gDesklets and Quote of the Day

For Ubuntu users, you install gDesklets using Synaptic. The desklets are of variable quality and value but some of the clocks are good, and perhaps the weather could be of some interest (if you can get it to work for your city ๐Ÿ˜‰ ). The keeper for me is Quote of the Day. Go to:

http://gdesklets.de/index.php?q=desklet/view/181&destination=desklet%2Fview%2F181
and download the tar.gz file. Install from within gDesklets, selecting the tar.gz file downloaded.

I put two Quote of the Day desklets on my desktop – one for Bible verses and one for general quotes. To add a desklet, double click it in the list and drag and drop it into place on your desktop. To configure or move a desklet, right click on it and make your selection.

To get gDesklets to automatically appear when starting up, go to System> Preferences> Sessions and click on Add. The command is simply gdesklets start.

Broadcom wireless under Hardy (Alpha 5)

Problem: wireless not working even using bcm43xx fwcutter trick used on previous occasions successfully.

Looked in system log – System>Administration>SystemLog kern.log
ERROR: Firmware file “b43/ucode5.fw” not found or load failed.
Huh? That’s not even a file made by bcm43xx.

Initial diagnostics:
lspci | grep Network
returned:
02:0b.0 Network controller: Broadcom Corporation BCM4306 802.11b/g Wireless LAN Controller (rev 03)

Solution:
Don’t use BCM43xx any more – doesn’t work. Use b43-fwcutter. See http://linuxwireless.org/en/users/Drivers/b43#fw-b43legacy

NB don’t bother using make etc on the tarball – just grab the suitable deb file from the hardy util site. Get by searching for Hardy util b43. Obviously, if I had access to the internet I could use synaptic or apt-get but no wireless = no internet (see Catch 22 ;-)).

then gksu b43-fwcutter -w /lib/firmware ~Desktop/wl_apsta-3.130.20.0.o
NB gksu not sudo when a graphical application

Working on the assumption that I can use the legacy v3 file (incorrect – see below).

Still a problem – there is now a folder called /lib/firmware/b43legacy
So tried:
sudo nautilus /lib/firmware/b43legacy and it contains ucode5.fw etc
Shame the folder has legacy as part of its name
Renamed the folder to b43 in a shameless attempt to get the system going

New issue (possibly progress though):
b43-phy0 ERROR: YOUR FIRMWARE IS TOO OLD. Firmware from binary drivers older than version 4.x is unsupported. You must upgrade your firmware files.

Hmmm – deleted all old bcm43xx firmware files from main /lib/firmware folder.
sudo rm -r /lib/firmware/b43legacy

Rebooted.

Nope.

Ah – download v4 wl_apsta.o file
wget http://downloads.openwrt.org/sources/broadcom-wl-4.80.53.0.tar.bz2

get wl_apsta.o from inside that file and put on desktop.

then sudo b43-fwcutter -w /lib/firmware ~Desktop/wl_apsta.o

Right – everything was automatically put it in b43 subfolder, not b43legacy. Starting to click into place. As is often the case, it all makes total sense backwards ๐Ÿ˜‰ .

Looking good – can see my wireless network showing with a decent signal. Supply password. Hmmm waiting for network key.

To be continued …

[Easter 2008] – use latest, v4 wl_apsta.o
sudo b43-fwcutter -w /lib/firmware ~Desktop/wl_apsta.o
Hmm still waiting for network key.
Even tried unsecured neighbourhood wireless network. Still no success.

And continued …

[Hardy Heron RC]
Waiting for network key (sigh …)
At least dual/multiple monitor support is getting much better.
Intrepid Ibex
Hmmm try v11 rather than v8 of b43-fwcutter deb (google b43 deb)
Nope

Kexi review (Access bruiser but not Access Killer yet)

After my disappointing experiences with OpenOffice Base last year I was worried that Kexi wouldn’t be much good. But it was – within its limits. The interface seemed excellent and intuitive and even though it wasn’t anywhere near as familiar to me as the Access interface (many thousands of hours) I found myself getting quite fast at it quite easily. My real issues relate to scripting. It was exciting to see that I could use python with Kexi as my scripting language but it is not clear yet how to add lots of sophisticated functionality e.g. when I update this field I want to check 3 other values and some data in the database and enable, disable some widgets and produce some messageboxes. This may well be easy (someone please correct me) but I don’t have lots of time to gamble on this if it is not ready. If I can get that sorted (including lots of good documentation on how to control and read from the widgets using python), I will take the next step and check out Kugar – the reporting part of Kexi. If all these parts work, and there is at least one decent book on Kexi in English, we might actually have an Access Killer. Kexi + MySQL + Kugar could be a winning combination for rapid application development – especially by non-programmers.

[update version 2.2 is released and looks promising – http://www.koffice.org/news/koffice-2-2-released/]

PyInstaller Round 2

Round 1 was nearly 18 months ago (PyInstaller1.2) and enabled me to successfully deploy a GUI application as a folder (although the XP button styles weren’t quite right – solved below).

Round 2 lasted about 8 hours and involved pyInstaller 1.3 on XP with python 2.5, wxPython 2.8.7.1, and win32com 2.1.0. If I’d had these tips at the beginning it would have taken 20 minutes max ๐Ÿ™ :

  1. Don’t name any folders “python”. It shouldn’t matter and usually it doesn’t but sometimes it does. Use mypy or something similar instead. That could have saved 6 hours right there ๐Ÿ˜‰ . If a module was referred to as python.msaccess, for example, the python part would be treated as a module and expected to have an __init__ method. NB everything worked fine except when it was processed into an executable by pyinstaller.
  2. When testing the build process of a spec file, set console to True (or 1) and debug to True (or 1).
  3. When running the build process from a batch file, add the command
    pause
    as the final line. Then you can see all the errors, if any, and have a chance at fixing them. Can also add something like raw_input(“Hit Enter to continue”) at the end of Config.py etc to ensure files like upx are configured successfully.
  4. To identify problems with the executable, run it from a batch file, and include pause as the final command on its own line. NB make the exe with debug=True and Console=True for these steps (revert when issues fixed).
  5. Set console=False (or 0) so that XP buttons look more attractive than the older, rectangular form.
    Rounded XP buttons
                     vs
    Rectangular XP buttons

    XP and wxPython and XP buttons etc
  6. If doing a single file executable, remember to add a.binaries after a.scripts
    a.scripts, a.binaries,
    and set exclude_binaries=False
  7. It really is quite easy to edit a spec file – it is just python after all.
  8. Scripts should be without comma separation and with single backslashes in the makespec process (Batch file), and with commas and double backslashes in the python spec file.
  9. Icon images etc can be kept in the same folder as the executable for ease of portability. Inside the script they should not have an absolute path.
  10. Start with a simple HelloWorld test as per the very helpful http://www.thescripts.com/forum/thread579554.html to check all systems are working before tackling a more complex, real-world example which will require getting down and dirty with the spec document.
  11. python “…Build.py” “…..” won’t work on my system – need “C:\Python25\python.exe” “etc …”

There is excellent documentation available at PyInstaller Manual

UPDATE: got a mysterious error on a script that worked well until it was processed by pyinstaller. Without all the gruesome details, it was because I had the wxPython application set to not redirect its output AND I had some print statements tucked away in some code. At some point (when the printed output reached 4096 bytes?) up popped an [Error 9] Bad file descriptor error. In future, diagnose by making an output.txt file and setting redirect to true. What is in the output file?

UPDATE: Use the -w parameter to skip the DOS window during execution.

UPDATE: Pyinstaller 1.5 with Python 2.6 (Round 3)

Sluggish MySQL because of ContentProtect

A program I wrote using python and MySQL ran much, much slower on my development computer than on the client’s computer. And their computer seemed about the same spec as mine. I upgraded from python 2.4 to 2.5. Nope. Was it the RAM. Nope – they had half a Gb to my 1Gb. Was it the hard drive? The L1 and L2 cache? Who could tell.

A friend (a very good friend) suggested it might be a MySQL setting. I was looking around in there when I noticed the named pipe alternative to the TCP/IP protocol. TCP/IP. Hmmmm I had sort of noticed the wireless network icon flickering in the taskbar when running MySQL. Could it be ….. ContentProtect!?

I reconfigured MySQL to not use TCP/IP:

connMySQL = MySQLdb.connect(host=maint.DB_HOST, user=maint.DB_USER,
passwd=maint.DB_PWD, db=maint.DB_DATABASE,
named_pipe=maint.NAMED_PIPE)

http://mysql-python.sourceforge.net/MySQLdb.html

… and the program suddenly flew! I could barely keep up with the screen output. Unfortunately my favourite MySQL manager, SQLyog, did not support named pipes.

http://www.webyog.com/forums//index.php?s=035d8d492234b760704ed30c35331bdd
&showtopic=2314&view=findpost&p=10917

But if I switched on support for TCP/IP the code relying on named pipes stopped working. I would get “error 2017: can’t open named pipe to host”.

The answer was to manually add one line to my my.ini file – namely:
enable-named-pipe

Now I could use the named pipes where possible, thus sidestepping the numbing effect of ContentProtect, while still being able to use tools that required or expected TCP/IP.

A useful reference is on:
http://dev.mysql.com/doc/refman/5.0/en/can-not-connect-to-server.html
#can-not-connect-to-server-on-windows

Collation in MySQL CONCAT function

This simple concat expression works in MySQL 4.1.21-standard:

select CONCAT(DATE_FORMAT(dtmTest,”%Y-%m-%d”), ” – “, strType)
FROM tbltest

but not in MySQL 5.0.45-community-nt:

Error Code : 1270
Illegal mix of collations (utf8_general_ci,COERCIBLE), (utf8_general_ci,COERCIBLE), (latin1_swedish_ci,IMPLICIT) for operation ‘concat’
(0 ms taken)

Solution: use COLLATE thus:

select CONCAT(DATE_FORMAT(dtmTest,”%Y-%m-%d”), ” – ” COLLATE utf8_general_ci, strType)
FROM tbltest;

This is relevant: http://dev.mysql.com/doc/refman/5.0/en/charset-collate.html
and the introduction of string repertoire could possibly make the problem go away in later versions?

http://dev.mysql.com/doc/refman/5.0/en/charset-repertoire.html

Running MySQL scripts (.sql files) from python

This should have been more obvious. All I wanted was to run a simple script e.g. myscript.sql in MySQL from python in Windows.

Some things worked fine from the DOS prompt but failed from within python (and RUN for that matter).

Here is the answer in the form of a simple function (NB to get your indentation right !):

def run_sql_script(scriptname):
    "Run a script in MySQL"
    import subprocess
    import time
    #this next line is too long for this blog but you will need it on one line to run
    args = "\"C:\\Program Files\\MySQL\\MySQL Server 5.0\\bin\\mysql.exe\" 
        -h%s -u%s -p%s --database=databasename < C:/Projects/projectname/3_scripts/%s" 
        % (DB_HOST, DB_USER, DB_PWD, scriptname)
    #print args
    child = subprocess.Popen(args=args, shell=True, executable="C:\\windows\\system32\\cmd.exe")
    #need to check whether finished or not every so often
    i = True
    elapsed = 0
    while i == True:
        time.sleep(10)
        elapsed = elapsed + 10
        if child.poll() == None:
            elapsed_mins = float(elapsed)/60
            print "%.2f minutes elapsed running %s" % (elapsed_mins, scriptname)
        else:
            print "Finished running script " + scriptname
            i = False

Key points: can't use call, must use full Popen and explicitly name the shell (and use it!).

NB scripts can run for a long time e.g. 30 minutes so it is a good idea to make the function keep the user informed.

If you want to kill it, open Task Manager and kill mysql.exe.

In Windows there are apparently some horrible compromises to be made when doing some simple things. See ...adventures-in-python-launching-subprocesses/

There is an alternative approach but it doesn't seem to ever end the subprocess:

#args = "cmd /k \"C:\\Program Files\\MySQL\\MySQL Server 5.0\\bin\\mysql.exe\" -h%s -u%s -p%s --database=databasename < C:\\Projects\\projectname\\test.sql" % (DB_HOST, DB_USER, DB_PWD) print args #http://www.realtechnews.com/posts/2777 re: the /k !!!!!

Real Interoperability

A recent Computerworld article “Microsoft loses appeal against EU antitrust ruling” contains some very good news for IT consumers everywhere. OK it took legal force, a massive fine (รขโ€šยฌ497 million), and it was resisted every step of the way, but Microsoft is finally going to enable genuine interoperability.
http://computerworld.co.nz/news.nsf/news/53A9FEF5880B1C7DCC25735900739D2F?
opendocument&utm_source=topnews&utm_medium=email&utm_campaign=topnews

How is this good? It means competitors to Microsoft finally get a fair go. Monopolies are bad for consumers and bad for innovation. It’s as simple as that.

From Computerword –

Carlo Piana, FSFE’s legal counsel, described the court ruling as “a milestone for competition. It puts an end to the notion that deliberate obfuscation of standards and designed lock-in is an acceptable business model, and forces Microsoft back into competing on the grounds of software technology.”

Also check out:

http://blogs.cnet.com/8301-13505_1-9779529-16.html?part=rss&tag=feed
&subj=TheOpenRoad

http://commandline.org.uk/more/microsoft/
eu-microsoft-decision-upheld-2007-09-17-11-00.html?showcomments=yes

This is especially good for Samba:

The Court of First Instance’s judgement, like the commission’s before it, sees Samba as the means for competition, in the Work group server space (i.e. file servers, print servers, etc). All potential competitors to Microsoft are using Samba, (the commissions own research found that 98% of competing products in this space use Samba), so it is good that the commission and the CFI are keen to get the documentation from Microsoft in a form that open source projects such as samba can use.

http://yro.slashdot.org/comments.pl?threshold=5&mode=thread&
commentsort=0&op=Change&sid=299827

Extracting text from PDFs using python and pdftotext

The answer was reasonably simple but it was very gruelling to obtain ;-). Firstly, the false leads:

1) Prescript proved to be an out-of-date, unsupported waste of time.

2) Ghostscript has never had much emphasis on user-friendliness or documentation. Was hoping to use its pdf2ascii functionality. Can’t remember precisely what happened but I think it only generated error messages for me.

3) pyPdf looks promising (the text extract functionality is still quite recent) but it didn’t get the text in the correct order – should probably revisit it later:

import pyPdf
"""http://pybrary.net/pyPdf/"""

def getPDFContent(path):
    content = ""
    # Load PDF into pyPDF
    pdf = pyPdf.PdfFileReader(file(path, "rb"))
    # Iterate pages
    for i in range(0, pdf.getNumPages()):
        # Extract text from page and add to content
        content += pdf.getPage(i).extractText() + "\n"
    # Collapse whitespace
    #content = " ".join(content.replace("\xa0", " ").strip().split())
    return content

print getPDFContent("pdfs/test.pdf")

—-

But I repeat – watch this option for the future. The developer is right onto it, as can be seen from the comment for the extractText method from the pdf.py module:

# Locate all text drawing commands, in the order they are provided in the
# content stream, and extract the text. This works well for some PDF
# files, but poorly for others, depending on the generator used. This will
# be refined in the future. Do not rely on the order of text coming out of
# this function, as it will change if this function is made more
# sophisticated.
#

# Stability: Added in v1.7, will exist for all future v1.x releases. May
# be overhauled to provide more ordered text in the future.
# @return a string object

http://pybrary.net/pyPdf/

4) pdftotext – bingo

Install pdftotext (a breeze in Ubuntu via Synaptic). In Windows refer to the brilliant, user-friendly documentation of Jeff Porter www.ire.org/training/nettour/pdf/PDFTOTEXT.pdf for step-by-step instructions.

http://www.foolabs.com/xpdf/download.html

pdftotext is part of XPDF – “Xpdf is an open source viewer for Portable Document Format (PDF) files. (These are also sometimes also called ‘Acrobat’ files, from the name of Adobe’s PDF software.) The Xpdf project also includes a PDF text extractor, PDF-to-PostScript converter, and various other utilities.

Xpdf runs under the X Window System on UNIX, VMS, and OS/2. The non-X components (pdftops, pdftotext, etc.) also run on Win32 systems and should run on pretty much any system with a decent C++ compiler. ” http://www.foolabs.com/xpdf/about.html

XPDF is GPL2

The python code is barely there but you can see the possibilities:

import os
os.system(“C:\\ … xpdf\\pdftotext -layout C:\\ … xpdf\\test.pdf”)
raw_input(“Finished”)

The text came out in the correct order thanks to the -format option.

DocuPrint 525A running on Ubuntu

[Updated Oct 2008]
Get printer driver (as RPM – need to convert to DEB for installation)
http://www.fujixerox.com.au/support/downloaddriver?productId=307&operatingSystemCode=Linux
or
http://www.xeroxprinters.co.nz/displayDvr.do?cid=8&langid=en&lpdtid=89&dvrOS=Linux&dvrPdtID=307
or
http://www.fujixerox.com.au/localDriverDownload.do

dpc525a_linux_.0.0.tar.zip

Extract Fuji_Xerox-DocuPrint_C525_A_AP-1.0-1.i386.rpm

Install Alien from synaptic if possible

Otherwise

alien_8.68_all.deb

http://www.howtoforge.com/converting_rpm_to_deb_with_alien
or
http://ubuntu.wordpress.com/2005/09/23/installing-using-an-rpm-file/

cd Desktop
sudo alien -k Fuji_Xerox-DocuPrint_C525_A_AP-1.0-1.i386.rpm
(use tab completion so no typos in file name – nice touch ;-))

will hopefully make fuji-xerox-docuprint-c525-a-ap_1.0-2_i386.deb on Desktop.

install deb

After installing deb,

System>Admin>Printing New etc

NB appears under FX

Older details – probably redundant:
usr/share/cups/model/FujiXerox/en/FX_DocuPrint_C525_A_AP.ppd
is the driver to use (may need to manually find – after which FX will appear in Printer list)

NB if you have problems printing longer documents (the job always shifts from Processing to Held) check out my solution at http://p-s.co.nz/wordpress/?p=227.

NB appears under FX