Shifting laptops in Ubuntu

My laptop HDD was failing and getting worse. Time to get a new machine and shift. Here is the highly simplified, sanitised version of how I got everything working.

Install latest Ubuntu (finally shifted from 32 to 64 bit). Use https://etcher.io/ to make live USB – it Just Works™. Was given the option of retaining Windows 10 so gave it 84GB/2TB and kept rolling.

Sorted out Nvidia as per Ubuntu 16.04LTS extended display not working (see also https://www.youtube.com/watch?v=_36yNWw_07g)

  1. Run sudo apt-get purge nvidia-*
  2. Run sudo add-apt-repository ppa:graphics-drivers/ppa
  3. Run sudo apt-get update
  4. Run sudo apt-get install nvidia-364 (Check what’s the latest version… in my case nvidia-375 was the latest in the repos)

Reboot. Success?

Set up launcher etc so not sticky and icons the right size. Remove unneeded ones etc.

sudo apt install chromium-browser dh-make gftp gimp git idle3 imagemagick inkscape keepass2 mysql-server openjdk-9-jdk python3-matplotlib python3-nose python3-numpy python3-requests python-pil python-wxgtk3.0 python-wxversion p7zip-full scribus shotwell shutter sqlitebrowser synaptic thunderbird vim virtualbox virtualbox-guest-additions-iso vlc wine64-development hardinfo

I got SQLYog working. It was fun finding the icon. Run using wine sqlyog.exe etc rather than trying to double click exes.

For MySQL I copied everything over and sorted out permissions. The innodb tables wouldn’t run (tables apparently did not exist) but I didn’t actually need them given they were local copies of remote tables. So I just transferred databases over using SQLYog.

For my cron jobs I ran crontab -e on source laptop and stored results then ran same command on new machine and pasted them. How cool is that :-).

Installed newest version of eclipse. As always, some faffing around to get desktop file sorted out etc. E.g.

sudo ln -s /home/g/eclipse/java-oxygen/eclipse/eclipse /usr/bin/eclipse

Install pydev.

Brought across lots of documents of course, and .gftp, .idlerc, .matplotlib, .openshot, .shh. Plus Music, Videos, Pictures (including photos)

Added thunderbird and firefox to startup.

Changed LibreOffice icon style to Galaxy under Tools>Options>LibreOffice>View>Icon style.

Then run some backups.

Karmic MySQL Server Problem Fixed

The Karmic upgrade went really well but MySQL server seemed to die along the way. A temporary solution that worked for me was found here – http://ubuntuforums.org/showthread.php?t=1040786.

sudo apt-get remove --purge mysql-server-5.1
sudo apt-get autoremove
sudo rm -rf /etc/mysql
sudo apt-get install mysql-server-5.1

[update] This solution doesn’t persist after reboot (at present). Still useful if I need to test MySQL but not a proper answer.
[update] running
sudo /etc/init.d/mysql start
does the trick. Not sure why it isn’t happening automatically on startup.
[SOLUTION]
The mysql file was in /etc/rc3/, rc4, and rc5, but not rc2. Why? No idea at all. Anyway, a quick gksu nautilus and copying S19mysql (yours may have a different number) and pasting it from rc3 into rc2 later, then a reboot to be sure. Working as needed. NB a prefix of K means deactivate.

To see if mysql is running:

ps -ef | grep mysql

a single line response (your mysql client?) is not a good sign.

Here is on thread of interest: http://ubuntuforums.org/showthread.php?p=8320705

Here are the main resources I found useful:

http://brainwreckedtech.wordpress.com/2008/04/25/ubuntu-804-bug-with-mysql-and-apparmor/
http://case.oncle-tom.net/2008/05/04/modifier-emplacement-donnees-serveur-mysql/
http://www.hobbes.ch/2009/11/mysql-stopped-working-after-update/
http://ubuntuforums.org/archive/index.php/t-1305921.html
http://ubuntuforums.org/showthread.php?t=1322070
http://bugs.mysql.com/bug.php?id=47195
http://www.mail-archive.com/ubuntu-server-bugs@lists.ubuntu.com/msg16084.html
NB My AppArmor mask denied etc didn’t affect the working of the system so was a bit of a side-track in my case. Mainly affects you if you don’t use standard path locations.
http://en.opensuse.org/AppArmor_Geeks
http://www.vanimpe.eu/blog/2009/01/13/requested_mask-denied_mask-errors-on-ubuntu-with-mysql/

Slow query yet indexed – collation mismatch on index?

I had a simple query with 3 tables linked with one inner and one left join. Ran indescribably slowly – unless I changed the left join to an inner join (which I didn’t want to do). Yet all the links were indexed. EXPLAIN showed that the indexes were not being used. Why? Answer – one indexed field had a collation of utf8 and the other of latin1. Aligned that and everything worked nice and fast. Was there any indication from the output of EXPLAIN that this was the issue? No! Anyway, the syntax is DEFAULT CHARSET = latin1 or whatever after the ENGINE = part.

Reset MySQL password in Ubuntu

A fresh install didn’t seem to let me set/reset the password and for some reason I couldn’t get in. So thanks to: http://ubuntu.flowconsult.at/en/mysql-set-change-reset-root-password/ for the answer (which worked in Hardy):

1. Stop the MySQL Server

sudo /etc/init.d/mysql stop

2. Start the mysqld configuration.

sudo mysqld –skip-grant-tables &

3. Login to MySQL as root.

mysql -u root mysql

4. Replace YOURNEWPASSWORD with your new password!

UPDATE user SET Password=PASSWORD(‘YOURNEWPASSWORD’) WHERE User=’root’; FLUSH PRIVILEGES; exit;

Note: This method is not regarded as the securest way of resetting the password. However it works.

Kexi review (Access bruiser but not Access Killer yet)

After my disappointing experiences with OpenOffice Base last year I was worried that Kexi wouldn’t be much good. But it was – within its limits. The interface seemed excellent and intuitive and even though it wasn’t anywhere near as familiar to me as the Access interface (many thousands of hours) I found myself getting quite fast at it quite easily. My real issues relate to scripting. It was exciting to see that I could use python with Kexi as my scripting language but it is not clear yet how to add lots of sophisticated functionality e.g. when I update this field I want to check 3 other values and some data in the database and enable, disable some widgets and produce some messageboxes. This may well be easy (someone please correct me) but I don’t have lots of time to gamble on this if it is not ready. If I can get that sorted (including lots of good documentation on how to control and read from the widgets using python), I will take the next step and check out Kugar – the reporting part of Kexi. If all these parts work, and there is at least one decent book on Kexi in English, we might actually have an Access Killer. Kexi + MySQL + Kugar could be a winning combination for rapid application development – especially by non-programmers.

[update version 2.2 is released and looks promising – http://www.koffice.org/news/koffice-2-2-released/]

Sluggish MySQL because of ContentProtect

A program I wrote using python and MySQL ran much, much slower on my development computer than on the client’s computer. And their computer seemed about the same spec as mine. I upgraded from python 2.4 to 2.5. Nope. Was it the RAM. Nope – they had half a Gb to my 1Gb. Was it the hard drive? The L1 and L2 cache? Who could tell.

A friend (a very good friend) suggested it might be a MySQL setting. I was looking around in there when I noticed the named pipe alternative to the TCP/IP protocol. TCP/IP. Hmmmm I had sort of noticed the wireless network icon flickering in the taskbar when running MySQL. Could it be ….. ContentProtect!?

I reconfigured MySQL to not use TCP/IP:

connMySQL = MySQLdb.connect(host=maint.DB_HOST, user=maint.DB_USER,
passwd=maint.DB_PWD, db=maint.DB_DATABASE,
named_pipe=maint.NAMED_PIPE)

http://mysql-python.sourceforge.net/MySQLdb.html

… and the program suddenly flew! I could barely keep up with the screen output. Unfortunately my favourite MySQL manager, SQLyog, did not support named pipes.

http://www.webyog.com/forums//index.php?s=035d8d492234b760704ed30c35331bdd
&showtopic=2314&view=findpost&p=10917

But if I switched on support for TCP/IP the code relying on named pipes stopped working. I would get “error 2017: can’t open named pipe to host”.

The answer was to manually add one line to my my.ini file – namely:
enable-named-pipe

Now I could use the named pipes where possible, thus sidestepping the numbing effect of ContentProtect, while still being able to use tools that required or expected TCP/IP.

A useful reference is on:
http://dev.mysql.com/doc/refman/5.0/en/can-not-connect-to-server.html
#can-not-connect-to-server-on-windows

Collation in MySQL CONCAT function

This simple concat expression works in MySQL 4.1.21-standard:

select CONCAT(DATE_FORMAT(dtmTest,”%Y-%m-%d”), ” – “, strType)
FROM tbltest

but not in MySQL 5.0.45-community-nt:

Error Code : 1270
Illegal mix of collations (utf8_general_ci,COERCIBLE), (utf8_general_ci,COERCIBLE), (latin1_swedish_ci,IMPLICIT) for operation ‘concat’
(0 ms taken)

Solution: use COLLATE thus:

select CONCAT(DATE_FORMAT(dtmTest,”%Y-%m-%d”), ” – ” COLLATE utf8_general_ci, strType)
FROM tbltest;

This is relevant: http://dev.mysql.com/doc/refman/5.0/en/charset-collate.html
and the introduction of string repertoire could possibly make the problem go away in later versions?

http://dev.mysql.com/doc/refman/5.0/en/charset-repertoire.html

Running MySQL scripts (.sql files) from python

This should have been more obvious. All I wanted was to run a simple script e.g. myscript.sql in MySQL from python in Windows.

Some things worked fine from the DOS prompt but failed from within python (and RUN for that matter).

Here is the answer in the form of a simple function (NB to get your indentation right !):

def run_sql_script(scriptname):
    "Run a script in MySQL"
    import subprocess
    import time
    #this next line is too long for this blog but you will need it on one line to run
    args = "\"C:\\Program Files\\MySQL\\MySQL Server 5.0\\bin\\mysql.exe\" 
        -h%s -u%s -p%s --database=databasename < C:/Projects/projectname/3_scripts/%s" 
        % (DB_HOST, DB_USER, DB_PWD, scriptname)
    #print args
    child = subprocess.Popen(args=args, shell=True, executable="C:\\windows\\system32\\cmd.exe")
    #need to check whether finished or not every so often
    i = True
    elapsed = 0
    while i == True:
        time.sleep(10)
        elapsed = elapsed + 10
        if child.poll() == None:
            elapsed_mins = float(elapsed)/60
            print "%.2f minutes elapsed running %s" % (elapsed_mins, scriptname)
        else:
            print "Finished running script " + scriptname
            i = False

Key points: can't use call, must use full Popen and explicitly name the shell (and use it!).

NB scripts can run for a long time e.g. 30 minutes so it is a good idea to make the function keep the user informed.

If you want to kill it, open Task Manager and kill mysql.exe.

In Windows there are apparently some horrible compromises to be made when doing some simple things. See ...adventures-in-python-launching-subprocesses/

There is an alternative approach but it doesn't seem to ever end the subprocess:

#args = "cmd /k \"C:\\Program Files\\MySQL\\MySQL Server 5.0\\bin\\mysql.exe\" -h%s -u%s -p%s --database=databasename < C:\\Projects\\projectname\\test.sql" % (DB_HOST, DB_USER, DB_PWD) print args #http://www.realtechnews.com/posts/2777 re: the /k !!!!!