Shifting laptops in Ubuntu

My laptop HDD was failing and getting worse. Time to get a new machine and shift. Here is the highly simplified, sanitised version of how I got everything working.

Install latest Ubuntu (finally shifted from 32 to 64 bit). Use https://etcher.io/ to make live USB – it Just Works™. Was given the option of retaining Windows 10 so gave it 84GB/2TB and kept rolling.

Sorted out Nvidia as per Ubuntu 16.04LTS extended display not working (see also https://www.youtube.com/watch?v=_36yNWw_07g)

  1. Run sudo apt-get purge nvidia-*
  2. Run sudo add-apt-repository ppa:graphics-drivers/ppa
  3. Run sudo apt-get update
  4. Run sudo apt-get install nvidia-364 (Check what’s the latest version… in my case nvidia-375 was the latest in the repos)

Reboot. Success?

Set up launcher etc so not sticky and icons the right size. Remove unneeded ones etc.

sudo apt install chromium-browser dh-make gftp gimp git idle3 imagemagick inkscape keepass2 mysql-server openjdk-9-jdk python3-matplotlib python3-nose python3-numpy python3-requests python-pil python-wxgtk3.0 python-wxversion p7zip-full scribus shotwell shutter sqlitebrowser synaptic thunderbird vim virtualbox virtualbox-guest-additions-iso vlc wine64-development

I got SQLYog working. It was fun finding the icon. Run using wine sqlyog.exe etc rather than trying to double click exes.

For MySQL I copied everything over and sorted out permissions. The innodb tables wouldn’t run (tables apparently did not exist) but I didn’t actually need them given they were local copies of remote tables. So I just transferred databases over using SQLYog.

For my cron jobs I ran crontab -e on source laptop and stored results then ran same command on new machine and pasted them. How cool is that :-).

Installed newest version of eclipse. As always, some faffing around to get desktop file sorted out etc. E.g.

sudo ln -s /home/g/eclipse/java-oxygen/eclipse/eclipse /usr/bin/eclipse

Install pydev.

Brought across lots of documents of course, and .gftp, .idlerc, .matplotlib, .openshot, .shh. Plus Music, Videos, Pictures (including photos)

Added thunderbird and firefox to startup.

Changed LibreOffice icon style to Galaxy under Tools>Options>LibreOffice>View>Icon style.

Then run some backups.

WiFi changes – printer stops working

My parents-in-law changed over their WiFi and their router. So the wireless printer stopped working. The solution was simple – we just needed to connect the printer to the new WiFi using the latest ssid and key (password).

Being a Canon MG7160 the first step was to swipe left or right on the printer touchscreen at WLAN, then Wireless LAN Setup > Standard Setup > and the ssid. Enter passphrase. Now the printer can connect to the WiFi and you can print to it over the WiFi.

ImageMagick cache resources exhausted resolved

My sofastatistics application relies on ImageMagick to convert PDFs to PNGs. The sort of command run under the hood was:

convert -density 1200 -borderColor "#ff0000" -border 1x1 -fuzz 1% -trim "/home/g/projects/sofastats_proj/storage/img_processing/pdf2img_testing/KEEPME/raw_pdf.pdf" "/home/g/projects/sofastats_proj/storage/img_processing/pdf2img_testing/density_1200_fuzz_on_#ff0000.png"

Recently, commands like this stopped working properly on my development machine. They wouldn’t handle high resolutions (600dpi seemed to be the limit for the images I was handling) and it took a very long time to complete.

I finally worked out what was going on by running the same tests on different machines.

Seemingly modest differences in CPU specs can create massive differences in the time required to convert PDFs to PNGs. What takes 4 seconds on an i7 can take 71 seconds on an i5. And creating a 1200 dpi image might take 0.5 minutes on an i7 and 18.5 minutes on an i5. So the slowdown was because I had shifted from a fast desktop to a (more convenient but slower) laptop.

The second issue was the error message about cache resources exhausted. This happened on a range of fast and slow machines and the amount of RAM seemed irrelevant. Interestingly, the problem only occurred on Ubuntu 17.04 and not 16.10. The reason was the policy.xml settings in /etc/ImageMagick-6/. It seems the following was set too low:

I changed it to:

and it would successfully create high-resolution PNGs even if it took a long time.

Hmmm – now I change the disk setting back and I am still able to make the higher-resolution images, even after rebooting. WAT?!

One other note – settings in policy.xml cannot be loosened through arguments supplied to the convert program via the CLI – they can only be tightened. It looks like these changes are all about security concerns with the intention of preventing malicious resource starvation.

Some references:

Eclipse Neon with Pydev on Ubuntu 17.04

  1. Check whether 32 or 64 bit – System Settings > Details > Overview – then download from https://www.eclipse.org/downloads/
  2. Right click on eclipse-inst-linux32.tar.gz and select Extract Here
  3. cd into eclipse-installer/ then run ./eclipse-inst
  4. Choose top item (standard Java version of eclipse)
  5. Make desktop icon for launcher as per step 7 here – How to install Eclipse using its installer remembering to point to java-neon instead of java-mars etc. Drag file onto launcher and icon will add itself and be operational
  6. If you don’t see toolbars in eclipse modify exec line Exec=env SWT_GTK3=0 as per eclipse doesn’t work with ubuntu 16.04
  7. I also added pydev in the usual way using link to http://www.pydev.org/updates as per http://www.pydev.org/download.html

Right click to set as wallpaper in Ubuntu

I have lots of images in my Pictures folder that I wouldn’t want to use as wallpaper. So I like to keep a separate wallpapers folder. But selecting from that via the standard Unity interface is a bit of a pain. I like to look at all the images at a reasonably large scale and choose what I feel like at the time. The answer is in http://askubuntu.com/questions/808736/set-as-wallpaper-right-click-option-in-nautilus-does-not-work.

gsettings set org.gnome.settings-daemon.plugins.background active true
gsettings set org.gnome.desktop.background show-desktop-icons true
xdg-mime default nautilus-folder-handler.desktop inode/directory

Shotwell not copying when importing

I use Shotwell for managing my photos and generally it works really well as a replacement for f-spot on my Ubuntu system. My work flow is to put photos in a folder ready to import, import them into Shotwell, and then empty the folder. I discovered very early on that it is important to copy photos rather than accept the default of linking to them otherwise they disappear from Shotwell after the input folder is cleared.

Unfortunately, after an upgrade, Shotwell stopped presenting the option of copy vs link and I imported anyway. Big mistake ;-). It seems to be important to locate the input folder (if using a work flow of add to input->import->clear input) outside of the folder Shotwell is storing its Photos under. E.g. if Shotwell is putting its images under ~/Pictures then don’t have your input folder anywhere like ~/Pictures/images to import.

Fixing the problem was a bit complicated because deleting the imported photos from Shotwell, relocating the input folder so it was outside the Shotwell storage path, and reimporting the images didn’t actually copy them even when the copy option was selected. It seems that Shotwell treats the image as already having been imported and so only does a link. Looking at the source code (yay for open source) gave some useful hints about how Shotwell was actually working esp https://github.com/GNOME/shotwell/blob/809b4cdd20ec6ec795c51ec046999f6f5106c07a/src/BatchImport.vala#L1857

The solution was to delete all records from the PhotoTable table in ~/.local/share/shotwell/data/photo.db Sqlite database where the filename field started with the poorly-located input folder. Obviously I copied the database file first ;-). E.g.

DELETE FROM PhotoTable WHERE filename LIKE '/home/username/Pictures/0 images to import/%'

Reimporting worked as it should have after this step. “DB Browser for SQLite” is brilliant, BTW.

Shifting my git repo up a folder

My SOFA Statistics git repo was in a location that made sense at the time but became increasingly annoying. And I needed to restructure things anyway to prepare for snap packaging. Time for a shift.

Step 1 – shift existing .git folder and .gitignore
Step 2 – shift all the folders and content
Step 3 – git mv to relocate all folders and files relative to new .git location so git recognises that the files are the same but relocated i.e. will keep all the history

Dual Boot Ubuntu 16.04 on Win 10 Acer Aspire E15

The goal – to turn a Windows 10 laptop into a dual boot which retains Windows but makes Ubuntu 16.04 the primary OS. Work being performed on parents-in-law’s new laptop.

The solution overview:

  1. Make recovery disk (on USB)
  2. Shrink Windows partition from within Windows 10
  3. Downgrade BIOS
  4. Set supervisor password on BIOS
  5. Disable Secure Boot (but not UEFI)
  6. Make successful 64-bit, EFI-friendly USB
  7. Install Ubuntu alongside Windows 10
  8. Add Ubuntu efi files and change boot order so Ubuntu grub efi comes first
  9. Re-enable Secure Boot, enable F12 Boot Menu, remove supervisor password
  10. Set up Qualcomm Atheros QCA9377 Wireless Network Adapter.
  11. Misc

Make recovery disk (on USB)

Just in case. And you’ll need over 16GB so a 32GB USB should be right. Format as FAT32.

Open Windows, click on Windows button on screen (bottom left), All apps > Acer > Acer Recovery Management > Backup > Create Factory Default Backup. Tick “Back up system files to the recovery drive.”

Wait a long time while system backup prepared.

Shrink Windows partition from within Windows 10

From Windows button on screen > All apps > Windows Administrative Tools > Computer Management > Storage > Disk Management. Right-click on main NTFS partition and select Shrink. The empty space left over will be used by Ubuntu later.

Downgrade BIOS

This really matters. Without it I got black screen after getting USB startup disk to provide GRUB option. No amount of mucking around with nomodeset or noapic or noacpi helped. It was the BIOS!

http://community.acer.com/t5/E-and-M-Series-Laptops/Acer-Aspire-e5-573g-You-can-not-install-any-one-Linux/m-p/386281/highlight/true#M3080

I’ve repeated the content below in case the link ever disappears:

I figured out how to downgrade the bios.

Go to: http://us.acer.com/ac/en/US/content/drivers

Search by Product Model:

Aspire E5-573G (EDIT – or whatever you have)
Change boot order so Ubuntu grub efi comes first

Select the right OS and download a bios. In my case I downloaded 1.15.

Run the ZRT_115.exe.

It will fail.

But before you close the installer, go to C:\Users\name\AppData\Local\Temp\

Search for a folder (random letters).tmp

There should be a H2OFFT-W.exe and zrt.rd file in there.

Just copy this folder and close the failing install.

In that copied folder, edit the platform.ini file.

Before:

[BIOSVersionFormat]
;[nb4-version] +
BIOSVFEnable=1
VersionFormat=XN.NN

After:

[BIOSV[BIOSVersionFormat]
;[nb4-version] +
BIOSVFEnable=1
VersionFormat=XN.DDersionFormat]
;[nb4-version] +
BIOSVFEnable=1
VersionFormat=XN.DD

The VersionFormat value now has ‘XN.DD’ instead of ‘XN.NN’.

This will ignore the fact that 1.25 -> 1.15 is a downgrade.

Prepare for a reboot. I.E. close unnecessary applications. Because it’ll happen automatically after running the installer.

Run H2OFFT-W.exe.
Set supervisor password on BIOS
Upon reboot, you’ll see a bios installing progress bar.

After that is done, press F2 during startup to get to bios. The version should now be 1.15.

At this point I set a password, turned off UEFI, and swapped my hard drive out for a fresh SSD. Ubuntu finally installed.

Set supervisor password on BIOS

The laptop has InsydeH20 BIOS Rev. 5.0 – access this by pressing F2 quickly after bootup. Then move to Security > Set Supervisor Password e.g. ‘a’ (we’ll be removing this later so a short password for convenience makes sense in this use case).

It is necessary to set this password (on Acer machines at least) to alter UEFI settings.

Disable Secure Boot (but not UEFI)

In the BIOS move to the Boot section and disable Secure Boot.

Don’t disable UEFI – that may have been necessary a few years ago but it probably causes more problems that it solves now (assuming it works at all). See http://www.rodsbooks.com/linux-uefi/

Make successful 64-bit, EFI-friendly USB

Download 64-bit ISO image. 32-bit apparently won’t work with UEFI.

I had trouble with unetbootin and the Ubuntu Startup Disk Creator. So I used good old dd to make my startup USB drive. This is probably what solved the “Missing operating system” problem I was having.

Format USB as FAT32 (maybe using Gparted or the general Gnome disk utility Disks).

sudo dd if=”/home/g/Downloads/ubuntu-16.04-beta2-desktop-amd64.iso” of=/dev/sdd

Note — won’t necessarily be sdd. Open Disks, select USB DISK, look at Device setting e.g. /dev/sdc1.

Install Ubuntu alongside Windows 10

In BIOS under Boot change boot order so USB HDD (that’s the USB stick actually) comes above Windows Boot Manager. Insert startup USB and reboot. Choose to install Ubuntu alongside Windows. I needed to do this with an ethernet cable plugged in given wireless wasn’t working for Ubuntu out of the box.

Add Ubuntu efi files and change boot order so Ubuntu grub efi comes first

In the BIOS, Security > Select an UEFI file as trusted for executing. Approve all ubuntu efi files. HDD0 > > > all the .efi files. I name them ubuntuorignameefi so they are easy to identify correctly and reorder in the Boot priority order section e.g. grubx64.efi -> ubuntugrubx64efi. Then in Boot > Boot priority order raise the grubx64 entry to the top.

I had 4 files to set (unlike 3 in some docs I found) – namely: grubx64.efi, fwupx64.efi, shimx64.efi, and MokManager.efi.

Re-enable Secure Boot, enable F12 Boot Menu, remove supervisor password

All straight forward. Remover supervisor password by setting it to an empty string by entering orig password then Enter (to register current password), Enter (to submit empty string as password, Enter again to confirm.

Set up Qualcomm Atheros QCA9377 Wireless Network Adapter.

Identify wireless first:

lspci

03:00.0 Network controller: Qualcomm Atheros Device 0042 (rev 30)

Installing the required driver was explained here by @chili555. As another grateful person said “You rock!”.

http://ubuntuforums.org/showthread.php?t=2300861&page=11&p=13400551#post13400551

open a terminal and do:
Code:

sudo mkdir /lib/firmware/ath10k/QCA9377/
sudo mkdir /lib/firmware/ath10k/QCA9377/hw1.0

If it reports that the file already exists, that’s fine, just continue.

With a temporary working internet connection:

Code:

sudo apt-get install git
git clone https://github.com/kvalo/ath10k-firmware.git
cd ath10k-firmware/QCA9377/hw1.0
sudo cp board.bin /lib/firmware/ath10k/QCA9377/hw1.0
sudo cp firmware-5.bin_WLAN.TF.1.0-00267-1 /lib/firmware/ath10k/QCA9377/hw1.0/firmware-5.bin
sudo modprobe -r ath10k_pci
sudo modprobe ath10k_pci

Your wireless should be working; if not, try a reboot.

Misc

I installed the system load indicator (set to traditional colours with Processor, Network, Harddisk), VLC, Shotwell (loading all photos with copy as the option), and brought thunderbird and firefox data over from old computer (merely copying xxxxxxx.default folders and updating profiles.ini Profile > Path settings. Then setting up icons on launcher and we’re done!

Launchpad – Bazaar to Git

I’ve stored my SOFA Statistics code on launchpad since 2009 and used bazaar (bzr) to do it. But a lot has changed since then and I know use git on a daily basis in my job. So I’d much rather use git for SOFA. Fortunately that is now possible on launchpad.

I found https://help.launchpad.net/Code/Git to be useful apart from the migration instructions. These didn’t work for me. For example, I had no luck installing sudo apt-get install bzr-fastimportInstead I found https://design.canonical.com/2015/01/converting-projects-between-git-and-bazaar/.

Need ~/.bazaar/plugins

If plugins folder not there, cd ~/.bazaar; mkdir plugins

cd ~/.bazaar/plugins


bzr branch lp:bzr-fastimport fastimport

cd ~/projects/SOFA/sofastatistics/sofa.repo/sofa.main/

git init

bzr fast-export --plain . | git fast-import

gitk --all

YES! It’s all there

Archive .bzr in case

USER is launchpad-p-s in my case (yes – a strange choice which made sense at the time)

PROJECT is sofastatistics

So as per
[url "git+ssh://USER@git.launchpad.net/"]
insteadof = lp:

added the following to ~/.gitconfig

[url "git+ssh://launchpad-p-s@git.launchpad.net/"]
insteadof = lp:

Note – if not using lp: etc I had trouble with my ssh key – possibly something to do with confusion between user g and user launchpad-p-s.

I own my own project so to implement git remote add origin lp:PROJECT I ran:

git remote add origin lp:sofastatistics

Note: would only work if insteadof setting added to ~/.gitconfig as described earlier https://help.launchpad.net/Code/Git

Otherwise I would have to git remote add origin git+ssh://git.launchpad.net/sofastatistics

As per https://launchpad.net/PROJECT/+configure-code. I.e.

https://launchpad.net/sofastatistics/+configure-code

Confirmed by making a check folder then cloning the code in: git clone git://git.launchpad.net/~launchpad-p-s/sofastatistics

Canon MG7160 on Ubuntu broken by update – solution

Another problem with my parents-in-law’s printer – it looks like an update broke the printer on two separate machines (an ancient desktop running Lubuntu 14.04 and an Asus Eee PC netbook running Ubuntu 14.04). Yet when I took the printer home it instantly worked with the default drivers on my Ubuntu 15.10 system there, albeit with the colours a bit off (see http://p-s.co.nz/wordpress/printer-driver-for-canon-mg-7100-on-ubuntu/). I fixed this by downloading and unpacking the driver at http://support-th.canon-asia.com/contents/TH/EN/0100551202.html. Then, from the terminal prompt I ran ./install.sh, answered some questions e.g. what I wanted the printer called, and voila – I had a working printer with the correct colours using Driver: CNMG7100.PPD.

Some things to note about setting up the printer:

1) the touchscreen on the top of the printer lets you swipe left and right to see more icons. The WiFi icon is where you can tell your printer the SSID and password/passphrase for you secured WiFi access point.

2) another setting lets you change the name of the printer e.g. to something user-friendly like MG7160.

3) on some routers (192.168.1.254 in the case of the one my parents-in-law were using as opposed to 192.168.178.1 on mine and 192.168.1.1 or 192.168.0.1 on many IIRC) you will be able to see the printer connected. Having given it a useful name in step 2) made this easier to spot. I could see it on my own router but not on my parents-in-law’s router for some reason but there did seem to be some connection. After I renamed the printer it was detected on the PC as a network printer under the new name so something was getting across.

BTW the error messages I was getting when trying to run the printer on my parents-in-law’s computers were inconsistent. More on that when I update this post from their place.