Shifting laptops in Ubuntu

My laptop HDD was failing and getting worse. Time to get a new machine and shift. Here is the highly simplified, sanitised version of how I got everything working.

Install latest Ubuntu (finally shifted from 32 to 64 bit). Use https://etcher.io/ to make live USB – it Just Works™. Was given the option of retaining Windows 10 so gave it 84GB/2TB and kept rolling.

Sorted out Nvidia as per Ubuntu 16.04LTS extended display not working (see also https://www.youtube.com/watch?v=_36yNWw_07g)

  1. Run sudo apt-get purge nvidia-*
  2. Run sudo add-apt-repository ppa:graphics-drivers/ppa
  3. Run sudo apt-get update
  4. Run sudo apt-get install nvidia-364 (Check what’s the latest version… in my case nvidia-375 was the latest in the repos)

Reboot. Success?

Set up launcher etc so not sticky and icons the right size. Remove unneeded ones etc.

sudo apt install chromium-browser dh-make gftp gimp git idle3 imagemagick inkscape keepass2 mysql-server openjdk-9-jdk python3-matplotlib python3-nose python3-numpy python3-requests python-pil python-wxgtk3.0 python-wxversion p7zip-full scribus shotwell shutter sqlitebrowser synaptic thunderbird vim virtualbox virtualbox-guest-additions-iso vlc wine64-development hardinfo

I got SQLYog working. It was fun finding the icon. Run using wine sqlyog.exe etc rather than trying to double click exes.

For MySQL I copied everything over and sorted out permissions. The innodb tables wouldn’t run (tables apparently did not exist) but I didn’t actually need them given they were local copies of remote tables. So I just transferred databases over using SQLYog.

For my cron jobs I ran crontab -e on source laptop and stored results then ran same command on new machine and pasted them. How cool is that :-).

Installed newest version of eclipse. As always, some faffing around to get desktop file sorted out etc. E.g.

sudo ln -s /home/g/eclipse/java-oxygen/eclipse/eclipse /usr/bin/eclipse

Install pydev.

Brought across lots of documents of course, and .gftp, .idlerc, .matplotlib, .openshot, .shh. Plus Music, Videos, Pictures (including photos)

Added thunderbird and firefox to startup.

Changed LibreOffice icon style to Galaxy under Tools>Options>LibreOffice>View>Icon style.

Then run some backups.

ImageMagick cache resources exhausted resolved

My sofastatistics application relies on ImageMagick to convert PDFs to PNGs. The sort of command run under the hood was:

convert -density 1200 -borderColor "#ff0000" -border 1x1 -fuzz 1% -trim "/home/g/projects/sofastats_proj/storage/img_processing/pdf2img_testing/KEEPME/raw_pdf.pdf" "/home/g/projects/sofastats_proj/storage/img_processing/pdf2img_testing/density_1200_fuzz_on_#ff0000.png"

Recently, commands like this stopped working properly on my development machine. They wouldn’t handle high resolutions (600dpi seemed to be the limit for the images I was handling) and it took a very long time to complete.

I finally worked out what was going on by running the same tests on different machines.

Seemingly modest differences in CPU specs can create massive differences in the time required to convert PDFs to PNGs. What takes 4 seconds on an i7 can take 71 seconds on an i5. And creating a 1200 dpi image might take 0.5 minutes on an i7 and 18.5 minutes on an i5. So the slowdown was because I had shifted from a fast desktop to a (more convenient but slower) laptop.

The second issue was the error message about cache resources exhausted. This happened on a range of fast and slow machines and the amount of RAM seemed irrelevant. Interestingly, the problem only occurred on Ubuntu 17.04 and not 16.10. The reason was the policy.xml settings in /etc/ImageMagick-6/. It seems the following was set too low:

<policy domain="resource" name="disk" value="1GiB">
I changed it to:
<policy domain="resource" name="disk" value="10GiB"/>
and it would successfully create high-resolution PNGs even if it took a long time.

Hmmm – now I change the disk setting back and I am still able to make the higher-resolution images, even after rebooting. WAT?!

One other note – settings in policy.xml cannot be loosened through arguments supplied to the convert program via the CLI – they can only be tightened. It looks like these changes are all about security concerns with the intention of preventing malicious resource starvation.

Some references:

Eclipse Neon with Pydev on Ubuntu 17.04

  1. Check whether 32 or 64 bit – System Settings > Details > Overview – then download from https://www.eclipse.org/downloads/
  2. Right click on eclipse-inst-linux32.tar.gz and select Extract Here
  3. cd into eclipse-installer/ then run ./eclipse-inst
  4. Choose top item (standard Java version of eclipse)
  5. Make desktop icon for launcher as per step 7 here – How to install Eclipse using its installer remembering to point to java-neon instead of java-mars etc. Drag file onto launcher and icon will add itself and be operational
  6. If you don’t see toolbars in eclipse modify exec line Exec=env SWT_GTK3=0 as per eclipse doesn’t work with ubuntu 16.04
  7. I also added pydev in the usual way using link to http://www.pydev.org/updates as per http://www.pydev.org/download.html

Right click to set as wallpaper in Ubuntu

I have lots of images in my Pictures folder that I wouldn’t want to use as wallpaper. So I like to keep a separate wallpapers folder. But selecting from that via the standard Unity interface is a bit of a pain. I like to look at all the images at a reasonably large scale and choose what I feel like at the time. The answer is in http://askubuntu.com/questions/808736/set-as-wallpaper-right-click-option-in-nautilus-does-not-work.

gsettings set org.gnome.settings-daemon.plugins.background active true
gsettings set org.gnome.desktop.background show-desktop-icons true
xdg-mime default nautilus-folder-handler.desktop inode/directory

Dual Boot Ubuntu 16.04 on Win 10 Acer Aspire E15

The goal – to turn a Windows 10 laptop into a dual boot which retains Windows but makes Ubuntu 16.04 the primary OS. Work being performed on parents-in-law’s new laptop.

The solution overview:

  1. Make recovery disk (on USB)
  2. Shrink Windows partition from within Windows 10
  3. Downgrade BIOS
  4. Set supervisor password on BIOS
  5. Disable Secure Boot (but not UEFI)
  6. Make successful 64-bit, EFI-friendly USB
  7. Install Ubuntu alongside Windows 10
  8. Add Ubuntu efi files and change boot order so Ubuntu grub efi comes first
  9. Re-enable Secure Boot, enable F12 Boot Menu, remove supervisor password
  10. Set up Qualcomm Atheros QCA9377 Wireless Network Adapter.
  11. Misc

Make recovery disk (on USB)

Just in case. And you’ll need over 16GB so a 32GB USB should be right. Format as FAT32.

Open Windows, click on Windows button on screen (bottom left), All apps > Acer > Acer Recovery Management > Backup > Create Factory Default Backup. Tick “Back up system files to the recovery drive.”

Wait a long time while system backup prepared.

Shrink Windows partition from within Windows 10

From Windows button on screen > All apps > Windows Administrative Tools > Computer Management > Storage > Disk Management. Right-click on main NTFS partition and select Shrink. The empty space left over will be used by Ubuntu later.

Downgrade BIOS

This really matters. Without it I got black screen after getting USB startup disk to provide GRUB option. No amount of mucking around with nomodeset or noapic or noacpi helped. It was the BIOS!

http://community.acer.com/t5/E-and-M-Series-Laptops/Acer-Aspire-e5-573g-You-can-not-install-any-one-Linux/m-p/386281/highlight/true#M3080

I’ve repeated the content below in case the link ever disappears:

I figured out how to downgrade the bios.

Go to: http://us.acer.com/ac/en/US/content/drivers

Search by Product Model:

Aspire E5-573G (EDIT – or whatever you have)
Change boot order so Ubuntu grub efi comes first

Select the right OS and download a bios. In my case I downloaded 1.15.

Run the ZRT_115.exe.

It will fail.

But before you close the installer, go to C:\Users\name\AppData\Local\Temp\

Search for a folder (random letters).tmp

There should be a H2OFFT-W.exe and zrt.rd file in there.

Just copy this folder and close the failing install.

In that copied folder, edit the platform.ini file.

Before:

[BIOSVersionFormat]
;[nb4-version] +
BIOSVFEnable=1
VersionFormat=XN.NN

After:

[BIOSV[BIOSVersionFormat]
;[nb4-version] +
BIOSVFEnable=1
VersionFormat=XN.DDersionFormat]
;[nb4-version] +
BIOSVFEnable=1
VersionFormat=XN.DD

The VersionFormat value now has ‘XN.DD’ instead of ‘XN.NN’.

This will ignore the fact that 1.25 -> 1.15 is a downgrade.

Prepare for a reboot. I.E. close unnecessary applications. Because it’ll happen automatically after running the installer.

Run H2OFFT-W.exe.
Set supervisor password on BIOS
Upon reboot, you’ll see a bios installing progress bar.

After that is done, press F2 during startup to get to bios. The version should now be 1.15.

At this point I set a password, turned off UEFI, and swapped my hard drive out for a fresh SSD. Ubuntu finally installed.

Set supervisor password on BIOS

The laptop has InsydeH20 BIOS Rev. 5.0 – access this by pressing F2 quickly after bootup. Then move to Security > Set Supervisor Password e.g. ‘a’ (we’ll be removing this later so a short password for convenience makes sense in this use case).

It is necessary to set this password (on Acer machines at least) to alter UEFI settings.

Disable Secure Boot (but not UEFI)

In the BIOS move to the Boot section and disable Secure Boot.

Don’t disable UEFI – that may have been necessary a few years ago but it probably causes more problems that it solves now (assuming it works at all). See http://www.rodsbooks.com/linux-uefi/

Make successful 64-bit, EFI-friendly USB

Download 64-bit ISO image. 32-bit apparently won’t work with UEFI.

I had trouble with unetbootin and the Ubuntu Startup Disk Creator. So I used good old dd to make my startup USB drive. This is probably what solved the “Missing operating system” problem I was having.

Format USB as FAT32 (maybe using Gparted or the general Gnome disk utility Disks).

sudo dd if=”/home/g/Downloads/ubuntu-16.04-beta2-desktop-amd64.iso” of=/dev/sdd

Note — won’t necessarily be sdd. Open Disks, select USB DISK, look at Device setting e.g. /dev/sdc1.

Install Ubuntu alongside Windows 10

In BIOS under Boot change boot order so USB HDD (that’s the USB stick actually) comes above Windows Boot Manager. Insert startup USB and reboot. Choose to install Ubuntu alongside Windows. I needed to do this with an ethernet cable plugged in given wireless wasn’t working for Ubuntu out of the box.

Add Ubuntu efi files and change boot order so Ubuntu grub efi comes first

In the BIOS, Security > Select an UEFI file as trusted for executing. Approve all ubuntu efi files. HDD0 > > > all the .efi files. I name them ubuntuorignameefi so they are easy to identify correctly and reorder in the Boot priority order section e.g. grubx64.efi -> ubuntugrubx64efi. Then in Boot > Boot priority order raise the grubx64 entry to the top.

I had 4 files to set (unlike 3 in some docs I found) – namely: grubx64.efi, fwupx64.efi, shimx64.efi, and MokManager.efi.

Re-enable Secure Boot, enable F12 Boot Menu, remove supervisor password

All straight forward. Remover supervisor password by setting it to an empty string by entering orig password then Enter (to register current password), Enter (to submit empty string as password, Enter again to confirm.

Set up Qualcomm Atheros QCA9377 Wireless Network Adapter.

Identify wireless first:

lspci

03:00.0 Network controller: Qualcomm Atheros Device 0042 (rev 30)

Installing the required driver was explained here by @chili555. As another grateful person said “You rock!”.

http://ubuntuforums.org/showthread.php?t=2300861&page=11&p=13400551#post13400551

open a terminal and do:
Code:

sudo mkdir /lib/firmware/ath10k/QCA9377/
sudo mkdir /lib/firmware/ath10k/QCA9377/hw1.0

If it reports that the file already exists, that’s fine, just continue.

With a temporary working internet connection:

Code:

sudo apt-get install git
git clone https://github.com/kvalo/ath10k-firmware.git
cd ath10k-firmware/QCA9377/hw1.0
sudo cp board.bin /lib/firmware/ath10k/QCA9377/hw1.0
sudo cp firmware-5.bin_WLAN.TF.1.0-00267-1 /lib/firmware/ath10k/QCA9377/hw1.0/firmware-5.bin
sudo modprobe -r ath10k_pci
sudo modprobe ath10k_pci

Your wireless should be working; if not, try a reboot.

Misc

I installed the system load indicator (set to traditional colours with Processor, Network, Harddisk), VLC, Shotwell (loading all photos with copy as the option), and brought thunderbird and firefox data over from old computer (merely copying xxxxxxx.default folders and updating profiles.ini Profile > Path settings. Then setting up icons on launcher and we’re done!

Canon MG7160 on Ubuntu broken by update – solution

Another problem with my parents-in-law’s printer – it looks like an update broke the printer on two separate machines (an ancient desktop running Lubuntu 14.04 and an Asus Eee PC netbook running Ubuntu 14.04). Yet when I took the printer home it instantly worked with the default drivers on my Ubuntu 15.10 system there, albeit with the colours a bit off (see http://p-s.co.nz/wordpress/printer-driver-for-canon-mg-7100-on-ubuntu/). I fixed this by downloading and unpacking the driver at http://support-th.canon-asia.com/contents/TH/EN/0100551202.html. Then, from the terminal prompt I ran ./install.sh, answered some questions e.g. what I wanted the printer called, and voila – I had a working printer with the correct colours using Driver: CNMG7100.PPD.

Some things to note about setting up the printer:

1) the touchscreen on the top of the printer lets you swipe left and right to see more icons. The WiFi icon is where you can tell your printer the SSID and password/passphrase for you secured WiFi access point.

2) another setting lets you change the name of the printer e.g. to something user-friendly like MG7160.

3) on some routers (192.168.1.254 in the case of the one my parents-in-law were using as opposed to 192.168.178.1 on mine and 192.168.1.1 or 192.168.0.1 on many IIRC) you will be able to see the printer connected. Having given it a useful name in step 2) made this easier to spot. I could see it on my own router but not on my parents-in-law’s router for some reason but there did seem to be some connection. After I renamed the printer it was detected on the PC as a network printer under the new name so something was getting across.

BTW the error messages I was getting when trying to run the printer on my parents-in-law’s computers were inconsistent. More on that when I update this post from their place.

Installing wireless USB modem driver on Ubuntu 15.10

It really was this simple:

How to install D-Link DWA-182 Wireless AC1200 Dual Band USB Adapter on Linux Ubuntu

Download zip from here:

rtl8812AU_8821AU_linux on GitHub

Unzip folder e.g. as “/home/mythbuntu/rtl8812AU_8821AU_linux-master”

cd "/home/mythbuntu/rtl8812AU_8821AU_linux-master"
make clean
make
sudo make uninstall
sudo make install

Restart and enable wireless and enter password when prompted (after selecting your own ssid).

Trouble copying audio file – until rdd-copy

I had trouble copying a audio file (wav) from a CD to my computer – the copy process was always getting stuck at exactly the same point 153MB in. And it didn’t matter whether I was using sound-juicer or nautilus. The answer was to install rdd (sudo apt-get install rdd). rdd copes with errors by supplying blanks (assuming multiple careful attempts to read the data have all failed) rather than halting.

The required command was rdd-copy src dest. But what to supply as src? I tried /media and similar but no luck. The final answer was ‘/run/user/1000/gvfs/cdda:host=sr0/Track 3.wav’. But how to find it? http://askubuntu.com/questions/222622/cdrom-is-mounted-but-where. Just drag the file from nautilus to the terminal and see what is displayed there. The following worked even though it took a long time to get the file:

rdd-copy '/run/user/1000/gvfs/cdda:host=sr0/Track 3.wav' /home/g/Desktop/track3.wav

F-spot vanished in Ubuntu 15.04 (Vivid)

F-spot has been removed from Ubuntu Vivid (15.04).

http://www.ubuntuupdates.org/package/core/utopic/universe/base/f-spot

Dependency is not satisfiable: liblcms1 (>= 1.15-1)

None of the data for f-spot was gone, just the ability to run the application – probably something to do with mono library deprecation.

~$ find / -name f-spot 2>/dev/null
/home/g/.gconf/apps/f-spot
/home/g/.config/f-spot

1.3MB in /home/g/.config/f-spot/photos.db

Anyway, opened Shotwell, “Import from Application”, “Import media from: F-Spot”, moving on.

Printer driver for Canon MG-7100 on Ubuntu

Once again I was solving a computer problem for my parents-in-law ;-). This time, it was yet another new printer they had bought – a Canon MG7100. Usually, I have had a really good experience with modern Ubuntu and popular printers. They Just Work. And this time it seemed things had gone well again. And they almost had. Except the colours were a bit off. On the Ubuntu printer test page Magenta was brown, bright green was instead a darker green, and yellow was very muddy – more like taupe. I wasted a lot of time cleaning ink nozzles etc etc but the actual solution was to choose a slightly different driver manually from the Canon list. There were two v4.0 options and it was the second that worked.

Category Good setting Bad setting
Job ID Canon-MG7100-2-671 Canon-MG7100-665
Driver CNMG7100.PPD STP00541.PPD
Driver Version 1.0 5.2.10-pre2
Description Canon MG7100 Canon MG7100
Driver Version Canon MG7100 Canon MG7100
Make and Model Canon MG7100 series Ver.4.00 Canon MG7100 series – CUPS+Gutenprint v5.2.10
Printer Canon-MG7100-2 Canon-MG7100

Hope this helps someone else.