Category Archives: Ubuntu

Mainly Ubuntu and its derivatives but potentially any Linux distro.

Adventures in Rye

Posted on by
Reply

Rye is a fantastic packaging solution for Python. See Python Package Management – Rye is All You Need by Elliot Simpson (Kiwi PyCon 2024)

There have been a couple of things to work out though.

Installing Rye on A Colleague’s Windows Computer in a Corporate Setting

The installation didn’t seem to properly work. Solution – switch off VPN when trying to install rye. Success! 🙂

Guardrails-AI

Most versions worked fine with rye but 0.5.10 didn’t make it possible to run the guardrails hub commands to install valid_sql e.g.

guardrails hub install hub://guardrails/valid_sql

So, having activated the virtual environment, I did a pip installation instead

python -m pip install guardrails-ai

and then added the dependency to pyproject.toml so rye wouldn’t remove it if I ran rye sync later. Installing valid_sql from guardrails hub worked after that.

wxPython

I ran:

rye add wxpython

and it built it successfully. Unfortunately, I didn’t build support for webview in.

Problem – trying to run the following didn’t work:

wx.html2.WebView.New() -> NotImplementedError

To build support for webview I needed to:

sudo apt install libwebkit2gtk-4.1-dev

and

sudo apt install clang

to avoid the error:

error: command 'clang' failed: No such file or directory

and possibly libwebkit2gtk-4.1-dev (although I haven’t checked whether removing it breaks things)

rye remove wxpython

Then, before running:

rye add wxpython --sync

I deleted ~/.cache/uv/built-wheels-v2/index/b2a7eb67d4c26b82/wxpython

Everything built correctly after that.

Extracting from Youtube on Ubuntu

Posted on by

How to extract a video from Youtube and optionally to extract mp3 from the video. On Ubuntu 20.10:

Install youtube-dl (for downloading from youtube):

sudo curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl
sudo chmod a+rx /usr/local/bin/youtube-dl

Install ffmpeg for sound extraction:

sudo apt install ffmpeg

How to extract video (note – has to be your paths to python3 and youtube-dl):

/usr/bin/python3 /usr/local/bin/youtube-dl https://www.youtube.com/watch?<youtubeid>

How to extract sound:

ffmpeg -i '<title>-<youtubeid>.mkv' '<title>.mp3'

With thanks to 3 Easy Ways to Download YouTube Videos in Ubuntu and Other Linux Distributions; How to Extract Audio From Video in Ubuntu and Other Linux Distributions; and Youtube-dl: Python not found (18.04)

First Impressions of Python Pattern Matching

Posted on by

Controversy – Python Adding Everything But the Kitchen Sink?

Is Python piling in too many new features taken from other languages? Is Pattern Matching yet another way of doing things for a language which has prided itself on there being one obvious way of doing things? In short, is Python being ruined by people who don’t appreciate the benefits of Less Is More?

Short answer: No ;-). Long answer: see below. Changed answer: see Second Impressions of Python Pattern Matching

I appreciate arguments for simplicity, and I want the bar for new features to be high, but I am glad Pattern Matching made its way in. It will be the One Obvious Way for doing, errr, pattern matching. Pattern matching may not be as crucial in a dynamically typed language like Python but it is still useful. And the syntax is nice too. match and case are pretty self-explanatory. Of course, there are some gotchas to watch out for but pattern matching is arguably one of the most interesting additions to Python since f-strings in Python 3.6. So let’s have a look and see what we can do with it.

What is Pattern Matching?

Not having used pattern matching before in other languages I wasn’t quite sure how to think about it. According to Tomáš Karabela I’ve been using something similar without realising it (Python 3.10 Pattern Matching in Action). But what is Pattern Matching? And why would I use it in Python?

I have found it useful to think of Pattern Matching as a Switch statement on steroids.

Pattern Matching is a Switch Statement on Steroids

The switch aspect is the way the code passes through various expressions until it matches. That makes total sense – one of the earliest things we need to do in programming is respond according the value / nature of something. Even SQL has CASE WHEN statements.

The steroids aspect has two parts:

Unpacking

Unpacking is beautiful and elegant so it is a real pleasure to find it built into Python’s Pattern Matching. case (x, y): looks for a two-tuple and unpacks into x and y names ready to use in the code under the case condition. case str(x): looks for a string and assigns the name x to it. Handy.

Type Matching

Duck typing can be a wonderful thing but sometimes it is useful to match on type. case Point(x=x, y=y): only matches if an instance of the Point class. case int(x): only matches if an integer. Note – case Point(x, y): doesn’t work in a case condition because positional sub-patterns aren’t allowed. Confused? More detail on possible gotchas below:

Gotchas

Sometimes you think exactly the same as the language feature, sometimes not. Here are some mistakes I made straight away. Typically they were caused by a basic misunderstanding of how Pattern Matching “thinks”.

Patterns aren’t Objects

case Point(x, y): seems to me to be an obvious way of looking for a Point object and unpacking its values into x and y but it isn’t allowed. It is the correct syntax for instantiating a Point object but we are not instantiating an object and supplying the object the case condition – instead we are supply a pattern to be matched and unpacked. We have to have a firm grasp on the notion that Python patterns are not objects.

Patterns Ain’t Objects

If we forget we get a TypeError:

case Point(0, 999):
TypeError: Point() accepts 0 positional sub-patterns (2 given)

Note, you must match the parameter names (the left side) but can unpack to any variable names you like (the right side). It may feel a bit odd being forced to use what feel like keyword arguments when the original class definition is positional but we must remember that we aren’t making an object – we are designing a pattern and collecting variables / names.

case Point(x=0, y=y): the x= and y= are the required syntax for a pattern. We insist on x being 0 but y can be anything (which we add the name y to). We could equally have written case Point(x=0, y=y_val): or case Point(x=0, y=spam):.

case Point:, case int:, case str:, case float: don’t work as you might expect. They match anything and assign it to the name Point or int or str etc. Definitely NOT what you want. The only protection is when you accidentally do this before other case conditions – e.g.

case int:
^
SyntaxError: name capture 'int' makes remaining patterns unreachable

This might become a common error because of our experience with isinstance where we supply the type e.g. isinstance(x, int). Remember:

case Patterns have to be Patterns

Instead, using the example of integer patterns, we need case int():, or, if we want to “capture” the value into, say, x, case int(x):.

Guards and Traditional Switch Statements

It is very common in switch / case when statements to have conditions. Sometimes it is the whole point of the construct – we supply one value and respond differently according to its values / attributes. E.g. in rough pseudocode if temp < 0 freezing, if > 100 boiling, otherwise normal. In Pattern Matching value conditions are secondary. We match on a pattern first and then, perhaps evaluating the unpacked variables in an expression, we apply a guard condition.

case float(x) if abs(x) < 100:
...
case float(x) if abs(x) < 200:
etc

Depending on how it’s used we could think of “Pattern Matching” as “Pattern and Guard Condition Matching”.

The most similar to a classic switch construct would be:

match:
case val if val < 10:
...
case val if val < 20:
...
etc

One final thought: there seems to be nothing special about the “default” option (using switch language) – namely, case _:. It merely captures anything that hasn’t already been matched and puts _ as the name i.e. it is a throwaway variable. We could capture and use that value with a normal variable name although that is optional because there’s nothing stopping us from referencing the original name fed into match. But, for example, case mopup: would work.

How to play with it on Linux

Make image using Dockerfile e.g. the following based on Install Python3 in Ubuntu Docker (I added vim and a newer Ubuntu image plus changed apt-get to apt (even though it allegedly has an unstable cli interface):

FROM ubuntu:20.04

RUN apt update && apt install -y software-properties-common gcc && \
add-apt-repository -y ppa:deadsnakes/ppa

RUN apt update && apt install -y python3.10 python3-distutils python3-pip python3-apt vim

docker build --tag pyexp .

(don’t forget the dot at the end – that’s a reference to the path to find Dockerfile)

Then make container:

docker create --name pyexp_cont pyexp

and run it with access to bash command line

docker container run -it pyexp /bin/bash

Useful Links

Pattern matching tutorial for Pythonic code | Pydon’t

Python 3.10 Pattern Matching in Action

PEP 622

Shifting laptops in Ubuntu

Posted on by

My laptop HDD was failing and getting worse. Time to get a new machine and shift. Here is the highly simplified, sanitised version of how I got everything working.

Install latest Ubuntu (finally shifted from 32 to 64 bit). Use https://etcher.io/ to make live USB – it Just Worksâ„¢. Was given the option of retaining Windows 10 so gave it 84GB/2TB and kept rolling.

Sorted out Nvidia as per Ubuntu 16.04LTS extended display not working (see also https://www.youtube.com/watch?v=_36yNWw_07g)

  1. Run sudo apt-get purge nvidia-*
  2. Run sudo add-apt-repository ppa:graphics-drivers/ppa
  3. Run sudo apt-get update
  4. Run sudo apt-get install nvidia-364 (Check what’s the latest version… in my case nvidia-375 was the latest in the repos)

Reboot. Success?

Set up launcher etc so not sticky and icons the right size. Remove unneeded ones etc.

sudo apt install chromium-browser dh-make gftp gimp git idle3 imagemagick inkscape keepass2 mysql-server openjdk-9-jdk python3-matplotlib python3-nose python3-numpy python3-requests python-pil python-wxgtk3.0 python-wxversion p7zip-full scribus shotwell shutter sqlitebrowser synaptic thunderbird vim virtualbox virtualbox-guest-additions-iso vlc wine64-development hardinfo

I got SQLYog working. It was fun finding the icon. Run using wine sqlyog.exe etc rather than trying to double click exes.

For MySQL I copied everything over and sorted out permissions. The innodb tables wouldn’t run (tables apparently did not exist) but I didn’t actually need them given they were local copies of remote tables. So I just transferred databases over using SQLYog.

For my cron jobs I ran crontab -e on source laptop and stored results then ran same command on new machine and pasted them. How cool is that :-).

Installed newest version of eclipse. As always, some faffing around to get desktop file sorted out etc. E.g.

sudo ln -s /home/g/eclipse/java-oxygen/eclipse/eclipse /usr/bin/eclipse

Install pydev.

Brought across lots of documents of course, and .gftp, .idlerc, .matplotlib, .openshot, .shh. Plus Music, Videos, Pictures (including photos)

Added thunderbird and firefox to startup.

Changed LibreOffice icon style to Galaxy under Tools>Options>LibreOffice>View>Icon style.

Then run some backups.

ImageMagick cache resources exhausted resolved

Posted on by

My sofastatistics application relies on ImageMagick to convert PDFs to PNGs. The sort of command run under the hood was:

convert -density 1200 -borderColor "#ff0000" -border 1x1 -fuzz 1% -trim "/home/g/projects/sofastats_proj/storage/img_processing/pdf2img_testing/KEEPME/raw_pdf.pdf" "/home/g/projects/sofastats_proj/storage/img_processing/pdf2img_testing/density_1200_fuzz_on_#ff0000.png"

Recently, commands like this stopped working properly on my development machine. They wouldn’t handle high resolutions (600dpi seemed to be the limit for the images I was handling) and it took a very long time to complete.

I finally worked out what was going on by running the same tests on different machines.

Seemingly modest differences in CPU specs can create massive differences in the time required to convert PDFs to PNGs. What takes 4 seconds on an i7 can take 71 seconds on an i5. And creating a 1200 dpi image might take 0.5 minutes on an i7 and 18.5 minutes on an i5. So the slowdown was because I had shifted from a fast desktop to a (more convenient but slower) laptop.

The second issue was the error message about cache resources exhausted. This happened on a range of fast and slow machines and the amount of RAM seemed irrelevant. Interestingly, the problem only occurred on Ubuntu 17.04 and not 16.10. The reason was the policy.xml settings in /etc/ImageMagick-6/. It seems the following was set too low:

<policy domain="resource" name="disk" value="1GiB">
I changed it to:
<policy domain="resource" name="disk" value="10GiB"/>
and it would successfully create high-resolution PNGs even if it took a long time.

Hmmm – now I change the disk setting back and I am still able to make the higher-resolution images, even after rebooting. WAT?!

One other note – settings in policy.xml cannot be loosened through arguments supplied to the convert program via the CLI – they can only be tightened. It looks like these changes are all about security concerns with the intention of preventing malicious resource starvation.

Some references:

Eclipse Neon with Pydev on Ubuntu 17.04

Posted on by
  1. Check whether 32 or 64 bit – System Settings > Details > Overview – then download from https://www.eclipse.org/downloads/
  2. Right click on eclipse-inst-linux32.tar.gz and select Extract Here
  3. cd into eclipse-installer/ then run ./eclipse-inst
  4. Choose top item (standard Java version of eclipse)
  5. Make desktop icon for launcher as per step 7 here – How to install Eclipse using its installer remembering to point to java-neon instead of java-mars etc. Drag file onto launcher and icon will add itself and be operational
  6. If you don’t see toolbars in eclipse modify exec line Exec=env SWT_GTK3=0 as per eclipse doesn’t work with ubuntu 16.04
  7. I also added pydev in the usual way using link to http://www.pydev.org/updates as per http://www.pydev.org/download.html

Right click to set as wallpaper in Ubuntu

Posted on by

I have lots of images in my Pictures folder that I wouldn’t want to use as wallpaper. So I like to keep a separate wallpapers folder. But selecting from that via the standard Unity interface is a bit of a pain. I like to look at all the images at a reasonably large scale and choose what I feel like at the time. The answer is in http://askubuntu.com/questions/808736/set-as-wallpaper-right-click-option-in-nautilus-does-not-work.

gsettings set org.gnome.settings-daemon.plugins.background active true
gsettings set org.gnome.desktop.background show-desktop-icons true
xdg-mime default nautilus-folder-handler.desktop inode/directory

Dual Boot Ubuntu 16.04 on Win 10 Acer Aspire E15

Posted on by

The goal – to turn a Windows 10 laptop into a dual boot which retains Windows but makes Ubuntu 16.04 the primary OS. Work being performed on parents-in-law’s new laptop.

The solution overview:

  1. Make recovery disk (on USB)
  2. Shrink Windows partition from within Windows 10
  3. Downgrade BIOS
  4. Set supervisor password on BIOS
  5. Disable Secure Boot (but not UEFI)
  6. Make successful 64-bit, EFI-friendly USB
  7. Install Ubuntu alongside Windows 10
  8. Add Ubuntu efi files and change boot order so Ubuntu grub efi comes first
  9. Re-enable Secure Boot, enable F12 Boot Menu, remove supervisor password
  10. Set up Qualcomm Atheros QCA9377 Wireless Network Adapter.
  11. Misc

Make recovery disk (on USB)

Just in case. And you’ll need over 16GB so a 32GB USB should be right. Format as FAT32.

Open Windows, click on Windows button on screen (bottom left), All apps > Acer > Acer Recovery Management > Backup > Create Factory Default Backup. Tick “Back up system files to the recovery drive.”

Wait a long time while system backup prepared.

Shrink Windows partition from within Windows 10

From Windows button on screen > All apps > Windows Administrative Tools > Computer Management > Storage > Disk Management. Right-click on main NTFS partition and select Shrink. The empty space left over will be used by Ubuntu later.

Downgrade BIOS

This really matters. Without it I got black screen after getting USB startup disk to provide GRUB option. No amount of mucking around with nomodeset or noapic or noacpi helped. It was the BIOS!

http://community.acer.com/t5/E-and-M-Series-Laptops/Acer-Aspire-e5-573g-You-can-not-install-any-one-Linux/m-p/386281/highlight/true#M3080

I’ve repeated the content below in case the link ever disappears:

I figured out how to downgrade the bios.

Go to: http://us.acer.com/ac/en/US/content/drivers

Search by Product Model:

Aspire E5-573G (EDIT – or whatever you have)
Change boot order so Ubuntu grub efi comes first

Select the right OS and download a bios. In my case I downloaded 1.15.

Run the ZRT_115.exe.

It will fail.

But before you close the installer, go to C:\Users\name\AppData\Local\Temp\

Search for a folder (random letters).tmp

There should be a H2OFFT-W.exe and zrt.rd file in there.

Just copy this folder and close the failing install.

In that copied folder, edit the platform.ini file.

Before:

[BIOSVersionFormat]
;[nb4-version] +
BIOSVFEnable=1
VersionFormat=XN.NN

After:

[BIOSV[BIOSVersionFormat]
;[nb4-version] +
BIOSVFEnable=1
VersionFormat=XN.DDersionFormat]
;[nb4-version] +
BIOSVFEnable=1
VersionFormat=XN.DD

The VersionFormat value now has ‘XN.DD’ instead of ‘XN.NN’.

This will ignore the fact that 1.25 -> 1.15 is a downgrade.

Prepare for a reboot. I.E. close unnecessary applications. Because it’ll happen automatically after running the installer.

Run H2OFFT-W.exe.
Set supervisor password on BIOS
Upon reboot, you’ll see a bios installing progress bar.

After that is done, press F2 during startup to get to bios. The version should now be 1.15.

At this point I set a password, turned off UEFI, and swapped my hard drive out for a fresh SSD. Ubuntu finally installed.

Set supervisor password on BIOS

The laptop has InsydeH20 BIOS Rev. 5.0 – access this by pressing F2 quickly after bootup. Then move to Security > Set Supervisor Password e.g. ‘a’ (we’ll be removing this later so a short password for convenience makes sense in this use case).

It is necessary to set this password (on Acer machines at least) to alter UEFI settings.

Disable Secure Boot (but not UEFI)

In the BIOS move to the Boot section and disable Secure Boot.

Don’t disable UEFI – that may have been necessary a few years ago but it probably causes more problems that it solves now (assuming it works at all). See http://www.rodsbooks.com/linux-uefi/

Make successful 64-bit, EFI-friendly USB

Download 64-bit ISO image. 32-bit apparently won’t work with UEFI.

I had trouble with unetbootin and the Ubuntu Startup Disk Creator. So I used good old dd to make my startup USB drive. This is probably what solved the “Missing operating system” problem I was having.

Format USB as FAT32 (maybe using Gparted or the general Gnome disk utility Disks).

sudo dd if=”/home/g/Downloads/ubuntu-16.04-beta2-desktop-amd64.iso” of=/dev/sdd

Note — won’t necessarily be sdd. Open Disks, select USB DISK, look at Device setting e.g. /dev/sdc1.

Install Ubuntu alongside Windows 10

In BIOS under Boot change boot order so USB HDD (that’s the USB stick actually) comes above Windows Boot Manager. Insert startup USB and reboot. Choose to install Ubuntu alongside Windows. I needed to do this with an ethernet cable plugged in given wireless wasn’t working for Ubuntu out of the box.

Add Ubuntu efi files and change boot order so Ubuntu grub efi comes first

In the BIOS, Security > Select an UEFI file as trusted for executing. Approve all ubuntu efi files. HDD0 > > > all the .efi files. I name them ubuntuorignameefi so they are easy to identify correctly and reorder in the Boot priority order section e.g. grubx64.efi -> ubuntugrubx64efi. Then in Boot > Boot priority order raise the grubx64 entry to the top.

I had 4 files to set (unlike 3 in some docs I found) – namely: grubx64.efi, fwupx64.efi, shimx64.efi, and MokManager.efi.

Re-enable Secure Boot, enable F12 Boot Menu, remove supervisor password

All straight forward. Remover supervisor password by setting it to an empty string by entering orig password then Enter (to register current password), Enter (to submit empty string as password, Enter again to confirm.

Set up Qualcomm Atheros QCA9377 Wireless Network Adapter.

Identify wireless first:

lspci

03:00.0 Network controller: Qualcomm Atheros Device 0042 (rev 30)

Installing the required driver was explained here by @chili555. As another grateful person said “You rock!”.

http://ubuntuforums.org/showthread.php?t=2300861&page=11&p=13400551#post13400551

open a terminal and do:
Code:

sudo mkdir /lib/firmware/ath10k/QCA9377/
sudo mkdir /lib/firmware/ath10k/QCA9377/hw1.0

If it reports that the file already exists, that’s fine, just continue.

With a temporary working internet connection:

Code:

sudo apt-get install git
git clone https://github.com/kvalo/ath10k-firmware.git
cd ath10k-firmware/QCA9377/hw1.0
sudo cp board.bin /lib/firmware/ath10k/QCA9377/hw1.0
sudo cp firmware-5.bin_WLAN.TF.1.0-00267-1 /lib/firmware/ath10k/QCA9377/hw1.0/firmware-5.bin
sudo modprobe -r ath10k_pci
sudo modprobe ath10k_pci

Your wireless should be working; if not, try a reboot.

Misc

I installed the system load indicator (set to traditional colours with Processor, Network, Harddisk), VLC, Shotwell (loading all photos with copy as the option), and brought thunderbird and firefox data over from old computer (merely copying xxxxxxx.default folders and updating profiles.ini Profile > Path settings. Then setting up icons on launcher and we’re done!

Canon MG7160 on Ubuntu broken by update – solution

Posted on by

Another problem with my parents-in-law’s printer – it looks like an update broke the printer on two separate machines (an ancient desktop running Lubuntu 14.04 and an Asus Eee PC netbook running Ubuntu 14.04). Yet when I took the printer home it instantly worked with the default drivers on my Ubuntu 15.10 system there, albeit with the colours a bit off (see http://p-s.co.nz/wordpress/printer-driver-for-canon-mg-7100-on-ubuntu/). I fixed this by downloading and unpacking the driver at http://support-th.canon-asia.com/contents/TH/EN/0100551202.html. Then, from the terminal prompt I ran ./install.sh, answered some questions e.g. what I wanted the printer called, and voila – I had a working printer with the correct colours using Driver: CNMG7100.PPD.

Some things to note about setting up the printer:

1) the touchscreen on the top of the printer lets you swipe left and right to see more icons. The WiFi icon is where you can tell your printer the SSID and password/passphrase for you secured WiFi access point.

2) another setting lets you change the name of the printer e.g. to something user-friendly like MG7160.

3) on some routers (192.168.1.254 in the case of the one my parents-in-law were using as opposed to 192.168.178.1 on mine and 192.168.1.1 or 192.168.0.1 on many IIRC) you will be able to see the printer connected. Having given it a useful name in step 2) made this easier to spot. I could see it on my own router but not on my parents-in-law’s router for some reason but there did seem to be some connection. After I renamed the printer it was detected on the PC as a network printer under the new name so something was getting across.

BTW the error messages I was getting when trying to run the printer on my parents-in-law’s computers were inconsistent. More on that when I update this post from their place.