ImageMagick cache resources exhausted resolved

My sofastatistics application relies on ImageMagick to convert PDFs to PNGs. The sort of command run under the hood was:

convert -density 1200 -borderColor "#ff0000" -border 1x1 -fuzz 1% -trim "/home/g/projects/sofastats_proj/storage/img_processing/pdf2img_testing/KEEPME/raw_pdf.pdf" "/home/g/projects/sofastats_proj/storage/img_processing/pdf2img_testing/density_1200_fuzz_on_#ff0000.png"

Recently, commands like this stopped working properly on my development machine. They wouldn’t handle high resolutions (600dpi seemed to be the limit for the images I was handling) and it took a very long time to complete.

I finally worked out what was going on by running the same tests on different machines.

Seemingly modest differences in CPU specs can create massive differences in the time required to convert PDFs to PNGs. What takes 4 seconds on an i7 can take 71 seconds on an i5. And creating a 1200 dpi image might take 0.5 minutes on an i7 and 18.5 minutes on an i5. So the slowdown was because I had shifted from a fast desktop to a (more convenient but slower) laptop.

The second issue was the error message about cache resources exhausted. This happened on a range of fast and slow machines and the amount of RAM seemed irrelevant. Interestingly, the problem only occurred on Ubuntu 17.04 and not 16.10. The reason was the policy.xml settings in /etc/ImageMagick-6/. It seems the following was set too low:

<policy domain="resource" name="disk" value="1GiB">
I changed it to:
<policy domain="resource" name="disk" value="10GiB"/>
and it would successfully create high-resolution PNGs even if it took a long time.

Hmmm – now I change the disk setting back and I am still able to make the higher-resolution images, even after rebooting. WAT?!

One other note – settings in policy.xml cannot be loosened through arguments supplied to the convert program via the CLI – they can only be tightened. It looks like these changes are all about security concerns with the intention of preventing malicious resource starvation.

Some references:

Launchpad – Bazaar to Git

I’ve stored my SOFA Statistics code on launchpad since 2009 and used bazaar (bzr) to do it. But a lot has changed since then and I know use git on a daily basis in my job. So I’d much rather use git for SOFA. Fortunately that is now possible on launchpad.

I found https://help.launchpad.net/Code/Git to be useful apart from the migration instructions. These didn’t work for me. For example, I had no luck installing sudo apt-get install bzr-fastimportInstead I found https://design.canonical.com/2015/01/converting-projects-between-git-and-bazaar/.

Need ~/.bazaar/plugins

If plugins folder not there, cd ~/.bazaar; mkdir plugins

cd ~/.bazaar/plugins


bzr branch lp:bzr-fastimport fastimport

cd ~/projects/SOFA/sofastatistics/sofa.repo/sofa.main/

git init

bzr fast-export --plain . | git fast-import

gitk --all

YES! It’s all there

Archive .bzr in case

USER is launchpad-p-s in my case (yes – a strange choice which made sense at the time)

PROJECT is sofastatistics

So as per
[url "git+ssh://USER@git.launchpad.net/"]
insteadof = lp:

added the following to ~/.gitconfig

[url "git+ssh://launchpad-p-s@git.launchpad.net/"]
insteadof = lp:

Note – if not using lp: etc I had trouble with my ssh key – possibly something to do with confusion between user g and user launchpad-p-s.

I own my own project so to implement git remote add origin lp:PROJECT I ran:

git remote add origin lp:sofastatistics

Note: would only work if insteadof setting added to ~/.gitconfig as described earlier https://help.launchpad.net/Code/Git

Otherwise I would have to git remote add origin git+ssh://git.launchpad.net/sofastatistics

As per https://launchpad.net/PROJECT/+configure-code. I.e.

https://launchpad.net/sofastatistics/+configure-code

Confirmed by making a check folder then cloning the code in: git clone git://git.launchpad.net/~launchpad-p-s/sofastatistics

Simple flask app on heroku – all steps (almost)

Note – instructions assume Ubuntu Linux.

See Getting Started with Python on Heroku (Flask) for the official instructions. The instructions below tackle things differently and include redis-specific steps.

Don’t need postgresql for my app even though needed for heroku demo app. Using redis for simple key-value store.

Main reason for each step is indicated in bold at start. There are lots of steps but there are lots of things being achieved. And each purpose only requires a few steps so probably hard to streamline any further.

  1. APP & BEST PRACTICE
    >> sudo apt-get install python3 python3-pip python-virtualenv git ruby redis-server redis-tools
  2. HEROKU
    Get free heroku account
  3. HEROKU
    Install heroku toolbelt Heroku setup. Sets up virtualenvwrapper for you too (one less thing to figure out)
  4. HEROKU
    Do the once-ever authentication
    >> heroku login
  5. APP
    Make project folder e.g.
    >> mkdir ~/projects/myproj
  6. APP
    >> cd ~/projects/myproj
  7. HEROKU
    >> echo “web: python main.py” > Procfile
  8. HEROKU & BEST PRACTICE
    >> git init
  9. HEROKU & BEST PRACTICE
    >> mkvirtualenv sticky

    So requirements for specific project can be separated from other project – lets heroku identify actual requirements. Normally “workon sticky” thereafter; deactivate to exit virtual env

  10. APP
    >> pip3 install flask
    Note – installed within virtualenv
  11. HEROKU
    Save the following as requirements.txt – needed by heroku so it knows the dependencies. Update version of redis as appropriate. gunicorn is a better approach than the flask test server
    flask
    gunicorn
    redis==2.10.3
  12. HEROKU
    So we can use Python 3.4 instead of the current default of 2.7:
    >> echo “python-3.4.3” > runtime.txt
  13. APP & HEROKU

    Make a toy app to get started from.

    Note – modify the standard demo flask app to add a port to ease eventual heroku deployment. Otherwise the app will fail because of a problem with the port when running

    heroku ps:scale web=1

    Starting process with command `python main.py`
    ...
    Web process failed to bind to $PORT within 60 seconds of launch

    Here is an example (may need updating if flask changes):

    import os
    from flask import Flask
    app = Flask(__name__)

    @app.route("/")
    def hello():
        return "Hello World!"

    if __name__ == "__main__":
        port = int(os.environ.get("PORT", 33507))
        app.run(host='0.0.0.0', port=port)

  14. BEST PRACTICE
    >> deactivate
  15. Make a module to make it easier to work with redis – let’s call it store.py:

    import os
    import urllib
    import redis

    url = urllib.parse.urlparse(os.environ.get('REDISTOGO_URL',
        'redis://localhost:6379'))
    redis = redis.Redis(host=url.hostname, port=url.port, db=0,
        password=url.password)

    We can then use redis like this:
    from store import redis

  16. APP
    Keep building app locally. The following is good for redis: Redis docs. And flasks docs are always good: Flask Docs – Minimal Application
  17. HEROKU & BEST PRACTICE

    Before deploying to production:

    1. Update git otherwise you’ll be deploying old code – heroku uses git for deployment
    2. set app.debug to False (although no rush when just getting started and not expecting the app to get hit much)
    3. probably switch to gunicorn sooner or later (will need to change ProcFile to
      web: gunicorn main:app --workers $WEB_CONCURRENCY
      )
    4. Example nginx.conf:

      # As long as /etc/nginx/sites-enable/ points to
      # this conf file nginx can use it to work with
      # the server_name defined (the name of the file
      # doesn't matter - only the server_name setting)
      # sudo ln -s /home/vagrant/src/nginx.conf ...
      #     ... /etc/nginx/sites-enabled/myproj.com
      # Confirm this link is correct
      # e.g. less /etc/nginx/sites-enabled/myproj.com

      server {
          listen 80;
          server_name localhost;

          location /static { # static content is

              # handled directly by NGINX which means
              # the nginx user (www-data) will need
              # read permissions to this folder
              root /home/vagrant/src;

          }

          location / { # all else passed to Gunicorn to handle
              # Pass to wherever I bind Gunicorn to serve to
              # Only gunicorn needs rights to read, write,
              # and execute scripts in the app folders
              proxy_pass http://127.0.0.1:8888;
          }
      }

    5. Example gunicorn.conf
      import multiprocessing

      bind = "127.0.0.1:8888" # ensure nginx passes to this port
      logfile = "/home/vagrant/gunicorn.log"
      workers = multiprocessing.cpu_count() * 2 + 1

  18. HEROKU
    >> heroku create

    Should now be able to browse to the url supplied as stdout fom command e.g.
    https://not-real-1234.herokuapp.com/. Note – not working yet – still need to deploy to new app

    >> git push heroku master

    Must then actually spin up the app:

    >> heroku ps:scale web=1

    A shortcut for opening is

    >> heroku open

  19. HEROKU
    Add redis support (after first deployment – otherwise

    ! No app specified.
    ! Run this command from an app folder or specify which app to use with --app APP.
    )
    >> heroku addons:create redistogo

    Note – need to register credit card to use any add-ons, even if free ones. Go to https://heroku.com/verify

Some other points: when developing on a different machine, I needed to supply my public key to heroku from that other machine (Permission denied (publickey) when deploying heroku code. fatal: The remote end hung up unexpectedly).

heroku keys:add ~/.ssh/id_rsa.pub

And the full sequence for upgrading your app after the prerequisites have been fulfilled is:

  1. git commit to local repo
  2. Then git push to heroku
  3. Then run heroku ps:scale web=1 again

And I had a problem when I switched from Python 2 to 3 with redis – my heroku push wouldn’t work. By looking at the logs (>> heroku logs –tail) I found that import imap wouldn’t work and searching on that generally found I needed a newer version of redis than I had specified foolishly in requirements.txt.

F-spot vanished in Ubuntu 15.04 (Vivid)

F-spot has been removed from Ubuntu Vivid (15.04).

http://www.ubuntuupdates.org/package/core/utopic/universe/base/f-spot

Dependency is not satisfiable: liblcms1 (>= 1.15-1)

None of the data for f-spot was gone, just the ability to run the application – probably something to do with mono library deprecation.

~$ find / -name f-spot 2>/dev/null
/home/g/.gconf/apps/f-spot
/home/g/.config/f-spot

1.3MB in /home/g/.config/f-spot/photos.db

Anyway, opened Shotwell, “Import from Application”, “Import media from: F-Spot”, moving on.

Printer driver for Canon MG-7100 on Ubuntu

Once again I was solving a computer problem for my parents-in-law ;-). This time, it was yet another new printer they had bought – a Canon MG7100. Usually, I have had a really good experience with modern Ubuntu and popular printers. They Just Work. And this time it seemed things had gone well again. And they almost had. Except the colours were a bit off. On the Ubuntu printer test page Magenta was brown, bright green was instead a darker green, and yellow was very muddy – more like taupe. I wasted a lot of time cleaning ink nozzles etc etc but the actual solution was to choose a slightly different driver manually from the Canon list. There were two v4.0 options and it was the second that worked.

Category Good setting Bad setting
Job ID Canon-MG7100-2-671 Canon-MG7100-665
Driver CNMG7100.PPD STP00541.PPD
Driver Version 1.0 5.2.10-pre2
Description Canon MG7100 Canon MG7100
Driver Version Canon MG7100 Canon MG7100
Make and Model Canon MG7100 series Ver.4.00 Canon MG7100 series – CUPS+Gutenprint v5.2.10
Printer Canon-MG7100-2 Canon-MG7100

Hope this helps someone else.

Eclipse and PyDev on Utopic

I upgraded to Utopic (Utopic Unicorn a.k.a 14.10) and eclipse wouldn’t complete loading anymore. Solution:

Download latest plain vanilla Eclipse from the standard downloads page. And feel free to donate something too.

sudo su

chown -R root:root /home/username/eclipse && mv /home/username/Downloads/eclipse /opt

ln -s /opt/eclipse/eclipse /usr/local/bin/eclipse && exit

Start by running:

eclipse

It didn’t even break PyDev so my luck’s finally turning ;-).

https://www.tumblr.com/search/install+eclipse+ubuntu

Canonical – ignore the hate and make your bet

Does Canonical deserve the hate that is being directed at it by some people? I doubt it – who could? Canonical deserves criticism for some things, of course, but not the ongoing and disproportionate spite some technical people direct at it. The only explanation I can come up with is that when some people love or believe in something strongly, and then are disappointed in that love, they can feel very, very angry. And as with romantic love, nothing will make them feel better till they have a new love to take the place of the old one. So I’m not sure there is much Canonical can do to appease them. Instead, the focus should be on the big bets it has to make to secure a future. The following comment (on Slashdot) captured the issue especially well so I quote it in full:

It’s not rocket science. Rightly or wrongly Canonical has decided that the future of general computing is in the mobile space and they are working on getting Ubuntu there and bridging the gap between the mobile computing experience and the desktop computing experience.

In simplest of terms, they’re trying to make a distro that can be both a phone and a desktop all in the same device. Again — rightly or wrongly — they have decided that they needed to move certain things in house to best accomplish that goal (Mir) and needed a specific interface they were in control of to scale between display form factors (Unity).

If you are a person that thinks this direction is wrong and will hurt Linux in the long run, then you belong in the “bad for Linux” category. I’m a person that thinks this is absolutely the best way for Linux to finally have its “year of the desktop” similar to how Apple made their comeback but with a twist — by providing a compelling mobile experience with a device that just so happens to be able to double as someone’s desktop when they want a bigger screen.

Pay attention to plunging desktop sales numbers. As people find ways to make mobile devices and tablets their only computing devices, this strategy will start to look smarter and smarter. Whatever else you think of Canonical (and by extension Ubuntu), this will either make them or break them.

Ubuntu good for linux?

As with most big bets, there are no guarantees of success. And Canonical will make mistakes. But I fully support their attempt and wish them well.

Cheaply Digitising Old Records

Digitising old records

The problem – a bunch of old records passed on to me by my Mum that used to be mine when I was a kid. An old Vera Lynn 45 record of children’s songs, my Rupert the Bear musical I was given when I was 8, a Looney Tunes collection, and misc others. The answer – a cheap USB turntable and Audacity for the actual recording. Of course, for better quality recording you need better equipment (proper record player + a pre-amp apparently) and you need to spend more time cleaning records, removing imperfections from the recording etc. But that would guarantee this job would never actually get done or be worth the time and money ;-). And for children’s records of the sort I needed to record, sound fidelity was not much of an issue. The USB turntable was a Megabeat MB-USBTR04, the computer was running Ubuntu Linux 13.04, and I used Audacity 2.0.3. One potential gotcha – you need to select the USB line in option as your input from the drop down list (very long in my case for some reason) on the menu. Then save your recordings in the lossless flac format and you’re done.