ImageMagick cache resources exhausted resolved

My sofastatistics application relies on ImageMagick to convert PDFs to PNGs. The sort of command run under the hood was:

convert -density 1200 -borderColor "#ff0000" -border 1x1 -fuzz 1% -trim "/home/g/projects/sofastats_proj/storage/img_processing/pdf2img_testing/KEEPME/raw_pdf.pdf" "/home/g/projects/sofastats_proj/storage/img_processing/pdf2img_testing/density_1200_fuzz_on_#ff0000.png"

Recently, commands like this stopped working properly on my development machine. They wouldn’t handle high resolutions (600dpi seemed to be the limit for the images I was handling) and it took a very long time to complete.

I finally worked out what was going on by running the same tests on different machines.

Seemingly modest differences in CPU specs can create massive differences in the time required to convert PDFs to PNGs. What takes 4 seconds on an i7 can take 71 seconds on an i5. And creating a 1200 dpi image might take 0.5 minutes on an i7 and 18.5 minutes on an i5. So the slowdown was because I had shifted from a fast desktop to a (more convenient but slower) laptop.

The second issue was the error message about cache resources exhausted. This happened on a range of fast and slow machines and the amount of RAM seemed irrelevant. Interestingly, the problem only occurred on Ubuntu 17.04 and not 16.10. The reason was the policy.xml settings in /etc/ImageMagick-6/. It seems the following was set too low:

<policy domain="resource" name="disk" value="1GiB">
I changed it to:
<policy domain="resource" name="disk" value="10GiB"/>
and it would successfully create high-resolution PNGs even if it took a long time.

Hmmm – now I change the disk setting back and I am still able to make the higher-resolution images, even after rebooting. WAT?!

One other note – settings in policy.xml cannot be loosened through arguments supplied to the convert program via the CLI – they can only be tightened. It looks like these changes are all about security concerns with the intention of preventing malicious resource starvation.

Some references:

F-spot vanished in Ubuntu 15.04 (Vivid)

F-spot has been removed from Ubuntu Vivid (15.04).

http://www.ubuntuupdates.org/package/core/utopic/universe/base/f-spot

Dependency is not satisfiable: liblcms1 (>= 1.15-1)

None of the data for f-spot was gone, just the ability to run the application – probably something to do with mono library deprecation.

~$ find / -name f-spot 2>/dev/null
/home/g/.gconf/apps/f-spot
/home/g/.config/f-spot

1.3MB in /home/g/.config/f-spot/photos.db

Anyway, opened Shotwell, “Import from Application”, “Import media from: F-Spot”, moving on.

Picasa 3.9 + F-spot together on Ubuntu

Some people want to use F-Spot and Picasa together – F-spot to import and tag photos; and Picasa to do minor editing ready for printing. Here’s their desired workflow:

  1. attach camera
  2. import into F-Spot
  3. do editing in Picasa
  4. want any changes to also be the same in F-spot.

This should be manageable given that Picasa can work on existing files, without having to separately import them to a different place on the file system. So the photo files are where F-Spot put them, and, irrespective of how Picasa organises them in its own “virtual” folders, changes made to the photo in Picasa alter the original photo in its F-spot location. So far, so good.

But there is one issue. If you install the last version of Picasa made for Linux, it is significantly out-of-date. It works as described with Picasa, but it is old. And if you install the latest version of Picasa (3.9) under WINE (instructions), a problem occurs when bringing all the photos into Picasa. Instead of merely pointing to the file locations for the photos, Picasa actually separately makes copies of them. Which means that when you edit them, the original in F-spot is unaltered.

Fortunately, there is a workaround and it involves changing the way Picasa watches folders. It requires some simple edits to the watchedfolders.txt file in: /home/username/.wine/dosdevices/c:/users/username/Local Settings/Application Data/Google/Picasa2Albums

Note: even though you might be using Picasa 3.9, the path is Picasa2Albums.

Make the following alterations as appropriate, depending on which folders you have photos in:

C:\users\username\My Pictures\ –>

/home/username/Pictures

/home/username/Pictures\

C:\users\username\My Documents\ –>

/home/username/Documents

/home/username/Documents\

C:\users\username\Desktop –>

/home/username/Desktop

/home/username/Desktop\

Note that both versions seem to be needed (see also How can I get Picasa to view my second HDD?).

[UPDATE]

Here are some extra issues I encountered:

  • One huge gotcha – Picasa cropping must be Saved to change the file on disk. Clicking on “Apply Change” is not enough to change the file on disk even though the change persists within Picasa. If it isn’t saved, the change is only visible when looking at the image in Picasa – it won’t show up when looking at the file otherwise (e.g. using ImageViewer) and that includes F-Spot. Can’t really blame F-spot for that.
  • F-Spot doesn’t automatically know about files put into its folder structure e.g. if you manually insert an image into /home/username/Photos/2014/03/03 f-spot won’t display it.
  • Picasa only grabs images from the watched folders if they have an image extension e.g. .jpg. Otherwise it ignores them.

The net result is that if you want a simple workflow where you view and tag photos in F-Spot and modify images using Picasa you should follow these instructions:

WORKFLOW INSTRUCTIONS

  • Only put images into F-Spot using import. Don’t manually put images into the standard folders F-Spot uses e.g. “Photos/2014/03/03/”.

    A simple workflow is to have a standard folder under Photos e.g. “Photos/0 Upload These” where you put copies of photos whether you get them from a camera or from another source e.g. as email attachments. You can give these human-readable names such as me_at_beach.jpg rather than IMG9805.JPG etc before importing them. Once imported into F-Spot (and thus copies put into the date-driven folder locations), empty the “Photos/0 Upload These” folder.

  • Save changes in Picasa – don’t just Apply Changes.
  • If you want multiple versions of an image, make them in F-Spot so F-Spot knows about them, then edit the extra versions in Picasa.

    To create new versions in F-Spot File, select a photo, then Photo>Create New Version in the menu.

PDF to PNG using PythonMagick

The problem: you have a large, crisp PDF image and you use PythonMagick to write it as a PNG but it comes out as a small, low-resolution, blurry PNG. You mess around with density, size, and quality settings to no avail. And the documentation is of little help. The answer is subtle, and here it is: you must only read the PDF image AFTER you have set the density to something high like 300. Otherwise it is the default dpi of only 72. Big thanks to PyBlosxom for providing a working example of code and restoring my sanity ;-).

FAIL
import PythonMagick
img = PythonMagick.Image("Desktop/test.PDF")
img.density("300") # too late, already read as 72 dpi when image instantiated
img.write("Desktop/test.PNG")

SUCCEED
import PythonMagick
img = PythonMagick.Image()
img.density("300")
img.read("Desktop/test.PDF") # read in at 300 dpi
img.write("Desktop/test.PNG")

Apparently, PIL is no good for this task as it can only write PDFs (http://www.velocityreviews.com/forums/t563423-convert-pdf-to-png.html)

And if you are confused about how to supply the Geometry argument, that is because there are several ways of doing it:

image.density(Geometry(150,150)); // could also use image.density("150x150")
Magick::Image Class
image manipulation with python

Here are some links that may be useful:

PythonMagick Readme

OpenShot – Finally, decent video editing for Linux

Using OpenShot I was able to make videos, edit them, insert new sections, add soundtracks, make a groovy 3D title sequence (using Blender behind the scenes). I exported output as AVIs and MP4s, was able to select a size format that perfectly suited my target (www.showmedo.com), and was able to work it all out without reference to the documentation. So this is a big vote of confidence in a brilliant project with a bright future. The developer is very talented (and not just as a developer, but also as a promoter etc) and I think it is safe to hitch my wagon to that program.

The few glitches I experienced were not very large and will probably not survive for too long.

As an example of what I was able to do with OpenShot, check out the Quick Overview video here:

SOFA taster video

Making custom emblems for Ubuntu

Have you ever wanted to make your own emblems for Ubuntu? It is very easy. Just use the GIMP (or something similar) to make a png image (make it at least 40x40px) and then open nautilus (e.g. Places>Home Folder>Edit>Backgrounds and Emblems and add it. I store all my emblems in ~/Pictures/emblems. For a brilliant overview of adding emblems etc visit http://www.go2linux.org/How-Customize-Ubuntu-with-Backgrounds-and-Emblems.

F-spot and the date sort order of pre-digital photos

F-spot is very useful but it has a few quirks which can be very confusing. Especially if you are working with photos which do not come from a digital camera (e.g. scanned, taken from the web etc). The order in which they display, and the dates under which they are stored, can be quite confusing. And quite difficult to adjust.

First, some background. When a digital camera takes a photo a whole lot of information (metadata) is stored with the photo e.g. camera band, exposure time, date taken. The latter seems to influence the order a photo displays in F-spot, not the date of creation*, or the name of the photo.

Here is an example:

> Image Type: jpeg (The JPEG image format)
> Width: 2272 pixels
> Height: 1704 pixels
> Camera Brand: Canon
> Camera Model: Canon PowerShot G2
> Date Taken: 2004:04:16 12:12:22
> Exposure Time: 1/199 sec.
> Aperture Value: 4.00 EV (f/4.0)
> Flash Fired: Flash did not fire, auto mode.
> Metering Mode: Pattern
> Focal Length: 7.0 mm
> Software: f-spot version 0.4.3.1

Photos like this are stored and sorted by F-spot much like you would expect. But what do you do with images which didn’t come from a digital camera?

One option is to manually change the datetime of the image. To do this, select a photo, then from the main f-spot menu Edit > Adjust Time. If you change the time, you also change the sequence.

For further information on F-spot there is a great article here – http://www.linuxjournal.com/article/9110.

* I wrote a little python script utilising the Python Imaging Library (PIL) which created new versions of the photos in the order I wanted and gave a creation date accordingly. Didn’t solve the F-spot ordering problem though.

Using Xsane without going Insane

Xsane works well enough once you figure out the interface. Whatever you do, don’t accidentally mess the colour settings up ;-). Here is a tentative set of instructions for mass scanning of photos my mother put together using Xsane on Intrepid. I’ll expand upon them later from a machine with a scanner on it:

Applications>Graphics>X-Sane
Choose appropriate scanner

Within X-Sane, Preferences
Window – Show Preview
Viewer – change to SAVE if doing multiple
Choose folder for saving to
Choose TIFF
Choose COLOR or GREY
Choose 300 for Dots per inch

SCAN

PREVIEW
Acquire preview then crop

PREFERENCES Name file in folder
Scan
Check folder