Simple flask app on heroku – all steps (almost)

Note – instructions assume Ubuntu Linux.

See Getting Started with Python on Heroku (Flask) for the official instructions. The instructions below tackle things differently and include redis-specific steps.

Don’t need postgresql for my app even though needed for heroku demo app. Using redis for simple key-value store.

Main reason for each step is indicated in bold at start. There are lots of steps but there are lots of things being achieved. And each purpose only requires a few steps so probably hard to streamline any further.

  1. APP & BEST PRACTICE
    >> sudo apt-get install python3 python3-pip python-virtualenv git ruby redis-server redis-tools
  2. HEROKU
    Get free heroku account
  3. HEROKU
    Install heroku toolbelt Heroku setup. Sets up virtualenvwrapper for you too (one less thing to figure out)
  4. HEROKU
    Do the once-ever authentication
    >> heroku login
  5. APP
    Make project folder e.g.
    >> mkdir ~/projects/myproj
  6. APP
    >> cd ~/projects/myproj
  7. HEROKU
    >> echo “web: python main.py” > Procfile
  8. HEROKU & BEST PRACTICE
    >> git init
  9. HEROKU & BEST PRACTICE
    >> mkvirtualenv sticky

    So requirements for specific project can be separated from other project – lets heroku identify actual requirements. Normally “workon sticky” thereafter; deactivate to exit virtual env

  10. APP
    >> pip3 install flask
    Note – installed within virtualenv
  11. HEROKU
    Save the following as requirements.txt – needed by heroku so it knows the dependencies. Update version of redis as appropriate. gunicorn is a better approach than the flask test server
    flask
    gunicorn
    redis==2.10.3
  12. HEROKU
    So we can use Python 3.4 instead of the current default of 2.7:
    >> echo “python-3.4.3” > runtime.txt
  13. APP & HEROKU

    Make a toy app to get started from.

    Note – modify the standard demo flask app to add a port to ease eventual heroku deployment. Otherwise the app will fail because of a problem with the port when running

    heroku ps:scale web=1

    Starting process with command `python main.py`
    ...
    Web process failed to bind to $PORT within 60 seconds of launch

    Here is an example (may need updating if flask changes):

    import os
    from flask import Flask
    app = Flask(__name__)

    @app.route("/")
    def hello():
        return "Hello World!"

    if __name__ == "__main__":
        port = int(os.environ.get("PORT", 33507))
        app.run(host='0.0.0.0', port=port)

  14. BEST PRACTICE
    >> deactivate
  15. Make a module to make it easier to work with redis – let’s call it store.py:

    import os
    import urllib
    import redis

    url = urllib.parse.urlparse(os.environ.get('REDISTOGO_URL',
        'redis://localhost:6379'))
    redis = redis.Redis(host=url.hostname, port=url.port, db=0,
        password=url.password)

    We can then use redis like this:
    from store import redis

  16. APP
    Keep building app locally. The following is good for redis: Redis docs. And flasks docs are always good: Flask Docs – Minimal Application
  17. HEROKU & BEST PRACTICE

    Before deploying to production:

    1. Update git otherwise you’ll be deploying old code – heroku uses git for deployment
    2. set app.debug to False (although no rush when just getting started and not expecting the app to get hit much)
    3. probably switch to gunicorn sooner or later (will need to change ProcFile to
      web: gunicorn main:app --workers $WEB_CONCURRENCY
      )
    4. Example nginx.conf:

      # As long as /etc/nginx/sites-enable/ points to
      # this conf file nginx can use it to work with
      # the server_name defined (the name of the file
      # doesn't matter - only the server_name setting)
      # sudo ln -s /home/vagrant/src/nginx.conf ...
      #     ... /etc/nginx/sites-enabled/myproj.com
      # Confirm this link is correct
      # e.g. less /etc/nginx/sites-enabled/myproj.com

      server {
          listen 80;
          server_name localhost;

          location /static { # static content is

              # handled directly by NGINX which means
              # the nginx user (www-data) will need
              # read permissions to this folder
              root /home/vagrant/src;

          }

          location / { # all else passed to Gunicorn to handle
              # Pass to wherever I bind Gunicorn to serve to
              # Only gunicorn needs rights to read, write,
              # and execute scripts in the app folders
              proxy_pass http://127.0.0.1:8888;
          }
      }

    5. Example gunicorn.conf
      import multiprocessing

      bind = "127.0.0.1:8888" # ensure nginx passes to this port
      logfile = "/home/vagrant/gunicorn.log"
      workers = multiprocessing.cpu_count() * 2 + 1

  18. HEROKU
    >> heroku create

    Should now be able to browse to the url supplied as stdout fom command e.g.
    https://not-real-1234.herokuapp.com/. Note – not working yet – still need to deploy to new app

    >> git push heroku master

    Must then actually spin up the app:

    >> heroku ps:scale web=1

    A shortcut for opening is

    >> heroku open

  19. HEROKU
    Add redis support (after first deployment – otherwise

    ! No app specified.
    ! Run this command from an app folder or specify which app to use with --app APP.
    )
    >> heroku addons:create redistogo

    Note – need to register credit card to use any add-ons, even if free ones. Go to https://heroku.com/verify

Some other points: when developing on a different machine, I needed to supply my public key to heroku from that other machine (Permission denied (publickey) when deploying heroku code. fatal: The remote end hung up unexpectedly).

heroku keys:add ~/.ssh/id_rsa.pub

And the full sequence for upgrading your app after the prerequisites have been fulfilled is:

  1. git commit to local repo
  2. Then git push to heroku
  3. Then run heroku ps:scale web=1 again

And I had a problem when I switched from Python 2 to 3 with redis – my heroku push wouldn’t work. By looking at the logs (>> heroku logs –tail) I found that import imap wouldn’t work and searching on that generally found I needed a newer version of redis than I had specified foolishly in requirements.txt.

Complex good – complicated bad

In the Zen of Python we are taught that complex is better than complicated. Which is fair enough if we understand the terms as follows:

It is ok if something is complex so long as it is not complicated.

complex: composed of many interconnected parts; compound; composite

complicated: difficult to analyze or understand

Complex vs Complicated

Any decent web framework is going to be a bit complex becauise of all the moving parts it has to handle. But it should make sense and be logically structured enough to avoid being overly complicated.

Eclipse and PyDev on Utopic

I upgraded to Utopic (Utopic Unicorn a.k.a 14.10) and eclipse wouldn’t complete loading anymore. Solution:

Download latest plain vanilla Eclipse from the standard downloads page. And feel free to donate something too.

sudo su

chown -R root:root /home/username/eclipse && mv /home/username/Downloads/eclipse /opt

ln -s /opt/eclipse/eclipse /usr/local/bin/eclipse && exit

Start by running:

eclipse

It didn’t even break PyDev so my luck’s finally turning ;-).

https://www.tumblr.com/search/install+eclipse+ubuntu

IDLE3 as default for py files on Ubuntu

Yes – I know, there are better alternatives to IDLE out there, but I am used to it for quick and dirty changes to python files (I use eclipse + pydev for more serious work). And I am increasingly making the switch to Python 3. So when I double click on a py file, odds are I want to open it with IDLE for Python 3 not Python 2.

Start by making sure you have a desktop file like the following:

gksudo gedit /usr/share/applications/idle-python3.4.desktop

[Desktop Entry]
Name=IDLE (using Python-3.4)
Comment=Integrated Development Environment for Python (using Python-3.4)
Exec=/usr/bin/idle-python3.4
Icon=/usr/share/pixmaps/python3.4.xpm
Terminal=false
Type=Application
Categories=Application;Development;
StartupNotify=true

Then make the desktop entry the default for python files:

gedit ~/.local/share/application/mimeapps.list

[Default Applications]
text/w-python=idle-python3.4.desktop

Note – no trailing semi-colon.

And in Linux Mint:

Linux Mint:

ls /usr/share/applications/

identify appropriate .desktop file

gedit /usr/share/applications/defaults.list

add the appropriate .desktop file reference at the front of the python line as appropriate.

Saddest Programming Concept Ever

Python has spoiled me for other languages – I accept that – but I still wasn’t fully prepared for some of the horrors I discovered in Javascript. Which made the satiric article by James Mickens, “To Wash It All Away“, all the more enjoyable. Here is a slice I especially liked:

Much like C, JavaScript uses semicolons to terminate many kinds of statements. However, in JavaScript, if you forget a semicolon, the JavaScript parser can automatically insert semicolons where it thinks that semicolons might ought to possibly maybe go. This sounds really helpful until you realize that semicolons have semantic meaning. You can’t just scatter them around like you’re the Johnny Appleseed of punctuation. Automatically inserting semicolons into source code is like mishearing someone over a poor cell-phone connection, and then assuming that each of the dropped words should be replaced with the phrase “your mom.” This is a great way to create excitement in your interpersonal relationships, but it is not a good way to parse code. Some JavaScript libraries intentionally begin with an initial semicolon, to ensure that if the library is appended to another one (e.g., to save HTTP roundtrips during download), the JavaScript parser will not try to merge the last statement of the first library and the first statement of the second library into some kind of semicolon-riven statement party. Such an initial semicolon is called a “defensive semicolon.” That is the saddest programming concept that I’ve ever heard, and I am fluent in C++.

Nice deal on programming books

First a disclosure – I will be getting two free e-books for promoting the Packt Sale ;-). But I wouldn’t bother writing this up unless I thought Packt books would be of some value to me – so that makes it a genuine endorsement. For reference, here are the three I’m weighing up:

  1. Learning IPython for Interactive Computing and Data Visualization
  2. Git: Version Control for Everyone
  3. Responsive Web Design with jQuery

Buy One - Get One Free

Apparently the deal is:

  • Unlimited purchases during the offer period
  • Offer is automatically applied at checkout

I procrastinated a bit so there are only a couple of days left (ends 26th March). So be fast! Here’s the promotion link

My favourite eclipse settings

Preferences>PyDev>Editor>Code Style>Code Formatter – Tick “Right trim lines?”

Preferences>PyDev>Editor>Typing – untick After ‘(‘ indent to its level (indents by
tabs if unchecked)

Number of indentation levels to add: 1

Switch pylint off so pydev code analysis can do its job:

Preferences>PyDev>Editor>PyLint Untick “Use pylint?”.

Preferences>General>Editors>Text Editors>Annotations>Occurences (PyDev) set to #6BF459 so bright enough to see when very small but can read text on top.

Preferences>General>Editors>Text Editors>

Displayed tab width: 4

Tick “Insert spaces for tabs”
Tick “Show print margin” and set to 80.

Tick “Show line numbers”.

Misc Moonpig Provocation

Mark Dominus writes a thought-provoking and entertaining piece on Moonpig. Here are some snippets I found especially interesting and some brief comment:

On ORMs and relational databases:

Right now the principal value of ORM software seems to be if your program is too fast and you need it to be slower; the ORM is really good at that. Since speed was the only benefit the RDB was providing in the first place, you have just attached two large, complex, inflexible systems to your program and gotten nothing in return.

Can’t say I agree, as I find a lot of value in relational databases irrespective of speed (transactions, incredible flexibility of analysis), but still worth mulling over.

On built it yourself versus reuse:

What we should have done, instead of building our own object store, was use someone else’s object store. KiokuDB is frequently mentioned in this context. After I first gave this talk people asked “But why didn’t you use KiokuDB?” or, on hearing what we did do, said “That sounds a lot like KiokuDB”. I had to get Rik to remind me why we didn’t use KiokuDB. We had considered it, and decided to do our own not for technical but for political reasons. The CEO, having made the unpleasant decision to have me and Rik write a new billing system, wanted to see some progress. If she had asked us after the first week what we had accomplished, and we had said “Well, we spent a week figuring out KiokuDB,” her head might have exploded. Instead, we were able to say “We got the object store about three-quarters finished”. In the long run it was probably more expensive to do it ourselves, and the result was certainly not as good. But in the short run it kept the customer happy, and that is the most important thing; I say this entirely in earnest, without either sarcasm or bitterness.

(On the other hand, when I ran this article by Rik, he pointed out that KiokuDB had later become essentially unmaintained, and that had we used it he would have had to become the principal maintainer of a large, complex system which which he did not help design or implement. The Moonpig object store may be technically inferior, but Rik was with it from the beginning and understands it thoroughly.)

Interesting to note there is a genuine trade off with uncertainty in it. If you rely on something external, you may spend longer trying to find the right input to your project and it may be abandoned after you have heavily invested in it. A lot depends on the reusable component.

And finally, on inheritance:

  1. Object-oriented programming is centered around objects, which are encapsulated groups of related data, and around methods, which are opaque functions for operating on particular kinds of objects.
  2. OOP does not mandate any particular theory of inheritance, either single or multiple, class-based or prototype based, etc., and indeed, while all OOP systems have objects and methods that are pretty much the same, each has an inheritance system all its own.
  3. Over the past 30 years of OOP, many theories of inheritance have been tried, and all of them have had serious problems.
  4. If there were no alternative to inheritance, we would have to struggle on with inheritance. However, Roles are a good alternative to inheritance:
    • Every problem solved by inheritance is solved at least as well by Roles.
    • Many problems not solved at all by inheritance are solved by Roles.
    • Many problems introduced by inheritance do not arise when using Roles.
    • Roles introduce some of their own problems, but none of them are as bad as the problems introduced by inheritance.
  5. It’s time to give up on inheritance. It was worth a try; we tried it as hard as we could for thirty years or more. It didn’t work.
  6. I’m going to repeat that: Inheritance doesn’t work. It’s time to give up on it.

I have used inheritance successfully to prevent repeated code but it has only really fitted a few (important) places. Delegation is more flexible than inheritance, but inheritance can be a natural fit for some problems.

Anyway, an interesting read with lots more in it than I’ve commented on (e.g. avoiding floating point rounding errors etc). Read it yourself.

Python tuples are immutable right?

Tuples are immutable, unchangeable right? Well yes – sort of. The issue is easier to illustrate than describe so here goes. If we have a list, we can add new items. E.g.

>>> l = ["apple", "banana", "cucumber"]
>>> l
['apple', 'banana', 'cucumber']
>>> l.append("date")
>>> l
['apple', 'banana', 'cucumber', 'date']

If we have a tuple, we can’t change which objects are contained in the tuple:

>>> t = ("apple", "banana", "cucumber")
>>> t
('apple', 'banana', 'cucumber')
>>> t[3] = "date"

You get TypeError: ‘tuple’ object does not support item assignment

So once you have a tuple, nothing will change, right? It is even called immutable so end of story right? Not quite.

Well try this:

>>> a = ["apple",]
>>> b = ["banana",]
>>> c = ["cucumber",]
>>> t = (a,b,c)
>>> t
(['apple'], ['banana'], ['cucumber'])
>>> b.append("A new banana in my immutable tuple! WAT?!")
>>> t
(['apple'], ['banana', 'A new banana in my immutable tuple! WAT?!'], ['cucumber'])

From the official documentation:

“Objects whose value can change are said to be mutable; objects whose value is unchangeable once they are created are called immutable. (The value of an immutable container object that contains a reference to a mutable object can change when the latter’s value is changed; however the container is still considered immutable, because the collection of objects it contains cannot be changed. So, immutability is not strictly the same as having an unchangeable value, it is more subtle.)” (Data Model) [emphasis added]

Subtle – yes – well said. Which objects are contained in a tuple is fixed – but what those objects are is not. In practice, most tuples are containers of numbers and strings and this potential confusion never arises. Incidentally, even though tuples are immutable, they can’t be used dictionary keys if they contain mutable objects. Try it! E.g.

d = {((1,2), (3,4)): 100}
vs
d = {((1,2), [3,4]): 100}

For more useful explanation of how Python handles variables, objects etc check out: Drastically Improve Your Python: Understanding Python’s Execution Model. It sometimes introduces too many new ideas at once in the example code but is very helpful if you focus on the salient parts of the examples only.

[Added later] See Python tuples: immutable but potentially changing

Eclipse + PyDev problems on Ubuntu 13.10

I wanted the latest eclipse + the latest PyDev. The problem was that PyDev wouldn’t install properly or show up. And no amount of using the Install New Software GUI path helped ;-). The solution was to manually unpack PyDev and put it in the appropriate folder. Should have blogged on this at the time (a week ago) because now I can’t remember all the details. But the following helped: PyDev not showing up in Eclipse