An easy way of using SuperHELP

Many people find using the command line a bit daunting and Jupyter notebooks unfamiliar so here is another way of using SuperHELP that may be much easier.

  1. Put the following at the top of your script:

    import superhelp
    superhelp.this()


  2. Run the script
  3. See the advice
  4. Learn something; make changes 🙂

If you don’t want the default web output you can specify other output such as ‘cli’ (command line interface) or ‘md’ (markdown):

import superhelp
superhelp.this(output='md')

If you don’t want the default ‘Extra’ level of detail you can specify a different detail_level (‘Brief’ or ‘Main’) e.g.

import superhelp
superhelp.this(output='md', detail_level='Brief')

or:

import superhelp
superhelp.this(detail_level='Main')

Edited to remove reference to explicit file_path argument e.g. superhelp.this(__file__). It is not needed anymore thanks to the Python inspect library. Also displayer -> output and message_level -> detail_level.

Why SuperHELP for Python?

I created SuperHELP to make it easier for people to write good Python code. You can find the project at https://pypi.org/project/superhelp/

SuperHELP logo

To summarise the rationale for SuperHELP I have thought of a few taglines:

SuperHELP – Python help that really helps!

SuperHELP – Help for Humans!

and even

SuperHELP – Make Python Pythonic!

Some context might help explain:

Python has secured its position in the top tier of programming languages and more people than ever are learning to write Python. But, let’s be honest, a lot of Python code being written is not what it could be. Even an elegant language like Python can be written badly or in a way that is hard to read or maintain.

To make it easier for people to write good Python we are already well-served by IDEs from IDLE upwards. People have easy access to style linters and IDEs clearly signal syntax errors and basic mistakes like unused variables. But wouldn’t it be great if people could check their code to see if there are better, more Pythonic ways of doing things? Or to learn more about basic language features and Python data structures than the standard help can offer?

That’s where SuperHELP can play a role.

So what exactly is SuperHELP? Basically it is an advice engine. SuperHELP reads a snippet of Python and provides advice, warnings, and basic information based on what it finds. For example, it might notice a function docstring is missing and show a template for adding one. Or identify the use of a named tuple and explain how to add docstrings to individual fields or the named tuple as a whole.

The intention is to make sure that everyone, from beginners upwards, learns something useful. Even an advanced Python programmer might not appreciate the benefits of using functools.wraps when creating their own decorator. Or an experienced Java programmer might not realise that Python properties are a much better option than getters and setters.

So how can people use SuperHELP? For most people, the easiest way will be to open a binder web notebook and enter their code there.

Binder

Of course, because SuperHELP is a pip package https://pypi.org/project/superhelp/ it can be installed alongside Python on a machine and used directly from the terminal e.g.

shelp --code "people = ['Tomas', 'Sal']" --output html --detail-level Main

If the output chosen is html (the default) output looks like:

And if –output cli (command line interface i.e. terminal or console) is selected, output looks like:

In all likelihood, there will be other ways of making SuperHELP advice more readily available – probably through integration with other platforms and processes.

SuperHELP needs your help:

  • If you have any ideas, or the ability to help in some way, please contact me at superhelp@p-s.co.nz.
  • Spread the word about SuperHELP through your social networks.

Eclipse Neon with Pydev on Ubuntu 17.04

  1. Check whether 32 or 64 bit – System Settings > Details > Overview – then download from https://www.eclipse.org/downloads/
  2. Right click on eclipse-inst-linux32.tar.gz and select Extract Here
  3. cd into eclipse-installer/ then run ./eclipse-inst
  4. Choose top item (standard Java version of eclipse)
  5. Make desktop icon for launcher as per step 7 here – How to install Eclipse using its installer remembering to point to java-neon instead of java-mars etc. Drag file onto launcher and icon will add itself and be operational
  6. If you don’t see toolbars in eclipse modify exec line Exec=env SWT_GTK3=0 as per eclipse doesn’t work with ubuntu 16.04
  7. I also added pydev in the usual way using link to http://www.pydev.org/updates as per http://www.pydev.org/download.html

Messed up Thunderbird folders – sharks circling

We’ve all done it – messed up something so badly while trying to do something clever we’d sell our first-born just to get back to where we were (see https://xkcd.com/349/). And if we succeed at merely restoring the status quo, we’re pitifully grateful.

It all started with trying to find some lost photos that Shotwell couldn’t find the originals of. Presumably they had been linked to only and then the originals deleted leaving only the reference and the thumbnail behind. So here was the plan:

  1. Identify all email attachments which are images
  2. See if any of them have the same name as the missing images
  3. Open email based on date and sender to recover image

The good news is the plan worked for lots of the missing photos. Thanks to Python3, import mailbox, and import email. The bad news was when I opened Thunderbird the next day. The folder I had been working on was missing. So in addition to my missing photos I also had seemingly lost 1.8GB of emails.

tldr; 1) Close TB; 2) rename the missing folder in your file system and delete the .msf version; 3) Open TB; 4) Close TB; restore original name; 5) Open TB – success? Inspired by http://kb.mozillazine.org/Empty_folders

Now back to the original problem. But I should probably run a full backup first.

Simple flask app on heroku – all steps (almost)

Note – instructions assume Ubuntu Linux.

See Getting Started with Python on Heroku (Flask) for the official instructions. The instructions below tackle things differently and include redis-specific steps.

Don’t need postgresql for my app even though needed for heroku demo app. Using redis for simple key-value store.

Main reason for each step is indicated in bold at start. There are lots of steps but there are lots of things being achieved. And each purpose only requires a few steps so probably hard to streamline any further.

  1. APP & BEST PRACTICE
    >> sudo apt-get install python3 python3-pip python-virtualenv git ruby redis-server redis-tools
  2. HEROKU
    Get free heroku account
  3. HEROKU
    Install heroku toolbelt Heroku setup. Sets up virtualenvwrapper for you too (one less thing to figure out)
  4. HEROKU
    Do the once-ever authentication
    >> heroku login
  5. APP
    Make project folder e.g.
    >> mkdir ~/projects/myproj
  6. APP
    >> cd ~/projects/myproj
  7. HEROKU
    >> echo “web: python main.py” > Procfile
  8. HEROKU & BEST PRACTICE
    >> git init
  9. HEROKU & BEST PRACTICE
    >> mkvirtualenv sticky

    So requirements for specific project can be separated from other project – lets heroku identify actual requirements. Normally “workon sticky” thereafter; deactivate to exit virtual env

  10. APP
    >> pip3 install flask
    Note – installed within virtualenv
  11. HEROKU
    Save the following as requirements.txt – needed by heroku so it knows the dependencies. Update version of redis as appropriate. gunicorn is a better approach than the flask test server
    flask
    gunicorn
    redis==2.10.3
  12. HEROKU
    So we can use Python 3.4 instead of the current default of 2.7:
    >> echo “python-3.4.3” > runtime.txt
  13. APP & HEROKU

    Make a toy app to get started from.

    Note – modify the standard demo flask app to add a port to ease eventual heroku deployment. Otherwise the app will fail because of a problem with the port when running

    heroku ps:scale web=1

    Starting process with command `python main.py`
    ...
    Web process failed to bind to $PORT within 60 seconds of launch

    Here is an example (may need updating if flask changes):

    import os
    from flask import Flask
    app = Flask(__name__)

    @app.route("/")
    def hello():
        return "Hello World!"

    if __name__ == "__main__":
        port = int(os.environ.get("PORT", 33507))
        app.run(host='0.0.0.0', port=port)

  14. BEST PRACTICE
    >> deactivate
  15. Make a module to make it easier to work with redis – let’s call it store.py:

    import os
    import urllib
    import redis

    url = urllib.parse.urlparse(os.environ.get('REDISTOGO_URL',
        'redis://localhost:6379'))
    redis = redis.Redis(host=url.hostname, port=url.port, db=0,
        password=url.password)

    We can then use redis like this:
    from store import redis

  16. APP
    Keep building app locally. The following is good for redis: Redis docs. And flasks docs are always good: Flask Docs – Minimal Application
  17. HEROKU & BEST PRACTICE

    Before deploying to production:

    1. Update git otherwise you’ll be deploying old code – heroku uses git for deployment
    2. set app.debug to False (although no rush when just getting started and not expecting the app to get hit much)
    3. probably switch to gunicorn sooner or later (will need to change ProcFile to
      web: gunicorn main:app --workers $WEB_CONCURRENCY
      )
    4. Example nginx.conf:

      # As long as /etc/nginx/sites-enable/ points to
      # this conf file nginx can use it to work with
      # the server_name defined (the name of the file
      # doesn't matter - only the server_name setting)
      # sudo ln -s /home/vagrant/src/nginx.conf ...
      #     ... /etc/nginx/sites-enabled/myproj.com
      # Confirm this link is correct
      # e.g. less /etc/nginx/sites-enabled/myproj.com

      server {
          listen 80;
          server_name localhost;

          location /static { # static content is

              # handled directly by NGINX which means
              # the nginx user (www-data) will need
              # read permissions to this folder
              root /home/vagrant/src;

          }

          location / { # all else passed to Gunicorn to handle
              # Pass to wherever I bind Gunicorn to serve to
              # Only gunicorn needs rights to read, write,
              # and execute scripts in the app folders
              proxy_pass http://127.0.0.1:8888;
          }
      }

    5. Example gunicorn.conf
      import multiprocessing

      bind = "127.0.0.1:8888" # ensure nginx passes to this port
      logfile = "/home/vagrant/gunicorn.log"
      workers = multiprocessing.cpu_count() * 2 + 1

  18. HEROKU
    >> heroku create

    Should now be able to browse to the url supplied as stdout fom command e.g.
    https://not-real-1234.herokuapp.com/. Note – not working yet – still need to deploy to new app

    >> git push heroku master

    Must then actually spin up the app:

    >> heroku ps:scale web=1

    A shortcut for opening is

    >> heroku open

  19. HEROKU
    Add redis support (after first deployment – otherwise

    ! No app specified.
    ! Run this command from an app folder or specify which app to use with --app APP.
    )
    >> heroku addons:create redistogo

    Note – need to register credit card to use any add-ons, even if free ones. Go to https://heroku.com/verify

Some other points: when developing on a different machine, I needed to supply my public key to heroku from that other machine (Permission denied (publickey) when deploying heroku code. fatal: The remote end hung up unexpectedly).

heroku keys:add ~/.ssh/id_rsa.pub

And the full sequence for upgrading your app after the prerequisites have been fulfilled is:

  1. git commit to local repo
  2. Then git push to heroku
  3. Then run heroku ps:scale web=1 again

And I had a problem when I switched from Python 2 to 3 with redis – my heroku push wouldn’t work. By looking at the logs (>> heroku logs –tail) I found that import imap wouldn’t work and searching on that generally found I needed a newer version of redis than I had specified foolishly in requirements.txt.

Listing Python scripts changed most recently

find / -name *.py -type f -mtime -60 -printf '%TY-%Tm-%Td %TT %p\n' 2>/dev/null | sort -r

Command/options Explanation
find Not as fast as locate but has many advantages. See locate vs find: usage, pros and cons of each other
-name *.py Only find python scripts
-mtime -60 -mtime is days
-printf ‘%TY-%Tm-%Td %TT %p\n’ Displays output with easy-to-read dates
2>/dev/null Pumps all the annoying output about lacking permissions into /dev/null i.e. ignores them
| Pipes output through to be sorted
sort -r Sort so most recent are at top

Also see
How to find recently modified files on Linux

Complex good – complicated bad

In the Zen of Python we are taught that complex is better than complicated. Which is fair enough if we understand the terms as follows:

It is ok if something is complex so long as it is not complicated.

complex: composed of many interconnected parts; compound; composite

complicated: difficult to analyze or understand

Complex vs Complicated

Any decent web framework is going to be a bit complex becauise of all the moving parts it has to handle. But it should make sense and be logically structured enough to avoid being overly complicated.

Eclipse and PyDev on Utopic

I upgraded to Utopic (Utopic Unicorn a.k.a 14.10) and eclipse wouldn’t complete loading anymore. Solution:

Download latest plain vanilla Eclipse from the standard downloads page. And feel free to donate something too.

sudo su

chown -R root:root /home/username/eclipse && mv /home/username/Downloads/eclipse /opt

ln -s /opt/eclipse/eclipse /usr/local/bin/eclipse && exit

Start by running:

eclipse

It didn’t even break PyDev so my luck’s finally turning ;-).

https://www.tumblr.com/search/install+eclipse+ubuntu

Deploying simple flask app on heroku

I’m now a fan of Heroku. How cool is it to be able to deploy a Python app to free hosting?!

Blackbox flask app on Heroku

But in spite of great docs at Getting Started with Python on Heroku there were a few issues I had to handle. The main problems were because the instructions assumed you wanted to start with their demo app and not your own – which meant that they only explained things like requirements.txt and Procfile after you needed to have already made them (they were already present in the demo version).

Note – I am already familiar with git so I don’t explain that here – see Starting a simple Flask app with Heroku for more fleshed-out instructions.

Anyway, here is what I needed to do at the start:

1) Change the app.run(host='0.0.0.0') line to
port = int(os.environ.get("PORT", 33507))
app.run(host='0.0.0.0', port=port)

Otherwise the app would fail because of a problem with the port when I ran

heroku ps:scale web=1

Starting process with command `python main.py`
...
Web process failed to bind to $PORT within 60 seconds of launch

2) I really needed to use virtualenvwrapper and create a requirements.txt file e.g.

cd <folder with code in it>
mkvirtualenv blackbox

Otherwise heroku wouldn’t know what dependencies my app needed fulfilled to work successfully.

To update requirements.txt after changes,

cd <folder with code in it>
workon blackbox
pip freeze > requirements.txt
deactivate blackbox

3) I needed to make a Procfile:

web: python main.py

Note, this was a toy flask app so not using gunicorn etc. Probably should look into that later:

4) Setting debug mode off probably isn’t essential for deployment but probably a good idea anyway: app.debug = False before deploying.

Some other points: when developing on a different machine, I needed to supply my public key to heroku from that other machine (Permission denied (publickey) when deploying heroku code. fatal: The remote end hung up unexpectedly).

heroku keys:add ~/.ssh/id_rsa.pub

And the full sequence for upgrading your app after the prerequisites have been fulfilled is:

  1. Make sure you have the port set for heroku
  2. Then git to local repo
  3. Then git push to heroku
  4. Then run heroku ps:scale web=1 again
  5. Revert from the heroku port back to local for local testing and dev.

heroku addons:add redistogo

To add redis support – NB need to register credit card to use any add-ons, even if free ones.

IDLE3 as default for py files on Ubuntu

Yes – I know, there are better alternatives to IDLE out there, but I am used to it for quick and dirty changes to python files (I use eclipse + pydev for more serious work). And I am increasingly making the switch to Python 3. So when I double click on a py file, odds are I want to open it with IDLE for Python 3 not Python 2.

Start by making sure you have a desktop file like the following:

gksudo gedit /usr/share/applications/idle-python3.4.desktop

[Desktop Entry]
Name=IDLE (using Python-3.4)
Comment=Integrated Development Environment for Python (using Python-3.4)
Exec=/usr/bin/idle-python3.4
Icon=/usr/share/pixmaps/python3.4.xpm
Terminal=false
Type=Application
Categories=Application;Development;
StartupNotify=true

Then make the desktop entry the default for python files:

gedit ~/.local/share/application/mimeapps.list

[Default Applications]
text/w-python=idle-python3.4.desktop

Note – no trailing semi-colon.

And in Linux Mint:

Linux Mint:

ls /usr/share/applications/

identify appropriate .desktop file

gedit /usr/share/applications/defaults.list

add the appropriate .desktop file reference at the front of the python line as appropriate.