Problem upgrading Dokuwiki after long neglect

When you upgrade Dokuwiki (brilliant wiki BTW) it does it in-place. Unfortunately, I hadn’t upgraded for about 5 years in spite of numerous warnings so the deprecated.php file bit me by assuming there were specific files in my instance of dokuwiki that needed deprecating and breaking when the system was so old they weren’t ;-). Once I figured the problem out the solution was simply to comment out the appropriate parts of deprecated.php. After all, there is no need to deprecate something you don’t even have.

Reminder: include

ini_set('display_errors', 1); ini_set('display_startup_errors', 1); error_reporting(E_ALL);

in index.php when trying to identify problems running a PHP app. And remove it afterwards.

Simple flask app on heroku – all steps (almost)

Note – instructions assume Ubuntu Linux.

See Getting Started with Python on Heroku (Flask) for the official instructions. The instructions below tackle things differently and include redis-specific steps.

Don’t need postgresql for my app even though needed for heroku demo app. Using redis for simple key-value store.

Main reason for each step is indicated in bold at start. There are lots of steps but there are lots of things being achieved. And each purpose only requires a few steps so probably hard to streamline any further.

  1. APP & BEST PRACTICE
    >> sudo apt-get install python3 python3-pip python-virtualenv git ruby redis-server redis-tools
  2. HEROKU
    Get free heroku account
  3. HEROKU
    Install heroku toolbelt Heroku setup. Sets up virtualenvwrapper for you too (one less thing to figure out)
  4. HEROKU
    Do the once-ever authentication
    >> heroku login
  5. APP
    Make project folder e.g.
    >> mkdir ~/projects/myproj
  6. APP
    >> cd ~/projects/myproj
  7. HEROKU
    >> echo “web: python main.py” > Procfile
  8. HEROKU & BEST PRACTICE
    >> git init
  9. HEROKU & BEST PRACTICE
    >> mkvirtualenv sticky

    So requirements for specific project can be separated from other project – lets heroku identify actual requirements. Normally “workon sticky” thereafter; deactivate to exit virtual env

  10. APP
    >> pip3 install flask
    Note – installed within virtualenv
  11. HEROKU
    Save the following as requirements.txt – needed by heroku so it knows the dependencies. Update version of redis as appropriate. gunicorn is a better approach than the flask test server
    flask
    gunicorn
    redis==2.10.3
  12. HEROKU
    So we can use Python 3.4 instead of the current default of 2.7:
    >> echo “python-3.4.3” > runtime.txt
  13. APP & HEROKU

    Make a toy app to get started from.

    Note – modify the standard demo flask app to add a port to ease eventual heroku deployment. Otherwise the app will fail because of a problem with the port when running

    heroku ps:scale web=1

    Starting process with command `python main.py`
    ...
    Web process failed to bind to $PORT within 60 seconds of launch

    Here is an example (may need updating if flask changes):

    import os
    from flask import Flask
    app = Flask(__name__)

    @app.route("/")
    def hello():
        return "Hello World!"

    if __name__ == "__main__":
        port = int(os.environ.get("PORT", 33507))
        app.run(host='0.0.0.0', port=port)

  14. BEST PRACTICE
    >> deactivate
  15. Make a module to make it easier to work with redis – let’s call it store.py:

    import os
    import urllib
    import redis

    url = urllib.parse.urlparse(os.environ.get('REDISTOGO_URL',
        'redis://localhost:6379'))
    redis = redis.Redis(host=url.hostname, port=url.port, db=0,
        password=url.password)

    We can then use redis like this:
    from store import redis

  16. APP
    Keep building app locally. The following is good for redis: Redis docs. And flasks docs are always good: Flask Docs – Minimal Application
  17. HEROKU & BEST PRACTICE

    Before deploying to production:

    1. Update git otherwise you’ll be deploying old code – heroku uses git for deployment
    2. set app.debug to False (although no rush when just getting started and not expecting the app to get hit much)
    3. probably switch to gunicorn sooner or later (will need to change ProcFile to
      web: gunicorn main:app --workers $WEB_CONCURRENCY
      )
    4. Example nginx.conf:

      # As long as /etc/nginx/sites-enable/ points to
      # this conf file nginx can use it to work with
      # the server_name defined (the name of the file
      # doesn't matter - only the server_name setting)
      # sudo ln -s /home/vagrant/src/nginx.conf ...
      #     ... /etc/nginx/sites-enabled/myproj.com
      # Confirm this link is correct
      # e.g. less /etc/nginx/sites-enabled/myproj.com

      server {
          listen 80;
          server_name localhost;

          location /static { # static content is

              # handled directly by NGINX which means
              # the nginx user (www-data) will need
              # read permissions to this folder
              root /home/vagrant/src;

          }

          location / { # all else passed to Gunicorn to handle
              # Pass to wherever I bind Gunicorn to serve to
              # Only gunicorn needs rights to read, write,
              # and execute scripts in the app folders
              proxy_pass http://127.0.0.1:8888;
          }
      }

    5. Example gunicorn.conf
      import multiprocessing

      bind = "127.0.0.1:8888" # ensure nginx passes to this port
      logfile = "/home/vagrant/gunicorn.log"
      workers = multiprocessing.cpu_count() * 2 + 1

  18. HEROKU
    >> heroku create

    Should now be able to browse to the url supplied as stdout fom command e.g.
    https://not-real-1234.herokuapp.com/. Note – not working yet – still need to deploy to new app

    >> git push heroku master

    Must then actually spin up the app:

    >> heroku ps:scale web=1

    A shortcut for opening is

    >> heroku open

  19. HEROKU
    Add redis support (after first deployment – otherwise

    ! No app specified.
    ! Run this command from an app folder or specify which app to use with --app APP.
    )
    >> heroku addons:create redistogo

    Note – need to register credit card to use any add-ons, even if free ones. Go to https://heroku.com/verify

Some other points: when developing on a different machine, I needed to supply my public key to heroku from that other machine (Permission denied (publickey) when deploying heroku code. fatal: The remote end hung up unexpectedly).

heroku keys:add ~/.ssh/id_rsa.pub

And the full sequence for upgrading your app after the prerequisites have been fulfilled is:

  1. git commit to local repo
  2. Then git push to heroku
  3. Then run heroku ps:scale web=1 again

And I had a problem when I switched from Python 2 to 3 with redis – my heroku push wouldn’t work. By looking at the logs (>> heroku logs –tail) I found that import imap wouldn’t work and searching on that generally found I needed a newer version of redis than I had specified foolishly in requirements.txt.

Complex good – complicated bad

In the Zen of Python we are taught that complex is better than complicated. Which is fair enough if we understand the terms as follows:

It is ok if something is complex so long as it is not complicated.

complex: composed of many interconnected parts; compound; composite

complicated: difficult to analyze or understand

Complex vs Complicated

Any decent web framework is going to be a bit complex becauise of all the moving parts it has to handle. But it should make sense and be logically structured enough to avoid being overly complicated.

Deploying simple flask app on heroku

I’m now a fan of Heroku. How cool is it to be able to deploy a Python app to free hosting?!

Blackbox flask app on Heroku

But in spite of great docs at Getting Started with Python on Heroku there were a few issues I had to handle. The main problems were because the instructions assumed you wanted to start with their demo app and not your own – which meant that they only explained things like requirements.txt and Procfile after you needed to have already made them (they were already present in the demo version).

Note – I am already familiar with git so I don’t explain that here – see Starting a simple Flask app with Heroku for more fleshed-out instructions.

Anyway, here is what I needed to do at the start:

1) Change the app.run(host='0.0.0.0') line to
port = int(os.environ.get("PORT", 33507))
app.run(host='0.0.0.0', port=port)

Otherwise the app would fail because of a problem with the port when I ran

heroku ps:scale web=1

Starting process with command `python main.py`
...
Web process failed to bind to $PORT within 60 seconds of launch

2) I really needed to use virtualenvwrapper and create a requirements.txt file e.g.

cd <folder with code in it>
mkvirtualenv blackbox

Otherwise heroku wouldn’t know what dependencies my app needed fulfilled to work successfully.

To update requirements.txt after changes,

cd <folder with code in it>
workon blackbox
pip freeze > requirements.txt
deactivate blackbox

3) I needed to make a Procfile:

web: python main.py

Note, this was a toy flask app so not using gunicorn etc. Probably should look into that later:

4) Setting debug mode off probably isn’t essential for deployment but probably a good idea anyway: app.debug = False before deploying.

Some other points: when developing on a different machine, I needed to supply my public key to heroku from that other machine (Permission denied (publickey) when deploying heroku code. fatal: The remote end hung up unexpectedly).

heroku keys:add ~/.ssh/id_rsa.pub

And the full sequence for upgrading your app after the prerequisites have been fulfilled is:

  1. Make sure you have the port set for heroku
  2. Then git to local repo
  3. Then git push to heroku
  4. Then run heroku ps:scale web=1 again
  5. Revert from the heroku port back to local for local testing and dev.

heroku addons:add redistogo

To add redis support – NB need to register credit card to use any add-ons, even if free ones.

Saddest Programming Concept Ever

Python has spoiled me for other languages – I accept that – but I still wasn’t fully prepared for some of the horrors I discovered in Javascript. Which made the satiric article by James Mickens, “To Wash It All Away“, all the more enjoyable. Here is a slice I especially liked:

Much like C, JavaScript uses semicolons to terminate many kinds of statements. However, in JavaScript, if you forget a semicolon, the JavaScript parser can automatically insert semicolons where it thinks that semicolons might ought to possibly maybe go. This sounds really helpful until you realize that semicolons have semantic meaning. You can’t just scatter them around like you’re the Johnny Appleseed of punctuation. Automatically inserting semicolons into source code is like mishearing someone over a poor cell-phone connection, and then assuming that each of the dropped words should be replaced with the phrase “your mom.” This is a great way to create excitement in your interpersonal relationships, but it is not a good way to parse code. Some JavaScript libraries intentionally begin with an initial semicolon, to ensure that if the library is appended to another one (e.g., to save HTTP roundtrips during download), the JavaScript parser will not try to merge the last statement of the first library and the first statement of the second library into some kind of semicolon-riven statement party. Such an initial semicolon is called a “defensive semicolon.” That is the saddest programming concept that I’ve ever heard, and I am fluent in C++.

Xdebug restored on Lucid for Netbeans

[Update for Ubuntu 14.10 and Netbeans 7.4

php.ini was in /etc/php5/mods-available/

Needed the following:

zend_extension = "/usr/lib/php5/20121212+lfs/xdebug.so"
xdebug.remote_autostart=on
xdebug.remote_enable=on
xdebug.remote_enable=1
xdebug.remote_handler="dbgp"
xdebug.remote_host=localhost
xdebug.remote_port=9000
xdebug.remote_mode=req
xdebug.idekey="netbeans-xdebug"

]

When I upgraded from Netbeans 6.7 to 6.8 I lost the ability to debug line by line. Later, I upgraded to 6.9 in the hope that this would fix it. It didn’t but the Netbeans error message for xdebug gave me a better steer towards useful resources. My system is Ubuntu Lucid (10.04) so I looked at http://wiki.netbeans.org/HowToConfigureXDebug#How_to_on_Linux. The answer for me was to edit a line in an xdebug file (I had previously added recommended lines * to php.ini to no avail – at least, not in isolation).

cd /etc/php5/conf.d

sudo gedit xdebug.ini

The first line of xdebug.ini was _almost_ correct 😉

Because I was using PHP 5.3 the key was zend_extension rather than zend_extension_ts. That part was right. But the path was not quite correct. Instead of:


zend_extension=/usr/lib/php5/2006 ... +lfs/xdebug.so

it needed to be (in my case):

zend_extension=/usr/lib/php5/20090626+lfs/xdebug.so

[Update – now /usr/lib/php5/20121212+lfs/xdebug.so]

Make sure the path actually exists and hasn’t become orphaned during an upgrade.

I then stopped and started apache and restarted Netbeans to be safe.

sudo /etc/init.d/apache2 stop

sudo /etc/init.d/apache2 start

—–

* Here are the lines you will probably already have added:

xdebug.remote_enable=1
xdebug.remote_handler=dbgp
xdebug.remote_mode=req
xdebug.remote_host=127.0.0.1
xdebug.remote_port=9000

SVG and the Future of Web Development

Sitepoint (Meitar Moscovitz) has an interesting article on SVG – SVG Is The Future Of Application Development. The blog item introduced me to the Sun Labs Lively Kernel project amongst other things. I have worked with SVG before – initially just making clipart in Inkscape, and more recently doing some geomapping. I found it clean and simple. Using SVG also meant I could develop my own tools for working with it which is what open standards is all about.

Which brings me to some of the comments on Moscovitz’s article. Some responses suggested that SVG was redundant because Adobe had shifted in another direction. The possibility of a single vendor abandoning a technology others have come to rely on is precisely why I think we should support open standards. We should support them in preference to proprietary systems unless the alternative is compelling (I have yet to find a good alternative to MS Access for some purposes, for example, but I am keeping an eye open). In addition to safeguarding against abandonment, using open standards increases competition in the tools space. MS IE 6 is an example of what can happen when a dominant vendor feels a lack of competition.

Other comments related to the lack of support for SVG in IE (for the foreseeable future). The argument was that if IE didn’t support it then there was no point using it. I don’t think it is that simple. IE is still dominant but there could be many cases where web application development could assume that a standards-compliant web browser would be used (e.g. kiosks; an intranet; a web application aimed at home users which the home users were highly motivated to use). Anyway, the future could surprise us. Who ever thought they would see Microsoft support ODF or portray itself as open-source friendly!?

I won’t be using this technology myself at this stage but I wouldn’t be surprised if it is something I need to add to my toolkit in 5 years time.

Apache and PHP on Windows XP (for Development Only)

Overview

Instead of allowing for all possibilities, the emphasis here is on providing a simple set of instructions that always work (fingers crossed). From this foundation, experimentation can occur in a controlled manner.

Step 1 – Get Apache 2.x.x installed (without PHP modules) – tested on 2.0.52 and 2.2.3
Step 2 – Get PHP 5.x.x working – tested on 5.1.4, and 5.2.0
Step 3 – Get mysqli module working (assumed to need MySQL) – tested on 4.1.9

Gotcha: in the Apache config file, change backslashes to forwards slashes – e.g “C:/PHP”. In the PHP config file leave backslashes as they are: “C:\Program Files etc”.

1 – Apache

Uninstall any previous version after backing up httpd.conf somewhere.

Install Apache (e.g. 2.2.2)

Having trouble finding a Windows binary? Check this out:

http://www.apachelounge.com/

Annoyed at how uncertain the process of getting a Windows binary can be? Read:

http://www.apachelounge.com/forum/viewtopic.php?p=1241#1241

Edit httpd.conf esp:

  • ServerAdmin webmaster@p-s.co.nz
  • ServerName localhost:80
  • DocumentRoot “C:/Projects” (or wherever you want to store web pages)
  • #
    # This should be changed to whatever you set DocumentRoot to.
    #
  • DirectoryIndex ………. index.php (usually index.html etc first)
  • # WINDOWS REQUIREMENTS ###
    EnableSendfile off
    EnableMMAP off
    Win32DisableAcceptEx
    ##############################

Make sure Apache is running BEFORE adding PHP functionality.

apache OK
If you see a red dot instead of a green triangle, apache is not running successfully.

If you have problems check out:

http://www.apachelounge.com/

2 – PHP

Create a “C:\PHP” folder

Extract the PHP windows binary file (preserving folder structure) to “C:\PHP”

Do NOT use the windows installer, just the zip file – extract to C:\Program Files\PHP maintaining the folders.  All dlls are correctly brought across (unlike installer).

Save “php.ini-recommended” as “php.ini” in “C:\PHP”.
“13. How do I know if my “php.ini” is being found and read?

It seems like it isn’t as my changes aren’t being implemented.

To be sure your “php.ini” is being read by PHP, make a call to phpinfo() and near the top will be a listing called “Configuration File (php.ini)”. This will tell you where PHP is looking for “php.ini” and whether or not it’s being read. If just a directory PATH exists then it’s not being read and you should put your “php.ini” in that directory. If “php.ini” is included within the PATH then it is being read. If “php.ini” is being read and you’re running PHP as a module, then be sure to restart your web server after making changes to “php.ini” http://www.php.net/manual/en/faq.installation.php

Anyhow, you need to edit php.ini, esp:

  • display_errors = On (NB this machine must only be used as a development machine)
  • include_path = “.;c:\php\pear” (NB the . tells PHP to look in its own folder first – don’t overlook it!)
  • extension_dir = “c:\php\ext”

Edit httpd.conf, so it will run PHP as a module:

  • # PHP ##########################
    LoadModule php5_module “c:/php/php5apache2.dll”
    AddType application/x-httpd-php .php
    PHPIniDir “c:/php”
    ###############################

The location of the loaded modules seems to result in numerous problems for people. Sometimes it is not because a file is missing, but because the file needs a companion file to operate. The Apache error message is probably misleading.

PHP searches for php.ini in the following locations (in
order):

* PHPIniDir directive (Apache 2 module only)
* HKEY_LOCAL_MACHINE\SOFTWARE\PHP\IniFilePath
* The PHPRC environment variable (possibly only relevant for CGI)
* Directory of PHP (for CLI), or the web server’s directory (for
SAPI modules)
* Windows directory (c:\windows or c:\winnt)

If you are running Apache 2, the simpler option is to use the
PHPIniDir directive

NB set the PATH in Windows to look at “c:\php”. Set this in Control Panel> System> Advanced> Environment Variables> System Variables> Path.

Plus set the PHPRC variable:

  • Go to Control Panel and open the System icon (Start -> Settings -> Control Panel -> System, or just Start -> Control Panel -> System for Windows XP/2003)
  • Go to the Advanced tab
  • Click on the ‘Environment Variables’ button
  • Look into the ‘System variables’ pane
  • Click on ‘New’ and enter ‘PHPRC’ as the variable name and the directory where “php.ini” is located as the variable value (e.g. “c:\php”
  • Press OK and restart your computer

A reboot is needed for this to take effect. This will enable PHP to locate files such as “php5ts.dll”

“NB the order of the paths in “Path” will be the order in which Windows looks for files. If you have multiple versions then make sure the correct version will be found first. Depending on your setup there is a good chance php.ini, or php5apache2.dll in “C:\Windows\System32” will take precedence over the one in “c:\php” if you have multiple.

One possible gotcha – if you have put copies of files such as php5apache2.dll, libmysql.dll, and php5ts.dll in “c:\Windows\System32” or “c:\Windows” in previous installation attempts, you will need to remove them (reboot in Safe Mode using F8) or overwrite them with newer versions. It seems that only files from the same PHP installation should be used in conjunction with each other.

“Don’t mix up your installation with DLL files from different PHP versions. You have the only choice to use the DLL’s and extensions that ship with your downloaded PHP version.”

http://www.php.net/manual/en/install.windows.apache2.php

Although lots of bulletin boards recommend putting files in the WINDOWS or SYSTEM32 directories, don’t.

“Previous editions of the manual suggest moving various ini and DLL files into your SYSTEM (i.e. “C:\WINDOWS”) folder and while this simplifies the installation procedure it makes upgrading difficult. We advise you remove all of these files (like “php.ini” and PHP-related DLLs from the Windows SYSTEM folder) before moving on with a new PHP installation. Be sure to backup these files as you might break the entire system. The old “php.ini” might be useful in setting up the new PHP as well. And as you’ll soon learn, the preferred method for installing PHP is to keep all PHP related files in one directory and have this directory available to your systems PATH.”

“The PHP manual used to promote the copying of files into the Windows system directory, this is because this directory “c:\Windows”, “c:\WINNT”, etc.) is by default in the systems PATH. Copying files into the Windows system directory has long since been deprecated and may cause problems.”

http://www.php.net/manual/en/faq.installation.php

Restart Apache from the Apache Service Taskbar icon in the tray so it will take on board the changes to the Apache and PHP config files.

Create a small file with as the contents, and save it as “c:\projects\testphp.php”.

In your browser type in “http://localhost/testphp.php” and enter. Hopefully, you will see the phrase “PHP is working – Phew!!!”.

3 – MySQL

Add the following to php.ini:

extension=php_mysqli.dll
extension=php_mcrypt.dll (nothing to do with MySQL but I usually add it for encryption needs while I’m there!)
mysqli.default_host = localhost

mysqli.default_user = xxxxx (whatever your user is)

Ref: http://www.php.net/manual/en/install.windows.extensions.php

PHP >= 5.0.0, requires “libmysql.dll” (libmysqli.dll in PHP <= 5.0.2) (bundled) PLUS: 2. COPY (do not move) libmysql.dll into the Apache2\bin directory (Even though there's no reference to this file in ANY error message I or my students have received, I have spent an interminable amount of time over the past seven weeks chasing this particular problem. All the references are to missing mysql functions when called from a PHP script and, literally, the only "mysql" dll that was not available in the extension_dir was libmysql.dll, so we tried putting libmysql.dll into the php\ext directory with no mas. Then, we put it into the Apache2\bin directory and we had connectivity! This goes against my natural inclinations to have a tidy installation, but it works, so I've passed it out to my students and the folks at the local LAMP that I've helped with their installations.) HTH. drj http://www.notestips.com/80256B3A007F2692/1/TAIO-64MEGD