I just came across the Ada Initiative – a cool project supporting women in open technology and culture. The name is quite significant too:
The Ada Initiative is named for Countess Ada Lovelace, widely acknowledged as the world’s first computer programmer. She is also the world’s first woman open source programmer.
http://adainitiative.org/about-us/
Check it out. They are also looking for supporters too.
Finally a video on topic which is short enough to share with others yet informative enough to be useful (and not just preaching to the choir): Clay Shirky: Why SOPA is a bad idea
Here are 4 articles on programming I really liked:
As an independent software developer I am 100% opposed to software patents. Software patents are unlikely to help my business and are much more likely to be used against it by a multinational software company, or worse, a parasitic patent troll. Here are some basic objections to software patents:
Here are some links with a local emphasis for further reading:
This post isn’t so much aimed at helping others (although maybe it will) as making my next upgrade easier. The intention is to add every tweak etc as I do them or remember them.
Here is what it looks like:
And here is the custom menu I made for the launcher:
See 15 Things I Did After Installing New Ubuntu 11.10 Oneiric Ocelot for how to make those and lots, lots more.
Initial video problems resolved
To begin with, I couldn’t even boot into Unity or anything. I had to boot into recovery mode and clear the path in various ways. I can’t remember exactly what I did but it involved wiping existing video drivers so I could even boot into Ubuntu 2D.
Had to get into recovery mode and purge the existing drivers. IIRC the file system was treated as being read-only and I had to remount as step 1 (nice and quick) before I could successfully make the other changes from root.
sudo apt-get remove --purge nvidia* fglrx*
[SOLVED] Going back to 10.10, Oneiric is a bust for me
I probably wiped /etc/X11/xorg.conf as well.
Once into Ubuntu 2D I was able to add proprietary drivers for nVidia and some combination worked and I was able to get Unity 3D working.
Booting into correct system
Want grub (grub2) to load my main system, not the test version I put in a new partition:
sudo grub-install /dev/sda
To change other grub settings edit /etc/boot/grub:
sudo gedit /etc/boot/grub
then edit the line –
GRUB_DEFAULT=0
to
GRUB_DEFAULT=4
save the file, then run
“sudo update-grub”
http://www.linuxquestions.org/questions/linux-software-2/changing-default-os-in-grub2-765011/
Re-ordering icons on launcher
In Unity 3D just drag them to the right then up or down and back in. In Unity 2D, hold for two seconds, then you can drag them around (http://ubuntuforums.org/showthread.php?t=1709136).
Removing overlay scroll bars
sudo apt-get remove overlay-scrollbar liboverlay-scrollbar-0.1-0
Things To Tweak / Fix After Installing Ubuntu 11.04 Natty Narwhal
Adding the system load indicator
Network, Memory And CPU Usage Indicator For Ubuntu: System Load Indicator
Adding a memory load indicator (useful when running a virtual OS at same time)
Things To Tweak / Fix After Installing Ubuntu 11.04 Natty Narwhal
Restore debugging for Netbeans
sudo gedit /etc/php5/apache2/php.ini
Find php.ini using phpinfo() then edit it thus:
xdebug.remote_enable=1
xdebug.remote_handler=dbgp
xdebug.remote_mode=req
xdebug.remote_host=localhost
xdebug.remote_port=9000
Next
sudo gedit /etc/apache2/apache2.conf
ServerRoot "/etc/apache2"
ServerName localhost
See Xdebug restored on Lucid for Netbeans
sudo /etc/init.d/apache2 stop
sudo /etc/init.d/apache2 start
sudo gedit /etc/php5/conf.d/xdebug.ini
xdebug.ini was correct:
zend_extension=/usr/lib/php5/20090626+lfs/xdebug.so
This folder did exist.
SQLyog problems
Managed to get the automated synchronisation working by deleting ~/.wine, reinstalling WINE, and reinstalling SQLyog (having saved
but the main system was flaky and crashed when displaying the results of a connection (amongst other things). The following gave a clue that WINE was involved: Mouse offset problem when using a WINE application maximized Downgraded from WINE 1.3 to 1.2 and the problems went away.
Also had to handle this when upgraded SQLyog: wine: cannot find L”C:\\windows\\system32\\plugplay.exe”. Fixed by making a copy of system.reg and wiping section on plugplay.
Adding an icon to the launcher involved the solution from How can I edit/create new launcher items in Unity by hand?
DVD playback even if using CSS encryption
sudo /usr/share/doc/libdvdread4/install-css.sh
Making Dash Searches Better
Keep Files From Showing Up In Unity (Dash) Or Other Programs That Use Zeitgeist With Activity Log Manager
Misc Links
Experiencing strange, intermittent faults on your computer? Don’t forget to check for a basic hardware problem such as faulty RAM before doing a lot of time-consuming testing of software installation and settings. If you are fortunate enough to be using Ubuntu (maybe using a Live CD or USB stick), it is easy to run memtest86 (assuming you are using a computer with an x86-compatible processor) and see if there is problem with your memory. Run it looking for patterns of bad RAM because you might be able to sidestep them. To display error reports in a useful format do the following once memtest has begun:
c for configuration
4 Error report mode
3 for Bad RAM Patterns
0 for continue
Let the system do multiple passes to be confident you have picked up any intermittent errors. This may take an hour or so. If bad RAM is detected and it is not too extensive, you can edit /etc/default/grub to sidestep the bad parts.
E.g. sudo gedit /etc/default/grub
Uncomment the example and use your own details as per the memtest output:
#GRUB_BADRAM=”0x01234567,0xfefefefe,0x89abcdef,0xefefefef”
Don’t forget to run the following afterwards to update your booting:
sudo update-grub
Dr. Grant Paton-Simpson
Lead Developer, SOFA Statistics
At the heart of this post is a puzzle. We have:
And this seems counter-intuitive – surely project popularity and revenue should go together. Perhaps the project isn’t sufficiently popular? Well let’s look at that part of the equation first.
SOFA Statistics has over 65,000 downloads so far, and while there is a difference between downloads, installations, and users, that is still a substantial number for a fully-fledged application. And based on feedback solicited by the project from first-time users, ratings given on Sourceforge, and comments from users interacting with the project, the application does seem useful to a wide range of people.
SOFA Statistics is easy to use and a very pleasant experience, and I’ve quickly found it’s simply the easiest way to access and query all sorts of databases. With the attractive charts and reports available, I would expect SOFA will take off quickly in terms of popularity and fandom. I think this has potential to be a revolutionary product in several ways. There are no other products quite like SOFA, and it represents a breakthrough in making statistics accessible to everyone. It’s easy enough to be used by the business analyst who doesn’t need SPSS, SAS or other systems — or who just doesn’t feel like spending thousands of dollars on a piece of desktop software. As such, it has potential to bring the benefits of statistics to a whole new population of business intelligence and academic users.
James Jackson
Gannett Co Inc
A recent review in Linux Journal was especially encouraging:
My hope is that this program becomes an adopted industry standard of sorts, mentioned in everyday conversation by organization workers the world over. And, given its free and multi-platform nature, … this hope of mine may not be an unrealistic one.
John Knight
Linux Journal
There are other positive signs as well. I liked this tweet suggesting that SOFA is “kinda like R for mere mortals”. AND BioPubMed thought SOFA “sets you free from your psychiatrist. At least, if it were statistics which was driving you crazy.” Blogger Boltgirl was especially enthusiastic:
SOFA, bitchez. My heart flutters ….
And now I found SOFA, and my life has changed. It is an open-source online stats package that evaluates your data and walks you through the process of selecting tests. It cranks out graphics. It produces results I can use. I understand, finally, and now I still weep, but this time in joy and relief.
Oh, science. We’re on again.
So SOFA seems popular enough. And it seems reasonable to expect an associated revenue stream if the right business model can be found. But no-one ever said it would be easy.
It may seem that project popularity will inevitably lead to financial success. But there are indications that this may not be so. As Dave Kellog, a technology commentator and former CEO of MarkLogic notes, “when you play the role of market spoiler it’s much easier to be famous than rich.”
The sad fact is that making money from software applications is not straight forward, whether a project is open source or proprietary (or hybrid). If you’re proprietary you are competing against existing proprietary vendors – some with large reputations and marketing budgets – or you’re competing against free. And if you are free and open source, you may not be able to get people to pay you. Furthermore, the market is global, fast-moving, and fiercely competitive.
There’s heaps of people out there that have said “Wow, the app store’s a goldmine. I’m going to get on this”. Reality is a bit harder. Knowing what will work and what won’t is just as hard as it is for any other business. Perhaps more so, because there’s so much competition.
Just look at how much duplication there is out there in the app market. I’ve got a friend who sends me ideas for apps every now and then. In almost every circumstance, somebody already has something that does it, and I just send him the link back.
Featured App: NodeDroid
So what would be a realistic expectation based on other people’s experiences? According to Taylor, conversion rates (into actual paying customers) of 0.5-2% are common for single-vendor commercial open source firms such as the company underpinning SOFA. Jacob Taylor. “User to Customer Conversion Rates.” Personal communication, March 2009. Cited in The Single-Vendor Commercial Open Source Business Model. Unfortunately, revenues for SOFA thus far have fallen well below these levels and there are no signs of improvement (assuming the current business model remains unchanged).
At this point it might be useful to elaborate on how things are meant to work. An interesting model I came across is called the Beekeeper Model (from http://jamesdixon.files.wordpress.com/2009/04/the-bees-and-the-trees-v2-draft-1.pdf).
In this scenario, the commercial open source company is like a Beekeeper – their job is to provide the bees what they need, keep them healthy and happy, and let them get on with making honey. The Beekeeper then sells the honey to customers. The bees are presumably the “community”. The SOFA Statistics problem so far has been that I write almost all the SOFA code and almost no users pay for it. Or, to follow the metaphor, I make nearly all the honey by myself, there are a few bees, and what honey there is the bears scoff! Before I offend what bees I do have I should make it very clear that I very much appreciate the efforts of all the people who have helped in various ways – e.g. translations, bug reports, encouragement etc. And I am not criticising people for using open source software for free. I have only contributed money to a handful of the many open source projects I rely on. The point is that the status quo for SOFA Statistics is not working from a business point of view. Something is going to have to change. Maybe a new business model is required.
The reasons will vary for different people but for me the main reasons for going open source are as follows:
Having expressed a strong preference for an open source business model, which are the most likely to succeed?
There are many different schema for describing different business models (e.g. Business models for open source software, The nine revenue streams for Open Source companies, and 11 open source business models) but here are some current exemplars of successful open source business models:
Note that I haven’t included the Donation-Ware model in the list as I am unaware of many successful examples (Calibre ebook management may be one exception). Whether the payments are micropayments (e.g. Flattr) or not I have not gained the impression that many projects will raise significant sums of money from this. Even the excellent OpenShot video editing project (which I have donated to) has only had 18 donors ever (as at Sept 2011).
There are other options as well, each holding variable levels of promise for the SOFA project.
And if the idea of earning money from the business is abandoned altogether, there is the option of Charity-Ware e.g. using Kiva:
“Kiva is a non-profit organization and website that allows you to lend as little as $25 to a specific low-income entrepreneur in the developing world. You choose who to lend to – whether a baker in Afghanistan, a goat herder in Uganda, a farmer in Peru, a restaurateur in Cambodia, or a tailor in Iraq – and as they repay the loan, you get your money back.”
This is not strictly a business model but it is worth mentioning.
OK – so the existing business model being used by the SOFA Statistics project hasn’t been working. Is it a case of refining the existing approach or trying something completely different? Either way there has to be a spirit of perseverance and experimentation. Joel Spolsky has a superb piece on the importance of persistence and morale. In How Hard Could It Be?: Start-up Static he compares an entrepreneur to a kid playing with their first shortwave radio.
He takes it home and turns it on, and what does he hear?
Nothing. Static.
This might be demoralizing. So he tries a different frequency.
Nothing. Static.
And this might be demoralizing again. Until his mom wanders by and plugs in the antenna on the radio, and suddenly, he picks up the ghost of a station! It sounds like it’s far away, and they seem to be speaking — what is that language they’re speaking? Never mind, it’s a station! …
Experimentation is also crucial:
As the business progresses, you start trying to turn all the various knobs on your fancy radio set in order to get better reception or to find a station you like. And fortunately, in business, we founders have a lot of knobs to play with. There’s price. Location. Employees. Marketing. Advertising. Return policies. Trade shows. Products. Search-engine optimization. …
At this moment, a founder who is incapable of careful morale management will think to himself, Maybe a career in HR management isn’t so bad after all. Meanwhile, the determined founder will start playing with the dials — rethinking the menu, trying new promotions, and adjusting prices. And what he’ll find is that, just like the tuner on a radio, certain aspects of a business can be off by only a little bit and then, one tiny adjustment, and BING! The thing starts working.
Returning to the SOFA project, numerous experiments have been tried with the precise wording and tone of messages, graphics, inclusion of free bonus themes etc. But, so far, there has been no significant success at all using the existing support/donation models. After two years the conclusion has been reached that it is time to give users a more compelling reason to pay the project money. Which brings us to the Open Core model.
Matt Assay, formerly of Alfresco and Canonical, reluctantly reached the conclusion that open core might be a necessity and made the following recommendation:
Add closed extensions to the core, still 100 percent open source project. Customers get full access to the source code to view and modify it. The user community loses nothing, but the company adds a compelling reason to pay it money for those … that otherwise won’t or can’t.
A time to reap, a time to sow: A phased approach for open-source businesses
If this approach is taken, the plan is to keep the existing version of SOFA 100% open source and to keep supporting and maintaining it. But any significant new features will probably be restricted to the proprietary version. This emphasis on the “pro version” would not be at the expense of the community though – the brutal fact is that without an ability to generate income from SOFA any significant extra work almost certainly wouldn’t be happening at all. It won’t be possible to justify working without an income when money is required to run a family. So the paying customers will be the people who keep the project going.
It is recognised that this is controversial to many (see Open Core Is Bad For You and Open core is not open source). Open Core is even labelled by some as fauxpen source (The Fauxpen Source Definition). Here is some anonymous feedback on the SOFA plans:
This is not a good idea, you have to choose another business model …. SOFA is an excellent tool. But pro version will kill the project, because free users fall on a second-class citizens category. Sell statistics books based on SOFA, or t-shirts, but don’t sell the software.
I am more inclined to agree with James Dixon of Pentaho fame:
[V]ery few people who complain about the open-core model have actually tried building a business using any of these models. I suggest you go and try it for yourself.
There are no guarantees that the approach being proposed will work but it is surely worth a try. Time will tell.
Each open source project is unique – with its own user base, community, and potential to generate revenue. Software which is simple, well documented, easy to install, and easy to use, for example, is going to provide less scope for revenue (assuming it is popular) than software which requires skilled support for installation, integration, and maintenance. And software which is used for mission critical tasks will similarly have more potential for a successful business model than software which performs a less critical task. Beyond the product itself, a lot depends on the surrounding context – are there other open source alternatives? And what is the price of competing proprietary products? So following advice based on projects with different characteristics may not work for your project and it is important to experiment and adapt. The overall experiment currently being pursued for the SOFA project is Open Core and there will be sub-experiments around the implementation of that strategy. Hopefully there will be good news to report back.
Based on a presentation made to the New Zealand Python Conference (Kiwi PyCon) in Wellington, NZ August 28th 2011. Thanks to Francois Marier for additional suggestions.
I had a Vista image that I foolishly set to 10GB. That was OK until I added the Java Runtime Engine, Eclipse, PyDev, Python, SOFA Statistics, another user for testing purposes (with non-English characters in their user name and home directory) etc. And once I ran the space-greedy system updates, I was out of disk space. A working solution was found in these two posts: Increase/Expand the Size of Windows 7 vdi file on VirtualBox Mac, and Cloning a VM with multiple snapshots?. If you want to copy the whole thing up to a particular snapshot (NB the result with be collapsed into one single VDI, unlike the original), get the hexUUID of the snapshot.
Step 1 – make a new and larger VDI. I did so by making a new virtual machine (testVista) with a new virtual disk image (testVista.vdi). An alternative approach would have been to make the vdi on its own but I couldn’t find a menu item to allow that. It was no great burden doing it as part of making a new virtual machine though.
Step 2 – clone the current state version of the old (and small) vdi. You need to find the hexUUID value of the snapshot you want. See Cloning a VM with multiple snapshots? for details. It gives guidance on how to tell which uuid you need.
The command that worked for me was (the path depends on where you are running the command from and your setup of course):
VBoxManage clonehd 0db42252-9240-4a0a-9393-a7d79778364d "../testVista/testVista.vdi" --existing
Note – testVista was the new VDI I was trying to clone into. It should take a few minutes to run.
Step 3 – disconnect to the old vdi and connect your machine to the larger vdi.
Step 4 – Boot into your system and resize the partition as per Increase/Expand the Size of Windows 7 vdi file on VirtualBox Mac On XP I needed EaseUS Partition. For a Ubuntu guest I downloaded the Gparted iso, added that to storage as a live CD, and booted in. Resized everything and exited. Then unlinked from GParted iso.
Step 5 – Success!
[Update – needed to do this again on another vdi – thus the clarified instructions here]
Foreign key support was added in SQLite version 3.6.19, Oct 14th 2009 (Release History of SQLite). There is no point trying to use this feature if you have an earlier version of SQLite. But which version do you have? Here are the relevant commands. NB version is the version of pysqlite, not sqlite. So 2.6.0 does not refer to the SQLite version, which is 3.7.4 i.e. very up-to-date as at the time of posting.
>>> import sqlite3 as sqlite
>>> sqlite.version
'2.6.0'
>>> sqlite.sqlite_version
'3.7.4'
The default installation of Python 2.6.6 (which I had to stay with for various reasons) installed an older version of SQLite. But this was easily remedied by installing a newer version (2.7.2) alongside 2.6.6 and overwriting the older version of sqlite3.dll in the C:\Python26\DLL folder with the version from C:\Python27\DLL. It was as simple as that.
Now foreign key constraints are disabled by default, so they must be enabled for each connection.
cur.execute("PRAGMA foreign_keys = ON")
Now to check that it is set, run the following:
cur.execute("PRAGMA foreign_keys")
to run the PRAGMA command you need, and
cur.fetchone()
to get the result back from it. Either (0,) or (1,). NB to do the fetchone() after “PRAGMA foreign_keys” not “PRAGMA foreign_keys = ON”. If nothing is returned, you either forgot the previous instruction and didn’t rerun “PRAGMA foreign_keys” before the fetchone() 😉 or SQLite was compiled with the wrong flags set.
Tip: If the command “PRAGMA foreign_keys” returns no data instead of a single row containing “0” or “1”, then the version of SQLite you are using does not support foreign keys (either because it is older than 3.6.19 or because it was compiled with SQLITE_OMIT_FOREIGN_KEY or SQLITE_OMIT_TRIGGER defined).
SQLite Foreign Key Support
See also:
SQLITE_OMIT_TRIGGER
Defining this option omits support for TRIGGER objects. Neither the CREATE TRIGGER or DROP TRIGGER commands are available in this case, and attempting to execute either will result in a parse error. This option also disables enforcement of foreign key constraints, since the code that implements triggers and which is omitted by this option is also used to implement foreign key actions.
Compilation Options for SQLite
If you are using SQLite as installed along with Python the most likely explanation for no result is that you made a mistake.
Pyinstaller is a wonderful way of making a single executable that can be run on any windows machine without requiring Python or associated libraries (e.g. wxPython for GUI toolkit) to be installed. It can make things much easier for users. Instead of installing a whole lot of libraries they just put a folder somewhere (even the Desktop is OK) and start using the executable. In my case I needed a simple toolkit for reading, editing, and manipulating secured MS Access databases and reporting results in either HTML or Excel spreadsheets. Python is much nicer to write code in than VB. But I had some major problems with getting win32com working.
If using a script with win32com inside a pyinstaller (made with 1.5 in this case) executable, there can be major problems with win32com.client.Dispatch(). I used two different approaches to get MS Access and Excel working. Using win32com.client.gencache.EnsureDispatch() seemed to work for both initially but that was incorrect. Using either approach (EnsureDispatch() or Dispatch() without the clsctx argument) there was a problem finding the module within PyInstaller. See EnsureDispatch and EnsureModule not working on win32 for a description of the problem even if the solution was not one I was able to use successfully.
Example error:
dbengine = win32com.client.gencache.EnsureDispatch(r'DAO.DBEngine.36')
File "C:\Program Files\exampledev\build\pyi.win32\admin\outPYZ1.pyz/win32com.client.gencache", line 536, in EnsureDispatch
File "C:\Program Files\exampledev\build\pyi.win32\admin\outPYZ1.pyz/win32com.client.gencache", line 520, in EnsureModule
File "C:\Program Files\exampledev\build\pyi.win32\admin\outPYZ1.pyz/win32com.client.gencache", line 287, in MakeModuleForTypelib
File "C:\Program Files\exampledev\build\pyi.win32\admin\outPYZ1.pyz/win32com.client.makepy", line 286, in GenerateFromTypeLibSpec
File "C:\Program Files\exampledev\build\pyi.win32\admin\outPYZ1.pyz/win32com.client.gencache", line 550, in AddModuleToCache
File "C:\Program Files\exampledev\build\pyi.win32\admin\outPYZ1.pyz/win32com.client.gencache", line 629, in _GetModule
File "C:\pyinstaller-1.5\iu.py", line 455, in importHook
ImportError: No module named win32com.gen_py.00025E01-0000-0000-C000-000000000046x0x5x0
For MS Access, I looked in C:\Python26\Lib\site-packages\win32com\genpy folder and found one the modules I had earlier generated by makepy (details on making it later). It started with the following details which made it clear it was the correct version:
# Created by makepy.py version 0.5.00
# By python version 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit (Intel)]
# From type library 'dao360.dll'
# On Tue Jun 21 21:47:34 2011
"""Microsoft DAO 3.6 Object Library"""
I saved and renamed the module to dao36_from_genpy.py and the starting point was:
dao36_from_genpy.DBEngine()
instead of win32com.client.Dispatch(r'DAO.DBEngine.36')
The idea came from py2exe/pyinstaller and DispatchWithEvents
Returning to using makepy.py:
“There are a couple of different ways to run makepy. Start Pythonwin, and from the menu select Tools->Com Makepy Utility. You should see a list of registered typelibs. Select “Microsoft Word x.y Object Library” and hit Ok. This can also be done programatically by initiating Word with
win32com.client.gencache.EnsureDispatch(‘Word.Application’) win32com.client.constants – AttributeError
For Excel, the solution was to use win32com.client.Dispatch(“Excel.Application”, clsctx = pythoncom.CLSCTX_LOCAL_SERVER). Remember to import pythoncom of course. I wasn’t able to get the other approach working very quickly because instead of a single module there was a folder, an extensive __init__.py, and a lot of submodules e.g. Workbook.py.
For more on Pyinstaller see Pyinstaller 1.5 with Python 2.6 (Round 3)