Negative Feedback

The impact of environment: how much do IDEs contribute?

2012-01-23T22:24:00.000-08:00

Those who know me will know this is a much-beloved topic of mine. I am setting this stuff down in an attempt to purge some of these thoughts from my brain.

During a conversation with a colleague yesterday, the idea of language productivity came up. It's an old argument: sure, you can write less lines of code in say Python than in Java (see my post on this topic), but with a modern IDE, you'll end up ahead as the IDE will be doing a lot of that typing for you. This forms part of a bigger conversation about the relative power of different languages, the relative merits of various IDEs and the way the brain works, before and after exposure to various models of computation. So, I've been maintaining a variety of links to research on these topics, and I'll try to state my current understanding succinctly in the next couple of paragraphs. I would really value any feedback that you could give me (for the handful of people who actually read this). Although I enjoy reading anecdotes of personal experiences, I would appreciate most if you could point to proper scientific studies on the topic.

So, this study, from 2000, compared C, C++, Java, Perl, Python, Rexx, and Tcl for a particular programming task: in this case a program to find which words could be spelled with particular phone numbers using the normal keypad letter mappings. They found that the "scripting language" programs were significantly shorter and took less time to write, but did not perform significantly poorer than the compiled languages. An interesting feature is that the compiled languages (C, C++ and Java) required considerably longer programs and took longer to write. The lines/hour for all languages were roughly comparable. They don't mention which IDEs the programmers were using, but one assumes that at least some of them were using "modern" IDEs. A 2003 interview with Guido van Rossum, highlights this idea of Python requiring less "finger typing". He's obviously biased as the creator of Python, but considering that he also wrote the Python interpreter (in C) and has a large realm of experience in various languages, one would think he has had some time to experience the differences.

Another study, from 2009 may explain a part of this. They found that reading code was largely a function of the number of tokens. This is important, because it implies that less tokens can aid in understanding already written code and have an effect on writing it. Importantly, they also find that simple domain mappings are easier to comprehend than complicated ones.

I have not been able to find much quantitative research on the benefits of IDEs. Although I acknowledge that environment makes a big difference, I tend to think about code in my head without an IDE, which makes the actual language quite important for me. To some extent one must acknowledge that there is a personal aspect as well - some people prefer a particular kind of approach, as shown in this article about language mavens vs tool mavens, which I've linked to before. I've slowly been learning Eclipse, and it is quite amazing how much of the mundane Java stuff it can automate. I'm also a great believer in refactoring tools (I use rope, with bindings for Emacs). At the same time it is clear to me that much of the stuff you get in Eclipse could be addressed by streamlining the language a bit.

Finally, the larger picture about fitting into an environment. This is where the established languages have the upper hand. Popular languages are more widely known, have more books and environments available and have the corporate OK. Also, if the libraries of the operating system you are using have been crafted for a particular language, there is a clear advantage to aligning yourself with that. As a researcher, though, it is often a bit of a conundrum: the real publishable results are more important than the interface, so you find the majority of computational research code has no GUI, and is only lightly coupled to the host OS. This is especially true of parallel code which inevitably has to run on headless cluster boxes.

I don't want to make this another post about choosing a programming language, but I will restate my final strategy: use the most expressive language available for prototyping - this will inevitably be a dynamically typed language with a REPL. For me, this is Python. This hooks into the constant lines per hour result of before. When the program produces the correct output and the algorithm is set, re-implement the slow bits in a lower level language. For me this is Fortran, but it could be C++ or whatever has good support for your problem domain. This strategy has placed me a in a poor position to benefit from IDE support. For that, you typically want to do everything in a combination of IDE and well-supported language (this is the classic Java/C++/C# for everything approach).

Anyone with links to proper quantifiable research on programmer productivity (especially the effect of IDEs)?

Unchecking heard podcasts in iTunes via AppleScript

2012-01-12T05:07:00.000-08:00

Because I have a small 16 GB iPhone which is way less than the size of my music library, I use the "sync only checked items" approach. However, I want podcasts I have listened to already be unchecked as well. For that reason I've written this little Applescript which runs through all the podcasts and unchecks the ones I've listened to.

tell application "iTunes"
 set thetracks to every track of library playlist 1 whose podcast is true and enabled is true and unplayed is false and bookmark is 0
 repeat with currentTrack in thetracks
  set enabled of currentTrack to false
 end repeat
end tell

This was slightly harder than it looks due to the fact that iTunes uses the "unplayed" property to indicate it has never been played, ever. When you're halfway through, though, a tracks' "unplayed" is false, but you don't want to uncheck it. In the iTunes GUI a blue dot indicates played, a half-filled blue dot played but not finished and no blue dot is played and finished. The breakthrough is in the "bookmark" field, which gets set to the position in the file you last listened to. Kindly, Apple sets this to 0 when a track is completely listened all the way, leading to the conditions above.

You can use the AppleScript editor to create this file and save it as "Uncheck heard podcasts.scpt", then save it in your Library/iTunes/Scripts folder. When you restart iTunes, it will show up in the script menu.

This is a good example of how knowing a scripting language can help you out.

The fallacy of the general purpose tool

2011-12-29T06:14:00.001-08:00

I carry a Leatherman on my belt every day wherever I go. It has become a bit of an extension of my body, which I only really notice when I have to fly somewhere and I'm forced to take it off for the flight. Then I get to my destination and I can't open the cable ties on my luggage. At the same time, I have a whole pinboard of tools up in my garage, which overlap somewhat with my Leatherman: a dedicated set of pliers, some dedicated screwdrivers and scissors and so on. This is because the is not as good as any of these dedicated tools.

I think the same goes for programming languages, and therefore I have learned quite a few, just like I have stocked my pinboard with tools that do one thing very well. Many people find learning new languages onerous and will try to find the one language that does everything they want to do well. Their arguments against learning a new language often involves something like "but I can do that just fine in my language, and then I don't have to learn a new language". Here's an example from real life that illustrates the advantages of special purpose languages.

My father approaches me with this problem. He uses an accounting package which can import OFX files (they're an SGML/XML format for financial records), but there is a slight problem. The ZAR needs to be changed to USD and the dates (which are tags of the form 20111201 - yes, no closing tag as it's SGML) have to be changed from YYYYMMDD to YYYYDDMM. Now, turns out that Python has a nice OFX module. So does Java. But the fastest way to make the changes to his files is probably sed:

sed -r -i -e 's/ZAR/USD/' -e 's/<DT(.*)>([0-9]{4})([0-9]{2})([0-9]{2})/<DT\1>\2\4\3/' filename.ocx

This does the replacement in-place, is really fast and is quick enough to throw together. This is the kind of thing that sed shines at. In fact, it is exactly what it was designed to do, so it is unsurprising that it does it so well.

To do the same thing in Python (even without using the OFX module) requires a bit more effort:

import os
import re
from tempfile import mkstemp

filename = "test.txt"
patternStrings = ["ZAR", r"<TD([^>]+)>([0-9]{4})([0-9]{2})([0-9]{2})"]
replacements = ["USD", r"<TD\1>\2\4\3"]

# Compile patterns
patterns = [re.compile(p) for p in patternStrings]

# Create temporary file to hold outputs
_, tempfile = mkstemp()

# Process file
outfile = open(tempfile, 'w')
for line in open(filename):
    for pattern, replacement in zip(patterns, replacements):
        line = pattern.sub(replacement, line)
    outfile.write(line)

outfile.close()

os.rename(tempfile, filename)

Needless to say, the difference is even more pronounced in Java due to the large amount of boilerplate needed.

import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.util.Scanner;
import java.util.regex.Pattern;

public class Fixer {
    static File infile = new File("test.txt");
    static String[] patternStrings = {"ZAR", "<TD([^>]*)>([0-9]{4})([0-9]{2})([0-9]{2})"};
    static String[] replacementStrings = {"USD", "<TD$1>$2$4$3"};

    public static void main(String[] args) throws IOException {
        Pattern[] patterns = new Pattern[patternStrings.length];
        Scanner in = new Scanner(new FileReader(infile));
        
        // Compile patterns
        for (int i=0; i<patternStrings.length; i++) {
            patterns[i] = Pattern.compile(patternStrings[i]);
        }
        
        // Create temporary file to hold outputs
        File tempfile = File.createTempFile("tmp", "tmp");

        // Process file
        FileWriter outfile = new FileWriter(tempfile);
        while (in.hasNextLine()) {
            String line = in.nextLine();
            for (int i=0; i < patterns.length; i++)
                line = patterns[i].matcher(line).replaceAll(replacementStrings[i]);
            
            outfile.write(line + System.getProperty("line.separator"));
        }        
        outfile.close();
        
        // move temp file back to filename
        tempfile.renameTo(infile);
    }
}

Now, at this point someone is bound to say "but wait, people don't write desktop apps in sed!". But that's quite the point - Java was designed with programming in the large in mind, but it is really tedious for short programs. In many cases, learning a small domain specific language and using it to solve your small problem is faster than learning how to do the same thing in your "one language".

I wish I had more time to put examples together, but this situation just presented itself and got me thinking.

Moral ergonomics

2011-12-26T06:51:00.000-08:00

It is quite hard to design a comfortable chair. The task is made more difficult by the variety of the human anatomy. While ergonomics has done some great things for our workplace comfort, it will never be able to come up with a truly one-size-fits-all chair.

Perhaps it is because the physical differences are so easy to perceive that the above statement seems so obvious. However, rephrase the whole paragraph substituting systems of ethics for chair, and perhaps include something about moral feelings being at least as variable as our limb measurements and you've got a big philosophical argument on your hands. It appears that there is renewed interest in a formal study of morality, from many camps. People like Sam Harris are trying to say that science can say something about our moral urges. He is being fought by people that feel that a deity or a book should be the final authority. Whichever way we turn, however, I am sure that everyone has felt the feeling of conflict that arises when your stated ethics lead to an uncomfortable conclusion.

As a moral noncognitivist I hold that moral statements like "X is wrong" can most appropriately be interpreted in a similar way to "I do not like X", perhaps with an additional "and I think you shouldn't like X either" thrown in for good measure. To borrow from a recent Facebook argument on a similar topic, it means that at the bottom of each moral statement lies a feeling. If you hold that killing people is wrong, it may be because that thing in itself feels wrong or because you have some connection to doing things against people's will. Keep asking yourself "and why is X wrong" for every answer like "X is wrong because Y is wrong, and X is like Y", and you'll end up saying "because Y feels wrong" at some point.

Of course, there are people who have tried to derive a consistent ethic. Is it so strange that none of them has really gained universal acceptance? Is it strange that no single chair design has been mass produced and rolled out all over the world? That our cars have adjustable seat backs? It's a question of ergonomics, not of logic.

I believe that we have to admit two things to go forward in the scientific exploration of morality:

There is no universally acceptable set of moral rules. No, not even consequentialism or human rights or the golden rule or anything like that. People feel differently just like they have different sized feet.
It may be more prudent to work on systems that don't terribly offend a critical number of people rather than to try to argue that they should just come around. I don't mean that we should coddle the extremists, but I do think it could be more productive to understand which feelings are causing the problem rather than to try to employ logic.

Unfortunately, many rational people fall into the trap of believing that anyone who reaches a different conclusion from them is irrational, neglecting the possibility that they may simply be shaped differently.

Scientific python setup for Mac OS X 10.6

2011-11-27T07:31:00.001-08:00

For posterity, and to save some people some trouble with their install, here is what I have done to get up to a working matplotlib on my Mac running Mac OS X Snow Leopard 10.6.8:

Download and install the python.org python 2.7 from the .dmg installer
Download this egg. To run it, you may have to rename it to have a .egg extention rather than a .egg.sh one.
Open a terminal
Execute the file you renamed before by typing "sudo bash path/to/filename" - to get the path to filename, you can open a Finder window showing the file, then drag the icon onto the terminal - that will insert the full path.
Install pip by typing "sudo easy_install pip"
Install GNU readline by typing "sudo pip install readline"
Install nose (used for testing) by typing "sudo pip install nose"
Install iPython by typing "sudo pip install ipython"
Download and install the NumPy binary distribution
Download and install the SciPy binary distribution
Download and install the Activestate tcl/tk - this is the one recommended on python.org. If you click through the python website to the Activestate one note that you don't have to fill in your info to download the files.

You should now have a working scientific environment that allows you to do most of the things you would be able to in a stock install of Matlab with a couple of packages thrown in, except for plotting. The next step is a bit more arduous, because the default binary install of Matplotlib does not work (at least not for me). This is what I had to do to get it working:

Download and install XCode
Install git
Open up a terminal
Type "git clone git://github.com/matplotlib/matplotlib.git"
Type "cd matplotlib"
Type "sudo su" and provide your password when prompted
Create a temporary directory (XXX in the following command) then do 'ARCHFLAGS="-arch x86_64" LDFLAGS="-arch x86_64" CFLAGS="-arch x86_64" FFLAGS="-m64" PREFIX=XXX make -f make.osx fetch deps mpl_build mpl_install'
Type "cp -r XXX/lib/site-packages/* /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/"

To test if all this worked, do the following:

Test ipython: "iptest'
Start ipython and do
Test numpy: "import numpy; numpy.test()"
Test scipy: "import scipy; scipy.test()"
Test matplotlib: "import matplotlib; matplotlib.test()"

Don't be too upset if some errors occur: it seems the test suites aren't entirely up to date. As long as you don't see a segfault or ipython crash, it should be ok.

My brain can't do that!

2011-09-07T07:35:00.000-07:00

When I was teaching computer programming to chemical engineering students I categorically denied that "I'm just not good at programming" was a valid excuse for flunking my subject. Lately I have become less sure of my position. Let's be clear: I am quite convinced that people have varying levels of aptitiude, it's just that I had thought that certain skills were so similar that one automatically led to the other. Chemical engineering students in their second year have already passed many mathematics courses and have shown themselves to be intelligent and able to follow the steps required to do algebra and calculus in a relatively mechanical way. Programming has always seemed entirely analogous to math for me, so I made the mistake of generalising from my experience to theirs. As any good lecturer will tell you, this is a fatal mistake.

I am constantly fascinated by how inaccessible our brains are to ourselves. The puzzle of exactly how my brain makes the connections between formulae and computer language syntax is probably better understood by some neuroscientist somewhere than by myself. This is, of course, entirely counter-intuitive. Many people would even deny it. But the thing that concreted it for me was listening to this episode of Radiolab (which if you haven't listened to it yet, is an entirely awesome podcast covering a wonderful array of different subjects in a wonderfully cock-eyed way). Spoiler alert: the skill displayed here is a guy who is able to run multiple symphonies simultaneously in his head. Try it. Start playing a song you know well in your head. A couple of seconds later, try starting another one by a different band. Now tell me where you are in the two different songs. If you are remotely normal you will be unable to even conceive of how to do this. The feeling I get when I try to do this makes me wonder: if we have such a clear example of a person who can do something that seems totally unrelated to skill or practice, but which normal people cannot do, who am I to say that those people in my class just couldn't "get" programming, ever?

The problem is that it leads to an uncomfortable end point. I cannot assume that any student who struggles simply cannot do the stuff he's trying with - that would invalidate the entire purpose of teaching! Well, unless you ascribe to the notion that teachers are only allowing their students to reach their potential, but that's a whole different post. Like the conundrums of free will, it appears the argument ends in a place where assuming something we know to be wrong in general (that "any student can do this") is both easier and less risky than correctly determining who has the ability in the first place and then only focusing our attention on them.

One good thing has come from these musings though: I find myself being much more sympathetic to the struggles my students have. And I suppose a little empathy never hurt anyone.

Public transport and a sense of society

2011-08-11T23:36:00.000-07:00

I feel more connected to people around me when I am in the street.

This seems pretty obvious, but the effect on your psyche is so subtle that you may not notice the long term changes. When I started road running, I obviously noticed more runners on the road (due to confirmation bias), but I also started to feel less afraid on the road. I had experienced a certain sense of trepidation leaving my "safe" home to go into the "unsafe" street, which diminished as I realised that the people on the street were also real people with real lives, by and large going about their business with no ulterior motives. Now that I walk more I am noticing a similar thing - I am feeling more connected to the community.

In the excellent book Traffic, Tom Vanderbilt posits the theory that being in a car makes us treat other people differently, while they perceive us differently. In effect, we treat the car as a whole new animal, only nominally controlled by its driver. It's easy to think of the people who inhabit the street as totally different from us when there is a nice glass wall between us. When you walk in South Africa, you also realise just how poor our pedestrian infrastructure is. It's amazing how often you get the "what are you doing in the road" look when you have no other option. Perhaps on a deeper level there is a question about why I am not in my car. I reckon that people would be more likely to contribute to their local communities if they walk or run.

The science of the general

2011-07-06T04:31:00.001-07:00

This is a draft of an upcoming comment hopefully to be published in Chemical Technology. Feel free to shoot with some comments, even if you're not a chemical engineer.

Chemical Engineering as a discipline by that name has been taught at universities since 1888. At this time it was seen as a small side-branch of Mechanical Engineering incorporating applied industrial chemistry, mainly serving the emerging chemical industries of the time. The curricula of these early courses would become more recognisable to modern Chemical Engineers when AD Little formalised the concept of unit operations in 1915, while the concept of material and energy balances had gained traction by 1935. Thermodynamics and process control had appeared by 1945, followed by chemical kinetics and process design by 1955 (see Origins of Chemical Engineering for a complete timeline). Henri Poincaré said "There is no science but the science of the general", and one sees this instinct for generalisation at work as fundamental knowledge and general analysis supplanted a curriculum filled with specific examples from industry.

During roughly the same time, computing devices were progressing from simple adding machines to fully programmable computers. Although computers had been used for process control on chemical plants since the late 1950s, the first personal computer was only released in 1974 and Chemical Engineering simulation software for the PC was released in 1981. In a recent Nature interview with Neil Turok from the African Institute for Mathematical Sciences, I was struck by his description of their curriculum design process. They had initially developed a "conventional syllabus involving math and physics", but realised there wasn't enough time to teach it. They then made a key observation: "... if you took a photograph of a scientist at work today, everyone looks the same ... sitting at a computer and trying to make sense of what's on the screen." I think one could confidently replace scientist with engineer in that observation.

The funny thing is that computers are often seen as a means to an end, something that can remain opaque to our understanding as long as the programs we use work. Although the same approach could arguably be applied to much of the equipment on our plants, we still teach Thermodynamics, Calculus and other subjects that are frequently described as not being used in the workplace. We do this because these subjects lead to a more fundamental, more general understanding of our plants. There is far greater likelihood of a particular Chemical Engineer using a computer in his day than any particular unit operation, but it is not common for us to teach the equivalent fundamentals behind computing as applied to the problems of Chemical Engineering. Instead, we focus on manual solution of sets of equations using methods that have very little in common with the computer techniques used by process simulators. A friend of mine summed it up well: "We're teaching as though it were still the 70s, as though there is a reasonable chance of having to solve a problem on site without the benefit of a computer".

Perhaps a part of the problem lies in the demarcation of domains. The fact that computer science is a discipline in itself, or that there may be an electronic or computer engineering department in the same school as our Chemical Engineering departments may lead us to focus on the "real Chemical Engineering", but the existence of an industrial chemistry course or a mechanical engineering course did not stop our predecessors from carving out the niche that would become Chemical Engineering. We would do well to remember that most of the simulators we use today were developed by Chemical Engineers rather than generic software houses.

Another argument is that the principles don't change much from manual calculation to computer implementation. However, the methods of solution and the fundamentals of computing have made such great strides in the last 100 years that understanding how to solve a set of mass balance equations by hand does not really enable one to reason about the algorithm being employed by a simulator. Not only that, but the algorithms and software that are available today enable the solution of whole different class of problems.

We need to move beyond the simple treatment of computers as tools to solve our problems and appreciate that there is a general science underpinning their operation. As much as understanding Reaction Kinetics or Thermodynamics enables us to understand the design and
failure modes of reactors, understanding the computational structure of our simulators, optimisation solvers and spreadsheets can help us to understand how to employ these new information unit operations more effectively. It may also enable us to develop the
next set of tools for ourselves.

China childbirth stats analysed

2011-06-07T07:14:00.000-07:00

I read this infographic today about China's one-child policy. One of the things that jumped out at me is that China has more men than women in their population, and the infographic (at the bottom right) mentions that some feel the policy makes this worse, along with infanticide and preferential abortion. So, I wanted to know how big the effect of the policy is and I did some stats.

The infographic explains the process as follows:

As it is traditionally enforced, couples are encouraged to have only one child
If that child is a girl, parents would be allowed to have a second child to hopefully bear a son
After the second child, the would be discouraged from having a second child.

Now, assuming that this is exactly enforced as written, there are only three outcomes of parents having children (let's assume the chance of a birth resulting in a boy is x and ignore things like multiple births etc):

Children	Boys	Girls	Probability
1	1	0	x
2	1	1	(1-x)x
2	0	2	(1-x)(1-x)

Now, we can calculate the expected number of children, assuming everyone wants children and follows this process, this is simply 1 child with probability x and 2 with probability (1-x), so expected value is 1x + 2(1-x)=2-x. x is typically near 0.5, and they say that the fertility is near 1.5. Success! But what about the expected ratio of boys to girls?

Using the table above, multiplying the "boys" column by the "probability" column and sum ming, we end up with 1x - (1-x)x=2x-x²=x(2-x) boys expected. For girls, it's 1(1-x)x + 2(1-x)(1-x)=(2-x)(1-x). If x is 0.5, this ends up with 0.75 boys and 0.75 girls. That doesn't explain what we observe in China... In terms of the ratio of boys to girls, we see that the (2-x) term is common to both expected values, so the ratio ends up being x/(1-x). But this is exactly the same ratio that we would expect if no birthing policy is in place, as every birth would produce x boys and (1-x) girls on average. So, either these assumptions don't hold, or we need to look elsewhere for the discrepancy.

MTU troubles with iBurst sorted

2011-05-22T06:30:00.000-07:00

When we moved into the new house, we went to Telkom to get our ADSL connected. They insisted it would take several weeks and they had to have physical access to the property, even though the previous owners had only cancelled their account two weeks before we moved in. This caused us to storm out of the Telkom offices, vowing never to darken their doorsteps again and get iBurst instead. The sign-up process was efficient and I left the store 20 minutes later with their desktop modem and an antenna in case our reception wasn't so good.

Installation was a breeze, and minutes later I was connected. However, the connection seemed erratic, with some sites loading faster than others and some strange freezing from time to time, even thought the iBurst speed tester reported really good speeds. I phoned support and they cleared it up for me: set the MTU to 1392 they said. No problem. Setting the MTU down made things work really well, but there was a slight snag. I had connected the iBurst device to our Time Capsule to access it wirelessly. Unfortunately, the Time Capsule is not capable of much in terms of routing. In router-speak, it is basically a bridge. The iBurst device is also a bridge, so the computer had to do the PPPoE dialing and the setting of the MTU. Things appeared fine until we tried to connect with our iPhones, which have no way of setting the MTU. So I fired up our little Netgear router and set it to do the PPPoE and the MTU, then chained the Time Capsule from that. Still no joy - turns out the Netgear WGR614 has a known issue with handling the MTU (we would later find out it wasn't the Netgear's fault as such, but read on).

For a long time (probably too long), we used the iBurst on our laptops, but had to use the data connections on our phones in our own house. Not good. When my father gave me a good price on his old iPad, with no 3G, it became really clear we had to do something. A friend at work suggested the MikroTik 750g as a really capable unit with lots of customisability. Some research showed that MikroTik has a huge community and that RouterOS surely had what I needed to get the problem sorted out.

Of course, with great power and flexibility comes great confusion to novice users, so it took me a couple of hours and several phone calls to friends to get the whole thing sorted out, but heres the lowdown:

My top tip is to do all the configuration up to step 3 before unplugging your working internet connection. The configuration can all be done with the box directly connected to a laptop and you don't need to have it connected to the iBurst device during config.

Step 1: Update the firmware on the MikroTik - pretty simple, and allows much more to be done via the web interface. The web interface also has a copy of WinBox, which enables a vast array of configuration options.

Step 2: Configure PPPoE dialout. Go to PPP on the left hand menu in WinBox or Webfig, then do "Add new"/"PPPoE client" and configure the username and password. Also check the "Add default route" and "Use Peer DNS" options. You can set the MTU to 1392 here, but this is not all you need to do - keep reading. I recommend using the first ethernet connection for this, this is what they assume you'll do anyway.

Step 3: Add a mangling rule to force lower MTU. Go to /IP/Firewall/Mangle - the PPPoE client setup should have added two rules to change the MSS. You'll notice you can't edit them, but copy the outgoing one and change the "New TCP MSS" to "clamp to pmtu". Also make sure this new rule is above the old one in the rule table (you can drag them around to re-order them).

Step 4: Plug it in and check that everything works. You can use WinBox to check that the PPPoE is working and that the mangling rule for clamping is getting the traffic before the other one. That's it. Hopefully this will save other people the hassle I had to go through.

Advertising statistics

2011-04-07T23:40:00.001-07:00

I saw an ad on TV yesterday that made me think about statistics. It featured a shot of three women, with voice over saying "1 in three women develop disease X" (can't remember what it was). The implication, of course, is that one of the three women shown in the picture will develop disease X. You're supposed to be thinking "which one will it be". This is vaguely reminiscent of my first class at university where a lecturer told us to look left and right - one out of three people pass the course, so work hard to make that person be you. As it happens, all three of us (myself and both of the people next to me) ended up passing.

The stats work out as follows. Let's say we have probability p of contracting the disease and that p=1/N. We are shown N people. The probability of no-one contracting the disease is (1-p)^N. For the 3 person scenario, that means that there is about a 30% chance that they're all fine. Of course, that means that there's a 70% chance that at least one of them will contract the disease, but that's not quite what they had in mind when they chose to show us three women. If they had chosen N=7 (I think that's for breast-cancer), there would be a 34% chance that everyone was fine.

Even more interesting is that the probability of everyone being fine converges to 1/e (about 37%) as N goes to infinity. Perhaps when I figure out a fast way to get nicely rendered equations and plots into this blog I'll go into more detail.

Moving your personal info to Google from Mac

2011-03-13T01:05:00.000-08:00

In my previous post I ended off frustrated because I couldn't see my calendars on my iPhone. This post is to serve as a nice step-by-step set of instructions to move your personal information from the Mac to Google's servers.

I assume when we start that you have a Gmail account, but have all your contacts and calendars on the Mac.

1. Sync contacts from Mac to Google

This part is easy. From Address Book Preferences, go to "Accounts" and check the "Sync with Google" checkbox. Click "Configure" to enter your Gmail username and password. Done.

2. Get your calendars on to Google Calendars

This part is easy but tedious. In iCal, select each of your calendars and go to "File|Export|Export" and save them somewhere. Now, go to Google calendar and create a corresponding calendar for each of your calendars. Then, import each of the files you created before into each of the calendars you have created. This is far more tedious than it should be, but when you're done you've got all your calendars mirrored on the Google servers.

3. Get the Google Calendars back on iCal

From iCal Preferences, go to the Accounts tab and add an account. Select "Google" and enter all your info. Close the preferences box. You will notice a new calendar showing up on the left hand pane in iCal. Open the Preferences again and move to the "Delegation", which should now reload to show all your calendars. Select them all and you should see a rather awkward nested structure appear and sync.

4. Get all this on the iPhone

To get the calendars and contacts synced on the iPhone, go to "Settings|Mail, Contacts, Calendar|Add Account" and select Microsoft Exchange. Enter the Google login details and when a new "Server" field appears, enter "m.google.com". You will now only see your primary calendar on the iPhone. Now, go to "m.google.com" from your phone's browser. Once you've logged in, select the "Sync" icon and then select all the calendars you want to have appear. Now, you should see all these calendars in the calendar app.

I must say that I would have far preferred to have the Google calendars transparently synced with my iCal calendars in a similar fashion to the contacts, but at least this works. Also, the groups I had defined in my Address book, which used to sync well with the iPhone have now disappeared. The fight continues, but after many hours, I now have nicely synced contacts and calendars.

Calendar syncing with iPhone, Mac and Google Calendar?

2011-03-12T03:34:00.000-08:00

During dinner with a geeky couple the other day, I mentioned my lack of calendar syncing with Google Calendar. I had tried to do this many times before but never got it working well. They professed total amazement, as they claimed it had been a total breeze to set up. Turns out we probably have different use cases and different expectations.

When you think about the personal information you typically want to have synced on your devices, many people will think of the functionality Microsoft Outlook:

Mail
Contacts
Calendar
To-dos/tasks

I currently have basic syncing set up between my iPhone and my Mac, but no syncing with any stuff from my work computer. I am not happy with this, but I am looking for a solution, and that solution seems to be Google.

Mail is really a breeze if you keep all of it on servers and simply access everything with IMAP. Contacts are synced pretty easily on the Mac with Address Book's support for syncing with Google Contacts. Infuriatingly, there is no standard support for To-dos on the iPhone, so that's not going to happen. The thing that remains is calendar support. This seems like it should work, but there are some problems.

The big problem appears to be that I have multiple calendars (10) set up in iCal. My workflow so far this morning has been as follows:

In iCal, select each calendar one by one and export to a .ics file
In Google calendar, create a calendar with that name one by one
One by one, import the .ics file to the calendar with the corresponding name
Add the gmail account, wait for syncing and then select all the calendars as "delegates"

Note: These calendars show up as totally different calendars, so I would have to delete my current calendars on my actual mac, and the new ones appear in this strange nested directory structure as shown in the picture below (the "On my Mac" calendars are the original ones, and the grey ones under "Deligates" are the mac ones):

That's it for the iCal setup. Not entirely transparent, but at least I can see all my calendars and access them with sync. So far, so good.

Now for the iPhone. No joy.

I have tried both the gmail account and using the exchange support of gmail. Neither of them allow me to see all my calendars on the iPhone. I only see what is known as the primary calendar, which means I now have two choices:

Combine all of the calendars into one calendar, which means I lose fine-grained control of what stuff to share with my wife and other people (does she want to see all my appointments with students, do my students want to see all my appointments with my wife?)
Undo all the work I've done this morning and go back to syncing my phone and my mac using iTunes, and forget about having access to my calendars online.

So now I am frustrated. Regardless of where the blame lies (many people have pointed out that this is really an iPhone problem), if I can't access all my calendars on my iPhone it's basically game over for the whole idea.

Impressions of superiority

2010-12-23T08:41:00.000-08:00

None of my compliments are as often or firmly rebuked as telling someone they have a natural ability for something.

It is a quirk of our society that our sense of achievement is diminished by aptitude. We praise above all individuals who have overcome adversity in their achievement. Many view being told they have an aptitude or ability as tantamount to saying they didn't have to overcome as much as other people would have.

I am aware of my natural aptitudes for some activities. I had no particular difficulty learning to play the guitar and appear to understand music pretty well. My mathematical intuition is sound. I find it easy to learn new programming languages. I also have certain... challenges. I am easily distracted, I find medium-term planning difficult and my learning mode is often highly inefficient as I require a lot of background and overview before things start making sense. It has taken me a lot of time to figure out what the population average abilties in some of these areas are. I am by no means there, but I have learned enough to know which properties are high and which are low.

The reason for listing my own abilities here is an attempt to defuse the knee-jerk reaction that I have perceived many times when you tell other people they have abilities above the norm. Many people will vehemently deny such abilities because they assume you are implying they didn't have to work at it. It seems more acceptable to lament the lack of such abilities in others than to claim them for yourself. Many people don't even do that - they will deny any particular variation in these abilities among people and instead end up lamenting their lack of application. I find this particularly objectionable, as the implication is often that the reason for failure is fundamentally a weak will.

Here's how a sample dialogue often plays out:

Me: I have left off marking my papers again, and will have to put in a late night tonight.

Them: Oh, I always get my marking out of the way first thing, then I feel much better.

Me: I wish I had your well-developed work ethic and drive to completion

Them: I don't really have any particularly well-developed drive to completion - you can do this too.

Me: I have tried and failed. Perhaps I am just weak-willed. But more likely I am more easily distracted than you are, or find this task more onerous than you do

Them: How dare you imply that I don't need to apply myself to get this stuff done!

Unequal distribution of abilities sits poorly with the doctrine of equality (or at least equal opportunity). Even worse, it is very difficult to think of oneself as anything other than "normal". When I lost 25 kg, I honestly did not seem any different to myself in the mirror. I can just remember seeing a lot more fat people around. When I started running, it never felt much different to run at a comfortable pace, even though that has changed radically as I trained. It's an all-too easy trap to fall into as ones abilities develop: renormalisation. As you become more knowledgable, you just start to see more idiots. As you become less fit, you start wondering how all these people can be so active.

This post is a bit of an impression rather than a well-though out argument. I wanted to get this down while I was still thinking about it after having discussed it at a party last night. There is much left out here: how much of these abilities are innate and how much are learned? How does practice feature into it and how well can we assess our own abilities, and what is the mechanism of self-control (in fact, what is being controlled). That will have to wait for later.

Lecturer rant reprise

2010-12-07T20:06:00.000-08:00

My post yesterday prompted many comments. This post attempts to address some of them.

I got a couple of people expressing the idea that my experiences with suboptimal systems are not unique to my institution. I appreciate your comments, especially those mentioning specific institutions, as they give a bit of insight into the scale of the problem. I find widespread problems that would be (relatively) simple to solve in a (relatively) general way incredibly irksome. I feel great desire to solve these problems and distribute the solutions to everyone. The way I feel is quite well described by the hacker ethic. I aim to do meaningful work, but many of the things I am forced to do at the university are menial work. It gives me little consolation that others are doing the same things. In fact, it makes it worse.

My boss chided me for putting the post online for everyone to see. His concern was for airing dirty laundry in public. I took the university's name out of the post but I stand by what was said and I don't think much is going to happen due to that rant one way or another, save perhaps serving as an aide memoire for the marking system ideas and a nucleation point for further discussion about it. The whole wikileaks furore was also being discussed copiously online at the time and I was struck by the urge for secrecy. I understand the need for privacy and feel the urge to confide in particular people myself, but I don't feel quite the same way about organisations. I tend to feel they have a responsibility to be transparent. What is wrong with the idea that by talking openly about problems, you increase the likelihood of them being solved? Especially if many places are having the same problems?

Later that day I had a bit of back-and-forth on Twitter with a mate who worked in the lab with me several years ago. He asked what I had done about the problems and suggested that I develop a prototype of a system to show to admin. His posts hit a nerve, because this is exactly the attitude that I have had up to now (don't complain - just fix it). In the years I have been at the university I have developed the following tools that led out from me suggesting that things could be done better (I list my estimated cumulative time investment into these systems and the number of years they have been in development in brackets):

A static web page generator that we used to generate our departmental webpage for a couple of years before the new CMS was rolled out (100 hours, 4 years)
A system that uses the above along with a Google Docs spreadsheet to collect information about the department's publications and generates a cross-linked static website suitable for burning to CD (50 hours, 3 years)
A project allocation system that uses GLPK to assign projects to students for our fourth-year project activity based on their preferences (150 hours, 3 years)
Scripts for typesetting the timetables for the school of engineering based on a flat format spreadsheet generated from syllabus. (250 hours, 4 years)
VBA code for use in Excel to calculate student number checksums and several shared calculations, including the supp calculation I mentioned in the previous post (10 hours, 2 years).
Roll-out of a departmental Wiki to capture information about these and other projects along with other documentation (50 hours 3 years)

This list is not meant to sing my praises, but I suppose it serves to establish that I do not simply moan about things - I generally just say "that can be done better" and then move to do things better. The problem with this attitude is that these things are way outside even the most liberal reading of my job description, which means that all the time I spent on these projects is time that I could have (should have?) spent on my students, or developing better lectures or my PhD or any of the things that are actually expected of me.

I completely understand why no-one is doing these things: because it is no-one's job. Now, I get a bit annoyed at the "fix it yourself" attitude because I encounter it most often when it has anything to do with computers. I am a relatively capable programmer, and often when I complain that I can't find a particular tool or about how a particular tool works I get this glib answer that complaining doesn't solve anything and I should do something about it myself or shut up.

Imagine a scenario: I post a tweet complaining that water is leaking from the ceiling of my office onto my keyboard. I decry the poor maintenance policies that led to the leak occurring and point out that a system should be in place which allows me to report such incidents and have them resolved in a timeous fashion. Would it be reasonable to ask me to develop a maintenance schedule for campus and perhaps implement a call-center? I think not, because such activities clearly belong with facilities management.

I can think of endless cases where I lack the skills or the time to fix things that bother me. I have already put in a lot of time developing half-baked solutions to problems that could be solved far better by people who are actually trained in this work. I am not a programmer, I am not a systems analyst.

Let's say I want to build a house. I think it is reasonable to approach an architect with a functional spec (I want a nice kitchen, I want the house to be cool, I like this kind of design) and expect him to develop a detailed design rather than drawing up plans for the house myself. He is bound to do a much better job because this is what he is trained to do. I think it is similarly reasonable to point out the gaps in the admin system and expect them to either streamline the system themselves or at least bring in someone who can.

Lecturer rant: there should be a system for marks

2010-12-05T21:00:00.001-08:00

Even after six years at the university I am often amazed at how stupid we are.

I use "we" there because I am part of faculty. I am amazed because of these stupid things (and these are just the ones I know of):

Yearbooks

The university yearbooks are written and maintained by hand. Yep, if you clicked on some of those links and saw lists of repetitive seemingly machine-generated content, you may infer that this is all on a database somewhere and simply sucked in to this document by some reporting system. You would be wrong. There are people in each department tasked with maintaining the system. Changes to the document are posed in the form of a manually generated diff and applied by humans. This means there is no automated consistency checking and no way of generating the dependency tree of a particular subject to answer a question like "I have failed GFU 320 - does this cost me a year?" because the dependencies are simply listed in the document. I have written scripts that scrape the PDFs for useful information like the number of periods allocated etc to check that they are consistent with the timetable (see the next point) and generate graphs like the one below that show the dependencies of our course. Why is there not an automatically generated graph like this with clickable links into the subject descriptions on the UP website? The graph is generated using Graphviz

Timetables

There is no single, up-to-date timetable for the university as a whole. Once a year the timetable book gets generated from a program called Syllabus Plus. After this point, all the advanced features of Syllabus, including distributed access by multiple people, custom web-based timetable views etc are ignored and the Word file containing the manually reformatted timetable information sorted by subject code is sent to each department. Each member of faculty then has to traverse this document searching for their subject and draw up a timetable in the traditional two-dimensional grid form themselves. This is a huge waste of effort! I have written scripts that read a dump prepared from Syllabus by our helpful IT staff in an Excel file that changes subtly each year even though I have given a detailed spec of the format I require and prepares timetables for the school of engineering.

Unfortunately, this is not the end, because the venue bookings are handled on a totally different system which does not write back to Syllabus. This means that at any given moment it is literally impossible to reconstruct the exact timetable as it stands without talking to every lecturer involved, as the ad-hoc changes are only relayed to them and not recorded centrally beyond the subject name (not the group or other information you may need to generate the timetable). The whole situation is silly and costs everyone involved hours of their time as they sift through the book trying to verify that the information is correct. What makes it worse is that several other universities are allowing staff access to the syllabus plus system while we are using what boils down to one step above manual scheduling.

Student marks

Student marks are calculated by each lecturer using home-rolled spreadsheets that are almost guaranteed to have an error in somewhere. The rules to determine for instance if a student qualifies for a supp are subtle and often incorrectly implimented. To give you an idea, here is the logic for if a student qualifies for a supp in the school of engineering from the 2010 yearbook:

In the School of Engineering a supplementary examination is only granted in instances where:

A final mark of between 45% and 49% was achieved;
A final mark of between 40% and 44% was achieved and where the candidate also achieved either a semester mark or an examination mark of 50% or higher;
A pass mark has been obtained, but the required subminimum in the examination section of the module or divisions thereof has not been obtained.
A final mark of between 40% and 49% has been obtained in first-year modules on 100 level.

I have made a graph showing the regions for a 50% semester mark exam split:

Is it not glaringly obvious that this should at least be coded up in a reusable spreadsheet function that can be distributed to everyone? Of course, that's just a sticking-plaster, because the system is broken in more subtle ways.

At the beginning of each course, an automatically-generated class list is sent to you by admin. It used to be a CSV file, now it is an Excel file (already my scripting has taken a knock right there). You then calculate marks in any way you want and send the marks back. Unfortunately they now send you a revised class list which does not have the same students in the same order as the previous ones. So what could have been a simple copy-and-paste now either becomes tedious lookup calculations or even more tedious manual lining up of the data. I think it is entirely obvious that we need things like the following ideas:

Accelerated marking system you scan in the memo and the papers, define active regions on the memo and mark directly into the system. The papers have a barcode sticker from a pile you issue the students at the beginning of each year, so your (lecturer) workflow involves going into the system and clicking through the memo for each paper. Totally transparent to the student and externals, a lot faster than carting around tons of papers and shuffling through them.
Grade calculation based on a tree structure with calculation nodes most all grade calculation can be seen as a tree-like aggregation of primary grades. Final mark is weighted average of semester and exam, each paper is sum of questions, which is sum of subquestions etc. You need a builder and navigator, with colour highlighting and live stats to answer questions like 'why is this guy's semester mark so low' or 'what will my class average look like if I adjust my semester mark'.
Audit trails all changes should be historied (BIG fail with spreadsheet-based systems). If an adjustment is made, this should be noted for this student so that one can track the total 'free marks' a particular student has received. This can also be used in
Automatic edge case detection: Students that are 'close' to particular boundaries like distinctions or failing or supps should be identified by the system and automatic adjustments should be suggested. This would of course be historied so that you can review how much action has been taken. With a proper system, this would include all the student's subjects, so that the system would warn you if a student (for instance) only needs your subject to pass his whole course and has not received a lot of free mark. I have even thought a good system could be that you start your course with 0.5 percent per test (so 0.5 for semester mark, 0.5 for exam and 0.5 for final mark), so 1.5 per course. This is the adjustment we allow today anyway, as we round each of those marks. Lecturers can use some of these marks for adjustments. The system could even find optimal distribution of the free marks to maximize the chance that the student succeeds in the course. I mean, we're only probably repeatable within about 2% to 5% on marks anyway.
Aggregated statistics lecturers should be able to see the distributions of the other courses this group has, and be warned about strange anomalies like an otherwise good student scoring really low or an average student doing particularly well.

The thing that amazes me most is that we have all this student manpower doing projects to design websites and database systems and intelligent algorithms for fault-finding and inference, but we're not using any of it on our own systems! I am sure that the development of a system with some of the features above would not be beyond our student body. A part of the problem is that no-one sees development of such systems as part of their job. If one person were to try to develop such a system they would end up wasting a lot of their own time, as the total loss of time is distributed among so many people. This means there should be a higher-up decision about introducing more efficiency in the system rather than a lone developer working on it. Unfortunately, the small losses aren't accounted for in an actionable way and things keep going like they're going. Perhaps I will should send a link to this blog post to my head of department...

Turing completeness is a trap

2010-11-16T20:43:00.000-08:00

Or why you can do everything in Excel but it may not be a good idea.

My wife's struggle with a spreadsheet from work yesterday got me thinking about this topic again. It has perplexed me for a long time that people use Excel for so much that other tools are clearly more suited for. There are many rants on the Internet about using Excel for database-like activities, and I will probably write a bit more on that later. However, I think the key phrase in the overuse of any powerful tool is "but I can do that in my tool, too".

What I mean is that one could approach someone who is an expert Excel user and say "I don't like Excel for engineering calculations because it doesn't allow me to track the units of the numbers like for instance Frink or Mathcad does". This guy would say "but you can do that in Excel - just do X or Y". Somewhere they are thinking "I will use another tool when I find something that Excel can't do". However, they are stuck in a Turing Tarpit. The problem is that all Turing complete languages are equivalent in power in this strange and abstract sense that it is possible to do the same calculation in both. This does not say anything about how easy it will be to do that calculation, or how maintainable the code will be - these are requirements that have very little to do with the computational power.

So, you if you are taking the view that you will use Excel as long as it is possible to do the thing you want to do, you literally don't need any other tool. Unfortunately the same can be said about any esoteric programming language like INTERCAL or x86 assembly. It is perfectly valid to say "but I can do that calculation in INTERCAL - I don't need any other languages". In a strict sense this is true, but this really just points out how little Turing completeness actually buys you in terms of useful programming structures.

At this point it seems like a good idea to mention the other thing that keeps people in Excel. I think the marketers of spreadsheets have been using the "no programming required" line for so long that everyone thinks that spreadsheets are different from programming. In fact, they can be understood as functional programming languages with a two-dimensional (or three-dimensional if you take sheets into account) structure. Of course, there's also VBA if you aren't sufficiently resourceful to figure out how to do everything using only the built-in functions and cells, but the spreadsheet itself is also a computational device. So a lot of people think that "using Excel" is different from "programming", when it differs mainly in the environment.

Bottom line: if you are a single-language kind of person (who only uses Excel or only uses C) you need to understand that it is true that you will never find a program that your favourite tool won't be able to solve in a strict sense, but that it is also true that other tools may make it a lot easier.

Life as a data source

2010-11-08T20:37:00.000-08:00

Tracking progress is a powerful mechanism for improvement, as we found out with our weight loss attempts the year before we started running. Using the power of feedback (I am a control man after all), I was able to reduce my weight and keep it off. My daily weight measurements are a part of my routine and I just love analysing the stats.

This graph shows my weight since I've started weighing myself at the end of November 2008. I estimate that I weighed above 100 kg when we started with the weight loss, but the first measurements are around 95 kg. This means I lost between 20% and 25% . Most of that loss was linear, around 1 kg/week as predicted by a 4600 KJ/day dietary energy deficit. As a rule of thumb, fat has about 32MJ/kg stored energy, so to lose a kg of fat you need to have that much energy deficit. These figures are approximately converted from the Hackers diet, which is a really nice reference for weight loss.

We have nice scale that also measures body fat via an impedance measurement. I don't trust the fat percentage much as an absolute thing, as it is highly correlated to the measured weight, as shown here:

However, it does seem to track my more subjective gauge of flabbiness, and as any control person will tell you, measurements that get the direction right are already helpful, even if they aren't accurate in an absolute sense. This is especially true here, as the target for body fat is just as low as possible (since it is highly unlikely that I will be able to get it down to anything approaching unhealthy levels).

Many people advocate exercise for weight loss. When you consider the kind of energy deficits you need to really lose weight, my personal take is that it's easier to eat less than to work out more. In fact, when you're losing, your energy levels go down so you're even less likely to work out. What worked for me was to lose most of my weight and start exercising when I was lighter.

Based on the success of tracking my weight, when I started running, I initiated the activity by buying a Garmin ForeRunner 305 so that I could track my progress. It turns out that the Mac software stores all the data in a sqlite database, so I whipped up some some python code that pulls the stuff out of there and analyses it. It's hosted on Github. A lot of the time spent writing the code was put into writing a routine that removes all the duplicate entries from the database. I've tried getting Garmin to do something about it by posting a request on their forums, to no avail.

After cleaning up the database, the scripts take each run and approximates the speed from the displacement data. Everything is resampled to one second intervals to make data processing easier. I started out plotting everything with gnuplot - at some point I suppose I should make the move to matplotlib for all the plotting, but separating the plotting process is a bit easier with gnuplot and a Makefile.

The results: Here's a speed histogram, showing the distribution of speed for each run. This is handy to see if for instance training for higher speeds is working.

The representation I am most proud of is this graph:

This shows the pareto frontier of my running performance over speed and distance. On the top right the world records are shown, and I also plot a percentage of the world record speed as a target. That percentage is at the moment calculated from a 20 minute 5 km target.

If people had told me how much data could be extracted from running, I may have started a lot sooner. Life is a great data source

Factors to consider when choosing a programming language

2010-10-26T23:31:00.000-07:00

This morning a colleague and I spoke briefly about choosing a programming language for some high-performance scientific computing (thermodynamic calculations) he wants to do. I started writing an e-mail with a couple of my thoughts, and then thought it would make a pretty good blog post for the same amount of time. So here's my list of things to consider when choosing a programming language. Note that these are not orthogonal to one another - many seem to describe the same thing from different angles. This is just my musings, not a research study.

Popularity. This is a very important one. A good place to start is the Tiobe index. You are more likely to find people to collaborate with if you use a popular language. You are also more likely to find reference material and other help. Unfortunately, the most popular language globally may not be a good match for your problem domain.
Language-domain match. Choose one that matches your problem domain. You can do this by looking at what other people in your field are using (after adjusting for popularity, so don't think the match with Java is good simply because a lot of people are using Java) or by looking at some code that solves problems you are likely to have and seeing how natural the mapping is.
Availability of libraries. Some would argue that this is the same as the point above, but I don't think so. If there's a library that solves your problem well, you'll put up with some ugly calling conventions or hassle in the language.
Efficiency. Languages aren't fast - compilers are efficient. Look at the efficiency of compilers or interpreters for your language. Be aware that interpreted code will run an order of magnitude slower than compiled code as a rule of thumb.
Expressiveness. The number of lines of code you create per hour is not a strong function of language, so favour languages that are expressive or powerful
Project-size. Do you want to be programming in the large or programming in the small? Choose a language that supports your use case.
Tool support. Popularity usually buys tool support (and some languages are easier to write tools for). If you are a tool-oriented user, choose a language with good tool support. Just read this article on tool mavens vs language mavens before you make a choice.

Note that all these things have no single right answer: they define the languages on my Pareto front. A good starting point to observe the trade-off between expressiveness and efficiency is the Computer Shootout. Also check out this analysis of some of the results.

Of course, a personal blog need some personal input, right?

I happen to know many languages, and the combination I use at the moment is Python for prototyping and Fortran (2003) for speed. Python is popular (it's the second interpreted language on the Tiobe index, after PHP, which sucks for scientific stuff). It's got a really nice set of libraries for the jobs that I am doing now and makes wrapping Fortran code easy. Fortran has some really good compilers (and the free gfortran is pretty good), and suits matrix-oriented programming really, really well. YMMV

Both of these languages have good tool support in Emacs (my favourite text editor) and Eclipse (which I am slowly picking up).

What does "it works" mean?

2010-10-21T00:02:00.000-07:00

Response to my rants about the hologram bracelets and articles like this one about magical thinking on SciAm have forced me to re-evaluate my language. I would like to say that the hologram bracelets "don't work" because they have no physical effect on the phenomena that they claim to affect. However, people who use the bracelet will probably have a measurable improvement in their performance as per the SciAm article. So what to do?

Ever since reading Ben Goldacre's Bad Science, I've been fascinated by the term "no better than placebo". The placebo effect is one of the reasons for using blind testing and shows what kind of rigour needs to be employed to avoid being fooled. Now, the idea is that a medicine is only really effective if it works better than placebo. Is there a corollary that says that if a medicine will not be particularly effective, it is better to attempt placebo treatment? I suppose this makes a kind of sense if the medical solution has physical side effects but the placebo doesn't. I just can't bring myself to think that I should start believing in crystals and holograms so that I can have cheap placebo improvements to my performance. Is this why superstition persists? I suppose it should be no surprise that it would have been selected away unless it conferred some advantage.

Unfortunately this brings us to an uncomfortable point. If the superstitions really do confer advantages (even if only mental) what is the sceptic's method of achieving similar benefits for similar (low) cost? It costs nothing to wear gold thongs to important interviews - what should you do if you honestly don't believe the thong will work? In fact, what does "work" even mean in this context? I hate semantics.

Programming language pragmatism

2010-10-17T12:09:00.000-07:00

So, it appears as though I have broken my long GUI drought by writing my first functional Mac OS X app in Objective C. In the process I have also discovered why GUI programming used to be so easy for me when I was younger: I was less picky. The app I have written is a really small thing that calculates pace and speed. However, the amount of duplicate and ugly code in there to get it to work is mind-blowing after having spent some time programming in more dynamic languages.

I guess a part of it is that I don't know the language very well, but a lot of it is just that Objective C is a very small extension of C, and that lacks the kind of mechanisms I have become used to. In another post, I'll elaborate on how the different languages one uses informs how you think about programming as an activity. I am definitely feeling the blub effect (read that link - it's really worth it), in that I'm missing a lot of features from other languages in Objective C.

The other interesting thing I'm noticing is how the tool (XCode) that you use to program is a part of the programming experience. I am discovering first hand how little many people feel the pinch of verbosity in their languages due to good tools. Even though this article is pretty old, I still feel it rings true. In the language of that article, I am a language maven rather than a tool maven. When I start learning a language I imagine writing code in a simple text editor. It's then that typing and elaborate method names really seem like they're a pain. In an environment that automates a lot of that stuff for you, you're less aware of the language and more of the overall tool experience. I still can't say that Objective C is a beautiful language, even with XCode. But then, I've never been a fan of the curly-brace languages in general other than a couple of years of loving C when I was in high school. I've kind of migrated to languages that scan more like spoken language.

Well - no-one said that blogging had to be deep all the time right? What do you guys think about the language vs tool debate? Just to get things started, I've noticed that most of the people who absolutely love static typing are tool users. Static typing makes the tools more useful, as they can figure out the types of things while you're typing and suggest completions of methods and so on. Obviously this is significantly harder for dynamically typed languages, so the tools seem much less capable. Maybe I should learn more tools.

Dogma and change fatigue

2010-08-03T21:41:00.000-07:00

There is never time to do it right, but there's time to do it again. Use the right tool for the job. If it's worth doing it's worth doing right. These are some items of dogma that I have lived by, and they are making me unhappy and a bad worker.

Many people have described me as an idealist. I have also recognised a certain evangelical streak in myself. Combine this with the values described by the phrases above and you have a recipe for Quixotic enterprise. This idea of using the right tool for the right job seems quite simple, but there's a hidden catch: what is the right tool? What is the right way to do something? When you really hold on to the dogma of "Do stuff right using the right tools" you are setting yourself up for a lot of pain. You start searching for better tools (to make sure you're using the "right" one), you start searching for better ways of doing things (to make sure you're doing it "right") and you eagerly evangelise the latest thing you've learned as the answer to these questions.

Of course, somewhere in the back of my mind I know the whole idea is bogus. I guess I was a bit happier when I knew less methods of doing things. Not because I was less enamoured of the dogma above, but because I was more convinced I was doing things right. Looking back, I find myself thinking I was doing things all wrong. So how was I able to be so productive? Because the dogma is wrong, or at least wrong when you interpret "right" as "best". The idea that there is only one right answer to vague questions like this is of course pretty immature. I am learning now how coupled the idea of "right" is to what you know, your skill level and the greater community. Perhaps I should try and focus on getting stuff done using what I have, and looking at projects that have actually delivered results for me, that's pretty much what I had to do.

But this brings me to the change fatigue part of the title. Once you realise that you should be freeing yourself from dogmatic adherence to infantile rules, you start searching around for better rules, right? Hang on - now we're back to the same problem, and we add a new one - change fatigue. Doing things in one way creates baggage. It affects the way you think and it creates artifacts built using those methods. It represents an investment. All of this means it is hard to change on so many levels that you may find yourself experiencing change fatigue - that feeling of unfamiliarity which drains your will to do things and makes you less productive. This is what I am experiencing now.

I should have a poster made, I think, with "Le mieux est l'ennemi du bien" (the best is the enemy of the good) printed on it. Or perhaps I should find a way to let go of dogma altogether and just get things done. After all, aphorisms may be pleasing but Voltaire also said "Un bon mot ne prouve rien" (a witty saying proves nothing).

Productivity: perception vs reality

2010-07-22T08:55:00.000-07:00

I am haunted by the thought that somewhere someone is doing what I am doing more efficiently. I pride myself on knowing efficient ways to do things. Note that this is somewhat different from "being efficient", which in many cases involves not learning new ways of doing things you will probably just do once. I have slowly learned that I am fooling myself when I figure out ways in which to do things better next time, as there probably won't be a next time and my time is largely wasted.

That said, the feelings aren't going away, and in fact I have learned things that make it even more difficult to figure out what's going on. Witness this discussion - the meat of which is that keyboard shortcuts feel faster but mousing is actually faster as measured by a stopwatch. Oh no! Now I need to time myself doing common tasks instead of relying how productive I feel? This brings up an even more interesting question: What is more important to me day to day, being productive or feeling productive?

The sad truth is that I have come to the conclusion that much of my investment into learning lots of different ways of doing things and even the environments that I enjoy the most have been focussed on making me feel productive rather than improving my actual productivity. Unfortunately I am now experiencing the sunk cost dilemma. I have spent almost 9 years of my life learning (and I stress that I have aimed to become proficient rather than just getting by) Linux, grep, sed, awk and all the unix tools and gnu plotutils so that I can do tons of stuff from the terminal. I have learned Emacs, memorising its arcane and non-transferrable shortcut keys and how to interface it with the shell commands. In Emacs I have learned AUCTeX for latex editing, where I often write code for pictures. I also use org-mode a lot for task management. I suppose my investment into learning Python was the least contentious of these decisions. On the way I have also shed Matlab and ignored powerful but expensive tools like Mathematica in favour of SciPy and Maxima, and lately Sage. I have replaced Simulink with Modelica.

I am now beset by doubts and I wonder if I shouldn't have taken the blue pill after all and when I started getting into Open Source when I graduated, just continued with my Windows box, installed Visual Studio and rekindled my original love for C++, used MS Word for my Masters and Matlab or even Excel for all my data processing, or perhaps laid my hands on Mathematica or MathCad. Would I not be more capable now and would I have not gotten more things done in the end by not going against the flow?

Unfortunately there is no easy answer. I am typing this on a MacBook Pro, which I love because Mac OS X has a nice unix layer beneath it but also because there are less hardware issues than I have endured using Linux. I have not yet been successful in using XCode to do anything meaningful because I don't want to write an app that I can't use on my Linux box at work. I have found myself developing code that I am forced to maintain and run simply because I am the only guy on my floor that has Linux on his machine and I have not wrapped the stuff in a GUI so that our secretary can run it. So I experience the need post facto of being able to create GUI apps and have other people run them, and of having a common environment that people can help me with.

However, every time I look into the "more productive" way I feel like gouging my eyes out. I hate word processors of pretty much every description, as the whole philosophy of "when it looks right it is right" galls me, and figures don't float - what's up with that? When I try to use Eclipse for development, I am up against the problems of using such environments for dynamic languages, in addition to the fact that all my current projects' directory structures don't lend themselves to Pydev's idea of what a Python project should look like. In fact, when I try to use statically typed languages, I feel totally put out by having to think of the type information in the first place. I cannot find an easy way to get from where I am now into the mainstream, and it annoys me.

I have vowed to stop looking around for now, until I have finished my PhD, but once that is done, should I become a good little MS drone? Have I wasted my time learning these skills or can I justify my rationalisation that I am more productive now? Does that even matter when I feel more productive (and happier) when I am using my current environment?

Why is no-one using electronic signatures?

2010-06-16T21:46:00.000-07:00

At the end of every semester we go through a little dance at the university called external examination. We send the exams to an external examiner and they have to indicate the paper is well set and fair, then we send them some of the marked papers and they indicate that they have been fairly marked and that the marks are reasonable. This process is monitored and our accreditation body (ECSA) requires that we keep the paper trail on file to check that everything was done right. They require a signature on a hard copy of the documentation at every stage of the process.

At the moment I am also going about the business of registering as a professional engineer, which also requires much signing of documents. After the signing by the people involved, it has to be signed by a commissioner of oaths and sent on to ECSA as a hard copy.

These situations have me wondering yet again why people don't use electronic means for these processes. Perhaps I should append "in South Africa" or "in the Engineering community in South Africa". An argument I hear quite often is that you need a handwritten signature for legal purposes. This is balony, as one can read in Act No. 25 of 2002: Electronic Communications and Transactions Act. A good summary of what the act means for digital signatures in SA can be found here, but I will give an even more summarised version: digital communication has exactly the same legal standing as hard-copy communication in South Africa. So there is no legal reason for people not to just e-mail one another and write their names at the bottom of the mail for non-contract communication just like any other time. When a legally binding contract is entered into, we need an "advanced digital signature", which boils down to signatures using public key systems like S/MIME or PGP.

So, why is no-one using these systems? I think the first is that most people still print out everything they want to keep, and digital signatures tend to be invisible. Printing out the document invalidates the security of a cryptographic signature and there is no formal way of ascertaining whether the document printed out was signed or not. Big problem for people who don't use computerised filing systems. Unfortunately, this means they do something very bad: they trust faxed or printed out signatures. Many people are ok with a document as long as it looks as if it has been signed. This is ludicrously simple to cheat, as all the things that make a physical signature difficult to fake are not present on the fax. I am absolutely certain that I could trace a signature and have it look legit after faxing. I am even more certain (because I have done it) that a couple of minutes with a photo manipulation software gives me a signature I can put on any document I wish.

Which brings me to the second reason no-one uses digital signatures: people think of computers primarily as an extension of the "real world". The mental model encouraged by Office software vendors is that the electronic document is just like the real thing, only on the computer. This is great when the metaphor is solid, but many things that are used in computers just don't have any physical equivalent, and these things become difficult to understand. This doesn't mean they can't be used (what's the real-world equivalent of Facebook?), it just means you need to package it right. I reckon Adobe's idea with a little ribbon that appears on a PDF when it's been validated is almost there, but it's not ubiquitous enough for people to recognise what it means, and I don't think there is a good way of validating the document after it has been printed.

I think there is a real gap in the market for a cryptographic signature that would work with mobile phones -- a datamatrix in the corner that would match an image hash of the whole document. Lots of interesting implementation details there, but a good way to make hard copies trustable.

So, what am I going to do to stop the madness of printing out documents, signing them and scanning them or faxing them? Pretty much what I'm doing already - affix a picture of my signature to the PDF and get on with my life. Legally, I could just as well have written my name, but it keeps people happy. I will also refer them to this blog entry and the law when they claim a legal necessity for hard-copy signatures.

What do you think - when will the hard copies go away?

Free will of the gaps

2010-05-07T03:27:00.000-07:00

The following is a paper I will be presenting at our Philosophy Department's colloquium entitled "SCIENCE, RELIGION AND THE FATE OF TRUTH" . The working title is "Free will of the gaps". I have tried to restrict myself to 2500 words (about 15 minutes of talking). Please drop a comment -- it's not entirely polished yet, but I think the gist is there. I think I need to focus on the ending a bit more than the premises, but the general idea is to draw a couple of implicit parallels between free will and religion, and how science pushes against both concepts.

When I was a child it was easy to imagine that I could control things at a distance. If I concentrated hard enough, it seemed like I could move a glass over the table at will just as I could move my hand. As I grew older I learned from experience and at school that the reach of my will was limited. I learned that, to move the glass I would need to use my hand; that to move my hand required the movement of internal muscles, which required electronic impulses from nerves which ultimately came from my brain.

I am a chemical engineer by training, and as such I tend to view the world with a certain pragmatism (not unlike philosophical pragmatism) that is shared by people who try to build things that will work. A nice theory is all well and good but it won't hold up a bridge, or generate electricity or produce better fuel, so engineers attempt to find models and use "will work'' and "won't work'' as close approximations of "true'' and "not true''. For this reason, I have studied for many years the way that the world works. A philosopher may point out that I have studied the way things appear to work, but an engineer would say that's irrelevant. A scientist may go a bit further and say that science takes great pains to distinguish between apparent correlation and actual causation, that the kinds of mistakes that people tend to make in their perceptions are well studied and corrected for by techniques like strong hypothesis testing and blinding.

Models of how things work are engineering axioms

I have also studied mathematics. John von Neumann (also qualified as a chemical engineer) said that you don't ever understand mathematics, you just get used to it. Although the field is viewed with some disdain by many, I have learned that mathematics is simply shorthand for logic: on the one hand, it is a way of using logical results without having to go through the trouble of proving them, on the other it helps to ensure that your own proofs are logically consistent. It also provides a conceptual framework that guides your thought along probably correct routes. Unfortunately, it is not a method of ensuring that all the statements you make are true in any sense, as Bertrand Russel had to learn the hard way thanks to Kurt Godel.

Engineers use math to work from the axioms to conclusions consistently

What I have learned from my studies of engineering and mathematics is that models like Newton's laws and Einstein's general relativity, together with some logic, predict the interaction of things larger than atoms with high accuracy. Whether engineers are designing buildings or spacefaring vessels, they have sufficient models without thinking about philosophy at all. Applied to the human body, these rules combine to place constraints on the effects of my thought. I cannot move the glass on the table with my mind. In fact, I cannot directly move my hand to move the glass. My hand moves due to electrical signals reaching muscles, the electrical signals originate in the brain or in autonomous systems in my body. The reach of my will has been beaten back to the brain.

Engineering pragmatism also demands parsimony of its models. Scientific parsimony is a rule of thumb also known as Occam's razor that holds that one should favour simple explanations over complicated ones. For scientists and engineers, this means that models that predict the results of experiments equally well should be distinguished based on their complexity, which is usually calculated from the number of assumptions or tunable variables. Newton's famous F=ma is a remarkably parsimonious model of movement. Einstein's model of relativity is more complex, but is more accurate than Newton's model when things are moving at high speeds. Einstein is often viewed as having proven Newton wrong, but Isaac Asimov warns us about the "Relativity of Wrong''. He explains that science (and the pragmatic view) is about improving the models, which means that Newton's models are less accurate than Einstein's, but both are accurate in certain circumstances.

So, it would appear (at least at the macroscopic level) that the universe proceeds by fixed, simple rules. Much of what we see is explained to great accuracy by such simple rules. If everything proceeds in this manner, with fixed causes leading to fixed effects via fixed models, the entire run of the universe is deterministic.

Engineers treat the macroscopic world as deterministic

This is not to say that the overall behaviour of the universe is simple or predictable. The complexity attainable by simple rules is amazing. Stephen Wolfram has shown that cellular automata (which are very simple rules for manipulation of numbers) can be computationally complete. This means that remarkably simple mechanisms yield systems with behaviour that is arbitrarily complex. Anyone who has ever zoomed around in a Mandelbrot fractal, or has struggled with the solution of the Navier-Stokes equations will attest to the beauty and complexity that can emerge from these simple manipulations. This complexity is the one of the chinks in the idea of predictability, embodied by Laplace's demon, which uses the rules governing the universe together with complete knowledge about the state of the universe at a single point in time to predict the future.

Chaos and complexity place limits on practical predictability

Another problem with predictability is that when we zoom in even further than atoms, the kind of models that engineers are used to using start to break down. Here we are in the quantum world, where things are much less intuitively satisfying. The best models we have are confusing and do not mesh with the macroscopic models. The Copenhagen interpretation of the wave equation implies that at a subatomic level, certain events are truly unpredictable beyond simple probabilities.

Quantum mechanics may imply limits on actual predictability

Daniel Dennet identifies three levels of model that people use to predict the behaviour of an object. The models engineers typically use are either physical stance or design stance models. That is, they use physics (like Newton's laws) or reasoning about the design of objects (a watch was designed to tell time, so it should show the correct time). Normal humans often predict how complicated things are going to behave by thinking about their intent or purpose. This is called the intentional stance. For instance, we may predict that a hungry lion will want to eat us and use this knowledge to move away from it. When interacting with humans, this prediction is harder, although it becomes easier with more intimate knowledge. Spouses or good friends may complete each other's sentences.

Unfortunately, this kind of intentive modelling is easy to over-apply. We may start trying to predict what a machine "wants''. Children may claim that a rock doesn't "want'' to move, and even adults often lament the unwillingness of their computers or bemoan how hard it is to find out what they want. The overeager application of intentive modelling is an example of how we think about the world may be wrong in the engineering sense of working -- a good understanding of the rules that govern a computer program is more accurate than trying to figure out what it wants.

Engineers tend to trust the physical stance more than the intentional stance

If the extent of my will has been restricted to my brain, and we try to get better understanding of the brain by moving from the intuitive intentional stance to the physical stance, we are drawn into the field of biology.

My exposure to chemistry leads naturally to an interest in biology. And what a time for an interest in biology! Every day we are making new discoveries about biological systems including our own bodies. Genetic research has accelerated as computer power has grown and methods for sequencing and correlating genetic information are making it easier than ever to understand the mechanisms behind the functioning of our bodies. Biologists are starting to develop physical models that engineers like: deterministic, calculable and predictive. Engineers are using the concepts behind evolution (selection and mutation) to optimise their models and finding that these techniques do work. In addition, they are making it more difficult to suppose that any external forces are in play other than selection and mutation leading to more specialised organisms over time. These models are quite good, but as we probe deeper into the brain, we seem to find nothing but physical matter where we expected to find our thoughts.

What is more certain than our thoughts? Renee Descartes surmised that everything is subject to doubt but the act of thought. Unfortunately, he realised that one could go no further, to probe the truth or validity of these thoughts. Douglas Hofstadter explores how self-awareness can emerge from the models that we have of how minds work by using equivalence of certain kinds of computation "Godel, Escher, Bach, an Eternal Golden Braid'', which should be required reading for anyone interested in the origin of self-awareness. Many other researchers have tried to defend the idea of self-awareness arising from mechanistic models even though no-one has as yet claimed to be able to build a self-aware machine. Few people contend that self-awareness is contradicted by modern science. However, there is an interesting stronger statement about what is called "free will'' or "agency''. For the purposes of this paper, we will use a colloquial definition of free will that involves

the possibility of more than one future that is uncontradicted by the present condition of the universe and
a property attached to a person that is able to make the final call about what ends up happening.

An example of the first condition a may be that eating pizza or pasta tonight may both be possible. An example of the second would be that you could choose either without any forcing reason.

The general consensus among modern philosophers seems to be that free will is an unprovable property. It is one of those topics that is hotly debated and well explored, but often ends in an "oh, let's agree to disagree'' embargo on further discussion. Why is it important? The largest impact of free will is in morality and ethics. A significant part of our legal systems is built around the idea of free will. It forms the basis for our concept of punishment and personal responsibility. If there is no free will, many of these constructs are logically inconsistent.

This ties in with math, science, and biology in an interesting way. I have mentioned that modern science appears to imply that at a macroscopic level things are predictable in the short term. There may be unpredictability due to chaos effects (where even small changes in inputs can cause large changes in outcomes) and quantum effects (where subatomic particles can exist in many states at once, only collapsing to one state when measured), but there is no science that shows that we can do what we like. It is a subtle point, but one must be both aware of one's existance and able to take independent action for us to be able to "do what we like''. Current science does not contradict the idea of a self-aware mind -- it is when we start saying that this mind can exert an influence that the problem begins.

It is clear from observation that certain physical constraints are in place. As near as we can tell, there is no "mind over matter". I cannot move the glass directly with my mind. But now, the glass has moved into the brain. If my brain is made up of the same kind of things as the glass: atoms, where is the "secret sauce''? It is remarkably elusive.

Religious apologists are critical of "God-of-the-gaps'' defences of the existence of a deity. Lay people may say things like "science hasn't yet created life'', and may point out "missing links'' in evolutionary chains, but they are cautioned by people like Dietrich Boenhoeffer that "If in fact the frontiers of knowledge are being pushed further and further back (and that is bound to be the case), then God is being pushed back with them, and is therefore continually in retreat''. In many ways free will has been defended in similar ways.

Everyone has their favourite example of free will: the existentialists had their ledge, almost everyone thinking of an example will try to do something unpredictable, or point out that my defence of my deterministic point of view contradicts itself. However, we are learning every day how our behaviour is determined by our physiology. Transcranial magnetic stimulation (TMS) can be used to selectively deactivate parts of the brain by placing the head in your head in a strong magnetic field. Experiments have shown that it is possible to induce feelings of awe, sensory illusions and other odd feelings by exposing the brain to a magnetic field. It is common knowledge that levels of various chemicals alter our perception and abilities. What is most interesting about results with TMS is that many of these effects are not as clearly categorised as altered states as for instance mind-altering drug use. A recent experiment reported in PNAS by Liane Young and coworkers shows that moral judgement can be altered by TMS. What makes this result interesting is how altering or disabling parts of the brain has such clear effects on things that people attribute to be a function of free will. From another angle, John Conway and Simon Kochen have proven that if we have free will, then so do subatomic particles. Now, I don't know about you, but the idea of a subatomic particle having free will is pretty strange to me.

Doing something unpredictable doesn't prove free will any more than adding unpredictable behaviour to a robot using dice or a quantum event would make us say that it had free will. Even more disconcertingly, it is difficult to think of an experiment that would conclusively prove the existence of free will. Because there is no requirement of predictability, it is unreasonable to demand a perfect predictionn of the outcomes of a human decision before we concede. In the end the idea of an unconstrained free will is pretty solidly contradicted by modern science and our personal experience. Our will is restricted to our body, then our brain, and even our brain is restricted by its physiology. On the other hand, mathematical consistency shows the limitations of any computational device, whether it is silicon or carbon.

Science is advancing ever further into our free will of the gaps

Installing scikits.timeseries on Mac OS X Snow Leopard

2010-03-28T00:19:00.000-07:00

Here is a nice list of steps (and dead ends) to get scikits.timeseries going on Mac OS X Snow Leopard. I am putting it up here for myself and for other people wishing to get going as soon as possible.

So, scikits.timeseries lists python >= 2.4, setuptools, numpy >= 1.3.0, Scipy >= 0.7.0, matplotlib >= 0.98.0 and pytables >= 2.0.

I am assuming you already have matplotlib and scipy and all that, and Snow Leopard comes with python 2.6 already installed. Now you need to get pytables going. For that you need to install HDF. If you go to the HDF site and you follow all the way to the Download page you may think you've hit the jackpot when they mention there's a nice pre-built binary for you to download. Don't do it, as it will not work with pytables! Rather, download the source for 1.8.4 patch 1 and compile it yourself (tar -xzf tarfile; ./configure --prefix=/usr/local && make && sudo make install). Should be fine if you have xcode installed.

Now you are ready for pytables. sudo easy_install tables works fine if you've got hdf installed as per above. Last step is to install timeseries itself. sudo easy_install scikits.timeseries works fine.

If the import fails (go into python and type import scikits.timeseries) with a reference to numpy 1.2.1, you may need to delete the numpy that Apple packaged with Snow Leopard. Simply do sudo rm -r /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/{numpy-1.2.1-py2.6.egg-info,numpy_APPLE_DEFAULT}

Timeseries is a great package for working with timeseries that have dates attached. Perfect for timeseries segmentation and analysis for your PhD!

Equation solving interface

2010-02-21T08:45:00.000-08:00

In moment of clarity I realised today that many of the things that I want to create as GUI applications can be generalised to a single application: An equation solving interface. I have played around with various equation solving environments like EES and PolyMath, also math environments like Mathcad but none of them work like the thing I am starting to see in my head.

The idea is like this: You set up a set of variables and constraints, which may be equalities or inequalities. You give the variables names and dimensions (length, time, etc). So far so normal. Traditionally you would now start to specify your problem, perhaps with the tool checking when there were enough specifications to solve the problem. In my vision, you get an interface consisting of entry boxes for each of the variables. You can enter values in any box, and as more information becomes available, the other boxes are filled. Some colour coding helps you to see which values you filled in recently and which ones have just been calculated. Hovering over a variable highlights the values that are linked to it, perhaps with an overall and direct distinction, perhaps with a drillable menu.

So the first mode of operation allows you to start typing the information you have and the unknowns are calculated as they can be. Now you can play with the values, and select which calculated values should stay the same and which should be recalculated (perhaps using that drill-down menu). You can select some variables to be "Fixed". You can elect to change some of the text boxes to sliders, which allows you to "play" with the solution. You can change some of the boxes to ranges, which will use interval arithmetic to calculate ranges on the other variables. You can change some of the variables to Min or Max, if you select more than one you get multiobjective optimisation. From the same interface, you can see visualisations of the current position in the design space and the constraints that you have placed via the equation setup and via the ranges you have imposed on the other variables. When you find a nice combo, you save it and it gets added to a list of tagged points in the plot views.

Other nice things you can do: automatic nondimensionalisation, perhaps with plotting of variables and implied boundraries (that's why you specify dimensions on the variables instead of units).

This generalises problems like doing brewing calculations, designing equipment and many of the things people use spreadsheets for.

A final note about why spreadsheets and programming languages aren't right for this job. It all has to do with the directionality of equations. Equations go both ways. In fact, a set of equations is like a graph connecting all the variables in it. On spreadsheets and in most programming languages, equations are one-way -- a cell can contain a value or a formula, not both. In some constraint programming languages, it's easier to be declarative, but most of them enforce complete specification and don't even try to calculate values if the whole problem hasn't been correctly specified. They also don't provide direct interaction that allows one to play with the solution.

Perhaps I'm missing something, but I haven't found anything like I've just described. I really want something like this, so perhaps this should the project that I finally break my GUI programming block on. If you have any information or comments, please pipe up!

Mr Fixit

2010-02-13T08:18:00.000-08:00

I am typing this from a 2009 model MacBook Pro, while my wife uses my previous laptop, a PowerBook G4 circa 2002. I loved my PowerBook, but there was a fatal design flaw in the power supply. The cord that came out of it was prone to weakening and breaking off. It failed in 2008, when I did a repair like this one, cutting off the bad bits of wire and soldering the good bits back together. This held out fine for a while, until it broke at a similar place (not where I fixed it before) due to the poorly positioned wire endpoint. I tried to fix it again, but something went wrong and the power supply blew.

Of course, by this time Apple has moved on. The power supply is still available, but it's pretty damn expensive (more than R800) and would only be here in two weeks if I ordered it today. So we went out and bought a cheap (R400) universal power adapter and spent some time researching the voltage and polarity of the G4 PowerBook power supply. For those of you wondering, it's 24V DC with a connector just like a 3.5 mm headphone jack. The tip and sleeve aren't connected to the power -- they're used for the light ring that indicates the status of the power. The power cord itself is positive on the ring connector and negative on the back connector.

So, after exhaustive testing to avoid frying my beloved laptop, I had a working power supply again. This is why every home should contain a multimeter and soldering equipment.

Applicable or unavoidable

2010-02-11T06:57:00.000-08:00

I have had a couple of moments of clarity recently, suddenly seeing how the words people use mean such very different things. Last year I realised that my approach to learning differs from most of the students in my class. I am an obsessive reader and need a lot of big picture structure with cohesive theory to understand stuff. My students are mostly experiential learners who don't like to read and need to do problems to understand things. I skip the examples in books, they skip everything else. This means that most of my sincere attempts at teaching them would be destined to fail until I started making a concerted effort at using numbers in my examples.

The title of this post is related to another realisation. I am an integrator. I try to find application of all the knowledge I have. I don't believe that truly useless knowledge exists, which is probably why I have such a vast store of trivia tucked away in the recesses of my mind -- I'm holding out hope that it will come in useful one day. This means I have often misinterpreted what people mean when they say they "haven't needed" a particular thing they learned in school or at university. When I say something like that I mean I couldn't find application for it. When they say it they mean they have been able to avoid application. This seems to be a very common goal in people's jobs; to avoid application of their knowledge as much as possible. I speak to people and they've come up with poorly implemented versions of what should have been Wiener filters because they've avoided dredging up some signal processing theory they learned a while back, or they don't program on principle, and spend hours doing a task that could have taken them seconds to Google and write down.

Now, as I read that paragraph I can see how superior it comes off, but I am pretty sure my obsession with application has cost me a lot of time. It has definitely cost me a lot of time arguing with people over needing a particular piece of theory because I didn't understand what they meant. To my mind, when you have a filtering problem you need a good filtering algorithm, which means you need to dredge up some filtering theory. To their mind they don't need it if they can come up with their own implementation, and if what they come up with works ok, it's good enough.

The take home message here is that I am trying harder to use numbers in my examples and to search for the proper semantics when people speak. Do they mean they don't need in my sense as in applicable or need as in unavoidable or need as unavoidable?

Where "added value" comes from

2010-01-12T09:41:00.000-08:00

One of the things that have often bothered me is the idea of added value. I'll admit to not having read enough market theory, but it's been pretty easy to understand how refining an ore leads to a product with higher value due to its increased utility. For bulk chemicals and products, the market is very efficient and bulk product prices tend to be marginally above production cost. Where things become weird is with services and branded products. Many of us have thought about the irrationality of paying twice as much for the same shirt simply because it bears a particular logo.

Then you start thinking about counterfeiting, especially about the concept of originality. Let's say you're willing to pay R100000 for an original Pierneef. If a person approaches you with a perfect copy of this painting, why would you not pay the same amount for the copy? The answer is that you are attaching some value to originality. I suppose it should be said that market theorists would probably explain this by saying perfect copies deminish the value of the painting by increasing the supply.

While driving to fetch some pizza from the best pizza place in Pretoria, it occurred to me that cognitive biases could help to explain much of the strategies associated with good business beyond the bulk manufacturing sense engineers tend to think of. I'm still working on the details, but briefly, people do not generally have access to perfect rationality. The biases listed in that wikipedia page all show how our perceptions and estimation of the odds are actually a bit off. When you can craft a strategy that bends as many of these biases in your favour as possible, you will end up with a product that will appear to be worth more than its utility to most people, and you will be able to get a lot more money for it. I'll be thinking about this more and posting, but leave a comment if you think I'm on to something (or even better, if you can point me to alternative explanations).

Don't trust science, look for evidence.

2009-12-29T08:02:00.000-08:00

My wife has the flu (along with several other people I know), so it's time for the usual recommendations from her mother to use a lot of vitamin C. Bad Science has fanned the flames of my natural combative streak, so I went out and tried to find a good study for the Vitamin C thing. I found this, and sent it on to the mother-in-law (who hasn't replied yet). I also posted it in reply to a friend on Facebook's status about dosing up on vit. C. Not long after that, a friend of hers replies with this link -- implying, of course, that anyone can prove anything by linking to stuff on the Internet.

So, of course, I reply, mentioning the number of people involved in the trials I had quoted, the significance of the results etc, which are all in the article and their links, while the other link has some hand-wavey trials with very little real evidence. And while I was posting this reply (which I'll quote, because I'm lazy):

"Your point was quite clear, which is why I had to reply: It's very popular at the moment to roll one's eyes and point to all the contradictory evidence on the internet, typically ending in "so in the end we don't know as much as we think" equivocation of the options. The reality is that in many cases we do know quite well what is happening (as in the case with Vit. C), and the underlying meme that science is always to be taken with a pinch of salt is damaging to proper understanding of scientific method and results.

This story repeats itself so many times that people become less likely to investigate the real results themselves (note that I'm not saying we should just trust scientists, we should be capable of understanding and evaluating the results). This goes for antiretrovirals, vaccines and many other treatments that have been shown to work, but are being used less than they should be because of general public distrust."

Of course, I can think of many other examples: Cell phones and/or their towers causing cancer (which is silly even on the face of it considering the power of the transmitters, but has also been pretty solidly disproved by long studies of cancer incidence), stuff in plastic water bottles causing cancer, overhead power lines, induction stoves causing sterility (same argument as with the cell phones). I think there is a general distrust of science, perhaps because people trusted science so much in the past and have become disillusioned.

So here's the thing: science isn't really a trust thing. The whole point of science is that the evidence should be so overwhelming when you make a positive statement that it cannot be avoided. In fact, many scientists would say that no hypothesis can ever be proved, we can just fail to disprove it after trying very hard. Reading popular media summaries of scientific work makes this difficult because headlines like "Scientists do many tests and find insufficient evidence to disprove this hypothesis" don't really sell papers. It also doesn't gel well with the way people talk and think in general. The problem is that most of our decisions every day are made in extremely unscientific ways, including a wide array of effects that mean we can't even reason accurately about our own experience. Trust is a proxy for research. If someone I trust a lot (or a source I trust a lot) says something, I am unlikely to repeat their experiments and likely to take them at their word. If I know something about the field in question, I can also interrogate my understanding of the situation to see if the results are consistent with other things I know to be true. In an ideal world, we would all be repeating all the experiments to make sure they work. In reality, we lack the time, skills and funds to do so, so trust is necessary.

Where does this leave us? We need to base our trust on good, objective measures. In science, this means being able to look at the results and weigh the evidence, not the authors or publishers. It means checking consistency of theories and avoiding repeating claims that have poor statistical significance. It means doing real science, and not cargo cult science. More on that later.

OpenModelica Mac OS X binaries

2009-12-28T04:14:00.001-08:00

If you are getting strange error messages when trying to run OMShell or other parts of the stuff that gets installed when you install OpenModelica from the Mac OS X .dmg, try using the .command files instead of the application files. This is discussed in the README.osx file.

The error you will see is

basic_string::_S_construct NULL not valid

PySUNDIALS problems

2009-12-21T05:24:00.000-08:00

I'm still installing JModelica. I am now in the annoying "nothing broke during install, but somethings still fail when I use them" phase. I am using the instructions here, and using the example to test if things are working because JModelica doesn't seem to have a test suite. This is what is happening now:


In [1]: import jmodelica.examples.cstr as cstr
/Library/Python/2.6/site-packages/jpype/_pykeywords.py:18: DeprecationWarning: the sets module is deprecated
import sets
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)

/Users/alchemyst/ in ()

/usr/local/Python/jmodelica/examples/cstr.py in ()
  9 from jmodelica.initialization.ipopt import NLPInitialization
 10 from jmodelica.initialization.ipopt import InitializationOptimizer
---> 11 from jmodelica.simulation.sundials import TrajectoryLinearInterpolation
 12 from jmodelica.simulation.sundials import SundialsDAESimulator
 13 from jmodelica.optimization import ipopt

/usr/local/Python/jmodelica/simulation/sundials.py in ()
 14
 15 try:
---> 16     from pysundials import cvodes
 17     from pysundials import ida
 18     from pysundials import nvecserial

/Library/Python/2.6/site-packages/pysundials/cvodes.py in ()
 39 import ctypes
 40 import sundials_core
---> 41 import nvecserial
 42
 43 realtype = nvecserial.realtype

/Library/Python/2.6/site-packages/pysundials/nvecserial.py in ()
 56 PVector = ctypes.POINTER(_NVector)
 57
---> 58 nvecserial.N_VNew_Serial.restype = PVector
 59
 60 nvecserial.N_VLinearSum_Serial.argtypes = [realtype, PVector, realtype, PVector, PVector]

/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ctypes/__init__.pyc in __getattr__(self, name)
356         if name.startswith('__') and name.endswith('__'):
357             raise AttributeError(name)
--> 358         func = self.__getitem__(name)
359         setattr(self, name, func)
360         return func

/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ctypes/__init__.pyc in __getitem__(self, name_or_ordinal)
361
362     def __getitem__(self, name_or_ordinal):
--> 363         func = self._FuncPtr((name_or_ordinal, self))
364         if not isinstance(name_or_ordinal, (int, long)):
365             func.__name__ = name_or_ordinal

AttributeError: dlsym(0x102e957d0, N_VNew_Serial): symbol not found

It's clear something is wrong with pysundials. On closer inspection, it seems that the dynamic library isn't loading correctly. A part of the problem is explained by this:


$ nm /usr/local/lib/libsundials*dylib

/usr/local/lib/libsundials_cvode.1.0.0.dylib:
0000000000000000 s __mh_dylib_header
              U dyld_stub_binder

And many more like it. It seems like the .dylib built by the sundials build process doesn't include the correct names. After much browsing and learning about how .dylibs work, I realise that this can be solved with a simple change to the configure script call (typical - it was actually documented: use --enable-shared in the build!):


FFLAGS="-arch x86_64" ./configure --prefix=/usr/local --enable-examples --enable-shared
make
sudo make install

Now nm shows the items in the dylib.

I know sundials is working right because the examples compile and run fine. Now for pysundials:


In [2]: from pysundials import cvode
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)

/Users/alchemyst/tmp/pysundials-2.3.0-rc2/examples/cvode/serial/ in ()

/Library/Python/2.6/site-packages/pysundials/cvode.py in ()
   350         if ret <> 352 cvode.CVodeSetFdata.argtypes = [ctypes.c_void_p, ctypes.c_void_p]
   353 cvode.CVodeSetFdata.restype = ctypes.c_int
   354

/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ctypes/__init__.pyc in __getattr__(self, name)
   356         if name.startswith('__') and name.endswith('__'):
   357             raise AttributeError(name)
--> 358         func = self.__getitem__(name)
   359         setattr(self, name, func)
   360         return func

/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ctypes/__init__.pyc in __getitem__(self, name_or_ordinal)
   361
   362     def __getitem__(self, name_or_ordinal):
--> 363         func = self._FuncPtr((name_or_ordinal, self))
   364         if not isinstance(name_or_ordinal, (int, long)):
   365             func.__name__ = name_or_ordinal

AttributeError: dlsym(0x1014d04f0, CVodeSetFdata): symbol not found

And the original thing I was trying to do:


In [1]: import jmodelica.examples.cstr as cstr
/Library/Python/2.6/site-packages/jpype/_pykeywords.py:18: DeprecationWarning: the sets module is deprecated
 import sets
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)

/Users/alchemyst/tmp/pysundials-2.3.0-rc2/examples/cvode/serial/ in ()

/usr/local/Python/jmodelica/examples/cstr.py in ()
     9 from jmodelica.initialization.ipopt import NLPInitialization
    10 from jmodelica.initialization.ipopt import InitializationOptimizer
---> 11 from jmodelica.simulation.sundials import TrajectoryLinearInterpolation
    12 from jmodelica.simulation.sundials import SundialsDAESimulator
    13 from jmodelica.optimization import ipopt

/usr/local/Python/jmodelica/simulation/sundials.py in ()
    14
    15 try:
---> 16     from pysundials import cvodes
    17     from pysundials import ida
    18     from pysundials import nvecserial

/Library/Python/2.6/site-packages/pysundials/cvodes.py in ()
    44 NVector = nvecserial.NVector
    45
---> 46 cvodes = sundials_core.loadlib("cvodes")
    47
    48 #Linear Multistep method constants


/Library/Python/2.6/site-packages/pysundials/sundials_core.pyc in loadlib(libname)
    62                 lib = ctypes.CDLL(libpaths[libname])
    63         except OSError, e:
---> 64                 raise OSError("%s\nCannot load shared library %s. Please check you config file and ensure the paths to the shared libraries are correct."%(e, libpaths[libname]))
    65         return lib
    66

OSError: dlopen(/usr/local/libsundials_cvodes.dylib, 6): image not found
Cannot load shared library /usr/local/libsundials_cvodes.dylib. Please check you config file and ensure the paths to the shared libraries are correct.

So the error has changed to it not finding the library. Something must be screwy with paths. Frustrating afternoon.

Matplotlib missing pylab

2009-12-19T06:53:00.000-08:00

I love matplotlib because it allowed me to move away from Matlab to Python as my platform of choice for quick problem solving. It is a very good plotting library. During my previous post on installing JModelica, I mentioned that matplotlib is failing. I recompiled a lot of stuff and finally thought I had it all licked (some of the problems had to do with gfortran not doing 64 bit code by default). Then I ran ipython, and import pylab fails. Strange, considering that import matplotlib works fine. Even stranger, ipython -pylab works fine, too.


In [1]: import pylab
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)

/Users/alchemyst/tmp/ in ()

/Library/Python/2.6/site-packages/pylab.py in ()
----> 1 from matplotlib.pylab import *
    2 import matplotlib.pylab
    3 __doc__ = matplotlib.pylab.__doc__

ImportError: No module named pylab

After some very frustrating searching around and reading of forum posts, it finally dawns on me that I am in my tmp directory, where the matplotlib directory I built it is still around. So that's what's happening! import matplotlib is importing the module from this directory. There is no pylab module there, so it's giving the error. Hopefully someone experiencing similar problems may find some help in this post.

Building JModelica on Mac OS X Snow Leopard

2009-12-15T21:58:00.000-08:00

I am again working on my PhD and I need to get my development environment sorted out. I have almost had it with OpenModelica, because it is so hard to get going on my Mac. I have tried to contribute code to the build process, but the configure script still bombs out based on environment variables that have to be set by hand and so forth. I have therefore decided to try JModelica instead. I duly followed the start of the install instructions, including grabbing a version of the repository using svn (I was not warned that the repo is about 130 MiB, though!).

Installing Ipopt

Ipopt is an interior point solver, part of the COIN-OR project. JModelica includes some nifty optimisation functionality via Optimica so it is quite good to have Ipopt (even though it is not required). So I download the latest Ipopt tarball and now I'm faced with more dependancies: BLAS and LAPACK. After some toying around with these packages, I learned the following:

The Mac dev tools inclode cBLAS and cLAPACK as part of the vecLib framework (something to remember). However, these don't play well with gfortran on 10.6
You can download and build them yourself with little effort, but most build tools assume you are going to use -lblas to access it, which is a bit harder to override.
Ipopt actually has scripts for downloading these for compiling Ipopt, as documented here (although, crucially, not in the install documentation I was following in the distribution)
The Ipopt instructions advise using the HSL libraries, but the process you have to go through to get them takes time, I've done it, but I still don't have access, so I'm just going for Mumps instead.

So, after wasting much time, all I really needed to do was


cd ThirdParty
for lib in Blas Lapack ASL Metis Mumps; do
cd $lib;
./get.$lib
cd ..
done
cd ..
./configure --prefix=/usr/local # to get stuff where I want it
make

Well, after that the configure script worked, but there was a build error (or a couple of them, all with the same form):


lpBlas.cpp:211: error: ‘dtrsm’ was not declared in this scope
IpBlas.cpp:211: error: ‘DTRSM’ was not declared in this scope
IpBlas.cpp:211: error: ‘F77_FUNC’ cannot be used as a function

Looking at the source, it was clear that F77_FUNC was supposed to be a macro that helps with fortran execution or something. Further investigation (grep -R "#define F77_FUNC") shows that it's supposed to be a mangling thing. Even further investigation shows that one of the configure messages that wizzed by is "WARNING: unknown Fortran name-mangling scheme"). The configure script then just silently doesn't define F77_FUNC, leading to these errors. After scrutinising

Ipopt/config.log

I noticed that ld was complaining about the archetecture. I modify my call to configure thus:

FFLAGS="-arch=x86_64"

Now make runs! Make test succeeds and I can do

sudo make install

and get to the main game.

Building JModelica

So, back to the original build instructions. I modify the configure again to point to the right places.


cd JModelica
mkdir build
cd build
../configure --with-ipopt=/usr/local/ --prefix=/usr/local
make

Three minutes later, success! Now do build docs (which requires doxygen, obtained via fink and graphviz, using the Mac install package), which takes ages.

Python support

Last step is getting Python support, one of the other cool things about JModelica. The docs list the following packages:

Package	URL	Steps I took to install
JPype	http://jpype.sourceforge.net/	Download, run sudo python setup.py install
lxml	http://codespeak.net/lxml/	sudo easy_install lxml
NumPy 1.2.0	http://numpy.scipy.org/	Already installed
Scipy 0.7	http://www.scipy.org/	Already installed
PySundials 2.3	http://sourceforge.net/projects/pysundials/	Install SUNDIALS, change reference in src/sundials_core_aux.c from sundials/sundials_smalldense.h to sundials/sundials_dense.h. sudo python setup.py install
pyreadline =1.5	http://ipython.scipy.org/dist/	sudo easy_install pyreadline
Matplotlib	http://matplotlib.sourceforge.net/	already installed
Nose	http://code.google.com/p/python-nose/	sudo easy_install nose

And now it still doesn't work, because Matplotlib is causing segfaults. I guess it's time to call it a day -- more in getting Matplotlib to go tomorrow.

You can't get there from here

2009-12-07T05:39:00.000-08:00

I am reading "The design of everyday things" by Donald A. Norman. While I was writing the previous post about Eclipse I realised that a part of my problem may be my lack of a consistent mental model for the Project and Workspace that Eclipse forces you to use.

I have seen the modern computer user interact with computers by observing my students and colleagues at university. One of the things that has become very fuzzy to most users is the concept of what a file "is". When I get questions about LaTeX from newbies, they often get confused about the editor they are using to edit their files, the files themselves, and the compiler that produces the final output. I lived through a low-level time and still use the terminal a lot, so the distinction between files and the programs that act on them is very clear in my mind, partly because I often operate on the same files with many different programs. The insight I had while working with Eclipse is that the project gives substance in the IDE to the grouping of files that I just assume are grouped quite well by directories.

When I think of my projects, I think of a directory with a memorable name that groups files, which can either be source files, scripts, library/module files or data. I sometimes use a Makefile to automate generation of outputs that need to be done in a certain sequence or when things change. I understand this process.

IDEs seem to operate on a different mental model, which I have not exactly figured out. You create a "Project". I am not exactly sure where this project "is". I assume when you add a file to the project, an entry is created in some kind of project file that is kept somewhere. In the case of Eclipse, these project files are in the workspace directory, in the case of Netbeans, they are in the directory you created to hold your project. So, fundamentally, you are not supposed to edit these files directly, you are supposed to interact with your project via the IDE. And perhaps this is the biggest problem for me. I don't see the gears turning in the IDEs, so it's hard for me to form a model of what will happen when I do things. And most importantly, I can't use my default method of trying out a few things in the terminal before automating them because there is aren't really separate programs working together.

This forms part of a larger problem for me. I have spent a lot of time (about 8 years) becoming proficient at the Unix commandline. I have learned a new language containing new words. In my mind "grep" is a verb meaning to look for something. I use it without thinking. Other things I use in the commandline without thinking: cut, paste, join, comm, diff, tr, sed (with its language), awk (with its language) and many of the features of bash that make it an efficient environment for entering commands. The keystrokes of readline (the library used by bash for input) are similar to the keystrokes in emacs (and as it happens, many of them are supported by the widgets in Mac OS X). I have also learned several computer languages (I will make that the topic of another post), and because I hate repetitive work so much, I find these tools help me to automate them away.

Now, I'm not saying all these things were easy to learn, but I have learned them and I am here now with all this knowledge in my head and an open terminal on every computer I use. I actually thought I would try to avoid the terminal today as an experiment, but it turns out I had to ssh into my computer at varsity to copy a file. I suppose I could do it with a vastly more involved setup that allowed me to browse that computer like one on my computer, but I am also sure that it would take me a lot longer than just going to a terminal and entering the commands.

So a lot of the cultural cues that are in graphical environments are not meaningful to me. In fact, many of them are downright confusing. Here I must say that most of the Mac's software is graphical and well enough designed that it doesn't bother me at all to use, with the exception of Xcode, which I still haven't quite figured out. Some of the stuff is pretty opaque. I have also realised that I have stopped trying to right click on stuff -- something Windows users do all the time, because the Mac interface guidelines require actions to be available without right clicking as well, so it's almost never required.

So, a long rant about where I am with computer use. What's the punchline? I find myself in a place where I am pretty efficent and can do most of the things I want to do without much trouble using the commandline or the tools I know. I am however aware that there are very good reasons to prefer environments that are more supportive and require less memory (from the user's perspective). I just don't know if I can there from where I am now without feeling stupid and helpless along the way. The irony is that the environments that make me feel stupid and helpless are the ones that are supposed to save people from feeling stupid and helpless in mine (the commandline).

Choosing an IDE - part 2, Eclipse

2009-12-07T04:29:00.000-08:00

I have been using Eclipse for some project editing and a couple of things have improved since the last time I looked at this. I have gotten a faster laptop, Eclipse Ganymede has come out and Pydev has improved hugely due to the commercial arm merging with the open source one.

The good bits

Editing python files is easy and efficient. The discovery of interpreters and installed modules is good and the type inference (which is the hardest part of getting Python editors to work well) is quite good -- better than Netbeans.

The strange bits

(things I think aren't broken as much as I just don't understand why it's done that way):

Why is it that, when I edit a file in another editor, then open Eclipse to that project, it does all the checking that allows it to say 'this resource is out of sync', but that I have to follow the instructions "Go to File|Refresh". Surely it should just refresh itself, or at least make the action clickable in the message. Why doesn't it just load the file as it is when it opens?

What is up with workspaces? I understand (kindof) what they are, but what are the advantages of having workspaces rather than just storing the project information in a file in a directory that I can sync and use on another machine, or zip up and send someone without using the IDE?

How does my workflow fit into the "Project" mold? Let's say I have my PhD, which is a real-world project. Normally I would have a PhD directory and bang stuff related to my PhD in there. Now, there's the thesis document itself which is written in LaTeX, which I can edit using TeXlipse. There's also loads of programs written in many different languages, many with some levels of support in Eclipse. What do I do? What seems to make a bit of sense is to create a PhD workspace and then have projects for each of the programs so that I can edit them in the right mode and have all the right compile options and whatnot. If I just have one big PhD project, how do I tell Eclipse to change behaviour based on the file I'm editing like Emacs does? I mean, if I have some Fortran sources in front of me, I probably want to compile them when I click on the play button. If I have a Python script open I probably want to run it. I guess this is from the use of Eclipse to edit "Applications" -- single, well-defined programs that consist of many files that all end up going into the application and where there is a clear build target to build and run. Perhaps with some tests. My projects just aren't like that.

The bad bits

Refactoring (one of the big reasons I want to use an IDE) is broken for me. Many simple things broke in strange ways. "Extract variable" would place the variable definition inside a function defined earlier in the file, which would break everything if I did it. It's still better than Netbeans, because the Python plugin for Netbeans has renaming as the only refactoring option. On the other hand, the rope support in Emacs means that refactoring just works.

Syntax highlighting is also strangely limited. I have grown very used to editing stuff with syntax highlighting and support for code formatting. Python is ok in Eclipse, but Makefile syntax is not supported (edit: Makefiles are supported by installing the C/C++ dev tools), neither are bash or awk, which I use quite a lot.

I am also very annoyed that stuff isn't indented as I type, but rather that I have to do periodic "clean my code" up formatting (even if it's a single key). I have become spoiled by Emacs, which can usually do the right thing when you press tab (who inserts a real tab in code ever anyway?). So your code always looks pristine. I don't know if it really boosts my productivity, but it's nice to have.

The last thing has to do with how I manage my files on my laptop and my desktop. I sync all my files to my destop when I get to work and sync back to my laptop when I leave. This is important functionality that has to work without me holding its hand. I use unison, which never breaks -- I have been using it daily for almost 10 years and it has not failed me in that whole time. The information that Eclipse stores in the workspaces just doesn't translate all that well between multiple machines. I am guessing this is because I have different versions of Eclipse running on my Mac and on my Linux box (because I haven't upgraded my Linux box). It's not really Eclipses fault, but it's annoying none the less.

The verdict

I get the persistent idea that I'm doing it wrong -- that if I just knew a couple of tricks in Eclipse it would all start to come together, and I must say I enjoy the passive information stuff a lot (the documentation pops up when you start typing, there's autocompletion of methods, there's a lot of highlighting that lets you know there are problems in the code. That's all great, and I would even consider doing my next project in pure python just for those benefits, which is what I suppose drives the one language for everything mentality I see so much these days, but for my existing code base, I'm not going to get the benefits without a lot of swearing using Eclipse.

If anyone reading this has used Eclipse (or another IDE I have not tried yet) and is thinking "Geez, he should just do this or read that", pleas post a comment. I read manuals. I figure stuff out. I can really see how Eclipse can help if you plan to write a Python program that's all in python all the time, but if there are other languages in there, it becomes less useful.

Choosing an IDE - part 1, The Problem

2009-11-27T03:54:00.000-08:00

As I have mentioned before, I am a cli guy. I honestly cannot imagine getting through a day without using the terminal. Drag and drop is all well and fine, but wildcard support and iteration in the shell is hard to do in finder (or explorer or nautilus or konqueror etc). That being said, I can see the benefit of many of the things that a modern IDE supplies. I am a big fan of Emacs, but the bells and whistles are very alluring.

I have some specific and interesting needs due to the variety of tools I have used to cobble together my solutions to problems that I have had in the past. Domain specific languages have filled many needs in these problems, so I need a lot of different syntaxes highlighted.

As an example of a more or less typical project I will describe my project assignment tools. The problem is assigning projects to students for our project subject. The idea is that each lecturer comes up with a couple of project proposals, the students choose their top 10 (in order) and a program then matches projects to students by minimising the "unhappiness" while obeying constraints on how many students each lecturer should get etc.

The process uses the following components, held together by a Makefile:

To allow the students to specify their projects, we use an Excel spreadsheet that verifies that their choices are valid etc. They fill in the project selection spreadsheet and e-mail it to us.
A Python script reads and compiles these spreadsheets into a table
A second Python script converts this table into a MathProg data file
GLPK is used to solve the problem (specified using MathProg)
Unix tools (grep, sed, awk, join, tr, xargs) are used to massage the run results
The massaged results are processed by python to generate an HTML file
The HTML file is coloured using CSS

The parameters of the optimisation will be tuned and the optimisation re-run many times. I will also need to answer many questions about specific students based on names or student numbers, which is easily obtained by grepping the intermediate results generated by the above process. These are the reasons why a single Python program doing the above would be a lot larger than the current files are: I would need to implement persistence and memoisation to avoid rerunning lots of code (which is currently handled by make), in addition to coding querying interfaces to allow queries of the results.

But back to IDEs. My ideal IDE does not force me to use one language for everything. It supports my use of domain specific languages by at least colouring the code, but ideally providing intelligent indentation, bracket matching and language help. It allows complex code refactoring and is aware of scope and different senses of variables. A debugger is a plus, but because I mostly use dynamic languages, I find I use the debugger infrequently. A last hurdle is that I either need the same program on both of the platforms I use (mac and linux) or use a scheme for projects that doesn't break when the project is edited by another program (plays well with others).

Much of what I've described is delivered by Emacs. It has code highlighting, tabbing and bracket matching for almost every language ever developed. It also has debugging and refactoring for python via interaction with pdb and rope, although I will admit it's not as slick as some modern IDEs.

After hearing a lot of people rave about the advantages of IDEs I have decided to give them another shot. So far I have installed Eclipse and Netbeans, as they both run under Mac and Linux. I will be posting some of my experiences here. Please leave a comment if you have some suggestions I should try out. I am really keen to boost my productivity, so I won't shoot down any suggestions out of and (except that I have tried vi and didn't like it that much).

Understanding and intent

2009-11-11T19:42:00.000-08:00

When our ancestors were attacked by animals, they needed a way to predict their behaviour to come up with a counter strategy. An effective way of doing this is to think about the animal's intent. "It's trying to catch me, so if I run here, it will run there" etc. Of course, humans who were good at avoiding being eaten were selected and here we are with our brains wired for modelling intent. Unfortunately, many things to not have intent. Rocks, the weather, computers are all examples of things that are not "trying" to do anything. They exist and behave according to deterministic rules. I say unfortunately, because our over-active intent modelling comes up with intent anyay. Children easily think that simple inanimate things are out to get them (or ascribe some kind of autonomy to them), but grown-ups are often caught by the same trap when they try to understand the motivations of the computers or other complex devices that they use.

I am in two minds about this phenomenon. On the one hand I think that it leads to sloppy thinking, where we are ascribing thought to something that has none, or leveraging a broken model to extrapolate far outside its realm of usefulness. On the other hand, it does make things easier to understand. I think my refusal to use intent when I was teaching programming is one of the reasons my students didn't understand what I was saying. How many times have you heard things like "So, the rock wants to fall, because it's being pulled by gravity, but there's this thing in the way, so it has to go around it" (perhaps only if you listen to the Naked Scientist or are an engineering educator). This is clearly wrong in the particulars: The rock has no intent, it doesn't "know" anything about what's in the way -- it is experiencing an acceleration proportional to the resultant force acting on it, that's about as far as it goes -- but it is right in allowing someone to understand what will happen.

So the riddle is this: how do I leverage this mode of thinking in myself and others to increase understanding or make explanations easier but avoid coming up with a new religion to explain the intent of rocks?

Front end loafing

2009-11-05T20:23:00.000-08:00

I am a big fan of planning. I spend a lot of time on increasing my future capabilities. I learn new programming languages, explore new environments and try to stay up to date on research directions. All of this means that I don't do as much right now as I probably could. Hopefully, however, it does mean that I do more now than I could have done if I hadn't followed this strategy in the past. My industry has a name for getting more stuff done up front: Front End Loading, hence the title of this post.

There is a fine balance between getting more done now and being more productive later. I tend to be a 'sharpen the axe' kind of guy -- I believe that you work more efficiently with sharp tools, that sharp tools require some time to learn and maintain and that that time tends to be less than what you save. Unfortunately, like any habit, this needs to be kept in check as well. When times are tough, my first instinct is to take a step back and find a better tool. Sometimes that is just procrastination. Sometimes it is time to put your head down and keep chopping.

PS: This whole post was really just a test of twitterfeed.

Optimal diamond cutting

2009-11-03T21:36:00.000-08:00

So, they found a big diamond in Argyle the other day, which got me thinking about finding the optimal way of cutting a rough diamond. Key parts of this is knowing the value of different cuts of different sizes, which a colleague of mine told me is documented in a thing called the rapaport. This gives the value of certain cuts of diamond for different weights and qualities.

So I was thinking that one could scan the diamond, build a 3d map of it and then try to maximise the value realised from the diamond by fitting cut diamonds into the rough diamonds. Things to consider:

Inclusions reduce the value of diamonds,
Diamond prices are nonlinearly related to weight
Cutting and polishing is a time-consuming and expensive process

Some of these points have been addressed with some significant simplifications in this paper, but there is a lot of scope for improvement. Too many ideas, too little time.

How can GUI programming be so hard (for me)?

2009-11-02T06:47:00.000-08:00

I do a lot of casual programming every day, to solve problems that I have at that moment. This mostly involves a quick one-liner in bash. Examples of this: finding out which of my students are repeating the subject by comparing this year's class list to last year's, figuring out which episodes of a series I still need to download, and so on. When I find myself doing the same thing more than once, that one-liner gets pushed into a script file that I can call at a later time. The shell allows me to pipe the output from one of these scripts into another or prepare input in many different ways, filtering by multiple criteria and allowing me to select files easily.
From time to time, I get the urge to convert one or more of these things into a nice GUI. To paraphrase a quote, now I have two or more problems (Apologies to Jamie Zawinski). First I need to choose a GUI toolkit that supports the platforms I use every day: Mac OS on the laptop and Linux on my desktop at work. All my scripting stuff 'just works'. I use a blend of (in order of size of job) grep, sed, awk, bash and python, all of which are transparently available on both platforms and have never given me a day of trouble (except perhaps python, but that's a different story). Enter the GUI and things get seriously complicated. Python ships with Tkinter, which is universally acknowledged to be pretty bad. It also has bindings for GTK, Qt and wxWidgets. I have not succeeded in getting any of these to work on both my computers at the same time.

A prime motivation for GUIfying these utility scripts is so that other people can use them more easily. Unfortunately that also means that this all has to work on Windows as well. This consideration, along with the fact that getting a fully functional version of Python running on a windows box takes time if not that much effort, has led me to consider Java for my GUI stuff.
So, now the problem -- I remember doing a lot of GUI programming with Delphi back in the day (when I was still at school). I wrote several programs that did useful things and didn't think it was all that hard. But now that I have mastered the command line, everything I need to do seems so hard in the GUI environment. Java has no standard GUI editor, so I have kind of drifted toward Netbeans for the nice GUI layout tool that it has. No matter how hard I try, though, I can't find the motivation to stick with figuring out GUI programming for the hours that it takes me to get a useful result; where useful is defined as something that actually does something instead of just popping up a window with buttons and all. I get that all of that functionality (having a clickable widget appear on screen, being able to close the window etc) comes at a cost, but figuring out the details takes be ages, and seems entirely orthogonal to the problem I am actually trying to solve.

For instance, here's the python code for a program that gives me a list of students in common between a couple of files:


import sys, csv
classlists = dict((f, [r for i, r in enumerate(csv.reader(file(f))) if i > 1]) for f in sys.argv[1:])
numbers = [set(student[0] for student in classlist) for classlist in classlists.itervalues()]
uniqnumbers = reduce(set.intersection, numbers)
for n in sorted(uniqnumbers): 
     print n

How long would that have taken to GUIfy? Frustratingly long.