April, 2004

I wrote this piece in February. I didn’t post at the time because I wanted to link to the online Vim help files at vimdoc.sourceforge.net, but it wasn’t working properly. Two months later, it is still not working properly, so I’m posting anyway.

Everybody’s DNS seems to have switched to point to this new server, so this will be the last of “Unseen Alan”.

Vim is my all-time favourite text editor. Productivity wise, I find it better than every IDE I’ve ever used, excepting Eclipse, which I prefer for Java¹.

I’m currently working on a C++ project. When we started here, we spent a few days setting up the development environment, including Vim. This is what we did:

• Installed a cvs plugin, which lets us see color-highlighted diffs.

• Set up tags over our source code and the library header files.

• Set standard C++ indenting and comment styles in a project-wide .vimrc file.

• Taught Vim how to interpret the error messages from the compiler, linker, and database preprocessor.

Function keys are all set up too:

• F2 opens a file explorer

• F3 swaps between a .cpp file and the related .h file

• F4 does the same as F3, but opens the related file in another window

• F5 kicks off a build

• F6 shows build errors

• F7 and F8 navigate to the next/previous build error

• F11 and F12 navigate to the next/previous tag

Some Vim features that have been helpful are:

• the ‘find’ command, because I am too lazy to remember what directory any given file is in,

• ‘grep’, which searches all of our project files by default,

• CTRL-N and CTRL-P, auto-completion, which I use constantly.

Would I recommend Vim to every programmer? No. Its vi-derived, modeful, “Look Ma, No Arrow Keys Runs On A VT100, A Mac And Anything In Between interface is daunting for those who haven’t had to use vi at some stage. But for a significant fraction of the development community, Vim is the ant’s pants.

¹ Though I haven’t used IDEA. Or Emacs.

My friend David has a way with words. Watch him avoid the saying the big “R”, while explaining how he spent the morning:

“I haven’t added any functionality, I’m just fixing it up, making the code a bit easier to work with.”

I’ve shortened some of the longer permalink URLs. Since the new URL scheme has onl been in use for a few days, this might cause a glitch with aggregators trying to work out what they have and haven’t seen, but I don’t think anyone else will be affected.

In other news, the new blog software is mostly going OK.

The current issue is that that the database file is sometimes being corrupted when the server shuts down. I am not sure why this is. I am taking daily backups.

Another old post from the “Unseen Alan” files. I wrote it after downloading a file full of UPPERCASE BABY NAMES. I can’t recall why I hadn’t posted this one.

Originally written November 11, 2003.

To put a file into title case:

First, make every character lower-case. ”\L” in the substitution pattern makes all following letters lower-case.

%s/./\L&/g

Second, change the case of the first letter in each word. ”\w” in the search pattern finds word characters. ”\u” in the substitution pattern makes the single next character upper-case.

%s/\w*/\u&/g

Simon points to a Register article by Tim Mullen, blaming “clueless users” for their own PC security problems. Tim writes:

We are calling for users to take a little time and to learn minimal skills before attaching their systems to the Internet.

A question that people seem to have stopped asking is, “Why can openning an email trash my PC?” The answer is that Microsoft thought that executing arbitrary code emailed from strangers was a good idea. This attitude pervades their entire OS. It means Windows users have to understand their OS in order to protect themselves from it, even though not all of them are capable of that level of understanding.

Users should be able to mouse and click and tap without having to think about computer security. The idea that waving my hand and wiggling my fingers can ruin a few thousand dollars worth of PC is ridiculous.

I’m saying that it should be safe for users to be clueless.

It works just fine on the Mac.

Here’s another draft post from the “Unseen Alan” archives. This one was originally dated Feb 3^rd, 2004. If I recall correctly, I never posted it because I was so overwhelmed that I was having trouble collecting my thoughts properly. Mind you, it had been a busy week.

If I ever get around to it, I’ll also blog about the simulator computers – a roomful of racked 8086, 80286 and 80386 processors for each simulator.

My friend Glenn works at Qantas, maintaining their Flight Simulators. Last Friday, he very kindly took me for a tour of their Sydney facilities. It was amazing.

The Big Lesson

The big lesson is that hand-piloting a 747 is tricky. This is because you are attempting to push 400 tons of metal around the sky by yanking on a three foot stick. I am not good at it.¹ I didn’t actually write off a plane, but in the real world my flights would have headlined the news for a week and re-ignited world fears over airline security. Passengers still in possession of their mental faculties may have complained, too.

For instance, here is a picture out the window of 767, showing a genuine 110-degree bank.

Graphics, Optics and Cockpit

Here is the view from the cockpit of a 747–400. This is on the main runway at Sydney airport. The new control tower is to the left, and the old one to the right.

The simulators have clever – and expensive – optics that give the view out the windows a genuine feel of depth. The optics make the view ‘feel’ real, despite the blocky, circa-1989 graphics.

Of course, the inside of the cockpit looks and feels real, since it is made of all-genuine airplane parts. A large part of the simulator system is devoted to fooling the avionics and instruments into thinking they are on a flying aircraft.

Movement

The other big ingredient in the realism recipe is the simlator’s physical movement.

The simulators – which look like Star Trek space shuttles – are mounted on hydraulic legs, standing in the centre of a three storey room. The legs have about six foot of movement in them, so they can make sim can move up and down, tilt, and twist.

I gave the motion system a good exercise. I got a good shudder going by spinning the plane sideways off the runway.² During several flights, my unusually aggressive style caused the motion safeties to cut out, leaving the simulator on a permanent tilt for the rest of the flight. Way too much fun.

Thoughts

As I mentioned before, I was shown around the sims by a technician. Glenn has never flown a 747, but he knows how to start one, taxi it, take off, fly it to a destination and land it. Granted, he may not know the finer points, or how to handle an emergency, but he can perform all the basics competently. Qantas has whole buildings full of simulator technians, all of whom can fly Jumbo Jets. I’m not sure where I’m going with this thought, but there’s a b-grade movie plot in here somewhere.

Links

• There are more and better pictures on this site.

• Anthony Hicks has video of a 747 simulator session.

¹ I made a ten bounce landing at Tullamarine. Apparently it was some kind of record.

² Don’t know if it was realistic. Never want to find out.

While building my blogging software, my test data has been all the entries I ever wrote. Amongst them all, I found five or six never-published drafts.

So while we’re waiting for the DNS for cardboard.nu to shake out, I am running a series that I think of as “Unseen Alan.” I’ve given these entries a grammatical tidy up, and re-written the most egregious ramblings, but the basic thoughts are unchanged.

Here’s one I wrote on 25 September, 2003. Note the embarrassing comment about Perl and Lisp. How in the world did I ever hold that view?

In a recent post, Charles points out that it takes a lot of (keyboard) typing to iterate over a list in Java, compared to the same piece of functionality Ruby, Perl, Python, Lisp, Smalltalk, OGNL and Haskell.

It would only be a fair comparison if these languages were suitable for large systems development.

At the time Java was created, Ruby, Perl, Python, Lisp, Smalltalk, OGNL and Haskell were either non-existent or suffered serious deficits as large-systems programming languages.¹ Back then, the in-vogue large-systems programming language was C++.

Update: It turns out that Smalltalk is the best ever language of all time, including future time. It is a better language than French, Classical Latin or Swahili. My apologies to anyone whose sense of reality this post may have previously offended. (True story: All the good guys ‘do’ Smalltalk. Gandalf used VisualWorks, the Hobbits were blessed with Enfin, and Aragorn wielded Digitalk. On the other hand, Saruman coded Java, the Oliphants were fueled by COBOL, and the language Orcs grunted to each other was C – except the Uruk Hai, who spoke ANSI C++.)

Java compares well to C++. As Phillip Pearson points out in the comments on Charles’ post, Java is an improvement over C++ for this task. Java is also well ahead of previous choices for large-systems programming: Assembler, COBOL, Fortran and C.

These days, however, some of the comparison languages are becoming suitable for large-systems development. Python particulary interests me, but Perl is developing in a promising manner and there is a slim, but frightening, chance that Lisp Smalltalk may yet gather a critical mass.

¹ I snort in the direction of the OGNL-Java comparison. OGNL is neat, but it cannot replace Java.

My new blog software is up, and looking stable.

I’m happy enough with it that I just switched the DNS over. This site will become the new http://cardboard.nu over the next two or three days.

Oh, and if you leave a comment, it will now stay.

Enjoy!

I have my new blog software up at http://alang.python-hosting.com/. Still to do:

• Import the project management checklists, downloads directory and other documents.

• Fix the RDF feeds, which are currently pointing to localhost.

• Sort out the Twisted daemon startup.

Hope to get these done tonight, and move the cardboard.nu domain across tomorrow.

Meanwhile, please have a play with the new site and leave a comment (on this blog, because I’ll wipe the new one and reimport tonight) if you find a breakage.

Update: The new site is looking OK. If it stays stable overnight, I’ll be “throwing the switch” tomorrow. Even has a new style sheet with readable sidebar text (thanks, Daniel).

Karen and I saw Paycheck at the cinema last night. I think we caught Hoyts’s last scheduled Australian session. It was exactly our kind of movie: heavy on the sci-fi, action and adventure, very light on the angst, insight and character development.

As the movie unfolded, I couldn’t help but notice similarities with Arnold Schwarzenegger’s 1990 sci-fi, Total Recall, including the hero with a memory wipe, the gigantic conspiracy, the pure-hearted girl, the sneaky substitute girl, the treacherous friend and fortune telling. The reason for this is that Paycheck and Total Recall are based on Philip K. Dick short story We Can Remember It For You Wholesale.

Paycheck’s fight scenes, on the other hand, are faster, lighter and more artistic than Total Recall’s. I didn’t twig as to why until a white dove appeared (for no apparent reason) just as the climactic confrontation scene began —John Woo‘s trademark.

I can’t explain why I typed ‘edlin’ at the Windows-XP command prompt. I haven’t used edlin since 1988, when I had to type up a 100 line Pascal assignment in a PC lab, because all the Unix terminals were full.

Imagine my surprise when it worked.

C:\>edlin
File name must be specified

C:\>edlin /?

Starts Edlin, a line-oriented text editor.

EDLIN [drive:][path]filename [/B]

/B   Ignores end-of-file (CTRL+Z) characters.

C:\>

That’s right. Windows-XP ships with edlin.exe in C:\windows\system32.

Mitchell, Connor, and their friends, Connor and Daniel, were watching Treasure Planet—Disney’s 2002 interpretation of Treasure Island—the other day.

To Disney’s great credit, their John Silver is more complex than the typical cartoon character. After the movie ended, the boys started discussing him. Connor and Connor summed up the confusion, “First he was good, then he was bad, and then he was good again.”

Mitchell replied, “No, he’s just somebody doing things.”

I call this a masterful insight for a nine year-old. It runs deep against pop-culture, which tries to paint every human act as a product of either “good” or “evil,” and reacts with shock if they are ever found together in one person.

Here is a mini-recipe for handling HTML forms that map to domain objects. This has cropped up in my work a few times lately, so I thought I’d write it down.

Scenario: An application has a web page for editting a domain object. The processing for the page can be called to do any one of:

• display a blank form,

• display an existing domain object,

• submit data for a new domain object,

• submit form data to update an existing domain object.

Steps:

1. Examine the incoming request for an ID argument. If you find one, retrieve the domain object with that ID from the database. If you the request does not contain an ID argument, create a new domain object in memory.
2. For each attribute of the domain object, examine the request for a corresponding argument. If the argument is present, validate it. If the argument is present and valid, update the object with the value of the argument.
3. If there were no errors, and the request was a ‘save’ request, then INSERT or UPDATE the domain object to the database.
4. Generate an HTML form to display the object’s data. If there was an ID argument in the request, include it in the generated form.

Commentary:

• Don’t be tempted to skip retrieving the domain object in the case of a save. This step ensures that any attributes of the domain object that are not also on the form are preserved. Bugs caused by disappearing attributes are often subtle and difficult to track down.

• The same request handler is used to display the initial form and to save the domain object. It is appropriate to use the HTTP method to distinguish between these two cases: use POST to save an object, and GET to save it.

• It is an error to have a save request without the attribute data. In a public web application, check for this condition.

• According to this recipe, the form is redisplayed to this user after the object has been successfully saved. In this case, ensure that the user receives clear feedback that their changes have been applied. Alternatively, you may choose to redirect the user’s browser or display a different page after the object has been saved.

• Handling invalid data can be tricky. Typically, if invalid data was entered, then the data the user entered (rather than data from the object) should be redisplayed to the user, along with an error message. Fortunately, most web frameworks provide some assistance with this process.

VoiceXML is schema for defining voice conversations. To date, its main use has been in IVR systems, providing an industry-standard protocol for conversations between applications and the expensive, specialised products that handle voice interaction with the user.

I couldn’t help noticing that with a couple of XSL style-sheets, it would be simple to convert a SOAP message to a VoiceXML message and back again. We could then transport Web Services over voice-grade telephone lines, without using a modem.

What bright vistas does this nifty piece architectural genius open? After many seconds of deep contemplation, I have what I suspect is the definitive list, numbered for your convenience.

1. Whenever your application wanted to order a thankyou gift for its sys-admin, it would send a SOAP request to the URI ”phone://1–800-FLOWERS.”

Behold the mighty power of XML, and tremble.

A Java version of Simon Brunning's Python and RPG UK postcode splitting algorithms.

Compared to the RPG version, the Java code has the virtue of being readable, even without comments.

Compared with the Python version, the Java is more verbose. Braces aside, this is mainly due to the need to return a whole object in Java where Python returns a simple tuple. The 'guts' of the algorithm is about the same size (7 lines vs 5), thanks to Apache Commons Lang.

import org.apache.commons.lang.StringUtils;
import org.apache.commons.lang.builder.ToStringBuilder;

public class Postcode {
  private String inCode = "";
  private String outCode = "";

  public Postcode(String str) {
    str = StringUtils.deleteWhitespace(str).toUpperCase();
    if (str.length() <= 4) {
      outCode = str;
    } else {
      int outEnd = str.length() - 3;
      outCode = str.substring(0, outEnd);
      inCode = str.substring(outEnd);
    }
  }

  public String getInCode() {
    return inCode;
  }

  public String getOutCode() {
    return outCode;
  }

  public String toString() {
    return ToStringBuilder.reflectionToString(this);
  }

  public static void main(String[] args)
  {
    System.out.println(new Postcode("SL18jt"));
    System.out.println(new Postcode("SL1 8jt"));
    System.out.println(new Postcode("N4"));
    System.out.println(new Postcode("GN43"));
    System.out.println(new Postcode("GN43qw"));
  }
}

At the turn of the century,¹ I was working at Qantas, an international airline. One day I saw one of the business analysts walking by with a fistful of petty cash vouchers, which turned out to be one voucher for the renewal of each of Qantas’s domain names, including http://www.qantas.com and http://www.qantas.com.au.

Apparently, one of Qantas’s domain names had expired few years previously, causing quite some grief, so Qantas instituted a new procedure: this one business analyst made a note in his diary of when the registration fees were due, paid them on his credit card and then claimed the money from petty cash. I shudder when I think of the fragility of such a critical business function.

The problem with domain names is that they are so inexpensive. For $19.99/yr (or less), it is just not worth the registrar doing any more at renewal time than sending a simple reminder email, and reminder emails often get bounced or be ignored.

I wonder if it would be possible to build a business out of maintaining domain name registrations for large companies. For the low, low price of 200USD/yr, you could offer domains that have 6 months grace on expiry, and send real, paper invoices. Even after the 1000% mark-up, such a service might represent good value to an enterprise that depends on having a web presence.

¹ I intend to be telling my grandkids stories. Thought I’d start practising the terminology now.

As of Python 2.1, the lambda keyword become a whole lot more useful. Instead of having to pass in each variable value, Python is able to reference variables from the containing containing scope.

For instance, let’s say we have a function called shout(f), which calls its parameter f and prints the result:

>>> def shout(f):
...     print f()
...

In the olden days, if you wanted to shout the numbers 10 through 12, you could pass a lambda function to shout, like this:

>>> for n in range(3):
...     shout(lambda n=n: n+10)
...
10
11
12

While that syntax still works, with Python 2.1 and later, we are able to omit the n=n and write:

>>> for n in range(3):
...     shout(lambda: n+10)
...
10
11
12

However, the two forms of lambda are not quite equivalent. Omitting the n=n causes the lambda to refer to the variable in the stack frame of the function that called the lambda. Consider what happens when we change the value of n after we make the lambda function, but before we call it:

>>> l = []
>>> for n in range(3):
...     l.append(lambda: n+10)
...
>>> for f in l:
...     shout(f)
...
12
12
12
>>>

Putting in the n=n causes the value of n to be copied at the time the lambda function is created, which is exactly what we want in this case:

>>> l = []
>>> for n in range(3):
...     l.append(lambda n=n: n+10)
...
>>> for f in l:
...     shout(f)
...
10
11
12
>>>

I found this little wrinkle while working on a multi-threaded application. One thread was producing lambda functions in a loop, and another thread was calling them – some before the loop was finished, and some after.

Today I got in touch with my ‘inner web designer’ and whipped up some prototype user interfaces for a potential project my bosses have their eyes on, deadline being today.

It turns out that my ‘inner web designer’ inherited his sense of beauty and grace from a 19th century Welsh coal-miner. I used two animated GIFs—one of a beating cartoon heart. The pages only work on IE, maybe only IE 6. There is a stylesheet, but it looks like a C++ program. An obfuscated C++ program. Bleah.

I’m going home to take a shower.

Bespoke software written for Government should be Open Source.

Government software is built under the authority of our elected officials, and paid for with our taxes. In a very real sense, Government software belongs to us, the public. And if the software belongs to the public, why shouldn’t the public have access to the source?

I see four big benefits to Open Source Government:

• Open Government: the public has the option of checking that software is peforming functions correctly and fairly, and according to legislation, regulation and policy.

• Reuse within Goverment: while not all the software will be directly useful in other contexts, open source will promote reuse of common componentry and infrastructure.¹ The aim would be to lower costs over the long term. There would also be concrete benefits to the IT industry.

• Reuse in the wider community: open source allows others to benefit from the work of government programmers, contractors and consultants.

• Developer Accountability: the possibility of public scrutiny and subsequent ridicule promotes quality in the finished product, even though it may raise in the short term.

There are also some questions that need to be thought through carefully:

• Licensing: Should software be released under a strict license or put into the public domain?

• What to keep Closed: Obviously, some software should not be public—perhaps smugglers shouldn’t have access to the Customs Department profiling algorithms—but where should the line be drawn?

Getting started is another matter. Since there is no incentive to government departments to take on the additional cost and risk of open sourcing their software, Open Source Government would need to be introduced as policy. There are precedents for policies like this. In the IT arena, State and Federal governments have policies mandating website accessibility be accessible, and ‘encouraging’ departments to choose to outsource their IT.

Open Source Government is possible. Spread the word.

¹ I shudder to think how many different implementations of, say, a Postcode Lookup function there would be across Australian Government deparments.

Writing my blog replacement software with Twisted I had some dynamic URLs that were difficult to handle the normal Twisted way, so I wrote some code that looked at each segment of the URL. That code was ugly, so I threw it away and wrote another version, which uses regular expressions to pull apart the URL.

The URLs share a certain amount of commonality in their structure, but some parts are optional, and URLs that mean different things look similar too:

• /blog/2003.html - the 2003 archive.
• /blog/python/2003.html - the 2003 Python archive.
• /blog/2004_02_18/blog_replacement_object_model - an article.

The code looks like this, minus comments and logging:

 class BlogResource(Resource):
    isLeaf = True

    def render(self, request):
        restOfPath = '/'.join(request.postpath)
        for (pat, pageClass) in self.MATCH_LIST:
            m = re.match(pat + '$', restOfPath)
            if m:
                return pageClass(request, *m.groups()).render()

        return common.NotFoundPage(request).render()

    MATCH_LIST = [
        (r'(?:index.html)?', BlogIndexPage),
        (r'(?:(\w+)/)?(\d\d\d\d)(?:\.html)?', 
                BlogYearArchivePage),
        (r'(?:(\w+)/)?(\d\d\d\d)_(\d\d)(?:\.html)?', 
                BlogMonthArchivePage),
        (r'(?:(\w+)/)?(\d\d\d\d)_(\d\d)_(\d\d)(?:\.html)?', 
                BlogDayArchivePage),
        (r'(\d\d\d\d)_(\d\d)_(\d\d)/(\w+)(?:\.html)?', 
                BlogEntryPage),
    ]

Step-by-step:

1. isLeaf = true causes Twisted to always call our render() method, even if there are more URL segments.
2. restOfPath is set to the value of the rest of the URL, not counting the '/blog' bit that got us to this class.
3. The for statement loops through MATCH_LIST, looking for a pattern that matches restOfPath.
4. When a match is found, a new instance of the matching pageClass is created, passing in the request, and all the matching groups out of the regular expressions as constructor parameters.
5. render() is called on the resulting page object to render the HTML.

Now that's the kind of trick you can pull in Python, but which would just be hard in Java.