March, 2004

—Thanks to Joachim and Jason.

A Java servlet container is a web server, but it’s rare to see a Java servlet container directly attached to the Internet. Most often we deploy servlet containers behind ‘real’ web servers like Apache. Why is this?

The reason is JSPs. JSPs need to be placed into the servlet container’s URL space. However, typical java web applications are based on model two and break if a user points their browser directly at a JSP. If we could configure a servlet container so that a user couldn’t send HTTP requests to JSPs, that would be fine, but the servlet container spec doesn’t provide a way to do that. So we end up proxying our Java applications through Apache to give us control over the visible URL space.

Of course, you can use model one and make all your JSPs user callable, but Java system designers reject model one for all the same reasons they reject ASP and PHP.

In large deployments, using Apache brings advantages in security, manageability and efficiency, but in small deployments, its only benefit is to hide part of the URL space. I find this unnecessary complication galling.

All that said, the standard servlets-behind-Apache model is workable and sustainable in the long term, and so I am not advocating a complete revision of the servlet container specification.

What I am advocating is that future web framework developers avoid Sun’s mistakes: ensure that page templates aren’t placed into the public URL space; better still, ensure that page templates don’t directly respond to HTTP requests at all.

My blog replacement is coming along—about 3800 lines so far. The database layer has been stable for the last two weeks, and I have all my current MT blog entries imported. I’m currently working on the web layer.

Real-life has temporarily got the better of this project. I was hoping to switch from my current ISP to a python server at the end of this month, but the switch will now have to wait until the next month.

Technology wise, sqlite works just fine, and I think I’m over the hump with Twisted and its Deferreds. To reduce the learning curve, I decided to leave Woven, Nevow and other templating systems alone for the time being, which means I’m well on my way to producing my own bastard child of Struts.

The two hours a day on the train is working out well. I can get a lot done in an hour with Python, and I’m finding it easy to pick up where I left off too.

I have lots more to write about the design, when I have a live system.

Virtual interfaces are unquestionably useful, but are a source of amusing anecdotes each time they fail to match the user’s intuition.

My step-father who, confusingly enough, is also named Alan, used to travel around Australia, explaining super-annuation to groups of his company’s miners, loggers, road-builders and factory workers.

Halfway though one such meeting, a gust of wind caught the edge of a slide on the overhead projector. To the audience’s delight, Alan flung out his arms and grabbed the projector screen, and then watched as the image continued to slide gracefully across the screen and off the other side.

Next tour, he got a laptop and data projector.

Phillip Greenspun points to this Infoworld article in which Jaron Lanier referred to Linux as “another Unix”.

“Here we have this wonderful opportunity of creativity in the open source movement,” but programmers built another Unix instead, he said. “What is this?” Lanier asked.

It’s an interesting combination of creativity, lawsuits and open source politics, is what it is.

Linux was started because Linus was frustrated with hacking on Minix.

Linux became the focus of open source developers, the world over, because there was no reasonable alternative; in the early nineties, BSD was buried in a cloud of legal uncertainty, and GNU/Hurd, although long-promised, wasn’t a viable option.

By the time BSD was free of its lawsuit, and had worked out its internal politics, Linux was well on its way to being the leading open source operating system.

It’s reasonable to lament that open source didn’t produce something more adventurous, or more elegant, or more user-friendly, or whatever it is you think an operating system should be. But lamentations won’t contribute to the common good in the way that Linux has.

Q: Why is two thirds of Iraq without electricity?

A: Because the US hasn’t handed over power.

Warning: Use Joke By June 30, 2004, or thereabouts.

Philip Greenspun notes that it is always more expensive to buy [theatre and concert] tickets over the Internet than in-person and usually more expensive on the Web than on the phone.

If a supplier sells their product via distribution partners and via the web, it is (almost inevitably) going to be cheaper for a customer to buy from one of the distribution partners.

The classic case is plane tickets. You can buy them from the airline over the phone or ‘net, or you can go to a travel agent. If an airline were to charge the same price to travel agents as they do to the general public, then travel agents will have a hard time making money out of the tickets. Eventually travel agents will refuse to sell the tickets.

Airlines need the business that travel agents bring in, so they sell tickets to travel agents at well below the the web price. Because they buy tickets on the cheap, travel agents have some margin to use to compete with each other. The overall effect is that it’s cheaper to buy tickets from a travel agent than from an airline.

I wonder if this is part of the reason why Philip is paying a ”‘net premium”.

Apropos Charles’s recent post on C++ and its relationship with Java, here are some notes from my recent, three month stint in C++.

From a technical point of view, the application we were working on was very simple: single user, two-tier design, green-fields development. As a result we side-stepped a lot of the truly interesting C++ problems, such as needing to implement half a garbage collecting memory manager.

1. Like Riding a Bike. I was surprised just how much C++ my brain had retained. I'm a bit annoyed too; I could be using that memory-space for something else.
2. I think I know why Joel doesn't use exceptions in C++. There are three main ways of indicating exceptional conditions in C++: through the global errno, through a return code and through exceptions. The three styles don't mix well at all -- you need to pick one and use it consistently.

   The Windows API uses return codes, so it makes sense to use return codes when programming C++ on Windows. In our case, we weren’t using Windows, but the client was more comfortable with the return codes than exceptions, so that’s what we used.

   Exceptions work well in Java because they are the single, standard way of indicating exceptional conditions, it therefore makes sense to use Exceptions in Java. I can only guess that the reason Joel doesn’t use exceptions in Java is that he is so used to the return-code style.

3. const is a major PITA. const is one of those ideas that is good in theory, but horrid in practice. The main problem for us was that if one bit of the code is made "const correct":http://www.cis.nctu.edu.tw/chinese/doc/research/c++/C++FAQ-English/const-correctness.html#faq-18.1, all the code around it is "sucked into the const vortex":http://www.cis.nctu.edu.tw/chinese/doc/research/c++/C++FAQ-English/const-correctness.html#faq-18.3.

   After a co-worker decided const-correctness was a Good Thing™, I spent several afternoons retrofitting consts all over the codebase. Those afternoons went like this: add a const keyword. Recompile. Find the newly caused const errors. Add another const keyword. Recompile. See one hundred new const errors. Decide a strategic cast of const to non-const might be appropriate. Repeat.¹

   To be fair, it became less of an issue over the period of a month or so as the consts wormed their way through the codebase. Even so, I don’t think const-ing caught a single error, or helped us write code any faster. Watch Scott Meyer shuffle his feet when Bill Venner puts this to him.

   One of the best decisions James Gosling made was to hold off on adding const to Java.

4. Garbage Collection is Good. There were just a few occasions when it was necessary to carefully think through who-is-responsible-for-deleting-what questions. Each time, I was reminded of just how useful garbage collection is.
5. I Modified the Literal Empty String. As in "". "I broke it":http://www.cardboard.nu/archives/000243.html. Seriously! It took four hours to figure out what was going on. Yay for Java immutable strings.
6. The Standard Template Library was useful. Good to see that Java will get Generics soon.
7. Long Compiles Are Good for Blogging. If one were to change certain important header files, one could expect a twenty minute compile, on a good day. Makes me appreciate Java's support for incremental compilation.

This all serves to confirm Java’s credentials for winning over C++ programmers.

¹ Actually I used a bit more thought than that, but that’s the general strategy. Next time I won’t whinge. I’ll just do this:

$ find . -name ’.cpp’ -o -name ’.h’ | xargs perl -i -p -e ‘s/const//’

$ cvs commit -m ‘fix compilation errors’

Mac fans: please don’t read this. Please. It’s quite offensive.

Linux/BSD zealots stop now too. Really. Pages like this are the reason that all open source browsers have a back button. Use it.

It may also make BeOS users sad, too.

After a week of using heavy office-train-home-train use, I’m convinced that Windows XP Pro is a world-class OS for developers. Here is what I am appreciating about it:

• Hibernate and standby-mode Just Work.
• CRT Monitor + LCD when I’m at work or home. LCD only on the train.
• Winamp 5. MAD Plugin. Works with the laptop’s play/pause/stop/next track/prev track soft buttons, so I can run it in the system tray.
• Virtual Dimension virtual desktop manager. I have my train-time on one desktop, and my office work on another, so I can quickly swap between them, without closing any windows.
• gVim is ever reliable.
• Cygwin gives me all the GNU tools and a bash shell, which is way more useful than CMD.EXE.
• FireFox is neat.
• I am off-line caching my mp3 directories from my home network and my project document directories from work. It makes the files look like they are on the network drives, even though they mysteriously are not. Windows lets me decide when to synchronise.
• Most drivers and applications seem to install/uninstall without a reboot.

From where I am now, it would take about a month to get an equivalent Linux setup, though this is partly due to the Linux-resistance of the HP hardware. No doubt the situation will improve with time. I’ll think about it again in six months or so.

The downsides? I have a bazzillion strange processes running, most of which I probably don’t need. MS Messenger insists that it can’t be uninstalled. I will probably have to pay for anti-virus software.

After six weeks of rather tedious to and fro, I picked up a brand new replacement for my HP nx7010 (1.6Ghz/512Mb/60Gb/15.4” WSXGA) last Thursday. It works fine and has no busted pixels (that I have noticed) either. I didn’t leave the store before I checked.

Here’s a quick way to test a machine for broken pixels. Create a little HTML file like this with notepad:

Save it as ‘test.html’ on the desktop, then open it up in Internet Explorer. Press F11 to get full screen mode. You should be able to see any bright pixels over most of the screen. Then change the bgcolor to white and look for dark pixels.

The store kindly¹ loaned me a Fujitsu Lifebook C2220 off their display stand when my first HP had to be returned. The machine worked fine and I was able to develop the database layer of my new blogging software.

However, the experience confirmed my choice of the HP. The HP has a bigger screen and a longer battery life, as well as being smaller, lighter and faster. How much faster? A test that consistently took 12 seconds for a thousand iterations on the Fujitsu takes 4 seconds on the HP. Other tests also show a 2.5x – 3.0x speed up.

¹ I say kindly, but if they hadn’t, I probably would have taken a refund instead of waiting for the replacement.

Charles says Struts and similar web frameworks should be thrown away. He writes:

Give a reasonably good OO developer the task of building a web framework, their first attempt will probably be something resembling Struts. which I suppose gives good support to the “Build one to throw away” theory.

Call me a l33t PHP programmer, but I don’t think Struts is the thing that needs throwing away. The problem runs deeper than that.

In the late 90s, Sun, fresh from their success with Applets, charged their Distinguished Engineers with defining a common standard for web frameworks. The Distinguished Engineers drew on the rush-to-market, look-at-us-we’ve-got-one-of-them-too mentality of the dot-com era, and gave us Servlets and JSPs.

The Servlet and JSP standard is a good first attempt. But, like all first attempts, it should have been thrown away once the world had learnt some lessons from it.

These are some of the lessons I learnt from Servlets and JSPs:

• A web framework must address the whole HTTP standard, including the difficult bits. The framework should understand multi-part HTTP requests, and be able to produce multi-part HTTP responses.

• Creating objects is not expensive. SingleThreadModel should be the default.

• Having HTTPServlet extend GenericServlet is cute design, but it complicates the API (for example, HTTPServletRequest extends ServletRequest, HttpJspPage extends JspPage, and so forth) for no concrete payoff.

• A templating mechanism is not the same as a servlet. JSPPage should not extend Servlet. On a practical level, JSPs should not need to be registered in the server’s URL space, where the user’s browser can call them directly.

• The Servlet authentication mechanism is unsuitable for many uses. A related defect is the lack of a simple, standard way for a web app to logout (or deauthenticate) a session.

The Servlets and JSP approach is so widely and thoroughly accepted by the Java community that it can’t be thrown away anytime soon. However, until it is thrown away, Java web frameworks will be building on crooked foundations.

Casey has some thoughts on web page annotations. It’s a good read.

A distributed annotation service could be based on blogging technology. Annotation blogging would address most of Casey’s concerns, at the expense of some wheel-re-inventation.

Your typical blog entry has a title, text, a creation date, and a “permalink” URL that identifies the entry. An annotation entry would have those attributes, as well as the URL of the annotated page.

I would publish my annotation entries on my website, and my friends would publish their annotation entries on their websites. When I browse a page that has been annotated by my friends, my annotation-aware browser would download my friend’s annotations and display them.

Threaded discussions in page annotations could be added by allowing one annotation to refer to another. This extension would also help annotaters to find other annotaters they might be interested in subscribing to.

The Third Voice annotation system that Casey describes suffered because it did not have a good mechanism for viewing just those annotations from credible sources. With annotation blogging, the source of annotations is as credible as a blog entry—which is good enough for most on-line community purposes.

Annotation blogging would encourage lively communities to form around news sites and popular pundits. Annotation and regular blogging would work in parallel, with blogs containing both annotations and regular entries. It would bring the blogosphere closer to mainstream current affairs.

The technological basis for annotation blogging is in place. If an enthusiastic bunch of programmers got working on it, we could all be annotation blogging by Christmas.

Well, it’s a nice thought.

The Internal Combustion Engine is here to stay. Due to the efforts of generations of engineers, it is acceptably quiet, reasonably safe and only moderately polluting.

I’m sure any sensible visitor to Earth would say, “You crazy humans! Abandon mediocrity! Lay hold of excellence! Work together and you can have silent, safe and clean hydrogen fuel cells in just ten years!”

But this is not going to happen, because the infernal combustion engine is good enough. We are used to it. Mechanics are trained in its care and maintenance. We have a huge fuel distribution network. Hydrogen fuel cells are competing not only with the petrol engine, but also with the infrastructure and culture behind it.

Storing source code in disk-based text files is also here to stay, and programmers are bound to their text files as tightly as society at large is bound to their petrol cars.

It’s not as if there is a lack of options. In a recent post, Chris Petrelli holds forth Smalltalk and Zope as examples of programming systems that don’t need text files. This paper by Roedy Green explores what could be achieved if we gave up text based source files. Others have been experimenting for years.

The problem with text files, as with petrol engines, is that they are good enough. We are used to text files. We are trained to work with text files. All our IDEs, OSes and version control systems are geared to text files. Any other system for storing and manipulating source is competing with computing’s technological and cultural heritage.

The internal combustion engine was invented in 1860. In 2004 it’s still going strong. Programmers have been putting source into disk based text files since 1960. It’s only fair to assume we’ll still be doing it in 2104.

While the reference documentation for the Python 2.3 logging module probably tells me all I need to know to use the logging module, I am just not grokking how it should integrate into my system. In particular, initialising and configuring logging is a bit of a mystery.

A quick Google turned up these links:

• Boudewijn Rempt’s Inform-IT article, Examining Python 2.3: New Additions contains a section on logging.

• PEP-282, authored by Vinay Sajip has a few usage examples.

• Vinay Sajip has published a comprehensive description of the logging package, with many examples.

I’m a software engineering fashion victim, so when somebody comes to ask if some new feature should go in the design or not, I reflexively yell, “YAGNI!” YAGNI requires little thought, and it’s usually a good not a bad a wise-sounding answer.

However, I think Cameron Purdy is riding the crest of a new wave when he writes:

“Use your brain.”

I predict that next week, once the Great YAGNI Flamewar is but a distant memory, the sloganeers will pick up Cameron’s thought, and acronymize it as “UYB”.

So, in future, when somebody asks me about design features, I’ll be yelling, “UYB, UYB!”

This last week has been a really interesting one for SCO:

• After threatening for months, SCO finally sued some Linux end-users. Their case against Autozone is basically this: Autozone converted all their Unix apps to Linux really smoothly, so they must have used SCO intellectual property to do it. Obvious isn’t it? Note the frequency with which the complaint uses the phrase, “on information and belief.”

• ESR got hold of a memo leaked from SCO, and promptly labelled it Halloween X. The memo indicates that Microsoft have been moving heaven and earth to ensure SCO don’t run out of money. I wonder why Microsoft would give money to a competitor?

• C|Net explore the revision history embedded in a SCO Word document, providing some insight into SCO‘s thinking. Unfortunately, C|Net don’t reveal the document’s source or question its authenticity.

Met with Keith, David and Charles for our traditional drink(s) followed by Japanese meal last night.

Keith and Charles spoke highly of the Spring framework, convincing me to give it a go. An interesting aspect of Spring is that Spring components are POJOs, and so can be unit tested independently of their container. This compares well with EJBs, which must have a full EJB container¹ before they can be tested.

¹ Or a reasonable fascimile. The point is, unit testing an EJB takes more effort than unit testing a POJO.

Early versions of Fortran, I am told, allowed the programmer to redefine the value of seven by writing 7 = 8.¹

We did something similar with C++, today, when we accidently redefined empty string (as in a literal ””) to be considerably less empty. It was surprising how much of our application didn’t break.

In the end, we traced the bug back to a call to the transform() STL function. When you call transform() on a string, you pass it a string::iterator telling it where to place the transformed characters. Unfortunately, string::iterator is a typedef for char *, so when you tell it to place the output into a new, empty string, guess what? The empty string literal gets clobbered.

¹ Assuming you hadn’t messed up too many other literals, you could get it back again by writing 7 = 8 – 1.

My mum asked me how the hit counter on her home page works. I found myself at a loss for how I might explain it in a few short paragraphs of clear English.

An explanation shouldn’t be hard, since mum ranks well ahead of ESR‘s Aunt Tillie in the technology stakes. Before she retired, Mum was a COBOL/4GL programmer, she can happily and competently maintain a web-site and she also has a few other projects going on her PC too.

Never-the-less, the web’s melange of protocols, products and pragmatism makes a lay explanation of even basic techniques an interesting exercise.

Here is my best shot:

Hi Mum,

The counter on your front page is a picture generated by a program on one of your ISP‘s servers. The program keeps a count of the number of times it has generated a picture for your page and make a picture out of the number.

A ‘hit’ is jargon for request from a browser to a server. The hit-counter counts the number of hits on the hit-counter picture. Since a browser will request and display the hit-counter picture almost every time it requests and displays your home page, the number on the hit counter roughly the number of times your home page has been viewed.

I went to your home page and selected “View Source” from my browser menu, to look your page’s HTML. Near the bottom of the HTML is a line that looks like this:

This line is what causes the browser to go and fetch the hit-counter image. “lhancock_1.dat” is the name of your counter, which the program needs to know since the same program is used to display hit-counters on many different people’s pages.

You could put a hit counter on every page on your site. If you use the same counter name on every page, it will count the total number of requests to all pages. If you use different counter names on every page, it will count requests to each page separately.

You might have wondered why a hit counter is displayed as an image rather than as text. The reason is that it is difficult to get a web page to include a piece of HTML or text from another source, but straight-forward to get it to include a picture.

Hope that helps.

Glad to hear the new rooster is settling in well.

Love,
  Alan.

What do you think? Did I leave out something important? Is this explanation unnecessarily complicated? What would you do differently?

There’s been quite some discussion on python-dev about PEP 318, “Function/Method Decorator Syntax”. PEP 318 proposes an extension that would allow function and class definitions to be modified—“decorated”—immediately after definition.

The PEP also proposes a similar extension for class syntax. Bob Ippolito commented, ”.. but who knows what else it might be useful for… I imagine it would be the end to some metaclass abuses…”.

Here are two uses of class decorators. Both could be done with metaclasses, but are clearer with PEP 318 syntax:

Singleton Classes

Defines a class with a single instance. The instance is assigned the name of the class. Pro: Design intent explicit. Con: The class object no longer has a module level name.

class Foo(object) [singleton]:
    def bar(self):
        print 'hi'

Foo.bar()

Register All Instances

Sometimes you need to register all instances of a class by some attribute. This is a classic use of metaclasses, but the PEP318 syntax is simpler to read and simpler to explain:

class Foo(object) [registerByName]:
    def __init__(self, name):
        self.name = name

Foo(‘fred)

print Foo.Get(‘fred’)

My $0.02

There hasn’t been a lot of discussion (this time around) about whether the list of decorators should go before or after the name being defined. I think it works better before the name, nearer the class keyword:

class [singleton] Foo(object):
    # ... 

I read this as, “A singleton class named Foo which inherits from object”. I prefer this to:

class Foo(object) [singleton] :
    # ... 

which I read as, “A class named Foo which inherits from object and is a singleton”.

I prefer the first form because the [singleton] decorator radically modifies the meaning of the class keyword. This argument carries less weight for a decorator such as [registerByName].

This same argument applies to function decorators: having the decorator list near the def reads more clearly where the decorator changes the meaning of the def keyword, such as with a [classmethod] decorator. There is less of a case for putting the decorator list first when using decorators with less semantic effect, such as [synchronized].