nihilanth

Ubuntu: "bad signature" problems prevent apt-get update from working

2008-07-27T19:28:00.000-07:00

When running a routine package update, I sometimes get errors like this:


$ sudo apt-get update
...
W: A error occurred during the signature verification. The repository is not updated and the previous index files will be used.GPG error: http://security.ubuntu.com hardy-security Release: The following signatures were invalid: BADSIG 40976EAF437D05B5 Ubuntu Archive Automatic Signing Key 
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/hardy-security/Release
W: Some index files failed to download, they have been ignored, or old ones used instead.
W: You may want to run apt-get update to correct these problems

This ubuntu bugs page suggests this:

$ sudo apt-get update -o Acquire::http::No-Cache=true

but the problem persists.
Solved it! By deleting the incorrectly signed Release and Release.gpg files that were downloaded in the last try:


$ sudo rm -f /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_hardy-security_Release*

Now apt-get is successful.
I got the idea above from the instructions in comment 20 to the bug

McCain vs Obama

2008-06-29T08:30:00.000-07:00

From CNN:

Muslim and Jewish groups on Monday sharply criticized Sen. John McCain's comments that he would prefer a Christian president to lead the United States.

The Arizona Republican's remarks came in an interview with Beliefnet, a Web site that covers religious issues and affairs.

"I just have to say in all candor that since this nation was founded primarily on Christian principles, personally, I prefer someone who has a grounding in my faith," the GOP presidential hopeful told the Web site in an interview published Saturday.

Its hilarious that other religious groups are attacking him for emphasizing his religion and excluding theirs, and not because this is blatantly false.

On the other hand, Barack Obama seems to take a more reasonable stand:

Whatever we once were, we're no longer just a Christian nation; we are also a Jewish nation, a Muslim nation, a Buddhist nation, a Hindu nation, and a nation of non-believers. (http://www.cbn.com/CBNnews/204017.aspx)

Here is another quote:

Democracy demands that the religiously motivated translate their concerns into universal, rather than religion-specific, values. It requires that their proposals be subject to argument, and amenable to reason. (from http://obama.senate.gov/speech/060628-call_to_renewal/)

2008-06-29T08:05:00.000-07:00

US Presidential Candiates on Politics vs Religion

I was returning from a social trip to Jakarta. While at the airport, wondering how to kill time, I ran into a bookshop and spied Richard Dawkin's "The God Delusion". I bought it immediately. Honestly, I did it more to dig up interesting factoids to tell my religious wife than because I thought the book would have any new perspectives for the godless geek that I am.

However, I had forgotten that Richard Dawkins writes very well, and it has been quite an entertaining read so far.

I could feel indignation bringing blood rushing to my eyes when I read chapters on how atheism is discriminated against in American politics and how the religious are blatantly lying in the press regarding the United States being founded as a Christian nation.

The chapter "Arguments for God' existence" does a very good job of summarizing the very few arguments from first principles put forward to justify the existence of God.

And the book led me to discover two blogs:

Pharyngula, a blog about "Evolution, development, and random biological ejaculations from a godless liberal", belonging to PZ Myers, a bilogist and associate professor at the Univertisty of Minnesota.

Atheists of Silicon Valley

Building and installing a local debian package

2008-05-27T18:21:00.000-07:00

Recently, I ran into trouble with the 'python-zeroc-ice' package on Ubuntu 8.04. It used to work fine on Ubuntu 7.10.

Some digging revealed that the package python-zeroc-ice-3.2.1-2 claims it supports python2.5, but the C module for ICE is built against python2.4.

So I compiled the package and installed it locally. This is how:

1. Downloaded the source package:
$ apt-get source python-zeroc-ice

2. Installed all other packages needed to *build* this pacakge:
$ sudo apt-get build-dep python-zeroc-ice

3. Fixed the problem:
$ cd zeroc-ice-python-3.2.1 # yes, the source package/dir is called zeroc-python-ice
$ vim config/Make.rules
-> set "PYTHON_VERSION = python2.5" in the appropriate place.
$ dch -i # bring up the changelog with new entry
Added a changelog messsage. The package version, at the top of the changelog, is now python-zeroc-ice-3.2.1-2ubuntu1
$ dpkg-buildpackage -uc -us

4. Install the new package
$ cd ..
$ dpkg -i python-zeroc-ice_3.2.1-2ubuntu1_i386.deb

And we are done. However, the change above means Ice won't be available for python2.4 anymore.

Gotcha with git-svn dcommit?

2008-04-09T01:57:00.000-07:00

http://kerneltrap.org/mailarchive/git/2007/8/21/255431: this should be kept in mind/verified when using git-svn.

Tracking multiple branches in svn repo with git-svn

2008-04-09T01:48:00.000-07:00

For some time now, have wanted to use git and git-svn to track the subversion repo at work. That repo has lots of branches, and individual changesets are merged very often between branches, making it a nightmare to find out what has already been merged, and what not.
Used to keep track of all this in a text file. Its becoming tedious. Have been trying to use git-svn to ease the pain.
On the first try, git-svn gave up halfway during the clone operation with an error message about protocol/network error. This git-svn was the one in ubuntu repositories.
Since then, installed git from source.
First I tried this way. It works, but then gitk will only show me one branch: the one I have currently checked out.
Now, I am trying this other way. Hopefully this will work better.

Code merging

2008-04-05T10:01:00.000-07:00

Wanting to merge two branches of code, both of which have had concurrent commits going on them for some time. Code is kept in svn, so the branches actually live in separate directories. Most changesets have been merged back and forth, yet residual changes remain.

I get diff to tell the files which are different between the branches.

diff -q -r -x '\.svn' -x '*.pyx' -x '*~' branch1 branch2

This produces lines like:


Files zeroc-ice-python_3.2.1/config/Make.rules and zeroc-ice-python-3.2.1-new/config/Make.rules differ
Only in zeroc-ice-python-3.2.1-new/config: Make.rules.GNU
Only in zeroc-ice-python-3.2.1-new/config: Make.rules.GNU_kFreeBSD

I need to munge the output of diff. I want to have 4 columns: directory (sans the branchname), filename, link to the file in branch1 if it exists there, link to file in branch2 if it exists there. I want this ordered/sorted by directory, and I want the output in emacs org mode format. I filter the output of the above diff command through this perl script:

while (<>) {
    if (/^Only in/) {
        m/Only in (.*): (.*)$/; $dirname=$1; $fname=$2; # sep dirpath and filename
        ($branch = $dirname) =~ s/([^\/]+).*/$1/; # the branch name is the first part of the dirpath (e.g. branch-xy/dir/path/file)
        $dirname =~ s/$branch\/?(.*)/$1/; # remove the 'branch-xy' from dirpath to get the remaining dirpath
        $_ = "| $dirname/$fname | [[file:~/workspace/couffable/branches/$branch/$dirname/$fname][Only in: $branch]] |";
    }

    if (/^Files /) {
        m/Files (.*) and (.*) differ/; $left=$1;
        $left =~ m/release-3-11\/(.*\/)?([^\/]+)/;
        if (defined($1)) { $dirname=$1; } else { $dirname = ""; }
        $left=$2; # remove branch name, sep dir and file
        $_ = "| $dirname$left | [[file:~/workspace/couffable/branches/release-3-11/$dirname$left][r311]] | [[file:~/workspace/couffable/branches/release-3-11-airtel/$dirname$left][r311-airtel]] |";
    }
    print $_ . "\n";
}

Tesla = Pylons + SQLAlchemy + Elixir

2008-04-04T02:37:00.000-07:00

Tesla is a Paster template creating Pylons applications using SQLAlchemy/Elixir ORM. Adds some simple database paster commands. Includes the following features:

Create model classes

Simple database commands (create/drop tables)

Migrations (using SoC migrate library)

Create and run batch scripts

Handles SQLAlchemy setup and session refresh

Now, will Tesla/Elixir allow me to have composite primary keys of my choosing? Can I have, for most models, an Elixir definition, and for some, a more flexible SQLAlchemy mapper? How?

The reason I'd like to do that is: ActiveRecord does a poor job of structuring database tables for performance

What a bunch of libraries! Sometimes you wonder if plain PHP with embedded SQL isn't better after all. No, no, that was heresy...

Blogging from GNOME Blog

2008-04-01T21:46:00.000-07:00

After several futile attempts at blogging from within Emacs to Blogger.com, I found the GNOME Blog applet and am using it now. It is not emacs, but at least it is less clunky than the blogspot web interface.

Nose: db setup and teardown

2008-03-31T01:54:00.000-07:00

In the last post I noted some documentation related to nose and ORM. Well, they did not work for me because my setup was not exactly like others'. Here is what worked for me. In

tests/__init__.py

I imported
```
from pylons import config
```
to get the SQLAlchemy engine embedded in pylons's config variable

I imported
```
import quickwiki.model as model
```
so that I could get hold of my models and metadata

I created a class
```
TestModel
```
inheriting from
```
TestCase
```
to hold the setup and teardown code

In the

tearDown

method, I do

model.metadata.drop_all(bind=engine)

to destroy all tables

In the
```
setUp
```
method, I call
```
tearDown
```
to destroy tables if they have not already been cleaned up, and then call
```
model.metadata.create_all(bind=engine)
```
to create the tables.

Here is the final code:


"""Pylons application test package

When the test runner finds and executes tests within this directory,
this file will be loaded to setup the test environment.

It registers the root directory of the project in sys.path and
pkg_resources, in case the project hasn't been installed with
setuptools. It also initializes the application via websetup (paster
setup-app) with the project's test.ini configuration file.
"""
import os
import sys
from unittest import TestCase

import pkg_resources
import paste.fixture
import paste.script.appinstall
from paste.deploy import loadapp
from routes import url_for

__all__ = ['url_for', 'TestController']

here_dir = os.path.dirname(os.path.abspath(__file__))
conf_dir = os.path.dirname(os.path.dirname(here_dir))

sys.path.insert(0, conf_dir)
pkg_resources.working_set.add_entry(conf_dir)
pkg_resources.require('Paste')
pkg_resources.require('PasteScript')

test_file = os.path.join(conf_dir, 'test.ini')
cmd = paste.script.appinstall.SetupCommand('setup-app')
cmd.run([test_file])

from pylons import config
import quickwiki.model as model

class TestModel(TestCase):
    """
    We want the database to be created from scratch before each test and dropped
    after each test (thus making them unit tests).
    """
    def setUp(self):
        self.tearDown()
        engine = config['pylons.g'].sa_engine
        model.metadata.create_all(bind=engine)
        
        page = model.Page()
        page.title = 'FrontPage'
        page.content = 'Welcome to the QuickWiki front page'
        model.Session.save(page)
        model.Session.commit()



    def tearDown(self):

        engine = config['pylons.g'].sa_engine
        model.metadata.drop_all(bind=engine)


class TestController(TestModel):

    def __init__(self, *args, **kwargs):
        wsgiapp = loadapp('config:test.ini', relative_to=conf_dir)
        self.app = paste.fixture.TestApp(wsgiapp)
        TestCase.__init__(self, *args, **kwargs)

Well, that almost worked. I fell foul of the 'setup-app' command and 'setup-config' in websetup.py. As you can see, the 'tests/__init__.py' file loads and executes the paster 'setup-app' command. Stands to reason: the app should be 'set up' before I run tests.

My setup-app is responsible for creating the DB and populating it with some initial data. But now I can't repeat my tests, because the first time I run the tests, the db is created and initial data put in, and the next time I run the tests... poof:


IntegrityError: (IntegrityError) column title is not unique
u'INSERT INTO pages (title, content) VALUES (?, ?)' ['FrontPage',
'Welcome to the QuickWiki front page']

Well, I thought, I would just drop everything in the db in the tearDown and ensure that tearDown is called before setUp is run. No go. For some reason, it seems, the tearDown() is not working.

Well, it seems I must have at least one test, for the fixtures to be run. So I created a dummy test, and all was well.

Pylons, Paste, Nose and ORMs

2008-03-31T01:17:00.000-07:00

Trying to do some unit tests in Pylons. Pylons uses nose. However, the Pylons Unit Testing guide is a little short on describing how to setup and teardown the database before each test. Here is all the docco to hand:

Pylons Unit Testing Guide

Testing Applications with Paste: more focused on the web part of the web application testiing

Unit Testing with Pylons + SQLAlchemy: explains the use of test setups and teardowns to bring the test DB up and down

2008-03-29T13:20:00.000-07:00

The Open Handset Alliance, led by Google, is developing and releasing the Android mobile OS.

But Android is not the only open mobile OS in town. Another noteworthy player is the LiMo foundation. Its mission is to produce an open, Linux based software platform for mobile devices. LiMo has been around longer and there are already LiMo devices in the market.

There are other efforts towards putting Linux on mobile devices:

Motorola has a Linux-Java based platform called MOTOMAGX. It is not an open polatform, AFAIK. Also, Motorola's mobile division's future is uncertain rigt now.

Openmoko is trying to create both a software platform and hardware; they release the design to the hardware as well.

Ubuntu Mobile

Nokia recently joined LiMo. Does this mean a Googls vs Nokia fight in the mobile OS arean? Not to mention, Symbian OS and Windows Mobile are the current dominant players and already have application developer mindshare.

Typing accented/european characters in Emacs

2008-03-16T22:50:00.000-07:00

Normally I never have to type anything but ASCII in Emacs. But I have been doing some internationalization testing/development, and now and then need to type some non-ASCII characters. Right now, I am happy to be able to type accented characters, though the ultimate goal is to be able to type in a wide variety of languages.

To get emacs to display most languages and scripts:

I ensure than my LANG environment is UTF-8
I install xfonts-intl-* and emacs-intl-fonts packages
I do a `C-h h' to see the multilingual hello file.... the languages of my interest should be present there

To type `interesting' characters in Emacs, I have found that this works:

Change input method to RFC1345 by:


C-x RET C-\ RET (select default input method RFC1345)

Type an accented chanracter using its RFC1345 mnemonic. E.g., to type the inverted exclamation mark used in Spanish I would type "&!I" (makes sense: '&' introduces the special character, ! is the excalamation mark, and I inverts it).

To be able to quickly lookup the RFC, I installed the `doc-rfc' meta-package. Now, to read RFC3412, for example, I do:


 C-x C-f /usr/share/doc/RFC/links/rfc3412.txt.gz

and since I have installed the `emacs-goodies-el' package, which contains `rfcview', the RFC shows up nicely formatted in Emacs.

Ticking noise from HDD

2008-03-15T10:06:00.000-07:00

Often, my laptop's HDD makes a chirp/ticking sound. Tracked it down to this issue and this fix.

In short, the issue is that the HDD's firmware is set to save power by parking its head; but the OS is accessing the HDD quite frequently. As a result, the HDD aggressively parks its head, and then has to unpark it because the OS does a read/write. Turns out this parking/unparking could wear out the driver faster than if there was no power management.

Additionally, this happens even when the laptop is on AC... where there is no power saving advantage.

The fix linked above simply turns off the HDD's power management.

How to choose clothes to wear

2008-03-01T20:59:00.000-08:00

Choose according to your station
- Choose according to what looks good on you
  - Exclude what you wore yesterday
    - Out of the rest, choose what you like

My First Emacs Lisp Function

2008-01-25T09:52:00.000-08:00


(defun how-many-chars ()
  (interactive)
  (message "We are %d chracters into this buffer."
           (- (point)
              (save-excursion
                (goto-char (point-min))
                (point)))))

Transporting and putting ssh public keys on remote servers the easy way

2008-01-09T00:13:00.000-08:00

I login to a lot of Linux boxes, and I use ssh public keys to login. When I create a new account/server, I have to:

copy over my ssh public key to the remote host
login to the remote host
put the public key in the authorized_keys file, set the permissions etc.
delete the copy of the public key

I figured there ought to be an easier way to do this, and here it is:

cat .ssh/id_rsa.pub | ssh parijat@192.168.1.97 "mkdir -p ~/.ssh ; cat - >> ~/.ssh/authorized_keys ; chmod -R go-rwx ~/.ssh"

Reactor vs Proactor

2008-01-08T03:39:00.000-08:00

I found a comparison of the Reactor and the Proactor pattern here. Both patterns talk about isses that crop up when building a concurrent network server. Both are related alternatives to thread based concurrency (or could work as a complement to thread based concurrency).
Both revolve around the concept of an IO De-multiplexer, event sources and event handlers. The driver program registers some event sources (e.g., sockets) with an IO de-multiplexer (e.g., select() or poll()). When an event occurs on a socket, a corresponding event handler is called. Of course, there must be some map between events from an event source to event handlers.
I found that these patterns are more or less embodied in the Python asyncore and asynchat modules, and want to discuss how the modules implement these patterns.

The Basics

We'll first compare the terminology of the patterns with that of the Python modules.

blocking IO: this would translate to a read()/write() on a blocking socket. The call would block until there was some data available to read or the socket was closed. The thread making the call cannot do anything else.
non-blocking, synchronous IO: this would translate to a read()/write() on a non-blocking socket. The call would return immediately, either with the data read/written, or with a signal that the IO operation could not complete (e.g., read() returns with -1, and errno set to EWOULBLOCK/EAGAIN. It is then the caller's responsibility to keep calling repeatedly until the operation succeeds.
non-blocking, asynchronous IO: this would translate to Unix SIGIO mechanisms (unfortunately, I am not familiar with this), or posix aio_* functions (not familiar with these either). Essentially, these IO calls return immediately, and the OS starts doing the operation in a separate (kernel level) thread; when the operation is ready, the user code is given some notification.

The Reactor Pattern: asyncore

According to the authors, here is how the Reactor pattern, which usually would use non-blocking synchronous IO, would work:

Here's a read in Reactor:
An event handler declares interest in I/O events that indicate readiness for read on a particular socket
The event de-multiplexer waits for events
An event comes in and wakes-up the demultiplexor, and the demultiplexor calls the appropriate handler
The event handler performs the actual read operation, handles the data read, declares renewed interest in I/O events, and returns control to the dispatcher

How does this work in Python? Its done using the asyncore module.

The IO demux is the asyncore.loop() function; it listens for events on sockets using either the select() or poll() OS call. It uses a global or user supplied dictionary to map sockets to event handlers (see below). Event handlers are instances of asyncore.dispatcher (or its subclasses). A dispatcher contains a socket and registers itself in the global map, letting loop() know that its methods should be called in response to events on its sockets. It also, through its readable() and writable() methods, lets loop() know what events it is interested in handling.
loop() uses select() or poll() to wait for events on the sockets it knows about.
select()/poll() returns; loop() goes through each socket that has an event, find the corresponding dispatcher object, determines the type of event, and calls a method corresponding to the event on the dispatcher object. In fact, loop() translates raw readable/writable events on sockets to slightly higher-level events using state information about the socket.
The dispatcher object's method is supposed to perform the actual IO: for example, in handle_read() we would read() the data off the socket and process it. Control then returns to loop(). Of course, one problem is that we should not do lengthy tasks in our handler, because then our server would not behave very concurrently and be unable to process other events in time. But what if we did need to do time-taking tasks in response to the event? Thats a subject for another post. For now we assume that our handlers can return quickly enough that as a whole the server behaves pretty concurrently.

The Proactor pattern: a psuedo-implementation in asynchat

According to the authors, here is how the Proactor pattern, which would usually use true asynchronous IO operations provided by the OS, would work:

Here is a read operation in Proactor (true async):
A handler initiates an asynchronous read operation (note: the OS must support asynchronous I/O). In this case, the handler does not care about I/O readiness events, but instead registers interest in receiving completion events.
The event demultiplexor waits until the operation is completed
While the event demultiplexor waits, the OS executes the read operation in a parallel kernel thread, puts data into a user-defined buffer, and notifies the event demultiplexor that the read is complete
The event demultiplexor calls the appropriate handler;
The event handler handles the data from user defined buffer, starts a new asynchronous operation, and returns control to the event demultiplexor.

How does this work in Python? Using the asynchat module.

Event handlers are instances of asynchat.async_chat (or rather, its subclasses). Taking read as an example, the handler would register interest in reading data by providing a readable() method that returns True.
loop() would then use it to wait on its socket until the socket was readable. When the socket become readable, instead of calling some OS function to read the data, async_chat.handle_read() is called.
This method will slurp up all available data.
Then, handle_read() would call the collect_incoming_data() method of the subclass. From the subclass's point of view, someone else has done the job of doing the actual IO, and it is being signaled that the IO operation is complete.
collect_incoming_data() processes the data, and by returning, implicitly starts a new async IO cycle.

The similarity between asynchat and Proactor is that from the application writer's point of view, he only has to write code to collect_incoming_data(). The difference is that, with asynchat, user level code is doing the IO, instead of true async facilities provided by the OS. The difference is greater when considering write operations. In a true Proactor, the event handler would initiate the write, and the event demultiplexer would wait for the completion event. However, in asynchat, the event handler (the subclass of async_chat) does not initiate the write per-se: it creates the data and pushes it onto a fifo, and loop(), indirectly through async_chat, writes it to the socket using synchronous non-blocking IO.

A Unified API

Basically, Python's asynchat is providing an emulated Proactor interface to application writers. It would be good if asynchat could be redone so that it could use true async IO operations on OSes that support them, and fall back to synchronous IO when it is not available.

Python's asynchat module

2008-01-06T01:30:00.000-08:00

Introduction

As mentioned in the previous post, I was going to look at how to write a network server using Python's asynchat module. To utilize the asynchat module's capabilities, I had to change the semantics of the echo server a little bit. The echo server using asyncore would echo back the data as soon as it got it. The echo server using asynchat will echo data back line by line, where each line should be terminated by the string "\r\n".

The async_chat interface for server writers

asynchat provides a higher level interface than asyncore. In this interface, to write a server, you subclass asynchat.async_chat, and override two methods:

collect_incoming_data(data): Unlinks asyncore, you don't have to bother with handle_read() event. The framework will read the data for you and call this method with the data. You probably want to save this data somewhere, in preparation for processing it later
found_terminator(): The framework calls this method when it detects that a 'terminator' has been found in the incoming data stream. The framework decides this based on information you give to the framework using the set_terminator() method.

Getting started

So how do you use this module to write a server? Just as with asyncore, you write a server class and instantiate and object; this object's socket is the server socket; you handle the event handle_accept() and create objects of class async_chat (or, rather, a subclass of async_chat that you created) to handle the client connection. The only difference between asyncore and asynchat is, so far, the object that you instantiate to handle the client connection.

Let's get started. First we look at the driver code:

server_main.py:


    import asynchat_echo_server
    ...
    server = module.EchoServer((interface, port))
    server.serve_forever()

The server class

Our 'EchoServer' class looks pretty much like before:

asynchat_echo_server.py


class EchoServer(asyncore.dispatcher):

    allow_reuse_address         = False
    request_queue_size          = 5
    address_family              = socket.AF_INET
    socket_type                 = socket.SOCK_STREAM


    def __init__(self, address, handlerClass=EchoHandler):
        self.address            = address
        self.handlerClass       = handlerClass

        asyncore.dispatcher.__init__(self)
        self.create_socket(self.address_family,
                               self.socket_type)

        if self.allow_reuse_address:
            self.set_resue_addr()

        self.server_bind()
        self.server_activate()


    def server_bind(self):
        self.bind(self.address)
        log.debug("bind: address=%s:%s" % (self.address[0], self.address[1]))


    def server_activate(self):
        self.listen(self.request_queue_size)
        log.debug("listen: backlog=%d" % self.request_queue_size)


    def fileno(self):
        return self.socket.fileno()


    def serve_forever(self):
        asyncore.loop()
    # TODO: try to implement handle_request()

    # Internal use
    def handle_accept(self):
        (conn_sock, client_address) = self.accept()
        if self.verify_request(conn_sock, client_address):
            self.process_request(conn_sock, client_address)


    def verify_request(self, conn_sock, client_address):
        return True


    def process_request(self, conn_sock, client_address):
        log.info("conn_made: client_address=%s:%s" % \
                     (client_address[0],
                      client_address[1]))
        self.handlerClass(conn_sock, client_address, self)


    def handle_close(self):
        self.close()

The difference is in the handlerClass, which is defined to be EchoHandler as before, but is coded differently. When we instantiate this object, it gets added to the global map of sockets that loop() is monitoring, and now loop() will monitor events on the client socket as well as the server socket. There can be any number of sockets. This behaviour is the same as that of asyncore.

handling per-client connections

Here is how we start our new EchoHandler:


class EchoHandler(asynchat.async_chat):

    LINE_TERMINATOR     = "\r\n"

    def __init__(self, conn_sock, client_address, server):
        asynchat.async_chat.__init__(self, conn_sock)
        self.server             = server
        self.client_address     = client_address
        self.ibuffer            = []

        self.set_terminator(self.LINE_TERMINATOR)

As can be seen, the init method calls async_chat.set_terminator() method with a string argument. The string argument tells async_chat that a message or record is terminated when it encounters the string in the data. Now, loop() will wait on this client socket and call async_chat's handle_read() method. async_chat's handle_read() will read the data, look at it, and call the collect_incoming_data() method that you define:


    def collect_incoming_data(self, data):
        log.debug("collect_incoming_data: [%s]" % data)
        self.ibuffer.append(data)

As you can see, we just buffer the data here for later processing.

Now, in the handle_read() method, async_chat will look for the string set by set_terminator(). If it finds it, then it will call the found_terminator() method that we define:


    def found_terminator(self):
        log.debug("found_terminator")
        self.send_data()

When we find that we have a complete line (because it was terminated by "\r\n") we just send the data back. After all, we are writing an echo server.

Sending data

Sending data back to peers is a common task. Using asyncore, we would create the data to be sent back and put it in a buffer. Then we'd wait for handle_write() events, writing as much data from the buffer to the socket as possible in each event.

asynchat makes this easier. We create the data, put it in a so called 'producer' object, and push the producer object to a FIFO. async_chat will then call each producer in turn, get data from it, send it out over the socket, piece by piece, until the producer is exhausted; it will then move on to the next producer.

If it encounters a None object in place of a producer, async_chat will close the connection.

All this can be accomplished with:


    def send_data(self):
        data = "".join(self.ibuffer)
        log.debug("sending: [%s]" % data)
        self.push(data+self.LINE_TERMINATOR)
        self.ibuffer = []

As you can see, putting the data in a producer object and pushing it on to the FIFO takes just one line of code: self.push(...). We dont have to define a producer class in the normal case because async_chat provides a simple_producer class for us, and the push() method creates an object of that class, populates it with whatever we supply, and then pushes it on to the FIFO. This behaviour can be over-ridden, using the async_chat module API, but we will look at that in another installment.

We have not bothered to push a None onto the FIFO, because we depend on the client closing the connection. We might have put a timer and when the timer expired, close the connection ourselves, to handle clients that go away without properly closing the connection.

Here is the full code:


import logging
import asyncore
import asynchat
import socket

logging.basicConfig(level=logging.DEBUG, format="%(created)-15s %(levelname)8s %(thread)d %(name)s %(message)s")
log                     = logging.getLogger(__name__)

BACKLOG                 = 5
SIZE                    = 1024

class EchoHandler(asynchat.async_chat):

    LINE_TERMINATOR     = "\r\n"

    def __init__(self, conn_sock, client_address, server):
        asynchat.async_chat.__init__(self, conn_sock)
        self.server             = server
        self.client_address     = client_address
        self.ibuffer            = []

        self.set_terminator(self.LINE_TERMINATOR)


    def collect_incoming_data(self, data):
        log.debug("collect_incoming_data: [%s]" % data)
        self.ibuffer.append(data)


    def found_terminator(self):
        log.debug("found_terminator")
        self.send_data()


    def send_data(self):
        data = "".join(self.ibuffer)
        log.debug("sending: [%s]" % data)
        self.push(data+self.LINE_TERMINATOR)
        self.ibuffer = []


    def handle_close(self):
        log.info("conn_closed: client_address=%s:%s" % \
                     (self.client_address[0],
                      self.client_address[1]))

        asynchat.async_chat.handle_close(self)

class EchoServer(asyncore.dispatcher):

    allow_reuse_address         = False
    request_queue_size          = 5
    address_family              = socket.AF_INET
    socket_type                 = socket.SOCK_STREAM


    def __init__(self, address, handlerClass=EchoHandler):
        self.address            = address
        self.handlerClass       = handlerClass

        asyncore.dispatcher.__init__(self)
        self.create_socket(self.address_family,
                               self.socket_type)

        if self.allow_reuse_address:
            self.set_resue_addr()

        self.server_bind()
        self.server_activate()


    def server_bind(self):
        self.bind(self.address)
        log.debug("bind: address=%s:%s" % (self.address[0], self.address[1]))


    def server_activate(self):
        self.listen(self.request_queue_size)
        log.debug("listen: backlog=%d" % self.request_queue_size)


    def fileno(self):
        return self.socket.fileno()


    def serve_forever(self):
        asyncore.loop()
    # TODO: try to implement handle_request()

    # Internal use
    def handle_accept(self):
        (conn_sock, client_address) = self.accept()
        if self.verify_request(conn_sock, client_address):
            self.process_request(conn_sock, client_address)


    def verify_request(self, conn_sock, client_address):
        return True


    def process_request(self, conn_sock, client_address):
        log.info("conn_made: client_address=%s:%s" % \
                     (client_address[0],
                      client_address[1]))
        self.handlerClass(conn_sock, client_address, self)


    def handle_close(self):
        self.close()

Writing a server with Python's asyncore module

2008-01-03T19:34:00.001-08:00

The Python asyncore and aynchat modules

The Python standard library provides two modules---asyncore and
asynchat---to help in writing concurrent network servers using
event-based designs. The documentation does not give good examples,
so I am making some notes.

Overview

The basic idea behind the asyncore module is that:

there is a function, asyncore.loop() that does select() on a bunch of 'channels'. Channels are thin wrappers around sockets.
when select reports an event on any socket, loop() examines the event and the socket's state to create a higher level event;
it then calls a method on the channel corresponding to the higher level event.

asyncore provides a low-level, but flexible API to build network
servers. asynchat builds upon asyncore and provides an API that is
more suitable for request/response type of protocols.

aysncore

The asyncore module's API consists of:

the loop method, to be called by a driver program;
the dispatcher class, to be subclassed to do useful stuff. The dispatcher class is what is called 'channel' elsewhere.


+-------------+           +--------+
| driver code |---------> | loop() |
+-------------+           +--------+
      |                      |
      |                      | loop-dispatcher API (a)
      |                      |
      |                  +--------------+
      |                  | dispatcher   |
      +----------------->| subclass     |
                         +--------------+
                             |
                             | dispatcher-logic API (b)
                             |
                         +--------------+
                         | server logic |
                         +--------------+

This is all packaged nicely in an object oriented way. So, we have
the dispatcher class, that extends/wraps around the socket class (from
the socket module in the Python standard library). It provides all
the socket class' methods, as well as methods to handle the higher
level events. You are supposed to subclass dispatcher and implement
the event handling methods to do something useful.

The loop-dispatcher API

The loop function looks like this:

loop( [timeout[, use_poll[, map[,count]]]])

What is the map? It is a dictionary whose keys are the
file-descriptors, or fds, of the socket (i.e., socket.fileno()), and
whose values are the dispatcher objects.

When we create a dispatcher object, it automatically gets added to a
global list of sockets. The loop() function does a select() on this
list unless we provide an explicit map. (Hmm... we might always want
to use explicit maps; then our loop calls will be thread safe and we
will be able to launch multiple threads, each calling loop on
different maps.)

Methods a dispatcher subclass should implement

loop() needs some methods from the dispatcher object:

readable(): should return True, if you want the fd to be observed for read events;
writable(): should return True, if you want the fd to be observed for write events;

If either read or write is true, the corresponding fd will be examined
for errors also. Obviously, it makes no sense to have a dispatcher
which returns False for both readable() and writable().

handle_read: socket is readable; dispatcher.recv() can be used to actually get the data
handle_write: socket is writable; dispatcher.send(data) can be used to actually send the data
handle_error: socket encountered an error
handle_expt: socket received OOB data (not really used in practice)
handle_close: socket was closed remotely or locally

Server sockets get one more event.

handle_accept: a new incoming connection can be accept()ed. Call the accept() method really accept the connection. To create a server socket, call the bind() and listen() methods on it first.

Client sockets get this event:

handle_connect: connection to remote endpoint has been made. To initiate the connection, first call the connect() method on it.

Other socket methods are available in dispatch: create_socket(),
close(), set_resue_addr().

How to write a server using asyncore

The standard library documentation gives a client example, but not a
server example. Here are some notes on the latter.

Subclass dispatched to create a listening socket
In its handle_accept method, create new dispatchers. They'll get added to the global socket map.

Note: the handlers must not block or take too much time... or the
server won't be concurrent.

These socket-like functions that dispatcher extends should not be bypassed. They do funky things to detect higher level events. For e.g., how does asyncore figure out that the socket is closed? If I remember correctly, there are two ways to detect whether a non-blocking socket is closed:

select() returns a read event, but when you call recv()/read() you get zero bytes;
you call send()/write() and it fails with an error (sending zero bytes is not an error).

(I wish I had a copy of Unix Network Programming by Stevens handy
right now.)

Will look at asynchat in another post.

The code for the server is below:

asyncore_echo_server.py


import logging
import asyncore
import socket

logging.basicConfig(level=logging.DEBUG, format="%(created)-15s %(msecs)d %(levelname)8s %(thread)d %(name)s %(message)s")
log                     = logging.getLogger(__name__)

BACKLOG                 = 5
SIZE                    = 1024

class EchoHandler(asyncore.dispatcher):

    
    def __init__(self, conn_sock, client_address, server):
        self.server             = server
        self.client_address     = client_address
        self.buffer             = ""

        # We dont have anything to write, to start with
        self.is_writable        = False

        # Create ourselves, but with an already provided socket
        asyncore.dispatcher.__init__(self, conn_sock)
        log.debug("created handler; waiting for loop")

    def readable(self):
        return True     # We are always happy to read


    def writable(self):
        return self.is_writable # But we might not have
                                # anything to send all the time


    def handle_read(self):
        log.debug("handle_read")
        data = self.recv(SIZE)
        log.debug("after recv")
        if data:
            log.debug("got data")
            self.buffer += data
            self.is_writable = True  # sth to send back now
        else:
            log.debug("got null data")

    def handle_write(self):
        log.debug("handle_write")
        if self.buffer:
            sent = self.send(self.buffer)
            log.debug("sent data")
            self.buffer = self.buffer[sent:]
        else:
            log.debug("nothing to send")
        if len(self.buffer) == 0:
            self.is_writable = False


    # Will this ever get called?  Does loop() call
    # handle_close() if we called close, to start with?
    def handle_close(self):
        log.debug("handle_close")
        log.info("conn_closed: client_address=%s:%s" % \
                     (self.client_address[0],
                      self.client_address[1]))
        self.close()
        #pass

            
class EchoServer(asyncore.dispatcher):

    allow_reuse_address         = False
    request_queue_size          = 5
    address_family              = socket.AF_INET
    socket_type                 = socket.SOCK_STREAM


    def __init__(self, address, handlerClass=EchoHandler):
        self.address            = address
        self.handlerClass       = handlerClass

        asyncore.dispatcher.__init__(self)
        self.create_socket(self.address_family,
                               self.socket_type)

        if self.allow_reuse_address:
            self.set_resue_addr()

        self.server_bind()
        self.server_activate()


    def server_bind(self):
        self.bind(self.address)
        log.debug("bind: address=%s:%s" % (self.address[0], self.address[1]))


    def server_activate(self):
        self.listen(self.request_queue_size)
        log.debug("listen: backlog=%d" % self.request_queue_size)


    def fileno(self):
        return self.socket.fileno()


    def serve_forever(self):
        asyncore.loop()


    # TODO: try to implement handle_request()

    # Internal use
    def handle_accept(self):
        (conn_sock, client_address) = self.accept()
        if self.verify_request(conn_sock, client_address):
            self.process_request(conn_sock, client_address)


    def verify_request(self, conn_sock, client_address):
        return True


    def process_request(self, conn_sock, client_address):
        log.info("conn_made: client_address=%s:%s" % \
                     (client_address[0],
                      client_address[1]))
        self.handlerClass(conn_sock, client_address, self)


    def handle_close(self):
        self.close()

and to use it:


    server = asyncore_echo_server.EchoServer((interface, port))
    server.serve_forever()

Convoluted C++

2007-12-09T07:49:00.000-08:00

When reading Bruce Eckel's TICCP volume 1 and doing an exercise on pointers-to-member syntax, I was reminded of one thing I like about Lisp and even more about Scheme: simple syntax. In comparison, C++ is so awful.

For e.g., Ex 25 in Ch 11 of Volume 1 asks us:

Create a class containing an array of int. Can you index through
this array using a pointer to member?

Well, here is the class:


class Foo {
public:
     int a[10];
     Foo() {
   for (int i = 0; i < 10; i++)
        a[i] = i;
     }
};

Now, how do I declare the pointer-to-member that will refer to a?

After several tries, I got it:
int (Foo::*mem)[10];, which I use like this:
mem = &Foo::a;

I'd tried:


int Foo::*mem;
...
mem = &Foo::a;

and


int Foo::*mem;
...
mem = &Foo::a[0];

and


int Foo::*mem[];
...
mem = &Foo::a[0];

(this crashed the compiler, GCC 3.4.4 on Cygwin)

and


int Foo::*mem[10];
...
mem = &Foo::a;

The point is that special syntax to do everything is so icky once you have tasted Scheme. I'd say half or more of Bruce's book is about the syntax to get to the various features of the language. You are focusing so much on learning syntax that you don't have any time to figure out how you'd use those features for real work. And with the convoluted syntax in your way, you are not likely to be using the appropriate feature where it would be really handy.

Nevertheless, TICPP does and excellent job of just that: describing the language's features and trying to give a scenario where that feature would be useful. Too bad, knowing the features requires memorizing so much syntax. In C, to get to a "feature", I'd just look at the man page for the relevant function. There are no man pages for syntax, unfortunately.

Convoluted C++

2007-12-09T07:42:00.000-08:00

Been going through Bruce Eckel's "Thinking in C++" books. I've not programmed in C++ for a while, as I mostly program in Python (and am learning Ruby, Lisp, Erlang, and stuff). However, I felt it would be good to sharpen up my C++ and Java chops.

So, I'd learnt C++ from Bjarne Stroustroup's book a long time ago. I thought I'd covered pretty much the whole book, except multiple inheritance. Alas, I'd learnt little. Well, enough to write a Python extension or two.

Bruce does an excellent job of explaining the language if you are a programmer that already knows C. Otherwise you should go and get familiar with C first. Recommened reading.

Design Patterns in Python: II

2007-09-15T08:31:00.000-07:00

As mentioned before, I am going through Part II of a talk on DPs in Python. The first part of the talk focuses on the Template Method design pattern. I have seen this pattern in the form of "algorithm" pattern in C++ template library and elsewhere (SAX parser library, various formatter libraries), but the video talk explains this very well. Alex Martelli offers "Self-Delgation" as an alternative name for this pattern. I suppose Algorithm could be another name, at least for some situations.

Well, we all know that the TM pattern concerns itself with separating the generic part of an algorithm or process into an "organizing method", which calls "hooks" at appropriate points to execute certain steps of the algorithm/process. So far, so good. What is revealing is that this separation can be done in three ways:

Inheritance TM

The organizing method is defined in an ABC. Sub-classes provide concrete implementations of hooks. Obvious variations: the parent class provides no-op, or reasonable default implementations of hooks. E.g., Queue.Queue follows the TM pattern and provides concrete hook implementations, which provide a full-implementation of a FIFO queue. However, because of the TM pattern, it can be sub-classed and the hooks overridden to make a LIFO queue, for e.g.

Factored TM

In this technique, the organizing method is in one class, and the hooks are implemented in another. The hook class can be passed to the class implementing the organizing method as an argument at instantiation time.

Further, hooks can be grouped. There can be a class per group.

Even further, each hook can be a class by itself. Then we get the Strategy pattern.

Mixin TM

This is applicable for languages allowing multiple inheritance. The idea is:

You multiply-inherit from a Mixin class that provides the "organizing methods"
You implement hooks to obtain the desired functionality.

An e.g.: you could derive your class from DictMixin, which causes your object to have the dictionary interface. All you have to do is implement a few hook methods, and automatically the rest of the rich dictionary interface (which is based on top of these hooks) is part of your class!

Combined Inheritance, Factored, and Mixin TM
One interesting use of these three techniques is in the SocketServer module:

TCPServer is a concrete implementations of network servers. They use factored TM: the TCP/UDPServer implements the organizing method, and a class derived from BaseRequestHandler provides the concrete hooks. Well, BaseRequestHandler provides some no-op hook implementations, that serve as documentation.
You can inherit from BaseRequestHandler and override some methods to specialize its behaviour. This is inheritance TM.
You can inherit from TCPServer and ThreadinMixin. ThreadingMixin overrides the TM of TCPServer and provides the multi-threaded TM. You just provide the hook implementation (the request handler) to have a multi-threaded tcp server.

The Strategy and State Patterns

The main idea in the Strategy Pattern seems to be: say you have a complicated process (the organizing method or OM) with many decision points. A decision point is impemented by an object which is invoked for action. That is, in TM we would have self.do_somthing(), but in Strategy we would have self.strat.do_something(). The strat is a reference to an object of some SC of a Strategy class that defines do_something(). To change the behaviour, we change strat from one SC instance to another. That is, we would do something like: self.strat = Strat1(), or self.strat = Strat2() depending on the situation. Obviously, Strat1 and Strat2 don't have to know much about each other for this to be scalable/appropriate.

In the State pattern, it is the state that knows how to switch behaviour. So we would have a decision point impemented as self.state.do_something(), and we would change behaviour by self.state.set_state(foo) or self.state.set_state(bar).

In Python, we can blur the lines:

1. Traditional

class Foo:
def bark():
print "bow"
class Bar:
def bark():
print "wow"
class OM:
def __init__():
self.strat = Foo()
def work():
print "working"
self.strat.bark()
print "work done"
def change_voice():
self.strat = Bar()

2. Change method

class OM:
def __init__():
def bark():
print "bow"
def bark2():
print "wow"
def work():
print "working"
self.bark()
print "work done"
def change_voice():
self.bark = self.bark2

3. Change __class__

class OM:
def work():
print "working"
self.bark()
print "work done"
def bark():
print "bow"
def change_voice():
self.__class__ = Cat

class Cat(OM):
def bark():
print "meow"

Design Patterns in Python: I

2007-09-15T05:50:00.000-07:00

I found some time to catch up on general tech reading. Or rather, viewing. Because I have been viewing some excellent Google EngEDU videos.

I just watched "Advanced Topics in Programming Languages Series: Python Design Patterns (Part 1)". The presenter, Alex Martelli, emphasized that Design Patterns can be useful, useless or dangerous/overkill depending on the particular language of choice. In other words, as the A good example would be Singleton, which is probably conceptually broken. Python provides modules instead that work handily. I found myself remembering a particular use of Singleton in my own Python code that is, well, icky and reeks of uselessness. Anyway, Alex covered Creational Patterns in brief, and then went on to talk about Structural Patterns. They were more interesting and Alex gave some examples from the python library modules.

Inheritance vs Composition

Alex mentioned that object composition is favored over inheritance for most cases. By composing or "wrapping" we can do things that inheritance cannot, like selectively modifying the interface to an object, restricting the interface provided. (Inheritance cannot restrict the interface to the object).

Hold vs Wrap

So we go with composition. Composing objects leads to the choice of the "Hold vs Wrap" choice . I have faced this choice myself and I have seen the irritating consequences of the wrong choice.

Say I have to compose objects O with subobject S. Hold: object O has a reference to S, nothing more. Then I would access S as O.S.some_method(). Wrap: hold (maybe via a private name), and provide delegation, so that you can say O.some_method().

The latter choice brings us into the realm of the Structural Design Patterns. At this point I would say that it becomes quite irritating to see code like this: foo.bar.some_method(). Basically O is exposing its internal structure to its clients and violating the "Law of Demeter":http://en.wikipedia.org/wiki/Law_of_Demeter.

Creational Patterns

Alex discussed the Singleton pattern and how it has fundamentally a problem with inheritance. He clarified that instead of a Singleton, what is usually required is a shared-state. He introduced the Borg/Monostate pattern that solves the same problem with more control possible.

The next few patterns were the Factory series of patterns. Luckily, in Python, any callable can be passed into client code and can behave as an object factory or factory method. In fact, each class is a kind of factory, which can be customized to your heart's content by implementing/overriding the __new__() method.

Structural Patterns

The next few patterns to be discussed were:

Adapter: you have code expecting interface C, supplier code providing interface S (a superset of C), you write an adapter from C to S;
Facade: you have one or more objects; you create a single interface concentrating and simplifying the needed functionality for clients; a discussion of Facade was mentioned: http://www.tonymarston.net/php-mysql/design-patterns.html (I have not looked at it yet)
Bridge: you have several implementations of abstraction A, all expecting functionality F, and you have (or want to have) several implementations providing F, and you dont want the client code to know of/depend on the F implementation. To do this, make sure that each implementation of A holds a reference R to F, and accesses the functionality F only by delegating to R. A good example is the SocketServer module, where BaseServer is the abstraction A, BaseRequestHandler is the functionality F. The appropriate implementation of BaseRequestHandler is passed at creation time to the appropriate sub-class of BaseServer, which then keeps a reference to it (the R). When a new request handler is required, R() is called to create a new object. Alex mentions that this is rather like a factory (but then, callables are factories), and that it is typical of Bridge DPs in Python to keep reference to a class and instantiate from it later. I wonder what a bridge to do a ORM would look like. I am now a little bit more confident of digging into the guts of SQLAlchemy.
Decorator: reuse/tweak without inheritance. Difference from Adapter/Facade would be that the interface does not change, but new functionality is somehow added. A buffering adapter to a file object would be an example: the starting and resulting interface is the same, and the difference is only whether data is written to disk with or without buffering. The gzip module is an example.
Proxy: a proxy looks like a decorator but deals with lifetime, access, location issues rather than adding functionality. Hmm.. well, CORBA comes to mind.

I am now going to watch Part II of the same series, which should deal with Behavioral Patterns.

The slides used for the talk are available at http://www.aleax.it/goo_pydp.pdf (8 MB PDF file).

What does Orkut have to do with crimes?

2007-08-21T09:21:00.001-07:00

When I came home from work, I found Star News Asia reporting on how Orkut is a channel for depravity and crime. There were blurbs like "Bach ke rehna" (Hindi: save yourself, or stay away), "Maut ki website" (Website of death), "Kishoron ko kiya ja raha hai gumraha" (Youngsters are being misled), "Orkut par ankush jaroori?" (Should Orkut be censored?).

It started with a report of a young girl murdered by a "friend" she met on Orkut. Google news shows similar cases being covered in various media. Too many to list. But what was strange about this news story was the exclusive focus on Orkut and blaming it for:

A means of disseminating information to its users about rave parties (where all-night dancing and drugs are mixed together);
Misleading youngsters: while Orkut started as a means of connecting people, it has (according to the news story) become a hive of people talking about sex, rave parties, drugs, etc.

I find it strange to see Orkut being targeted so exclusively and being portrayed as promoting these activities. Several objections come to mind:

So this girl met this guy online, befriended him, and met him in a hotel room alone; the guy turned out to be not that friendly and caring and murdered the girl. Agreed, one knows less about people you meet online. That requires some care and judgment on behalf of the user, no?
This meeting could have happened off line. I know of people being harmed by friendly looking acquaintances. What about that?
Is Orkut a motivator for rave parties, or are people wishing to go to rave parties using Orkut as a medium?
Why Orkut, and not other social networking sites? Because it is more popular in India? If Orkut were not popular, would people have not used any other social networking platform? If none were available, would they not use email lists, IRC channels, BBSes? Shall we ban them?
Agreed, all kinds of users interact online and some are not so moral. Is it Orkut's responsibility for what people post and tell each other? Is it a parent/guardian's responsibility or Orkut's that young children are not exposed to adult content?
The internet is a dangerous place. Many sites and forums contain adult content. Places other than Orkut are used to organize parties, some of them "not so nice". Shall we block the internet?

The news story seems to me to be more an attempt to sensationalize and demonize and less to point to a problem, inform people, or offer a balanced view.

Here is where we could go from here:

Let's ban Orkut. Users will go to MySpace. Lets ban that, next. Then Facebook. Then ...
Finally we will have stomped out the evil social nets. Next target: BBSes and forums. They make it too easy for people to reach each other.
Ergh... we forgot about personal web-pages. No more personal websites. All media to be controlled and all publications to be approved by Almighty Censorship Alliance of Sensational News Channels.
Oops, we forgot about email. Bye bye, you nasty little disseminator of information about parties, drugs and personal details.
Now we plug the network cable out of the PC.
Ban books: too many porno flimsies around.
No more phone: you better come home and get interviewed by the family before you meet my daughter.
No talking to strangers on the roadside.
No roads.

Funny, I don't see anything about responsible and judicious use of communication anywhere in the list. I guess only news channels know how to do that. Ornery folks like us should just ban everything.

Erlang bit syntax wrinkles

2007-08-19T05:16:00.000-07:00

The Erlang bit syntax is wonderful, if complicated.

Say you have a Binary like:

84> Bin1 = <<1,0,2>>.
<<1,0,2>>

Then, this will match:

85> <<A:1/binary, B:8, C:1/binary>> = Bin1.
<<1,0,2>> %% A => 1, C => 2

But this will not match:

86> <<A:1/binary, B:1/binary, C:1/binary>> = Bin1.

=ERROR REPORT==== 19-Aug-2007::17:46:17 ===
Error in process <0.139.0> with exit value: {{badmatch,<<3>>},[{erl_eval,expr,3}]}

** exited: {{badmatch,<<1,0,2>>,[{erl_eval,expr,3}]} **

It seems that I can specify a literal '0' as an int type (the default type) but not a binary type (see http://erlang.org/doc/reference_manual/expressions.html#bit_syntax for the gory details). Wonder why?

GNU Emacs, SLIME, Allegro

2007-08-17T10:59:00.000-07:00

For some reason, I decided that I just had to switch to GNU Emacs from XEmacs on my WinXP laptop. So I went and downloaded Emacs 22.1 for Windows and installed it.

Setting it up for Erlang was easy: I added this to C:\Documents and Settings\Parijat\.emacs:


;; Erlang:
;; Add the location of the elisp files to the load-path
(setq load-path (cons  "C:/erl5.5.4/lib/tools-2.5.4/emacs" load-path))
;; set the location of the man page hierarchy
(setq erlang-root-dir "c:/erl5.5.4")
;; add the home of the erlang binaries to the exec-path
(setq exec-path (cons "c:/erl5.5.4/bin" exec-path))
;; load and eval the erlang-start package to set up everything else 
(require 'erlang-start)

Now, to setup Common Lisp. I use Allegro CL 8.0 Free Express Edition. (Yes, one day I'll buy the full version and repay Franz, but only after getting myself Lisp Certified ;-)) I was following the instructions at http://www.franz.com/emacs/slime.lhtml and http://common-lisp.net/project/asdf-install/tutorial/setup.html .

This is what I did:

1. Opened up ACL and loaded asdf into it:


International Allegro CL Free Express Edition
8.0 [Windows] (Aug 17, 2007 22:59)
Copyright (C) 1985-2006, Franz Inc., Oakland, CA, USA.  All Rights Reserved.

This development copy of Allegro CL is licensed to:
   Trial User

CG version 1.81.2.23 / IDE version 1.80.2.21
Loaded options from C:\Documents and Settings\Parijat\My Documents\allegro-prefs.cl.

;; Optimization settings: safety 1, space 1, speed 1, debug 2.
;; For a complete description of all compiler switches given the current optimization settings
;; evaluate (EXPLAIN-COMPILER-SETTINGS).

[changing package from "COMMON-LISP-USER" to "COMMON-GRAPHICS-USER"]
CG-USER(1): (require :asdf)
; Fast loading C:\Program Files\acl80-express\code\ASDF.001
T

2. Then, downloaded asdf-install and unpacked it into "C:\Documents and Settings\Parijat\asdf-install" (the source tarball has several nested directories called asdf-install; the innnermost contains the .asd file, and that's the one I am referring to here).

3. Told asdf about this directory:


CG-USER(4): (pushnew "c:/Documents and Settings/Parijat/asdf-install/" asdf:*central-registry* :test #'equal)

and compiled and loaded asdf-install:


CG-USER(5): (asdf:operate 'asdf:compile-op 'asdf-install)
CG-USER(6): (asdf:operator 'asdf:load-op 'asdf-install)

4. Well, all this stuff was in the end to get SLIME. I went to http://common-lisp.net/project/slime/#downloading and found the link to the latest SLIME. Now, on Windows, it seems asdf-install can't download and install tarballs off net (at least on Allegro CL). Then I downloaded and put the slime tarball in C:\Temp. Then:


(setq asdf-install::*verify-gpg-signatures* nil)
(setq *cygwin-bash-program* "c:/cygwin/bin/bash.exe")

and then:


CG-USER(7): (asdf-install:install "C:/Temp/slime-2.0.tgz")

This did not work, complaining:


Warning: Cannot find tar command "C:\\PROGRA~1\\Cygwin\\bin\\bash.exe".
Error: Unable to extract tarball C:/Temp/slime-2.0.tgz.

It seems my mod to *cygwin-bash-program* did not take effect inside asdf. Wonder why. I even quit Allegro, restarted it, defined *cygwin-bash-program* before compiling and loading asdf-install. No luck. (One thing I did not try was to define asdf-install:*cygwin-bash-program* instead of vanilla *cygwin-bash-program*). So I just went and hardcoded tar-command() in installer.lisp to return "C:\\cygwin\\bin\\bash.exe".

Now the above command works.

5. The Franz documentation instructs me to start "mlisp.exe" or "alise.exe" LISP image (see next step). But my ACL 8.0 express did not come with these images; it only has "allegro-ansi.exe". So, following the README in the acl80-express directory, I create the images, by starting allegro-ansi.exe, and in the "Debug window", typing the following:


CG-USER(1): (build-lisp-image "sys:mlisp.dxl" :case-mode :case-sensitive-lower
                      :include-ide nil :restart-app-function nil
        :restart-init-function nil)
CG-USER(2): (sys:copy-file "sys:allegro-ansi.exe" "sys:mlisp.exe")

Now, I have a mlisp.exe and mlisp.dxl. Sure enough, clicking on mlisp.exe launches a console LISP without the GUI.

6. Now I have to load slime into Emacs. According to the instructions on the Franz webpage, I created a file "C:\Documents and Settings\Parijat\.slime.lisp", with the contents:


(load "C:/Documents and Settings/Parijat/.asdf-install-dir/site/slime-2.0/swank-loader.lisp")

(sys:with-command-line-arguments (("p" :short port :required)
      ("ef" :long ef :required))
  (restvar)
  (swank::create-server :port (parse-integer port :junk-allowed nil)
          :style :spawn
                 :dont-close t
   :coding-system (or ef "latin-1")))

and added this to my .emacs:


;; Allegro LISP + SLIME
(push "c:/Documents and Settings/Parijat/.asdf-install-dir/site/slimw-2.0/" load-path)
(require 'slime)
(slime-setup)

(setq slime-multiprocessing t)
(setq *slime-lisp* "C:/Program Files/acl80-express/mlisp.exe")
(setq *slime-port* 4006)

(defun slime ()
  (print "this is my slime")
  (interactive)
  (shell-command 
   (format "%s +B +cm -L \"c:/Documents and Settings/Parijat/.slime.lisp\" -- -p %s --ef %s &"
    *slime-lisp* *slime-port*
    slime-net-coding-system))
  (delete-other-windows)
  (while (not (ignore-errors (slime-connect "localhost" *slime-port*)))
    (sleep-for 0.2)))

I restart emacs. I say "M-x slime". And wait. The mlisp.exe window shows up, but no connection.

I figure that the problem is in (swank:create-server :port ...) which is supposed to start the swank server, does not work. If I run it in the mlisp.exe window, the command does not return (okay, maybe it is not supposed to). I can't telnet to port 4006 on the localhost, though.

Puff!!! It used to be simpler.

Concurrency: Erlang vs Java

2007-08-07T10:24:00.000-07:00

Its fashionable to extol the high-performance of Erlang concurrency these days. Programming Erlang, Section 8.11, has this problem:

Write a ring benchmark. Create N processes in a ring. Send a message
round the ring M times so that a total of N * M messages get
sent. Time how long this takes for different values of N and M.

Well, I copied off the Java program from here: http://www.sics.se/~joe/ericsson/du98024.html

And then I wrote this Erlang program:

-module(threadperftest).
-compile(export_all).

timeit() ->
 register(mainproc, self()),
 io:format(" Procs,  Mesgs, SpawnTotal,  SpawnProc,   RunTotal,    RunMesg~n", []),
 timeit_aux(1000, 1000, 10000),
 init:stop().

timeit_aux(NProcs, NMsgs, NProcsMax) ->
 if NProcs =<>
     main([NProcs, NMsgs]),
     timeit_aux(NProcs+1000, NMsgs, NProcsMax);
    true -> void
 end.
    
main([N, M]) ->
 {_, _W0} = statistics(wall_clock),
 FirstPid = spawn(fun loop0/0),
 % io:format("First: ~p~n", [FirstPid]),
 LastPid = setup(N-1, FirstPid),
 FirstPid ! LastPid,
 {_, W1} = statistics(wall_clock),
 LastPid ! M,
 receive
 stop ->
     void
 end,
 {_, W2} = statistics(wall_clock),
 io:format("~6B, ~6B, ~10g, ~10g, ~10g, ~10g~n", [N, M, W1*1.0, 1.0*W1/N, W2*1.0, W2*1000.0/(M*N+N)]).

setup(0, PrevPid) ->
 PrevPid;
setup(HowMany, PrevPid) ->
 Pid = spawn(fun() -> loop(PrevPid) end),
 % io:format("~p: Linking to: ~p~n", [Pid, PrevPid]),
 setup(HowMany - 1, Pid).

loop0() ->
 receive
 LinkedPid when is_pid(LinkedPid) ->
     % io:format("First: ~p: Linking to ~p~n", [self(), LinkedPid]),
     loop1(LinkedPid);
 X ->
     erlang:error({unknownMessageError, X, self()})
 end.

loop1(LinkedPid) ->
 receive
 M when is_integer(M), M > 0 ->
     % io:format("~p: Received: ~p~n", [self(), M]),
     LinkedPid ! (M-1),
     loop1(LinkedPid);
 M when is_integer(M), M =:= 0 ->
     % io:format("~p: Received: ~p, Terminating~n", [self(), M]),
     mainproc ! stop,
     done;
 X ->
     erlang:error({unknownMessageError, X, self()})
 end.
  
loop(LinkedPid) ->
 receive
 M when is_integer(M), M > 0 ->
     % io:format("~p: Received: ~p~n", [self(), M]),
     LinkedPid ! M,
     loop(LinkedPid);
 M when is_integer(M), M =:= 0 ->
     % io:format("~p: Received: ~p, Terminating~n", [self(), M]),
     LinkedPid ! M,
     done;
 X ->
     erlang:error({unknownMessageError, X, self()})
 end.

The performance is something like this (all times are in milliseconds, except the last one, which is in microseconds):

 Procs,  Mesgs, SpawnTotal,  SpawnProc,   RunTotal,    RunMesg
 1000,   1000, 0.00000e+0, 0.00000e+0,    266.000,   0.265734
 2000,   1000,    16.0000, 8.00000e-3,    562.000,   0.280719
 3000,   1000, 0.00000e+0, 0.00000e+0,    969.000,   0.322677
 4000,   1000, 0.00000e+0, 0.00000e+0,    1672.00,   0.417582
 5000,   1000,    15.0000, 3.00000e-3,    2469.00,   0.493307
 6000,   1000,    16.0000, 2.66667e-3,    3234.00,   0.538462
 7000,   1000,    16.0000, 2.28571e-3,    4015.00,   0.572998
 8000,   1000,    16.0000, 2.00000e-3,    4750.00,   0.593157
 9000,   1000,    31.0000, 3.44444e-3,    5469.00,   0.607060
10000,   1000,    31.0000, 3.10000e-3,    6375.00,   0.636863

The numbers in the RunMesg column are the time it takes to send one message from one process to another. Why is it increasing? Does the runtime have to do more work to find the target process?

Here is the Java output (program below):

 Procs,  Mesgs, SpawnTotal, SpawnProcs,   RunTotal,   RunMesg
  100,   1000,     16.000,      0.160,  15531.000,     15.531
  200,   1000,     62.000,      0.310,  27047.000,     27.047
  300,   1000,     94.000,      0.313,  37828.000,     37.828
  400,   1000,    125.000,      0.313,  46859.000,     46.859
  500,   1000,    141.000,      0.282,  54578.000,     54.578
  600,   1000,    203.000,      0.338,  60594.000,     60.594
  700,   1000,    234.000,      0.334,  65282.000,     65.282
  800,   1000,    406.000,      0.508,  68156.000,     68.156
  900,   1000,    484.000,      0.538,  69782.000,     69.782
 1000,   1000,    735.000,      0.735,  70015.000,     70.015

The last column is in milliseconds whereas for Erlang it was in microseconds, AND the number of procs is 100-1000, whereas for Erlang they were 1000-10,000.

Programming Erlang exercise problem: trouble in the concurrent universe?

2007-08-06T05:47:00.000-07:00

Going through Programming Erlang, encountered this exercise problem in Section 8.11:

Write a function start(AnAtom, Fun) to register AnAtom as spawn(Fun). Make sure your program works correctly in the case when two parallel processes simultaneously evaluate start/2. In this case, you must guarantee that one of these processes succeeds and the other fails.

Here is my first solution, and it seems to compile and run.

-module(register_function).
-export([start/2]).

start(AnAtom, Fun) ->
  case whereis(AnAtom) of
       undefined -> register(AnAtom, spawn(Fun));
       _ -> error
  end.

Now, is there race condition in the above? I mean, consider Processes A and B both executing start/2:

Process A calls whereis(AnAtom) which evaluates to 'undefined';
Process B calls whereis(AnAtom) which evaluates to 'undefined';
Process A calls register(AnAtom, spawn(Fun)), which succeeds and returns true;
Process B calls register(AnAtom, spawn(Fun)), which fails as AnAtom has been registered

Superficially, we have satisfied the requirements of the problem: Process A succeeded and B failed. But the sinister problem is that spawn(...) has been called by both processes anyway. Thus we have two instances of Fun being executed, with one of them in an anonymous process that I, at least for now, don't know how to access.

Ironically, the problem seems to be because presumably register puts AnAtom in some "global" table. Exactly the kind of concurrency problem that Erlang tries to avoid.

How would this be solved? Here is an idea:

Create a "register_function" server;
start/2 sends a message to this server with AnAtom and Fun as contents
presumably the server processes one message at a time and the code between matching a message and executing some expressions is atomic with respect to other messages to the same server

Puff. You'd think Erlang would come with built in functionality for this. Is this important?

That free lunch that you were talking about....

2007-04-12T10:56:00.001-07:00

Reading on BBC:

From his office in Nevada, entrepreneur Dennis Hope has spawned a multi-million-dollar property business selling plots of lunar real estate at $20 (£10) an acre.

This guy has made $9m till now.

Buyers include Hollywood stars, large corporations - including the Hilton and Marriot hotel chains - and even former US presidents Ronald Reagan and Jimmy Carter. George W Bush is also said to be a stake holder.

What drives me nuts is:

day. He allocates land by simply closing his eyes and pointing to a map of the Moon.

So, uh, who said there was no free lunch? I mean, by the time this guy's claims are challenged in court, he would have enjoyed much of that $9m.

Technorati Tags: greed stupid

Emacs and bitstream fonts

2007-04-10T05:25:00.000-07:00

I use emacs on Ubuntu heavily and naturally want a nice font to use. I chose to use the "bitstream vera sans mono" font. Well, easy does it: Options->Customize Emacs->Specific Face: "default", and setting the value of "Font Family" to "bitstream-bitstream vera sans mono" and value of "Height" to "100" gave me the nice font that I wanted. "Custom" created the settings for me:

(custom-set-faces
;; custom-set-faces was added by Custom -- don't edit or cut/paste it!
;; Your init file should contain only one such instance.
'(default ((t (:inherit nil :stipple nil :background "gray12" :foreground "green" :inverse-video nil :box nil :strike-through nil :overline nil :underline nil :slant normal :weight normal :height 117 :width normal :family "bitstream vera sans mono")))))

I should mention that I am using the "color-scheme" package and using a nice looking color scheme. The color information probably comes from there.

But then I tried to create a new frame (C-x 5 2) and lo! the new frame had my old font. Reading the emacs faq gave the answer: I'd have to set the font again in the variable that determines the options for all new frames. So I set in my "~/.emacs":

(setq default-frame-alist
'((font . "-bitstream-bitstream vera sans mono-medium-r-normal--0-0-0-0-m-0-iso8859-1")))

and new frames would now have the specified font, which happens to be the same as the original. I wonder if it is a coincidence that the size of the font in new frames and initial matches, because I have not specified the height of the font in this variable.

This seems like a ugly way: why specify the font properties in two different places? So I commented out the face created by custom, and oops! new frames would have a background of white. So I had to undo the change and let the fonts be specified in two places. But at least the initial and new frames have the same good looking font.

Ubuntu Edgy and X11 font bogosity

2007-04-10T05:17:00.000-07:00

Using Ubuntu Edgy recently. I'd installed the great bitstream vera fonts and could use them in gnome, but could not see them in X applications, in particular emacs.

Turned out that /etc/X11/xorg.conf had a wrong configuration for fonts. It originally said:

Section "Files"
FontPath "/usr/share/X11/fonts/misc"
FontPath "/usr/share/X11/fonts/cyrillic"
FontPath "/usr/share/X11/fonts/100dpi/:unscaled"
FontPath "/usr/share/X11/fonts/75dpi/:unscaled"
FontPath "/usr/share/X11/fonts.Type1"
FontPath "/usr/share/X11/fonts/100dpi"
FontPath "/usr/share/X11/fonts/75dpi"
FontPath "/usr/share/X11/fonts/misc"
# path to defoma fonts
FontPath "/var/lib/defoma/x-ttcidfont-conf.d/dirs/TrueType"
EndSection

Turns out that the directory "/usr/share/X11/fonts" is wrong and should be "/usr/share/fonts/X11", I changed the section to:

Section "Files"
FontPath "/usr/share/fonts/X11/misc"
FontPath "/usr/share/fonts/X11/cyrillic"
FontPath "/usr/share/fonts/X11/100dpi/:unscaled"
FontPath "/usr/share/fonts/X11/75dpi/:unscaled"
FontPath "/usr/share/fonts/X11/Type1"
FontPath "/usr/share/fonts/X11/100dpi"
FontPath "/usr/share/fonts/X11/75dpi"
FontPath "/usr/share/fonts/X11/misc"
# path to defoma fonts
FontPath "/var/lib/defoma/x-ttcidfont-conf.d/dirs/TrueType"
FontPath "/var/lib/defoma/x-ttcidfont-conf.d/dirs/CID"
EndSection

Then I restarted X. But no go. xlsfonts would still not list the bitstream fonts.
The bitstream fonts were linked to from "/var/lib/defoma/x-ttcidfont-conf.d/dirs/TrueType". I went into that directory, and ran

$ sudo ttmkfdir -o fonts.dir -d .

Then I restarted X again. And yes, this time xlsfonts could see the bitstream fonts. So the problems were the wrong font paths and the fact that the fonts.dir file had not been created in the defoma directory. It seems that this is a new bug in Edgy.

Software is knowledge

2007-03-29T09:44:00.001-07:00

People screaming about software engineering were pulling the wool over our eyes and trying to make a quick buck, all along, as we all suspected when we attended those boring software engineering classes in CS courses.

Robert Glass got it right. Software is NOT an artifact of engineering. Making software is not like making a bridge.

Software is an embodiment of knowledge. Executable knowledge. Knowledge that when pasted over some silicon and stimulated with electricity can cause things to happen. By itself very little beyond flipping voltage levels in areas of said silicon. But give it some tools (like a serial port, cable and lcd screen) and it can communicate. Give it more tools (serial port, cable, robot arm) and it can create physical changes in its environment.

Yeah, you got it. Software is less like a scalpel, more like a doctor.

Why the sudden light?

Well, I was trying to document high-availability solutions with MySQL. When going through the nitty-gritty of MySQL Cluster's limitations (no offense intended: it is probably a good product; I am not dissing the software) it occured to me that they warranted noting down, so that my team would not have to read so much should they deal with MySQL Cluster. I found myself writing a short note with a link to the original page for details, because I felt that I could not tell the whole truth in a shorter way, and copying and pasting would be a waste. So, my documentation is a little more than some summary notes peppered with links to the full documentation.

Anyone reading any code that tried to deal with the limitations above---written in an arguably already readable language---would still scratch their heads at why some weird stuff was being done. There would have to be comments explaining why this was done. The comment could be just a link to the MySQL documentation page. Or, it could have the full explanation, with the MySQL version in which said limitation lurked. Or both. The person reading the code would have to understand the documentation before they could say they understood the code.

The point is that: if there is good documentation about some topic, I can do little than point others to it and say "this is important". The documentation is the communication.

And the more important point is: you cannot claim to have understood a software unless you understand its intention and motivation. And in the above case, the motivation means reading the documentation. Most times, reading documentation means you come back with knowledge. In other words, to understand the software, you must, well, understand. Err.. I mean, you must acquire the knowledge the original developer had when creating the software.

Which brings me to a wicked question: the reason I was writing the original notes about clustering was so that not everyone in my team has to know the nitty-gritty and just know enough to design and write code to be handle clusters if we decided to use clusters. But it looks like no one can actually decide if the code will be right without having read the full documentation anyway.

What about code? That's worse. You now must understand the intention of the code. Then you must understand the motivation behind the code. To do that you must read the documentation.

Software is knowledge.

And my documentation is little more than a reminder note or a collection of related links.

Singapore Entreupreneur Web 2.0 Unconference

2007-02-15T09:19:00.000-08:00

Choon Keat came around at work to remind me of this unconference. I'd wanted to attend and therefore we both left together. CK was a bit surprised at the number of people there---ought to be close to 200---because the last time he had attended it, it was a small group in a "pathetic room with pizza and beer on the side".

What intrigued me about this event was the format. The self-styled unconference claims to have an informal setting where many parallel discussions are going on side by side and where you are either presenting, debating or absorbing. Should you be not doing any of these, you are supposed to use the Law of Two Feet to take yourself elsewhere where you can be useful.

It wasn't quite like that. This is the fourth instalment of the event, and this is the first time the organizers were trying to have three "tracks". IMHO the tracks made it all too formal, because I was scientifically trying to decide how to split my time instead of just popping in at different talks and seeing whether something interesting was going on. Still, not having attended the previous events, I can't compare and this may very well be a good format.

There was lots to hear, see and talk about. The debate was sometimes spirited and people were not afraid to speak. The questions (to presenters from startups) came fast and the presenters were able to roll with the punches.

The first presentation I attended was from a startup called ActiveDeals which puts movies in your MSN with some advertising along with it. It claims its product is interactive TV, about which I could only make out that people could chat about the video they were seeing with their friends and (presumably) on their website. If there is more interactivity the presented did not say. Haven't fully checked it out yet, but this is the kind of startup that depends a lot on execution and the main barrier to entry is the work to get there. Their main points were:

The videos would be short, on-demand, and focus on local stuff; something people would be comfortable watching as a short break at work;
The interactive options would lead to user-generated content: links, reviews, etc., which could then be linked against the videos and become content.
activedeals would be doing the video production, so that's some leg work.
Sponsorship of the video content would pay the bills.
The sponsors would have to be educated to the use the interactivity of the medium as opposed to just sell, sell, sell.

I don't know how far they have come along---there were some mockups in the presentation, and the presenter Todd Murray referred to them as such---but the presenter mentioned they would launch in March.

I attended only 5 minutes of the next presentation, which about a website called Wisheus. Here you can "publish your wishes and your wares". It might attract some teens. Could not see whats new about it. But I've not tried to sign-in, and I did not sit through the whole presentation, so if it has some twist, I don't know about it.

I quit this talk to go to another about complex-systems and networks. The presentation started out with fractals, then on to complex systems with emergent behaviour (you know, the ones where autonomous agents which have simple behavioural rules but can interact with each other, produce complex, unpredictable behaviour), and then on to networks. Finished off with the observation that by observing a network, one could find out those nodes which are highly influential. For e.g., in the blogospere, there could be a blogger who is connected to just the right people and has interest in the idea you wish to popularize. This blogger does not have to be an A-list blogger with lots of readership, just rightly connected. Okay, but the relationship with fractals or complex systems is tenuous. The talk could have been better.

The next talk was titled "The Two Towers (of Web 2.0)". I walked in a little bit late and heard people going on about Ajax. Asked someone what the other pillar was, and heard that one pillar was not ajax but technology and the other was marketing. Marketing? Well... not surprising considering that the talk was anchored by a group called Singapore Entrepreneurs, a group that runs a blog and also provides vc funcding in some way. I chipped in with an observation that while Ajax is a great technology enabler, another interesting aspect is the opening of the web as services with APIs through which people create and transform existing content into new forms. Guess people did not like it.

The only interesting points in this talk were that singapore is not ready for web 2.0 and that the talk should have been about doing stuff, not about whether we can or should do it. The rest of the time was spent in regurgigating stuff from techcrunch.com (did not know it was so popular), and blaming timid singaporean VCs.

The next talk I was interested in attending was by Yahoo about widgets and gadgets. But at this point I had to leave.

Interesting event and I look forward to attending more. If I could stand up to the barrage of questions that other starter-uppers got then I would be proud of my pitching skills (don't have any, I think). I do hope the level of the discussion goes up a little bit. It is most interesting when people talk about local circumstances and local problems and not debate about whether Singapore is too small. It is. Apparently some people want to start-up anyway. What could they do better?

GPACS: when to refactor

2007-02-12T02:20:00.000-08:00

Here is a situation:

You are working on specs for the next sprint and they are not complete;
You are considering a new technology to replace one you are using;
Experimentation has shown that it is good.

I say, take the time to refactor! Use whatever knowledge you have now. Don't overdo it, because the new specs may call for some changes.

People struggle to allocate time to refactor. One should do it whenever one can. Especially when:

You have time on your hands;
You know going forward you will be using the new tech
You have the luxury of replacing some tech with another without simultaneously having to deal with changing requirements (at least while you are doing the replacement).

Does it also help not to have to think about the future design, while you are re-factoring something in the current design? Sometimes I think one needs to consciously choose to juggle only one variable at a time.

Keeping the fires fed

2007-02-05T20:03:00.000-08:00

I did a mid-sprint review yesterday. Work seemed to be going on at a rapid clip, even when some developers were involved in support activities or other projects. What I realized is that a steady stream of well-defined work activities keeps people occupied, purposeful, and productive.

Where does re-design fit into this? When people start thinking about design or design changes, I've noticed that sometimes very blue-sky ideas come in. The good part is that the design might be very good (although most often it is hard to tell whether a design idea is significantly better that what you already have).

The way to control design efforts to be useful, incremental and low-risk is to---probably---focus on implementing some part of a use case. Design can then evolve.

Of course, radically new design may need re-writes
and revolution, not evolution, may be the way to go. But that way lies risk without the certainty that the new design would lead to a significantly improved situation. Especially in the face of evolving requirements.

GPACS: the early days

2007-02-03T02:10:00.001-08:00

GPACS is the company's accepted software development process. It is a quite like Scrum. More about it at some other time. The point is that I am managing an enhancements to an existing software while also considering re-doing or modifying it for some new requirements. One of the first things I need to plan is to how to get various teams working together. The other thing I need to figure out is to how can the same team work simultaneously on "enhancements" to existing code-base while trying to implement something that has a slightly different set of requirements.
.
I came across a bunch of articles by Chad Fowler: "The big rewrite".
And then another article
by someone called Kevin Barnes.

Oh, I was already thinking that a complete rewrite was out of the question because of the risks involved. But I could not have listed down the issues/risks as well as these two articles have.

Audacity

2007-01-01T02:14:00.001-08:00

From http://www.iftf.org/features/map_of_the_decade.html:

Want a quick tour of the future? Each year, the Institute for the Future plots the big trends that will define the coming decade on an AT-A-GLANCE graphic map. Scanning the Map is a little like swallowing a time capsule from the future--and getting very smart very fast!

More cool words:

The Group Economy: Organizing for sociability
Lightweight Infrastructures: Rethinking the movement of goods and services
Sensemaking: The new science and technology of meaning
Farming the Planet: Ground zero for global turbulence
Transformational Geography: The new role of the global south
X-People: The intentional evolution of human being

Wow, this site really reminds me of http://huhcorp.com.

Who funds these guys? Looks like it is a membership driven privately funded organization.

Who are these guys? The roots were engineers from RAND corporation. Now? A group of researchers with technical degrees, and some technical experience.

What's the difference?

2007-01-01T01:51:00.001-08:00

From http://www.iftf.org/:

We provide the foresight to create insights that lead to action.

From http://huhcorp.com/:

We provide distinct clients with groundbreaking business strategies and cutting-edge designs to aggressively and creatively compete in a changing economy.

Vote for me, get a TV

2007-01-01T01:05:00.001-08:00

December 29, 2006: Declaring color television a basic necessity, an Indian political party promised free sets to the poor and swept to power in May in the southern state of Tamil Nadu. So far, the Dravida Munnetra Kazhagam party, or DMK, has handed out 60,000 sets and plans to give away 30,000 more in coming months....As far as political platforms go, this one clearly has been a winner. "I will always vote for DMK," said Parimala, a 51-year-old widow who like many in these parts uses only one name....An electoral stunt for sure, but one that illustrates how populism has emerged as the default position of parties across India's political spectrum as they contend with the fading of once-dominant ideologies....But no one has gone as far as Tamil Nadu's DMK. Part of the reason is money: Tamil Nadu is one of India's wealthiest states, giving its government full tax coffers.

From free rice to free televisions. Obviously Tamil Nadu has been progressing a lot.

Tenth deadly sin of project planning

2006-12-30T10:18:00.001-08:00

In an IEEE Software 'From the Editor' column, Steve McConnell writes (at http://www.stevemcconnell.com/ieeesoftware/eic19.htm), that the nine deadly sins of project planning are:

(1) Not planning at all

(2) Failing to account for all project activities

(3) Failure to plan for risk

(4) Using the same plan for every project

(5) Applying pre-packaged plans indiscriminately

(6) Allowing a plan to diverge from project reality

(7) Planning in too much detail too soon

(8) Planning to catch up later

(9) Not learning from past planning sins

I am managing a project right now, and I just can't figure out which sin I am committing: (2) or (7)? I suppose the answer is that is depends on the project and its circumstances. But of course. It always does.

And uh, so at least now I know that these are all the deadly sins of project planning and other sins are presumably not deadly. They are merely extremely toxic, presumably. My project won't exactly die if I forget about some other practical issues, it will just go into convulsions or stop breathing for a while, or sway gently from side to side when it should be rocking.

In other words, I still have to worry about the non-fatal issues in some particular order which is highly project dependent, and these so called fatal sins have to watched out for but exactly when a sin is being committed is also dependent on the project.

Duh!

Why do these wise men write these articles with zero information? I just wasted a perfectly good 5 mins which I could have spent more usefully reading about a API, framework, or some such, that would actually have solved a few real problems.

The tenth deadly sin of project management: trying to get wisdom from anything other than hard facts when you should be doing project management: learning, building consensus, disseminating information, talking to your team, and writing code.

When unix processes are too much

2006-12-16T02:43:00.000-08:00

I want to ssh into a server, run vmstat for a while, then get back the output of vmstat. All automatically, via a Python program. There are many problems. Here is a simulation of the problems through the command line.

Take 1

The first thought was to ssh in, run vmstat and redirect the output of vmstat to a file on the remove server, like so. Then later, I would kill ssh, and use scp to retrieve the logfile from the remote server. Here is the command line equivalent:

$ ssh 'couffable@asgard' 'vmstat 1 2>&1 >/home/couffable/remote-vmstat.log' &
[2] 10405

On remote machine:
[couffable@asgard ~]$ ps ax | grep vmstat
20921 ? Ss 0:00 bash -c vmstat 1 2>&1 >/home/couffable/remote-vmstat.log
20938 ? S 0:00 vmstat 1

This starts things nicely and gives me the PID of the ssh process. The trouble is when I want to stop the process:

On local machine
$ kill 10405
$ fg # to get bash to retrive ssh's status and remove it from zombie list

But on the remote machine, vmstat is still running!
[couffable@asgard ~]$ ps ax | grep vmstat
20921 ? Ss 0:00 bash -c vmstat 1 2>&1 >/home/couffable/remote-vmstat.log
20938 ? S 0:00 vmstat 1

This won't do at all.

Take 2

I noticed that if I run vmstat though ssh without redirecting vmstat's output on the remote machine, ssh shows vmstat's output on my terminal. So I said, why not redirect ssh's output?. Here is what I tried:

On local machine:
$ ssh 'couffable@asgard' 'vmstat 1' 2>&1 >/home/couffable/local-vmstat.log &
[2] 10569

On the remote machine:
[couffable@asgard ~]$ ps ax | grep vmstat
21239 ? Ss 0:00 vmstat 1

Bingo! Now vmstat's output is being stored in a local file, not on the remote machine. Or is it? I notice that on the local machine:
$ # press enter
[2]+ Stopped ssh 'couffable@asgard' 'vmstat 1' 2>&1 >/home/couffable/local-vmstat.log

That is, vmstat/ssh's output is not getting logged because ssh stops when backgrounded. Puff!

Take 3

I decided that I won't background ssh.

On local machine:
$ ssh 'couffable@asgard' 'vmstat 1' 2>&1 >/home/couffable/local-vmstat.log

On remote machine:
[couffable@asgard ~]$ ps ax | grep vmstat
21624 ? Ss 0:00 vmstat 1

After some time, again on local machine (in another terminal):
$ ps ax | grep vmstat
10880 pts/1 S+ 0:00 ssh couffable@asgard vmstat 1
$ kill 10880

And yes, the vmstat on remote machine dies too. Now, will this work in Python?

Take 4

The trouble with running the results of Take 3 in Python is that I can't directly invoke ssh: the redirection won't happen. For redirection, I need a shell. So I could do something like:

>>> pid = os.spawnlp(os.P_NOWAIT, "sh", "sh", "-c", "ssh couffable@asgard vmstat 1", "2>&1", ">vmstat.log")
>>> procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 144 82808 231972 1293708 0 0 2 2 0 2 1 0 99 0

What? No redirection? Not good. After killing ssh processes, I try again:
>>> pid = os.spawnlp(os.P_NOWAIT, "sh", "sh", "-c", "ssh couffable@asgard vmstat 1 2>&1 >vmstat.log")
>>> pid

Redirection is working now. I kill the process:
>>> os.kill(11935,9)
>>> pid, status = os.waitpid(11935, 0)
>>> status
9

Ok, so far so good. But the ssh and vmstat processes are still running! (Killing ssh does stop both ssh and vmstat, like Take 2 and unlike Take 1.) What gives? Killing sh does not stop ssh since sh spawned ssh. Hmm.... I don't really want sh to spawn ssh. I want sh to redirect standard output and error and then exec ssh.

Take 5

>>> pid = os.spawnlp(os.P_NOWAIT, "sh", "sh", "-c", "exec ssh couffable@asgard vmstat 1 2>&1 >vmstat.log")
>>> pid
12080

On local macine:
$ ps ax | grep vmstat
12080 pts/4 S+ 0:00 ssh couffable@asgard vmstat 1

On remote machine:
[couffable@asgard ~]$ ps ax | grep vmstat
24081 ? Ss 0:00 vmstat 1

Well and good. Now let's kill the ssh process.
On local machine:
>>> os.kill(12080, 9)
>>> pid, status = os.waitpid(12080, 0)
>>> status
9

On local machine:
$ ps ax | grep vmstat
{nothing}

On remote machine:
[couffable@asgard ~]$ ps ax | grep vmstat
{nothing}

Yay!

The question is: all this mucking around with Unix plumbing is wonderful, but could I have have done it faster if I'd written my own fork(), redirect, and exec mini-program instead of trying to invoke os.spawnlp(...) using sh magic?

Jeff Darcy's notes on really high performance servers

2006-12-09T08:35:00.001-08:00

In his article "High-Performance Server Architecture" (http://pl.atyp.us/content/tech/servers.html)

Jeff Darcy talks about what kills performance. He mentions the following:

data copies
context switches
memory allocation
lock contention

as the biggest four reasons for poor performance, especially at high concurrency. I did not quite understand his suggestions on reducing lock contention. But perhaps it will become clearer by putting his idea to work on a real problem.

He also mentions:

How does your storage subsystem perform with larger vs. smaller requests? With sequential vs. random? How well do read-ahead and write-behind work?
How efficient is the network protocol you're using? Are there parameters or flags you can set to make it perform better? Are there facilities like TCP_CORK, MSG_PUSH, or the Nagle-toggling trick that you can use to avoid tiny messages?
Does your system support scatter/gather I/O (e.g. readv/writev)? Using these can improve performance and also take much of the pain out of using buffer chains.
What's your page size? What's your cache-line size? Is it worth it to align stuff on these boundaries? How expensive are system calls or context switches, relative to other things?
Are your reader/writer lock primitives subject to starvation? Of whom? Do your events have "thundering herd" problems? Does your sleep/wakeup have the nasty (but very common) behavior that when X wakes Y a context switch to Y happens immediately even if X still has things to do?

I am now itching to try a few things. However, how does one begin in a dynamic language like Python? I can't do much about memory allocation. Umm... also, Python does its own reference counting, so data copies should not be a huge problem: I just have to ensure that my own code does not make unnecessary copies. Context switches and lock contention look like the primary and secondary targets to focus on for performance and design.

This statement is rather interesting:

It's very important to use a "symmetric" approach in which a given thread can go from being a listener to a worker to a listener again without ever changing context. Whether this involves partitioning connections between threads or having all threads take turns being listener for the entire set of connections seems to matter a lot less.

Now, how does one go about doing that and still have clean, understandable, and maintainable code?

Will you look into the crystal ball for me, please?

2006-12-06T10:39:00.001-08:00

I wonder what this guy has to say about mobile advertising.