Erlang - Parallel Programming without Pain

Joe Armstong's Lecture on InfoQ. Must watch.

My First Mac and Virtual Box Experience

I am officially a Mac convert.

Still learning how to survive on one mouse button and use these fn, control, option and command keys. But so far so good. My MBP experience is absolutely positive.

Before the purchase, I was slightly worried about if I may pay too much for a fine GUI. After last few days of hand-on experience, I think, as many MBP users has attested, the premium is well worth it.

I, in particular, appreciate the BSD root of Mac OSX. Whenever I cannot use the Mac OSX GUI to get thing done, I can always fall back to command line alternatives.

Vista is a staggering failure of MS Corp in every sense (I have used Vista for a year). As Paul Graham puts it, Microsoft has ceased to be relevant. They still have market share, but it is a market share going to erode fast, under the attacks of Sun, Linux and FOSS. The unwillingness to waste any more time on Vista is my prime motivation to switch.

The very first application I have downloaded is the VirtualBox. Basically within seconds I can start a new virtual instance of Ubuntu 8.04 beta. By default I can only ssh into the box via NAT. Will need to do some research to see how to get bridging done in Mac OSX.

The second application I tried is, not surprisingly, QuickSliver, an utility that warranted an hour of Google Tech Talk.

The Importance of Being Nosy (Nosetests, Really)

Unit testing

One of a few important lessons I learn from Java development is the importance of unit test. It is of course not a silver bullet to the software reliability/quality problem, but the consequence not adopting it can only be more dire.

I used only JUnit to implement unit tests in Java (Since JUnit is "good enough", I never explore the alternative framework TestNG.). Actually, problems in writing unit tests usually have very little to do with the frameworks themselves, but more with:

A. The legacy code base you want to test against. For example, if the existing code base design is not modular enough, you have to go through some refactoring process to make unit testing possible. It is not the kind of luxury we can afford very often.

B. Mock objects. EasyMock has made mocking a dependency easier, but maintaining the mock object codes still requires considerable effort.

Unit testing and Python

I think unit testing is even more important for Python programmers.

Since everything happens in runtime for Python, errors in your scripts, no matter syntactical or logical, will not be caught until you actually execute the line of code where the bug lives*.

On top of it, a majority of commercial/in-house software contains so many features. The sheer number of possible combination of the execution pathways means thorough manual testing is simply unattainable, if not unrealistic.

If at the end of the day, a production issue in your python software arises and it turns out to be "incorrect indentation", the managers and users will look down on you pretty hard ("Did you really test it?"). Unit test + coverage tools can help us to avoid these kind of situation.

Besides, unit tests can be used to safeguard corner cases and exceptional conditions. For financial applications, one of those commonly-seen mistakes is that a program fails to perform properly when a certain event (e.g. rollover) happens on a public holiday. With unit testing, we can subject the relevant codes to different combination of date and instruments, so we can make sure it will, for instance, calculate accrual correctly no matter what.

Comparing Java, I find it easier to do unit testing in Python because, with Python's "Duck Typing" capability, we do not have to worry about maintaining mock objects at all. Whenever you need to mimic a dependency's behavior, just write a dummy class with just enough attributes/methods the testee expects. End of story.

No more record and play APIs, as in EasyMock.

Nosetests makes it easy

So, you can understand why I am so glad to find nose. Nosetests is, in a nutshell, a unit test discovery tool. It can traverse down a python source code directory and pick up classes that extends unittest.Testcase, doc test, and any classes/methods that follow the default naming convention. I can now mix any testing styles depending on the task at hand.

I usually use the following parameters with nosetests:

--verbosity=3 --with-doctest -s --exe

"--exe" is necessary because in cygwin, my primary development environment, py files are usually set to be "executable".

Feedback

Do you have any tips and idea on unit testing/software quality assurance to share? Feel free.



* It is not entirely true. If you want to catch syntactic error before run-time, you can use the compiler module to parse each py file individually. It is useful when you try to refactor by renaming modules/variables using tools like sed. This allows to make sure the regex does not do anything stupid with your code.

Python in the Corporate World

Python is used widely in academia, open source projects and start-up. However, when it comes to the corporate world, it seems to me it is not making a lot of inroad. At least this is true in the finance industry, which I am more familiar with.

A quick check on the python.org's success stories, only two companies are listed as Fortune 500, and none of them is in the finance.

Occasionally there are ads looking for people with python skill in the financial sector, but the positions are usually of support or operation nature, not application development.

From first glimpse, it is a classic chicken-and-egg problem: there is not enough skillful python programmers out there to fill the jobs, it is difficult to start new projects in python. However, if python programming job market is weak, there is little incentives for students or programmers to pick up this non-mainstream language.

But there is something more than just supply-demand. I believe a common perception that, somehow, dynamic scripting languages are not for large scale projects and is difficult to maintain also make people shy away from using Python.

The not for large scale projects argument is a not too hard to counter. We just need to find some good examples of high visibility. For example, Google Groups and mercurial.

However, there may not be a clean-cut answer to the question of maintainability. First of all, we have a generation of programmers who have used to statically typed languages and their accomplishing IDEs (such as Java and Eclipse). To this group of audience, the idea that one can build a full-fledged application using just API documentation, command line tools and vi/emacs is beyond imagination. It is easy to conclude that, without IDE support, it is difficult to refactor the code base, which is an essential step in keeping the application in good heath.

Secondly past experience may still leave a bad taste. If you have done maintenance of legacy applications written in perl, some of them could make you feel like you have just visited a programmer's hell. Cryptic code and inconsistent style are everywhere. (But it'd still be way better than tcl. Luckily I have not come across any such programs yet). This bad memory certainly make people think twice before embracing dynamic scripting language in general, Python in particular.

A disclaimer I must make: what I wrote above is based on my experience only, and your organization may be free of the above issues, especially if yours is relatively smaller in scale (so it is relatively easier to innovate) or relatively young (so there are less legacy applications). But today in most of the giant, multinational financial institutions that inspire to be a "financial supermarket" for all, the decision markers, especially the "professional manager" type, have a total different mind-set. First of all they will not be benefited much, personally, by giving green light to the use of non-mainstream technology/language that promise better result and higher productivity. On the other hand, they have everything to lose if the brave decision does not deliver. Java (or C# if it is a MS Window shop) is always a safe bet, even though it means one need an army of programmers to get thing done. In essence, it is an agency problem.