Exploring chaos with Python

Having just written another small python script, I have a greater appreciation for python than I did before. This particular script explores how a certain function becomes chaotic for specific values of a coefficient. It's pretty simple, but the results are pretty interesting:
The above image is are some graphs of the system (see the link for more details) as it becomes chaotic. Python was pretty cool for this script because of the simplicity of generating the graphs. Using the NumPy and matplotlib libraries, generating the data set and graphing it is very straightforward. In fact, the code to plot is a simple oneliner:

plt.plot(xValues,yValues,',')

The two parameters are just lists of data values, and the "," character means that each data point is represented by a pixel on the graph. Everything else is handled by the plotting library.
Oh and one more very important thing: go grab the PyScripter IDE before any python development. It's a pretty nice tool and has what you would expect from an IDE (especially stepping through code in a debugger!). It even has support for unit testing.

Approaches to supplying software updates to your end-users

Let's say (hypothetical ... of course) that you have released a piece of software and users have downloaded it. Suddenly you find a bug that you immediately smash in the next revision. Unfortunately, this particular bug is pretty annoying for you (the developer) and you don't want your users getting rid of your software. How do you go about ensuring that they get an update? Here are some approaches I thought of:

1. Hope the users navigate to your website every so often to check for updates
2. Add an automatic update program to your application
3. Have a "check for updates" option in your program

Unfortunately, options (2) and (3) require that you have done this before placing your application for download. Option (1) is a bad idea (at least according to my website traffic) because people don't come back to your home page very often (if at all). So what do we do?

The way I see it, we have to settle for a compromise. Implement option (1), but also go ahead and either do (2) or (3) for future releases of your application. That way anyone who downloads it from now on will have update capability. Depending on how much time you want to invest, you might go with a full blown automatic updater or just settle for a "check for updates" option.

How do you go about checking for updates? If your application tracks its version, you can compare its version number to an official version number on your website (stored in a text file perhaps) by requesting it from your application. Then if the versions dont match, you can inform the user and direct them to the update website.

Now the next step is to find time to actually implement those ideas...

What happens when you just set a Stream position to zero (C#)

This morning I was fixing a few bugs which I had found in HTML Combiner and I noticed a few of my unit tests that loaded configuration files were failing. They were using a MemoryStream object so that they read from a string instead of the file system. The first thing I thought was that it was just a problem reading from a string, and wouldn't happen when reading from an actual file.

Of course then I ran the program and notice that when I loaded the configuration file for my website, my tags were showing up twice. After setting up NUnit to run in visual studio and stepping through the debugger, it was in fact reading the file multiple times.

After spending about an hour wondering what was wrong, I casually glanced at a few links on google about resetting a stream position (which I do at the beginning of the method). Finally, I made it to the StreamReader.DiscardBufferedData example code on the MSDN library website.

If you change the stream position using Seek, the actual position of the buffer is now out of sync. So you have to call DiscardBufferedData in order to force the stream to go to the desired position. After I added this line everything worked once again:

sr.DiscardBufferedData();          // where sr is a StreamReader
sr.BaseStream.Seek(0, SeekOrigin.Begin);

This made me glad that I've been adding unit tests to catch things like this -- I might have overlooked the problem the tests weren't failing. Another similar incident happened a few days ago when a small bug was introduced after I made a change, the bug would result in the build path not being loaded from a configuration file. The actual problem looked pretty innocent, it would have taken time to isolate the cause. However, I was able to catch it because the test failed shortly after I had made the change.

Unit testing a method that makes heavy use of FileStream (C#)

While working on the HTML Combiner, I ran into a situation where I was trying to test a method that wrote a configuration file. This method makes heavy use of FileStream, and so the resulting test would have been coupled to the filesystem. This seems like a common issue you would run into, and there are a few ways to approach the unit test for such a method:

1. Allow the filesystem dependency
2. Create an "IDataSource" interface and make an implementation that wraps the FileStream object
3. Instead of using FileStream, use its parent, Stream, in the method you're trying to test

Option (1) is out since that's exactly what we're trying to avoid! Allowing the dependency on the file system is a bad idea, because it allows anything that can go wrong with file IO to potentially fail your unit test at an unpredictable time. In addition, if you're writing to files, you have to worry about cleaning up so your tests don't read the file of a previous test run.

Option (2) seems like a good idea because you can stub the IDataSource interface in the unit test. In fact, I probably will do this later on (I admit, I was rushed for time!). This means that you will have to change your method's code to use the interface instead of changing "FileStream" to "Stream," but it's a more elegant solution that does not depend on Stream at all.

Option (3) is the one I went with, and isn't a terrible solution. By using a Stream object in the target method, you no longer depend on the FileStream (and consequently the file system). In your test for example, you can pass in a MemoryStream which contains a test "file" whose contents are in a string you declare. However, I did encounter some disadvantages:

• You still have a dependency on Stream, instead of using a stub
• You have to specify the encoding your string uses in order to create a MemoryStream out of it
• Creating and writing to the MemoryStream is not painless

Well, we still achieve decoupling from the filesystem, which was the main goal. However, in the future, I will probably switch to an "IDataSource" interface and use that. This way, we get total independence from Stream and we could also write an implementation that, say, reads/writes an SQL database.

HTML Combiner -- a small utility for inserting HTML into multiple pages

No doubt if you've worked on websites before you've wanted to reuse common blocks of HTML code across multiple pages -- such as the link bar on the right of this page. Normally you could accomplish this through Server Side Includes (for pure HTML), iframes, or even Javascript. Now my web hosting plan doesn't support SSI, and after reading about the downsides of iframes I'm not very ecstatic about using them either. Finally, I can't see myself using Javascript just for a links bar -- if someone has it disabled then they can't navigate my site!

So, wanting an excuse to write some C# code, I wrote a small utility that allows you to reuse HTML code by inserting tags throughout your website. The program then inserts your block of code where those tags appear in a the pages you want. In fact, I've been using it across my website for the links bar and also some statistics tracking scripts. Here is a screenshot of the main screen

Hoping that someone else might get some use out of it, I've put it up for download on its own page and submitted it to several software download sites. I haven't distributed my own programs since 2007, so it was an interesting process. In fact, it reminded me of a nice fact:

Programming is easy. Releasing software is hard

I have a very simple utility, but I have to put up with quite a few issues:

1. Making sure there are error handlers everywhere
2. Setting up an installer (an adventure in itself
3. Targeting a version of .NET that is widespread (I chose 2.0)
4. Writing a help file
5. Building the product website
6. .... (more) ....

Actually it took me a good 6-7 hours to finish everything, and there are still alot of things I need to add. But it's up now, so I'll be able to add to it pretty easily.
That's all for today -- in my next post I'm going to talk about my recent experiences with adding integration testing and also my attempts at unit testing methods that are heavily dependent on FileStreams.

Using the Google Chart API (cont'd) and caching images for retrieval

Yesterday I talked about using the Google Chart API on the server to send chart images to a client in response to a query. Today after meeting with my colleague, he revealed a more straightforward approach than trying to cache the generated images and retrieve them from the image tag.

Well, we're still doing that, but instead of generating the image data and sending that, we generate just the url string to send to the google chart site. Then we send that to the client, where it is embedded into an image tag. Now we don't have to cache any image data. However, we still might end up caching it for a simple reason. It would be nice for users to get a quasi-permanent link to a generated graph, so that they could type "address/graph?id=..." instead of having to put up with a Google Chart url string (which might be long depending on the dataset).

There are still some more things to add to the code before it is ready to integrate with the main project, but overall it will make a pretty nice addition.

Using the Google Chart API to send images in response to a GET request

While we are finalizing the design for our project, we've each been working on a different small task. The task I've been working on is implementing the ability for the user to make a chart based on the data set they are looking at. They should be able to say "hey, I want to see the number of concerts each year from 1990-2010," and it should work. To do this, I've been using the Google Chart API to generate the chart on the servlet and then send it to the client.
To accomplish this, I've been following this answer to a question on stackoverflow.com The process works like this:

1. Client sends a POST request to the servlet with parameters specifying the data set and ranges to use
2. Servlet builds a GraphBean object from the client's request and sends it to a GraphBuilder
3. GraphBuilder constructs a url and sends a GET request to the Google Chart API url
4. The image generated is given a unique id and stored in the servlet class
5. The servlet writes a response that references the unique image id in an image tag
6. The image tag sends a new request to the servlet with that image id, and the servlet sends the image back
7. The image is removed from the servlet

Actually, the last 4 steps aren't implemented yet, but I expect to have them done soon. I've been developing the first steps in a regular java application instead of using servlets. However, the integration with the servlet will be (hopefully) painless. Once this is done, we should get some nice pictures out of it when our database is set up.

Forcing mandatory unit tests when pushing to the central repository

For the past few days I've been working on getting our build process automated to deploy to a local server (since we have no remote host as of yet). It's deceptively simple:

1. build java source files (including tests)
2. copy compiled class files to deploy directory

Sounds easy, right? It should be, but my relative inexperience with web apps translates into spending extra time configuring environment variables, searching out libraries and putting them in the right place, etc. But hey, that's finished now!


The real issue I wanted to tell you about is related: We want to make sure our unit tests are run each time someone tries to push to the production branch of our central repository, to make sure they haven't broken. I figure we also want to test them outside are local environment, let's say the directory of the local server we test on.
So we add a preoutgoing hook to mercurial that simply launches an ant script that will run the unit tests and the push will fail if the tests fail. To get this working though, my process resembled this less-than-ideal workflow:

1. build java source files (including tests)
2. copy compiled class files to deploy directory
3. call junit task from within Ant
4. Observe that _x_ is not in the right place
5. Put _x_ in the right place
6. Repeat

Where _x_ usually refers to hibernate and one of its xml files. Figures. However, after hours of pain and realizing that I had pointed the ant script to the wrong version of Tomcat, this process now resembles the ideal process at the beginning. I'm pretty happy with it, I just hope it continues to work...