Wednesday, November 30, 2011

Use screencasts for your bug reports

Note: So finals are about to start, you might not see me for awhile.

Anyway,

On the project I'm on now, we're working directly with the client (the owner of a local business) on a web application. We took over the project from another company working on it and started with some bug fixes. In my experience the most difficult part of tracking and killing bugs reported by someone else is simply reproducing them.

Let's take a look at a bug report style you might have encountered (I certainly have):
Bug: Clicking on the SaveLotsOfMoney button causes the browser to explode

Login as Jim
Click "SaveLotsOfMoney" button
"Server error" is displayed right before the browser explodes and crashes

Attached: a screenshot of the browser exploding. 

Alright, this is actually a good one because at least they stuck a screenshot on it and they were nice enough to enumerate the steps they had taken (as opposed to "it just crashed").

Now you've just taken the project and you have no idea how the code works. So you try and reproduce it, but the SaveLotsOfMoney button works fine for you. Oops?

Turns out the actual problem was that during previous testing the reporter did something unrelated, data was changed, and the guy before you forgot to do a null check and now the browser has exploded. It doesn't happen for you because you're making your tests reproducible (of course).

How do you prevent this? Make them do it again? Now they can't reproduce it! They didn't know that what they did previously affected that test, because it's not their job to know the details, so of course they don't include it in their bug report.

The solution to this that our current client uses is to make screencasts of the bug in action. This is very important, because it means you can see everything that they were doing that caused the bug. It can vary, and in my case it was a detailed error message (a "foreign key violation -- big help") that was shown. In another case I was able to see signs of data they had previously changed, which led to reproducing it.

So: don't use those old crusty bug reporting methods, try making screencasts of them instead. The client mentioned uses a tool called Jing (disclaimer: I don't work for them) to record their screencasts and it looks good to me, though I have no personal experience with it.

No comments:

Post a Comment