Monday, December 12, 2011

Scrutinizing another project code base: what to look for

So when you're evaluating an unfamiliar project's codebase, make sure to be careful and not be deceived by how it first looks. On the current project we're working on the code looked like it had a good architecture to begin with, but after working with it for a few weeks you begin to notice some things that are just weird.

Here's some things to watch out for (keep in mind we have an ASP .Net MVC 2 project):
  • Controller logic mixed into the view layer (this is a terrible sign). Not very common in this project but I've seen plenty of it.
  • (Off that bullet) Too much C# code in the view layer that's not really controller logic, but stuff like loops, if blocks, etc. that render different HTML elements are just ugly. 
  • Embedded JavaScript code inside aspx files: In my opinion this is sloppy and pollutes the markup with JavaScript code. What's even worse is JavaScript code that also contains C# code in it!
  • Large JavaScript files. JavaScript code should be treated like your other code -- this includes having some kind of separation to give a modular structure (makes it unit testable too) 
  • Code duplication. This is a big one: why is the code being duplicated? A severe case in this project is two JavaScript files containing the same code but in different places. They even had the same typos! A really annoying case of this is when you have three classes with the same name but in different namespaces.
  • Doing too much in the data layer. Having too much stuff (such as branching logic etc.) in the data layer makes it hard to unit test the bugs that will pop up from it.
  • Commented out code: This is not a red flag but should still be a warning sign. Why is there code that's been commented out? If it's under source control (and it better be) you already have a copy of it, if not then why isn't there an explanation of why it's been commented out?
  • Swallowing exceptions: This is my favorite because often to the user it just looks like their button doesn't work, when in reality an exception was caught, swallowed, and the action just didn't proceed.
So, I've seen all of these things in the project we're on now and this is based on that experience. Your results may vary.

Thursday, December 8, 2011

Selenium tests that fail "sometimes" and Jenkins

We're running Jenkins as our CI server, and part of the automated deployment is to make sure the UI tests pass. Well, sometimes the Selenium tests fail for one reason or another after passing for say, 20 builds. Initially I hypothized the following:
  • Click events don't register
  • IE has a bad day
  • Some kind of timing thing
All of these things have occurred in the past -- especially with IE. So with the help of stackoverflow I went ahead and implemented some things to fix this problem.

The first thing was to Re-run the UI tests that failed a second time. Of course this comes with all the drawbacks and this points to problems with the test itself. In this case that was ok because the test was still useful for confirming a bug was fixed, but we still need the build to pass. But I'll come back to this issue later on in the post.

Well it turns out that while JUnit has the @Rule annotation (see this post) which allows you to intercept test calls, NUnit doesn't have that. Oops. I tried using the NUnit extensibility API but that didn't turn out to well (my fault I'm sure).

So I resorted to this method in our UI test base class, SeleniumTest :

public void RetryFailure(Action testMethod)
{
    if (_retrying)
    {
        _retrying = false; 
        return;
    }
    try
    {
       _retrying = true; 
       testMethod();
    }
    catch (Exception e)
    {
        Console.WriteLine("::::::retrying:::::::::::");
        Console.WriteLine("error was {0}", e.Message); 
        WebDriver.Quit();
        SwitchBrowser(_currentBrowser); 
        testMethod(); 
    }
    _retrying = false; 
}

I'm not too excited about the code quality of this method, but it does work. All it does is attempt to run the test, catch any failures, and then rerun the test again. Here's what the beginning of a test that uses this looks like:

[TestCase] 
public void AdminCancelANeedViaOfficeCalendar()
{
    if (!Retrying)
    {
        RetryFailure(AdminCancelANeedViaOfficeCalendar);
        return;
    }

// ... more code

As you can see, we have to have this additional bit of logic at the beginning of each test. Now, it would have been easy to just encapsulate the test code in some kind of loop and then repeat it that way, but this allows the SeleniumTest.RetryFailure method to have all of the control.

Alright, so the next step was to generate screenshots of test failures. This is pretty easy as Selenium already provides a way to do this by casting the WebDriver to ITakesScreenshot. So nothing exciting there.

Now to integrate this into Jenkins. At this point, we have a bunch of failure screenshots being archived as artifacts, but you have to actually go look for them to view them. I'm lazy so instead I took a shot at providing a custom HTML report of the failures with their screenshots:
Summary of test failures

So the screenshot is from a page that is accessible via the Jenkins project page. This means that we can be lazy and still get the information we need.


 
UI Test Failures link shows up on the project page
 The HTML Publisher plugin takes care of the details, all we have to is generate the actual page. This is done once again inside SeleniumTest.

 private void BuildReport(string reportFile, string testName, string testImagePath)
{
    if (!File.Exists(reportFile))
    {
        using (StreamWriter sw = new StreamWriter(new   FileStream(reportFile, FileMode.Create, FileAccess.Write)))
        {
            sw.WriteLine("<html><body>");
             sw.WriteLine("<h1>Test Failure Screenshots</h1>");
         }
    }
    try
    {
          using (StreamWriter sw = new StreamWriter(new FileStream(reportFile, FileMode.Append,  FileAccess.Write)))
         {
             sw.WriteLine("<div>");
             sw.WriteLine("<h2>" + testName + "</h2>");
             sw.WriteLine("<img src=\"" + testImagePath + "\"/>");
             sw.WriteLine("</div>");
         }
    }
    catch (Exception e)
    {
         Console.WriteLine("Problem writing report: {0}", e.Message); 
    }
}

Pretty simple, probably could be moved to somewhere more fitting. In the mean time though, it works well enough.

Anyway, the point of all of this was to just allow us to see why tests fail but also retry them to see if they fail reproducibly. It turns out that a couple of the tests repeatedly fail the first time but not the second, so there's probably some work to do.

See you next time.

Sunday, December 4, 2011

Another abstraction layer on top of PageObjects with Selenium

Hi everyone,

As I mentioned before, on this project our Selenium 2 tests are extremely important. We've tried to keep the test code on the level of the production code, so it naturally has its own object model. We already use PageObjects extensively in the project.

Recently a few bugs popped up that we reproduced with Selenium tests that were pretty similar and required similar actions. Here's an example of some duplicated code:

// first step: log in as a temp to accept a need
LogOn login = new LogOn(WebDriver);
login.Go();
login.Login("Temp", "temporary");

TempNeeds need = new TempNeeds(WebDriver);
need.ViewFirst();
need.Accept();

login.Logout();
login.Go();
login.Login("Chris", "foo");

Billing billing = new Billing(WebDriver);
billing.Go();
billing.CreateNewInvoices();
billing.UpdateAndReview();
billing.FinishCreate();

Turns out this bit of logic is common in our tests, so it gets duplicated everywhere. So in the interest if minimizing the amount of code, I modified it to look more like this:

Temp temp = new Temp(WebDriver, "Temp", "temporary");
temp.AcceptFirstAvailableNeed();

Admin admin = new Admin(WebDriver, "Chris", "foo");
admin.CreateInvoices();

This code does the same thing, it's just been hidden by another abstract layer that represents the different users of the site and what they would do. Usually, each method off of a user would interact with many different PageObjects. The idea is to make the code reveal its purpose more by telling you what the user is trying to do, not the different PageObjects that need to be used.

Anyway, we're trying this approach out now for newer tests so we'll see how it goes. Hopefully the more tests we write and the more objects we build up, the less new Selenium code we'll have to write.

Wednesday, November 30, 2011

Use screencasts for your bug reports

Note: So finals are about to start, you might not see me for awhile.

Anyway,

On the project I'm on now, we're working directly with the client (the owner of a local business) on a web application. We took over the project from another company working on it and started with some bug fixes. In my experience the most difficult part of tracking and killing bugs reported by someone else is simply reproducing them.

Let's take a look at a bug report style you might have encountered (I certainly have):
Bug: Clicking on the SaveLotsOfMoney button causes the browser to explode

Login as Jim
Click "SaveLotsOfMoney" button
"Server error" is displayed right before the browser explodes and crashes

Attached: a screenshot of the browser exploding. 

Alright, this is actually a good one because at least they stuck a screenshot on it and they were nice enough to enumerate the steps they had taken (as opposed to "it just crashed").

Now you've just taken the project and you have no idea how the code works. So you try and reproduce it, but the SaveLotsOfMoney button works fine for you. Oops?

Turns out the actual problem was that during previous testing the reporter did something unrelated, data was changed, and the guy before you forgot to do a null check and now the browser has exploded. It doesn't happen for you because you're making your tests reproducible (of course).

How do you prevent this? Make them do it again? Now they can't reproduce it! They didn't know that what they did previously affected that test, because it's not their job to know the details, so of course they don't include it in their bug report.

The solution to this that our current client uses is to make screencasts of the bug in action. This is very important, because it means you can see everything that they were doing that caused the bug. It can vary, and in my case it was a detailed error message (a "foreign key violation -- big help") that was shown. In another case I was able to see signs of data they had previously changed, which led to reproducing it.

So: don't use those old crusty bug reporting methods, try making screencasts of them instead. The client mentioned uses a tool called Jing (disclaimer: I don't work for them) to record their screencasts and it looks good to me, though I have no personal experience with it.

Monday, November 21, 2011

Get up to speed on a new project by writing Selenium tests

Alright. So I've taken the task of maintaining and extending an ASP .Net MVC 2 project with a friend of mine. It's not too big, but there are some things working against us:
  • Neither of us have done ASP .Net development before
  • We had to set up a new database (this was actually the hardest part) 
  • There are no tests for the project
  • The javascript looks like our javascript back in the beginning of the Recordings project ... we've learned better since then.
Actually, let me emphasize that last point. I'm a big fan of automated testing, so much that I feel hesitant to modify code that isn't under test (excluding the simple stuff, of course). Especially on an unfamiliar code base. That's why one of the things we've been doing and really emphasizing on this project is automated UI tests with Selenium 2 WebDriver.

Writing tests using Selenium has helped us learn the functionality of the application as well as establish a good suite of tests for future modifications. And at the moment they are more useful than unit testing due to the difficulty of unit testing parts of the code.

In addition, we're making sure to keep our tests nice and clean using PageObjects and also making them reproducible when they mess with the database. Eventually we'll have these tests picked up by our CI server when it builds the project before deployment.

Of course, that doesn't mean we're going to go without unit testing the code. There are a couple static classes from the ASP .Net libraries that are mixed in with areas we'd like to test -- but using some questionable hacks we've been able to work those out. And after burying said hacks in another layer of abstraction, we can write nice clean unit tests like we all enjoy.

So: Now that we're on a new project things should pick up with this blog again. Recordings is live and complete, so there hasn't been much programming for a few weeks.

See you next time, though.


    Monday, November 7, 2011

    Log4net in a not-.NET application

    Hi everyone,

    So I've moved on to a couple of other projects (Recordings is now live and complete enough). One such project is an old legacy VB6 application that I'm extending with C# via COM interop.

    The situation I have is this:
    • I would like to use Log4net in the C# code, compiled as DLLs and imported into the VB6  project
    • I would like to configure Log4net via an xml config file, but it won't have to change at runtime. 
    So, here's how that's accomplished:

    The log4net.config file is added as an embedded resource in the C# class library project. It looks like this:

    <?xml version="1.0" encoding="utf-8" ?>
    
    <log4net>
      <appender name="DebugFileAppender" type="log4net.Appender.RollingFileAppender">
        <file value="J:\Chris\Source\citysiege\main\logs\" />
        <appendToFile value="true" />
        <rollingStyle value="Date" />
        <maxSizeRollBackups value="30" />
        <datePattern value="yyyy-MM-dd'.txt'" />
        <staticLogFileName value="false" />
        <layout type="log4net.Layout.PatternLayout">
          <conversionPattern value="%date [%thread] %-5level %logger - %message%newline"/>
        </layout>
        <lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
      </appender>
    
      <root>
        <level value="DEBUG" />
        <appender-ref ref="DebugFileAppender" />
      </root>
    </log4net>
    

    A static class method serves as a wrapper around the log4net library and an entry point to configure it:

                var assembly = Assembly.GetExecutingAssembly();
                var stream = assembly.GetManifestResourceStream("CitySiegeLogic.log4net.config");
                log4net.Config.XmlConfigurator.Configure(stream);

    Simple right? Well, took a little while to figure out. Also, if the config file is in a subfolder be sure to add that to the path along with the namespace, so for example "CitySiegeLogic.logging.log4net" if its under /logging.

    Thursday, September 22, 2011

    Workflow on the 2-person Recordings project

    Hi everyone. Today I'm going to share the workflow we use on our project. Mostly for reference, I'll come back in a year and see how things change. Hopefully we'll be on other projects by then.

    So let's just see the process for fixing a bug and deploying the fix to production. That will go through the entire process.

    Step 1: Observe the bug happening

    The bug I'm talking about is the canned graphs problem. Certain types of graphs showed this result:

    The Undesirable Result

    Great. After confirming that it's a Real Bug and not just user error, I create a new ticket on our bug tracker, Unfuddle: 

    The bug report: You can see it's "fixed," that's because this is a current screenshot. 


     Step 2: Reproduce and isolate the bug with tests

    In general I try to trap every bug with a unit or integration test. This particular bug had a good 'integration' test case for it, but I also wrote a UI test with Selenium

    You can see that I'm exciting about Selenium because I used the phrase "real live selenium ui test."

    Step 3: Bug fix

    The build jobs that run during merges
    The actual bug fix in this case was trivial (see the post mentioned above). After making the fix, the changes are merged from my dev branch into the Test branch (our main 'stable' branch). This merge is picked up by the build server, which builds the project, runs the tests, packages it into a WAR file, and archives it.

    Step 4: Deploying to production 

    Jobs running for Production
    Once the build is good, the next step would be more thorough manual testing to determine any tweaks, additions, etc. before we stop working on the feature. In this case though, we can just merge to our Production branch. 

    Now the fix is ready to deploy, so we can run the "Production Deploy" job. This will upload our WAR file to the live server which tomcat will automatically redeploy. After running "Production Status" to confirm the website is still running, we can visit it to confirm it's working.


    Well, that's pretty much it! The process is more exciting for new feature additions, but it's basically the same process. Check back in a year to see how the process changes....

    Sunday, September 18, 2011

    More bug-fixing with Selenium WebDriver and PageObjects

    A problem, I'm definitely connected to the internet.
    Hi everyone. I spent the past few days tracking and trying to reproduce another problem we've been having on the server. The problem is with the "recommend box" -- basically a way you can vote on your favorite songs or concerts. The problem happened seemingly randomly when you tried to recommend a song or a concert:

    Oops. This bug had a few annoying characteristics:
    • It was hard to reproduce and had to obvious order to causing the problem
    • I couldn't reproduce it locally 
    • The server stack trace was useless -- because we use an open session in view filter the root cause was a database problem but only showed up at the end of the filter when the transaction is committed 
    Not a good combination. After spending a couple days just experimenting and try to reproduce it, I finally sat down and decided to squash it for real. It was clear that without being able to reproduce it consistently, it was going to be a hard fix. It was also clear that it was a database related problem, so the first steps were:
    1. Restore the database with a set of test data
    2. Try to reproduce the problem locally 
    3. Record each action taken until the problem occurs
    4. Repeat until its consistent
    Luckily, that actually didn't take too long. I was able to isolate it to about 4 or 5 manual steps. The next part was to automate this process. Since it had no obvious starting point in the code, it was a good candidate for a UI test with Selenium WebDriver.

    As with the previous post, we make heavy use of PageObjects to abstract the details of the markup and what-not from the test case:

    package http.voting;
    
    import static org.junit.Assert.assertEquals;
    import http.SeleniumTest;
    
    import org.junit.Test;
    
    import pages.IndividualRecording.IndividualRecordingPage;
    import pages.song.SongPage;
    import pages.voting.RecommendPanel;
    
    public class TestUpvote extends SeleniumTest
    {
        protected boolean needsDatabaseReset()
        {
            return true; 
        }
        
        @Test
        public void reproduceErrorForUserActivityAndDuplicateKeys()
        {
            // go to recording 8 
            IndividualRecordingPage recording8 = new IndividualRecordingPage(driver, 8);        
            driver.get(recording8.getURL());
            
            // upvote it
            recording8.focus(); 
            RecommendPanel votingPanel = new RecommendPanel(driver);         
            assertEquals(RecommendPanel.VotingResult.SUCCESS,  votingPanel.upvote());
            
            // go to song page 
            SongPage song257 = recording8.gotoSong(257);
            // upvote song twice
            assertEquals(RecommendPanel.VotingResult.SUCCESS,  votingPanel.upvote());
            assertEquals(RecommendPanel.VotingResult.ALREADY_VOTED, votingPanel.upvote()); 
    
            // go to "July 8th, 1978 - Roslyn etc.." 
            song257.gotoAssociatedRecording(26);
            // upvote (didn't work in bug)
            assertEquals(RecommendPanel.VotingResult.SUCCESS, votingPanel.upvote());         
        }
    }
    

    Minor note: the needsDatabaseReset() method tells the SeleniumTest class to restore the database to a fresh state each time a test is run.

    You can see the test case deals with specific actions such as "upvote" and "go to song" instead of searching the website for different elements to click. This is all handled by the different page objects. For example, here is the relevant section of the RecommendPanel class:

         public VotingResult upvote()
        {
            final int currentVoteCount = getCurrentVoteScore(); 
            getUpvoteButton().click(); 
            
            try
            {
                // success if the votes increase or the "error box" pops up -- although we still need to check it 
                (new WebDriverWait(source, 5)).until(new ExpectedCondition<Boolean>() {
                    public Boolean apply(WebDriver d) {
                        return getCurrentVoteScore() > currentVoteCount || getVisibleErrorBox() != null ; 
                    }
                });
                
                int newVoteScore = getCurrentVoteScore(); 
                
                if(newVoteScore > currentVoteCount) 
                    return VotingResult.SUCCESS;             
                else if(getVisibleErrorBox() != null && getVisibleErrorBox().getText().contains("You already voted"))
                    return VotingResult.ALREADY_VOTED; 
                else
                    return VotingResult.ERROR; 
            }
            catch(Exception exc)
            {
                exc.printStackTrace();
                return VotingResult.ERROR; 
            }
            
        }
    

    Here, we're simply clicking the "recommend" button and ensuring that the result is either an increase in the current vote score, a popup saying "you already voted," or an error message.

    With this test, the problem is now easily reproducible. The actual problem is very unexciting. We log user actions by IP address to decrease the chance of duplicate voting. The table we use is mapped to multiple classes with the id "generator" set to increment. This caused duplicate primary keys to be generated when new entries were added. Changing the generator to "identity" fixed the problem.

    In contrast to the previous bug, this one was hard to reproduce but an easy fix. Stay tuned for more bugs that are both irreproducible and hard to fix!*

    And for reference, the PageObjects used for this test:

    IndividualRecordingPage
    SongPage
    RecommendPanel

    * hopefully not

    Wednesday, September 14, 2011

    The PageObject pattern for Selenium WebDriver UI tests

    There was a bug recently encountered while browsing the graphs page. Clicking on two of the canned graph options yielded this friendly result:

    Woops. So at that moment I was also experimenting with Selenium WebDriver for automating some UI tests. So I figured "hey, why not reproduce the problem with some of these tests before fixing it."


    At first I just stuck the Selenium test code in each test case, but after awhile I refactored to use the page object pattern. Basically you have a class that represents some part of the page (in this case, GraphsPage) which serves as an interface the the component's services, such as "generate graph." in our case.



    So before the first step, it's time to define a base class called PageObject which we'll subclass for the graphs page. At this point I'll also note that Selenium seems to have some kind of support integrated called a PageFactory that you use on your page object class. In the near future I might refactor them again to use that.

    But anyway, here's the PageObject:
    package pages;
    
    import org.openqa.selenium.By;
    import org.openqa.selenium.WebDriver;
    import org.openqa.selenium.WebElement;
    
    public abstract class PageObject 
    {
        protected WebDriver source; 
        
        protected PageObject(WebDriver source)
        {
            this.source = source; 
        }
                    
        public abstract String getURL(); 
        
        public abstract void focus(); 
    }
    

    Nothing too exciting here, it's just some boiler plate.

    On to the fun part, the GraphsPage class:

    package pages.graphs;
    
    import org.openqa.selenium.By;
    import org.openqa.selenium.WebDriver;
    import org.openqa.selenium.WebElement;
    import org.openqa.selenium.support.ui.ExpectedCondition;
    import org.openqa.selenium.support.ui.Select;
    import org.openqa.selenium.support.ui.WebDriverWait;
    
    import pages.PageObject;
    
    public class GraphsPage extends PageObject
    {
        private String GRAPH_URL = "graphs"; 
        
        public GraphsPage(WebDriver driver)
        {
            super(driver); 
        }
        
        private WebElement getGeneratedURLLabel()
        {
            return source.findElement(By.cssSelector("#graphpanel #graphlink")); 
        }
        
        private  Select getCannedGraphsList()
        {
            return new Select(source.findElement(By.cssSelector("#cannedgraphs select")));
        }
        
        private  WebElement getGenerateGraphsButton()
        {
            return source.findElement(By.className("getgraph"));
        }
        
        public String getURL()
        {
            return GRAPH_URL; 
        }
        
        public boolean generateGraph()
        {
            getGeneratedURLLabel().clear(); 
            getGenerateGraphsButton().click();  
            
            try
            {            
                (new WebDriverWait(source, 5)).until(new ExpectedCondition<Boolean>() {
                    public Boolean apply(WebDriver d) {
                        return getGeneratedURLLabel().getText().contains("?"); 
                    }
                });
                
                return true; 
            }
            catch(Exception exc)
            {
                return false; 
            }
        }
        
        public void selectCannedGraph(String cannedGraphTitle)
        {
            Select cannedGraphs = getCannedGraphsList();         
            cannedGraphs.selectByVisibleText(cannedGraphTitle); 
            cannedGraphs.getFirstSelectedOption().click(); 
        }
    
        @Override
        public void focus() 
        {
            getGeneratedURLLabel().click(); 
        }
    }
    

    Alright, now for the explanation. First, I'd like to explain the presence of the focus method. It seems in my experience that using FireFox 6 and IE, the click() method would sometimes fail to work unless you focused the page the first time. So that's what that does.

    Here are the service methods:
    • generateGraph(): clicks the "generate graph" button, and returns true if the graph was retrieved successfully. This is done by checking for the presence of a generated URL for that specific graph. 
    • selectCannedGraph(): selects the canned graph on the canned graph list based on it's title.
    You also see a few private methods, those are there mainly to abstract references to specific HTML elements and only have one reference to them.

    Now we can write a test case to reproduce the problem. The problem occurs when you select the canned graph "Most played in cities," so let's write one to reproduce that:

    package http.graphs;
    
    import static org.junit.Assert.assertTrue;
    import http.SeleniumTest;
    
    import org.junit.Test;
    
    import pages.graphs.GraphsPage;
    
    public class TestGraphs extends SeleniumTest
    {
        
        @Test
        public void testMostPlayedInCities()
        {
            testCannedGraph_helper("Most played-in cities");        
        }
        
        private void testCannedGraph_helper(String graphToCheck)
        {
            final GraphsPage page = new GraphsPage(driver);         
            driver.get(super.getPage(page.getURL())); 
            
            page.selectCannedGraph(graphToCheck); 
            assertTrue(page.generateGraph()); 
        }        
    }
    
    

    Nothing too exciting, you can see I've added an additional helper method so we can test other canned graphs as well. All we do here is select the canned graph, generate it, and assert that the graph was generated correctly. If you run it, it will fail.

    One last thing, you can see that this test class extends SeleniumTest. That's just a boiler plate class, it looks like this:

    package http;
    
    import org.junit.After;
    import org.junit.Before;
    import org.openqa.selenium.WebDriver;
    import org.openqa.selenium.firefox.FirefoxDriver;
    
    public class SeleniumTest 
    {
        protected WebDriver driver;
        protected final String ROOT = "http://localhost:8080/Recordings/"; 
        
        protected WebDriver getNewDriver()
        {
            return new FirefoxDriver(); 
        }
        
        @Before
        public void Before()
        {
            driver = getNewDriver(); 
        }
        
        @After
        public void After()
        {
            driver.quit(); 
        }
        
        public String getPage(String page)
        {
            return ROOT + page; 
        }
    }
    

    It just initializes the Firefox drvier and restarts it after each test. Alright, so next time I'll show you what the actual problem is and how it was fixed. Until then, take care!

    Saturday, September 10, 2011

    The joys of automating your build and deployment

    Hi everyone,

    Alright, so you know like everyone I'm sure (if not, start) I like to browse stackoverflow and it's more subjective version programmers. Take a look at this question about the benefits of nightly builds for a one man team.

    Having gone all these years on my own projects without a real build server setup -- I can say that I would never hesitate to use one for a non-trivial project, whether it was just me or N other people. Now the post above is about "nightly builds," but the precursor to that is an automated build. After that, you can run it nightly or by the hour -- there's no difference.

    For me, using a tool like eg. Hudson makes your life a lot more easier. It guarantees you have a "working", reproducible copy of your project and if you want ready for deployment. It allows you to do other fun things like run your unit/integration tests on SCM changes (because you will forget!), run static code analysis tools and other metrics, and see trends over time.

    We (my friend and I working on Recordings) have a separate Amazon EC2 instance running Hudson 24/7 just for that purpose. Our project is (relatively) tiny and there's only two of us. We've spent some hours setting it up -- and I wouldn't hesitate to do it again. We have it doing all kinds of stuff, such as:
    • Monitoring our dev branch for changes; building and running unit tests whenever a change is detected
    • Monitoring our production branch for merges (which have been previously tested); building, running unit and integration tests, minifying js/css code, and packaging the project into a WAR file for deployment. 
    • Jobs to synchronize our local test databases with the production database
    • A job that polls the website every so often to make sure it's still online
    • A job that runs findbugs and compiles tasks (ie  TODOs) in source code
    • Lastly: a job that deploys the last successful build from production to our production server
     We use this Hudson tray tracker tool to see the status of everything at a given time, so we always know what's going on.

    This allows us to be as lazy as possible, because the last thing you want to do is waste your time manually going through the build process when you're trying to fix bugs or merge new features!

    And finally, as I said I find this useful even for solo projects that still have actual users* (other than yourself). Having your CI server able to build your application, create installers/docs/etc., and generate a setup file ready for distribution is very handy and will save you lots of time.


    * I've used Hudson for the Recordings java webapp and some smaller C# desktop applications.

    Tuesday, September 6, 2011

    How transparently can you implement open session in view late in development?

    On the Recordings project, we set up hibernate to eagerly load everything. I'm not sure why we made that decision (it was 8 months ago) but we've been busy getting rid of that practice. When deciding how to handle the lazy loading problem, we found a few different solutions:

    1.  Create different presentation objects for different pages and eagerly load everything into those -- pages that only need certain properties would get a class only containing those properties. 
    2. Add flags to our data access methods that indicate what to fetch
    3. Keep the db connection open while the view is being rendered (known as the Open Session in View pattern)
    Approach (1) seems like a sound approach (and we've done it before in other projects) but would add too much additional maintenance and work this late in the development, as we're trying to restrict ourself to bug fixes and performance enhancements as necessary.

    Approach (2) really seems ugly because not only does the data layer get convoluted -- it exposes the idea of "lazy loading" to other layers that shouldn't care about it, since they now have to decide what items they need.

    So we've taken approach #3. We've implemented it by using a servlet filter that runs before any servlets. It simply opens a new session and then closes the session after the request is completed. One drawback is that if anything goes wrong with the database, the code that's responsible isn't shown in the stack trace. Instead, it shows the servlet filter as the cause which is not helpful.

    A good (though painful) thing is that it's forced us to consolidate all of our data access code into one class. We have more than one class in our data layer, but they all extend one class (called BasicHibernateDataSource) which handles acquiring the session object. Our db tests also failed after doing this until we set up a basic database test class that mimics the servlet filter.

    Although it seems like a lot, I think the effort required to implement it has still been "less" effort than the other two approaches, or at least the most transparent. This has allowed us to avoid introducing many new bugs due to adding a bunch of different functionality a this stage -- we're trying to push on so we can get it live as soon as possible.

    Sunday, August 21, 2011

    Let's use hudson as our CI server on Amazon EC2

    Hey everyone,

    Since we're both lazy, we don't want to build our project manually. So we've been using Hudson as our CI server. Over the past few weeks we've been working on the build process and it's coming along pretty nicely. Here's our situation:
    • 2 person "distributed" team
    • Can't setup a physical server ourselves (lack of equipment and connection guarantees)
    • Need remote access so we can both use it
    Well, after searching around we decided to settle on an Amazon EC2 instance. The biggest reason is that you can try a micro instance free* for a year. That allows us to see how it performs in the meantime.

    So we've set it up, and it's pretty decent so far. We have a ubuntu instance deployed running Hudson, with our jobs polling Mercurial for changes and building the project.

    A couple things I've run into:
    • Don't install ant via apt-get. Don't ask me why (maybe I'm doing something wrong), but when I install via the package manager it doesn't get all the required libraries -- it seems to be missing some tasks.
    • CPU Usage - During the running of our unit tests CPU usage occasionally goes 'past' 100%. This really slows down the tests some times. Our tests that hit the db take about 2 minutes normally but took 14 minutes at one point.
    I guess the problem might be solved by upgrading the instance, but it's not a big problem for us. Since the db tests only run on production builds (which run nightly) speed isn't vital for them. Our regular unit tests which run when the Test branch is changed run pretty fast.

    And finally, I'd like to give a shout out to this hudson tray tracker application. It puts an icon on your system tray that shows the status of your hudson jobs. You can also bring up a window to run builds or view the output.
    
    Hudson Tray Tracker.
     It's pretty convenient, although I wish it had TFS style build notifications where you get a pop up when every a build is started or completed. Overall though I can't complain.

    Next time I will share the shell script we use to set up hudson on our instance -- this includes installing ant, tomcat, mysql, mercurial, and a couple other things. But we really want that process automated.







    * actually, it seems I currently owe $0.31 for I/O requests. I still think it's worth it...

    Wednesday, August 17, 2011

    Back in school

    Hey guys,

    Apologies for the delay. I'm back in school and things have been pretty hectic lately. We're still working on deploying the web app, but in the mean time I've been working with Clojure both on the JVM and the CLR. So expect something related to that at some point. I'm still not done unpacking!


    Thursday, July 21, 2011

    Recordings is moving on!

    Hey everyone -- this is going to be a short post. I plan on posting a few more things related to jqPlot (our favorite plotting library) hopefully in the next week but in the mean time...

    My colleague is finishing up our last feature complete task and I've been moving to our next stage: setting up the build system. Right now I have Hudson building our main branches, running tests, findbugs, etc. and packaging it all into a nice war file. The next step is to set up an Amazon EC2 micro instance to do this, as I've been doing it on a local VM with a Ubuntu image.

    In the meantime, here are some screenshots of our project.

    The main data page, where you can browse/filter/search the database

    The graphs page (my favorite!)

    The recording detail page, where you can comment and recommend it
    And remember ... our user interface changes a lot since we're always tweaking it, so don't consider anything final!

    Sunday, July 10, 2011

    Choosing a text scroller plugin for the side bar

    So a few quick notes: We are slightly behind on our feature complete milestone because we decided to add a new feature. But we should be good within a week or so.

    In the meantime, I've been working on implementing a "recent changes" text scroller on the left side of our website. This will show the most recent comment posts, recording/song additions, and any additions we make as well. I'm personally not doing the data side of this (although I might if we get time constrained) so my job is just the front-end.

    Naturally I'm very lazy, so I prefer a pre-existing solution to making blocks of text scroll vertically. I like javascript and jQuery, but I don't want to spend too much time writing something like that. So I went looking for such a plugin.

    And after looking at a good dozen or so different plugins, which for one reason or another I couldn't get to work with our text blocks, I finally settled on one called SerialScroll. Here's a list of features relevant to us:
    Recent Changes Side-bar
    1. Vertical scrolling
    2. Support for arbitrary scrollable items specified with CSS selectors
    3. Auto scrolling
    4. ~4 kb  total size

    Starting from the "News Ticker" demo at the demo page, I was able to customize it for our page and retrieve its content via AJAX.

    I realize it's not too exciting since you can't see it scrolling, but just trust me that it does, in fact, scroll. Although we might change the look, this looks like the plugin we'll stick with.

    Oh and for reference, here's our javascript:

     $('#changelist').load('recentchanges .changeitem',function()
     {  
      $('#changelist').serialScroll({
     
       force: true,
       axis: 'y',
       items:'.changeitem',
       interval: 2000,
       easing:'linear',
       lazy: true,
       step:1,
       start:0,
       cycle:true,
       jump: true,
       event: 'click'
      }); 
     }); 
    

    Sunday, June 19, 2011

    Writing and testing a 2007 MS Word Add-in using NUnit, Moq, and the MVP pattern

    Hey everyone,

    Today I want to show you an approach to writing a testable UI. To do this I'm going to write a 2007 Microsoft Word Add-in using Microsoft's Visual Studio Tools for Office. The goal is to write an application level add-in and test it using NUnit and Moq.

     I'm going to design it around the Model-View-Presenter pattern: We'll have two interfaces: An IView, which will be an Office ribbon and contain no logic. Then we'll  have an IPresenter, which will handle all of the "UI logic" such as taking actions based on user dialog results etc.

    Our Presenter won't actually have any domain logic in it: we'll be delegating that to another class (I'll explain that in a minute). This means that when we test it, we'll be doing behavior testing instead of testing for object states. This also means we'll be using Moq to setup our Mock domain objects and verifying that calls are made to them.

    Of course it's important to note that this kind of testing involves some coupling between your tests and the implementation of the class under test, but I think it's acceptable since the Presenter is behavior based.

    So what kind of add-in will we be doing? Well at first I was going to be boring and use a very contrived example. But then I came up with a nice domain-specific idea.

    I figured why not do something chemistry related? That's always fun. So we're going to make an addition to the Word ribbon that allows the user to enter either a chemical formula or an entire reaction equation. When the user clicks the "Action" button, the add-in will either render the equation as an image and insert it, or if the user enters a single compound it will insert the molar mass at the cursor position.

    Here's the user interface:


    Don't worry about the actual logic behind rendering and molar mass calculation. I've already taken care of that using my chemical database program I wrote last year. Now of course I think the code is crap, but it still works and that's the main thing.

    So let's get started by defining our interfaces. Our IPresenter will only contain a single method: Action() which returns void and expects no parameters. Our Ribbon will implement IView. To start with, we'll give IView two properties: MolarMass and EquationText:


    IView.cs:
    namespace Chemistry.UI
    {
        public interface IView
        {
            double MolarMass
            {
                set;
            }
            string EquationText
            {
                get; 
            }
        }
    }
    
    IPresenter.cs:
    namespace Chemistry.UI
    {
        public interface IPresenter
        {
            void Action(); 
        }
    }
    

    In our first iteration we'll simply assume that the user-entered text is a compound and we want to calculate the molar mass of it. Let's write a simple test that will confirm our IPresenter implementation makes a call to our domain object (an instance of IFormulaParser):

    Presenter.cs:
    namespace Chemistry.UI
    {
        public class Presenter : IPresenter
        {
            public void Action()
            {
                throw new NotImplementedException();
            }
        }
    }
    
    IFormulaParser.cs:
    namespace Chemistry.UI
    {
        public interface IFormulaParser
        {
            double MolarMass(string compound); 
        }
    }
    
    And now our test file, PresenterTests.cs:

    using NUnit.Framework;
    using Moq;
    using Chemistry.UI; 
    namespace Tests
    {
        public class PresenterTests
        {
            private IPresenter _presenter;
            private Mock<IFormulaParser> _formulaParser; 
    
            [SetUp] 
            public void SetUp()
            {
                _presenter = new Presenter();
                _formulaParser = new Mock<IFormulaParser>();            
            }
    
            [TestCase] 
            public void Action_UserEntersCompound_CallsMolarMassMethod()
            {
                _presenter.Action(); 
                _formulaParser.Verify(parser => parser.MolarMass("O2")); 
            }
        }
    }
    

    So in our test class we initialize a new Presenter object and a mock FormulaParser. For now we've set the IFormulaPresenter to return a value of 32 when MolarMass("O2") is called. In our test case, we make sure that our FormulaParser is called correctly. Of course the test fails since Presenter doesn't do anything. Let's change it so that it uses the text entered by the user instead our hard-coded value.

    using NUnit.Framework;
    using Moq;
    using Chemistry.UI; 
    namespace Tests
    {
        public class PresenterTests
        {
            private IPresenter _presenter;
            private Mock<IFormulaParser> _formulaParser;
            private Mock<IView> _view; 
    
            [SetUp] 
            public void SetUp()
            {
                _presenter = new Presenter();
                _formulaParser = new Mock<IFormulaParser>();            
                _view = new Mock<IView>(); 
            }
    
            [TestCase("O2",32.0)] 
            [TestCase("H2", 2.0)]
            [TestCase("C6H6",78.0)]
            public void Action_UserEntersCompound_CallsMolarMassMethod(string compound, double molarMass)
            {
                _view.Setup(view => view.EquationText).Returns(compound);
                _formulaParser.Setup(parser => parser.MolarMass(compound)).Returns(molarMass);                 
    
                _presenter.Action(); 
    
                _formulaParser.Verify(parser => parser.MolarMass(compound)); 
            }
        }
    }
    


    Now we've done a couple things. First, we've created a new mock IView. We also changed our test to accept parameters and set up the view's EquationText property with the compound string. Finally, we check again to make sure the presenter calls the FormulaParser.MolarMass method. Now let's make the Presenter pass the test.

    public class Presenter : IPresenter
    {
     private IFormulaParser _formulaParser;
     private IView _view;
    
     public Presenter(IView view)
     {
      _view = view;
      _formulaParser = new FormulaParser(); 
     }
    
     public void Action()
     {
      _view.MolarMass = _formulaParser.MolarMass(_view.EquationText); 
     }
    }
    

    Alright, so now the Presenter is using the IView's equation text property, calculating the molar mass, and setting the IView's MolarMass property to the result. Let's also update the test cases to verify that View's MolarMass property is also set:

    [SetUp] 
    public void SetUp()
    {
     _view = new Mock<IView>();
     _formulaParser = new Mock<IFormulaParser>();                  
     _presenter = new Presenter(_formulaParser.Object, _view.Object);                  
    }
    
    [TestCase("O2",32.0)] 
    [TestCase("H2", 2.0)]
    [TestCase("C6H6",78.0)]
    public void Action_UserEntersCompound_CallsMolarMassMethod(string compound, double molarMass)
    {
     _view.Setup(view => view.EquationText).Returns(compound);
     _formulaParser.Setup(parser => parser.MolarMass(compound)).Returns(molarMass);  
                  
     _presenter.Action(); 
    
     _formulaParser.Verify(parser => parser.MolarMass(compound));
     _view.VerifySet(view => view.MolarMass = molarMass); 
    }
    

    At this point, our test passes. And at this point if we implement IView in our Ribbon object, everything should work. So let's proceed and implement rendering equations as well. Let's start by writing a test to make sure our rendering object is called. To do thish we need to add our IEquationRenderer interface, which has a reference to System.Drawing:

    public interface IEquationRenderer
    {
     Image RenderEquation(string equation); 
    }
    

    And modify the Presenter constructor:

    public Presenter(IEquationRenderer equationRenderer, IFormulaParser formulaParser,  IView view)
    {
     _view = view;
     _formulaParser = formulaParser;
     _equationRenderer = equationRenderer; 
    }
    

    Finally add our test case:

    [TestCase("H2 + O2 ----> H2O")]
    public void Action_UserEntersEquation_CallsEquationRenderer(string equation)
    {                        
     _view.Setup(view => view.EquationText).Returns(equation);            
     _presenter.Action();
     _equationRenderer.Verify(renderer => renderer.RenderEquation(equation));
     _view.VerifySet(view => view.EquationImage = It.IsAny<Image>()); 
    }
    

    An important note here: I wasn't able to mock the Image interface that we're passing around, so we're verifying that the IView gets its EquationImage property set to "any" Image object instead of a dummy Image that we would pass around normally.

    If we make this test pass by just adding a call to our EquationRenderer, we could also still be calling the molar mass calculation which we don't want. So let's add another test that ensures that when we parse an equation, we don't try a molar mass calculation:

    [TestCase("H2 + O2 ----> H2O")]
    public void Action_UserEntersEquation_DoesNotTryMolarMassCalculation(string equation)
    {
     _view.Setup(view => view.EquationText).Returns(equation);
     _presenter.Action();
     _formulaParser.Verify(parser => parser.MolarMass(It.IsAny<string>()), Times.Never()); 
    }
    

    This test will fail because we need some conditional logic to differentiate compounds and equations. Let's do this by adding a method to IFormulaParser:

    bool IsEquation(string text); 
    

    Let's set this up in our equation tests to return true by adding this setup line to our two test cases:

    _formulaParser.Setup(parser => parser.IsEquation(equation)).Returns(true); 
    

    The actual implementation will just check for the presence of "---->," but that could always change. Now we can modify the Presenter to take this into account:

    public void Action()
    {
     if (_formulaParser.IsEquation(_view.EquationText))
     {
      _view.EquationImage = _equationRenderer.RenderEquation(_view.EquationText); 
     }
     else
     {
      _view.MolarMass = _formulaParser.MolarMass(_view.EquationText);
     }            
    }
    

    And now our tests pass. At this point, the rest of the work is implementation details. Here's an example of the final result:

    The important thing to note with this approach is that we've created an extremely testable UI layer. Since all of the logic is in the Presenter class, we can test all aspects of the UI. This becomes even more exciting when your UI logic involves the user making decisions by answering prompts and responding to dialogs. You could for example wrap a call to MessageBox.Show in an IView DisplayMessage() method that returns a DialogResult. This would allow you to test the Presenter and just mock the IView.

    Anyway, just for reference, the full source code of this add-in is on my website. Apologies in advance for the weird project structure, the Test project is under the "Tests" folder. The solution uses Visual Studio 2010.

    Sunday, June 12, 2011

    Updates, progress, what's coming up...

    Hey everyone. Not much for you today. I would like to say what kind of topics will be appearing on here in the next few weeks though.

    First: I've been working on an MS Office add-in in C# and would like to share my experience unit testing it with NUnit and Moq.

    Then, the Recordings project is moving along well. We're on schedule to have all "core features" in by the end of the month. At that point we won't be making any new feature additions -- just working on testing/bug fixes/etc. and also the build process.

    We've decided to set up a build server using Hudson to build/test/deploy our web application -- so I'm sure of course there will be plenty of problems that will arise for us to resolve. I for one am pretty excited about automation -- we'll be doing a clean check out from our production branch in Mercurial, running our unit/integration tests, minifying our javascript/css, and I'm sure all kinds of fun tasks. By the end I'll write about our build process (for reference -- and if anyone finds it useful that's always good).

    Apologies for nothing more concrete (ie: some code or a problem or something).But hey, I'm sure some stuff will come up!

    Sunday, June 5, 2011

    Labeling the top 5 results on a graph with jqPlot

    Hey everyone,


    As you can guess by the title, we're back with more on jqPlot! Last time I talked about jqPlot I had wanted to use tick marks in the Highlighter plugin for axes where the data was non-numerical. The whole reason of using the highlighter in the first place was because some of our graphs have large numbers of data points, so we have to hide the axes labels or it looks terrible.

    A large data set (129 points). 
    So we hide the axes labels and it looks better, but now you can't tell what any of the points are. We added the highlighter plugin so that at least when you hover over a data point, you see its value. That's helpful, but what if you wanted to, for example, print this graph out?

    We decided to label the graph for the points with the top X (where X=5 at the moment) y-values on the graph. In the end we get this:

    Labeling the top 5 results looks pretty nice. 
    Now if you print it out, at least you see the most important values on the graph.

    The Plugin

    So in order to accomplish this I started from an existing plugin (Highlighter since that's what I'm familiar with). I'm going to explain how I changed the highlighter plugin to accomplish what I needed, but you don't have to start with it at all. There are a couple things we have to change to make it work:

    • Modify the postPlotDraw hook so that it sets up labels for the number of points we want to draw the label for.
    • Make a call to our own function that will determine which points to draw labels
    • Modify the showTooltip function so that it uses more than one label 
    None of these are really hard, so let's just start with the first step.

    The first thing is to add two new variables to the plugin object itself:

    $.jqplot.TopResultDisplay = function(options) {
            this._tooltipElements = []; 
            this.numberOfPoints = 5; 
    
            /*   the rest of the options .... */ 
    };
    

    TopResultDisplay is the name of the plugin. _tooltipElements is empty right now but will contain a number of div elements that represent the labels. numberOfPoints determines how many top points we want to label.

    Now we need to modify the postPlotDraw() function so that it creates those divs. It will also make a call to our function that will eventually draw the labels. After modifying it, the postPlotDraw() function looks like this:

    // called within context of plot
        // create a canvas which we can draw on.
        // insert it before the eventCanvas, so eventCanvas will still capture events.
        $.jqplot.TopResultDisplay.postPlotDraw = function() {
            this.plugins.TopResultDisplay.highlightCanvas = new $.jqplot.GenericCanvas();
            
            this.eventCanvas._elem.before(this.plugins.TopResultDisplay.highlightCanvas.createElement(this._gridPadding, 'jqplot-highlight-canvas', this._plotDimensions));
            this.plugins.TopResultDisplay.highlightCanvas.setContext();
            
            var p = this.plugins.TopResultDisplay;
            
            for(i =0;i<p.numberOfPoints;i++)
            {
                elem = $('<div class="jqplot-highlighter-tooltip" style="position:absolute;display:none"></div>');
                p._tooltipElements[i] = elem;
                this.eventCanvas._elem.before(elem);
            }        
            
            drawLabelsForPoints(this); 
        }; 

    The changes I made start at line 10: we simply create a number of div elements and insert them on the _tooltipElements array.

    Then we add a call to drawLabelsForPoints(this) which will be our label drawing function that takes in a plot object.

    Now let's write that function. When I was changing the Highlighter plugin this function was originally handleMove, but I ended up changing all of the functionality. The function itself is not that exciting:

    function drawLabelsForPoints(plot) 
        {
            var hl = plot.plugins.TopResultDisplay; 
            
            var sortedData = []; 
                    
            for(i =0;i<plot.series[0].data.length;i++)
            {            
                var newPoint = new Object(); 
                newPoint.data = plot.series[0].data[i];
                newPoint.seriesIndex = 0; 
                newPoint.gridData = plot.series[0].gridData[i];             
                            
                sortedData.push(newPoint);                                        
            }        
            
            sortedData.sort(function(a, b) { return b.data[1] - a.data[1]; }); 
                    
            for(i=0;i<hl.numberOfPoints;i++)
            {            
                showTooltip(plot, plot.series[0], sortedData[i]);
            }    
        }
    

    First we create a new array called sortedData. Then we fill this array with all of our unsorted points, but we use the format that showTooltip of Highlighter expects. The data variable refers to the actual data array of the form [x,y]. seriesIndex is the series that the point is on, and gridData refers to the points x/y pixel coordinates on the canvas. You might notice that I'm only using one series here since all of our graphs have only one series. You could however modify it to support more than one if needed.

    After filling the array, we sort in place using a function that sorts the data in descending order based on the y-value. Finally, we call showTooltip on the top results with the number of points that was specified.

    The next step is to modify showTooltip to use more than one label element. To do this I changed a small part of the showTooltip function at the top to look like this:

        var hl = plot.plugins.TopResultDisplay;
            
            if(!hl.enabled) return;        
            var elem = hl._tooltipElements.splice(0, 1)[0];         
            if(!elem) return; 
            
            if (hl.useAxesFormatters) {
            /* .. . the rest of the function */
    

    The important line is line 4, where we make the call to _tooltipElements.splice. This removes one element from the array of divs and returns it, which means that as long as it returns non-null we still have labels to use. After that the rest of the code remains the same and references the elem object to draw on.

    The very last step is to remove the line

    $.jqplot.eventListenerHooks.push(['jqplotMouseMove', handleMove]);
    
    from the top of the code. This line wires up the handleMove method to be called whenever the mouse is moved over the canvas.

    After that, we're done with changing the code. All that's left is to enable the plugin. We do that just as any other plugin. Below is the configuration for the plugin as we have it on our graphs page.

    TopResultDisplay:
    {
     tooltipAxes: 'xy',
     useAxesFormatters:true,
     tooltipLocation: 'n', 
     numberOfPoints: 5
    }, 

    And the results again for reference:

    So that's it! If you have any questions/comments/etc. feel free to leave a comment.

    Monday, May 30, 2011

    Fun with Hibernate: Retrieving an object that should have no references

    So I think I'm just going to call this a series, since there seem to be a lot of episodes of me and Hibernate not getting along.

    This particular problem arose a couple weeks ago. It turned out that when you were on the website and clicked on a recording to bring up its detail page, a NullPointerException would be thrown. Great! There wasn't an obvious solution so I just logged it in our issue tracker (issue #68)

    Then it looked like it only caused a problem with a specific recording (with id=4). Even more exciting was that the spot where the exception occurred was completely unrelated to the cause. Instead, it was being generated in the data layer when the query was made. Well that's never good.

    Eventually we found that our Recording object, which contained a List of SongInstances, was causing the problem. One of the SongInstance objects was null. The weird part was that each SongInstance had a foreign key to a Recording. Since no-one had a reference to SongInstance, deleting a SongInstance should not have impacted anything.

    Well, turns out the problem was caused by two things. First, one of our 'integration' tests tested deletion from the database (oops) and we don't have a test database yet. So it deleted one of the entries from Recording #4. The other problem was that our hibernate mapping Recording contained the following entry for mapping a list of SongInstances:

    <list-index column="trackListing" base="1" />

    Which caused Hibernate to retrieve the SongInstances sequentially by their "track listing." And when the test ran, it deleted the song instance that was set as track #2, causing a null pointer when Hibernate retrieved it. Oops.

    So, our problem wasn't really a bug, but a side-effect of our test needing a real test database. Just one more thing to learn I guess.

    In other news: we are setting a 'deadline' for feature-completeness by the end of June. Our schedule is progressing pretty well and we should be ready for The Next Step at that time. Yay!

    See you next time.

    Friday, May 20, 2011

    Unknown Entity in Hibernate and other things

    Hey everyone,

    Let me get this out of the way first: I made a really careless mistake while writing the hibernate mapping files the other day (probably because I should have gone to bed) so I'm going to share it with you for my reference and so you can laugh at me.

    What happened was that I had added a mapping for a class (RecordingCommentUserActivity.hbm.xml) and forgot to add a new entry to the hibernate.cfg.xml file. Oops. One of the side-effects was that Hibernate complained about requiring the fully qualified class name, which should have clued me in because you shouldn't ever need it.

    The other thing that tripped me up was that a simple select query (using the full path) did not throw any errors in the log (that I saw). So I thought there was something wrong in another aspect of the project.

    Of course when I tried to do something real like persist an object to the database, then it complained, giving one of those fun "Unknown Entity" exceptions.

    As with a lot of problems, the hardest part was figuring out what was wrong. The fix took a few seconds. And everyone was happy.

    In even more happy news, our updated security logic is nearly complete. We now track user activity and prevent duplicate voting of comments/songs/recordings/etc. I've been doing this off of another head on my branch so I can keep my main branch head free of the changes until they are more scrutinized. In the next few days I'll be switching away from security and finishing up an unimplemented voting feature for songs. That's good, because we allocated 1 week for the task on our estimates.

    See you next time with more updates.

    Tuesday, May 17, 2011

    New directions

    Hey everyone,

    I'm out of school (for a bit) and working at a C# shop for the summer, so you'll probably see some C# issues crop up, although I'm still doing Java web development on the side. I might be rolling out some small WCF-based projects to learn. Maybe rewrite my chemical database project using it. That could be fun.

    As for the Recordings project, we came up with a "feature complete" schedule which lists all of the tasks needing completion before we could consider pushing to production. Looks like we still have a few more months left (although we did make pessimistic estimates ... isn't that what you have to do though...)

    As for myself, I'm focusing on some revamped security logic. I think I mentioned our very simplified security model in a post about Mockito, but we decided that it needed to be expanded. For example, our users are able to like/dislike (or vote up/down if you want) songs, recordings, and comments on both of those. Before we just limited the user to a certain number of votes per day, but they could still vote more than once.

    This time, we're making it so that we track user activity so they can't do that until we delete the user activity entries (every X days or so). This functionality is almost done, in fact, the logic is all there. All that's missing is the Hibernate mappings and the actual data access. Right now the data layer is basically stubbed out. Mostly because writing Hibernate mapping files and data classes is very tedious and not as fun as the rest of it. But we'll get to it eventually.


    Stay tune for more exciting programming related updates in the Near Future...

    Tuesday, May 10, 2011

    Updates.

    Hey guys,

    It's been a week, and I'm still going through finals. In fact, I have a physical chemistry test that's going to kill me. Expect more updates next week.

    Tuesday, May 3, 2011

    AJAX and being lazy with tablesorter

    Now I love using other peoples' code.

    I'm extremely lazy in terms of programming: if someone's written it, I'm going to use it. In fact, the first thing I do (except when I'm reinventing the wheel for learning purposes) is check to see if there's a library for problem X.

    An example: we're using the DataTables jQuery plugin to make our tables look pretty. Now, data tables has some nice features:
    • Pagination 
    • Sorting
    • Searching (which updates the results instantly)
    • Easy to integrate (just a one liner) 
    All these things make DataTables very attractive. So what's the problem? Well, recently we've been ajax-ifying one of our pages so that it's constantly reloading the same table. For some reason, DataTables doesn't like this (at least with our setup, if it's working for you feel free to share). The end-result is that we lose sorting ability when the AJAX reloads the table. That's a problem.

    After spending some time looking for some solutions, we decided to drop DataTables in favor of tablesorter, another jQuery table plugin.

    We decided to use tablesorter over DataTables for a couple reasons:
    • It wasn't working with our AJAX setup (as mentioned)
    • At first we were using the other features, but then we stopped using pagination and searching
    • tablesorter integrates just as easily
    • tablesorter is more light-weight (12kb vs 60kb) 
    • tablesorter works for our purposes with no exciting changes (we add a line in our success function to refresh the table) 
    So for our purposes, tablesorter was the better option. And the best part is, it let us be as lazy as possible since integrating it took exactly three steps*:
    • download minified js file
    • add to project
    • add two lines of code to existing script files
    Very convenient.


    * apologies for the multiple bulleted lists, I do kind of like them though.

    Tuesday, April 26, 2011

    More servlet testing with Mockito and JUnit

    Hello everyone,

    I had some time to implement some more functionality for The Project today, so I thought I'd share more with you about testing our servlets.

    Last time we were testing a server's JSON response to make sure it called the right validators and handled bad inputs.

    Today we're testing the response of a POST request (also via AJAX). Our target: A simple controller that handles some very basic security (sufficient for our purposes). We want to send a request to the server and set a session variable if the user is authenticated. We are using JUnit integrated into Eclipse.

    So first, in our TestAdminController*:

    import static org.junit.Assert.*;
    import static org.mockito.Mockito.mock;
    import static org.mockito.Mockito.verify;
    import static org.mockito.Mockito.when;
    import org.junit.Before;
    import org.junit.Test;
    import org.mockito.Mockito;
    

     Naturally we'll have more than one test case, so we will put some of the common code into a setup() method that is run before each test case is executed:

    private AdminController controller; 
    private HttpServletRequest stubRequest;
    private HttpServletResponse stubResponse; 
    private HttpSession stubSession;  
    
    @Before
    public void setup()
    {
        controller = new AdminController();     
        stubRequest = mock(HttpServletRequest.class);
        stubResponse = mock(HttpServletResponse.class); 
        stubSession = mock(HttpSession.class);  
    }
    
    

    As always we've stubbed the request and response. This time we're also stubbing a session since we'll be setting a session attribute.

    Now we set up one of our test cases: simply testing that, given a valid user, the session variable is set by the server. To do this we first hook up our HttpServletRequest so that when the server attempts to retrieve the current session, it returns a stub instead. We also setup a stub user manager (which handles authentication) so that we don't have to worry about usernames and passwords and all those things:

    when(stubRequest.getSession()).thenReturn(stubSession);
    when(stubRequest.getParameter("password")).thenReturn(UserManager.SUPER_SECRET_ADMIN_PASSWORD); 
    UserManager stubUserManager  = new UserManager(); 
    when(stubUserManager.userIsAuthenticated(Mockito.anyString(), Mockito.anyString())).thenReturn(true)
    

    We use Mockito.anyString() because we don't care what credentials have been retrieved, but we always declare the user as valid. This decouples the test from the UserManager class.

    Finally, we are ready to initialize the controller:
    AdminController controller = new AdminController();
    controller.setUserManager(stubUserManager);
    controller.doPost(stubRequest, stubResponse);         
    

    To ensure that the server set the session variable, we use verify() to determine that the HttpSession.setAttribute() method was called:

    verify(stubSession).setAttribute("authenticated", true);
    

    Now we can run the test and if everything is working, it should pass. In this way, we have ensured that at least some parts of the controller are working so any problems will be on the client.

    Well, that's all for now!

    *I put the imports here because I hate it when I find a solution to something, but the author doesn't list the imports and the library isn't easily searchable!