Rely on your (well written) tests when adding new features

Hi everyone!

Well, I found some time to make a post here. On the ASP .Net project we're working on, things are going pretty well (although they have slowed a bit -- working on getting back into it). Still don't have my dev machine though.

Anyway, our previous work over the past two months has been mostly bug fixing, but we've recently implemented an actual new feature. The feature in question required some pretty substantial modification of various parts of the site and so the potential for breaking things was there.

Luckily, both our unit test suite and functional tests have grown pretty substantially and so there is a lot of coverage to ensure things don't break. In fact, the feature in question affects areas that happen to be heavily tested, so we can prevent regressions that way.

But it's not just about having test coverage of the code you're modifying -- you have to treat your test code right. That is to say, the test code is not "just test code" and it should be held to certain quality standards like your production code. We've spent a good chunk of our time establishing a test architecture when we could spend that time cranking out more badly written, fragile, hard to maintain tests.

Instead, we've taken the time to isolate potential changing parts of the code (such as element locaters) and build an abstraction layer on top of the Selenium API specific to our project. This makes the tests easier to maintain and less susceptible to UI changes. Another nice benefit is that it's reduced the time it takes to write functional tests like this substantially. Most of the actual Selenium code has already been written, and writing a test means creating different page objects and calling their methods.

When we do add to the page objects, it's usually a small amount of code and just means we'll be reusing it later.

So, the moral of this story: treat your tests right and you will benefit from it (we certainly have).


See you next time!

Selenium tests that fail "sometimes" and Jenkins

We're running Jenkins as our CI server, and part of the automated deployment is to make sure the UI tests pass. Well, sometimes the Selenium tests fail for one reason or another after passing for say, 20 builds. Initially I hypothized the following:

• Click events don't register
• IE has a bad day
• Some kind of timing thing

All of these things have occurred in the past -- especially with IE. So with the help of stackoverflow I went ahead and implemented some things to fix this problem.

The first thing was to Re-run the UI tests that failed a second time. Of course this comes with all the drawbacks and this points to problems with the test itself. In this case that was ok because the test was still useful for confirming a bug was fixed, but we still need the build to pass. But I'll come back to this issue later on in the post.

Well it turns out that while JUnit has the @Rule annotation (see this post) which allows you to intercept test calls, NUnit doesn't have that. Oops. I tried using the NUnit extensibility API but that didn't turn out to well (my fault I'm sure).

So I resorted to this method in our UI test base class, SeleniumTest :

public void RetryFailure(Action testMethod)
{
    if (_retrying)
    {
        _retrying = false; 
        return;
    }
    try
    {
       _retrying = true; 
       testMethod();
    }
    catch (Exception e)
    {
        Console.WriteLine("::::::retrying:::::::::::");
        Console.WriteLine("error was {0}", e.Message); 
        WebDriver.Quit();
        SwitchBrowser(_currentBrowser); 
        testMethod(); 
    }
    _retrying = false; 
}

I'm not too excited about the code quality of this method, but it does work. All it does is attempt to run the test, catch any failures, and then rerun the test again. Here's what the beginning of a test that uses this looks like:

[TestCase] 
public void AdminCancelANeedViaOfficeCalendar()
{
    if (!Retrying)
    {
        RetryFailure(AdminCancelANeedViaOfficeCalendar);
        return;
    }

// ... more code

As you can see, we have to have this additional bit of logic at the beginning of each test. Now, it would have been easy to just encapsulate the test code in some kind of loop and then repeat it that way, but this allows the SeleniumTest.RetryFailure method to have all of the control.

Alright, so the next step was to generate screenshots of test failures. This is pretty easy as Selenium already provides a way to do this by casting the WebDriver to ITakesScreenshot. So nothing exciting there.

Now to integrate this into Jenkins. At this point, we have a bunch of failure screenshots being archived as artifacts, but you have to actually go look for them to view them. I'm lazy so instead I took a shot at providing a custom HTML report of the failures with their screenshots:

Summary of test failures

So the screenshot is from a page that is accessible via the Jenkins project page. This means that we can be lazy and still get the information we need.



 

UI Test Failures link shows up on the project page

 The HTML Publisher plugin takes care of the details, all we have to is generate the actual page. This is done once again inside SeleniumTest.

 private void BuildReport(string reportFile, string testName, string testImagePath)
{
    if (!File.Exists(reportFile))
    {
        using (StreamWriter sw = new StreamWriter(new   FileStream(reportFile, FileMode.Create, FileAccess.Write)))
        {
            sw.WriteLine("<html><body>");
             sw.WriteLine("<h1>Test Failure Screenshots</h1>");
         }
    }
    try
    {
          using (StreamWriter sw = new StreamWriter(new FileStream(reportFile, FileMode.Append,  FileAccess.Write)))
         {
             sw.WriteLine("<div>");
             sw.WriteLine("<h2>" + testName + "</h2>");
             sw.WriteLine("<img src=\"" + testImagePath + "\"/>");
             sw.WriteLine("</div>");
         }
    }
    catch (Exception e)
    {
         Console.WriteLine("Problem writing report: {0}", e.Message); 
    }
}

Pretty simple, probably could be moved to somewhere more fitting. In the mean time though, it works well enough.

Anyway, the point of all of this was to just allow us to see why tests fail but also retry them to see if they fail reproducibly. It turns out that a couple of the tests repeatedly fail the first time but not the second, so there's probably some work to do.

See you next time.

Another abstraction layer on top of PageObjects with Selenium

Hi everyone,

As I mentioned before, on this project our Selenium 2 tests are extremely important. We've tried to keep the test code on the level of the production code, so it naturally has its own object model. We already use PageObjects extensively in the project.

Recently a few bugs popped up that we reproduced with Selenium tests that were pretty similar and required similar actions. Here's an example of some duplicated code:

// first step: log in as a temp to accept a need
LogOn login = new LogOn(WebDriver);
login.Go();
login.Login("Temp", "temporary");

TempNeeds need = new TempNeeds(WebDriver);
need.ViewFirst();
need.Accept();

login.Logout();
login.Go();
login.Login("Chris", "foo");

Billing billing = new Billing(WebDriver);
billing.Go();
billing.CreateNewInvoices();
billing.UpdateAndReview();
billing.FinishCreate();

Turns out this bit of logic is common in our tests, so it gets duplicated everywhere. So in the interest if minimizing the amount of code, I modified it to look more like this:

Temp temp = new Temp(WebDriver, "Temp", "temporary");
temp.AcceptFirstAvailableNeed();

Admin admin = new Admin(WebDriver, "Chris", "foo");
admin.CreateInvoices();

This code does the same thing, it's just been hidden by another abstract layer that represents the different users of the site and what they would do. Usually, each method off of a user would interact with many different PageObjects. The idea is to make the code reveal its purpose more by telling you what the user is trying to do, not the different PageObjects that need to be used.

Anyway, we're trying this approach out now for newer tests so we'll see how it goes. Hopefully the more tests we write and the more objects we build up, the less new Selenium code we'll have to write.

More bug-fixing with Selenium WebDriver and PageObjects

A problem, I'm definitely connected to the internet.

Hi everyone. I spent the past few days tracking and trying to reproduce another problem we've been having on the server. The problem is with the "recommend box" -- basically a way you can vote on your favorite songs or concerts. The problem happened seemingly randomly when you tried to recommend a song or a concert:

Oops. This bug had a few annoying characteristics:

• It was hard to reproduce and had to obvious order to causing the problem
• I couldn't reproduce it locally 
• The server stack trace was useless -- because we use an open session in view filter the root cause was a database problem but only showed up at the end of the filter when the transaction is committed 

Not a good combination. After spending a couple days just experimenting and try to reproduce it, I finally sat down and decided to squash it for real. It was clear that without being able to reproduce it consistently, it was going to be a hard fix. It was also clear that it was a database related problem, so the first steps were:

1. Restore the database with a set of test data
2. Try to reproduce the problem locally 
3. Record each action taken until the problem occurs
4. Repeat until its consistent

Luckily, that actually didn't take too long. I was able to isolate it to about 4 or 5 manual steps. The next part was to automate this process. Since it had no obvious starting point in the code, it was a good candidate for a UI test with Selenium WebDriver.

As with the previous post, we make heavy use of PageObjects to abstract the details of the markup and what-not from the test case:

package http.voting;

import static org.junit.Assert.assertEquals;
import http.SeleniumTest;

import org.junit.Test;

import pages.IndividualRecording.IndividualRecordingPage;
import pages.song.SongPage;
import pages.voting.RecommendPanel;

public class TestUpvote extends SeleniumTest
{
    protected boolean needsDatabaseReset()
    {
        return true; 
    }
    
    @Test
    public void reproduceErrorForUserActivityAndDuplicateKeys()
    {
        // go to recording 8 
        IndividualRecordingPage recording8 = new IndividualRecordingPage(driver, 8);        
        driver.get(recording8.getURL());
        
        // upvote it
        recording8.focus(); 
        RecommendPanel votingPanel = new RecommendPanel(driver);         
        assertEquals(RecommendPanel.VotingResult.SUCCESS,  votingPanel.upvote());
        
        // go to song page 
        SongPage song257 = recording8.gotoSong(257);
        // upvote song twice
        assertEquals(RecommendPanel.VotingResult.SUCCESS,  votingPanel.upvote());
        assertEquals(RecommendPanel.VotingResult.ALREADY_VOTED, votingPanel.upvote()); 

        // go to "July 8th, 1978 - Roslyn etc.." 
        song257.gotoAssociatedRecording(26);
        // upvote (didn't work in bug)
        assertEquals(RecommendPanel.VotingResult.SUCCESS, votingPanel.upvote());         
    }
}

Minor note: the needsDatabaseReset() method tells the SeleniumTest class to restore the database to a fresh state each time a test is run.

You can see the test case deals with specific actions such as "upvote" and "go to song" instead of searching the website for different elements to click. This is all handled by the different page objects. For example, here is the relevant section of the RecommendPanel class:

     public VotingResult upvote()
    {
        final int currentVoteCount = getCurrentVoteScore(); 
        getUpvoteButton().click(); 
        
        try
        {
            // success if the votes increase or the "error box" pops up -- although we still need to check it 
            (new WebDriverWait(source, 5)).until(new ExpectedCondition<Boolean>() {
                public Boolean apply(WebDriver d) {
                    return getCurrentVoteScore() > currentVoteCount || getVisibleErrorBox() != null ; 
                }
            });
            
            int newVoteScore = getCurrentVoteScore(); 
            
            if(newVoteScore > currentVoteCount) 
                return VotingResult.SUCCESS;             
            else if(getVisibleErrorBox() != null && getVisibleErrorBox().getText().contains("You already voted"))
                return VotingResult.ALREADY_VOTED; 
            else
                return VotingResult.ERROR; 
        }
        catch(Exception exc)
        {
            exc.printStackTrace();
            return VotingResult.ERROR; 
        }
        
    }

Here, we're simply clicking the "recommend" button and ensuring that the result is either an increase in the current vote score, a popup saying "you already voted," or an error message.

With this test, the problem is now easily reproducible. The actual problem is very unexciting. We log user actions by IP address to decrease the chance of duplicate voting. The table we use is mapped to multiple classes with the id "generator" set to increment. This caused duplicate primary keys to be generated when new entries were added. Changing the generator to "identity" fixed the problem.

In contrast to the previous bug, this one was hard to reproduce but an easy fix. Stay tuned for more bugs that are both irreproducible and hard to fix!*

And for reference, the PageObjects used for this test:

IndividualRecordingPage
SongPage
RecommendPanel

* hopefully not

The PageObject pattern for Selenium WebDriver UI tests

There was a bug recently encountered while browsing the graphs page. Clicking on two of the canned graph options yielded this friendly result:

Woops. So at that moment I was also experimenting with Selenium WebDriver for automating some UI tests. So I figured "hey, why not reproduce the problem with some of these tests before fixing it."


At first I just stuck the Selenium test code in each test case, but after awhile I refactored to use the page object pattern. Basically you have a class that represents some part of the page (in this case, GraphsPage) which serves as an interface the the component's services, such as "generate graph." in our case.



So before the first step, it's time to define a base class called PageObject which we'll subclass for the graphs page. At this point I'll also note that Selenium seems to have some kind of support integrated called a PageFactory that you use on your page object class. In the near future I might refactor them again to use that.

But anyway, here's the PageObject:

package pages;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;

public abstract class PageObject 
{
    protected WebDriver source; 
    
    protected PageObject(WebDriver source)
    {
        this.source = source; 
    }
                
    public abstract String getURL(); 
    
    public abstract void focus(); 
}

Nothing too exciting here, it's just some boiler plate.

On to the fun part, the GraphsPage class:

package pages.graphs;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.ui.ExpectedCondition;
import org.openqa.selenium.support.ui.Select;
import org.openqa.selenium.support.ui.WebDriverWait;

import pages.PageObject;

public class GraphsPage extends PageObject
{
    private String GRAPH_URL = "graphs"; 
    
    public GraphsPage(WebDriver driver)
    {
        super(driver); 
    }
    
    private WebElement getGeneratedURLLabel()
    {
        return source.findElement(By.cssSelector("#graphpanel #graphlink")); 
    }
    
    private  Select getCannedGraphsList()
    {
        return new Select(source.findElement(By.cssSelector("#cannedgraphs select")));
    }
    
    private  WebElement getGenerateGraphsButton()
    {
        return source.findElement(By.className("getgraph"));
    }
    
    public String getURL()
    {
        return GRAPH_URL; 
    }
    
    public boolean generateGraph()
    {
        getGeneratedURLLabel().clear(); 
        getGenerateGraphsButton().click();  
        
        try
        {            
            (new WebDriverWait(source, 5)).until(new ExpectedCondition<Boolean>() {
                public Boolean apply(WebDriver d) {
                    return getGeneratedURLLabel().getText().contains("?"); 
                }
            });
            
            return true; 
        }
        catch(Exception exc)
        {
            return false; 
        }
    }
    
    public void selectCannedGraph(String cannedGraphTitle)
    {
        Select cannedGraphs = getCannedGraphsList();         
        cannedGraphs.selectByVisibleText(cannedGraphTitle); 
        cannedGraphs.getFirstSelectedOption().click(); 
    }

    @Override
    public void focus() 
    {
        getGeneratedURLLabel().click(); 
    }
}

Alright, now for the explanation. First, I'd like to explain the presence of the focus method. It seems in my experience that using FireFox 6 and IE, the click() method would sometimes fail to work unless you focused the page the first time. So that's what that does.

Here are the service methods:

• generateGraph(): clicks the "generate graph" button, and returns true if the graph was retrieved successfully. This is done by checking for the presence of a generated URL for that specific graph. 
• selectCannedGraph(): selects the canned graph on the canned graph list based on it's title.

You also see a few private methods, those are there mainly to abstract references to specific HTML elements and only have one reference to them.

Now we can write a test case to reproduce the problem. The problem occurs when you select the canned graph "Most played in cities," so let's write one to reproduce that:

package http.graphs;

import static org.junit.Assert.assertTrue;
import http.SeleniumTest;

import org.junit.Test;

import pages.graphs.GraphsPage;

public class TestGraphs extends SeleniumTest
{
    
    @Test
    public void testMostPlayedInCities()
    {
        testCannedGraph_helper("Most played-in cities");        
    }
    
    private void testCannedGraph_helper(String graphToCheck)
    {
        final GraphsPage page = new GraphsPage(driver);         
        driver.get(super.getPage(page.getURL())); 
        
        page.selectCannedGraph(graphToCheck); 
        assertTrue(page.generateGraph()); 
    }        
}

Nothing too exciting, you can see I've added an additional helper method so we can test other canned graphs as well. All we do here is select the canned graph, generate it, and assert that the graph was generated correctly. If you run it, it will fail.

One last thing, you can see that this test class extends SeleniumTest. That's just a boiler plate class, it looks like this:

package http;

import org.junit.After;
import org.junit.Before;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;

public class SeleniumTest 
{
    protected WebDriver driver;
    protected final String ROOT = "http://localhost:8080/Recordings/"; 
    
    protected WebDriver getNewDriver()
    {
        return new FirefoxDriver(); 
    }
    
    @Before
    public void Before()
    {
        driver = getNewDriver(); 
    }
    
    @After
    public void After()
    {
        driver.quit(); 
    }
    
    public String getPage(String page)
    {
        return ROOT + page; 
    }
}

It just initializes the Firefox drvier and restarts it after each test. Alright, so next time I'll show you what the actual problem is and how it was fixed. Until then, take care!