We're running Jenkins as our CI server, and part of the automated deployment is to make sure the UI tests pass. Well, sometimes the Selenium tests fail for one reason or another after passing for say, 20 builds. Initially I hypothized the following:
- Click events don't register
- IE has a bad day
- Some kind of timing thing
All of these things have occurred in the past -- especially with IE. So with the help of
stackoverflow I went ahead and implemented some things to fix this problem.
The first thing was to
Re-run the UI tests that failed a second time. Of course this comes with all the drawbacks and this points to problems with the test itself. In this case that was ok because the test was still useful for confirming a bug was fixed, but we still need the build to pass. But I'll come back to this issue later on in the post.
Well it turns out that while JUnit has the @Rule annotation (see
this post) which allows you to intercept test calls, NUnit doesn't have that. Oops. I tried using the NUnit extensibility API but that didn't turn out to well (my fault I'm sure).
So I resorted to this method in our UI test base class,
SeleniumTest :
public void RetryFailure(Action testMethod)
{
if (_retrying)
{
_retrying = false;
return;
}
try
{
_retrying = true;
testMethod();
}
catch (Exception e)
{
Console.WriteLine("::::::retrying:::::::::::");
Console.WriteLine("error was {0}", e.Message);
WebDriver.Quit();
SwitchBrowser(_currentBrowser);
testMethod();
}
_retrying = false;
}
I'm not too excited about the code quality of this method, but it does work. All it does is attempt to run the test, catch any failures, and then rerun the test again. Here's what the beginning of a test that uses this looks like:
[TestCase]
public void AdminCancelANeedViaOfficeCalendar()
{
if (!Retrying)
{
RetryFailure(AdminCancelANeedViaOfficeCalendar);
return;
}
// ... more code
As you can see, we have to have this additional bit of logic at the beginning of each test. Now, it would have been easy to just encapsulate the test code in some kind of loop and then repeat it that way, but this allows the
SeleniumTest.RetryFailure method to have all of the control.
Alright, so the next step was to generate screenshots of test failures. This is pretty easy as Selenium already provides a way to do this by casting the WebDriver to
ITakesScreenshot. So nothing exciting there.
Now to integrate this into Jenkins. At this point, we have a bunch of failure screenshots being archived as artifacts, but you have to actually go look for them to view them. I'm lazy so instead I took a shot at providing a custom HTML report of the failures with their screenshots:
 |
| Summary of test failures |
|
|
So the screenshot is from a page that is accessible via the Jenkins project page. This means that we can be lazy and still get the information we need.
 |
| UI Test Failures link shows up on the project page |
The
HTML Publisher plugin takes care of the details, all we have to is generate the actual page. This is done once again inside
SeleniumTest.
private void BuildReport(string reportFile, string testName, string testImagePath)
{
if (!File.Exists(reportFile))
{
using (StreamWriter sw = new StreamWriter(new FileStream(reportFile, FileMode.Create, FileAccess.Write)))
{
sw.WriteLine("<html><body>");
sw.WriteLine("<h1>Test Failure Screenshots</h1>");
}
}
try
{
using (StreamWriter sw = new StreamWriter(new FileStream(reportFile, FileMode.Append, FileAccess.Write)))
{
sw.WriteLine("<div>");
sw.WriteLine("<h2>" + testName + "</h2>");
sw.WriteLine("<img src=\"" + testImagePath + "\"/>");
sw.WriteLine("</div>");
}
}
catch (Exception e)
{
Console.WriteLine("Problem writing report: {0}", e.Message);
}
}
Pretty simple, probably could be moved to somewhere more fitting. In the mean time though, it works well enough.
Anyway, the point of all of this was to just allow us to see why tests fail but also retry them to see if they fail reproducibly. It turns out that a couple of the tests repeatedly fail the first time but not the second, so there's probably some work to do.
See you next time.
