- Click events don't register
- IE has a bad day
- Some kind of timing thing
The first thing was to Re-run the UI tests that failed a second time. Of course this comes with all the drawbacks and this points to problems with the test itself. In this case that was ok because the test was still useful for confirming a bug was fixed, but we still need the build to pass. But I'll come back to this issue later on in the post.
Well it turns out that while JUnit has the @Rule annotation (see this post) which allows you to intercept test calls, NUnit doesn't have that. Oops. I tried using the NUnit extensibility API but that didn't turn out to well (my fault I'm sure).
So I resorted to this method in our UI test base class, SeleniumTest :
public void RetryFailure(Action testMethod)
{
if (_retrying)
{
_retrying = false;
return;
}
try
{
_retrying = true;
testMethod();
}
catch (Exception e)
{
Console.WriteLine("::::::retrying:::::::::::");
Console.WriteLine("error was {0}", e.Message);
WebDriver.Quit();
SwitchBrowser(_currentBrowser);
testMethod();
}
_retrying = false;
}
I'm not too excited about the code quality of this method, but it does work. All it does is attempt to run the test, catch any failures, and then rerun the test again. Here's what the beginning of a test that uses this looks like:
[TestCase]
public void AdminCancelANeedViaOfficeCalendar()
{
if (!Retrying)
{
RetryFailure(AdminCancelANeedViaOfficeCalendar);
return;
}
// ... more code
As you can see, we have to have this additional bit of logic at the beginning of each test. Now, it would have been easy to just encapsulate the test code in some kind of loop and then repeat it that way, but this allows the SeleniumTest.RetryFailure method to have all of the control.
Alright, so the next step was to generate screenshots of test failures. This is pretty easy as Selenium already provides a way to do this by casting the WebDriver to ITakesScreenshot. So nothing exciting there.
Now to integrate this into Jenkins. At this point, we have a bunch of failure screenshots being archived as artifacts, but you have to actually go look for them to view them. I'm lazy so instead I took a shot at providing a custom HTML report of the failures with their screenshots:
 | ||
| Summary of test failures |
 |
| UI Test Failures link shows up on the project page |
private void BuildReport(string reportFile, string testName, string testImagePath)
{
if (!File.Exists(reportFile))
{
using (StreamWriter sw = new StreamWriter(new FileStream(reportFile, FileMode.Create, FileAccess.Write)))
{
sw.WriteLine("<html><body>");
sw.WriteLine("<h1>Test Failure Screenshots</h1>");
}
}
try
{
using (StreamWriter sw = new StreamWriter(new FileStream(reportFile, FileMode.Append, FileAccess.Write)))
{
sw.WriteLine("<div>");
sw.WriteLine("<h2>" + testName + "</h2>");
sw.WriteLine("<img src=\"" + testImagePath + "\"/>");
sw.WriteLine("</div>");
}
}
catch (Exception e)
{
Console.WriteLine("Problem writing report: {0}", e.Message);
}
}
Pretty simple, probably could be moved to somewhere more fitting. In the mean time though, it works well enough.
Anyway, the point of all of this was to just allow us to see why tests fail but also retry them to see if they fail reproducibly. It turns out that a couple of the tests repeatedly fail the first time but not the second, so there's probably some work to do.
See you next time.
No comments:
Post a Comment