Performing an action when a test fails

Filed under: Java, Selenium, TDD, Test automation, — Tags: @ClassRule, @Rule, JUnit, Screen shot on failure — Thomas Sundberg — 2012-07-08

JUnit supports annotations so a method can be executed first in a test class or before each test method is executed. It also has annotations that supports methods to be executed after each test method or after the test class. This is very good if you need a common setup and a common tear down.

The after methods don't give you access to the test result. You cannot know if a test method failed or not. This may pose a problem if you have a need to perform some specific actions after a failed test. A solution could be to implement an onError() method. But JUnit doesn't support it.

A solution is to employ a @Rule annotation.

A JUnit rule is a way to change the test executions for a JUnit test. Or stated as in the Javadoc 'A TestRule is an alteration in how a test method, or set of test methods, is run and reported.'

You add a rule to a test using the annotations

on either a public static field or a public field.

A ClassRule is honored before the test class is executed and a Rule is honored before each test method is executed.

An implementation of a rule must implement the interface org.junit.rules.TestRule. A stub implementation is provided in JUnit so one way of implementing a rule is to subclass org.junit.rules.TestWatcher, override the method failed() and implement your failure code in it. It will only be executed if a test fails.

The simplest possible implementation may be:

package se.waymark;

import org.junit.rules.TestWatcher;
import org.junit.runner.Description;

public class SimpleOnFailed extends TestWatcher {
    @Override
    protected void failed(Throwable e, Description description) {
        System.out.println("Only executed when a test fails");
    }
}

All that is done is extending TestWatcher and overriding it's failed() method. This is all you need to do to get started. By extending this example, you are able to perform a lot of different things.

To use this rule, you have to declare an instance variable in the test class that should follow the rule and annotate it using the @Rule annotation. The simplest possible example may be:

package se.waymark;

import org.junit.Rule;
import org.junit.Test;

import static org.junit.Assert.assertTrue;

public class SimpleRuleExampleTest {
    @Rule
    public SimpleOnFailed ruleExample = new SimpleOnFailed();

    @Test
    public void shouldPass() {
        assertTrue(true);
    }
}

This example will never trigger the rule. To see that it actually works, change the assert to false and force a failure to execute the failed() method.

Taking a screen shot on failure

A more interesting example of using a rule may be to use it to take a screen shot and save the image when a test fails.

Let's assume that we are testing a web application using Selenium. Whenever there is a test failure, we want to capture the browser so we are able to understand why we had a failure by viewing what the browser just saw.

Let's start with a Selenium test that opens up a web page and trigger a failure.

package se.waymark;

import org.junit.After;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;

public class ScreenShotTest {
    private WebDriver browser = new FirefoxDriver();

    @Test
    public void shouldFail() {
        browser.get("http://www.yr.no");
        By link = By.partialLinkText("I do not expect to find a link with this text");
        browser.findElement(link);
    }

    @After
    public void tearDown() {
        browser.close();
    }
}

Three things are done in this example

It is not obvious why the test fails if you try to look at the browser while it executes. Chances are that you don't even see the entire browser before it shuts down and reports a failure. To be able to review what the browser saw, we would like to save a screen shot and review it.

The way to do this is of course to write a rule that will take a screen shot on failure and use it in our test.

package se.waymark;

import org.apache.commons.io.FileUtils;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;

import java.io.File;
import java.io.IOException;

public class ScreenShotRule extends TestWatcher {
    private WebDriver browser;

    public ScreenShotRule(WebDriver browser) {
        this.browser =  browser;
    }

    @Override
    protected void failed(Throwable e, Description description) {
        TakesScreenshot takesScreenshot = (TakesScreenshot) browser;

        File scrFile = takesScreenshot.getScreenshotAs(OutputType.FILE);
        File destFile = getDestinationFile();
        try {
            FileUtils.copyFile(scrFile, destFile);
        } catch (IOException ioe) {
            throw new RuntimeException(ioe);
        }
    }

    @Override
    protected void finished(Description description) {
        browser.close();
    }

    private File getDestinationFile() {
        String userDirectory = FileUtils.getUserDirectoryPath();
        String fileName = "screenShot.png";
        String absoluteFileName = userDirectory + "/" + fileName;

        return new File(absoluteFileName);
    }
}

It will use the browser, take a screen shot and save it as a png image called 'screenShot.png' in your home directory.

To use it, add a @Rule annotation to the test class.

package se.waymark;

import org.junit.*;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;

public class ScreenShotTest {
    private WebDriver browser = new FirefoxDriver();

    @Rule
    public ScreenShotRule screenShootRule = new ScreenShotRule(browser);

    @Test
    public void shouldFail() {
        browser.get("http://www.yr.no");
        By link = By.partialLinkText("I do not expect to find a link with this text");
        browser.findElement(link);
    }
}

Unfortunately, it turns out that you need to move where the browser is closed as well. Instead of closing it in the @After method I will close the browser in the finished() method in the rule. Closing in the @After method closes the browser before we are able to capture the screen shot.

Finally, a Maven pom that will provide all dependencies for this example:

<?xml version="1.0" encoding="UTF-8"?>
<project>
    <modelVersion>4.0.0</modelVersion>
    <groupId>se.waymark.educational</groupId>
    <artifactId>example</artifactId>
    <version>1.0</version>
    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.10</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>2.24.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

Resources



(less...)

Pages

About
Events
Why

Categories

Agile
Automation
BDD
Clean code
Continuous delivery
Continuous deployment
Continuous integration
Cucumber
Culture
Design
Executable specification
Gradle
Guice
J2EE
JUnit
Java
Linux
Load testing
Maven
Mockito
Pair programming
PicoContainer
Programming
Public speaking
Quality
Recruiting
Requirements
Scala
Selenium
Software craftsmanship
Software development
Spring
TDD
Teaching
Technical debt
Test automation
Tools
Web
Windows
eXtreme Programming

Authors

Thomas Sundberg
Adrian Bolboaca

Archives

Meta

rss RSS