Acceptance Testing: Cucumber JVM

Cucumber-JVM is a java version of the popular Cucumber BDD tool for Ruby platform. The cucumber community (cukes) is one of the most vibrant community and are expanding the framework from Ruby to Java, .NET, Python, Perl, PHP etc. The core festures of cucumber being similar to jbehave with a story (feature) file, corresponding scenario implementation and an entry point class to execute all the stories. But there are some key differences between them which are discussed as follows:

1. All the parameters in the story are parsed using regular expression instead of parsing the parameter matcher in jbehave ($ by default).
2. JBehave allows to extend scenario implementation classes (i.e. StorySteps class) in order to reuse the scenarios. Cucumber on the other hand directly finds the scenario implementation for the story regardless of their classes, and blocks extending the implementation classes.
3. In JBehave, only single instance of the Step Definition class (scenario implementation classes) is maintained during the execution of the story retaining the values of instance variables. In cucumber though, for each scenario a new instance of the Step Definition class is created and all the previous values of instance fields are lost.
4. JBehave support annotations such as @BeforeStory and @AfterStory, which allow to execute the methods before and after the entire execution of the story respectively. Cucumber on the other hand has @Before and @After annotation which by default execute before or after every scenario. If a parameter is passed along with the annotation, such as @Before("@SETUP") or @After("@SETUP"), then the method will be executed before or after the scenario with the tag "@SETUP" in the feature file.
5. JBehave is flexible to have the Step Definition class (scenario method implementation) anywhere in the package structure, but requires the story entry point class (AllStories) to specify the instance of the class. On the other hand Cucumber-Jvm mandates to have the Step Definition classes in the same package of the story entry point class (AllStories). This enables cucumber to automatically find the implementation methods for the scenario specified in the story.
6. The tabular input format in Jbehave uses a ExampleTable class, which is a list of maps, each map representing a row, with table header's as the key to retrieve row values. In contrast to this approach, cucumber-jvm requires to create classes representing the table row structure. Cucumber then returns a List of Objects of the table type created earlier. This also helps to classify the text fields from the numeric fields using the data types of the instance variables.
7. The Configuration class for JBehave provides rich set of customization from Reports, Input Parameter converters, and story path. While cucumber does provide some of configuration options, major customization still doesn't seems to be straight forward.

The configuration of cucumber as mentioned above consists of an entry point class to execute all the features in the feature path. It loads the features using the Cucumber class providing options for execution and report generation. Below is the list of options available:

1. tags: specify the tagged scenarios and stories to execute or to skip. Only run scenarios tagged with tags matching TAG_EXPRESSION.
2. strict: Usually, when cucumber can’t find a matching Step Definition the step gets marked as yellow, and all subsequent steps in the scenario are skipped. The strict option causes Cucumber to exit with 1 for pending and undefined steps.
3. format: specifies how the results are formatted. Available formats: junit, html, pretty, progress, json, pretty:
   html: Generate an html report in the targeted location
   json: Generate a compact json report in the targeted location
   json-pretty: Generate a well formatted json report in the targeted location
   junit: Generate a cucumber junit report in the targeted location (xml format)
   progress: It causes a regular JUnit test to be stuck at yellow
    
4. features: specifies the path to the feature file (story). E.g. @Cucumber.Options(features = "classpath:simple_text_munger.feature")
5. glue: specifies the path where glue code (step definitions and hooks) is loaded from.
6. name: runs only the scenarios whose names match REGEXP.
7. dry-run: skips execution of glue code.
8. monochrome: doesn't color terminal output.

Below is the code which loads Cucumber feature files using Cucumber class with the options as described above.

@RunWith(Cucumber.class)

@Cucumber.Options(tags = { "~@WIP", "~@BROKEN" }, strict = true, 

     format = { "pretty", "html:target/cucumber", "json-pretty:target/cucumber.json" })

public class AllStories { }

Features in Cucumber-JVM are similar to the jbehave stories with Given-When-Then scenarios and support for tabular input as well. Further tags can be referenced in the feature file in order to tag scenarios and stories to execute or skip them.

@TESTS
Feature: Add a customer to the records.

@SETUP
Scenario: Customer account "John" is created with default settings.
Given a customer with the name "John" and table
        | ROW_ID | NAME | VALUE |
        | 3232323  | John12  | abc        |
        | 6454560  | John42  | xyz        |

When a customer tries to create an account
Then get an customer account id which is not null and greater than zero

Similar to Jbehave, cucumber also provides step definitions for execution of the scenarios in the features. As mentioned above, the step definition class cannot be extended for reuse, but cucumber automatically scans the package of its entry-point class, to find the step definitions for the corresponding scenarios. Further @StepDefAnnotation is used to mark the class of step definitions, later scanned by cucumber-jvm.

@StepDefAnnotation
public class OrgTerminalMachineSetupSteps{

@Before("@SETUP")
public void cleanup(){ ... }

@Given("^a customer with the name \"([^\"]*)\" and table$")
public void a_customer_with_the_name(String customerName, List<Row> list) throws Throwable { .. }

@When("^a customer tries to create an account$")
public void dealer_tries_to_create_an_account() throws Throwable { .. }

@Then("^get an customer account id which is not null and greater than zero$")
public void get_customer_accid_not_null_and_greater_thanzero() throws Throwable { .. }

  class Row {
    public String rOW_ID;
    public String nAME;           
    public String vALUE;
   }
}

For each step execution of the scenario in the feature, cucumber scans and finds the step definitions. Then it creates the instance of Step-Definition class before executing each scenario and executes the corresponding step methods. So in case the scenarios are required to be inter dependent in order to carry out an operation, all the instance fields of step definition class need to be singletons. So either a singleton factory class can be used to get field instance or spring can be used to inject such instances.
     Cucumber supports spring integration and requires "cucumber-spring" jar and "cucumber.xml" file in the source main resources directory. The cucumber.xml specifies the beans or component scans to load the beans required for cucumber acceptance tests. Also the spring config files can be imported into cucumber.xml for more organized configuration. The cucumber.xml is loaded by default using cucumber-spring before it initializes the step definition classes for tests execution. All the across scenario fields should be Autowired to grab the instances loaded by cucumber.xml. With such an spring integration, it allows to maintain the field instances across scenarios, access properties and take advantage of most of the spring related features.
     Moving ahead with the Jenkins setup for running the cucumber tests, it is necessary to run the tests in maven using maven-failsafe-plugin. The problem though with the failsafe-plugin is it requires all the tests to be inside the source test directory instead of source main directory. Although this seems logical as we are running tests and not any development code, it does require to load all the spring related beans from cucumber.xml in test resources by importing spring config files in the main resources directory. This seemed not to be working with both the spring configs (in test and main directories) and none of the beans were loaded. Copying all the spring related configuration files from the main resources folder to the test resources allows only to load/component scan the beans from the classes present in its codebase test or main. Hence the only solution we found is to copy all the source from main to test directory which seemed a lot of change. To avoid such major change for just running the tests using the maven-plugin, the configuration of the maven plugin was modified to load all the tests from the source main directory. Below are the changes and the config of the maven-failsafe-plugin:

 <plugin>

   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-failsafe-plugin</artifactId>
   <version>2.12</version>

   <configuration>
     <includes>
       <include>**/AllStories.java</include>
     </includes>
     <testSourceDirectory>${project.build.sourceDirectory}</testSourceDirectory>
     <testClassesDirectory>${project.build.outputDirectory}</testClassesDirectory>
     <reportsDirectory>${project.build.outputDirectory}/failsafe-reports</reportsDirectory>
     <additionalClasspathElements>
       <additionalClasspathElement>${project.build.sourceDirectory}/resources</additionalClasspathElement>
     </additionalClasspathElements>
   </configuration>

   <executions>
     <execution>
       <id>integration-test</id>
       <goals>
         <goal>integration-test</goal>
         <goal>verify</goal>
       </goals>
     </execution>
   </executions>

 </plugin>

The above changes in the testSourceDirectory and testClassesDirectory causes the maven-plugin to change its path to load the tests from the source main directory, thus running the acceptance test. Moving on to the Jenkins configuration, cucumber provides a nice plugin for Jenkins which enables it to provide well organized reports. The configuration for the Cucumber-Reports (latest version 0.0.14) Jenkins plugin is very simple as described in the documentation. The Json Report generated is usually in the target folder by default, hence we specify "Json Reports Path" as target. Also the "Plugin Url Path" is used to make the ""Back To Jenkins" link work in the Cucumber Reports by pointing to the right Jenkins Url.

One important note while running the Cucumber-Reports plugin: In the feature file if there is only Scenario wihout any Given-When-Then statements, then the cucumber tests fo run and generate the report in json. But the generated json report cannot be parsed by the cucumber-reports and it throws below exception,

[CucumberReportPublisher] Compiling Cucumber Html Reports ...
[CucumberReportPublisher] copying json from: file:/c:/.jenkins/workspace/cucumber-acceptance-tests/to reports directory: file:/e:/.jenkins/jobs/cucumber-acceptance-tests/builds/2012-11-01_16-13-02/cucumber-html-reports/
[CucumberReportPublisher] Generating HTML reports
ERROR: Publisher net.masterthought.jenkins.CucumberReportPublisher aborted due to exception
java.lang.NullPointerException
at net.masterthought.cucumber.util.Util.collectSteps(Util.java:104)

The reason behind it is the cucumber-reports plugin expects the scenarios to at least contain a Given statement in order to parse the generated json report successfully. Hence if we specify the scenario with atleast a Given step as below, the cucumber-reports jenkins plugin generates the report successfully.


@SETUP
Scenario: Setup.
Given Something

Although there are still some unresolved issues with the Cucumber-Reports Jenkins plugin. In the Cucumber-Reports in the Feature Statistics table, the time duration populated is "35 secs and 55 ms" but actually its supposed to be around 30 minutes. Also in the Feature Report details we see a message such as "Result was missing for this step". This message is displayed because the json report generated by cucumber doesn't have the result section in the report for every step: "result": { "duration": 776000, "status": "passed" }
If cucumber-jvm version 1.0.14 the json report does not have result section, but if 1.0.8 or 1.0.9 is used the json report does contain the result section. The cucumber-report plugin both version 0.0.14 and 0.0.12
cannot parse the result section the json report generated and the issue still persists. An quick fix will be to try using cucumber-reports jenkins plugin version 0.0.9 as shown the web documentation or wait till the issue is resolved in the later versions.