Test Driven Development

Test Driven Development is famous software development process which relies on the developer to write an automated test case before writing any piece of functional code. It emphasizes series of unit tests and re-factoring to provide a simple design.

   Everyone is accustomed to the general practice of software development which looks as below:

• Design: Figure out how you're going to accomplish all the functionality.
• Code: Type in the code that implements the design.
• Test: Run the code a couple of times to see if it works, then hand it over to QA.

On the other hand Test Driven Development modifies this approach as below:

• Test: Figure out what the next chunk of function is all about.
• Code: Make it do that.
• Design: Make it do that excellently.

As described above TDD completely inverts the accepted ordering of 'design-code-test'. So, from one view, TDD just puts the design after the test and the code. Refactoring is considered as pure design in TDD.

   In TDD world we are not allowed to figure out a complete or excellent design to get our test (and all existing tests) to pass, before we start coding it. Although there is sometimes a debate on whether there should be some kind of initial design phase were interfaces (along with methods signature) for the future classes needs to be defined. Further it is not allowed to reduce or skip the "refactor" step during the TDD development. Hence after each iteration of passing test, there should be refactoring done on the code which indirectly contributes to the design. Also once a test is written, TDD allows us to do either of the following during implementation to pass the test:

1. Reuse some existing code
2. Introduce meaningful new class(es) and method(s)
3. Copy existing method(s) and change the copies

TDD helps in certain aspects of the integration, as the entire process a divided into a series of small steps. The more often we check in the code in version control system, and the smaller our changes are, the less likelihood of getting any 'merge conflicts' with others. Also every commit is a guaranteed fallback position, a piton in the rock that we can easily go back to if we slip and fall.

Below is the Red-Green-Refactor Rule for Test Driven Development:

RED	When you write the test, you are designing the behavior you expect the code-under-test to perform.
GREEN	When you write the code to pass the test, you are designing the internal implementation of that behavior.
REFACTOR	Your micro-focus on getting to green probably 'un-designed' the code. When you refactor you are re-designing.

The Stepwise Premise for TDD goes as below:
   -  Can gigantic complex architectures really be created using nothing other than red-green-refactor?
   -  Consider these issues:

• All large solutions don't just materialize out of nowhere; they are ultimately created in modest steps anyway.
• Even if we have analysis and design phases for large-scale architectural features, we can still develop using TDD.
• Considerable data is available to support the idea that complex global design processes frequently don't work.
• TDD has a serious track record: it is being used all over the world to create complex systems.

Below are the commonly used TDD patterns:

Specify It

• Essence First: What is the most basic functionality needed, not including anything fancy
• Test First:       What exactly will we be testing? Capture that in the test method name.
• Assert First:    What behavior would you like to check?  Writing the assert statement will lead us to produce the structure backwards by "backfilling the method" by declaring the objects and methods we need to create as well as the expected result of calling the new code.

Frame It

• Frame First: Create whatever class(es), constructor(s) and method(s) are needed by our assert statement.

Evolve It

• Do The Simplest Thing That Could Possibly Work: Focus on minimalism by asking oneself to program only what is absolutely necessary to pass a test.
• Break It To Make It: Write a new test code that we know will fail because as our production code isn't capable of handling the new test.
• Refactor Mercilessly: Make design improvements continuously, aggressively, mercilessly avoiding really bad code.
• Test Driving:  In TDD, we don't want to stray too far from the Green Bar.

Finally, Robert Martin, one of the fanatic devotee of Test Driven Development provides the three laws of TDD in his book Clean Code as below:

• First Law: You may not write production code until you have written a failing unit test.
• Second Law: You may not write more of a unit test than is sufficient to fail, and not compiling is failing.
• Third Law: You may not write more production code than is sufficient to pass the currently failing test.

Refactoring generally involves by taking an existing class that's too complex, and break it into smaller classes, each of which takes part of the old class's responsibility, and both of which work together. There are numerous advantages of refactoring the classes to smaller ones, some listed as as follows:

   1)  By making classes smaller, thus easier to grasp at one time.
   2)  By aligning the smaller classes with a well-understood functional breakdown of the underlying problem.
   3)  By making the couplings between classes mirror the couplings between functionality.
   4)  By (ultimately) allowing complex systems to be built by composing many simpler objects.
   5)  By making each smaller class easier to test.

Refactoring also involves Decremental Development, which means finding ways to shrink the code even as we continue to add new features. All the common functionality are moved as a part of library, while pre-existing libraries (core as well as external) with required implementation is searched for instead of re-inventing the wheel.

GUI Applications

In order to apply TDD on GUI applications, they need to have clear separation between user interface and operational logic most commonly achieved by MVC pattern. Although the model/view split isn't the only technique for TDD'ing GUI's, but it does represent the meta-pattern for all of them.
Following can be achieved by splitting responsibilities:

• We can test the Model by having our TestCase pretend to be the View.
• The most important interactions are on the Model, enabling to test core functionality.
• We can use fake domain objects for testing which are in turn are used by the Model.
• We can test the View by creating a fake Model and driving it that way.
• The View can be tested by driving the window's programmatically.

A lot of enhancements can be applied to the Model-View split further such as follows:
 - Add Publisher-Subscriber to allow multiple Views on the same Model.
 - Add a Controller class to translate View-gestures into Model-commands.
 - Add a Command system to isolate and manipulate individual commands.


Test Driven Development Shortcomings

TDD is a development process which assures quality by enforcing unit tests. Although the quality of the code mainly depends on the quality of tests, not when the tests are written during development or how many lines are covered. The essential purpose for writing unit tests is to reduce the possibly of defects in the development phase itself and provide a set of automated tests to validate future changes without introducing new defects. Although such approach is greatly beneficial, the question raised often is to what extent should the tests be written ? When does this approach looses efficiency over the value of auto-tested code ? Does this provide optimal solution to the complex process of software development and unforeseen defects. Is the time and effort spent in writing unit tests to prevent and decrease defects the best approach ?

Most of the Unit Testing tutorials, TDD books and sites describe the approach with basic examples such as processing students grades, calculating wages etc. Although it does gives us a perspective and seems to make the approach by far the best one, but when applied in the co-operate world, such approach has some inherent issues listed as below:

1) Testing a piece of code completely, may involve huge number of scenarios to be considered. Even to select the subset of critical cases and write the test cases for them, it involves almost similar effort as writing the original functional code. But even after selecting a subset of critical cases, we still open ourselves to the possible defects occurring from the ignored scenarios. How to decide which cases are critical and which should be ignored. Some cases may be ignored before, but considering the entire system, such cases could lead to vital failures. Hypothetically, even if we painstakingly compile all the critical cases and wrote unit tests for the entire application, we cannot be sure that there wouldn't be any defects coming up from the unit tested code. Often times, the unit tests validate obvious scenarios (mostly by replicating the code/object in unit test or verifying if the method does get called) thus providing us with a false sense of security. This mostly is caused when the same person writes both the test and the code.

2) Compared to most of the unit testing examples in tutorials, books and articles, the professional code is not that simple or straight forward to isolate. Many real world systems involves, file handling, calling external services, databases, invoking external processes and multi-threading operations. The outcome of these operations is hard to predict. We cannot comprehend the possible values returned by the external services, or by the database all the times. Some of the scenarios such as concurrent operations, server timeout, etc are difficult to recreate in unit test environment. Even if a unit test could be written to check the handling of possible service failures, it would require a substantial amount of efforts compared to manual or integration testing.

3) The basic premise of TDD is that the test drives the system design and implementation. Hence if the line of code cannot be tested then it shouldn't have be written at all. Sometimes due to the limitations of Unit Testing tools such as Junit, Mockito and others the unit test cannot isolately test a certain piece of code. Static methods is one of such cases were despite using Powermock there are many questions raised over the effectiveness of those tests. Also private class fields/methods mostly tend to be changed to lower access modifiers to facilitate unit testing as far as Junit is concerned. Concerns are also raised about the use of Mockito's InjectMocks in unit tests and recommended to use constructor based auto-wiring instead of setter or field based auto-wiring. This ultimately restricts the usage of some features of the programming language or the frameworks inside the boundaries of testability often tagged as bad design.

5) As mentioned previously by Robert Martin, no production code should be written without the corresponding failing test. This totally ignores the fact that whether the unit test is effective, productive and valuable in catching issues. Further it blurs the line between writing a unit test on the behavior/functionality of the code rather than mapping each line of production code with the corresponding unit test. For example creating a new object, setting values to an object, non-conditional calls to library's void methods, logging etc sure compounds to numerous lines of production code, but they hardly articulate any logic or behavior. Consider the following code below:

Properties properties = new Properties();
properties.setProperty("key", "value");
properties.store(new FileOutputStream("C:/test.properties"), null);

The above code creates a Properties object and uses built-in store method of API to create properties file without any conditional logic. There could be many what if arguments made such as what if the store method is not called or file path is incorrect, or properties are not set or incorrectly set etc which often is a slippery slope. But mandating the existence of a line of code or their order is not the purpose of unit test, but is to make sure an independent chunk of code behaves as intended. Any piece of code which only has a single logical flow and returns same or similar results no matter the input has no concrete behavior. Further, if the code does not provide any behavior by itself or relies on external library methods for its behavior then unit testing such code not only adds to overhead and maintenance but fails to provide any productive feedback to detect real problems.
    Further, mandating TDD during a proof of concept or trial and error to fix a known problem not only increases the development overhead exponentially but also distracts the developer from the core task/problem.

4) Someone has said "the line of code that is fastest to write, that never breaks, that doesn't need maintenance is the line you never have to write". In Test Driven Development, as the unit test drives the development (rather than us choosing the critical methods to unit test), there is a lot more test code involved. Multiple scenarios for the given piece of code may encourage duplicate code unless only a single person works on it. In the co-operate projects such big chunks of test code adds up to the maintenance of the system. Badly written unit tests which often involves hardcoded error strings further consume time/effort to maintain. Fragile tests which generate false failures mostly tend to be ignored even in case of valid errors. Modifying the existing functionality using TDD becomes quite challenging as we need to deal with a mesh of interconnected mock objects and a series of test cases.

 Finally the root issue with TDD is not the effort or time required to write them, but their value compared to the effort i.e. Developer Productivity. TDD is much easier to be applied when the design documents dictates the classes/methods and their functionality beforehand. It also would help if all the possible test cases are listed (usually by testers) for the pre-designed classes.


Was it really Behavior Driven Development ?

Since writing this 2013 blog post, many others have joined to question the effectiveness of TDD. David Heinemeier Hansson, the creator of Ruby on Rails has described TDD as "Test-first fundamentalism is like abstinence-only sex ed: An unrealistic, ineffective morality campaign for self-loathing and shaming". After the blog post Kent Beck put forward his sarcastic defense on TDD which later was followed by conversation with Martin Fowler on whether TDD is Dead. Though the conclusion of the conversation was that TDD is valuable in some contexts, but much disagreement prevailed over the number and type of contexts in which it should be applied. Then in the DevTernity 2017 conference Ian Cooper gave a talk on "TDD, Where Did It All Go Wrong" which was promoted by Uncle Bob Martin. In the talk Cooper pointed out that TDD is being practiced incorrectly since we are focused on testing the implementation details instead of testing the system behavior. Due to this we often write more test code than implementation code. Such implementation driven tests with spaghetti of mocks makes refactoring painful, maintenance a nightmare and decreases the overall development productivity. Developers too often don't understand the intent of such tests and are unable to deduce the system behavior by reading them. Enhancements and re-designs becomes difficult as changing the implementation also requires to change the tests which is long haul process.

TDD is mainly practiced by using 'adding a new method to a class' as trigger to write a test. Such test-case per class approach fails to capture the true ethos for TDD. Adding a new class or method is not the trigger for writing tests. The trigger is implementing a requirement. Write tests to cover the use cases or user stories, not the implementation classes or methods. The system under test is not a class but the exports from a module or its facade. The 'unit' of 'unit testing' here really means module, not a class. A class by itself can be the facade, but many classes are implementation details of the module. Do not write tests for implementation details, these change. Write tests only against the stable contract of the (public) API (which can be within a module). Ian Cooper referenced the first book on TDD, "Test-driven Development: By Example" by Kent Beck and pointed out that Kent has explicitly stated that we need to be testing behavior not the implementation. On page 4 of the book Kent writes "What behavior will we need to produce the revised report? Put another way, what set of tests, when passed, will demonstrate the presence of code we are confident will compute the report correctly ?", which clearly refers to test over behavior not implementation. Kent further states that "When we write a test, we imagine the perfect interface for our operation. We are telling ourselves a story about how the operation will look from the outside. Our story won't always come true, but its better to start from the best-possible application program interface (API) and work backward than to make things complicated, ugly, and 'realistic' from the get-go", which affirms testing API's not implementation methods. The tests should run in isolation from other tests, but not the system under test. The unit of isolation is not the class under test, but the tests themselves. Although tests can and should test several classes working together if that is what is needed to test the behavior. We avoid file system, database, simply because these shared fixture elements prevent us from running in isolation from other tests, or the tests become slow. But if there is no shared fixture problem (one test does not affect another) then its perfectly fine to talk to database (though in-memory) or file system in unit tests. Focusing on methods for testing creates tests which are hard to maintain and code which is difficult to refactor because implementation details are exposed to the tests. Such tests do not capture the behavior we want to preserve and becomes difficult to understand. Refactoring is the process of changing a software system in such a way that it does not alter the external behavior of the code yet improves its internal structure. It is the step were we improve our design/implementation, produce clean code, remove duplication, sanitize code smells and apply design patterns. During refactoring to clean code we should not write new unit tests since we are not introducing new public APIs / classes. Dependency is the key problem in software development at all scales. Dependency between the tests and the code should be eliminated by avoiding mocking. Tests should not depend on implementation details by using Mocks because changing the implementation breaks such tests.  Hence mocks should be avoided at all costs except to isolate the tests on the module boundaries (databases, external services, file systems).

Unit Testing using Mockito and PowerMock

Unit Testing is a vital task in any development cycle. It involves writing tests around the actual classes and methods developed as part of the project. Mostly all the applications today involve external service calls, database calls, system calls which cannot be invoked by the unit tests as it will effect the state of the application in most circumstances. One possible solution would be to create an equivalent Test classes for such classes making external calls, using the Test Double pattern. But this increases redundant code, drives up efforts to develop unit-test counterpart of the original class and increases code maintenance. On the other hand mocking an object from the original class can be easily used to check for expected results while writing unit tests. There are many frameworks which support mocking of objects such as Mockito, PowerMock, JMock, EasyMock, SevenMock, rMock and Unitils. All the mocking frameworks use reflection mechanism and sometimes byte-code to create a mocked object mostly during runtime. The usual working of these frameworks involves following steps:

1. Creation of a mock.
2. Definition of the stubbed methods (what the method should do when a call happens). Sometimes definition is combined with expectations.
3. Definition of the expectations (how many times this method will be called, etc).
4. Execution of the test code.
5. Verification of the expectations.

  There are various patterns and popular styles for unit testing using mock objects. The Chicago-style Testing and London-style Testing are the popular ones which preached in most of the schools. Chicago-style Testing focuses on asserting that the subject-under-test changes to the expected state. While London-style Testing focuses on writing tests by asserting that the subject-under-test does the expected calls to the components to which it must interact. London-style tests usually use mock objects to assert interactions and to isolate the subject-under-test from its dependencies, facilitating the task of testing. London-style testing is also referred as Interaction or Behavioral style TDD or mockist-style testing. Chicago-style on the other hand is also referred to as Detroit-style or classic TDD.
   Self Shunt pattern is another approach besides mocking the objects for unit testing. It is usually used to test whether an object under test communicates correctly with its collaborator i.e. to check that an object has been called correctly. With self-shunt, the test case passes itself to the object under test, the object under test then interacts with the test case, and then the test case checks its own state. Self-shunt is a specialized case of mock object pattern, where the test case itself acts as a mock. Self shunt pattern can be applied typically by creating complete stub of the object or using the test case object itself as stub. Stub objects provide canned responses (and can be autogenerated by helper libraries), but typically do not directly cause the unit test to fail unlike the mock objects. They are typically just used so that the object you're testing gets the data it needs to do its work. Self shunt pattern does violates the single responsibility principle which states that every class should handle only one responsibility. With self shunt pattern, the test stub class changes if the test case changes or if method signature of the interface it implements changes thus making the test class responsible for both the test case as well as to implement the interface. But even if the test class is mocked, the attributes of the class still must be semantically coupled with the original test class making any meaningful separation difficult.

   In real unit tests stubs are a lot more complex than dummy objects because usually it needs a way to modify the return value on the stub object. Further it starts to get really complex when the system under test requires certain methods on a collaborator to be called (possibly in a certain order). Then we need to use a mock that can record how it is used and be verified later on. Using a Test Spy, is a much simpler way to test how collaborators were used than creating a record/playback style mock. A Test Spy is a real object with one or many mocked methods. It allows to record method invocations for later verification of the behavior and stub methods.

Below is the common terminology specified in xunits-patterns used across various testing frameworks:

• A Dummy Object is a placeholder object passed to the system under test but never used.
• A Test Stub is a hard coded object used for testing. It provides the system under test with indirect input.
• A Test Spy provides a way to verify that the system under test performed the correct indirect output. The verification occurs after the method under the test has been called.
• A Mock Object provides the system under test with both indirect input and a way to verify indirect output. All the expectations are configured before the calling of the method under test.

Mockito
Mockito is one of the testing framework used to create mock objects for automated junit tests in Test-driven development or Behavioral-driven development. Mockito allows to mock both classes as well as interfaces unlike EasyMock which requires class extensions to do so. Mockito also allows to chain the method calls similar as EasyMock, producing less imperative code. Mockito also supports Hamcrest matchers allowing 'match' rules to be defined declaratively, such as assertThat() contruct and its standard set of matchers. It is primarily used for Interaction testing in order to verify the interactions between various objects.
    When using mockito for mocking objects, we don't need to specify an exact argument. We can use argument matchers such as anyString(), anyList(), anyLong(), anyMap(), anySetOf(), anyListOf() etc. Warning, If you are using argument matchers then all the arguments must be provided by matchers.
e.g. when(person.getAddress(anyInt(), eq("abc")).thenReturn("53rd Street, IL");   // FAILS

doNothing() is used to set the void methods to do nothing which generally is by default on mock objects. It is mostly used when we make consecutive calls on the method were we want alternate call to fail, or when we spy on the actual object and want the void method to do nothing.
doAnswer() is used when we want to answer the call to the stub object's void method with Mockito's generic Answer type.
doThrow() is used when we want to throw an exception when the stub object's void method is called.
doReturn() is used to when we are calling real methods on spy objects or overriding previous exception stubbing. We can specify the object to be returned when the specified stub method is called.
when() is used when we want the particular method of the mock object to return a particular value (or throw a particular exception) when it is called.
stub() is used to stub a method call with return value or an exception. It is same as Mocktio.when which is recommended over stub() method.
verify(mock) is used to check if certain behavior happened once. verify(mock, times(n)) is used to check if the certain behavior happened n number of times. Verify will work only after calling the actual method.

Further @Mock annotation is used to create a mock object similar to Mockito.mock(). The @InjectMocks annotation on the other hand is used to inject the mock or spy objects (from current class) in the specified class to instantiate an object. Currently it only supports setter injection. Mockito tries to inject the objects by type, but does not throw anything when injection fails.

Below is the required maven dependencies for using Mockito:

  <dependency>
   <groupId>org.mockito</groupId>
   <artifactId>mockito-all</artifactId>
   <version>1.9.5-rc1</version>
  </dependency>

  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <version>4.10</version>
   <scope>test</scope>
  </dependency>

PowerMock requires @RunWith(PowerMockRunner.class) annotation at the class level inorder to initialize powermock. Then @PrepareForTest annotation is required to tell PowerMock to prepare the specified classes for testing. The classes passed to @PrepareForTest annotation contains the static methods which are needed to be mocked. In order to mock a static class PowerMockito.mockStatic() is called passing the static class to be mocked. Mockito.when is used to return the expected value, do nothing or throw exception. PowerMockito.when() method can be used too which just delegates to the original Mockito.when(Object) method. The PowerMockito.doNothing() is used for setting void methods to do nothing. The PowerMockito.doCallRealMethod() method is used to called the real implementation of the static method. Static methods can be verified by first calling the PowerMockito.verifyStatic() to start verifying certain behavior followed by the call to the static method to be verfied. Mockito.VerificationMode can be used with PowerMockito.verifyStatic(Mockito.times(2)) to verify the exact number of calls on a static method. Below is the sample test using Mockito:

@RunWith(MockitoJUnitRunner.class)
public class TestHarnessWebServiceTestServiceTest {

    @Mock OrganizationAccountMapper organizationAccountMapper;
    @Mock UserService userService;
    @InjectMocks TestHarnessWebServiceTestService service;

    @Test
    public void authorizeUser_shouldThrowAnErrorIfWeFailToGetRoles() throws Exception {

      Sample sample = Mockito.mock(Sample.class);

      when(userService.getTaxDetails(anyListOf(Integer.class))).thenThrow(new ContextException(DB_ERROR));
      when(userService.login(anyString(), anyInt(), anyString())).thenReturn(null);
      when(userService.findRolesFor("userId")).thenReturn(new String[0]);

      assertEquals(true, userService.testAccountFlexOrgList("userId", "password"));
      assertSame(error, e.getCause());
      assertTrue(userService.authorizeUser("userId", "password"));

      verify(organizationAccountMapper).getRelatedAccounts(0, "1");
      verify(userService,times(2)).updateContext(organizationAccountMapper);

      service.sendMessage("userId", "Some Message");
   }
}

public class TestHarnessWebServiceTestService {

   private UserService userService;

   public void setUserService(UserService userService) {
      this.userService= userService;
   }

   public void sendMessage(String userId, String message) {
      String emailId = userService.getEmailAddress(userId);
      sendEmail(emailId, message);
   }
}

While using mockito when clause on mock/live objects, it is important to consider the following:
1) When Mockito.mock() is used and methods are called on it, none of the actual methods get called. The when clause configured works in this case.
2) When a new Object() instance is used and mockito config such as Mockito.when(....).thenReturn(....) is applied to it (object in the when clause), then this Mockito configuration does not work for the actual object. In such case a mock object needs to be injected e.g. in case of service or database objects, which will configured with mockito's when...thenReturn... clauses.

Mockito Answer is used to provide the mock object with the ability to act as a bean, by recording the value being passed to the actual method. It allows stubbing with generic Answer interface and return the same argument instance on a mocked method using Answer interface. The doAnswer() is used to stub a void method with a generic Answer and capture arguments passed for verification. The thenAnswer() method sets a generic Answer for the method and is similar to the thenReturn() method were the answer() method is executed everytime returning the value specified during the when clause.

Do Answer Example:

      Mockito.doAnswer(new Answer() {
          public Object answer(InvocationOnMock invocation) {
              Object[] args = invocation.getArguments();
              Mock mock = invocation.getMock();
              return null;
          }
      }).when(mock).someMethod();

Then Answer Example:

      Mockito.when(mock.someMethod(anyString())).thenAnswer(new Answer() {
          Object answer(InvocationOnMock invocation) {
             Object[] args = invocation.getArguments();
             Object mock = invocation.getMock();
             return "called with arguments: " + args;
          }
      });

      System.out.println(mock.someMethod("foo"));

Spy

All the methods of a spy object are real unless they are stubbed, as opposed to a mock object were all the methods are stubbed unless callRealMethod() is called. Hence spy object allows partial mocking retaining real methods of the object to be tested. Partial mocking is widely debated as it usually means that the code complexity has been moved to a different methods of the same object which generally is not considered as a best practice.

Mockito Spy Example:

      ConfigurationEvent configurationElement=Mockito.spy(new ConfigurationEvent());
      Mockito.doNothing().when(configurationElement).begin();
      Mockito.doReturn(true).when(configurationElement).isSoftDeleted();
      Mockito.doReturn(new ElementType(12,"event",true,true)).when(configurationElement).getElementType();

Matchers

Matchers provides a set of static methods which allows flexible verification and stubbing. There are two implementations of matchers, Hamcrest matchers and Mockito matchers. Hamcrest matchers are generic-typed objects that check that an arbitrary value matches specific criteria and return Matcher objects of type Matcher<T>. Mockito matchers are static methods specific to when and verify that apply only to argument values, and return object of type T. Mockito matchers often implement Hamcrest Matcher interface providing standard hamcrest methods.

BaseMatcher is a base class for all the Hamcrest Matcher implementations. Some of the frequently used hamcrest matchers include ArgumentMatcher and TypeSafeMatcher. The ArgumentMatcher is a type of hamcrest Matcher which provides a predefined describeTo() method while an abstract method matches() to be implemented. TypeSafeMatcher implements null checks and checks for specific type before casting. CustomMatcher implements the describeTo() method providing the description of the object, with the remaining methods to be implemented. BaseMatcher can be extended directly to provide custom matcher implementations. The methods to implement are matches() which evaluates the matcher for an item and describeMatch() method which generates the description providing the reasoning for non accepted item.


Matcher Example:

    private static class HealthCheckMatcher extends BaseMatcher< Healthcheck > {

        private HealthCheck expected;

        HealthCheckMatcher(HealthCheck healthCheck) {
            assert healthCheck != null;
            this.healthCheck = healthCheck;
        }

        @Override
        public boolean matches(Object item) {
            if (!(item instanceof HealthCheck)) {
                return false;
            }
            HealthCheck actual = (HealthCheck) item;
            return expected.getKey().equals(actual.getKey()) && expected.getName().equals(actual.getName());
        }

        @Override
        public void describeTo(Description description) {
            description.appendText("HealthCheck with key: ").appendValue(expected.getKey())
                    .appendText(", test name: ").appendValue(expected.getName());
        }

        @Override
        public void describeMismatch(Object item, Description description) {
            description.appendText("was ");
            if (!(item instanceof HealthCheck)) {
               description.appendValue(item == null ? "null" : item.getClass());
               return;
            }

            description.appendValue(item);

            HealthCheck actual = (HealthCheck) item;
            
            if(!expected.getKey().equals(actual.getKey())) {
               description.appendText("Key mismatch. Expected: ").appendValue(expected.getKey())
                          .appendText(", Actual: ").appendValue(actual.getKey());
            }
            // Similarly equality check for expected.getName() and actual.getName().
        }
    }

    @Test
    public void healthCheckListInitializedOnlyReturnsThoseValuesExpected() {

        HealthCheck expectedHealthCheck = new HealthCheck("354", "Pulse per minute");

        // Check if an expected item is present in the list.
        Collection< HealthCheck > list = makeSomeCall();
        Assert.assertThat(list, HealthCheckMatcher.hasItem(expectedHealthCheck));

        // Check if the actualHealthCheck object matches the expected object. 
        // This is one way match two objects without any equals() method implementation.
        HealthCheck actualHealthCheck = makeSomeOtherCall();
        HealthCheckMatcher healthCheckMatcher = new HealthCheckMatcher(expectedHealthCheck);
        Assert.assertThat(actualHealthCheck, healthCheckMatcher);
    }

ArgumentCaptor is a specialised ArgumentMatcher that records the matched argument for later inspection using the capture() method. It enables to assert certain arguments after verifying actual call. First an ArgumentCaptor is created for the class we wish to inspect. Then the ArgumentCaptor is used as an ArgumentMatcher in the verify call. No matter what values the object contains, the ArgumentCaptor will always match thus allowing the verify call to succeed. After capturing the object, its values can be inspected by calling getValue() method and the original object been passed to the actual method can be accessed.

Agrument Capture Example:

      ArgumentCaptor< alerteventpreference > modelPrefCaptor = ArgumentCaptor.forClass(AlertEventPreference.class);    
      verify(unitOfWork).registerNew((modelPrefCaptor.capture()));  
      AlertEventPreference alertEventPreference = modelPrefCaptor.getValue();
      assertEquals(CAN_ALERT_ID, alertEventPreference.getAlertEventId());

In order to Capture an argument which is an object of Collection or List interface using ArgumentCaptor gives issues due to generic typed objects. This is resolved by using the @Captor annotation as shown below:

      @Captor
      private ArgumentCaptor< List< Machine > > machineListArgumentCaptor;    
      ....  
      verify(machineListBroker).populateEmbeds(eq(ANY_ORG_ID), machineListArgumentCaptor.capture(), anyList());
      assertEquals(machineListArgumentCaptor.getAllValues().get(0).size(), 2);

PowerMock
Powermock is the testing framework extending other standard libraries such as EasyMock and Mockito. It  uses a custom classloader and bytecode manipulation to enable mocking of static methods, constructors, final classes and methods, private methods, removal of static initializers.
Below is the required maven dependencies for using PowerMock:

    <dependency>
      <groupId>org.powermock</groupId>
      <artifactId>powermock-module-junit4</artifactId>
      <version>${org.powermock.version}</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.powermock</groupId>
      <artifactId>powermock-api-mockito</artifactId>
      <version>${org.powermock.version}</version>
      <scope>test</scope>
    </dependency>

PowerMock requires @RunWith(PowerMockRunner.class) annotation at the class level inorder to initialize powermock. Then @PrepareForTest annotation is required to tell PowerMock to prepare the specified classes for testing. The classes passed to @PrepareForTest annotation contains the static methods which are required to be mocked. In order to mock a static class PowerMockito.mockStatic() is called by passing the static class to be mocked. PowerMockito.when() method is used to return the expected value, do nothing or throw exception when invoked. Since PowerMockito.when() method just delegates to the original Mockito.when(Object) method, it can be used alternatively with Mockito.when() method. The PowerMockito.doNothing() is used for setting void methods to do nothing. The PowerMockito.doCallRealMethod() method is used to call the real implementation of the static method. Also Mockito matchers are may still applied to a PowerMock mock.

Static methods can be verified by first calling the PowerMockito.verifyStatic() to start verifying certain behavior followed by the call to the static method to be verfied. Mockito.VerificationMode can be used with PowerMockito.verifyStatic(Mockito.times(2)) to verify the exact number of calls on a static method.
Below is the sample test using PowerMock:

public class AppHelper {
 
  public static Integer getOrganizationId(String ldapID) throws HttpException {
   
    Integer organizationId = 0;
    ComponentRegistry.getInstance().getLog().writeTrace("getOrganizationId", "Retrieving org details for userid:" + 
                                                                              ldapID + " from Application");
    MaintainAccountService service = getMaintainAccountServiceProxy();

    OrgAccountsByUserIDIP orgAccountsByUserIDIP = new OrgAccountsByUserIDIP();
    orgAccountsByUserIDIP.setUserID(ldapID);
    OrgAccountsByUserIDOP response = service.getOrgAccountsByUserID(orgAccountsByUserIDIP);

    parseResponse(appController, response, businessKey);
    appPostProcess(appController,response);
 
    if(response != null && !response.getOrganizationAccount().isEmpty()){
      if(response.getOrganizationAccount().size() > 1) {
       throw new HttpException(HttpStatus.SC_PRECONDITION_FAILED,"Precondition failed");
      }
      else {
       organizationId = response.getOrganizationAccount().get(0).getId();
       ComponentRegistry.getInstance().getLog().writeTrace("getOrganizationId", "Got organization: " + organizationId + 
                                                           " for userid:" + ldapID + "     from Application");
      }
    }
   
    if(organizationId == 0){
      ComponentRegistry.getInstance().getLog().writeTrace("Organization Id is not found for the LDAP Id :"+ldapID);
    }
 
    return organizationId;
  }
  .............
}
 
@RunWith(PowerMockRunner.class)
@PrepareForTest({ComponentRegistry.class,AppHelper.class})
public class AppHelperTest {
 
  @Test
  public void powerMockTest() {

    PowerMockito.mockStatic(ComponentRegistry.class);
    PowerMockito.mockStatic(AppHelper.class);
 
    Log logMock = Mockito.mock(Log.class);
    Mockito.doNothing().when(logMock).writeTrace(Mockito.anyString(),Mockito.anyString());
    Mockito.doNothing().when(logMock).writeTrace(Mockito.anyString());
 
    ComponentRegistry registry = Mockito.mock(ComponentRegistry.class);
    Mockito.when(registry.getLog()).thenReturn(logMock);
    PowerMockito.when(ComponentRegistry.getInstance()).thenReturn(registry);
   
    MaintainAccountService service = Mockito.mock(MaintainAccountService.class);
    Mockito.when(service.getOrgAccountsByUserID(Mockito.any(OrgAccountsByUserIDIP.class))).thenReturn(createTestIP());
 
    PowerMockito.when(AppHelper.getMaintainAccountServiceProxy()).thenReturn(service);
 
    PowerMockito.doNothing().when(AppHelper.class,"parseResponse",Mockito.any(AppController.class), 
                                                  Mockito.any(OrgAccountsByUserIDOP.class), Mockito.anyString());
 
    PowerMockito.doNothing().when(AppHelper.class,"appPostProcess",Mockito.any(AppController.class), 
                                                  Mockito.any(OrgAccountsByUserIDOP.class));
 
    PowerMockito.doCallRealMethod().when(AppHelper.class,"getOrganizationId","user");

    PowerMockito.verifyStatic(AppHelper.class, Mockito.times(3));

    AppHelper.getOrganizationId("user");
  }
}

Acceptance Testing: Cucumber JVM

Cucumber-JVM is a java version of the popular Cucumber BDD tool for Ruby platform. The cucumber community (cukes) is one of the most vibrant community and are expanding the framework from Ruby to Java, .NET, Python, Perl, PHP etc. The core festures of cucumber being similar to jbehave with a story (feature) file, corresponding scenario implementation and an entry point class to execute all the stories. But there are some key differences between them which are discussed as follows:

1. All the parameters in the story are parsed using regular expression instead of parsing the parameter matcher in jbehave ($ by default).
2. JBehave allows to extend scenario implementation classes (i.e. StorySteps class) in order to reuse the scenarios. Cucumber on the other hand directly finds the scenario implementation for the story regardless of their classes, and blocks extending the implementation classes.
3. In JBehave, only single instance of the Step Definition class (scenario implementation classes) is maintained during the execution of the story retaining the values of instance variables. In cucumber though, for each scenario a new instance of the Step Definition class is created and all the previous values of instance fields are lost.
4. JBehave support annotations such as @BeforeStory and @AfterStory, which allow to execute the methods before and after the entire execution of the story respectively. Cucumber on the other hand has @Before and @After annotation which by default execute before or after every scenario. If a parameter is passed along with the annotation, such as @Before("@SETUP") or @After("@SETUP"), then the method will be executed before or after the scenario with the tag "@SETUP" in the feature file.
5. JBehave is flexible to have the Step Definition class (scenario method implementation) anywhere in the package structure, but requires the story entry point class (AllStories) to specify the instance of the class. On the other hand Cucumber-Jvm mandates to have the Step Definition classes in the same package of the story entry point class (AllStories). This enables cucumber to automatically find the implementation methods for the scenario specified in the story.
6. The tabular input format in Jbehave uses a ExampleTable class, which is a list of maps, each map representing a row, with table header's as the key to retrieve row values. In contrast to this approach, cucumber-jvm requires to create classes representing the table row structure. Cucumber then returns a List of Objects of the table type created earlier. This also helps to classify the text fields from the numeric fields using the data types of the instance variables.
7. The Configuration class for JBehave provides rich set of customization from Reports, Input Parameter converters, and story path. While cucumber does provide some of configuration options, major customization still doesn't seems to be straight forward.

The configuration of cucumber as mentioned above consists of an entry point class to execute all the features in the feature path. It loads the features using the Cucumber class providing options for execution and report generation. Below is the list of options available:

1. tags: specify the tagged scenarios and stories to execute or to skip. Only run scenarios tagged with tags matching TAG_EXPRESSION.
2. strict: Usually, when cucumber can’t find a matching Step Definition the step gets marked as yellow, and all subsequent steps in the scenario are skipped. The strict option causes Cucumber to exit with 1 for pending and undefined steps.
3. format: specifies how the results are formatted. Available formats: junit, html, pretty, progress, json, pretty:
   html: Generate an html report in the targeted location
   json: Generate a compact json report in the targeted location
   json-pretty: Generate a well formatted json report in the targeted location
   junit: Generate a cucumber junit report in the targeted location (xml format)
   progress: It causes a regular JUnit test to be stuck at yellow
    
4. features: specifies the path to the feature file (story). E.g. @Cucumber.Options(features = "classpath:simple_text_munger.feature")
5. glue: specifies the path where glue code (step definitions and hooks) is loaded from.
6. name: runs only the scenarios whose names match REGEXP.
7. dry-run: skips execution of glue code.
8. monochrome: doesn't color terminal output.

Below is the code which loads Cucumber feature files using Cucumber class with the options as described above.

@RunWith(Cucumber.class)

@Cucumber.Options(tags = { "~@WIP", "~@BROKEN" }, strict = true, 

     format = { "pretty", "html:target/cucumber", "json-pretty:target/cucumber.json" })

public class AllStories { }

Features in Cucumber-JVM are similar to the jbehave stories with Given-When-Then scenarios and support for tabular input as well. Further tags can be referenced in the feature file in order to tag scenarios and stories to execute or skip them.

@TESTS
Feature: Add a customer to the records.

@SETUP
Scenario: Customer account "John" is created with default settings.
Given a customer with the name "John" and table
        | ROW_ID | NAME | VALUE |
        | 3232323  | John12  | abc        |
        | 6454560  | John42  | xyz        |

When a customer tries to create an account
Then get an customer account id which is not null and greater than zero

Similar to Jbehave, cucumber also provides step definitions for execution of the scenarios in the features. As mentioned above, the step definition class cannot be extended for reuse, but cucumber automatically scans the package of its entry-point class, to find the step definitions for the corresponding scenarios. Further @StepDefAnnotation is used to mark the class of step definitions, later scanned by cucumber-jvm.

@StepDefAnnotation
public class OrgTerminalMachineSetupSteps{

@Before("@SETUP")
public void cleanup(){ ... }

@Given("^a customer with the name \"([^\"]*)\" and table$")
public void a_customer_with_the_name(String customerName, List<Row> list) throws Throwable { .. }

@When("^a customer tries to create an account$")
public void dealer_tries_to_create_an_account() throws Throwable { .. }

@Then("^get an customer account id which is not null and greater than zero$")
public void get_customer_accid_not_null_and_greater_thanzero() throws Throwable { .. }

  class Row {
    public String rOW_ID;
    public String nAME;           
    public String vALUE;
   }
}

For each step execution of the scenario in the feature, cucumber scans and finds the step definitions. Then it creates the instance of Step-Definition class before executing each scenario and executes the corresponding step methods. So in case the scenarios are required to be inter dependent in order to carry out an operation, all the instance fields of step definition class need to be singletons. So either a singleton factory class can be used to get field instance or spring can be used to inject such instances.
     Cucumber supports spring integration and requires "cucumber-spring" jar and "cucumber.xml" file in the source main resources directory. The cucumber.xml specifies the beans or component scans to load the beans required for cucumber acceptance tests. Also the spring config files can be imported into cucumber.xml for more organized configuration. The cucumber.xml is loaded by default using cucumber-spring before it initializes the step definition classes for tests execution. All the across scenario fields should be Autowired to grab the instances loaded by cucumber.xml. With such an spring integration, it allows to maintain the field instances across scenarios, access properties and take advantage of most of the spring related features.
     Moving ahead with the Jenkins setup for running the cucumber tests, it is necessary to run the tests in maven using maven-failsafe-plugin. The problem though with the failsafe-plugin is it requires all the tests to be inside the source test directory instead of source main directory. Although this seems logical as we are running tests and not any development code, it does require to load all the spring related beans from cucumber.xml in test resources by importing spring config files in the main resources directory. This seemed not to be working with both the spring configs (in test and main directories) and none of the beans were loaded. Copying all the spring related configuration files from the main resources folder to the test resources allows only to load/component scan the beans from the classes present in its codebase test or main. Hence the only solution we found is to copy all the source from main to test directory which seemed a lot of change. To avoid such major change for just running the tests using the maven-plugin, the configuration of the maven plugin was modified to load all the tests from the source main directory. Below are the changes and the config of the maven-failsafe-plugin:

 <plugin>

   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-failsafe-plugin</artifactId>
   <version>2.12</version>

   <configuration>
     <includes>
       <include>**/AllStories.java</include>
     </includes>
     <testSourceDirectory>${project.build.sourceDirectory}</testSourceDirectory>
     <testClassesDirectory>${project.build.outputDirectory}</testClassesDirectory>
     <reportsDirectory>${project.build.outputDirectory}/failsafe-reports</reportsDirectory>
     <additionalClasspathElements>
       <additionalClasspathElement>${project.build.sourceDirectory}/resources</additionalClasspathElement>
     </additionalClasspathElements>
   </configuration>

   <executions>
     <execution>
       <id>integration-test</id>
       <goals>
         <goal>integration-test</goal>
         <goal>verify</goal>
       </goals>
     </execution>
   </executions>

 </plugin>

The above changes in the testSourceDirectory and testClassesDirectory causes the maven-plugin to change its path to load the tests from the source main directory, thus running the acceptance test. Moving on to the Jenkins configuration, cucumber provides a nice plugin for Jenkins which enables it to provide well organized reports. The configuration for the Cucumber-Reports (latest version 0.0.14) Jenkins plugin is very simple as described in the documentation. The Json Report generated is usually in the target folder by default, hence we specify "Json Reports Path" as target. Also the "Plugin Url Path" is used to make the ""Back To Jenkins" link work in the Cucumber Reports by pointing to the right Jenkins Url.

One important note while running the Cucumber-Reports plugin: In the feature file if there is only Scenario wihout any Given-When-Then statements, then the cucumber tests fo run and generate the report in json. But the generated json report cannot be parsed by the cucumber-reports and it throws below exception,

[CucumberReportPublisher] Compiling Cucumber Html Reports ...
[CucumberReportPublisher] copying json from: file:/c:/.jenkins/workspace/cucumber-acceptance-tests/to reports directory: file:/e:/.jenkins/jobs/cucumber-acceptance-tests/builds/2012-11-01_16-13-02/cucumber-html-reports/
[CucumberReportPublisher] Generating HTML reports
ERROR: Publisher net.masterthought.jenkins.CucumberReportPublisher aborted due to exception
java.lang.NullPointerException
at net.masterthought.cucumber.util.Util.collectSteps(Util.java:104)

The reason behind it is the cucumber-reports plugin expects the scenarios to at least contain a Given statement in order to parse the generated json report successfully. Hence if we specify the scenario with atleast a Given step as below, the cucumber-reports jenkins plugin generates the report successfully.


@SETUP
Scenario: Setup.
Given Something

Although there are still some unresolved issues with the Cucumber-Reports Jenkins plugin. In the Cucumber-Reports in the Feature Statistics table, the time duration populated is "35 secs and 55 ms" but actually its supposed to be around 30 minutes. Also in the Feature Report details we see a message such as "Result was missing for this step". This message is displayed because the json report generated by cucumber doesn't have the result section in the report for every step: "result": { "duration": 776000, "status": "passed" }
If cucumber-jvm version 1.0.14 the json report does not have result section, but if 1.0.8 or 1.0.9 is used the json report does contain the result section. The cucumber-report plugin both version 0.0.14 and 0.0.12
cannot parse the result section the json report generated and the issue still persists. An quick fix will be to try using cucumber-reports jenkins plugin version 0.0.9 as shown the web documentation or wait till the issue is resolved in the later versions.

Acceptance Testing: JBehave

Acceptance testing is one of the crucial phase in product testing as it determines whether the system operates based on the specifications set. It ensures that the system functions as expected, integrating with numerous components/services to provide accurate results. Such automated testing of the product as a whole, based on a pre-decided set of scenarios (mainly from testers) ensures that we catch the faults before the manual testing takes over. It not only saves time for both testers/developers but also boosts developer confidence while making crucial changes in legacy code. There could be various approaches followed to write acceptance tests. Either the data needed for the test is created from scratch in a regular or in-memory database before and deleted once the test is completed in case of the regular database, or a static database for acceptance test is used\maintained were the required data needed for the test is essentially always present.

  There are 5 core principles for writing acceptance tests mentioned as below:

1. Acceptance tests should be isolated and external to the application under test.
2. Acceptance tests should be executed against the live application.
3. Acceptance tests should be independent of any development environments.
4. Acceptance tests should always be executed against actual data.
5. Acceptance tests should imitate the manual verification criteria.

  With all said about the advantages of acceptance testing, there are two major Java frameworks supporting such testing, mainly JBehave and Cucumber. Both have a basic idea of writing stories which contain various test scenarios, using Give-When-Then clauses. All the scenarios are executed by mapping Give-When-Then clauses to corresponding methods and executing the mapped methods based on the order in the story. Upon the completion of execution, a report is generated based on the story and providing the execution results. But    JBehave and Cucumber are differ in some aspects of their workings. JBehave on one hand requires the story to be tightly coupled with its java implementation class, Cucumber only requires such coupling based on the scenarios in the story, irrespective of its implementation class. Lets dive in to have a closer look at each of the frameworks.

JBehave

JBehave is been quite a framework for acceptance testing has most of the basic set of features such as reusing scenarios, skip scenarios, html/json/xml/text reporting, running multiple stories, jenkins plugin etc.

In maven world, jbehave can be configured by adding a dependency in the pom.xml for "jbehave-core" (version=3.6.8) in the group "org.jbehave". Also in order to execute all the stories using maven (mvn integration-test) a plugin entry must be added in the plugins section as follows:

      <plugin>

        <groupId>org.jbehave</groupId>

        <artifactId>jbehave-maven-plugin</artifactId>

        <version>${jbehave.core.version}</version>

        <executions>

          <execution>

            <id>unpack-view-resources</id>

            <phase>process-resources</phase>

            <goals>

              <goal>unpack-view-resources</goal>

            </goals>

          </execution>

          <execution>

            <id>embeddable-stories</id>

            <phase>integration-test</phase>

            <configuration>

              <includes>

                <include>${embeddables}</include>                        <!-- include all stories -->

              </includes>

              <excludes />

              <storyTimeoutInSecs>5200</storyTimeoutInSecs>

              <generateViewAfterStories>true</generateViewAfterStories>

              <ignoreFailureInStories>false</ignoreFailureInStories>

              <ignoreFailureInView>false</ignoreFailureInView>

              <threads>1</threads>

              <metaFilters>

                <metaFilter>-skip</metaFilter>     <!-- specify annotation to filter and skip the scenario -->

              </metaFilters>

            </configuration>

            <goals>

              <goal>run-stories-as-embeddables</goal>

            </goals>

          </execution>

        </executions>

      </plugin>

Once maven is configured and ready, we can write story scenarios and its implementation. Now, in order to invoke all the stories from Eclipse, a java class inheriting JUnitStories is implemented which specifies the similar configuration as in the maven plugin above.

   public class AllStories extends JUnitStories {


    public AllStories() {

        configuredEmbedder().embedderControls()

        .doGenerateViewAfterStories(true)

        .doIgnoreFailureInStories(false)            // stop rest of the scenarios if any scenario fails

        .doIgnoreFailureInView(false)              //

        .useThreads(1)                                     // specify number of threads to use

        .useStoryTimeoutInSecs(300);           // story execution timout in seconds

        // specify annotation to filter and skip the scenario

        configuredEmbedder().useMetaFilters(Arrays.asList("-skip"));

    }


    public Configuration configuration() {

        Class<? extends Embeddable> embeddableClass = this.getClass();

        // Enables to decorate and format non-Html reports

        Properties viewResources = new Properties();

        viewResources.put("decorateNonHtml", "true");

        // Start from default ParameterConverters instance

        ParameterConverters parameterConverters = new ParameterConverters();

        // factory to allow parameter conversion and loading from external resources (used by StoryParser too)

        parameterConverters.addConverters(new DateConverter(new SimpleDateFormat("yyyy-MM-dd")));

        return new MostUsefulConfiguration()
        

   .useStoryControls(new StoryControls().doDryRun(false).doSkipScenariosAfterFailure(true))

            .useStoryLoader(new LoadFromClasspath(embeddableClass))

            .useStoryPathResolver(new UnderscoredCamelCaseResolver())

            .useStoryReporterBuilder(new StoryReporterBuilder()

                .withCodeLocation(CodeLocations.codeLocationFromClass(embeddableClass))

                .withDefaultFormats()

                .withPathResolver(new ResolveToPackagedName())

                .withViewResources(viewResources)

                // generates report in the following formats

                .withFormats(CONSOLE, TXT, HTML, XML)

                .withCrossReference(xref)

                // displays full exception stacktrace in the generated report

                .withFailureTrace(true).withFailureTraceCompression(true))

            .useParameterConverters(parameterConverters);
    }


   // Specify the class which implements the methods mapped to the scenarios

    public InjectableStepsFactory stepsFactory() {

        return new InstanceStepsFactory(configuration(), new Object[] { new StorySteps() });

    }


    // Specify the relative path to the stories with the stories to include and exclude.
    protected List<String> storyPaths() {
    
      String codeLocation = codeLocationFromClass(this.getClass()).getFile();    

      return new StoryFinder().findPaths(codeLocation, Arrays.asList("**/**/*.story"), Arrays.asList("**/excluded*.story"));

    }

The configuration above enables report generation by setting doGenerateViewAfterStories to true. It sets jbehave to stop the execution of scenarios in case of failure in executing any scenario. The execution is configured to run on a single thread in order to show an accurate execution duration in the report. Also it prints the scenario statements one by one with the debug results providing clear understanding. In order to prevent timeout of the story due to long service calls, it is set to a comfortable amount of 5 minutes. Also a "@skip" meta matcher is added in order to skip the scenarios in the story.
    In the configuration method we specify the properties in order to format non-Html reports. The formats in which reports are generated are, Text, Html, Xml and in Eclipse/Command console. The stack trace is enabled in the report on failure of the scenario using the withFailureTrace() method.

    After the configuration and story loader class is ready, we write the actual story with scenarios as follows:

Story: A customer with name "John" needs to setup a account.

Scenario: Customer account is created with default settings.
Given a customer with the name "John" with
| a | b | c |
| 1 | 0 | 1 |
| 2 | 6 | 4 |
When customer tries to create an account
Then get an customer account id which is not null and greater than 0
........
Scenario: ......

Meta: @skip
Given ......

Note that in the last scenario above we use the Meta information providing the property with the name "skip" but no value. The meta matchers can also be used as name-value pair such as "@ignore true".  The order of the scenarios in the story is the order of the execution of the scenarios.
   Moving forward we write the corresponding implementation for the scenarios in a class which is referenced in the stepsFactory() method of AllStories class. Annotations such as @Given, @When, @Then are used to bind the methods to the corresponding scenario's Given, When, Then clause. The @BeforeStory and @AfterStory annotations are used to initialize the story and clean up after its execution respectively. It is important to know that the statements following Given-When-Then in the story should match the ones in the annotations in order for the method to bind to the corresponding statement. Further we use quotes to highlight the parameters and "$" to identify the parameters for parsing. The identifying character for the parameter can be changed to "%" for example using the following statement in the configuration of AllStories class.
return new MostUsefulConfiguration().useStepPatternParser(new RegexPrefixCapturingPatternParser("%")) 

Further, the parameters parsed using "$" from the variables in the story are assigned to the method's parameters of the type String, Integer or ExampleTable. Any text in the appropriate position is converted to String while number is converted to Integer. The table specified is converted to ExampleTable, one of the JBehave object types. ExampleTable is mainly a list of Maps consisting of values from the table assigned to the header acting as the key for the Map.

public class StorySteps{

 ...........

@BeforeStory

public void initialize(){ .... }

 ................

@Given("a customer with the name \"$customerName\" with $someTable")

public void setupOrg(String customerName, ExamplesTable someTable) throws Exception {

    ....

    List<Map<String, String>> rows = someTable.getRows();

for (Map<String, String> row : rows) {

             String a = row.get("a");

             ......

        }
 }


@When("dealer tries to create an account")
public void whenCustomerTriesToCreateAnAccount() throws Exception {


@Then("get an customer account id which is not null and greater than $number")
public void thenGetAnCustomerAccountIdWhichIsNotNullAndGreaterThan(Integer number) {

    Assert.assertThat...

}


@AfterStory
public void cleanUp() throws Exception  { ... }

 .............

}

The reports generated by jbehave are impressive providing a list of all stories along with their execution time, total scenarios, success, failures etc. Each story then provides the details of its scenarios as Given-When-Then and, colored Green for success and Red with stacktrace for failure. The only odd thing in the report is the Given section which when in a tabular form in the story, gets converted to a chunk of text without indentation and spacing. Even using the "{trim=false}" property before the table doesn't work to preserve the spacing of the columns on the report.

Jbehave also provides a plugin for Jenkins, Continuous Integration system, which parses the report generated in xml format to provide Test statistics similar to Junit. The configuration is simple, in the "Post-build Actions" add "Publish testing tools result report", then add "JBehave-3.x". Usually the pattern "**/jbehave/*" works, but with more specific pattern such as "**/jbehave/stories.*.xml" it certainly works. The jenkins report of jbehave is nothing fancy but a list of all the scenarios executed or failed, and the current testing trend.

Next we will continue with our next discussion on Cucumber-JVM Framework.