Why I dislike Cucumber - BDD differently

At the end of last year we started using BDD process at Masabi as part of our future continuous delivery system. We are using Cucumber as probably the most popular BDD framework.
With some Specs2 experience (at Strollimo), I shortly realised, I do not like Cucumber because of:

1. The feature files
2. The global step definitions
3. The missing feature scope
4. Tooling: Junit integration, runner, maven integration, test separation

The programmable feature file

Let's start with the first two. Personally I don't like the feature files at all. I don't mean that I don't like the Feature/Scenario/Given/When/Then structure. I mean I don't like that this structure is separated out into files. This separation has two consequences:

1. the feature files are programmable, the wording of a sentence determine the behaviour, you can not write your specification freely in English, because you shold reuse the sentences.
2. Changes must be performed on two places:in the feature files and in the step definitions.

And here we are at the global step definitions. The global scope make the feature file programming (1) even worse. The sentences are reused globally but sometimes the same sentence means completely different behaviour, therefore we must apply small changes on the sentence to make it unique. Eg. Given a user logged in. We had to create two sentences: Given a user logged in to the web page, Given a user logged in to the mobile client, even though the feature unambiguously identified the target system. On the other hand, the sentences should be remembered after month/years , otherwise different sentences will mean same behaviour implemented potentially in different places.

Other problem with the reuse, that everybody is fond of the code reuse. Nobody wants to write a step definition twice. In order to achieve the code reuse many people make the step definitions parametrized, nicely reusable; unfortunately it makes the feature description less readable. The worst case is probably the "touch <<css parameter selector >>" style steps. But I can't blame anybody apart from the system forcing the step definition reuse.

What is the solution?

I'm more and more convinced that the feature file should not be programmable. The feature releated step definitions would make thing a bit better. (It is possible but really painful in cucumber-JVM) I believe the right approach is what Specs2 does. The specification is copied into the code and the sentences are used for logging not for programming.

   import org.specs2._

  class BDDAsIWantSpec extends Specification { def is = s2"""

    This is normal scenario declaration
    where we describe all the steps
      Given I am a user                             $pickUser
      When I want to log in                         $loginLogic
      Then I get to the My Account page             $verifyAccountPage
                                                        """

    def pickUser = ...
    def loginLogic = ...
    def verifyAccountPage = ...
  } 

Abstraction

The problem above can also be described as the lack of abstraction. When the test is written, sometimes there is a need to express abstraction in the test to avoid code reuse. For example at Masabi we wanted to write "Then I purchase a Single Ride ticket", but this sentence actually represents a much more complex scenario, where the user must select the product filters (origin and destination station), the product and provides her payment card details.
I believe the easiest tool to express such an abstraction is a programming language because those are designed for it. We could add abstraction to Gerkin and we could start building a new programming language around the current descriptive language but why shall we do if we have many of them already?

What is the solution?

The solution is quite straightforward: Use programming language! In a programming language you can easily solve the abstraction problem moving code snippets around, pushing up or down with safe refactoring operations.

Feature scope

In cucumber the fixtures are handled on Scenario (or Step) level. I like this as default behaviour but sometimes, for slow tests it would be great to have feature (or even higher) level fixtures. We are using cucumber for end-to-end testing and we had only two possibilities: setting up the environment before the tests or setting up the environment for each scenario. The second is definitely more reliable, it guarantees that each test will run always on the same initial state but it is VERY slow. The first solution - one environment for all tests - is also not good because certain properties must be changed - eg. timeouts - and controlling the initial state of each test is not easy.

Solution

Feature level before and after methods, potentially feature and scenario scopes in the dependency injection system.

Tooling

I don't want to analyse the weaknesses of cucumber tools but I want to mark:

• No Maven plugin for cucumber (like SBT plugin)
• No proper isolation: cucumber is used for end-to-end tests but the tests are not isolated, the static states are visible everywhere, memory leaks can cause out of memory exception, etc.
• Running only one scenario is not easy.
• HTML reporting is ugly.

In order to integrate with JUnit (and Maven) you have to add a JUnit test file, and hack the integration tests around not to run on mvn install.

The Future

I started an experimental project addressing the problems above. This project uses Java8 lambdas but if there is high demand I have idea how to implement the functionality in Java 5/6/7. Java 8 will be released next week. Come an join to the project: MByHave8 on BitBucket.