Cucumber Retrospective

I’ll be presenting a lightning talk on my experience to date using Cucumber at Tuesday’s Tester Meetup. Since I needed to keep it short and sweet, I’ve kept the scope of my talk to a kind-of plus & minus format. My notes are below in advance of the presentation. I’ll add additional comments if I get some useful feedback on the day.

Things I learnt about Cucumber (and test automation generally) from recent project experience.

The Good

[1] Great feedback

Continuous integration testing using Cucumber acceptance tests resulted in high level of confidence in delivered builds meeting known requirements/specifications (i.e. passing tests designed from requirements).

Although – the same degree of confidence could be obtained from any [correctly implemented] automated test framework.

The Bad

[1] Mind-numbingly boring to write;

Admittedly this is likely due to the manner in which the tests were scripted.

Lack of immediate feedback was also an issue for me since I scripted the tests and then passed them to developers who implemented them – this meant I was removed from the results of the tests.

There were a lot of them.

[2] Governance;

Implementation and execution of acceptance tests needs to be monitored otherwise, as in my case, scripts don’t get implemented and/or failing tests get de-scoped rather than fixed.

[2] Checks not tests;

Tests have either passed or failed, and if they have failed are fixed by developers with little involvement from me (fortunately communication with developers was good so I did get plenty of useful information from them). Not as much root cause analysis as I would potentially carry out with own automated tests.

For example – I built my own layer of sanity checking in addition to the acceptance tests using JMeter, giving me the immediate feedback desired and the ability to identify faults myself.

[3] High maintenance cost:

In the early stages of a project where business requirements may not be fully understood (even by the business), many tests are likely to require further attention. Early tests are likely to need refactoring in later stages of the project, both from a Cucumber script perspective and from a step implementation [Java in our case] perspective.

Tests must be named/modularised coherently in order to track coverage effectively. When requirements change or are added to etc and the application grows, tests may become difficult to track.

The Ugly

[1] Quote from Cuke4Ninja by David de Florinier & Gojko Adzic – “… use [Cucumber] to automate functional validation in a form that is easily readable and understandable to business users, developers and testers.”

Our tests often looked something like the below, to enable some degree of dynamism in field validations (pairwise tests could be similarly implemented) – coming in quite useful when incorporating 3rd party changes to said validations.

@changeBankDetails
@High                          
Scenario Outline: 40 new bank details validation
Given I load data file "classpath:changebankdetails_data.feed" and use rowid: '1'
And I am logged in with userID <table2.userID> password <table2.password>
And I am on the 'Enter new details' page
And I see And I see |checkbox |contractNumber  |
                    |checked  |<<<TBA>>>       | # any preferred contract can be hardcoded here
And I see |sortCode  |accountName|accountNumber|rollNumber|
          |<sortcode>|<acctName> |<acctNum>    |<rollNum> |  
When I click 'Update selected contracts'
Then I see <message> 

Examples:
|rowID|sortcode|accountName|accountNumber|rollNumber |message|
|1    |        |           |             |           |You must enter the following information to proceed: Sort Code Account Name Account Number B/soc number|
|2    |123456  |           |             |           |You must enter the following information to proceed: Account Name Account Number B/soc number|
|3    |        |MR A Smith |             |           |You must enter the following information to proceed: Sort Code Account Number B/soc number|
|4    |        |           |123456789    |           |You must enter the following information to proceed: Sort Code Account Name B/soc number|
|5    |        |           |             |12345678911|You must enter the following information to proceed: Sort Code Account Name Account Number|
|6    |        |MR A Smith |123456789    |12345678911|You must enter the following information to proceed: Sort Code|
|7    |123456  |           |123456789    |12345678911|You must enter the following information to proceed: Account Name|
|8    |123456  |MR A Smith |             |12345678911|You must enter the following information to proceed: Account Number|
|9    |123456  |MR A Smith |123456789    |           |You must enter the following information to proceed: B/soc number|
|10   |12345   |MR A Smith |123456789    |12345678911|Sort code must be six digits|
|11   |1234567 |MR A Smith |123456789    |12345678911|Sort code must be six digits|
|12   |ABCDEF  |MR A Smith |123456789    |12345678911|Sort code must be six digits|
|13   |!!!!!   |MR A Smith |123456789    |12345678911|Sort code must be six digits|
|14   |"      "|MR A Smith |123456789    |12345678911|Sort code must be six digits|
|15   |123456  |MR A Smith |123456       |12345678911|Account number must be seven or eight or nine digits|
|16   |123456  |MR A Smith |1234567891   |12345678911|Account number must be seven or eight or nine digits|
|17   |123456  |MR A Smith |ABCDEFGHIJ   |12345678911|Account number must be seven or eight or nine digits|
|18   |123456  |MR A Smith |!!!!!        |12345678911|Account number must be seven or eight or nine digits|
|19   |123456  |MR A Smith |"      "     |12345678911|Account number must be seven or eight or nine digits|
|15   |123456  |"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"|123456789    |12345678911|Account name is a maximum of 31 characters|
|15   |123456  |MR A Smith |123456789    |1234567890123456789|Building Society reference is a maximum of 18 character|

Business users are unlikely to want to read such “codified” tests however. In fact, good BDD philosophy states that such a test should instead be implemented in a more readable fashion, e.g:

Given I am logged in as a registered user
And I am on the Enter New Bank Details page
When I change my Bank details to an invalid Account Name
Then I see the message “Account name is a maximum of 31 characters”

However my present understanding of the way in which Cucumber is actually implemented leads me to believe that writing tests in the fashion above would simply mean that the detail (e.g. the invalid information inserted into the account name field to make it invalid) must be hidden in the underlying implementation code (Java in our case) instead. This may be ok for the business user, but it means that the usefulness of the test (the iterations through various invalid entries and checking of different error messages) has been buried in the implementation steps and further scripts will be required to check the same behaviour, rather than one multi-iteration script as before.

I’m not presenting myself as any kind of an expert, in fact I’m off down to the BDD Exchange event at Skills Matter on Friday to try and educate myself further! The above is just my reflections on my Cucumber experience thus far. I’d appreciate comments[/corrections] if you have them.

- Simon

4 thoughts on “Cucumber Retrospective

  1. Hi Simon

    Thanks for posting – Cucumber is something we want to trial, so this was an interesting read for me.

    One question I do have is around loosely defined requirements at the beginning of the project.

    Isn’t the point of cucumber that the tests are effectively the requirements? So if the requirements do change then it’s a good thing the acceptance checks fail – rather than having a stagnant set of requirements in a word doc?

    Or am I missing something?

    Good luck for your meetup! Are you going to post some videos?

    1. In a “truly” agile environment – yes that would be the case. However only the web application was delivered in a quasi-agile fashion. The underlying database was being delivered by a large consultancy in a distinctly waterfall fashion, with documented requirements up-front delivered to us some way downstream. Not ideal.

      Very much a learning curve for me in any event since it was my first experience using Cucumber and I certainly wouldn’t say I’m anywhere near mastering it!

Leave a Reply

Your email address will not be published. Required fields are marked *

GiottoPress by Enrique Chavez