Using SpecFlow for End-to-End Regression Testing

Question

We are employing BDD and using SpecFlow to drive our development (ATDD).

Our QA team would like to define their own 'end-to-end regression tests (in Gherkin/SpecFlow) and re-use the Steps we have already defined.

(Please note - I know that this is not a great example but it should provide enough details)

A test may include..

1. Log in
2. Search for a product
3. Select a product to buy
4. Create an order
5. Select delivery option.
6. Submit the order.
7. Cancel the order.

This would suggests a scenario like..

Given I am logged in
When I Search for a product
And I Select a product to buy
And I Create an order
And I Select delivery option
And I Submit the order
And I Cancel the order
Then ??!!

Which is clearly wrong as we are not checking the output at each step.

So this may be resolved as a sequence of scenarios:

Scenario 1:
Given I am logged in
When I Search for a product
Then I see a list of products

Scenario 2:
When I select a product to buy
Then I can create an order

Scenario 3:
When I create the order
And I Select delivery option
Then I can submit the order

etc etc

The main issue with this is that there seems no way to specify the order/sequence that the scenarios are run in (a characteristic of nUnit?). Because there are dependencies between scenarios (they are not set to a know starting point) they must be run in sequence.

My questions are:

a) Are we trying to fit a square peg in a round hole?!

b) Does anyone know if there is a way to use SpecFlow/Gherkin in this way?

c) Or does anyone know what alternatives there are?

Many thanks!

Answer 1

I would say that you are writing your scenarios on the wrong abstraction level. But that depends on what you want to use them for;

If you want to write test-scripts then you are on the right track... but it will be a nightmare to maintain as it, in the first case (long script) will be very brittle and the second case (several scenarios) need to ensure a certain execution order. Both of them are discouraged and considered anti-patterns.

I would suggest that you merge the ATDD-tests you are writing and talk to the test department to get their view on the matter and include the test-cases they need to ensure that the system is thoroughly tested. Who know? You might even learn something from each other :P

And when you write those "specifications" (as I rather call them) you write them on a higher level. So instead of writing:

Given I am logged in
When I Search for a product
  And I Select a product to buy
  And I Create an order
  And I Select delivery option
  And I Submit the order

you write something like

When I submit an order for product 'Canned beans'

In the step-definitions behind that step you perform all that automation (login, browsing to the product page, select the delivery options, submit the order).

All of this can be read about in these great articles on how to write maintainable UI Automation tests:

• http://gojko.net/2010/04/13/how-to-implement-ui-testing-without-shooting-yourself-in-the-foot-2/
• http://elabs.se/blog/15-you-re-cuking-it-wrong
• http://www.marcusoft.net/2011/04/clean-up-your-stepsuse-page-objects-in.html
• http://dhemery.com/pdf/writing_maintainable_automated_acceptance_tests.pdf
• http://gojko.net/2010/01/05/bdd-in-net-with-cucumber-part-3-scenario-outlines-and-tabular-templates/
• http://chrismdp.github.com/2011/09/layers-of-abstraction-writing-great-cucumber-code/
• http://benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories.html
• http://mislav.uniqpath.com/2010/09/cuking-it-right/

I hope this helps