Should BDD scenarios include actual test data, or just describe it?

Question

We've come to a point where we've realised that there are two options for specifying test data when defining a typical CRUD scenario:

Option 1: Describe the data to use, and let the implementation define the data

Scenario: Create a region     Given I have navigated to the "Create Region" page       And I have typed in a valid name       And I have typed in a valid code     When I click the "Save" button     Then I should be on the "Regions" page      And the page should show the created region details 

Option 2: Explicitly state the test data to use

Scenario: Create a region     Given I have navigated to the "Create Region" page       And I have filled out the form as follows         | Label | Value  |         | Name  | Europe |         | Code  | EUR    |     When I click the "Save" button     Then I should be on the "Regions" page      And the page should show the following fields         | Name   | Code |         | Europe | EUR  | 

In terms of benefits and drawbacks, what we've established is that:

Option 1 nicely covers the case when the definition of say a "valid name" changes. This could be more difficult to deal with if we went with Option 2 where the test data is in several places. Option 1 explicitly describes what's important about the data for this test, especially if it were a scenario where we were saying something like "has typed in an invalid credit card number". It also "feels" more abstract and BDD somehow, being more concerned with description than implementation.

However, Option 1 uses very specific steps which would be hard to re-use. For example "the page should show the created region details" will probably only ever be used by this scenario. Conversely we could implement Option 2's "the page should show the following fields" in a way that it could be re-used many times by other scenarios.

I also think Option 2 seems more client-friendly, as they can see by example what's happening rather than having to interpret more abstract terms such as "valid". Would Option 2 be more brittle though? Refactoring the model might mean breaking these tests, whereas if the test data is defined in code the compiler will help us with model changes.

I appreciate that there won't be a right or wrong answer here, but would like to hear people's opinions on how they would decide which to use.

Thanks!

Answer 1

I would say it depends. There are times when a Scenario might require a large amount of data to complete a successful run. Often the majority of that data is not important to the thing we are actually testing and therefore becomes noise distracting from the understanding we are trying to achieve with the Scenario. I started using something I call a Default Data pattern to provide default data that can be merged with data specific to the Scenario. I have written about it here:

http://www.cheezyworld.com/2010/11/21/ui-tests-default-dat/

I hope this helps.

Answer 2

I prefer option 2.

To the business user it is immediately clear what the inputs are and the outputs. With option 1 we don't know what valid data is, so your implementation may be wrong.

You can be even more expressive by adding invalid data too, when appropriate

Scenario: Filter for Awesome     Given I have navigated to the "Show People" page     And I have the following data     | Name  | Value  |     |  John | Awesome|     |  Bob  | OK     |     |  Jane | Fail   | When I click the "Filter" button Then the list should display         | Name   | Value   |     |  John  | Awesome | 

You should however keep the data so its described in terms of the domain, rather that the specific implementation. This will allow you to test at different layers in your application. e.g. UI Service etc..