How detailed should a customer acceptance test be?

Question

Here is a test description, testing the "Create New Widget" use-case.

• Confirm that you can enter a new widget into the system.

---

Here is another test description, testing the "Create New Widget" use-case.

• Bring up the application.
  
• Create a new widget by the name of "A-008", with the description being "Test Widget for Acceptance Test 3-45".
  
• Confirm that the widget is now visible in the leftmost widget tree view.
  
• Select another widget in the tree view, then select the widget "A-008" again. Confirm that the values in the widget display equal the values you entered.
  
• Delete widget "A-008" and close the application

---

Here is another test description, testing the "Create New Widget" use-case.

• Bring up the application.
• Bring up a second instance of the application viewing the same data.
  
• In the first instance of the application, right-click on the "Widgets" node. In the ensuing context menu, activate the "Create New Widget" menu item.
  
• A "New Widget" window should be activated. Leave every field blank, and press the OK button. A message box should come up saying "Please enter a Widget name". Press OK on this message box.
  
• Enter "A-008" in the "Name" field.
  
• Set the description field to "The llama (Lama glama) is a South American camelid, widely used as a pack animal by the Incas and other natives of the Andes mountains. In South America llamas are still used as beasts of burden, as well as for the production of fiber and meat. The height of a full-grown, full-size llama is between 5.5 feet (1.6 meters) to 6 feet (1.8 meters) tall at the top of the head. They can weigh between approximately 280 pounds (127 kilograms) and 450 pounds (204 kilograms). At birth, a baby llama (called a cria) can weigh between 20 pounds (9 kilograms) to 30 pounds (14 kilograms).
• Press the OK button. A message box should appear saying "The description must be 512 characters or less"
  
• Set the description to "'); DELETE FROM WIDGET WHERE 1=1;" in the "Description" field. Press the OK button.
• In the left-most tree view, a new widget by the name of "A-008" should have appeared.
  
• Activate a window in the second instance of the application, and confirm that Widget "A-008" has automatically appeared in that tree view as well.
  
• In the first instance of the application, right-click on the "Widgets" node. In the ensuing context menu, activate the "Create New Widget" menu item. A "New Widget" window should be activated.
  
• Set the name to "A-008", and press OK. A message box must come up, saying "A Widget with this name already exists. Please select another Widget name".
  
• Press the OK button on this message box, then press the Cancel Button to exit the "Create Widget" dialog box.
• Display the widget page for widget "A-008" in the second instance.
• In the first instance, press the "Undo" menu item
• Confirm that the second instance is now displaying the start page.
• .................etc..............

Each example tests that you can create a new widget. In the third test, I was testing the functionality as an experienced programmer, thinking "OK, where are all of the places a bug can appear", and checking every one of these. Is the third one appropriate for a customer acceptance test?

How comprehensive is "too comprehensive"?

Answer 1

The user acceptance test cases should be detailed and simple but not as detailed as your third example. Acceptance testing is about making sure the customer gets what they agreed to. If you simply say, "click this then click that, etc, etc..." that is more like a functional test. You are not eliciting users to verify that functionality meets the test case laid out in the acceptance test. You are only asking them to click through tests that you could have simply automated.

User acceptance tests should be more along the lines of "create widget, verify that it appears, delete widget, etc." This will also encourage users to seek out individual features and (as a side effect) flush out any usability problems you may have overlooked.

Answer 2

I think that your acceptance tests should primarily be good path tests. Sometimes the "good" path will ensure that errors are properly handled. You should have other tests that validate your security and exercise the corner cases, but an acceptance test is more about making sure that the correct application has been built than making sure that every possible condition is handled correctly. If you have good unit tests and use best practices, then I think that the good path testing is entirely appropriate.

For example, I wouldn't necessarily test that I don't have problems with SQL injection if I've used a technology that enforces parameterized queries or where I generate queries by hand (I don't), that the unit tests validate that injection fails. Addressing the corner cases in unit tests makes it less important to focus on them in acceptance tests. If you need to include a few as examples to the customer that your backend implementation addresses their concerns, then by all means do so, but I wouldn't test things that I know I've addressed adequately via other testing.