Can use Specflow scenario with Visual Studio 2013 Load Test

Question

I plan to reuse existing Specflow scenarios (Currently is using for acceptance and automated test) for VS Load Test as well, to avoid duplication and extra work. Specflow works fine for those test since it runs them once but in context of Load test when it executes each Specflow scenario more than one time and parallel it runs into issues and errors and with higher number of user it gets more

These errors can fail some of the test which at end creates incorrect test result, for instance using one Specflow scenario as test scenario with load test of 20 users and time period of 2 minutes can caused 50 errors similar to below. So test result shows that particular scenario is executed 200 times where 150 passed and 50 failed test and failure is caused by Specflow errors. In context of Load test this result is totally wrong and incorrect since the test itself has issues.

Error message:

ScenarioTearDown threw exception. System.NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object.
TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.HandleBlockSwitch(ScenarioBlock block)
TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(StepInstance stepInstance) TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.Step(StepDefinitionKeyword stepDefinitionKeyword, String keyword, String text, String multilineTextArg, Table tableArg)
TechTalk.SpecFlow.TestRunner.Then(String text, String multilineTextArg, Table tableArg, String keyword)

After some investigation it seems Specflow cannot generate and run same scenario parallel which caused this conflict and fails some test but I also have some doubt about that and seeking to see if there is any workaround about this or if I am missing anything and wondering if Specflow scenarios can be used for Load Test at all?

Answer 1

I understand wanting to reuse your tests for load testing (Don't Repeat Yourself), however a load test has a very different purpose than acceptance tests. Load tests should take realistic every day usage scenarios, and throw increasing numbers of users at them. For this reason I would urge you to keep your load tests separate from your acceptance and automated tests. They really are testing different things.

Load tests should test the performance of the application under high usage for every day scenarios, and acceptance and automated tests make sure the application is functioning according to spec.

Load testing is the process of putting demand on a system or device and measuring its response. Load testing is performed to determine a system’s behavior under both normal and anticipated peak load conditions. It helps to identify the maximum operating capacity of an application as well as any bottlenecks and determine which element is causing degradation.

Source: Wikipedia: Load testing

An acceptance test is a formal description of the behaviour of a software product, generally expressed as an example or a usage scenario. ... For many Agile teams acceptance tests are the main form of functional specification; sometimes the only formal expression of business requirements. In other cases, they merely complement a specification document resulting from a less specifically Agile technique or formalism, such as uses cases or more narrative documents.

Source: Agile Aliance: Acceptance Testing

They are different things, so the tests, and the test frameworks, should be different as well. You aren't really "repeating yourself" by keeping a separate suite for load tests.

As for the technical reason why this is failing? SpecFlow, when run using the normal Visual Studio test runner, was not built to run tests in parallel. There are parallel test runners available, but most are paid software.