What should be the scope of a Pact provider test?

Question

My organization started to use Pact for creating/verifying contracts between REST services/micro services written in Java about half a year ago. We have a hard time deciding what the appropriate scope or grasp of a provider test should be and would love some input from the experience of other pact users out there.

Basically the discussion evolves around where to mock/stub in the provider tests. In a service you would have to mock external calls to other services at least but you have the option of mocking closer to the REST resource class as well.

We boiled it down to two options:

1. The first option is that a provider test should be a strict contract test and only exercise the provider service's REST resource class, mocking/stubbing out the service classes/orchestrators etc. used from there. This contract test would be augmented with component tests that would test the parts stubbed/mocked by the provider test.

2. The second option is to use the provider test as a component test that would exercise the entire service component for each request. Only transitive external calls to other components would be mocked/stubbed.

These are thoughts of pro's for each option

Pro's for option 1:

• Tests will be simpler to implement and will get a smaller footprint
  => higher degree of isolation.
• We probably need other component tests anyway to cover use cases typically not covered in the consumer's expectancies (error flows etc). This way we wouldn't mix different kinds of component tests (Pact and other) in one bag, making the test suite easier to understand.

Pro's for option 2:

• Tests are exercising more of the "real" code => less risk for test bugs due to bad mocking/stubbing.

I would be really interested to hear how your provider tests typically look in this regard. Is there a best practice?

Clarifying what we mean by "Component": A component is a microservice or a module in a larger service application. We took the definition of 'component from Martin Fowlers http://martinfowler.com/articles/microservice-testing/.

A provider service/component typically has a REST endpoint in a Jersey resource class. This endpoint is the provider endpoint for a Pact provider test. An Example:

@Path("/customer")
public class CustomerResource {

    @Autowired private CustomerOrchestrator customerOrchestrator;

    @GET
    @Path("/{customerId}")
    @Produces(MediaType.APPLICATION_JSON)
    public Response get(@PathParam("customerId") String id) {
        CustomerId customerId = CustomerIdValidator.validate(id);
        return Response.ok(toJson(customerOrchestrator.getCustomer(customerId))).build();
    }

In the above example the @Autowired (we use spring) CustomerOrchestrator could either be mocked when running the provider test or you could inject the real "Impl" class. If you choose to inject the real "CustomerOrchestratorImpl.class" it would have additional @Autowired bean dependencies that in turn may have other... etc. Finally the dependencies will end up either in a DAO-object that will make a database call or a REST client that will perform HTTP calls to other downstream services/components.

If we were to adopt my "option 1" solution in the above example we would mock the customerOrchestrator field in the CustomerResource and if we were adopting "option 2" we would inject Impl-classes (the real classes) for each dependency in the CustomerResource dependency graph and create mocked database entries and mocked downstream services instead.

As a side note I should mention that we rarely actually use a real database in provider tests. In the cases where we adopted "option 2" we have mocked the DAO-class layer instead of mocking the actual database data to reduce the number of moving parts in the test.

We have created a "test framework" that automatically mocks any Autowired dependency that is not explicitly declared in the spring context so stubbing/mocking is a light weight process for us. This is an excerpt of a provider test that exercises the CustomerResource and initiates the stubbed CustomerOrchestrator bean:

@RunWith(PactRunner.class)
@Provider("customer-rest-api")
@PactCachedLoader(CustomerProviderContractTest.class)
public class CustomerProviderContractTest {

    @ClassRule
    public static PactJerseyWebbAppDescriptorRule webAppRule = buildWebAppDescriptorRule();

    @Rule
    public PactJerseyTestRule jersyTestRule = new PactJerseyTestRule(webAppRule.appDescriptor);

    @TestTarget public final Target target = new HttpTarget(jersyTestRule.port);

    private static PactJerseyWebbAppDescriptorRule buildWebAppDescriptorRule() {
        return PactJerseyWebbAppDescriptorRule.Builder.getBuilder()
            .withContextConfigLocation("classpath:applicationContext-test.xml")
            .withRestResourceClazzes(CustomerResource.class)
            .withPackages("api.rest.customer")
            .build();
    }

    @State("that customer with id 1111111 exists")
    public void state1() throws Exception {
        CustomerOrchestrator customerOrchestratorStub = SpringApplicationContext.getBean(CustomerOrchestrator.class)
       when(customerOrchestratorStub.getCustomer(eq("1111111"))).thenReturn(createMockedCustomer("1111111));

    }
    ...

Answer 1

This is a question that comes up often, and my answer is "do what makes sense for each service". The first microservices that pact was used for were so small and simple that it was easiest to just test the whole service without any mocks or stubs. The only difference between a call to the real service and a call in the verification test was that we used sqlite for the tests. Of course, we stubbed calls to downstream services.

If it's more complex to set up real data than it is to stub, then I'd use stubs. However! If you are going to do this, then you need to make sure that the calls you stub are verified in the same way that pact works. Use some sort of shared fixture, and make sure that for every call that you stub in a pact provider test, you have a matching test to ensure the behaviour is as you expect it to be. It's like you're chaining collaboration/contract tests together like this:

Answer 2

I say go with Option 2.

The reason is because the whole raison d'etre for Pact is to have confidence in your code change - that won't break the consumer, or that if it does, find a way to manage that change (versioning) while still keeping the interactions intact.

To be fully confident, you must use as much of the 'real' code as possible, while the data can be mocked or real, it doesn't really matter at that point. Remember that you want to be able to test as much of the production code before deploying it.

The way I use it, I have 2 types of tests, unit tests and Pact tests. The unit tests make sure that my code doesn't break on stupid mistakes or bad inputs, which is great for mocking dependencies, while the Pact tests are used to test the interactions between the consumer and the provider and that my code change doesn't affect the request or the data format. You could potentially mock dependencies here, but that might leave something open to breaking in production because that dependency might affect the request or data.

In the end though, it's all up to preference on how you use Pact, as long as you use it to test out the contracts between consumer and provider.