Is endpoint unit testing possible based on only OpenAPI 3.0 file?

Question

After some googling I decided to directly ask this:

Is it possible to run REST API endpoint "build tests" based on solely an OpenAPI 3.0 specification file?

I would like to add example request-response pairs to the OpenAPI spec file and have either an existing or self-written library run through all the such defined tests and report any failures.

As far as I understand the standard, it would be easily possible to include examples for requests and test the response in a schematic way (thinking http://json-schema.org/). But I don't yet see a way to incorporate more specific tests, such as testing a number in a specific response field to be an exact value. Also, it would be nice to have a way to sequentially test requests, but I don't expect to achieve that just from the spec file.

An optimal solution would be included in my repository and run the tests on localhost. Thank you all very much in advance.

Answer 1

Is it possible to run REST API endpoint "build tests" based on solely an OpenAPI 3.0 specification file?

Yes. The spec has at least two components that allow automatic tests generation:

• Examples. You could specify examples for defined endpoints, which describe concrete data samples you send/receive.
• Schemas. They describe data models, which you can use to generate requests to the API.

But I don't yet see a way to incorporate more specific tests, such as testing a number in a specific response field to be an exact value.

It can be done using the "enum" keyword with a single value in a list in the schema for the desired field. Open API doesn't support the "const" keyword in contrast with JSON Schema.

Also, it would be nice to have a way to sequentially test requests, but I don't expect to achieve that just from the spec file.

To achieve that you could use Open API links, which allows you to specify how to construct a request to endpoint B from request or response to endpoint A. For example:

paths:
  /users:
    post:
      summary: Creates a user and returns the user ID
      operationId: createUser
      responses:
        '201':
          description: Created
          content:
            application/json:
              schema:
                type: object
                properties:
                  id:
                    type: integer
                    format: int64
                    description: ID of the created user.
          links:
            GetUserByUserId:
              operationId: getUser
              parameters:
                userId: '$response.body#/id'
  /users/{userId}:
    get:
      summary: Gets a user by ID
      operationId: getUser
      parameters:
        - in: path
          name: userId
          required: true
          schema:
            type: integer
            format: int64

In this example, the id value returned in the 201 response to the POST /users can be used as the userId parameter in GET /users/{userId}. In this way, you can define sequences of requests to different endpoints.

I want to note two tools that can generate and execute test cases based only on the input Open API spec:

1. Schemathesis uses both sources and doesn't require configuration by default. It utilizes property-based testing and verifies properties defined in the tested schema - response codes, schemas, and headers. It supports Open API 2 & 3 and stateful testing via Open API links and automatically can generate sequences of API requests based on their definition in the spec file.
2. Dredd focuses more on examples and provides several automatic expectations. It supports only Open API 2, and the third version is experimental.

Both provide a CLI and could be extended with various hooks to fit the desired workflow.