Beta

We highly recommend fully setting up SDK tests in all your SDK repositories before exploring custom contract tests.

Custom end-to-end API contract tests with Arazzo

You can use Speakeasy to create custom end-to-end contract tests that run against a real API.

This document guides you through writing more complex tests using the Arazzo Specification, as well as through the key configuration features for these tests, which include:

Server URLs
Security credentials
Environment variable-provided values

Arazzo is a simple, human-readable, and extensible specification for defining API workflows. Arazzo powers custom test generation, allowing you to define rich tests capable of:

Testing multiple operations
Testing different inputs
Validating that the correct response is returned
Running against a real API or mock server
Configuring setup and teardown routines for complex end-to-end (E2E) tests

The Arazzo Specification allows you to define sequences of API operations and their dependencies for contract testing, enabling you to validate whether your API behaves correctly across multiple interconnected endpoints and complex workflows.

When a .speakeasy/tests.arazzo.yaml file is found in your SDK repo, the Arazzo workflow is used to generate tests for each of the workflows defined in the file.

Prerequisites

Before generating tests, ensure that you meet the testing feature prerequisites.

Writing custom end-to-end tests

The following is an example Arazzo document defining a simple E2E test for the lifecycle of a user resource in the example API:


arazzo: 1.0.0
info:
  title: Test Suite
  summary: E2E tests for the SDK and API.
  version: 0.0.1
sourceDescriptions:
  - name: The API
    url: https://example.com/openapi.yaml
    type: openapi
workflows:
  - workflowId: user-lifecycle
    steps:
      - stepId: create
        operationId: createUser
        requestBody:
          contentType: application/json
          payload:
            {
              "email": "Trystan_Crooks@hotmail.com",
              "first_name": "Trystan",
              "last_name": "Crooks",
              "age": 32,
              "postal_code": 94110,
              "metadata":
                {
                  "allergies": "none",
                  "color": "red",
                  "height": 182,
                  "weight": 77,
                  "is_smoking": true,
                },
            }
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == "application/json"
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          - condition: $response.body#/postal_code == 94110
        outputs:
          id: $response.body#/id
      - stepId: get
        operationId: getUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == "application/json"
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          - condition: $response.body#/first_name == Trystan
          - condition: $response.body#/last_name == Crooks
          - condition: $response.body#/age == 32
          - condition: $response.body#/postal_code == 94110
        outputs:
          user: $response.body
          age: $response.body#/age
      - stepId: update
        operationId: updateUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id
        requestBody:
          contentType: application/json
          payload: $steps.get.outputs.user
          replacements:
            - target: /postal_code
              value: 94107
            - target: /age
              value: $steps.get.outputs.age
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == "application/json"
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          - condition: $response.body#/first_name == Trystan
          - condition: $response.body#/last_name == Crooks
          - condition: $response.body#/age == 32
          - condition: $response.body#/postal_code == 94107
        outputs:
          email: $response.body#/email
          first_name: $response.body#/first_name
          last_name: $response.body#/last_name
          metadata: $response.body#/metadata
      - stepId: updateAgain
        operationId: updateUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id
        requestBody:
          contentType: application/json
          payload:
            {
              "id": "$steps.create.outputs.id",
              "email": "$steps.update.email",
              "first_name": "$steps.update.first_name",
              "last_name": "$steps.update.last_name",
              "age": 33,
              "postal_code": 94110,
              "metadata": "$steps.update.metadata",
            }
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == "application/json"
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          - condition: $response.body#/first_name == Trystan
          - condition: $response.body#/last_name == Crooks
          - condition: $response.body#/age == 33
          - condition: $response.body#/postal_code == 94110
      - stepId: delete
        operationId: deleteUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id
        successCriteria:
          - condition: $statusCode == 200

This workflow defines four steps, each of which feeds into the next:

Create a user
Retrieve that user via its new ID
Update the user
Delete the user

This is possible because the workflow defines outputs for certain steps that serve as inputs for the following steps.

The workflow generates the test shown below:

Input and outputs

Inputs

There are various ways to provide an input for a step, including:

Defining it in the workflow
Referencing it in a previous step
Including it as an inline value

Workflow inputs

You can provide input parameters to the workflow using the inputs field, which is a JSON Schema object that defines a property for each input that the workflow wants to expose. These workflow inputs can be used by any step defined in the workflow.

Test generation can use any of the examples defined for a property in an inputs JSON schema as a literal value, which it then uses as an input for the test. Because tests are non-interactive and cannot ask users for input, the test generation randomly generates values for the inputs if no examples are defined.


arazzo: 1.0.0
# ....
workflows:
  - workflowId: some-test
    inputs: # This is the JSON Schema for the inputs each property in the inputs object represents a workflow input
      type: object
      properties:
        email:
          type: string
          examples:
            - Trystan_Crooks@hotmail.com # Examples defined will be used as literal values for the test
        firstName:
          type: string
          examples:
            - Trystan
        lastName:
          type: string
          examples:
            - Crooks
    steps:
      - stepId: create
        operationId: createUser
        requestBody:
          contentType: application/json
          payload: {
              "email": "$inputs.email", # The payload will be populated with the literal value defined in the inputs
              "first_name": "$inputs.firstName",
              "last_name": "$inputs.lastName",
            }
        successCriteria:
          - condition: $statusCode == 200

Step references

Parameters and request body payloads can reference values via runtime expressions from previous steps in the workflow. This allows for the generation of tests that are more complex than a simple sequence of operations.

Speakeasy’s implementation currently only allows the referencing of a previous step’s output, which means you will need to define which values you want to expose to future steps.


arazzo: 1.0.0
# ....
workflows:
  - workflowId: some-test
    steps:
      - stepId: create
        operationId: createUser
        requestBody:
          contentType: application/json
          payload: #....
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == "application/json"
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
        outputs:
          id: $response.body#/id # The id field of the response body will be exposed as an output for the next step
      - stepId: get
        operationId: getUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id # The id output from the previous step will be used as the value for the id parameter
        successCriteria:
          - condition: $statusCode == 200

Inline values

For any parameters or request body payloads that a step defines, you can also provide literal values inline to populate the tests (if static values are suitable for the tests).


arazzo: 1.0.0
# ....
workflows:
  - workflowId: some-test
    steps:
      - stepId: update
        operationId: updateUser
        parameters:
          - name: id
            in: path
            value: "some-test-id" # A literal value can be provided inline for parameters that matches the json schema of the parameter as defined in the associated operation
        requestBody:
          contentType: application/json
          payload: # literals values that match the content type of the request body can be provided inline
            {
              "email": "Trystan_Crooks@hotmail.com",
              "first_name": "Trystan",
              "last_name": "Crooks",
              "age": 32,
              "postal_code": 94110,
              "metadata":
                {
                  "allergies": "none",
                  "color": "red",
                  "height": 182,
                  "weight": 77,
                  "is_smoking": true,
                },
            }
        successCriteria:
          - condition: $statusCode == 200

Payload values

If you use the payload field of a request body input, its value can be any of the following:

A static value
A value with interpolated runtime expressions
A runtime expression by itself

You can then overlay the payload value using the replacements field, which represents a list of targets within the payload that will be replaced with the value of the replacements. These replacements can be static values or runtime expressions.


arazzo: 1.0.0
# ....
workflows:
  - workflowId: some-test
    steps:
      - stepId: get
        # ...
        outputs:
          user: $response.body
      - stepId: update
        operationId: updateUser
        parameters:
          - name: id
            in: path
            value: "some-test-id"
        requestBody:
          contentType: application/json
          payload: $steps.get.outputs.user # use the response body of the previous step as the payload for this step
          replacements: # overlay the payload with the below replacements
            - target: /postal_code # overlays the postal_code field with a static value
              value: 94107
            - target: /age # overlays the age field with the value of the age output of a previous step
              value: $steps.some-other-step.outputs.age
        successCriteria:
          - condition: $statusCode == 200

Outputs

As shown above, you can define outputs for each of the steps in a workflow, allowing you to use values from things such as response bodies in following steps.

Currently, Speakeasy supports only referencing values from a response body, using the runtime expressions syntax and json-pointers.

Any number of outputs can be defined for a step.


arazzo: 1.0.0
# ....
workflows:
  - workflowId: some-test
    steps:
      - stepId: create
        operationId: createUser
        requestBody:
          contentType: application/json
          payload: #....
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == "application/json"
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
        outputs: # Outputs are a map of an output id to a runtime expression that will be used to populate the output
          id: $response.body#/id # json-pointers are used to reference fields within the response body
          email: $response.body#/email
          age: $response.body#/age
          allergies: $response.body#/metadata/allergies

Success criteria

A step’s successCriteria field contains a list of criterion objects used to validate the success of the step and form the basis of the test assertions for test generation.

The successCriteria may be as simple as a single condition testing the status code of the response or as complex as multiple conditions testing various individual fields within the response body.

Speakeasy’s implementation currently only supports simple criteria and the use of the equality (==) and inequality (!=) operators for comparing values and for testing status codes, response headers, and response bodies.

Note: While the Arazzo specification defines additional operators like >, <, >=, <=, ~, and !~, Speakeasy currently only supports == and !=.

To test values within the response body, due to the typed nature of the SDKs, you need criteria for testing the status code and content type of the response to help the generator determine which response schema to validate against.


arazzo: 1.0.0
# ....
workflows:
  - workflowId: some-test
    steps:
      - stepId: create
        operationId: createUser
        requestBody:
          contentType: application/json
          payload: #....
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == "application/json"
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          # or
          - context: $response.body
            type: simple
            condition: |
              {
                "email": "Trystan_Crooks@hotmail.com",
                "first_name": "Trystan",
                "last_name": "Crooks",
                "age": 32,
                "postal_code": 94110,
                "metadata": {
                  "allergies": "none",
                  "color": "red",
                  "height": 182,
                  "weight": 77,
                  "is_smoking": true
                }
              }

Testing operations requiring binary data

Some operations require you to provide binary data, for example, for testing file uploads and downloads.

To provide the test with the test files, use the x-file directive in the example for the relevant field.


arazzo: 1.0.0
# ....
workflows:
  - workflowId: postFile
    steps:
      - stepId: test
        operationId: postFile
        requestBody:
          contentType: multipart/form-data
          payload:
            file: "x-file: some-test-file.txt"
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == "application/octet-stream"
          - context: $response.body
            condition: "x-file: some-other-test-file.dat"
            type: simple

The files are sourced from the .speakeasy/testfiles directory in the root of your SDK repo, where the path provided in the x-file directive is relative to the testfiles directory.

The content of the sourced file is used as the value for the field being tested.

Configuring an API to test against

By default, tests are generated to run against Speakeasy’s mock server, which has a URL of http://localhost:18080. The mock server validates that the SDKs are functioning correctly but does not guarantee that the API is correct.

The generator can be configured to run all tests against another URL or against individual tests using the x-speakeasy-test-server extensions in the .speakeasy/tests.arazzo.yaml file.

If the extension is found at the top level of the Arazzo file, all workflows and tests will be configured to run against the specified server URL. If the extension is found within a workflow, only that workflow will be configured to run against the specified server URL.

The server URL can be either a static URL or an x-env: EXAMPLE_ENV_VAR value that pulls the value from the environment variable, EXAMPLE_ENV_VAR (where the name of the environment variable can be any specified name).


arazzo: 1.0.0
# ...
x-speakeasy-test-server:
  baseUrl: "https://api.example.com" # If specified at the top level of the Arazzo file, all workflows will be configured to run against the specified server URL
workflows:
  - workflowId: some-test
    x-speakeasy-test-server:
      baseUrl: "x-env: CUSTOM_API_URL" # If specified within a workflow, only that workflow will be configured to run against the specified server URL. This will override any top-level configuration.
    # ...

You may provide a default value in the x-env directive if the environment variable is not set. This can be useful for local development or non-production environments.


x-speakeasy-test-server:
  baseUrl: "x-env: CUSTOM_API_URL; http://localhost:18080" # Run against the local mock server if the environment variable is not set

Reserved Environment Variable

The TEST_SERVER_URL environment variable is reserved for use by Speakeasy’s mock server. When running tests via speakeasy test, if the mock server is generated and enabled, TEST_SERVER_URL is automatically set to the URL of the running mock server and overwrites any existing value for that environment variable while running.

If you want to use a custom test server instead of the mock server, you can:

Use the --disable-mockserver flag when running speakeasy test to prevent the automatic setting of TEST_SERVER_URL
Use a different environment variable name (like CUSTOM_API_URL in the examples above) for your custom server configuration

If all tests are configured to run against other server URLs, you can disable mock server generation in the .speakeasy/gen.yaml file:


# ...
generation:
  # ...
  mockServer:
    disabled: true # Setting this to true will disable mock server generation

Configuring security credentials for contract tests

When running tests against a real API, the SDK may need to be configured with security credentials to authenticate with the API. To configure the SDK, add the x-speakeasy-test-security extension to the document, workflow, or individual step.

The x-speakeasy-test-security extension allows static values, values pulled from the environment, or runtime expressions referencing outputs from previous steps to be used when instantiating an SDK instance and making requests to the API.

Important: The keys under value must exactly match the names of the securitySchemes defined in your OpenAPI document’s components.securitySchemes section.

For example, if your OpenAPI document defines:


components:
  securitySchemes:
    myApiKeyScheme:
      type: apiKey
      in: header
      name: X-API-Key
    myBasicAuthScheme:
      type: http
      scheme: basic
    myBearerTokenScheme:
      type: http
      scheme: bearer
    myOAuth2Scheme:
      type: oauth2
      flows:
        clientCredentials:
          tokenUrl: https://api.example.com/oauth2/token
          scopes: {}

Then your Arazzo test configuration should reference these exact scheme names:


arazzo: 1.0.0
# ...
x-speakeasy-test-security: # Defined at the top level of the Arazzo file, all workflows will be configured to use the specified security credentials
  value:
    # The keys below MUST match the securitySchemes names from your OpenAPI document
    myApiKeyScheme: "x-env: TEST_API_KEY" # Values can be pulled from the environment
    myBasicAuthScheme:
      username: "test-user" # For schemes requiring multiple values, provide a map
      password: "x-env: TEST_PASSWORD"
    myBearerTokenScheme: "x-env: TEST_BEARER_TOKEN"
    myOAuth2Scheme:
      clientId: "x-env: MY_CLIENT_ID"
      clientSecret: "x-env: MY_CLIENT_SECRET"
      tokenURL: "http://test-server/oauth2/token" # Redirect OAuth flow to test server
workflows:
  - workflowId: some-test
    x-speakeasy-test-security: # Security can be defined/overridden for a specific workflow
      value:
        myApiKeyScheme: "test-key"
    # ...
    steps:
      - stepId: step1
        x-speakeasy-test-security: # Or security can be defined/overridden for a specific step
          value:
            myBearerTokenScheme: "x-env: TEST_AUTH_TOKEN"
        # ...
      - stepId: step2
        # ...

Note: For OAuth2 schemes, you can override the tokenURL to redirect the authentication flow to a test server instead of the production endpoint. This allows you to test OAuth2 flows against mock servers or staging environments without affecting production authentication systems.

Using Runtime Expressions for Dynamic Security

The x-speakeasy-test-security extension also supports runtime expressions, allowing you to populate security credentials dynamically from the outputs of previous steps. This is particularly useful for workflows that require authentication tokens obtained from login operations.

For example, you can use a runtime expression to reference a token from a previous authentication step:


arazzo: 1.0.0
info:
  title: Example
  summary: Example of a test suite
  version: 0.0.1
sourceDescriptions:
  - name: ./example.yaml
    url: https://example.com
    type: openapi
workflows:
  - workflowId: createUser
    x-speakeasy-test-security:
      value:
        apiKey: $steps.authenticate.outputs.token
    steps:
      - stepId: authenticate
        workflowId: authenticate
        requestBody:
          contentType: application/json
          payload:
            {
              "username": "trystan.crooks@example.com",
              "password": "x-env: TEST_PASSWORD",
            }
        successCriteria:
          - condition: $steps.authenticate.outputs.token != ""
        outputs:
          token: $steps.authenticate.outputs.token
      - stepId: create
        operationId: createUser
        requestBody:
          contentType: application/json
          payload:
            {
              "email": "Trystan_Crooks@hotmail.com",
              "first_name": "Trystan",
              "last_name": "Crooks",
              "age": 32,
              "postal_code": 94110,
              "metadata":
                {
                  "allergies": "none",
                  "color": "red",
                  "height": 182,
                  "weight": 77,
                  "is_smoking": true,
                },
            }
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == "application/json"
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          - condition: $response.body#/postal_code == 94110
        outputs:
          id: $response.body#/id
      - stepId: get
        operationId: getUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == "application/json"
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          - condition: $response.body#/first_name == Trystan
          - condition: $response.body#/last_name == Crooks
          - condition: $response.body#/age == 32
          - condition: $response.body#/postal_code == 94110

In this example:

The workflow defines x-speakeasy-test-security at the workflow level with apiKey: $steps.authenticate.outputs.token
The first step (authenticate) calls an authentication workflow and captures the token in its outputs
All subsequent steps in the workflow will use this dynamically obtained token for authentication
The runtime expression $steps.authenticate.outputs.token references the token output from the authenticate step

This approach enables complex authentication flows where tokens must be obtained dynamically during test execution, rather than being provided as static values or environment variables.

Configuring environment variable provided values for Contract tests

When running tests against a real API, you may need to fill in certain input values from dynamic environment variables. Use the Speakeasy environment variable extension to do so:


arazzo: 1.0.0
# ....
workflows:
  - workflowId: my-env-var-test
    steps:
      - stepId: update
        operationId: updateUser
        parameters:
          - name: id
            in: path
            value: "x-env: TEST_ID; default" # Provide an environment variable and an optional default value if that env variable is not present.
        requestBody:
          contentType: application/json
          payload: {
              "email": "x-env: TEST_EMAIL; default", # Provide an environment variable and an optional default value if that env variable is not present.
              "first_name": "Trystan",
              "last_name": "Crooks",
              "age": 32,
              "postal_code": 94110,
              "metadata":
                {
                  "allergies": "none",
                  "color": "red",
                  "height": 182,
                  "weight": 77,
                  "is_smoking": true,
                },
            }
        successCriteria:
          - condition: $statusCode == 200

Last updated on September 16, 2025