schemathesis 3.25.0

Release 3.25.0

on Python PyPI

In this release of Schemathesis, there are significant technical advancements to improve its effectiveness in real-world API testing scenarios. The focus has been on:

• GraphQL response validation
• Expanded compatibility with explicit examples in schemas
• Rigorous testing against the Open API directory

Running tests against the Open API directory

It has been a while since tests against the Open API directory were first implemented, but now they are updated and run in our CI every day to ensure compatibility with real-world APIs. They include generating tests against every API and sending them to a mock server.

Some schemas in the directory are too large to be executed in the current CI setup, so they are not included in the following statistic, totaling 4067 syntactically valid API schemas.

• Total API operations: 118771
• Successfully tested:: 116286 (97.90%)
• With errors in definitions: 2346 (1.97%) - self-contradicting constraints, unresolvable references, missing path parameter definitions
• Unsupported: 139 (0.12%) - generating images, Java regex syntax, deeply nested recursive references)

Excluding API operations with errors, Schemathesis was able to generate and execute test cases against up to 99.88% (*) of all API operations in the Open API directory.

(*) I'd still investigate more into self-contradicting constraints and unresolvable references as Schemathesis may miss support for some Open API / JSON Schema keywords that are relevant to that behavior.

🚀 Added

• --hypothesis-no-phases CLI option to disable Hypothesis testing phases. #1324
• Support for loading GraphQL schemas from JSON files that contain the __schema key.
• Response validation for GraphQL APIs.
• Support tag in filters for custom auth.
• Support for testing examples inside anyOf / oneOf / allOf keywords.
• Support for the text/xml media type in XMLSerializer.
• Support for the text/json media type in JSONSerializer.
• Support for pytest 8.

🔧 Changed

• CLI: Eagerly check for permissions when writing output to a file, including JUnit XML and other reports.
• Python: Explicitly note that combining @schema.given with explicit examples from the spec is not supported. #1217
• Clarify error message when a state machine has no transitions. #1992
• Do not consider missing the paths key an error for Open API 3.1.
• Improved formatting of multiple errors within the same API operation.
• Allow arbitrary objects in an array for application/x-www-form-urlencoded payloads.

🗑️ Deprecated

• The --contrib-unique-data CLI option and the corresponding schemathesis.contrib.unique_data hook. The concept of this feature
  does not fit the core principles of Hypothesis where strategies are configurable on a per-example basis but this feature implies
  uniqueness across examples. This leads to cryptic error messages about external state and flaky test runs, therefore it will be removed in
  Schemathesis 4.0

🐛 Fixed

• CLI: Do not duplicate the error message in the output when the error has no traceback and the --show-trace option is provided.
• Open API: Internal error on path templates that contain . inside path parameters.
• Open API: YAML serialization of data generated for schemas with format: binary.
• Create parent directories when saving JUnit XML reports and other file-based output. #1995
• Internal error when an API operation contains multiple parameters with the same name and some of them contain the examples keyword.
• Internal error during query parameter generation on schemas that do not contain the type keyword.
• Example generation for request body parameters using $ref.
• Generating examples for properties that have deeply nested $ref.
• Generating examples for properties with boolean sub-schemas.
• Validating responses with boolean sub-schemas on Open API 3.1.
• TypeError on non-string pattern values. This could happen on values in YAML, such that when not quoted, they are parsed
  as non-strings.
• Testing examples requiring unsupported payload media types resulted in an internal error. These are now correctly reported as errors
• Internal error on unsupported regular expressions in inside properties during example generation.
• Missing XML definitions when the media type contains options like application/xml; charset=utf-8.
• Unhandled timeout while reading the response payload.
• Internal error when the header example in the schema is not a valid header.
• Handle KeyError during state machine creation.
• Deduplicate network errors that contain unique URLs in their messages.
• Not reporting multiple errors of different kinds at the same API operation.
• Group similar errors within the same API operation.