grafana/k6 v0.31.0

on GitHub

k6 v0.31.0 is here! 🎉 It's a smaller release with some significant performance improvements, a new http_req_failed metric and changes to the output subsystem that enable output extensions with xk6!

New features

Output cleanup and extensions (#1874)

The state of k6's output packages has been a development pain point for a long time, which made it difficult to add new outputs in a consistent way. In the refactor done in v0.31.0, this has been mostly addressed and outputs now implement a simpler and cleaner Output interface.

In addition to this, it is now possible to implement custom k6 output extensions in Go with xk6! This is very useful if you use a system that's currently not supported by the built-in outputs, or need some custom handling of the metrics k6 produces.

Writing output extensions is done very similarly to how JS module extensions are currently written, though instead of calling js/modules.Register(), you should implement the new Output interface and call output.RegisterExtension() with your constructor.

We are working on the proper documentation for this, as well as the overdue xk6 documentation about JS extensions, so keep a lookout for those on k6.io/docs.

Marking requests as failed (#1856)

It's now possible to declare expected HTTP response statuses for either the entire test or for individual HTTP requests, and k6 will emit a new http_req_failed metric as well as tag HTTP metrics with expected_response: <bool>. By default, k6 will now fail requests that return HTTP 4xx/5xx response codes.

For example:

import http from 'k6/http';

// Set expected statuses globally for all requests.
http.setResponseCallback(http.expectedStatuses({min: 200, max: 399}, 418));

export default function () {
  // This request will be marked as failed.
  http.get('https://httpbin.test.k6.io/status/400');
  // This request will be considered as "passed" because of the responseCallback override.
  http.get('https://httpbin.test.k6.io/status/400', { responseCallback: http.expectedStatuses(400) });
}

Running this script will produce a summary like:

http_req_duration..............: avg=204.57ms min=203.31ms med=204.57ms max=205.82ms p(90)=205.57ms p(95)=205.7ms
  { expected_response:true }...: avg=203.31ms min=203.31ms med=203.31ms max=203.31ms p(90)=203.31ms p(95)=203.31ms
http_req_failed................: 50.00% ✓ 1   ✗ 1

Note the new http_req_duration sub-metric for expected responses only, and the new http_req_failed Rate metric. This new metric and metric tag have many potential use cases, and one of the most important ones is the ability to set better thresholds. For example:

• 'http_req_failed': ['rate<0.1'], i.e. fail the test if more than 10% of requests fail.
• 'http_req_duration{expected_response:true}': ['p(95)<300', 'p(99.9)<500'] - fail the test if the the 95th percentile HTTP request duration is above 300ms or the 99.9th percentile is above 500ms; specifying expected_response:true here may be important, because a lot of times failed requests may return more quickly than normal ones, thus skewing the results and wrongly satisfying the threshold.

If the response callback is not specified, the default expected statuses will be {min: 200, max: 399}. The previous behavior of not emitting anything can be achieved by setting the callback to null, i.e. http.setResponseCallback(null). Additionally, the expected_response tag can be disabled by removing it from the default list of system tags, e.g. k6 run --system-tags 'proto,subproto,status,method,url,name,group,check,error,error_code,tls_version,scenario,service'.

The http.setResponseCallback() is planned to allow arbitrary JS functions to process responses in the future, but for now only the http.expectedStatuses() callback is supported.

Other enhancements and UX improvements

• JS: Because of the awesome improvements to goja, the JS runtime k6 uses, it's no longer necessary for k6 to load core.js to polyfill missing JS features when using the default --compatibility-mode=extended. So in v0.31.0 core.js has been dropped entirely, yielding some significant CPU and memory usage improvements. The actual numbers will depend on the use case, but for simple tests users can expect a memory drop of about 2MB per VU (from ~2.7MB to ~600KB), and a slight CPU decrease of about 5-10%. For more complex tests with a lot of JS code this benefit won't be as pronounced. Another benefit of this change is that initializing VUs and starting a test is substantially faster than before! (#1824)
• JS: Also because of goja improvements, some unused Babel plugins were disabled which should have minor performance benefits as well. (#1822)
• JS: Expanded ArrayBuffer support in most internal modules, so now you can pass ArrayBuffer to http.file(), in k6/encoding and k6/crypto functions. This makes working with binary files more efficient as it doesn't require string translations. In upcoming versions we plan to expand this to the WebSocket module, as well as make some potentially breaking changes for APIs that currently return an array of integers or string (see the details in the Breaking Changes announcement below). (#1800)
• The Docker image base was updated to Alpine 3.13. Thanks @andriisoldatenko! (#1821)
• The Debian package now includes ca-certificates as a dependency. Thanks @Bablzz! (#1854)

Bugs fixed!

• Execution: Aborting a test during VU initialization (e.g. with ^C) will now properly propagate to any used outputs. (#1869)
• Execution: A race condition between the Engine and the outputs' finalization code was fixed, ensuring that all metrics are properly emitted before exiting. (#1869)
• Execution: Another race condition in the Engine was fixed, which may have resulted in the end-of-test summary missing some of the last test metric data. (#1888)
• Cloud: the test name is now properly validated and will raise an error if not set via the ext.loadimpact.name JS option or config, or the K6_CLOUD_NAME environment variable. (#1870)
• JS: Babel is now also run on compilation errors, which improves support of some obscure language features. (#1861)
• JS: SharedArray introduced in v0.30.0 can now be iterated with forEach. (#1848)

Internals

• JS: SharedArray was rewritten using goja.DynamicArray making it more performant and easier to reason about. (#1848)
• JS: Some TC39 tests for unsupported features were disabled, improving the runtime of the test suite. (#1816)
• CI: Some more tests were enabled on Windows. (#1855)

Breaking changes

• JS: While we don't expect the core.js removal and Babel changes to impact the vast majority of users, those were substantial changes in how k6 interprets JS and a minority of users might experience issues with their tests. Please report any unexpected JavaScript errors by creating a GitHub issue. In particular Promise is now undefined, and some unused Babel plugins like transform-es2015-for-of and transform-regenerator were also removed. This means that some workarounds like the ones mentioned here and here also won't work as is and will need additional polyfills and plugins to work properly.

Planned future breaking changes

The following are not breaking changes in this release, but we'd like to announce them so users can prepare for them in upcoming releases (likely k6 v0.32.0).

• JS: The ArrayBuffer changes in this release are backwards compatible and shouldn't cause any issues, but in v0.32.0 some JS APIs that currently return an array of integers or string for binary data will return ArrayBuffer instead. This is the case for open() when used with the 'b' argument, response bodies for requests that specify responseType: 'binary', crypto.randomBytes(), hasher.digest('binary'), and encoding.b64decode(). Response.json() and Response.html() will also probably stop working when used with requests that specify responseType: 'binary'. These changes shouldn't be a problem for most users that were simply using these values to pass them to other internal modules (e.g. opening a binary file and passing it to http.post()), but if the scripts modified the binary data or depended on the current array of integers or string values they will need to be adapted to use typed arrays instead. You can follow the discussion in PR #1841 and issue #1020.
• As part of the rebranding of Load Impact to k6, the k6 GitHub repository will be moved from https://github.com/loadimpact/k6 to https://github.com/k6io/k6 . Additionally because of Go's usage of URLs in package imports, the URL will be changed throughout the codebase. Since GitHub will maintain a redirect from the old location we don't expect this to impact a lot of users, or even external k6 contributors and xk6 developers, but be aware that the new URL should be used moving forward.
• Because of the sunsetting of Bintray, the DEB, RPM, MSI and Chocolatey package repositories currently hosted on Bintray will be moved to a self-hosted solution sometime in the upcoming weeks. We'll communicate these changes via our blog as well as the official documentation.