v1.17.1

Failed tests in the text report

The text report only gave a count for a failing request or step. 1.17.1 lists each failed test with its message, and workflow step details start under the step text instead of one or two columns to its left. A step with a failed JavaScript test in 1.17.0:

FAIL WORKFLOW check-user [0 passed, 1 failed, 0 skipped in 1.739666ms]
  1. FAIL Get [1 test(s) failed]
    Source Target: {{base.url}}/users/1
    Effective Target: http://localhost:3000/users/1

And in 1.17.1:

FAIL WORKFLOW check-user [0 passed, 1 failed, 0 skipped in 1.435875ms]
  1. FAIL Get [1 test(s) failed]
     Source Target: {{base.url}}/users/1
     Effective Target: http://localhost:3000/users/1
     Failed Tests:
       user has an email

• Passing tests are left out. A message over several lines stays lined up under its first line.
• With 10 or more steps, the step numbers are right-aligned so the details of every step start in the same column.
• This applies to -f text, to the default auto format whenever it prints the text report, and to the headless Text format.

JavaScript asserts show their text once

tests.assert and tests.fail used their text as both the test name and its message, so you saw it twice:

Tests:
  [PASS] status is 200
  [FAIL] user has an email - user has an email

Now it's [FAIL] user has an email. In JSON, failure.message is user has an email instead of user has an email: user has an email, and the failed test has no message field. client.test hasn't changed and still shows what failed it, as in user has an email - Error: no email for user 1.

--timeout stops JavaScript tests

@assert and RTS test blocks already stopped at the request timeout. JavaScript tests didn't, so a test stuck in a loop ran for 30 seconds whatever the timeout was. This one never moves to the next page:

### Orders
# @name Orders
# @script test
> {%
let page = response.json();
let count = 0;
while (page.next) {
  count += page.items.length;
}
tests.assert(count > 0, "has orders");
%}
GET {{base.url}}/orders

With --timeout 5s, 1.17.0 still waited 30 seconds and exited with 24:

Errors:
  Script error:
    error[timeout]: script exceeded the 30s time limit
    test script 1

1.17.1 stops it after 5 seconds and exits with 20:

Errors:
  Script error:
    error[timeout]: response checks timed out after 5s
    test script 1
    ╰─> context deadline exceeded

• All checks share one timeout that starts after captures. They run in order: @assert, RTS test blocks, then JavaScript tests. A check that hasn't started when the time runs out is skipped.
• @timeout on a request works the same way.
• A timeout in the checks exits with 20, like any other timeout. 1.17.0 exited with 24, the code for script errors, even when the error said timeout. A script that throws still exits with 24.
• A script block still stops after 30 seconds at most. Pre-request scripts don't use --timeout.

See exit codes.

Loop variables in step conditions

A step's @when or @skip-if is checked once, before the @for-each of the request it runs, so it can't read that loop's variable. For most names this fails with a hint. If the variable had the name of a built-in, such as list, json or len, the condition read the built-in instead and carried on without an error.

# @workflow sync-lists
# @skip-if list.archived
# @step Sync using=SyncList

### Sync list
# @name SyncList
# @for-each json.file("lists.json") as list
POST {{base.url}}/lists/{{= list.id }}/sync

old-promos is archived, and 1.17.0 synced it anyway:

PASS WORKFLOW sync-lists [3 passed, 0 failed, 0 skipped in 2.967291ms]
  1. PASS Sync (1/3) [200 OK in 1.218709ms]
    Source Target: {{base.url}}/lists/{{= list.id }}/sync
    Effective Target: http://localhost:3000/lists/news/sync
  2. PASS Sync (2/3) [200 OK in 568.917µs]
    Source Target: {{base.url}}/lists/{{= list.id }}/sync
    Effective Target: http://localhost:3000/lists/old-promos/sync

1.17.1 stops with the same error any other name gets:

FAIL WORKFLOW sync-lists [0 passed, 1 failed, 0 skipped in 59.75µs]
  1. FAIL Sync [@skip-if: lists.http:2:1: undefined name "list"]
     Error:
       error[script]: undefined name "list"
       --> lists.http:2:1
            |
          2 | # @skip-if list.archived
            | ^
       @skip-if
       help: @skip-if is checked before the @for-each of request "SyncList" starts, so "list" is not set. Put the condition on the request to check each item

Move @skip-if list.archived under the @for-each line and step 2 shows SKIP Sync (2/3) [@skip-if evaluated to true: list.archived].

The same goes for a @use alias with the loop variable's name. If you meant the built-in in such a condition, write rts.len or rts.list.

See the workflows docs.

+ needs text on one side

1.16.0 stopped + from joining two lists or two dictionaries. Any other pair was still turned into text, so [1] + 2 gave "[1]2" and true + false gave "truefalse". Now + adds two numbers or joins when one side is a string. Every other pair fails with an error such as cannot add list and number.

null fails on either side too. The @apply example from the docs sent an empty bearer token when auth.token wasn't set:

### Reports
# @name Reports
# @apply {headers: {"Authorization": "Bearer " + vars.get("auth.token")}}
GET {{base.url}}/reports

1.17.0 sent Authorization: Bearer. 1.17.1 doesn't send the request:

error[script]: cannot add string and null
--> reports.http:3:48
     |
   3 | # @apply {headers: {"Authorization": "Bearer " + vars.get("auth.token")}}
     |                                                ^

If an empty value is what you want, give it a fallback: "Bearer " + (vars.get("auth.token") ?? ""). Joins like "id=" + 1000000 work as before.

See operators.

Upgrading from 1.17.0

Most request files keep working. If you have CI checks or scripts that read the output, look out for these:

1. + fails when neither side is a string, as in [1] + 2, true + false or 1 + true, and when either side is null. Convert with str() or add a ?? fallback.
2. A timeout in @assert, RTS test blocks or JavaScript tests exits with 20 instead of 24, and the JSON failure code is timeout instead of script.
3. --timeout and @timeout limit JavaScript tests now. A slow test that finished within 30 seconds can time out with a shorter --timeout or @timeout. Raise @timeout on that request.
4. A step @when or @skip-if that reads the loop variable of its request's @for-each fails for every name, including built-ins and @use aliases. Put the condition on the request, or write rts.len and so on for the built-in.
5. Failed JavaScript asserts:
   • JSON failure.message no longer repeats the text, and the test has no message field.
   • The workflow summary reads Workflow check-user failed at step Get: test failed: user has an email instead of Workflow check-user failed at step Get: user has an email.
   • Text with line breaks shows on one row, with \n, in the text report and terminal output.
6. The text report lists failed tests, and workflow step details start under the step text. Update anything that parses it, or switch to -f json.