unkn0wn-root/resterm v1.15.0

on GitHub

v1.15.0

You can now write response tests directly in RestermScript, without switching to JavaScript. There's also json.equal and json.diff, plus a couple of bug fixes.

RestermScript test blocks

You can use @rts test to validate responses in RestermScript. Call t.error to report a failure and continue checking, or t.fatal to stop the current block.

### Orders
# @name orders
# @use ./validators.rts
# @rts test "orders are valid"
> {%
for let _, msg range validators.orders(response.json().orders) {
  t.error(msg)
}
%}
GET {{base.url}}/orders/bad

validators.rts:

module validators

export fn orders(list) {
  let errs = []
  for let _, o range list {
    if o.total < 0 {
      errs = rts.list.append(errs, "order " + str(o.id) + ": total " + str(o.total) + " is negative")
    }
    if len(o.items) == 0 {
      errs = rts.list.append(errs, "order " + str(o.id) + ": no items")
    }
  }
  return errs
}

Tests:
  [FAIL] orders are valid - orders.http:7: order 1: total -5 is negative
                            orders.http:7: order 3: no items (77.416µs)

• Each block produces one test result. Names are optional, but names containing spaces must be quoted. # @script test lang=rts does the same thing.
• Test blocks can access response, vars, env, captures, and imported modules. They are read-only, though: vars.set and request modifications will fail. Use @capture if you need to save something.
• Resterm runs @assert first, then RTS test blocks in file order, and finally JavaScript tests. An assertion failure won't prevent the other tests from running.
• Runtime errors, such as accessing a field on null or calling fail(), fail the current block and show the file and line. The remaining blocks still run. resterm run exits with 1 for failed tests and 24 for script errors.
• If an imported @use module can't load, Resterm reports the error once and marks all test blocks as failed.

See @rts and t in the docs.

Comparing JSON

json.equal(a, b) compares values recursively, including nested lists and dicts. You can also use it in @assert:

# @assert json.equal(response.json().tags, ["ops", "billing"]) => "tags changed"

json.diff(got, want) gives you a message for each difference. You can pass those messages to t.error to report all mismatches at once:

### User
# @name user
# @rts test "matches fixture"
> {%
let want = {id: 7, name: "Ada", role: "owner", tags: ["ops", "billing"], active: true}
for let _, d range json.diff(response.json(), want) {
  t.error(d)
}
%}
GET {{base.url}}/users/7

Tests:
  [FAIL] matches fixture - users.http:7: $["display.name"]: unexpected "Ada L."
                           users.http:7: $.role: got "admin", want "owner"
                           users.http:7: $.tags[1]: missing, want "billing" (61.209µs)

Resterm shows up to 100 differences, followed by a count of any remaining ones (for example, and 3 more differences). See the JSON helpers docs.

One-line {% %} blocks

Single-line {% %} scripts work with RestermScript too:

# @rts pre-request
> {% request.setQueryParam("expand", "roles") %}

In 1.14.1 this failed with dict key must be string or ident.

JSON paths

JSON paths behave the same way across @capture, stream captures, response.json(path), json.get, json.has, mock responses and @auth command options.

• Negative indexes count from the end everywhere. In 1.14.1, response.json("items[-1].id") returned null in scripts.
• Captures can read keys that contain a dot: {{response.json["display.name"]}}. This used to capture an empty value.
• A key with a space has to be quoted, for example $["first name"].
• Stream paths can go deeper, as in {{stream.events[0].json.id}} for a gRPC stream. Field names in stream paths are case-sensitive.

Invalid paths produce errors instead of silently returning unexpected values. For example, in 1.14.1, items[*] returned the entire list, so this assertion passed:

# @assert len(json.get(response.json(), "items[*]")) == 3

In 1.15.0, it fails with:

error[script]: json.get(value[, path]) invalid path "items[*]": invalid array index "*"
--> items.http:4:23

The same goes for captures. {{response.json.items[*].id}} fails even when capture.strict is off. With capture.strict on, a missing key says key not found instead of segment not found.

See the captures docs.

Canceling a run

Ctrl+C during tests counts as a cancellation, not a script error. The exit code is 130 instead of 24.

1.14.1:

FAIL GET orders [test script 1: interrupt signal received]
  Script Error:
    error[script]: test script 1
    ╰─> interrupt signal received

1.15.0:

FAIL GET orders [interrupt signal received]
  Error:
    error[canceled]: interrupt signal received

A cancel before the request is sent, or during @capture, @apply, @when and pre-request scripts, shows error[canceled] too. In the TUI, a cancel during checks keeps the response and shows Request canceled.

If you use the headless API and cancel with a cause (context.WithCancelCause or context.WithTimeoutCause), the run is reported as canceled or timed out instead of a network or unknown error. This includes poll and retry waits.

Response check timeout

@assert and RTS test blocks get their own full request timeout (--timeout or the request-specific setting), starting when response checks begin. Previously, HTTP assertions only had whatever time was left after the request, which could be almost nothing for slow responses. If the checks run out of time, the error says response checks timed out after 30s and blocks that haven't started are skipped.

JavaScript tests keep their own time limit.

Script warnings

Resterm warns you about unsupported or unrecognized @script directives instead of silently ignoring them:

WARN users.http:3: @script language "python" is not supported. Use js or rts
WARN kind.http:3: @script kind "check" is not supported. Use pre-request or test
WARN extra.http:3: @script ignores "python". Use lang=js or lang=rts to choose a language
WARN opt.http:3: unknown @script option "foo"

The first two are skipped and the last two run as JavaScript, as before. Warnings don't change the exit code.

@rts is stricter about what follows it. A test name with spaces needs quotes, and unknown options such as foo=1 are a parse error:

error[parse]: @rts test takes one name. Quote a name with spaces
--> words.http:3:18
     |
   3 | # @rts test user shape
     |                  ^

Other fixes

• Numbers are written in full when they turn into text. str(1000000), "id=" + 1000000, {{= 1000000 }} and captures give 1000000 instead of 1e+06. They follow json.stringify, so captures of very large or very small numbers, like 1e+21 or 1e-7, changed too.
• An error inside a .rts module shows the line from the module. Before, it showed the line with the same number from the .http file. When several test blocks fail, each error shows a line from its own file. If a file changed after its code was loaded, an error from the old code shows the line that ran.
• resterm run and headless output show the failing line under errors from @assert, @capture, @when, @apply and templates, as the TUI already did. The TUI no longer prints these errors twice.
• When an @assert and a JavaScript test both fail with errors, each error keeps its own file, line and stack. Before, they were merged into one error[unknown].
• Multi-line test messages line up in resterm run and the TUI.
• FAIL rows in --format text and JUnit output stay on one line. Multi-line errors are joined with ; .
• Control characters in test names and messages are escaped, so a stray escape code can't mess up your terminal.
• Workflow summaries stay on one line, and every line of a multi-line step message is indented. Profile failure reasons print on one line.
• str() and json.stringify no longer hang on huge nested values. They fail with string too long.
• The LIVE badge on the response tabs no longer disappears when the tab row is full.
• Completion after @rts offers test as well. After @script test it offers only lang=.

Upgrading from 1.14.1

If you have existing request files or CI checks watch out for these changes:

1. {% %} is no longer stripped from JavaScript in 2 cases - an included .js file wrapped in {% and %}, and a block where {% or %} shares a line with code across several > lines (> {% a ... > b %}). These fail with SyntaxError: ... Unexpected token %. For an included file, delete the braces. Inline, put > {% and %} on their own lines. One-line blocks like > {% code %} still work.
2. Invalid JSON paths, such as items[*] or an unquoted key with a space, are errors in scripts and captures. Stream capture paths are case sensitive, so {{stream.Kind}} no longer matches kind.
3. Unknown options after @rts, such as foo=1, stop the file from parsing and resterm run exits with 2. 1.14.1 ignored them.
4. Ctrl+C during checks exits with 130 instead of 24, and a strict capture miss says key not found instead of segment not found. Update CI scripts that look for these.
5. A module can't keep vars, request, t or other host objects for a later request. Using one fails with vars belongs to an evaluation that has ended. Save the value you need instead, for example vars.get("token").
6. @script test lang=rts blocks were skipped without a warning in 1.14.1. In 1.15.0 they run.

Upgrading from 1.13.3 or earlier

If you're upgrading from 1.13.3 or earlier, there are a few changes in 1.14.0 worth checking:

• On macOS and Linux, history files are now readable only by their owner (0600), and new directories use 0700. Existing history files have group and other permissions removed when opened.
• resterm history backup no longer overwrites a file that isn't a SQLite database. Previous versions would delete it.
• resterm history compact exits with 1 if another process prevents the database from shrinking.
• The headless Go API has breaking changes to usage errors, failure rules, and JSON field names. If you use it from Go, read the release notes first.