unkn0wn-root/resterm v1.11.0 on GitHub

v1.11.0

This release fixes two security issues where values from a server response could cause Resterm to send local files or secrets. It also changes how scripts and @apply handle templates, so some existing scripts will need updating. Other fixes cover resterm run, gRPC TLS, SSH tunnels, OAuth refresh, OpenAPI import, and resterm record.

Breaking security change: scripts and `@apply` no longer expand variable templates

Values passed to request.setHeader, setURL, setQueryParam, setBody, and the other set* helpers are now data. Templates in them are not expanded. Only dynamic helpers such as {{$uuid}} are rendered.

### Profile
# @script pre-request
> request.setHeader("Authorization", "Bearer {{api.token}}");
GET https://api.example.com/me

In 1.10.3, this request sent the token. Now it sends the text Bearer {{api.token}}. Build the value from variables instead:

# @script pre-request
> request.setHeader("Authorization", "Bearer " + vars.get("api.token"));

The same applies to @apply and @patch. A patch returns data, so headers, query, body, auth, and vars in it are not expanded:

# @apply {headers: {"Authorization": "Bearer {{api.token}}"}}             sends {{api.token}} as text
# @apply {headers: {"Authorization": "Bearer " + vars.get("api.token")}}  sends the token

An @ path line in a body set by setBody or @apply {body: ...} is sent as text. The file is not read.
A {{= ... }} expression read with vars.get and written back is sent as text. Compute the value in the script, or read it through a getter.
A token printed by @auth command is sent as printed, even when it contains {{.

Getters now resolve variables and expressions before returning a value, so code that reads a value and writes it back keeps working:

# @script pre-request
> request.setURL(request.getURL() + "?debug=1");
GET {{base}}/users

request.getURL() and request.getHeader() in JavaScript, and request.url, request.headers, request.header(), and request.query in RestermScript, now expand variables and {{= ... }} expressions. In 1.10.3, request.getURL() returned the text {{base}}/users.

Values set by the script itself are returned as written.
An undefined variable in the value is an error.
A helper written in the URL or a header, such as {{$uuid}}, stays as text in the getter, because each use makes a new value. Declare it with @request to read the value that is sent (see below).

Security fixes

Response values could read local files

A value from a server, placed in a body with a template, could add an @ path include line:

### Get note
# @capture global note {{response.json.note}}
GET https://api.example.com/note

### Save note
POST https://api.example.com/save
Content-Type: text/plain

{{note}}

If the server returned "note": "hi\n@ /home/me/.ssh/id_rsa", 1.10.3 read that file and sent its content in the body of Save note. Now only include lines written in the request file are read. A value placed by a template is sent as text.

The same happened when a script passed text with an @ path line to request.setBody.

Response values could expand secrets

A script that copied a server value into the request also expanded templates in that value:

### Next page
# @script pre-request
> request.setHeader("X-Next", vars.global.get("next"));
GET https://api.example.com/next

If the captured next was {{api.token}}, 1.10.3 sent the value of api.token. A name such as {{AWS_SECRET_ACCESS_KEY}} read the OS environment. Now the header is sent as {{api.token}}.

gRPC TLS settings were ignored

### Health
# @grpc grpc.health.v1.Health/Check
# @setting grpc-root-cas ./certs/ca.pem
GRPC api.internal:443

The docs say any gRPC TLS setting turns on TLS. In 1.10.3, Resterm connected without TLS unless the request also had @grpc-plaintext false. This covered grpc-root-cas, grpc-client-cert, grpc-client-key, and grpc-insecure. If the server also accepted plaintext, metadata and @auth tokens were sent unencrypted.

Now these settings turn on TLS. @grpc-plaintext true still forces plaintext.

Scripts read the values the request sends

A helper in a declared value is now rendered once, when the request starts:

### Create order
# @request id {{$uuid}}
# @script pre-request
> request.setHeader("X-Script-Id", vars.get("id"));
POST https://api.example.com/orders
X-Id: {{id}}

In 1.10.3, vars.get("id") returned the text {{$uuid}}, so the request sent two different UUIDs. Now X-Id and X-Script-Id are the same. @run var declarations read the same value too.

trace in @capture and @assert now belongs to the current response. In 1.10.3 it read the last traced response, so trace.enabled() was true on a request without @trace that ran after one with it.

resterm run

Ctrl-C now stops the run and still prints the report:

FAIL GET slow [perform request: Get "http://127.0.0.1:8080/slow": interrupt signal received]
SKIP GET next [skipped after cancel]
Summary: total=2 passed=0 failed=1 skipped=1

The exit code is 130. In 1.10.3, the process ended with no output.

A request skipped by @when no longer stops --fail-fast. In 1.10.3, the rest of the run was skipped.
A skipped workflow step no longer fails the workflow. JSON reports show the reason in skipReason.
@else fail= and @default fail= count as assertion failures and exit 1. In 1.10.3 they exited 3 (internal error).
An unknown environment in @compare fails before any request is sent and exits 2. In 1.10.3, the requests above it were sent first, and the run exited 3.
-H (--headers) works as the first flag. In 1.10.3, resterm run -H api.http printed the help. Only -h, --help, and help print help now.
Errors in @assert and @capture expressions point to the column in the file, not the column in the expression.

Exit codes and transports

gRPC certificate failures exit 22 (TLS). In 1.10.3 they exited 21 (network).
SSH tunnel failures, such as a refused connection or a rejected handshake, exit 27 (route). In 1.10.3 they were reported as network or unknown failures, and the error ended with an extra use of closed network connection.
On macOS, an untrusted HTTPS certificate exits 22. A TLS alert from the server, such as certificate required, exits 22 too.
WebSocket results show the ws:// or wss:// URL. In 1.10.3, the effective URL showed http:// or https://.

Auth, import, and record

OAuth refresh sends client credentials the same way as the first fetch. In 1.10.3, a client without client_secret, such as authorization_code with PKCE, sent client_id in the form on fetch, but a Basic header with an empty secret on refresh.
# @auth oauth2 cache_key=myapi takes grant and client_auth from the line that seeded the key. In 1.10.3, the short line always set grant=client_credentials and client_auth=basic.
Tokens saved by 1.10.x with --persist-auth are still found.
--openapi-server-index now applies to every request. In 1.10.3, the file's baseUrl used the chosen server, but each request set baseUrl back to the first server.
resterm record checks the .http or .rest extension before it starts. If the recorder cannot start, e.g., the port is busy, it no longer leaves an empty output file that blocks the next try.

SHA-512 in RestermScript

crypto.sha512(text) and crypto.hmacSha512(key, text) return hex-encoded digests, the same as the SHA-256 helpers:

### Signed
# @rts pre-request
> request.setHeader("X-Signature", crypto.hmacSha512(vars.get("api.secret"), request.url))
GET https://api.example.com/orders

Docs

Each docs topic now has its own page under docs/, the same pages as resterm.app. docs/resterm.md links to the new pages.
The docs now clarify several requirements. @auth apikey needs the placement, the name, and the value. A second @auth, @name, @poll, @retry, or other single-value directive in a request is a parse error, not a warning. The JavaScript response.body is a property, not body().

New contributors

Thanks @quelcom for adding SHA-512 and HMAC-SHA512 to RestermScript in #441.