yhirose/cpp-httplib v0.62.0

on GitHub

What's Changed

Behavior changes

Several fixes below change behavior that existing code may rely on:

• Server: an HTTP/1.1 request without Host, a request line whose parts are not separated by a single SP, and a request-target containing # are answered with 400.
• Server: a Content-Length or Transfer-Encoding set by a handler is replaced by the one worked out from the body (a HEAD response keeps the handler's length). 1xx, 204 and 304 responses carry no Content-Length.
• Server: a response from a content provider with neither a length nor chunked coding closes the connection after it.
• Server: the request line and headers together must arrive within the read timeout (5 seconds by default). Each read used to get the full timeout.
• Server: only regular files are served, and set_mount_point() returns false for a directory whose real path cannot be resolved.
• Server: a content setter replaces a Content-Type already in the response, including one set with set_header() before it. Pass the type to set_content(), set_content_provider(), set_chunked_content_provider() or set_file_content() instead.
• Client: a URL with userinfo, or with a backslash in its authority, gives an invalid client whose requests fail with Error::Connection.
• Client: a CONNECT answered with anything but 200 fails with Error::ProxyConnection. Only a 407 is still returned as the response.
• Client: a buffered response body over set_payload_max_length() fails with Error::ExceedMaxPayloadSize instead of Error::Read.
• Client: with set_follow_location(true), a 3xx without Location is returned as the response instead of failing with Error::Unknown.
• Client: the error logger receives each failed request once, with the request the caller made. A redirect client no longer has the error logger.
• TLS: OpenSSL servers that verify client certificates now let clients resume sessions, unless a verify callback is set. wolfSSL servers that verify client certificates no longer do. OpenSSL servers refuse client-initiated renegotiation.
• Limits: 0 now means no limit for set_keep_alive_max_count(), CPPHTTPLIB_WEBSOCKET_MAX_PAYLOAD_LENGTH, CPPHTTPLIB_FORM_URL_ENCODED_PAYLOAD_MAX_LENGTH, CPPHTTPLIB_MULTIPART_FORM_DATA_FILE_MAX_COUNT and CPPHTTPLIB_RANGE_MAX_COUNT. At 0, the last three used to reject every request with a URL-encoded form body, a multipart body or a Range header, so a build that set one of them to 0 to turn that off now accepts such requests. CPPHTTPLIB_REQUEST_URI_MAX_LENGTH, CPPHTTPLIB_HEADER_MAX_LENGTH, CPPHTTPLIB_HEADER_MAX_COUNT and CPPHTTPLIB_MAX_LINE_LENGTH cannot be lifted and must be positive (checked by static_assert). A WebSocket ping interval of 0 or less turns pings off.
• Header lookups refuse REMOTE_ADDR, REMOTE_PORT, LOCAL_ADDR and LOCAL_PORT in any letter case, as they already did in upper case.

Security-relevant fixes

• Request parsing and message framing:
  • Read the request line strictly. Any run of spaces or tabs separated its parts, a fragment in the target was stripped, and an HTTP/1.1 request without Host was served, so a server or cache in front that follows RFC 9112 could see a different request. The parts must now be separated by a single SP, and a target with # or an HTTP/1.1 request without Host is answered with 400 (RFC 9112 §3, §3.2).
  • Close the connection after a partly read request body. When a content receiver returned false or threw partway through a body, the server drained a whole Content-Length again from where reading stopped, swallowing the start of the next request and parsing the rest as a request of its own. The response now carries Connection: close, and a request body can be read only once.
  • Close the connection after ambiguous framing. A request or response with both Transfer-Encoding and Content-Length, or with Transfer-Encoding in HTTP/1.0, left the connection open, though RFC 9112 §6.1 has it closed. The server and the client now close it, and the server sends Connection: close whenever it closes after a response.
  • Frame response bodies in one place. A content provider that wrote past the length it was given put the extra bytes after the body, where a keep-alive peer read them as the next message; such a write now fails, on the server and on the client. A Content-Length or Transfer-Encoding set by a handler was sent next to the one the library worked out, and a post-routing handler could change the body after the length was set. Both are now worked out from the body about to be written, after the post-routing handler (RFC 9110 §8.6).
  • Keep a single content source in a Response. set_content() did not drop a content provider installed earlier, so the provider was asked for more bytes than it had: a file under a mount point could go out under the error status a file request handler set to hide it, and the file mapping or a user provider was read past its end. Every content setter now replaces the others, and so does assigning res.body directly.
  • Honor quoted-pairs in parameter values. An escaped quote ended a quoted-string, so a Content-Disposition parameter could be split into other parameters than an RFC 9110 parser in front of the server sees. \" and \\ are now unescaped in Content-Disposition and media type parameters (other backslashes, such as those of a Windows path, are kept), and the multipart writer escapes a backslash.
  • Keep a %u escape that does not decode to a character. decode_path_component() dropped a %u escape of a surrogate and joined the text on either side of it, so the decoded path could differ from the one a filter in front had checked.
  • Compare names as strings, without case. Content type lookup, the compressible type check and the guard on the reserved names REMOTE_ADDR, REMOTE_PORT, LOCAL_ADDR and LOCAL_PORT compared only a 26-bit hash of the name, so a file with an unknown extension could be served as text/html. The guard was also case-sensitive, so a lookup spelled in another case returned a header the client sent.
• Static files:
  • Serve no file whose real path cannot be resolved. When resolving it failed, the check that the file stays inside the mount point was skipped and the file was served.
  • Redirect a directory request with a relative Location (#2628, #2630). The trailing-slash redirect for a directory could produce a Location that clients read as a network-path reference to another host, and under a mount point other than / it pointed to the wrong path. The Location now names only the last segment, percent-encoded and prefixed with ./, so it always resolves against the request URL.
• TLS:
  • Keep TLS verify callbacks per context. tls::set_verify_callback(), which set_server_certificate_verifier() uses, kept the callback in one thread_local slot, so a verifier was skipped when the handshake ran on another thread, ran for other clients on the thread that set it, and could run after its client was destroyed. The callback is now stored with the context on every backend, and an exception thrown from a callback rejects the certificate instead of unwinding through the TLS library.
  • Keep client certificates required after setting a verify callback. Setting the callback reset the verify mode, so a server that required client certificates accepted a client without one (OpenSSL, Mbed TLS and wolfSSL). The mode is now derived from both settings in one place.
  • Turn off session resumption on wolfSSL servers that verify client certificates. wolfSSL shares its session cache between contexts, so a session resumed on such a server could carry a certificate checked against another server's CA, or against a client CA replaced since.
  • Complete server certificate updates. update_certs_pem() replaced the certificate, key and client CAs while handshakes on other threads were reading them, which could crash a server rotating certificates under load. It now waits for running handshakes, and OpenSSL servers no longer allow client-initiated renegotiation, which runs a handshake the update cannot wait for. wolfSSL added new client CAs to the old ones instead of replacing them, and invalid material left Mbed TLS and wolfSSL half updated; new material is now checked before it is swapped in.
  • Match certificate names as X509_check_host() does on Mbed TLS and wolfSSL. These backends accepted wildcard patterns that OpenSSL rejects (a wildcard followed by fewer than two labels, a wildcard in an IDNA label, a name with an empty label), and fell back to the subject CN even when the certificate had a DNS SAN (RFC 6125 §6.4.4). Every backend now accepts the same names, and a trailing dot on the host is ignored.
  • Keep an accepting verifier from overriding Mbed TLS verification. With built-in verification on, a server certificate verifier that returned true made Mbed TLS accept a certificate its own check had rejected, while OpenSSL and wolfSSL still failed with SSLServerVerification. The verifier is now an additional check on every backend.
  • Bound a TLS handshake by its timeout as a whole. Every wait for the socket got the full timeout again, so a peer that sent a byte now and then could hold a handshake, and the worker serving it, indefinitely.
• Client:
  • Keep a client's limits and TLS checks across redirects (#2632, #2634). The client created for a cross-host redirect did not get the payload limit, the max timeout, the microseconds of the connection timeout or the hostname address map. Over HTTPS it also lost the CA store given to set_ca_cert_store(), falling back to the default CAs, and the server certificate and session verifiers, so a pinning verifier never ran on the host redirected to. These now carry over. After such a redirect, the original client also answered a 401 or 407 from the new host with its own Digest credentials; the redirected response is now returned as it is.
  • Fail a refused CONNECT instead of returning the proxy's answer. A non-200 answer was returned as the response, so a proxy, or anyone on the plain connection to it, could answer an HTTPS request as if the origin had. Such a request now fails with Error::ProxyConnection. A 407 is still returned as the response, so the caller can see that the proxy wants credentials.
  • Reject userinfo and backslashes in a URL authority. The URL parser took userinfo as part of the host, where parsers that honor userinfo, or read a backslash as /, see another host. Userinfo is not supported, so such a URL now gives an invalid client.
  • Use a non-deterministic source for the WebSocket masking key and Sec-WebSocket-Key, multipart boundaries and the Digest cnonce (#2629). They came from a std::mt19937 seeded once with 32 bits, while RFC 6455 §5.3 requires an unpredictable masking key. This also fixes an all-zero masking key on 64-bit big-endian builds.
  • Close the connection after an early answer to Expect: 100-continue. When the final response came before 100 Continue, the body was never sent but the connection was kept, so the server could read the next request as that body. A retry or redirect after it opens a new connection.
  • Skip every interim response, with its header fields. Only a 100 Continue without header fields was skipped, so a 103 Early Hints was taken for the final response and a 100 with fields threw the parsing off. Every 1xx other than 101 is now skipped, up to CPPHTTPLIB_INTERIM_RESPONSE_MAX_COUNT (100), past which the request fails with Error::Read.
• Resource limits:
  • Bound the request head by the read timeout as a whole. The read timeout applied to each read of the request line and headers, so a client sending them a little at a time could hold a worker for as long as it liked. A request body is still bounded per read; the README describes what to put in front of a server that faces untrusted clients.
  • Grow a WebSocket payload only as it arrives. The payload was allocated and zero-filled to the length a frame header declared before any of it was read, on the server and on the client. Each fragment of a message is now also read against what is left of the message limit, as a fragmented message could take about twice the limit.
  • Do not reserve a response body by its claimed length. The client reserved what Content-Length claimed before any of the body arrived, without bound when there was no payload limit. It now reserves at most 1 MiB up front.
  • Bound the size of an SSE event. The SSE client buffered a line or the data of an event with no limit of its own. Past the new CPPHTTPLIB_SSE_MAX_EVENT_LENGTH (16MB), it ends the connection with Error::ExceedMaxPayloadSize and does not reconnect.
  • Resume the multipart header search where it stopped. A part header arriving a few bytes at a time was rescanned from the start on every read, which made parsing it quadratic.
  • Remove recursion that a long input could drive into a stack overflow. std::regex recursed once per character of a status line's reason phrase on the client, and the static file type lookup ran the same kind of regex over the request path. case_ignore::hash recursed once per character of a header or trailer name in builds that keep the recursion (-O0). All of them now run in loops.

New features

• WebSocketClient supports HTTP proxies (#2635): set_proxy(), set_proxy_basic_auth(), set_proxy_bearer_token_auth(), set_no_proxy() and, in SSL builds, set_proxy_digest_auth(). connect() opens a CONNECT tunnel for both ws:// and wss:// (RFC 6455 §4.1). A proxy that does not answer 200 is reported as Error::ProxyConnection, with its status and headers in the Result.
• WebSocketClient::set_server_certificate_verifier() and set_session_verifier(), as SSLClient has, so a wss client can pin a certificate.
• Keep the TLS error behind a failed read or write in the client's Result. On TLS 1.3, where a server rejects a client without a certificate only after the handshake, ssl_error() and ssl_backend_error() were 0. They now report the TLS error, and are reset at the start of each request so a later success does not report an earlier error. error() stays Error::Read or Error::Write.

Bug fixes

• Server:
  • Spawn a dynamic thread when queued jobs outnumber idle threads (#2633). Threads already notified for a job still counted as idle, so a burst of connections could leave jobs queued with no thread while the pool was below max_threads. With keep-alive, such a connection waited until another one closed.
  • Close the connection after a response body with no length. A content provider with neither a length nor chunked coding sends a body that only the end of the connection delimits (RFC 9112 §6.3), yet the response offered keep-alive, so a keep-alive client waited for the server's idle timeout.
  • Replace the Content-Type along with the content. set_content_provider() and file content added their Content-Type next to one set by an earlier set_content(), so the response carried two. A Content-Type set after set_file_content() is now used for the file.
  • Read served files with pread() and serve only regular files (POSIX). Over TLS a file was copied out of its memory mapping, so a file truncated while being served raised SIGBUS and took the server down. Files are now opened close-on-exec and without blocking on a FIFO, and a device is no longer served as an empty body.
  • Run the pre-routing handler once for an unmatched WebSocket upgrade. When no WebSocket route matched, the request was routed as plain HTTP and the handler ran a second time.
  • Match file extensions, media types and media type parameter names without case (RFC 9110 §8.3.1). INDEX.HTML got the default type, Multipart/Form-Data was not taken for multipart, and user extension mappings were case-sensitive.
  • Treat a limit of 0 as no limit everywhere it can be lifted. set_keep_alive_max_count(0) tripped an assert, and the form length, multipart file count and range count macros rejected every request at 0. A negative WebSocket ping interval sent pings in a tight loop.
• Client:
  • Return a 3xx without a Location instead of failing the redirect. With set_follow_location(true), a 304 Not Modified or any 3xx without Location failed with Error::Unknown. It is now returned as the response, as curl, Go, requests and urllib3 do, and response handlers and content receivers get it too.
  • Report a response body over the limit as Error::ExceedMaxPayloadSize, also when the limit is reached while decompressing. A buffered request failed with Error::Read, while the stream API already reported ExceedMaxPayloadSize.
  • Report each failed request to the error logger once. A failure could be reported twice, once without the request when it came from the TLS setup or the proxy CONNECT, a canceled content receiver or progress callback was reported twice, and a stream that failed to open was not reported. The outermost call now reports it once, with the request the caller made, also when the failure happened on a redirect.
  • Send the client's credentials and proxy target on a stream request. open_stream(), and with it stream::Get() and SSEClient, left out the Authorization from set_basic_auth() or set_bearer_token_auth(), and through a plain HTTP proxy the absolute-form target (RFC 9112 §3.2.2) and Proxy-Authorization.
  • Treat a payload limit of 0 as no limit when the stream API decompresses. Every compressed response read through the stream API failed with set_payload_max_length(0).
  • Parse chunked trailers when the stream API decompresses. response->trailers stayed empty for a compressed chunked response. Trailers are now parsed only after the last chunk.
  • Do not report a body that ends with the connection as an error. For a response with neither Content-Length nor chunked coding (RFC 9112 §6.3), the stream API set read_error() to ConnectionClosed at the end of the body, and SSEClient passed it on to on_error.
  • Sign the request-target that is sent in a Digest uri. The Digest uri used the raw path, while the request line carried it encoded, with the params and without the fragment, so a server comparing the two rejected the credentials.
  • Do not send the fragment of a URL (RFC 9110 §7.1), and insert Params before a fragment instead of after it.
  • Keep a Client usable when its URL cannot be used. An unsupported scheme (without exceptions) or a port out of range left Client with no implementation, and the first request dereferenced a null pointer. It is now an invalid client whose requests fail with Error::Connection.
• SSE client:
  • SSEClient::stop() interrupts a blocked read. stop(), and the destructor with it, waited until the server sent something or the read timeout passed (300 seconds by default).
  • An event stream is no longer cut by payload_max_length (100MB by default), which made the client reconnect. Each event stays bounded by CPPHTTPLIB_SSE_MAX_EVENT_LENGTH.
  • Dispatch an event whose event field is empty as "message", as the HTML standard does. It never reached on_event("message").
• WebSocket:
  • Wake a blocked read() on a pong timeout on Windows too. The fix in v0.61.0 shut down the read side of the socket, which does not wake the wait on Windows, so read() still waited for its read timeout. read() now also returns Fail rather than Timeout for a connection that has closed, since Timeout means the connection is still usable.
  • Validate a Close frame before echoing it. Any Close payload was echoed back, including a 1-byte one, a code that may not be sent, and a reason that is not UTF-8. An invalid one is now answered with 1002. WebSocket::close() cuts the reason back to a whole UTF-8 character and sends no payload with a code that may not be sent.
  • Treat CPPHTTPLIB_WEBSOCKET_MAX_PAYLOAD_LENGTH of 0 as no limit. It rejected every frame with a payload.
• TLS:
  • Let a client resume its session with an OpenSSL mTLS server. Without a session id context, OpenSSL answered a resuming client with a fatal alert once client certificates were verified, which curl and browsers ran into on their second connection. A server with a verify callback does not resume sessions, since a resumed handshake does not run the callback, and a change of client CAs flushes the session cache and rotates the ticket keys.
  • Honor the PEM lengths in PemMemory. cert_pem_len and key_pem_len were ignored and every backend read up to a NUL, so a buffer holding exactly the PEM was read past its end. A length of 0 still means a NUL-terminated string.
  • Stop leaking CA certificates from tls::get_ca_certs() on Mbed TLS. It handed out copies that free_cert() does not free. It now returns the context's own certificates, valid as long as the context's CA certificates are unchanged.
  • Pass the session wrapper to wolfSSL verify callbacks. VerifyContext::session held the raw WOLFSSL pointer, while every other tls:: function takes the session wrapper.
  • Set the client verify mode under the context lock on Mbed TLS and wolfSSL, where it is a setting of the context shared with session creation.

Build

• Avoid getaddrinfo_a() under ThreadSanitizer (#2631). glibc runs its lookups in a thread TSan never initializes, which crashed. The std::thread based fallback is used instead.
• Fix compiler warnings from clang, 32-bit builds, BoringSSL and MSVC.

Documentation

• README: describe what follows a cross-host redirect, when an SSL client loads its CA certificates, how the read timeout bounds a request and what to put in front of a server facing untrusted clients, how multipart names with backslashes are read, and that a Content-Length or Transfer-Encoding a client caller passes has to match the body.
• README: note that the file request handler is not called for HEAD or a 304 and should not be used for access control, and that regex routes need simple patterns, especially on threads with a small stack.
• README-websocket: note that the handler is called whatever the Origin of the handshake, so a server that authenticates with cookies should check it.
• README, README-stream, README-websocket, README-sse and the docs site (en/ja): fix a client sample that did not compile and statements that no longer matched the code, describe the behavior changes in this release, complete the API references of WebSocket, WebSocketClient and the stream:: helpers, and fix cookbook samples, among them a multipart sample that trusted the client's file name.

Development

• Fuzz the buffered path of stream_line_reader::getline().

Full Changelog: v0.61.0...v0.62.0