hatchet-dev/hatchet v0.114.2 on GitHub

Hatchet v0.114.2 adds rate limit pausing, RetryAfter errors, and FIPS images. It also includes a set of security fixes and a large batch of DAG, scheduler, and database fixes.

Highlights

Rate limit pause: a rate limit key can be paused. Tasks that use the key wait, and do not consume the limit, until the pause ends (#5209).
RetryAfter errors: a task can return a RetryAfterError to choose when its next retry runs. See Retry Policies (#5162).
FIPS images: every server image is also published with a -fips suffix, built against the validated Go Cryptographic Module. See FIPS 140-3 Images (#4997).
Workflow pause in the Go SDK: workflows can now be paused and unpaused from the Go SDK (#5025).
Dashboard: the trigger dialog has a Form tab built from the workflow's input schema, the runs table has a "Replay as new" action that re-triggers a run with its original input, and the workflows page shows triggers, description, and last deploy (#5183, #5140, #5242).
Engine: the lookup, DAG data, and DAG-to-task tables in the core and OLAP databases are now partitioned by time, like the task tables already were (#5038, #5050).
Engine: payloads are stored once and shared between events, DAGs, and tasks instead of being written several times (#5196, #5198, #5224).
Self-hosting: default entitlements for new tenants can be set with env vars (#5266).

Security

Build: Go 1.26.10, newer base images, and fixed Node releases block the compromised eMudhra CAs (CVE-2026-111943) (#5256, #5258, #5275).
Auth: an API token's JWT subject must match the tenant stored for that token (#5215).
Engine: SNS message verification is stricter. Signed fields other than Message may not contain newlines, the signing certificate URL must be a regional SNS host with a .pem path, outbound requests have a timeout and a 1 MiB response cap, and messages signed more than an hour ago are rejected (#4926, #5279).
Engine: worker slot units must be zero or more and are summed without overflow, so a worker cannot skip the tenant slot limit (#5274).
API: listing workflow runs with payloads checks payload permissions before querying (#5205).

Fixed

Engine: DAG replays no longer hang or retry forever when a running task is replayed, a child is spawned twice, or tasks finish out of order (#5186, #5189, #5207, #5265).
Engine: the DAG operator no longer claims idempotency keys that its children need, no longer reports a cycle when a task shares its workflow's name, and no longer checks idempotency keys for DAG steps (#5136, #5137, #5211).
Engine: task errors are passed to the on-failure task, returned verbatim without a dag failed: prefix, and internal errors no longer leak into task run errors (#5178, #5302, #5212).
Engine: skip_if and cancel_if conditions are evaluated when a parent returns empty output, empty parent outputs still appear in the parents map, and an invalid CEL condition is ignored instead of failing the DAG (#5295, #5296).
Engine: a failing CEL expression no longer causes a flood of retries (#5151).
Engine: the DAG operator is registered for workflows put by very old SDKs, skips deleted tenants, and is metered once per run instead of once per step (#5203, #5177, #5158).
Engine: CANCEL_* concurrency strategies no longer cancel a task's own retry (#5229).
Engine: when a later concurrency batch fails, results from batches that already committed are still published, so cancelled tasks get their cancellation and filled tasks are not delayed (#5155).
Engine: the in-memory concurrency index uses much less memory per key, and its build timeout is 30s (#5156, #5214, #5228).
Engine: idle queues go inactive again instead of being polled forever, the rate limit requeue runs without a transaction and only when there is work, and REQUEUED_RATE_LIMIT events are reported at most once a minute per task (#5144, #5138, #5245, #5170).
Engine: retry decisions for a batch of failed tasks match each task to its own row. Large batches previously got wrong results or timed out and never retried (#5243).
Engine: the trigger path caches event-to-workflow lookups, workflow versions, step expressions, and match conditions, and runs its preflight checks before opening a transaction (#5251, #5259, #5165).
Engine: cron and scheduled pollers no longer rewrite rows they already own on every poll (#5301).
Engine: new indexes for DAG orchestration action lookups, stuck evicted durable orchestrators, and cron name search, and a redundant OLAP runs index is dropped (#5288, #5160, #5267, #5090).
Engine: the durable task connection runs one worker-status handler per worker, fixing an out-of-memory crash when a worker stops reading, and a wait_for condition without a base returns an error instead of crashing the handler (#5201, #5260).
Engine: statement timeouts now apply behind a transaction-mode PgBouncer, with a startup warning when they are missing. See Using PgBouncer (#5065, #5218).
Engine: the Go memory limit is set to 90% of the container limit to reduce OOM kills (#5180).
Engine: orchestrator terminal events are written to OLAP once, retried tasks are counted by their final status in the workflow duration metric, and is_dag_orchestrator is recorded on task events (#5161, #5176, #5199).
Engine: task durations were sometimes matched to the wrong task (#5227).
Engine: empty and null payloads are no longer written to the payloads table (#5252, #5253).
Engine: authentication returns a 5xx error instead of "invalid token" when the database is unreachable (#5173).
Engine: analytics events from in-process callers carry the tenant id (#5120).
API: listing workflow runs with payloads returns 400 above 1,000 runs or 1,000 DAG child tasks, and DAG outputs are no longer duplicated per child task (#5205).
Dashboard: run counts load separately from the runs list, which makes the runs page much faster for tenants with many runs (#5164).
Dashboard: workflow search UX fixes, the Manage Billing button is hidden on legacy plans, and the support chat widget follows the app theme (#5036, #5150, #5219).
Go SDK: waiting for a workflow result ends when the caller's context ends, and WithToken overrides the API token for the OTel collector (#5234, #5217).
Go SDK: durable status polling fixes, and shutdown and evict acks are serialized (#5128).

What's Changed

fix(engine): dag operator cycles from task and workflow sharing names by @mrkaye97 in #5137
fix(dashboard): workflow search ux tweaks by @mrkaye97 in #5036
fix(ts, ruby, go): durable status polling fixes, serialize shutdown and evict acks by @mrkaye97 in #5128
fix(engine): dag operator causes idempotency key collisions by default by @mrkaye97 in #5136
feat(go,ts,rb): add workflow pause to the rest of the sdks by @mrkaye97 in #5025
fix(operatorsvc): set the analytics tenant id on in-process contexts by @grutt in #5120
perf(engine): requeue rate limited items without a transaction by @grutt in #5138
fix(ts): shutdown causes heartbeat error flood in logs by @mrkaye97 in #5143
fix(scheduler): let idle queues go inactive again by @igor-kupczynski in #5144
chore: dependabot fixes by @mnafees in #5146
fix: do not sign tags when originated from Taskfile by @mnafees in #5148
feat(engine): skip stale NATS pub/sub messages instead of working through a backlog by @igor-kupczynski in #5103
feat(engine): run NATS pub/sub handlers on a bounded per-subscription pool by @igor-kupczynski in #5104
fix: Consolidate entitlement logic by @juliusgeo in #5149
fix(frontend): hide Manage Billing button on legacy plans by @jbsouvestre in #5150
fix(engine): retry flood on CEL evaluation failures in DAG operator by @mrkaye97 in #5151
feat(durable-streams): large messages, metadata by @juliusgeo in #5139
perf(olap): drop ix_v1_runs_olap_tenant_ins_at_status by @igor-kupczynski in #5090
feat(engine): partition core tables by @mrkaye97 in #5038
fix(engine): charge DAG-operator runs once for TASK_RUN by @igor-kupczynski in #5158
fix(engine): run analyze in migration by @mrkaye97 in #5159
fix(engine): avoid seq scan on ListStuckEvictedDurableOrchestrators by @mrkaye97 in #5160
test(python): couple small fixes to python tests by @mrkaye97 in #5163
feat: partition more olap tables by @mrkaye97 in #5050
fix: throttle REQUEUED_RATE_LIMIT monitoring events to once per task per minute by @mnafees in #5170
fix(engine): stop double-writing orchestrator terminal events to OLAP by @igor-kupczynski in #5161
fix(dashboard): split count queries to improve page performance by @mrkaye97 in #5164
feat: add RetryAfter(Exception|Error) to SDKs and engine by @juliusgeo in #5162
fix(engine): return 5xx if we fail to reach the db on auth by @mrkaye97 in #5173
fix(python): re-add log queue max size by @mrkaye97 in #5175
fix(engine): keep committed concurrency results when a later batch fails by @igor-kupczynski in #5155
fix: ignore deleted tenants in DAG operator claim by @mnafees in #5177
fix(engine): dag operator not propagating task run errors back to on failure task by @mrkaye97 in #5178
chore: Bump github.com/rabbitmq/amqp091-go from 1.13.0 to 1.14.0 in /examples/go/embedded by @dependabot[bot] in #5174
feat(dashboard): add Replay as new action to the runs table by @jbsouvestre in #5140
fix: SNS ingestor outbound requests hardening by @mnafees in #4926
feat: publish -fips images built against the validated Go crypto module by @mnafees in #4997
fix: automemlimit by @jbsouvestre in #5180
fix(engine): dag replays causing infinite retry loop by @mrkaye97 in #5186
fix(engine): dag replays, part ii by @mrkaye97 in #5189
test(loadtest): add a concurrency strategy workload by @igor-kupczynski in #5192
chore(frontend): add knip and remove unused dependencies by @jbsouvestre in #5191
perf(engine): shrink the in-memory concurrency index by @igor-kupczynski in #5156
feat(durable-streams): add Reset method for durable streams by @juliusgeo in #5194
chore: reporters need to mention if they used AI to find security findings by @mnafees in #5202
fix(dispatcher): run one durable worker-status handler per stream by @igor-kupczynski in #5201
fix(engine): dag operator not registered on very old SDKs by @mrkaye97 in #5203
fix(engine): statement timeouts behind transaction poolers by @igor-kupczynski in #5065
fix(engine): wire up is_dag_orchestrator col on the task events by @mrkaye97 in #5199
chore(engine): duped migration version by @mrkaye97 in #5206
chore(engine): improve in-memory concurrency index visibility by @mrkaye97 in #5197
fix(engine): durable dags hang on replays of running tasks by @mrkaye97 in #5207
fix(engine): don't check idempotency keys for dag steps by @mrkaye97 in #5211
fix(engine): stop leaking errors from the operator by @mrkaye97 in #5212
fix(api): cap tasks per workflow-run list read that includes payloads by @igor-kupczynski in #5205
fix(auth): verify JWT subject matches stored API token tenant by @igor-kupczynski in #5215
chore: Bump the go-modules group across 1 directory with 20 updates by @dependabot[bot] in #5182
chore: Bump the npm-dev-deps group across 2 directories with 4 updates by @dependabot[bot] in #5193
feat(go-sdk): option to override API token for Hatchet OTel collector by @mnafees in #5217
fix: bump index build timeout for concurrency Strats by @juliusgeo in #5214
feat(frontend): sync Pylon chat widget theme with app theme by @jbsouvestre in #5219
fix: cross-batch terminal events to OLAP with retried tasks would cause incorrect Prom metrics by @juliusgeo in #5176
chore(engine): add exhaustiveness and sum type checking linters by @mrkaye97 in #5221
feat(dashboard): JSON Schema form for workflow trigger input by @jbsouvestre in #5183
chore: Bump github.com/getkin/kin-openapi from 0.144.0 to 0.149.0 in /examples/go/embedded by @dependabot[bot] in #5223
fix: GetTaskDurationsByTaskIds could get wrong row for taskId by @juliusgeo in #5227
fix: set concurrency index creation timeout to 30s by @juliusgeo in #5228
chore: dependabot fixes by @mnafees in #5230
fix(test): reduce flake on cancellation during engine startup by @abelanger5 in #5231
fix(test): flake on sending action deltas in operator client by @abelanger5 in #5232
fix: match each failed task to its own locked row when deciding retries by @igor-kupczynski in #5243
fix: cancel_* concurrency strategies could cancel a task's own retry by @juliusgeo in #5229
chore(engine): bump pgoutbox by @mrkaye97 in #5244
test: use timeout_func_only = true for pytest by @abelanger5 in #5247
fix(go-sdk): end the workflow result wait when the caller's context ends by @abelanger5 in #5234
perf(scheduler): don't run the rate limit requeue on every queue iteration by @abelanger5 in #5245
fix(docs): return 405 for SSE GET requests on stateless MCP endpoint by @grutt in #5249
feat(frontend): add Storybook to the dashboard app by @jbsouvestre in #5250
feat(dashboard): show triggers, description and last deploy on the workflows page by @jbsouvestre in #5242
fix(dev): set statement timeouts through local PgBouncer by @igor-kupczynski in #5218
test(typescript): deflake durable eviction test by @abelanger5 in #5246
refactor(engine): simplify task data shape by @mrkaye97 in #5198
fix(engine): don't write null-string payloads for task inputs by @mrkaye97 in #5252
fix(engine): remove one more source of null::jsonb payloads by @mrkaye97 in #5253
perf: use caches properly in the trigger path by @abelanger5 in #5251
chore: bump Go to 1.26.10 for eMudhra CA distrust by @igor-kupczynski in #5256
refactor(engine): typed payload shapes by @mrkaye97 in #5196
chore: upgrade ca-certificates on publishing CI jobs (eMudhra distrust) by @igor-kupczynski in #5258
feat: serverless operator by @abelanger5 in #4912
refactor(engine): shared payloads by @mrkaye97 in #5224
fix(engine): handle durable wait_for conditions without base by @igor-kupczynski in #5260
perf: add some more caching to the trigger path by @abelanger5 in #5259
fix: exhaustive checks from serverless operator by @juliusgeo in #5261
fix: missing eventKeyWorkflowsCache in task_test.go by @juliusgeo in #5263
fix(ts-core): properly export HatchetError and redirect settings by @abelanger5 in #5264
feat(engine): add env vars for default tenant entitlements by @mrkaye97 in #5266
ci: use fixed Node in publish jobs for eMudhra CA distrust by @igor-kupczynski in #5275
fix(engine): tighten SNS signature verification by @igor-kupczynski in #5279
fix(engine): reject negative worker slot units by @igor-kupczynski in #5274
fix(engine): replays of dag tasks can cause hang by @mrkaye97 in #5265
feat(engine): index cron names for search by @mrkaye97 in #5267
docs: list Sydney as a live data plane region by @grutt in #5284
feat: add durable streams to the core SDK by @abelanger5 in #5282
refactor: add read_payloads as sql method which can be shared across queries by @abelanger5 in #5281
feat(engine): rate limit pause by @juliusgeo in #5209
fix(db): index DAG orchestration action lookup by @grutt in #5288
fix: all open Dependabot alerts by @grutt in #5287
fix(engine): [dag operator] return outputs even when empty, and ignore invalid CELs on conditions by @mrkaye97 in #5295
fix(engine): evaluate dag conditions against empty parent output by @igor-kupczynski in #5296
fix(engine): return errors out of the dag operator verbatim by @mrkaye97 in #5302
chore(engine): fix migration numbering by @mrkaye97 in #5303
perf(ticker): stop rewriting already-owned cron and scheduled rows on every poll by @grutt in #5301
feat: REST API, refreshTimeout, and releaseSlot in the /core package by @abelanger5 in #5292
feat: add durable streams to the serverless operator by @abelanger5 in #5293
fix(ts): durable stream shutdown, worker label propagation by @mrkaye97 in #5304
fix: allow batching messages on publish for durable streams by @juliusgeo in #5308
fix: accidentally broke backwards compatibility with engine by @juliusgeo in #5311
fix: bump typescript SDK version by @juliusgeo in #5312
chore: Bump the npm-dev-deps group across 1 directory with 9 updates by @dependabot[bot] in #5324
chore: Bump the github-actions group across 1 directory with 7 updates by @dependabot[bot] in #5323
chore: Bump zod from 4.3.6 to 4.6.6 in /frontend/docs by @dependabot[bot] in #5306
chore: Bump the go-modules group across 1 directory with 18 updates by @dependabot[bot] in #5322
chore: Bump github.com/labstack/echo/v4 from 4.15.4 to 4.16.1 in /sdks/go/examples/embedded by @dependabot[bot] in #5310
refactor(engine): run trigger preflight before the owned transaction by @igor-kupczynski in #5165
fix: dependabot alerts by @mnafees in #5325
feat(durable-streams): add a frontend UI by @juliusgeo in #5285
feat(engine): add fairpool connection gate by @igor-kupczynski in #5166
chore: changelog entries for next OSS release by @mnafees in #5329

Full Changelog: v0.110.5...v0.114.2

hatchet-dev/hatchet v0.114.2 Hatchet 0.114.2 on GitHub

Highlights

Security

Fixed

What's Changed

hatchet-dev/hatchet v0.114.2
Hatchet 0.114.2

on GitHub