This is a recommended release focused on verifiable offsets, data correctness, and resilient peer and chunk retrieval. Key highlights include signed absolute Arweave offsets on data responses and from /ar-io/offsets (now answering L1 IDs), so a client can verify an item's chunks against the chain from any source; importing verified parquet-l2 bands, such as ardrive-l2, into ClickHouse or SQLite so GraphQL answers for their items without unbundling; a fix for trusted-gateway fetches that cached arweave.net's JSON wrapper in place of the data (copies already cached start with {"accept-ranges" and need purging); X-AR-IO-Stable reported correctly for the first time, with verified stable data given the 30-day immutable cache tier; chunk fetches that time out answered 503 rather than a cacheable 404; AR.IO peer weights that recover after an outage, a peer list kept across restarts, and anchor lookups that fail over between peers; GraphQL fixes for signatures and owner keys read from wrong offsets, duplicated tags, upstream fan-out loops and fields lost from upstream results; blocked IDs answered 451 instead of being redirected to their sandbox; and opt-in metering of /graphql through the rate limiter and x402.
Added
-
Data responses for an item or transaction whose root is a stable L1 transaction with an offset recorded in the local index now carry
X-AR-IO-Root-Arweave-Start-OffsetandX-AR-IO-Root-Arweave-Data-Size(otherwise neither is sent): the root's place in the weave, in Arweave's own numbering (start =/tx/{id}/offset'soffset - size + 1), read from the local index only and signed with the other root headers (#1140). With the root-relative offsets already sent, a client can fetch the item's chunks by absolute offset from any gateway or node and check them against the block'stx_root;docs/arweave-offset-verification.mdgives the recipe.GET /ar-io/offsets/:idgainsrootArweaveStartOffset,rootArweaveDataSize,itemArweaveStartOffsetanddataArweaveStartOffset, and now answers L1 IDs. Items in a nested bundle whose place in its root may be unknown (#907) get no absolute offsets. -
An operator can capture the trace of one request: sent with
X-AR-IO-Trace: 1and the admin bearer key, the request runs in its own root span, which the head sampler always samples (whateverOTEL_TRACING_SAMPLING_RATE_DENOMINATORis) and the bundled OTEL collector always keeps (newforce-sample-policy), and the response carriesX-AR-IO-Trace-Idto find it by. Unforced requests are sampled exactly as before. The trace already records every retrieval source tried, so one curl answers "where did this come from, and what failed first" (#626). Ignored without the admin key. -
arweave_chunk_fetch_orphaned_total{result}counts shared Arweave peer chunk fetches that finished after every caller waiting on them had gone (#885). They are left running on purpose, as cache warming, which helps a later caller only if the result lands within the chunk promise cache's TTL. On one gateway 99.3% of peer-origin chunk requests were cancelled by the caller, so this measures how much fetch work outlives its callers. That work is bounded by the number of peer attempts, not by time (the per-attempt timeout stops once headers arrive); this is now documented where it arises (getChunkByAny). -
POST /ar-io/admin/invalidate-data({"id": "<id>"}) forgets what the gateway recorded for one ID: cached hash, stored location, trust and verification, in the index and in memory, so the next request resolves it afresh (#959 item 6). It is the way to correct a wrong record that later writes cannot replace, such as one already marked verified, without editing SQLite by hand. The cached bytes are kept (they are stored by hash and may belong to other IDs); caches in front of the gateway still need their own purge. -
ENABLE_GRAPHQL_METERING(default off) puts/graphqlunder the rate limiter and x402, at a flatGRAPHQL_OPERATION_COST_BYTESper operation (default 10 KiB, 10 tokens; a batch pays per operation), charged before the query runs. Until now the most expensive route a gateway serves was the one it could not meter. Only the client's IP bucket is charged: the per-path resource bucket would be a single allowance shared by every GraphQL client./ar-io/inforeports the cost and the price. -
CONTIGUOUS_DATA_CACHE_INDEX_MAX_BYTES_PER_SWEEP(default 0, off) caps the bytes one contiguous-cache eviction sweep frees, so eviction is paced by bytes per interval instead of arriving in bursts. On turbo-gateway.com's btrfs cache HDD one 10,000-file sweep freed 258 GB over 30 minutes and stalled the disk (p99 latency 232 s for two minutes). The budget is ignored while free space is belowCONTIGUOUS_DATA_CACHE_MIN_FREE_BYTES. Docker Compose now also passesCONTIGUOUS_DATA_CACHE_INDEX_UNLINK_CONCURRENCYandCONTIGUOUS_DATA_CACHE_INDEX_MAX_BATCHES_PER_SWEEP, which it did not. -
ar-io-node index-l2-import --target clickhouseand its worker (L2_BAND_IMPORT_ENABLED, off by default) load installedparquet-l2bands, such asardrive-l2, into a gateway's ClickHouse, so its GraphQL answers for their data items without unbundling them. Each band is verified first (bound to the publisher's signed publication, its roots found in the L1 index, a sample of items read on chain) and refused whole if a check fails. One placement is kept per item, items ClickHouse already holds are left alone and disagreements counted (l2_band_import_conflicts_total), and nothing above ClickHouse's top height is written. Jobs are queued underETL_STAGING_PATH/l2-importand loaded byclickhouse-auto-import, one at a time between its own batches (CLICKHOUSE_AUTO_IMPORT_L2_POLL_INTERVAL), withclickhouse-import's newCLICKHOUSE_IMPORT_KEEP_EXISTING, which inserts only keys ClickHouse lacks, so a row the gateway exported meanwhile is never replaced. A ledger,parquet_l2_importsinbundles.db(new migration), records each band;--forget <band>removes what an import wrote. One 3.2M-item band ofardrive-l2verifies and queues in about two minutes and loads in about 40 seconds.--target sqlite(L2_BAND_IMPORT_TARGET=sqlite) does the same for a gateway without ClickHouse, writing the items straight intobundles.dbbeside the running gateway, up to its stable L1 tip, in short transactions it can resume from; it is refused whereCLICKHOUSE_URLis set. All ofardrive-l2is about 370 GB in SQLite.
Changed
-
AR.IO peer selection weights now drift back to their category default when nothing reports on a peer, halfway every
PEER_WEIGHT_DECAY_HALF_LIFE_MS(default 15 min;0turns it off) (#1119). A peer marked down during an outage used to keep its low weight until it was picked and succeeded, which its low weight made unlikely. For that to reach the peers tried first, a key's home set (the hash-ring peers asked before any others) is now ordered by a weighted draw rather than strictly by weight: decay keeps weights in order, so a strict sort left a recovering peer behind its neighbours for good. Heavier peers still usually go first, and a peer near the minimum weight rarely does.PEER_WEIGHT_DECAY_HALF_LIFE_MS=0turns off both the decay and the drawn order, restoring the previous peer selection exactly. -
/chunk/<offset>and/chunk/<offset>/datanow answer 503 withRetry-After: 5andCache-Control: no-store, not 404, when a chunk could not be fetched in time: the serve deadline (CHUNK_SERVE_DEADLINE_MS), an upstream timeout, or an internal cancellation (#917). A 404 for an offset reads as permanent, so clients and peers cached it, peer weighting took it for "lacks the data", and the observer's reference-gateway client stopped at it instead of trying the network. Under load nearly every failure was this kind: 99% of requests at 500 concurrent, with a working gateway. A boundary lookup skipped because the chain fallback was at its concurrency limit is also a 503 (boundary_unavailable), not a 404. A 404 now means no source found the chunk; one case is still ambiguous: when every peer fetch timed out, the peer cascade does not report why, and that ends in a 404 (fetch_failed). Dashboards will show these 503s as 5xx. Serve-deadline 503s are counted, as before, bychunk_serve_deadline_exceeded_total; all of them carry their reason in thechunk.retrieval.errorspan attribute; they are logged at debug rather than warn so a busy gateway does not flood its logs. -
The default
GRAPHQL_ROOT_TX_GATEWAYS_URLSis now{ "https://arweave.net": 1 }, replacinghttps://arweave-search.goldsky.com, which now answers every request with a 307 tohttps://arweave.net/graphql: nodes on the default were already querying arweave.net, through an extra hop. Behaviour is otherwise unchanged. arweave.net rate-limits some addresses on its gateway endpoints (429,retry-after: 299);{}turns the source off, and a gateway with its own indexer can list that instead. The variable is now documented indocs/envs.md.
Fixed
-
A blocked ID requested outside its sandbox (on the apex,
/<id>or/<id>/<path>, or on an IP host such as a load balancer's) was still redirected with a 302 to its sandbox subdomain, which answers 451. Abuse checkers stop at the 302, so a host's abuse desk kept reporting the URL as live after the takedown. The sandbox middleware now checks the local block list first and answers 451 directly, with the short not-found TTL (these hosts are not revalidated on unblock). An ID that is not blocked is redirected as before; if the block list check fails, so is a blocked one. -
Data fetched from a trusted gateway could be a JSON wrapper of the item instead of the item. arweave.net (now HyperBEAM) answers a request whose
Acceptincludesapplication/jsonwith a JSON document of the data (raw-id,body,commitments,hashpath, ...), for full and range requests alike, and axios sent its defaultapplication/json, text/plain, */*because the gateway set noAccept. arweave.net is in the defaultTRUSTED_GATEWAYS_URLS, so a gateway whose first upstream missed cached the wrapper and served it as the item, with the wrong content type and digest (observers reportdataHashDigestandcontentTypemismatches). Byte fetches from trusted gateways, AR.IO peers and HTTP range sources now sendAccept: */*. Copies already cached keep serving the wrapper until purged: they start with{"accept-ranges". -
Two gateways that listed each other in
GATEWAYS_GQL_URLScalled each other's/graphqlwithout end, until every hop timed out (#680). Fanned out queries now carryX-AR-IO-Hops, and a request that has already passed throughGATEWAYS_GQL_MAX_HOPSgateways (default 2) is answered from the local index only. A gateway that excludes its local index (GATEWAYS_GQL_INCLUDE_LOCAL=false) answers such a request with nothing and aFANOUT_HOP_LIMITwarning, so a federation-only gateway that other gateways query through one more hop should raiseGATEWAYS_GQL_MAX_HOPS(for example to 3). -
The Iceberg metadata scripts declared
root_parent_offsetasint; it is a byte offset into a root bundle and passes 2^31 in bundles over 2 GiB, so it is nowlong, like the other offsets (#907). A nested bundle unbundled without a known offset in its root, whose items are recorded atroot_parent_offset0, is now logged and counted (bundles_unbundle_unknown_parent_offset_total) so its producers can be found. -
POST /ar-io/admin/queue-bundlequeued every ID as its own root, so for a bundle that is itself a data item each of its items was recorded with the bundle as root and offsets relative to the bundle (#959 item 5, whose writer was unknown). Serving then rejected those locations, since the "root" is not an L1 transaction, and items failed intermittently. The route now looks up a nested bundle's root and payload offset first, and answers 409 when the root is known but the offset is not. Rows written this way before can be cleared withPOST /ar-io/admin/invalidate-data. -
A data item stored in two bundles could be given a location made of one copy's root and the other copy's offsets: data attributes took the root from the location recorded at fetch time and filled missing offsets from the bundles index, even when the two named different roots (#959). The index now fills in only when it names the same root. Separately, an ID found only through its data root (an L1 transaction whose bytes are also a data item's) borrowed that other item's root and offsets; it no longer gets any location that way, and the ID's own row now always wins over one found through the data root.
-
AR.IO peer weighting no longer treats a busy peer like one without the data: a 429 or 503 from a peer costs it a fifth of a failure's weight (#1118). Chunk peers were also penalised twice for every refusal (once where it was detected and again in the request's catch); each now counts once.
-
Chunk-metadata anchoring asked only the first peer of its pool, so when that peer failed every lookup fell through to the slower chain search. It now asks the first peer not backed off: a peer that is unreachable, refuses (429, 402, 401, 5xx; a 404 is a healthy peer without the chunk) or contradicts the chain is skipped for 30 s, doubling to 10 min, and the next peer takes its probes. The pool now keeps every entry in priority order (previously
GATEWAYS_ROOT_TX_URLSin object order, or only the firstTRUSTED_GATEWAYS_URLSentry), and has its own setting,CHUNK_METADATA_ANCHOR_PEER_URLS, so anchoring can be pointed elsewhere or turned off ({}) without changing root-TX lookups. -
A gateway that started while the AR.IO gateway registry was unreachable had no peers until the next hourly refresh, so every peer retrieval failed for up to an hour. The peer list is now saved after each successful registry read (
AR_IO_PEER_LIST_CACHE_PATH, defaultdata/peers/ar-io-peers.json; Docker Compose mounts it fromPEERS_DATA_PATH, default./data/peers) and loaded at startup, and while the list is empty a failed read is retried after 10 s, backing off to 5 min. A registry read that returns no gateways at all is treated as a failure rather than applied as an empty list. -
X-AR-IO-Stablewasfalseon every response, and stable data never gotCACHE_STABLE_MAX_AGE(30 days,immutable): since 2023 the gateway compared SQLite's1withtrue, which never matched. A transaction at height 1,500,000 on vilenarios.com, long past the fork depth, was servedX-AR-IO-Stable: falsewith a two-hourmax-age. Stable data is now reported stable, and directly addressed stable data (/<id>,/raw/<id>) that is also verified and served from the gateway's own cache getsCACHE_STABLE_MAX_AGE,immutable, for the first time; other stable data keeps the unstable tiers, and ArNS responses keep their own TTL. A data item the bundles flush has not yet promoted is also stable once its root L1 transaction is, rather than waiting for the flush (an item whose bundle has nostable_block_transactionsrow was never promoted). This is the same test the flush applies before promoting it, so no placement is trusted more than before. With no local record of the data,X-AR-IO-StableandX-AR-IO-Verifiedare now sent asfalseinstead of being left out. -
GraphQL no longer lists a data item's tags twice when the item was indexed from two bundles (or imported from an L2 band and later unbundled). The stable flush keeps the first stored row of an item, but wrote the tags of every placement beside it; it now writes tags only where it stores the item. Tag rows already duplicated are not removed.
-
GraphQL no longer serves, or caches, a data item's signature or owner key read from a wrong offset. A signature is used only when its SHA-256 is the id, and an owner key only when its SHA-256 is the owner address; anything else counts as
attribute_fetch_total{outcome="invalid"}and is logged with the offset that produced it. Cached entries that fail either check are removed. Rows with item-relative owner and signature offsets had been served wrong signatures, and, through the owner cache keyed by address, wrong owner keys for every item of the same owner. -
GraphQL results merged from upstream gateways (
GATEWAYS_GQL_URLS) lost fields the query had asked for, because the selection forwarded upstream left out what the gateway reads back. Amounts read 0 AR when a query selectedquantity { ar }orfee { ar }withoutwinston;parentwas null withoutbundledIn; and a field selected twice under different aliases (a: tags { name } b: tags { value }) kept only the first selection's sub-fields, which failed the whole query for tags. The forwarded selection now always asks forwinston, merges every selection of a field, and the parent is read fromparentorbundledIn. Stored data was never wrong, so nothing needs reindexing.
Docker images
ENVOY_IMAGE_TAG:9a28d6538aca7fb88d96c862062b13a0ed0fce69CORE_IMAGE_TAG:e93047a244eb7f1cdb305df0b7c12f151e245e87CLICKHOUSE_AUTO_IMPORT_IMAGE_TAG:40d7e43ff21e9de42039f8db0fd89fb22e36ff22LITESTREAM_IMAGE_TAG:be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8OBSERVER_IMAGE_TAG:2aa908dc40e0c6fcf3c91bda92565af10876288e