ar-io/ar-io-node r74

Release 74

on GitHub

This is a recommended release focused on cache performance, multi-domain ArNS support, and content moderation correctness. Key highlights include background caching for range request cache misses to improve video/media streaming performance, multiple ArNS root hosts for serving ArNS names across multiple domains from a single gateway, contiguous data cache hit/miss Prometheus metrics for improved observability, and configurable cache control for blocked responses. It also corrects HTTP 451 handling for blocked content, simplifies the parquet export pipeline, and adds ClickHouse and block verification to auto-verify.

Added

• Background Caching for Range Request Cache Misses: When a range request (e.g., byte-range for video seeking) misses the local cache, the gateway now optionally fetches and caches the full item in the background so subsequent requests (range or full) are served locally. Controlled by BACKGROUND_CACHE_RANGE_MAX_SIZE (default: 0 / disabled) and BACKGROUND_CACHE_RANGE_CONCURRENCY. Includes deduplication, capacity-based drop semantics, and Prometheus metrics for monitoring cache activity

• Multiple ArNS Root Hosts: Operators can now serve ArNS names across multiple domains from a single gateway instance by providing a comma-separated list in ARNS_ROOT_HOST (e.g., ARNS_ROOT_HOST=arweave.dev,g8way.io). The first host is used as the "primary" for gateway identity headers. ArNS resolution, apex content, and sandbox redirects work per-matched host with longest-suffix matching (#621)

• ClickHouse Verification in Auto-Verify: Auto-verify is a data validation tool that checks consistency of indexed blockchain data (blocks, transactions, data items) across multiple backends (SQLite, Parquet, ClickHouse). This adds an optional ClickHouse source for verification in addition to the existing SQLite and Parquet sources.

• Block Verification in Auto-Verify: Verify block data alongside transactions and data items

• Bundle Data Prefetch in Auto-Verify: During auto-verify runs, raw bundle bytes are now fetched from the local gateway while it is still running, then parsed after shutdown. Previously the bundle-parser source had to fetch from arweave.net after the gateway was stopped, which was significantly slower.

• Contiguous Data Cache Hit/Miss Metrics: New contiguous_data_cache_hits_total and contiguous_data_cache_misses_total Prometheus counters in ReadThroughDataCache, labeled by request_type (range vs full). Enables operators to monitor cache performance per request type.

• Accurate Cache Miss vs Not-Found Metrics: Cache miss counter now fires only after a successful upstream fetch (data exists but wasn't cached locally). A new contiguous_data_not_found_total counter tracks requests where data is unavailable in any source, preventing not-found requests from inflating the miss count and skewing the cache hit rate.

• Configurable Cache Control for Blocked (451) Responses: New CACHE_BLOCKED_MAX_AGE env var (default: 30 days, matching stable data TTL) controls the Cache-Control max-age sent with 451 responses. Previously, blocked responses used the short not-found TTL, causing CDNs and proxies to re-request blocked content too frequently.

• Parent Bundle ID in Missing Data Item Errors: Auto-verify compareItems now includes the parent bundle ID in missing_in_source discrepancy messages for data items, making it easier to identify which bundle a missing item belongs to when debugging bundle-parser failures.

Changed

• Parquet Export Pipeline Simplification: Eliminated DuckDB intermediate tables from the export pipeline. All core export logic moved from the bash script into src/workers/parquet-exporter.ts, with the CLI script becoming a thin wrapper around the admin API. The CLI now uses --api-host/--api-port instead of --core-db/--bundles-db.

• Removed Legacy Auto-Verify CLI Options: Cleaned up deprecated verification flags

Fixed

• ClickHouse ETL Height Range: Fixed off-by-one errors in height range calculations in clickhouse-auto-import

• ClickHouse ETL Exit Code Capture: Fixed $? capturing the exit code of a variable assignment instead of the curl command

• HTTP 451 for Blocked Content: Corrects r73's blocked-content status code from 452 (non-standard) to 451 ("Unavailable For Legal Reasons"), the IANA-registered standard for content blocked due to legal or policy reasons.

• Trusted Gateway ArNS 451 Handling: The TrustedGatewayArNSResolver now accepts HTTP 451 responses from trusted gateways instead of treating them as errors. When a trusted gateway indicates a name is blocked, the local gateway respects that moderation signal and returns 451 to the client rather than falling through to the on-demand resolver.

• Serving Cached Data with Undefined or Zero Content-Length: The read path of ReadThroughDataCache now skips cache entries with a missing or zero dataSize, preventing responses without a Content-Length header. The write path already rejected zero-size entries; this closes the corresponding read-side gap.

• Parquet Export and ClickHouse Import Robustness: Parquet-export script now uses curl -o with temp files instead of head/tail parsing to handle multiline JSON API responses correctly. Auto-verify's importToClickHouse switches to execFileSync with an args array, preventing shell injection in ClickHouse import invocations.

• Parquet Export Verify-Count Non-Zero Exit: parquet-export --verify-count now exits non-zero when row counts don't match, making it useful in CI and automation pipelines. Also validates curl availability at startup alongside python3.

• High-Severity Dependency Vulnerabilities: Resolved known vulnerabilities in transitive production dependencies via yarn resolutions: path-to-regexp (ReDoS, via express/express-openapi-validator), h3 (request smuggling), picomatch (glob injection), preact (VNode injection), socket.io-parser (unbounded binary attachments), undici (multiple HTTP smuggling/memory issues), and bumped fast-xml-parser from 5.3.6 to 5.5.9.

• JSON Data Files Missing from Production Build: offset-block-mapping.json was excluded from dist/ because the build copy step only matched .graphql, .sql, and .lua files. This caused a startup warning and fallback to slower full-range block searches in containers. .json files are now included in the copy step.

• Auto-Verify Prefetch Timing and Empty ClickHouse URL: Removed the last_fully_indexed_at filter from the bundle prefetch query — this flag is set asynchronously by BundleRepairWorker, causing prefetch to find 0 bundles even when indexing was complete. Also handles AUTO_VERIFY_CLICKHOUSE_URL="" (empty string) explicitly to prevent a crash when the variable is set but empty in .env.

Docker Images