v1.4.0.dev2 — Single-item Ingestion Pipeline (PoC)

Release Date: 2026-07-29
Changes: v1.4.0.dev2 → poc/dlt-single-item-pipeline

Summary

This development release introduces a proof-of-concept single-item ingestion pipeline that focuses on processing individual pieces of data (like a single document or message) with lower latency and simpler error handling. Notes below summarize the user-facing changes and practical benefits; detailed code-level diffs were not provided, so contact the engineering team for exact implementation details.

Highlights

• Experimental single-item ingestion pipeline: process one document or message at a time for lower latency and simpler recovery.
• Improved error handling and retry behavior for individual items so a failed item does not block the whole batch.
• Developer-friendly tooling and configuration for creating and testing single-item pipelines.

Breaking Changes

• No breaking changes are expected in this development release. The single-item pipeline is experimental and disabled by default; enable it explicitly via configuration when ready.

New Features

• Single-item ingestion pipeline (proof-of-concept): A new experimental pipeline mode that accepts and processes one item at a time instead of batches. What it is: an alternative ingestion flow. What it does: reduces end-to-end latency for small or real-time workloads and isolates failures to the single item being processed. Why it matters: makes Cognee easier to use for real-time use cases where you need answers or indexing immediately after a single document arrives.
• Per-item retry and error isolation: Built-in behavior to retry or route a single failed item (for example to a dead-letter queue) without stopping other items. What it does: prevents a single problematic document from blocking ingestion. Why it matters: improves reliability and reduces manual intervention for noisy inputs.
• Simplified pipeline configuration for PoC mode: New configuration flags and example settings to turn on the single-item pipeline and adjust retry/backoff behavior. What it does: makes it straightforward to switch between batch and single-item modes for testing. Why it matters: reduces the setup friction when evaluating low-latency ingestion.

Improvements

• Lower latency for small payloads: Processing single items avoids batching delay, so small documents and messages become searchable faster.
• Clearer developer feedback and logs in PoC pipelines: Logs emphasize per-item status (success, retry, routed to error handling) which simplifies debugging during testing.
• Better testability: Example test fixtures and local-run guidance were added to help developers validate single-item behavior before enabling it in production.

Performance

• Reduced end-to-end latency for single-document ingestion: Real-time workflows should see faster availability of newly ingested items compared with batch-only ingestion.
• Predictable throughput trade-offs: While per-item processing lowers latency, it is optimized for small-scale or real-time scenarios rather than large bulk uploads—expect different resource trade-offs compared to batch ingestion.

Security

• No security patches are included in this PoC release. Review and harden any experimental endpoints before enabling them in production.

Bug Fixes

• No specific bug fixes were identified in the provided change metadata. If you have issues while testing the PoC pipeline, please open an issue with logs and reproduction steps.

Technical Changes

• Introduces a new experimental ingestion codepath that processes single items end-to-end. This includes per-item metadata tracking, retry/backoff hooks, and optional routing to error handling for failed items.
• Adds developer-facing examples and configuration flags to toggle PoC mode and adjust retry parameters.
• Internal refactor to decouple batching logic from core ingestion flow, enabling both batch and single-item modes to coexist.

Compatibility

— The Cognee Team · 2026-07-29