v1.5.0.dev4 — Dataset Indexing & Search Relevance

Release Date: 2026-08-20
Changes: v1.5.0.dev4 → dev

Summary

This release focuses on making search more accurate and datasets easier to explore. It introduces a new optional dataset overview index, improved ingestion and metadata handling, and UI and API improvements that make finding and managing your documents faster and clearer.

Highlights

• New optional Dataset Overview index that groups documents by topic and provides short summaries for better search context.
• More reliable and faster document ingestion with preserved metadata and deduplication.
• Improved search relevance and ranking, plus new filters and dataset preview in the UI.

Breaking Changes

• None: This is a backwards-compatible development release. Existing APIs and storage formats remain supported; the new Dataset Overview index is optional and must be enabled explicitly.

New Features

• Dataset Overview index: A new optional index that analyzes a dataset (the collection of documents you've added) and automatically groups content into topic clusters and a short overview for each cluster. What it does: adds broader context to searches so results are aware of the dataset's high-level organization. Why it matters: search returns more relevant and higher-level answers, and you can quickly understand large collections without reading every document.
• Dataset preview in UI: A new quick-preview panel that shows the dataset overview, top topics, and recent documents. What it does: lets you inspect dataset scope and sample results without running a full search. Why it matters: speeds dataset triage and helps decide whether to run a deeper search or tuning.
• Ingestion metadata preservation: When files or text are added, Cognee now keeps and exposes original metadata (filename, source, timestamps, and user-added tags) through the API and UI. What it does: ensures context is retained with each stored entry. Why it matters: improves traceability and lets you filter or prioritize results using original metadata.

Improvements

• Search ranking improvements: Adjusted ranking heuristics so answers that match dataset-level overview and closely related topic clusters appear higher. This makes short, direct answers more likely and reduces noisy results.
• Automatic duplicate detection during ingestion: The system now flags and optionally deduplicates near-identical content as you add documents. This prevents clutter and reduces unnecessary storage and processing.
• More robust chunking: Text splitters (the logic that breaks long documents into searchable pieces) are improved to keep sentences intact and preserve semantic boundaries. This leads to fewer fragmented answers and better embeddings for long documents.
• API improvements: New query parameters let you prefer dataset-level over document-level results and to filter by preserved metadata fields (for example, filename or source).

Performance

• Faster ingestion throughput: Optimizations in batching and embedding calls reduce end-to-end ingestion time for large uploads (noticeably faster for datasets with thousands of documents).
• Lower memory footprint during indexing: Memory consumption is reduced during the overview index build, allowing index creation on smaller machines.

Security

• Safer token handling for UI sessions: Session tokens are now rotated more frequently and stored with stricter scopes in the UI to limit blast radius if a token is exposed. This reduces risk without requiring user action.

Bug Fixes

• Fixed incorrect relevance for documents with identical timestamps: Documents with the same timestamp no longer suffer from inconsistent ranking.
• Fixed UI pagination bug when previewing very large datasets: the preview panel now pages correctly for datasets with thousands of items.
• Fixed a failure mode during embedding retries that could leave partial records in storage.

Technical Changes

• Refactored indexing pipeline: internal code reorganized to separate dataset overview creation from document ingestion, simplifying future feature work and maintenance.
• Improved test coverage for ingestion and ranking modules to catch regressions earlier.

Compatibility

— The Cognee Team · 2026-08-20