LogicLabs-OU/OpenArchiver v0.5.2

v0.5.2: Advanced Search, Upload Progress, PST/Mbox Import Fixes, and Bulk Deletion

on GitHub

This release adds a new advanced search experience and a substantial round of data-integrity work across ingestion and storage. Several of the ingestion fixes close paths where an import could skip or duplicate messages while reporting success, so upgrading is recommended for anyone running large or recurring imports.

Advanced search

Search now supports structured filtering alongside keyword matching. The search page has a new filter panel that lets an operator narrow results by:

• Ingestion source — include or exclude specific sources. Selecting a merged ingestion covers its whole merge group. This control appears only for roles that can list sources.
• Sender and recipient — include or exclude sender addresses, and include or exclude recipient addresses (matched across To, Cc, and Bcc).
• Mailbox — restrict results to specific archived mailbox accounts, with type-ahead suggestions as an address is typed.
• Date range — restrict to a sent-date window. Both bounds are inclusive and interpreted in UTC.
• Attachments — restrict to emails with or without attachments.

Keyword matching can also be scoped to specific parts of an email — subject, body, attachment name, attachment content, sender, or recipients — so a search for invoice scoped to attachment names finds messages with a file named like invoice.pdf rather than every message that merely mentions the word. Results can be sorted by newest, oldest, or relevance. When at least one filter is active, keywords become optional, so the filters double as a way to browse the archive.

The full search state — keywords, matching strategy, and every filter — lives in the page URL, so a search can be bookmarked, shared, or reloaded and reproduced exactly. Pagination preserves all active filters.

Each search result and the email metadata section now also show the mailbox an email was archived from, and the metadata section links the ingestion source to its email list. The source relation in the detail response is trimmed to id and name so stored credentials are never included.

Everything available in the UI is also available through GET /v1/search using the same parameters. See the search guide and the Search API reference for details.

Reindex required. The attachments filter relies on a hasAttachments field in the search index, and improved sender display relies on a new fromName field. Emails archived before this release need a one-time reindex before the attachments filter counts them correctly or search results show a resolved sender name. See Search Indexing & Reindexing.

Upload progress and feedback

File uploads for PST, EML, and Mbox ingestion sources were previously fire-and-forget — a spinner with no progress, no way to cancel, and a failure reported only as a transient toast while the file input reset, so a slow or failed upload looked like a hang. All three file-based providers now share one upload field driven by XMLHttpRequest that shows a progress bar with percentage, a cancel button, a persistent success panel with the file name and storage path, and a persistent error panel with the failure reason.

Ingestion and imports

Several changes close paths where an import could report success while losing or duplicating messages:

• Mbox imports no longer drop messages (#412). The mbox splitter's readable side was a byte stream, so Node re-concatenated the per-message buffers and the consumer received runs of messages glued into one, which were then archived as a single email. The splitter now runs in object mode so each message stays a discrete buffer, and it logs the message count so a source-versus-archive mismatch is visible.
• Corrupted MIME and re-sync duplication in PST imports are fixed (#376, #417). PST message construction emitted invalid MIME — two top-level Content-Type headers, a headerless first body part, and raw binary injected into the header block — so parsers rendered HTML bodies as raw source, dropped recipients, and could not resolve inline cid: images. Messages are now rebuilt as valid RFC 5322 / 2046 with proper multipart nesting, CRLF line endings, base64-encoded bodies, RFC 2047 encoded headers, and Content-ID on inline attachments. Senders carrying an Exchange X.500 distinguished name are resolved to their SMTP address. The constructed PST mailbox identity is now stable across runs, which stops a re-sync from re-archiving the entire PST as duplicate rows; two paths that could run a second import concurrently with a live one were also closed by raising the ingestion worker's BullMQ lock duration and heartbeating sync sessions on wall-clock time.
• Long email IDs, folder names, and attachment file names no longer drop emails (#405). Attachment file names from MIME headers were used raw in storage paths, so names over the 255-byte filesystem limit failed the write and dropped the whole email, and names containing slashes or .. could create unintended directories. Attachment file names are now sanitized and byte-clamped with a short hash suffix, email-id limits are measured in bytes, and folder path segments are clamped. Original file names and paths are still stored unmodified in the database.
• Large imports no longer fill the disk (#400). Local PST files are read in place instead of copied into a temp directory, and leaked temp directories are swept by owner PID.
• Unreadable files now fail the job instead of crashing the worker; source-stream errors in the file connectors are propagated so the job rejects cleanly.
• Ingestion sources are sorted alphabetically by name (#407), rather than by creation order, which makes the source list and every source dropdown easier to navigate at scale.

Two deduplication lookups were also given supporting indexes to remove full-table scans on large re-syncs: msgid_header_source_idx on (message_id_header, ingestion_source_id) for the default Message-ID dedup path (#394), and storage_hash_source_idx on (storage_hash_sha256, ingestion_source_id) for the preserve-original (GoBD) byte-hash path. Before these, every ingested email sequential-scanned archived_emails, pinning CPU during re-syncs of large archives.

Storage

• Non-ASCII attachment downloads are fixed (#406). Downloading an attachment or .eml whose file name contained non-ASCII characters previously threw ERR_INVALID_CHAR and returned HTTP 500, because the raw name was interpolated into the Content-Disposition header. Downloads now use Express res.attachment(), which encodes the file name per RFC 6266 / 5987. The download error path also moved from console.error to the project logger.
• NFC/NFD file name mismatches are tolerated (#409). On storage layers that rewrite file-name bytes on write while staying byte-sensitive on read — notably Docker Desktop bind mounts on macOS and some network shares — a file written under an NFC name could land on disk as NFD, so a download by the stored path returned 404. Local storage lookups now try the exact path first, then the NFC and NFD forms. Standard Linux filesystems were never affected.

Display and readability

• Resolved sender names (#413). Exchange mail often carries an X.500 legacy distinguished name in the From address while the display name parses correctly, and earlier versions showed the raw distinguished name in the dashboard and search. The list and detail views now prefer the parsed display name, the detail page shows Name <email>, and the thread sidebar fetches the name. The search index stores the name as a new searchable fromName field while keeping from as the address, so search and filtering by email still work; search results display fromName. The Top Senders widget resolves display names from the database, so it reads correctly without a reindex.
• Bulk email deletion (#366). The archived emails list page now supports selecting and deleting emails in bulk.

Operations

• Quieter routine logs (#401). The continuous-sync scheduler heartbeat and no-work dispatch moved to debug, and per-email and per-attachment deduplication logs were demoted to debug, so idle systems no longer flood centralized log dashboards. Lifecycle and startup logs stay at info; LOG_LEVEL=debug restores the verbose output.

Upgrade notes

• Reindex for full search coverage. After upgrading, run a reindex so existing emails populate the new hasAttachments and fromName search fields. Until then, older emails are treated as attachment-less by the attachments filter and may show an address rather than a display name in results. The Top Senders widget does not require a reindex.
• New environment variable. PDF_PARSE_TIMEOUT_MS controls the built-in (non-Tika) PDF parse timeout in the indexing worker. Default is 20000 ms.

Contributors

• @weishest / wayneshn — advanced search, upload progress, ingestion and storage fixes, sender-name resolution, and logging changes.
• @opniskanen — corrupted MIME and re-sync duplication fixes for PST imports (#417).
• @waffen29 — bulk email deletion on the archived emails list page (#366).

Thanks to everyone who reported the issues that shaped this release.

Full Changelog: v0.5.1...v0.5.2