v1.1.0-beta.001 - feat(metadata): Multi-Provider Ecosystem (Metron & ComicVine Parity)
This massive overhaul fundamentally restructures the core architecture of Omnibus to support multiple metadata providers as equal, first-class citizens. Admins can now explicitly choose between ComicVine and Metron.Cloud as the primary engine driving their library, searches, and discovery pages.
🚀 New Features & Capabilities
- Primary Metadata Selection: Added a new Primary Metadata Source dropdown in the Admin Settings. Switching to Metron seamlessly redirects all automated background jobs, Discover page endpoints, and Interactive Searches to the Metron.Cloud API.
- Hybrid Series Monitor (queue.ts): Completely rebuilt the SERIES_MONITOR background job into a highly efficient hybrid engine. It now pulls a 90-day forward-looking global release calendar from Metron (bypassing ComicVine's lack of future dates) while simultaneously running a slow-drip fallback scan on older ComicVine-sourced manga/indies.
- Smart Matcher Integration: The Admin Smart Matcher now includes a Metadata Source dropdown, allowing admins to manually input a Metron ID or a ComicVine ID to instantly match and organize unrecognized local files.
- Dynamic Routing: All external URL buttons across the app (Series Page, Admin Requests, User Queues, Modals) now dynamically read the database source and render either View on ComicVine or View on Metron with correct external links.
🛠️ Architecture & Backend Updates
- Metron Rate Limit Compliance: Completely overhauled the MetronProvider to strictly adhere to Metron's official API best practices. Engineered a smart fetchWithBackoff utility that parses X-RateLimit-Burst-Remaining and Retry-After headers to gracefully navigate the 20-request/minute burst limits without throwing 500 errors.
- Robust Caching Engine: Implemented a new 12-hour database cache layer for all search queries (/api/search) to act as a burst-shield. Search spams and duplicate requests are now served instantly from memory, aggressively protecting indexer API quotas.
- Safe Pagination Mapping: Built a mathematical slicer into the API layer that safely translates the UI's 10-item visual pages into Metron's 50-item API pages. This enables endless scrolling through search results while strictly capping sub-fetches to prevent rate-limit hanging.
- Database Schema Update (schema.prisma): Abstracted the hardcoded cvId integer columns into universal metadataId (String) and metadataSource (String) columns across the Series, Issue, Request, and ReadingListItem models. Added a strict @@unique([metadataSource, metadataId]) constraint to prevent cross-provider ID collisions.
- ComicInfo.xml Compliance & Custom Tags: Updated metadata-extractor.ts and metadata-writer.ts to fully support Metron IDs. Omnibus now natively reads and writes the community-adopted and custom XML tags to perfectly sync with tools like ComicTagger. It also embeds the Metron Cloud URL directly into the tag as a fallback.
- API Standardization & Alert Unification: Updated the entire /api ecosystem to accept, pass, and return provider and metadataSource parameters. Refactored importer.ts to utilize a single, unified SystemNotifier.sendAlert(...) method, stripping out redundant code for Discord and Mailer integrations.
🖥️ Frontend & UI Improvements
- Component State-Drop Prevention: Plugged a critical UI state bug in comic-grid.tsx where clicking related issues in the More in this Series grid dropped the provider flag. Modals now seamlessly track and carry the metadataSource across all sub-selections.
- Graceful Image Fallbacks: Implemented beautiful, stylized fallback icons () across the Discover grid and Search modals. If a provider's rate limit is hit or cover art is missing, the UI renders cleanly without broken browser image tags, loading the deep art later upon user click.
- Strict Type Safety: Abstracted the UI badge logic (In Library, Monitored, Requested) to cast all identifiers as strings (Set). This completely eliminates JavaScript strict equality (===) bugs caused by comparing database integer hashes with API string IDs.
- Settings Page UI: Reorganized the Settings page to feature the Primary Metadata Source prominently. The UI automatically disables the Popular Issues toggle if Metron is selected (as Metron lacks a popular endpoint) and safely disables the provider dropdown if API credentials are missing.
- Details Modals: Component details modals now fetch their internal data dynamically. Clicking a Metron comic pulls creators, arcs, and characters from the Metron API, while clicking a ComicVine comic pulls from CV.
- Continue Reading Shelf: The DynamicCover component on the homepage successfully pulls the correct cached imagery regardless of origin.
🔍 Observability & Debugging
- Deep Importer Traces: Injected detailed [Debug] logs across the CSV, CBL, and AniList importer routes. Admins can now trace exactly what strings were extracted from uploaded lists and view the exact reasons why a fuzzy-match to local files succeeded or failed.
- Smart Matcher API Visibility: Added comprehensive logs to match-series/route.ts to expose metadata engine routing and raw payload retrieval results, making it trivial to troubleshoot API anomalies during manual matches.
- Metron Endpoint Tracking: The MetronProvider now logs the exact paginated endpoints hit and outputs rate-limit ceiling parameters.
- Cross-Provider Fallback Tracing: Added [Library ID Debug] logs to the central IDs router, allowing admins to inspect the exact normalized string combinations the system is generating to power the UI's cross-provider badge fallbacks.
🧪 Comprehensive Test Coverage
- Metron Engine Validation: Added a complete test suite for the MetronProvider (tests/lib/metadata/metron.test.ts) guaranteeing mathematical precision for pagination slicing and strict adherence to 429 Retry-After backoffs.
- OPDS Page Streaming (PSE): Created tests/api/opds-streaming.test.ts to ensure external reading apps (like Panels and Chunky) can successfully and securely extract image buffers from internal zip archives without authorization bypasses.
- Reading List Auto-Builder: Implemented tests/api/reading-list-autobuild.test.ts to verify the automated pagination loops and dynamic source routing when converting massive ComicVine and Metron story arcs into readable lists.
- Smart Matcher Isolation: Added tests for the match-series API to explicitly prove that when a user selects a specific provider (e.g., Metron), the backend completely skips the opposing provider (CV), ensuring data purity.
- Interactive Search Logic: Tested the global hoster_priority settings to guarantee the GetComics scraper is fully bypassed when third-party hosters are disabled.
- XML Extractor & Writer Updates: Expanded existing test suites to assert that and tags, as well as metron.cloud Web URLs, are successfully extracted from archives and correctly generated during repack operations.
🐛 Fixes & Improvements
- Cache Cleanup Memory Leak: Updated the CACHE_CLEANUP cron job to target the new meta_details_ keys alongside legacy cv_details_cache_ keys, preventing infinite database bloat over time.
- Duplicate Scanner Blindspot: Upgraded the Admin Diagnostics duplicate scanner to recognize all official metadata sources, preventing it from flagging valid Metron series as rogue local duplicates.
- Ghost Folder Cleanup: Request cancellation now strictly respects the exact metadataSource of the volume, ensuring empty Metron folders are properly purged from the disk when a download is aborted by an admin.
- Failed Download Alerts: Dynamic metadata source routing is now safely applied to failed download alerts, ensuring Discord/Email notifications successfully fetch and display the synopsis for failed Metron downloads.
- Generic Title Sanitization: Added Regex interceptors (/^Issue\s*#?\s*\d+$/i) to the API routes to detect when Metron returns unformatted generic names (e.g., Issue #3) and intelligently override them with proper formatting (Series Name #3).
- Deep ID Recovery: Fixed an issue where Metron list-endpoints occasionally omitted the series.id, causing Next.js to trap null IDs. The provider now executes rapid micro-searches to dynamically recover missing identifiers behind the scenes.
- Importer Resilience: importer.ts safely parses generic metadataIssueId formats, meaning local files dumped into the /watched folder will auto-match to Metron records perfectly.
- Reading List Auto-Linker: api/reading-lists/route.ts correctly honors the original source of an imported CBL/CSV list item so downloaded Metron files accurately link back to the reading list.
- Fallback API Handling: Ensured the Manual Fallback DDL search passes the metadataSource token down to the database so stalled queue items don't accidentally revert to ComicVine.
⚠️ Migration Safety
- Added src/scripts/migrate-metadata.js to securely iterate through all existing database records, copying legacy cvId integers to the new metadataId string column and flagging them as COMICVINE. This ensures upgrading admins will experience zero data loss or broken links.
📖 Admin How-To: Swapping Priority
- Navigate to Admin Settings -> Metadata Tab.
- Ensure you have populated credentials for the provider you wish to use (e.g., Metron Username & Password).
- Select your desired provider from the Primary Metadata Source dropdown.
- Click Save All Changes. New searches and the Discover page will immediately begin using the new provider. (Note: This does not retroactively change your existing library).
📖 Admin How-To: Updating Existing Series (CV -> Metron)
Your existing library will continue to use ComicVine for updates unless you explicitly convert them. To convert a series to Metron:
- Navigate to the specific series in your library and click Fix Match.
- In the Interactive Search modal, change the Source dropdown to Metron.Cloud.
- Search for the series name, select it from the results, and Omnibus will automatically overwrite the old ComicVine data, update the DB source flag, and download the new Metron metadata.