Xentraxx/GooglePhotosTakeoutHelper_Neo v6.2.1

on GitHub

⚡ Performance

• JSON sidecar path and content caching across Steps 2 and 4 — Every media file's JSON sidecar was previously located and parsed twice: once in Step 2 (partner-sharing detection) and again in Step 4 (date + GPS extraction). Each lookup involved 5–10+ File.exists() calls trying different naming patterns (supplemental-metadata, truncated variants, numbered forms, etc.), and each parse read and decoded the full JSON file. Step 2 now caches the resolved sidecar path and the isOwnSidecar confidence flag directly on the FileEntity, and Step 4 reuses both — skipping the expensive findJsonForFileWithConfidence lookup entirely. A new LRU cache (50,000 entries) in JsonMetadataMatcherService stores parsed JSON content keyed by sidecar path, so each sidecar file is read and decoded exactly once across both steps. For a 50,000-file takeout this eliminates ~100,000 redundant filesystem stat calls and ~50,000 duplicate JSON reads. The isOwnSidecar flag is preserved through MediaEntity normalization (which creates new FileEntity instances during ranking), ensuring the issue #139 cross-photo GPS guard remains intact.
• Parallelized Step 8 creation-time updates — File creation timestamps were updated one file at a time in a sequential loop, each call opening a Win32 handle (CreateFile + SetFileTime + CloseHandle) or POSIX utimensat call. These operations are independent and are now batched with Future.wait using the fileIO concurrency limit. For a 50,000-file takeout this turns the final step from fully single-threaded to parallel, significantly reducing wall-clock time on Windows.
• Extension-first filter in wherePhotoVideo() — The discovery filter called lookupMimeType() for every file during recursive directory listing, including non-media files (JSON sidecars, etc.). The extension check (MediaExtensions.additional set lookup + isMotionPhotoExtension) is now performed first; only files that don't match the special-extension fast path fall through to the more expensive lookupMimeType call. For a 100,000-file takeout this avoids ~100,000 unnecessary MIME map lookups during discovery. The fast path only matches extensions that lookupMimeType does not recognize (.mp, .mv, .dng, .cr2, .cover, .mp~N), so standard extensions like .jpg/.png/.mp4 still go through lookupMimeType exactly as before — Step 1's content-based extension fixing (which reads file headers, not the filter's MIME result) is unaffected.
• Eliminated redundant stat() + length() in MediaHashService.calculateFileHash — The hash calculation called file.stat() to generate the cache key, then file.length() separately to get the file size — two syscalls per file. The size is now reused from the already-fetched FileStat, eliminating ~50,000 redundant syscalls during Step 3 for a 50,000-file takeout.
• Partial JPEG read for EXIF date-tag check in Step 7 — hasExistingExifDateTag loaded the entire JPEG into memory (file.readAsBytes()) just to check whether EXIF date tags exist. EXIF data is always near the start of a JPEG (the APP1 marker follows SOI + JFIF), so only the first 64KB is now read (file.openRead(0, 65536)). For a 10MB JPEG this reduces memory allocation from 10MB to 64KB per file.

🛠️ Maintenance

• Test suite for JSON sidecar caching — Unit tests (test/unit/json_sidecar_cache_test.dart) cover the FileEntity sidecar path and confidence flag fields (defaults, set/get, JSON round-trip), extractAllFromJsonCached with cached own-sidecar (returns date+GPS), cached isOwnSidecar=false (drops date+GPS per issue #139), fallback to full lookup when no cached path exists (secondary files), graceful handling of deleted sidecar files, readJsonContentCached content caching and invalid-input handling, and Step 2 → Step 4 integration verifying the cached path is populated during discovery and reused during date extraction.

Full Changelog: v6.2.0...v6.2.1