1
[Feature] CLI: Single-pass upload (hash + dedup + upload) instead of reading files twice
Source: immich-app/immich#27977 · opened by @abo-chen
I have searched the existing feature requests, both open and closed, to make sure this is not a duplicate request. • Yes The feature Currently, the immich upload CLI reads all files twice during a single upload session: 1. Hashing files — reads every file to compute perceptual hashes 2. Uploading assets — reads non-duplicate files again to upload For large libraries (e.g., 1TB+ with 100k+ files), this effectively doubles the I/O time. On my setup, hashing 1TB takes ~5 hours, and the upload adds another significant chunk of time — much of which is re-reading the same files from disk. Proposed solution: Merge the three phases into a single streaming pipeline: For each batch of N files: 1. Hash files 2. Send batch hashes to server for dedup check 3. Upload only non-duplicate files 4. Move to next batch This way each file is read at most once (hash + upload in the same pass), or twice only for non-duplicat…
No pledges yet. Be the first to back this.
Comments
Similar requests
multi cli upload hash first
1 vote · 0 comments
[Feature] API to blacklist a hash or attach multiple hashes to an asset
1 vote · 0 comments
Find duplicates and deduplicate globally across all users
8 votes · 0 comments
Deduplication in shared albums
26 votes · 0 comments
Android: does not show file size unless uploaded
1 vote · 0 comments
No comments yet.