FeatureFuel
1

[Feature] CLI: Single-pass upload (hash + dedup + upload) instead of reading files twice

Source: immich-app/immich#27977 · opened by @abo-chen
I have searched the existing feature requests, both open and closed, to make sure this is not a duplicate request. • Yes The feature Currently, the immich upload CLI reads all files twice during a single upload session: 1. Hashing files — reads every file to compute perceptual hashes 2. Uploading assets — reads non-duplicate files again to upload For large libraries (e.g., 1TB+ with 100k+ files), this effectively doubles the I/O time. On my setup, hashing 1TB takes ~5 hours, and the upload adds another significant chunk of time — much of which is re-reading the same files from disk. Proposed solution: Merge the three phases into a single streaming pipeline: For each batch of N files: 1. Hash files 2. Send batch hashes to server for dedup check 3. Upload only non-duplicate files 4. Move to next batch This way each file is read at most once (hash + upload in the same pass), or twice only for non-duplicat…

No pledges yet. Be the first to back this.

Make a pledge

Pledge your monetary support if this feature is added.

$

Comments

No comments yet.

Replying to

Add a comment

What do you think about this feature request?


Similar requests