Improve machine learning resilience in low VRAM scenarios (CUDA)

The bug Hi there, I'm currently evaluating Immich and really liking it so far. My Immich instance is running via Docker on a Debian 12 VM that is hosted on ESXi. The VM has a vGPU profile with 4 GiB VRAM assigned. During a stress-test of the hardware accelleration (both, transcoding and machine learning), I noticed that the machine learning Python module does not seem to be very resilient against low VRAM scenarios. After uploading some initial photos and videos, I run a "stress-test" by starting re-running all relevant processing tasks simultaneously (face detection, smart search, transcoding). At the same time, I ran a smart search query from the main dashboard. The following observations were made: • Especially the smart search query allocates a lot of VRAM • When the smart search query request fails due to low memory: 1. A corresponding exception (failed to allocate memory) is logged in the container 2. The smart search container…