1
[Feature Request] AI chat/RAG: cache the embedding model instead of reloading it on every query
Source: paperless-ngx/paperless-ngx#13567 · opened by @lunetics
The problem load_or_build_index (src/paperless_ai/indexing.py) calls get_embedding_model(config) on every invocation, and that runs once per chat/RAG query (_stream_chat_with_documents in src/paperless_ai/chat.py, inside the read_store() block). There is no caching of the embedding instance, so each query reloads the full embedding model before it can retrieve anything. What we measured Per-query model reload cost on a production archive (about 1,050 documents, HuggingFace/CPU backend), by embedding model: | Embedding model | RAM loaded per query | |---|---| | intfloat/multilingual-e5-small | ~0.5 GB | | intfloat/multilingual-e5-base | ~1.1 GB | | BAAI/bge-m3 | ~2.3 GB | The reload dominates end-to-end query latency (it outweighs the actual vector search) and produces a RAM spike on every request. On memory-constrained instances this effectively caps the usable embedding model size for reasons that have nothing to do with retrieval quality. Suggested direction …
No pledges yet. Be the first to back this.
Comments
Similar requests
[Feature Request] Allow overriding the embeddings encoding_format for OpenAI-compatible LLM providers
4 votes · 0 comments
[Feature Request] Add detailed debug logging for RAG/LLM requests
2 votes · 0 comments
[Feature Request] AI Suggestions Cache
6 votes · 0 comments
[Feature Request] Open the AI document chat as a full page/tab instead of a small overlay window
6 votes · 0 comments
[Feature Request] AI chat: size the synthesis prompts by context window, not embedding chunk size
3 votes · 0 comments
No comments yet.