3
[Feature Request] AI chat: size the synthesis prompts by context window, not embedding chunk size
Source: paperless-ngx/paperless-ngx#13924 · opened by @00BraiNdeaD00
I've been evaluating Paperless' RAG on a local Ollama setup and ran into something I can't tell is intended or not. (Diagnosis done with LLM; tried on 3.1.0 ) _stream_chat_with_documents passes the embedding chunk size to the response synthesizer's prompt helper: get_response_synthesizer() defaults to ResponseMode.COMPACT, and llama_index's PromptHelper._get_available_chunk_size() ends with result = min(result, self.chunk_size_limit). So each synthesis pack is capped at the embedding chunk size rather than at what the context window allows. One chat question turns into roughly one LLM call per retrieved chunk via the refine chain, while the retrieved context would comfortably fit in a single call Counting POST /api/chat at the Ollama side, 12 questions over a 37-document archive: | PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE | LLM calls per question | |---|---| | 1024 (default) | 3.5 | | 512 | 5.0 | There's a reading where this is …
No pledges yet. Be the first to back this.
Comments
Similar requests
[Feature Request] AI chat/RAG: cache the embedding model instead of reloading it on every query
1 vote · 0 comments
[Feature Request] Allow overriding the embeddings encoding_format for OpenAI-compatible LLM providers
4 votes · 0 comments
[Feature Request] Open the AI document chat as a full page/tab instead of a small overlay window
6 votes · 0 comments
[Feature Request] Optimize AI suggestion generation
5 votes · 0 comments
[Feature Request] Different keys for LLM vs. LLM embeddings
6 votes · 0 comments
No comments yet.