FeatureFuel
3

[Feature Request] AI chat: size the synthesis prompts by context window, not embedding chunk size

Source: paperless-ngx/paperless-ngx#13924 · opened by @00BraiNdeaD00
I've been evaluating Paperless' RAG on a local Ollama setup and ran into something I can't tell is intended or not. (Diagnosis done with LLM; tried on 3.1.0 ) _stream_chat_with_documents passes the embedding chunk size to the response synthesizer's prompt helper: get_response_synthesizer() defaults to ResponseMode.COMPACT, and llama_index's PromptHelper._get_available_chunk_size() ends with result = min(result, self.chunk_size_limit). So each synthesis pack is capped at the embedding chunk size rather than at what the context window allows. One chat question turns into roughly one LLM call per retrieved chunk via the refine chain, while the retrieved context would comfortably fit in a single call Counting POST /api/chat at the Ollama side, 12 questions over a 37-document archive: | PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE | LLM calls per question | |---|---| | 1024 (default) | 3.5 | | 512 | 5.0 | There's a reading where this is …

No pledges yet. Be the first to back this.

Make a pledge

Pledge your monetary support if this feature is added.

$

Comments

No comments yet.

Replying to

Add a comment

What do you think about this feature request?


Similar requests