[Feature Request] overhaul worker management

Description You configure paperless to do nothing. no crons, no consume, nothing. You add 2.000 documents. You manually tell paperless to do the llm index. (docker exec paperless-ngx python manage.py document_llmindex update) You now have a single task with 2.000 sub-tasks. Paperless does its job. The embedding ai is getting pounded. Since the whole system is still near idle, you manually tell paperless to do the ai suggestions. (docker exec paperless-ngx python manage.py shell -c 'from documents.tasks import check_scheduled_workflows; check_scheduled_workflows()') Nothing. System still near idle. little embedding ai still getting pounded. big ai still sleeping. Paperless waits for the 2.000 sub-tasks to complete so it can mark the first main-task as complete and only then it will start with the second main-task. Parallelization PLEASE! And when you go the normal way, paperless loads the embedding model to do the llm index, then it loads the big model to do …