一台 DGX Spark 推理主機,應對多種用途
One inference box (DGX Spark) and multiple use cases...
RAG-LCC 的開發者建立了模型管理工具,讓一台 DGX Spark 可配合 RAG 工作流程及本地編碼輔助等用途切換模型設定。工具支援兩套設定、啟停模型、啟動前記憶體檢查,並以 vLLM Docker runtime 和 LiteLLM 作為中央入口。相關程式已整理成 GitHub 專案 harinezumigel-llm-stack。
While developing my RAG-LCC experimental RAG, I had multiple use cases:
RAG-LCC uses LLMs for:
- Prompt guard
- Inference (query answering)
- Reply judgement
- Query rewrite
- Query split
Although some of the tasks can be handled by the same LMM it is good practice to have other models then the inference one for prompt guard and judgement.
I use 2 sets of configurations, which I want to switch as convenient as possible.
Coding assistance:
I tried different models, which would be a separate discussion. But when I use local coding assistance, I cannot run the RAG tests because my Spark DGX can't provide it.
What it does
- Two configuration files
- Starts and stops models. Pre-flight memory check before startup
- Uses the NVIDIA provided vLLM docker runtime
- Integrates LiteLLM as central entry point (LLM proxy)
- Supports model groups, e.g. rag and rag_old
- Define aliases. So you must not remember the model name, only what it does, e.g. "judge"
- Status shows running and stopped models
- Log interaction allows you to attach and detach from logs
- Container creation and recreation. You can protect a models container from beeing recreated
Conclusion
Building a RAG system taught me that managing the LLMs I use is an important part of my development flow. What started as some lines of python code has become a GitHub repo on its own.
If you want to have a look: harinezumigel-llm-stack
Feedback and ideas are welcome.
來源:DEV Community · dev.to


