In-Process ONNX Query Routing & Roman Urdu Detection
"A compact MiniLM or logistic regression classifier exported to ONNX Runtime can reliably predict intent, route (RAG vs. deterministic), and detect Roman Urdu on CPU in under 5ms, eliminating LLM latency and API spend."
Trained a 4-class multi-output classifier in scikit-learn / Hugging Face, exported to ONNX format, and mounted in-process inside FastAPI at application startup.
Sub-4ms classification latency on single CPU core; 100% precision on high-confidence greetings, contact lookups, and clear Roman Urdu queries.
Boundary queries with ambiguous phrasing (e.g. asking about both design and pricing) exhibited lower confidence, requiring safe deterministic fallback routing.
Benchmark INT8 quantized transformer models to improve semantic nuance while staying strictly within the 5ms latency window.