Use ← and → arrow keys or Spacebar to navigate
AMD Developer Hackathon ACT II · Track 1

Enterprise Inference Overhead

"Every task does not need a premium model."

  • The Cost Problem: Production pipelines wastefully route basic sentiment, summarization, and math requests to cloud-hosted premium models.
  • Scale Issues: High network roundtrips, strict rate limits, and soaring monthly invoices for trivial tasks.
  • Unified Interface Requirement: Enterprises need a single gateway that guarantees 100% correctness while optimizing expenditures.
Baseline Single-Model Route
Sentiment Review Basic Sum Factual Q Clamp Function Logic Puzzle
↓
Expensive Premium API ($$$)
ZeroToken Router ACT II
Slide 1 of 5
AMD Developer Hackathon ACT II · Track 1

ZeroToken Router

"Prove locally. Escalate only when necessary."

  • Dynamic Classification: Instant, zero-token categorization maps input queries to specific domains on startup.
  • Zero-Token Local Execution: High-confidence heuristic math, summarization, NER, and sentiment logic are executed directly inside the container.
  • Fallback Guarantee: Remote API exceptions or timeouts automatically fall back to deterministic emergency outputs, avoiding container crashes.
In-Memory Gateway Classifier
Sentiment, NER, Summaries Local Heuristics (0 Tokens)
Simple Arithmetic Fast Regex Solver (0 Tokens)
Complex Logic & Coding Fireworks Escalation
ZeroToken Router ACT II
Slide 2 of 5
AMD Developer Hackathon ACT II · Track 1

Technical Architecture

"Engineered for strict hardware and runtime limits."

  • Bundled Model: CPU-only Qwen2.5 3B GGUF model compiled inside the Docker image using optimized `llama-cpp-python`.
  • Targeted Premium Routing: Escalates factual/logic tasks to MiniMax M3 and coding queries to Kimi K2.7 Code.
  • Grading Constraints Compliant: Fits easily inside the 4 GB RAM, 2 vCPU, and 10 GB compressed image thresholds.
Input Batch: tasks.json
↓
Deterministic Classifier
↓
Local Solver & Qwen
Fireworks API Route
↓
Atomic results.json Output
ZeroToken Router ACT II
Slide 3 of 5
AMD Developer Hackathon ACT II · Track 1

Empirical Evidence & Metrics

"Maintaining 100% accuracy while cutting cost."

Profile Local Tasks % Simulated Eval Score Fireworks Token Cost Latency
Safe 0% 19 / 19 (100%) 100% (Baseline) 12.5s
Hybrid (Ours) 52.6% 19 / 19 (100%) 47.4% (-52.6% Cost) 5.8s
Local 100% Qwen Dependent 0% (Free) 45.0s
Fireworks Token Expenditure
Safe Profile
100%
Hybrid
47.4%
Local
0%
ZeroToken Router ACT II
Slide 4 of 5
AMD Developer Hackathon ACT II · Track 1

Inference as Infrastructure

"Automated cost optimization at the routing layer."

52.6% Savings

More than half of typical development queries resolved with zero API tokens.

Zero Latency Spill

Parallel remote execution and instantaneous heuristic checks speed up evaluation.

Production Ready

Dockerized, schema-guaranteed fallback prevents down-time on network errors.

"Maximize Accuracy. Minimize Cost. Zero Complexity."

ZeroToken Router ACT II
Slide 5 of 5