Home/Research/Federated Hybrid LLM Inferencing
Research · Architecture

Cloud-class accuracy, generated locally.

A multi-tier framework that distributes inference across cloud, client, edge and IoT. A reward model scores every token and escalates only what falls short; every cloud correction is cached with O(1) lookup, so the hierarchy gets cheaper and more accurate the longer it runs.

Talk to a solutions architect
51–65%
Cloud cost reduction at launch, before the cache matures
~3%
Of tokens that require the cloud LLM at all
90%+
Local generation once the cache has matured
O(1)
Cache lookup, via suffix automaton
What it is

Intelligence distributed across a hierarchy of devices.

Rather than sending every token to a frontier endpoint, the framework keeps generation local and escalates only what a reward model rejects — then caches the correction so the same escalation never happens twice.

Multi-tier architecture

Routes inference across cloud, client, edge and IoT tiers by complexity and requirement.

Token-level reward routing

A reward model scores each token, escalating to cloud only when local generation falls short.

Active-learning N-Gram cache

Every cloud correction is cached at O(1) lookup, continuously improving local accuracy.

NVIDIA-optimized

Built for B200, H200, H100, DGX Spark, RTX and Jetson platforms.

The hierarchy

Four tiers, each sized to its workload.

Rejected tokens travel up; corrections travel back down. The N-Gram cache propagates cloud → client → edge → IoT, so intelligence earned once is reused everywhere.

Cloud70B+ params · B200 / H200 / H100
Source of reasoning and token verification. Invoked only when the reward model rejects an SLM token.
100% source
Client600M–4B · DGX Spark / RTX 5090
Subject-matter experts: domain SLMs for banking, legal and healthcare on DGX Spark; developer and analyst SLMs on RTX; 4B MoE (4×600M) on laptops and phones; real-time contextual reasoning on AR glasses.
90%+ hit rate
Edge1M–500M · RTX A-Series / Jetson AGX
Context experts: in-cabin automotive AI with offline capability, smart-hub orchestration for energy and security, and factory-floor predictive maintenance.
75% hit rate
IoT1M–50M · Jetson Orin Nano
Task experts: sensor agents interpreting data streams and raising alerts, robot controllers with cached cloud reasoning, and wearables running cached diagnostic patterns.
60% hit rate
How it works

Four steps, repeated per token.

1 · Edge SLM generationThe edge SLM generates candidate tokens autoregressively on local NVIDIA hardware — DGX Spark, RTX or Jetson.
2 · Reward model scoringEach token is scored against the cloud LLM’s distribution. Accepted tokens stay local; rejected ones escalate.
3 · Suffix automaton cacheEvery cloud correction is stored at O(1) lookup. Recurring patterns bypass both reward evaluation and the cloud entirely.
4 · Cache propagationThe N-Gram cache propagates down the hierarchy, so a correction earned at one tier benefits all of them.
Benefits

What the architecture buys.

Cost reduction

51–65% lower cloud cost at launch, falling further as the cache matures.

Ultra-low latency

90%+ of tokens generated locally, removing the network round-trip.

Data sovereignty by design

Private data never leaves the device during inference — sensitive tokens are processed on-device.

Offline capability

Edge devices keep working without connectivity, serving from cached intelligence.

Continuous learning

Every cloud correction improves the whole hierarchy, automatically.

Hardware flexibility

From B200 cloud servers to Jetson Orin Nano, across the full NVIDIA stack.

Tunable quality-cost dial

Move the reward threshold per use case — no retraining, no redeployment.

Decreasing marginal cost

Cloud reasoning is cached after first invocation, so the system gets cheaper as the cache saturates.

Use cases

From the datacentre to the wearable.

Enterprise hub

Domain SLMs for banking, legal and healthcare on DGX Spark, with full compliance and data residency.

600M–4BDGX Spark90%+ cache hit

Developer workstation

Code completion, documentation and analysis locally on RTX 5090/5080 at cloud-level accuracy.

RTX 5090Code-optimized

Personal AI assistant

4B MoE models on laptops and smartphones for always-available, privacy-first assistance.

4B MoE (4×600M)Mobile-ready

On-device coding assistant

A 3B code model on RTX 5090; the cloud handles architecture decisions and project-specific patterns are cached.

3B code modelPattern caching

Real-time translation

A 2B multilingual SLM on phones, with domain phrases cached from cloud.

2B multilingual<5% cloud calls

Automotive AI

In-cabin navigation, driver assistance and infotainment, fully offline-capable for safety-critical paths.

1M–500MJetson AGXOffline-ready

Smart home hub

Energy, security and scheduling orchestrated locally for privacy.

RTX A-Series75% cache hit

Industrial intelligence

Predictive maintenance, quality control and process optimization with real-time inference.

Industrial-gradeReal-time

Sensor agents & wearables

Single-task interpretation of sensor streams, robot controllers with cached reasoning, and health monitoring on ultra-low power.

1M–50MJetson Orin Nano60% hit rate

Want this sized against your device fleet?

The economics depend on your token mix and how fast the cache saturates. A solutions architect can model both.

Talk to a solutions architect