A Brain Too Big to Carry — On-Device vs Datacenter Inference

Author: Ivan Chiam, Gianluca, Zane Fong, Bryan Shan, Dylan Patel and Reyk Knuhtsen (SemiAnalysis) URL: https://newsletter.semianalysis.com/p/a-brain-too-big-to-carry-on-device One-line: Robotics inverts the LLM rule that hardware bends to the model; for generalist robots, though, moving the planning layer off the robot to shared datacenter GPUs wins on silicon, DRAM and cost per FLOP, and the thing standing in the way is the wireless network, specifically jitter.

Key claims

• Robotics inverts the LLM order of operations. With LLMs “the model comes first; the hardware second.” Robots face two constraints LLMs don’t: hard real-time control deadlines, and the manufacturer paying upfront for compute on every unit. So on-robot hardware is fixed and the model is designed to fit it — “intelligence becomes something you ration against latency and unit economics.”
• Robot models are still small, and the sizes are constraint-driven. Generalist robot models sit in the billions of parameters: Physical Intelligence’s π0 class ~3B, π0.7 5B, ByteDance’s GR-3 4B, Generalist’s ~10B, Nvidia’s DreamZero 14B (which needs two GB200s off-robot to run in real time). Months later Nvidia’s RoboTTT went the other way: a 3B policy using test-time training, small enough to run onboard. Early results from Generalist and Dyna suggest scaling works for robots too; data and the network are the binding inputs.
• Frequency decides what can leave the robot. Action and servo loops run at hundreds of Hz (a 100 Hz loop has 10 ms per output; a wireless round trip costs 10–50 ms), so “anything above roughly 100 Hz can never leave the robot.” The planning layer runs below ~20 Hz — a 5 Hz planner gets 200 ms per decision — so it can be offloaded. The real problem is jitter, not average latency. Jetson Thor, the best robot compute available, delivers about 1/10th the FLOPs of a GB200 and ~1/30th its memory bandwidth.
• The supply chain is pointed at the datacenter. Jetson is “the thin red sliver” in Nvidia’s accelerator output, and Blackwell earns gross margins in the mid-to-high 70s versus mid-60s for Jetson. Robot silicon has also moved onto contested nodes: Orin ran on Samsung SF8, Thor is on TSMC N4, and successors likely follow Rubin to N3 and Feynman to N2. Wafer volume isn’t the strain — a million ~400mm² Jetson-class chips in 2030 is ~10,000 wafers — so the question is silicon efficiency.
• Shared GPUs win on silicon and DRAM past a few robots per GPU. Wafers per robot cross over at ~7 robots per GPU; DRAM per robot at ~5. Jetson DRAM has climbed from 32GB (Xavier) to 64GB (AGX Orin) to 128GB LPDDR5X (Thor), while incremental DRAM capacity is absorbed by HBM, leaving robots competing for “a shrinking pool of non-HBM wafers” (see Memory).
• The TCO math favors offload once utilization is counted. Benchmarking a reconstruction of RoboTTT (GR00T N1.7 checkpoints with 16 TTT modules, 151 MB of per-robot fast-weight state), one B300 sustains 12 robots inside a 500 ms chunk deadline versus 4 on an RTX PRO 6000. For a 96-robot fleet: B300 offload ~$554K capex / $18.63/hr; RTX 6000 Pro $436K / $15.61/hr; 96 Jetson Thors at $3,500 ~$394K / $14.97/hr (four-year life, shorter because robots vibrate and get knocked around). Before utilization, cost per FP4 PFLOP is $0.15/hr (B300), $0.16 (Thor), $0.39 (RTX). Net of utilization (~90% server vs ~40% onboard) it’s $0.17 vs $0.37, so offload costs ~46% of on-robot for industrial deployments and ~12% for home robots that run 1–2 hours a day. Offload wins from about 5 robots per B300.
• Deployed companies split by generality, wireless quality and site control. Boston Dynamics puts System 1 (the VLA) on an onboard Jetson Thor and System 2 (a planner they compare to DeepMind’s Gemini Robotics ER, estimated at hundreds of billions of parameters if not 1T) on Google TPUs via Orbit, querying it every few seconds. Agility Robotics (Digit; customers include GXO, Schaeffler, Amazon, Toyota and Mercado Libre) keeps inference onboard so it can drop into sites unchanged and because safety decisions can’t depend on a link. Verne Robotics runs few-billion-parameter policies locally. Sunday Robotics planned on cloud inference and reversed “two days into testing ACT-2 in real homes” because of jitter; Starlink and a 5G router dropped out the same way. Weave Robotics runs locally but still relies on Wi-Fi for training uploads and teleoperation.
• The network wall is about hardware, and it’s solvable. Robots are metal, they move, and their motors make noise on the frequencies they transmit on. Wi-Fi and 5G were built for downlink-heavy, stationary clients; robots stream video uplink while moving, and a typical router goes silent for 100 ms to several seconds during a handoff. The proposed fixes: robot-aware access points (uplink scheduling, location-aware beamforming, centralized timing, multi-link, 6 GHz, fast handoffs), shrinking the perception load, 4x4 uplink MIMO mainboards, and putting every robot on one clock so datacenter batching works. Autonomous Vehicles are the counterexample where compute has to stay onboard.
• Three filters decide who climbs the wall: how much control you have over RF conditions, the task’s latency budget, and whether the task is complex enough to justify a datacenter GPU. Factories and homes pass the first filter; street-roaming robots don’t.

Notable quotes

“Because of these two constraints, in robotics the on-robot hardware is fixed, and the model is designed to fit what we can reasonably manufacture.”

“Robot model sizes tell you where the constraints sit, not where sizes will settle.”

“So a low-volume, lower-margin Jetson will always be competing for advanced-node wafers from the back of the queue.”

“Bolt a datacenter GPU onto the robot and the chip alone would out-draw the entire machine, demand cooling the robot cannot carry, and flatten the battery in minutes.”

“If the robot loses connection to an offboard safety model, it cannot tell whether anything is wrong, so its only option is a blind, worst-case fail-safe. That is why complex safety decisions must always stay on the robot.”

“Latency was mostly fine. Jitter was the problem. A robot trying to hold 99% success over hours can’t shrug off the occasional hiccup the way a video call can.”

“Add shared fiber to the picture, and your neighbor turning on Netflix while your humanoid is working can, quite literally, kill you.”

“A single-task arm running a simple policy has no reason to leave the robot. A 200B parameter model automating all the busy work in a car plant, and coordinating with the dozens of other robots on the same floor, has every reason.”

Charts and figures

How it connects

• Robotics — the main subject. It covers generalist robot models from Boston Dynamics, Agility Robotics, Verne, Sunday Robotics and Weave, and where each one decided to run inference.
• Nvidia — Jetson Thor (on-robot) vs Blackwell/B300 (datacenter) is the core TCO comparison. The margin gap (mid-to-high 70s vs mid-60s) explains why Nvidia has little reason to ramp robot silicon yet.
• AI Infrastructure — the piece is about inference infrastructure for physical AI. Its conclusion is that robot cognition should be pooled in the same datacenters that serve LLMs.
• Machine Learning — the robot models (π0, GR00T, RoboTTT, DreamZero) are the workloads being sized; their parameter counts and latency budgets drive every architecture decision.
• SemiAnalysis / Dylan Patel — the publisher and co-author. The cost numbers come from SemiAnalysis’s Accelerator, Memory and AI TCO models.
• TSMC — Thor moved from Samsung SF8 to TSMC N4, which puts robot silicon in the same queue as datacenter accelerators.
• Memory — HBM is absorbing most new DRAM capacity, which squeezes the LPDDR supply robot brains need.
• DeepMind / Google — Boston Dynamics serves System 2 on Google TPUs through the DeepMind partnership and the Gemini Enterprise Agent Platform.
• Physical Intelligence and ByteDance — their π0/π0.7 and GR-3 models anchor the model-size chart; Weave Robotics also runs Physical Intelligence models.
• World Models — DreamZero, a 14B “world action model” on a video-diffusion backbone, is the high-compute end of the range, and world models are named as one answer to the robot-data shortage.
• Edge Computing — this is the edge-vs-cloud question for embodied AI. The piece’s answer is to keep the fast loops on the edge and pool the slow, heavy ones.
• Autonomous Vehicles — the counterexample: huge coverage area, no RF control and fast movement force compute onboard.
• Starlink / 5G — Sunday tried both as fallbacks and both dropped out; 5G is cast as the complement to Wi-Fi on larger and outdoor sites.
• Tariff Armageddon and the GPU Loopholes and What Is So Hard About Behind-The-Meter Power for Datacenters Part 1 — other SemiAnalysis long-reads on the supply side of the same datacenter buildout.