A model on the device across a fleet of sites: update it like firmware, not like a chat

Red Hat opened a conversation about managing models on thousands of devices, Hugging Face released version 1.0 of a library that lets an iPhone or Mac app work with a local model, NVIDIA showed how to link eight desktop DGX Sparks, and n8n put the whole model set for a small company on one such box with no cloud API. For a business owner with several sites, the main question is who updates the model at each of them, and how.

What came out

Red Hat on October 20 published an article on MLOps for edge AI: once a model is optimized and packaged, the management questions remain. Which version runs where, and how to update it without an incident across the whole fleet. I only have the beginning of the article, so I'm not retelling Red Hat's answers.

Hugging Face on October 20 released AnyLanguageModel 1.0, a replacement for Apple's Foundation Models framework: you change the import line, and the same session code works with the system model, with open models through llama.cpp, MLX and Core ML, or with cloud providers, including OpenAI, Anthropic, Gemini and Ollama. Tool calling and structured output work with every provider. The API key is set through a function rather than a string, so the app can fetch short-lived tokens from its own server. For a start they name Gemma 4 E2B, around 3.4 GB, and LFM2.5 2.6B, around 1.7 GB, both of which fit on an iPhone; Qwen3.8 27B takes around 16 GB and needs a Mac with 32 GB of memory.

NVIDIA on October 22 announced at GTC Berlin that up to eight DGX Spark nodes can be linked for models that need hundreds of gigabytes of memory. It's an announcement, and there's little detail in it.

n8n on October 19 showed how to keep all the models for a small company's workflows on one DGX Spark with 128 GB of unified memory: chat and data extraction on Qwen3.6-35B-A3B, embeddings, a reranker, Whisper for speech, GLM-OCR for documents, image generation and speech synthesis, all behind one OpenAI-compatible address through LiteLLM with per-team keys. The author writes that a clinic, a law firm or an accountant often can't send client data to a cloud API, and gives numbers: 120 tokens per second for one user and 359 in total across eight. The most useful part for me is about discipline. Each model is downloaded once, with the exact commit recorded in models.lock, containers are pinned by version, and the vLLM version inside the image is checked before launch. The author's nightly tag turned out to be five months old, with vLLM 0.19, and the model failed to load with an error that looked like a corrupted checkpoint.

Example: a farm with two sites

Take a farm with two sites and controllers at each. Suppose the sites run a small model that identifies plant disease from a photo. At one site it was updated, at the other it wasn't. For a month both sites report “all fine,” but by different rules. The discrepancy gets found a month later, and no one remembers which version was where and since what day.

A model on a device has to be run like firmware: with a version number, with the install date on each device, with a rollback plan. A chat in the cloud gets updated for you; a model at your site, you update yourself.

How it works for me

Since 2023 I've been responsible for the server and network of a hydroponic farm in the US. The equipment there is run by 25 Raspberry Pi based controllers at 2 sites. The site networks have no public IP addresses, so I set up reverse proxies, NAT, a firewall and a VPN, and the result is one network in which a Debian server in Google Cloud with MariaDB sees every controller. Data from the controllers goes into the database, and commands go back: irrigation, pumps.

The system on the controllers is closed. It had no REST API, only local sockets. I figured out how that interface works and built the integration with the server on top of it; that link is still running. The automation logic and the interface are done by another team; the network, the database, updates and support are on me. I call it DevOps.

This work taught me to plan updates in advance: I do updates and debugging remotely, through the same network, from one server. If the farm needed a model at a site, I'd run it the same way as the rest of the controller software: version, date, device, a way back. The hardware and stacks from these stories I haven't tried myself.

What I would do

Set up one server that knows which model version and which container is on each device. At the farm the controller data already goes to one server with MariaDB; I'd keep the version records there too.

Pin versions in a file, like n8n's models.lock, and launch only from it. A latest tag at a site with no internet means nothing.

Update in stages: one device, one site, all of them. Between stages, compare the old and new versions' results on the same data.

Test the rollback path before the update, not in the middle of an incident.

Decide what the model does on site and what still goes to the server. Recognizing a photo can happen at the site; what to do with a batch of plants is better decided where both sites are visible.

Related reading

Why a system needs a shared center that keeps state and coordinates processing, I wrote in the article The server: data, access and process execution. Services: greenhouse and hydroponics automation and equipment monitoring and alerts. The project itself: Hydroponic farm in the US: 25 controllers at 2 sites. Nearby are my reviews on routing between your own model and an API, on observability on a single server and on zero trust for a 30-person company. Earlier I wrote that releasing a model on a device matters more than one fast answer, and about analytics next to an industrial historian. If you have equipment at several sites and are thinking of putting a model there, tell me what's already there and how it gets updated now. Discuss your task.