Rethinking local-first AI assistants: Building an always-on OpenClaw-based runtime across Arm platforms

Running a local model is no longer unusual. Developers can deploy vLLM, Ollama, and other inference engines on servers, laptops, and on-premises Linux systems, then watch those models generate content directly on hardware they control.

The harder problem begins when that model becomes part of an everyday workflow. How does it retain explicitly saved information, answer questions about uploaded documents, retrieve current public information, or send scheduled notifications?

An inference endpoint does not automatically provide persistent memory, tool routing, document retrieval, or scheduling. Without those capabilities, the application remains a passive request-and-response system.

A local model is not an operational AI assistant.

A different architecture: Own the runtime, control the data, and adapt the compute

Building a long-running local-first assistant requires more than a model. It requires an architecture that combines an interaction channel, persistent data, tools, scheduling, and local inference services on infrastructure the developer controls.

The design follows three principles:

Figure 1. OpenClaw-based local-first reference runtime across replaceable Arm inference platforms.

This distinction matters. OpenClaw provides the assistant and Gateway foundation. The reference implementation extends that foundation with a Telegram gateway, deterministic routing, Ollama, Qdrant, Playwright, and a cron worker. These integrations are not part of the standard OpenClaw Gateway. The reference implementation adds them.

1. Own the runtime

An operational runtime does more than invoke an LLM API. It uses Telegram as a mobile and desktop interaction channel while integrating explicitly saved persistent memory, document RAG, public web search, and proactive scheduling.
From one interface, users can store information, query local content, invoke specific tools, and receive notifications triggered by the cron worker. The application moves beyond passive chat to become a continuously operating service.

2. Control the data

Local-first does not mean fully offline. It means that storage locations and external connections are explicit, inspectable, and controlled by the operator.

• Local data and computation: LLM inference, embeddings, Qdrant collections, local copies of uploaded documents, task history, cron state, and Gateway state.
• Explicit external boundaries: Telegram transports messages and uploaded files. The /search command enables Playwright to access a search engine and selected public pages. The weather skill contacts a public weather service, and setup requires downloading models and container images from external registries.

The runtime does not require a public cloud LLM API for inference. Telegram and explicitly enabled tools still cross the local data boundary. Sensitive deployments therefore require deliberate decisions about the communication channel, network access, host security, access control, and backups.

3. Adapt the compute

Because the assistant layer depends on an API contract rather than one inference engine, developers can reuse the same application architecture across different classes of Arm-based compute. The reference configurations demonstrate that portability with GPU-backed vLLM on NVIDIA DGX Spark and CPU-based llama.cpp on the Radxa Orion O6.

When the platform changes, the application workflow remains the same. The main adjustments are the endpoint, model name, context budget, and platform-specific configuration. The implementation demonstrates that one application contract supports different Arm compute configurations and local inference backends.

A customizable reference implementation

To validate this architecture, we built the openclaw-arm-continuum reference implementation and used a local-first household assistant as the scenario for an Arm Learning Path.

The household assistant is an accessible example, not a fixed product definition. The same architecture can support independent developers, engineering teams, and other use cases that require local inference and explicit data boundaries. The Learning Path uses only synthetic or public data. Readers do not need to enter real household or organizational information.

1. Persistent memory and semantic retrieval

A user can explicitly save a synthetic household memory from Telegram:

/mem #home The boiler should be inspected every October.

The user can later retrieve that information from the local Qdrant collection:

/rag memory: When should the boiler be inspected?

Ollama generates embeddings locally, Qdrant manages vector storage and retrieval, and the local LLM then generates a response from the retrieved context. Telegram still transports the original messages, while persistent memory, embedding, retrieval, and generation run on hardware controlled by the operator.

2. Document upload and asynchronous RAG

The user creates a synthetic household-maintenance.txt file and uploads it through Telegram. Any caption other than /tracker or /mem routes the upload to knowledge indexing by default, so the user uses /knowledge as the caption to make the destination explicit. The reference runtime saves a local copy, while the Memory Watcher performs content chunking, embedding, and Qdrant indexing in the background.

The bot returns a stored filename with a timestamp prefix. After indexing completes, the user can use that filename for a reproducible document-specific query:

/rag <returned-file-name> When should the heating filter be cleaned?

This workflow enables a user to upload a document from a phone while keeping the saved copy, vector index, and generation process inside the local runtime.

3. Explicit browser search

When a request requires current public information, the user can explicitly select the browser search route:

/search Arm Learning Paths local AI development

The Playwright worker connects to a public search engine and selected pages, saves the retrieved content as local Markdown, and passes that context to the local LLM for summarization. The user explicitly triggers this operation with the /search command because it crosses the external network boundary.

4. Proactive scheduling and Telegram notifications

The runtime uses a separate cron worker to manage time and schedules. It does not assume that the LLM knows the current time. A user can create a scheduled task directly from Telegram:

/cron add daily 21:15 Heating check :: Remind the household to review the heating maintenance notes.

At the configured time, the cron worker executes the task and delivers the result to an allowlisted Telegram chat. The user can also inspect the schedule through the Gateway dashboard and runtime logs.

Moving from DGX Spark to an Armv9 CPU

The proof of concept first deploys a vLLM-based runtime on NVIDIA DGX Spark. It then replaces the generation backend with llama.cpp on a Radxa Orion O6. This change validates the same upper-layer architecture across different Arm compute configurations and local inference backends.

Both deployments retain the same Telegram interface, Ollama embeddings, Qdrant collections, skills, browser search, cron behavior, and deterministic routing. On the Orion O6, the runtime connects to llama.cpp through the same OpenAI-compatible API contract and uses systemd to manage the local server.

The migration validates the article’s central claim: the assistant workflow remains stable while the inference service changes underneath it. Telegram, memory, retrieval, tools, scheduling, and routing remain intact; developers change the endpoint, model, context budget, and platform-specific service configuration for the target Arm system.

Build a local-first runtime you control

Local-first AI does not eliminate every trade-off between privacy and functionality. The important requirement is to keep inference, persistent data, external tools, and communication boundaries explicit. Operators can then choose and constrain those components according to the deployment.

Developers can extend a local-model demo into an operational and inspectable AI runtime. The architecture combines an OpenAI-compatible API with replaceable local inference backends, persistent memory, RAG, tools, and scheduling. The resulting runtime can work across different Arm platforms.

The reference implementation is available under the Apache 2.0 license. To build it yourself:

• Arm Learning Path: Extend OpenClaw for a Local-First AI Assistant Across Arm Platforms
• GitHub repository: odincodeshen/openclaw-arm-continuum. The tutorial uses the v1.2 release tag to keep the instructions reproducible.

This reference implementation is not a finished product. It provides a foundation for further development. You can preserve the explicit data boundaries and OpenAI-compatible API contract as you extend the implementation. You can add skills, specialist agents, or local inference backends that match your hardware, privacy requirements, and application. This approach provides a practical path to a local-first AI assistant that you control. The assistant can continue to evolve with the underlying Arm compute platform.

Extend OpenClaw for a Local-First AI Assistant Across Arm Platforms