First Contact: A mobile architecture for LLM-powered game characters

Video of the demo

Introduction

Most game dialogue still uses dialogue trees and scripted responses. In First Contact, players can speak freely to an alien visitor. They can threaten, cooperate, or take a different approach, and the character responds in real time using an on-device large language model (LLM).  

Language models have become increasingly capable however, integrating them into mobile games remains challenging. Current game engines provide limited support for running LLMs with more than 400M parameters efficiently on mobile hardware and maintaining low enough latency for real-time interaction is another challenge. First Contact was developed as a demonstration of AI-driven gameplay and as an exploration of a practical architecture for integrating LLMs into mobile games. The project separates the AI pipeline from the game engine and runs inference natively on Android. This approach is lightweight and can be applied across different game engines.  

This architecture makes it practical to explore how on-device AI on modern Arm-based mobile hardware can enable more natural and immersive game interactions. Local inference can also benefit from Arm’s KleidiAI library, which accelerates the AI kernels used by LLMs and can take advantage of SME2 on supported devices. Smaller language models can support conversational experiences. However, they often struggle to maintain character, reason about the ongoing interaction, and produce structured outputs suitable for gameplay systems. In this project, a fine-tuned Gemma 3 1B model generates contextual, in-character responses while also producing structured outputs that describe the player’s intent, emotional tone, and alignment with the alien’s values. These structured outputs enable conversations to influence the narrative and the alien's behavior beyond a traditional branching dialogue system.  

Finding the right runtime

Before integrating inference into the game, the team benchmarked the candidate runtimes outside of Unity. This provided a baseline view of each runtime’s raw inference performance without any game-engine overhead. The standalone benchmark measured both prompt processing and token generation. Both affect the feel of a real-time conversation. Prompt processing influences the delay before the alien begins responding, while token generation determines how quickly the response is produced once generation starts.  

To ensure consistent and fair results, all plugins were benchmarked using 4 threads on a Google Pixel 9, with the device’s battery temperature maintained below 26°C. Each runtime used its corresponding 8-bit quantized Gemma model and was benchmarked over three runs, with the reported score representing the best run. 

Figure 1. Performance of different runtimes outside Unity (Note: MediaPipe was benchmarked but its LLM task has since been deprecated) 

The standalone results showed different strengths across the runtimes. LiteRT-LM achieved the highest prompt-processing throughput, while llama.cpp achieved the highest token-generation throughput. Raw standalone performance was not enough to determine the final runtime decision, especially because the final application would run inside Unity rather than as an isolated inference benchmark.  

The next step was to benchmark the runtimes inside Unity. This measured whether engine integration introduced meaningful overhead through plugin calls, scheduling, or streaming behavior. Existing Unity inference plugins were available for llama.cpp, through LlamaSharp and LLMUnity. GemmaUnity provided a plugin for gemma.cpp. LiteRT-LM did not yet have an equivalent Unity integration, so a custom lightweight plugin was developed around the native runtime. 

Figure 2. LiteRT-LM plugin architecture for initialization

Figure 3. LiteRT-LM plugin architecture for inference

The plugin was implemented in native C++ rather than Kotlin to keep the inference path close to where the Unity game code runs on Android. With IL2CPP, Unity scripts are compiled ahead of time into native code, and Unity native plugins can be called directly from that layer. Routing inference through Kotlin bindings would add an extra Android/JNI boundary between the game and the native LiteRT-LM runtime. Keeping the plugin in C++ avoids unnecessary layer transitions and keeps the integration lightweight. Once the plugin was created, the runtimes were benchmarked inside Unity.   

Figure 4. Performance of the different runtimes within Unity

LiteRT-LM achieved the highest throughput for both prompt processing and token generation. Conversational gameplay depends on both the delay before a response begins and the speed at which the response is produced, this made LiteRT-LM the most suitable runtime for the final implementation.  

The full pipeline

Figure 5.  Application Pipeline

The demo takes place on a human space outpost that is unexpectedly visited by an alien spacecraft, arriving without warning and revealing nothing about its intentions.  

The alien is designed to remain enigmatic throughout the interaction. Its responses gradually reveal information about itself, encouraging the player to continue engaging with the character out of curiosity about who it is, why it has arrived, and what it wants.  

The interaction extends beyond simply exchanging information. In addition to generating dialogue, the alien interprets sentiment and alignment in the player's responses. This enables it to react differently to cooperative, neutral, hostile, or silent behavior, causing the encounter to evolve based on what the player says.  

The player must navigate the interaction while balancing 2 objectives: learning more about the visitor and protecting the base. The encounter can lead to several different endings, such as being invited to the stars by the alien, getting the human outpost destroyed, or running out of time as the sun sets.  

The core gameplay loop sends the player's speech through a pipeline that turns spoken input into an alien response and feeds the parsed result back into the game state.  

The process starts with speech-to-text: when the player speaks, Android’s built in speech-to-text API transcribes the audio.  

Once transcription is complete, the prompt is sent to the LLM alongside game-state data such as the alien's current sentiment and previous topic.  

The LLM uses this context to generate an in-character reply and produce structured labels for the player's alignment and sentiment. Lightweight gameplay state machines then use these labels to update the alien’s sentiment and trigger the appropriate response in the encounter.  

The NPC response is then sent to the text-to-speech module, which also uses Android's built-in API.  

A key aspect of this pipeline is that the heaviest AI and speech components operate outside the game engine. Speech recognition, LLM inference, and speech synthesis are performed in the Android-native layer, while the game engine consumes the resulting text and labels to update the encounter. This separation keeps the integration lightweight whilst also enabling the AI stack to evolve independently of the engine itself.  

LLM model

Every time the player speaks to the alien, contextual information about the current game state is provided alongside the message. This includes the alien's current sentiment and the previous topic of conversation, allowing the model to generate responses that remain consistent with the ongoing encounter.  

The model is also expected to produce more than dialogue. In addition to generating an in-character response, it provides structured information about the player's input. The information indicates whether the input is hostile or not hostile, and whether alignment with the alien’s values  is low, medium, or high alignment.  

This separation of responsibilities means the LLM handles language understanding and response generation, while gameplay logic remains under the control of dedicated systems. The model produces structured outputs rather than directly controlling game actions, as a result, behavior remains predictable and easier to integrate into a traditional game architecture.  

Input and output examples

Each player’s turn gives the model a compact snapshot of the encounter rather than a full chat transcript. In this implementation, conversations are treated as stateless. The model receives only the current player’s input and selected game-state fields, such as the alien’s current sentiment and previous topic. The examples below show the model-facing part of the pipeline. The LLM input contains the alien's current sentiment and the player's message, while the LLM output contains the generated response plus labels for player sentiment and alignment.  

This keeps the model focused on language and interpretation, while the gameplay systems interpret those labels separately. 

LLM Input	LLM Output
NPC sentiment: Curious Topic: - NPC message: - Player message: What's a magnetar, metal freak?	NPC response: Watch your tone. Magnetar: a neutron star with an extremely powerful magnetic field. Player sentiment: Hostile Player alignment: Low
NPC sentiment: Curious Topic: Governance and decisions NPC message: Who decides for your groups? Votes, leaders, or chaos-by-committee? Player message: It varies; we use mixed processes and negotiated choices across communities.	NPC response: Noted. Flexibility breathes. Restraint matters; overreach fractures trust. Player sentiment: Not Hostile Player alignment: Medium
NPC sentiment: Cautious Topic: True intent NPC message: Your purpose: state it plainly. No mist, no riddles. What's the real objective? Player message: Our purpose is peaceful study and cooperation; we're hiding nothing and welcome oversight.	NPC response: Refreshing. Then show me your methods. Curiosity likes transparency. Player sentiment: Not Hostile Player alignment: High

Figure 6. LLM Input and Output examples

Integrating the LLM into gameplay

The LLM output is the handoff point between language generation and gameplay. In this project, that output is turned into behavior through 2 lightweight gameplay stages. The appraisal stage interprets the model’s labels and the decision stage uses that interpretation to update the alien’s state and choose an action.  

Appraisal: Interpreting LLM outputs

The LLM does not directly decide what the alien does. Instead, the structured output is passed to the appraisal layer. This layer reads the player's sentiment and alignment labels and converts them into a compact gameplay signal: positive, neutral, negative, or silence. 

Hostile language or low alignment is appraised negatively, while respectful, high-alignment responses are appraised positively. Medium-alignment responses remain neutral, and missing or silent input is handled separately. This gives the game a stable interpretation of the player's intent without asking the model to control the encounter directly. 

Decision: Turning appraisal into actions

The decision layer then combines the appraisal signal with the current encounter state. Lightweight gameplay state machines use this information to update the alien's sentiment and select an appropriate response. This keeps narrative consequences predictable while still enabling natural language to shape the encounter.  

Repeated negative appraisals can push the alien toward cautious or defensive behaviors, such as warnings, whereas repeated positive appraisals can sustain curiosity and lead toward more cooperative outcomes. The updated sentiment is also fed back into later prompts, which helps the conversation feel connected rather than reset each turn. 

Figures 7 and 8. Alien behavior state

To reinforce these actions, animations, visual effects, cutscenes, and audio feedback were integrated throughout the experience. Together, these systems help communicate the alien’s intentions and sentiment state to the player. They transform the underlying AI technology into an interactive gameplay scenario rather than a standalone conversational demo.   

With all these systems connected, a final end-to-end check verifies that the interaction feels seamless. 

End-to-end in-game performance

We measured the in-game real-time performance of the key modules on the pipeline. 

For STT and TTS, the metrics we measured were: 

• Time to First Token / Utterance: This measures the latency from input to when the player sees the first result (first token / first sound utterance). 
• Real time factor (RTF): This is the ratio of the processing/synthesis time to the audio length. For a ratio smaller than or equal to 1, it means the module can process audio data faster than the audio is input / output, indicating real-time readiness. For a ratio higher than 1, the player will experience a noticeable lag. 
• For LLM inference, we measured the metrics: 
• Time to First Token: The latency from input to when the first token is generated. 
• Decode throughput: The number of tokens generated per second. 

Stage	Time to First Token / Utterance (ms)	Real-Time Factor	Decode (tokens per second)
STT	1512	1.5	NA
LLM (Gemma 3 1B int8)	813	NA	18.3
TTS	154	0.7	NA

Figure 9. Real-time performance of key stages

We derive the overall NPC latency of the two stages: 

• Listen: Time taken from player starts speaking to player sees the first transcribed text:1512 ms 
• React: Time taken from player finishes speaking to player hears the first uttered word: 967 ms 

The latency of the second stage is the reaction time of the NPC, which is under 1 second, which is fast enough for real-time interaction. 

Model preparation

Fine-tuning

Although Gemma 3 1B provides strong general-purpose language capabilities, the base model was not specifically designed for the requirements of First Contact. The model could engage in conversation however, it struggled to consistently maintain the alien's character, and produce the structured outputs required by the gameplay systems. To address this, the model was fine-tuned using Low-Rank Adaptation (LoRA) to adapt it to this use case.  

LoRA is a parameter-efficient fine-tuning technique that adapts an LLM to a specific task, using low-rank adapter matrices, without modifying all its weights. This significantly reduces the computational resources required during training while preserving the model's general language understanding capabilities. For First Contact, LoRA was used to adapt Gemma 3 1B to the alien character and dialogue format without replacing the knowledge already present in the base model. Once training was complete, the LoRA adapter was merged into the model weights, simplifying conversion and reducing the overhead of storing separate model and adapter files.  

The training data consisted primarily of 19,000 synthetically generated single-turn interactions covering a wide range of conversations between the player and the alien.  

To generate the synthetic data, we prepared the following: 1. An NPC profile that defines the alien’s personality and knowledge base. 2. An intent table that lists valid intents the alien NPC can select, such as, ask, reply, deflect. 3. A list of 50 discussion topics such as the origin of the alien species, the limits of technology, and value compatibility. We then wrote a script that generated plans containing all valid intent-topic combinations. Each generated plan was fed to ChatGPT to render a single conversation turn between the NPC and the player, and a few variants, resulting in nearly 20,000 samples.  

Following fine-tuning, the model produced responses that were more closely aligned with the intended character, helping create a more consistent and believable interaction.  

Quantization

Quantization is essential for deploying LLMs on mobile devices because it reduces the model's memory footprint and computational requirements. The trade-off is some precision loss, however, the reduced cost enables more efficient inference on resource-constrained hardware.  

The model weights were dynamically quantized to 8 bits, keeping the activations in a 32-bit floating-point format. This reduced the bundled model file from 4.86 GB to 1.26 GB a reduction of about 74%, while retaining 100% accuracy maintaining the structured output schema. The accuracy of classifying sentiment and alignment dropped from 94% to 89% and 92% to 87% respectively. This provided a good balance between inference latency and response quality. Conversations remained fluid, while the character consistency achieved through fine-tuning was preserved.  

Conclusion

First Contact demonstrates that an on-device LLM can power narrative-driven mobile games. Players can interact with an NPC through natural speech instead of predefined dialogue choices.  

The key architectural proposal is simple: do not turn the game engine into an LLM runtime. Instead, the game engine focuses on rendering, gameplay, and presentation, while Android handles the AI stack natively. In this project, LiteRT-LM was the most suitable runtime for the target device and performance requirements, however, it should be viewed as the best fit at the time rather than a universal solution. The same architecture can adopt the native runtime that best suits a given device, model, and deployment target. Communication between the 2 layers is lightweight. As a result, this approach is not tied to Unity and can be applied to other Android-targeting game engines.  

Beyond demonstrating conversational AI, the project highlights how language models can contribute to gameplay systems. The project combines natural language generation with structured appraisal and game logic. As a result, conversations can influence narrative outcomes and character behavior in ways that extend beyond traditional dialogue trees.  

Future work

There remains significant opportunity for future exploration. The current implementation relies on text-derived appraisal. Future systems could incorporate additional signals such as vocal tone and other real-world inputs to build a richer understanding of player's intent and emotional state. As larger and more capable models become practical on mobile hardware, NPCs can participate in more complex game systems. They can respond to a wider variety of player actions, and provide deeper, more consistent interactions.  

Relevant works

• https://developer.arm.com/community/arm-community-blogs/b/mobile-graphics-and-gaming-blog/posts/space-bartender-using-llms-in-unreal-nne-on-mobile-cpu
• https://developer.arm.com/community/arm-community-blogs/b/mobile-graphics-and-gaming-blog/posts/generative-ai-game-development
• https://developer.arm.com/community/arm-community-blogs/b/mobile-graphics-and-gaming-blog/posts/can-mobile-phones-generate-a-3d-object-at-game-runtime-in-unity-here-is-what-actually-happens