Hybrid Voice Interaction: Local STT/TTS with LLM Integration

[twistking]

Hello,
Since ED is currently prototyping new ATC and presumably other AI interactions, I hope I can provide some useful suggestions.

I'm very critical of LLM "AI," but for video games - especially dynamic and immersive simulation environments like DCS - the technology is, in my opinion, extremely useful. To my surprise, the technology has now reached a point where it is also technically viable for a product like DCS, with very lightweight and fast models that can run locally on client machines, thereby solving a lot of economic obstacles. Also, most of these models are open source and relatively easy to implement.

The cornerstone would be TTS/STT on the local machine, meaning speech generation and natural speech processing from the player (through a microphone). Models are now fast enough to run alongside DCS without significantly impacting performance. However, minimum system requirements would increase, as at least 2 CPU cores and a few GB of RAM would need to be dedicated to this task.

This approach would not allow free-flowing conversations with the AI. The logic would still be scripted. A full LLM capable of handling open-ended conversation with the player would require a high-end workstation PC if run alongside DCS.

However, models are becoming more efficient and average player PCs are becoming more powerful. ED should therefore anticipate the feasibility of locally running LLMs to handle the entire stack.

In the meantime, ED could create an API for external LLMs to hook into their system, either to completely take over the logic or to augment the scripted logic with some flair and personality.
Such an external LLM could be cloud-hosted (e.g., using the user's paid account with providers like OpenAI) or locally hosted, either on the player's machine or on a dedicated PC on the network.

The proposed system would be fully multiplayer-compatible, since all communication with the AI is exchanged as text, while TTS and STT are handled locally on each client. If a larger LLM is desired for more free-flowing conversation, it could be run either on the server or in the cloud, with the server API managing the interaction between the game and the LLM.

Thanks for reading.

PS:
One major advantage of this hybrid approach is that the LLM component could be easily maintained and expanded by the community. With a clean API separation, there would be no security risks for multiplayer or core gameplay, but plenty of room for creative extensions. For example, different LLM “personalities” could be developed - from a strict flight instructor helping beginners step by step, to a more casual RIO adding flavour and banter to the experience.
Neither development nor maintenance of these LLMs would be on ED, as long as they provide a robust API for the external LLM to hook into.

Edited 46 minutes ago by twistking

[MAXsenna]

I really hope the first part of your post is where we're going! To my knowledge BeyondATC runs locally, and is pretty much a "free flow", though you can't crack jokes with it. 

Cheers! 

[SharpeXB]

Great idea but just realize these things in other sims are paid subscriptions and have probably 10x the player base DCS has. Plus they only do ATC and not all the combat comms this game has. I can’t imagine ED would have the resources to do this.

[twistking]

54 minutes ago, MAXsenna said:

I really hope the first part of your post is where we're going! To my knowledge BeyondATC runs locally, and is pretty much a "free flow", though you can't crack jokes with it. 

Cheers! 

Interesting! I just watched a video about Beyond ATC, but it’s not entirely clear to me how they handle it technically. The underlying logic is probably still scripted.

Placing an LLM between DCS logic and the text output could, however, add some useful functionality. Not strictly necessary for ATC, but for communicating with an AI backseater/RIO/Wizzo or C2/AWACS it could make a big difference.

32 minutes ago, SharpeXB said:

Great idea but just realize these things in other sims are paid subscriptions and have probably 10x the player base DCS has. Plus they only do ATC and not all the combat comms this game has. I can’t imagine ED would have the resources to do this.

Yes, I’ve considered the economic hurdles, but please note: the STT/TTS components are open-source, proven, and reliable. ED would mainly need to build a robust scripted system that covers as many situations as possible. That’s busy work, but not rocket science. After that, they could develop a stable API that optionally connects to either a locally hosted or a cloud-based LLM.

The LLM part would remain optional. Players or server owners could do one of the following:

• run a model themselves (a modern 16-core CPU with 64 GB RAM can already handle a very lightweight LLM alongside DCS)
• connect the API to a paid cloud-hosted service
• simply skip the LLM entirely and still enjoy high-quality, low-latency voice communications with AI in DCS through the scripted system

Edited 3 hours ago by twistking

[MAXsenna]

6 minutes ago, twistking said:

Interesting! I just watched a video about Beyond ATC, but it’s not entirely clear to me how they handle it technically. The underlying logic is probably still scripted.

They just use the Microsoft Speech Recognition included in Windows, like VoiceAttack/(VAICOM), but they use AI to "restructure" the speech-to-text, in case you say something in the incorrect order. Right now that's all we need in DCS in my opinion. And @SharpeXB is correct of course, they only need to use it for ATC. But that shouldn't stop us from giving commands to wingmen/JTAC in DCS. Hell, we can even do it now. But it's limited to exact commands limited by how VoiceAttack functions. Maybe VoiceAttack will include some sort of AI down the line. Then we're golden. Sort of. None of this will work until ED has fixed the overall AI in the game.

12 minutes ago, twistking said:

Placing an LLM between DCS logic and the text output could, however, add some useful functionality. Not strictly necessary for ATC, but for communicating with an AI backseater/RIO/Wizzo or C2/AWACS it could make a big difference.

And we can already do that by now with VAICOM, except you need to say the correct commands perfectly. That's where we need the AI like they've done in BeyondATC. I have no need to strike up conversations, but have the AI "understand" me in stressful situations.

Maybe I'm missing your point, but what more do you really need?  

Cheers! 

[twistking]

18 minutes ago, MAXsenna said:

They just use the Microsoft Speech Recognition included in Windows, like VoiceAttack/(VAICOM), but they use AI to "restructure" the speech-to-text, in case you say something in the incorrect order. Right now that's all we need in DCS in my opinion. And @SharpeXB is correct of course, they only need to use it for ATC. But that shouldn't stop us from giving commands to wingmen/JTAC in DCS. Hell, we can even do it now. But it's limited to exact commands limited by how VoiceAttack functions. Maybe VoiceAttack will include some sort of AI down the line. Then we're golden. Sort of. None of this will work until ED has fixed the overall AI in the game.

And we can already do that by now with VAICOM, except you need to say the correct commands perfectly. That's where we need the AI like they've done in BeyondATC. I have no need to strike up conversations, but have the AI "understand" me in stressful situations.

Maybe I'm missing your point, but what more do you really need?  

Cheers! 

Ok, i think that's a NLU model, which allows you to say stuff out of order with the system still understanding your intent from a limited set of possible commands. These are lightweight and would also be part of my proposal. I just did not mention them specifically to make the post easier to digest.

The benefits of optionally integrating an LLM would be that the AI could actually react dynamically to situations in the sim. The API would not only pass along player communications (in text, after STT), but also selected sim parameters. For example, a flight instructor LLM could assist new players, while a RIO could comment on your flying skills or provide input on the tactical situation. The API could expose different sets of parameters depending on the role the LLM is meant to "play." A RIO would receive aircraft states and the tactical picture (within the realistic limits of what it could actually know).

The model could also be trained on DCS manuals and NATOPS. I think the advantages are obvious, but i admit that currently it might push most computers too far. However, in just a few years the situation will be very different. There are already extremely lightweight LLMs that, if trained on the right data and fed with real-time game states, could easily take on the role of an instructor or a cocky RIO.

Edited 3 hours ago by twistking

[SharpeXB]

1 hour ago, twistking said:

ED would mainly need to build a robust scripted system that covers as many situations as possible. That’s busy work, but not rocket science.

Look how long it takes to accomplish “simple” things here and you want this? I agree it’s a cool idea.

How long as the “improved ATC” without all this voice recognition been in the works, 8 years?

Sure this is a wishlist so wish away…

[MAXsenna]

Look how long it takes to accomplish “simple” things here and you want this? I agree it’s a cool idea.
How long as the “improved ATC” without all this voice recognition been in the works, 8 years?
Sure this is a wishlist so wish away…

@twistking I have to agree with SharpeXP here. Give me a better more realistic ATC which I can speak to along with EWR, AWACS, JTAC and wingmen in the near future! I'll be very happy and dream about your proposal!

Sent from my SM-A536B using Tapatalk

[twistking]

@SharpeXB@MAXsenna 

Fair enough. If ED released something like this tomorrow, most people simply wouldn’t have the hardware to run it locally anyway, and only a few would be willing to pay an AI company to make it work. In a few years, however, even mainstream PCs will be able to run a lightweight LLM alongside DCS without issue.

The STT/TTS and NLU integration is something we’ll hopefully get with the improved ATC (and it would be a disappointment if not). If ED is reading this, I’d just want them to consider future-proofing it by designing the system to be modular enough for an API hook. Those optional LLMs could then be fully community-developed. ED would “only” need to provide a robust API, which - in the grand scheme - isn’t that much work actually.

Assuming STT/TTS/NLU are all in place for the new ATC/AI communication: the API hook would sit in front of the TTS in the AI response pipeline. Instead of the TTS receiving input directly from the scripted system, the script-generated reply would go to the API together with auxiliary game-state data. The community-developed LLM would then generate a richer reply from those data points, which is finally sent back to the TTS.

I would argue the difficult part is the STT/TTS/NLU and scripted logic, which ED is hopefully already working on. The API is comparatively easy. Tuning the LLM through LoRA to work with the provided data in an optimal way could then be a community effort.

Edited 1 hour ago by twistking

[MAXsenna]

6 minutes ago, twistking said:

The STT/TTS and NLU integration is something we’ll hopefully get with the improved ATC (and it would be a disappointment if not).

Absolutely!  

7 minutes ago, twistking said:

If ED is reading this, I’d just want them to consider future-proofing it by designing the system to be modular enough for an API hook. Those optional LLMs could then be fully community-developed. ED would “only” need to provide a robust API, which - in the grand scheme - isn’t that much work actually.

Totally!

8 minutes ago, twistking said:

Assuming STT/TTS/NLU are all in place for the new ATC/AI communication: the API hook would sit in front of the TTS in the AI response pipeline. Instead of the TTS receiving input directly from the scripted system, the script-generated reply would go to the API together with auxiliary game-state data. The community-developed LLM would then generate a richer reply from those data points, which is finally sent back to the TTS.

I would argue the difficult part is the STT/TTS/NLU and scripted logic, which ED is hopefully already working on. The API is comparatively easy. Tuning the LLM through LoRA to work with the provided data in an optimal way could then be a community effort.

Let's hope they read this, and have had a good look at BeyondATC!

[SharpeXB]

1 hour ago, twistking said:

most people simply wouldn’t have the hardware to run it locally anyway

I don’t notice that BATC has any significant hardware requirements. The main obstacle I think DCS would face is simply not having enough users to fund such an add-on. And again these are actually subscriptions not just a buy once product. As cool as BATC is and it’s very cool, I wonder how many gamers in DCS who just blast off down the taxiways are willing to write down and read back taxi instructions