In-vehicle AI Agent
Edgenesis partnered with a pioneering EV manufacturer in Japan to develop and implement the industry's first in-car LLM-based AI Agent. The system enables comprehensive vehicle control and natural conversation while offering personalized experiences through secure user recognition. This groundbreaking solution integrates with vehicle systems to control functions like lighting and climate control, while actively engaging with drivers and responding to environmental contexts such as location and weather.
The solution combines edge computing with cloud architecture, powered by Qualcomm's Hexagon NPU for accelerated processing. Built on Android Automotive Operating System, the system uses function calling to convert user intent into actions, while abstracting different devices through vendor properties for unified control via AAOS's native SDKs. This architecture ensures both rapid response times and robust functionality across all vehicle systems.
Edgenesis delivered the following to the EV manufacture :
- In-vehicle AI Agent: Voice-enabled assistant for natural communication, device control, and context-aware responses with external information integration
- Edge/Cloud Hybrid Architecture: Optimized processing distribution between in-vehicle systems and cloud AI services, with local processing for critical functions and cloud capabilities for complex tasks
- AAOS Integration: Native integration with Android Automotive Operating System for seamless control of vehicle functions, applications, media and subsystems
- Personalization & Security: Facial recognition for driver identification with secure on-vehicle data storage, edge-level encryption, and authorized access control
- LLM Optimization: Fine-tuned model for accurate function calling, quantized and converted for Qualcomm AI Engine Direct for optimal performance on automotive hardware
- RAG Implementation: Retrieval-Augmented Generation system enabling access to vehicle documentation, user preferences, and external knowledge sources
- Vehicle Systems Integration: Comprehensive control over climate, lighting, entertainment, and other vehicle subsystems through unified interface
- Application Integration: Seamless interaction with vehicle applications for navigation, music, messaging, and other entertainment functions
- External API Integration: Real-time integration with weather, traffic, and third-party services for enhanced contextual awareness
Challenges
- LLM Fine-tuning: Ensuring accurate function calling and response formatting while maintaining consistent behavior across vehicle operations
- Edge Optimization: Quantizing and converting models for Qualcomm chip while preserving performance within resource constraints
- Vehicle Integration: Implementing seamless integration with Android Automotive OS and various vehicle subsystems through vendor properties
- Security Implementation: Developing robust edge-level encryption and secure storage systems that meet automotive industry standards
- Performance Tuning: Achieving consistent sub-second response times while managing resource allocation and battery consumption
Edgenesis Solution
Industry First
Successfully delivered the first production-ready in-vehicle AI Agent for electric vehicles with comprehensive system control
Hybrid Architecture
Implemented edge-cloud solution leveraging Qualcomm's Hexagon NPU for optimal performance and reliability
Seamless Integration
Achieved complete integration with Android Automotive OS, enabling unified control across all vehicle subsystems
Enhanced Privacy
Developed secure edge-processing system keeping sensitive user data on-device with encrypted storage
Resource Efficiency
Optimized LLM for automotive hardware while maintaining high accuracy and sub-second response times
User Experience
Delivered personalized interaction through facial/voice recognition with context-aware responses
Offline Capability
Ensured critical functions remain available without network connectivity through edge processing
Knowledge Integration
Successfully implemented RAG system providing instant access to vehicle documentation and external information
