The Paradigm of Cognitive Twins
Most AI integrations are transactional: the user submits an isolated context, the model produces a stateless output, and the context is immediately cleared. To build a genuine cognitive twin—a digital mirror that retains context, learns across domains, and communicates fluidly in natural spoken audio—requires bypassing this stateless constraint.I explored this pattern through the DEXter AI Voice & Chatbot Hub, an independently developed personal portfolio prototype. It does not represent an employer or client system and contains no employer/client confidential information.
1. 🎛️ The Microservice Architecture
The system is built as a self-contained API route on our Next.js edge stack, handling:Client-Initiated E2EE Tunnels: All chatbot interactions are encrypted before transmission to prevent intermediate packet snooping.
Strict Rate Limiting: Uses a lightweight in-memory request limit to protect the prototype's inference endpoint.
Embeddable Demo Surface: Uses standard CORS and preflight
OPTIONS handling so the portfolio widget can be tested across approved demo surfaces.2. 🧠 Persistent Memory Graphs: Mem0 Integration
Instead of overloading the prompt context with static data, the prototype experiments with a dynamic RAG retrieval loop powered by Mem0:Anonymized User IDs: On first load, the client generates a unique cryptographic UUID (
dexter_user_id) saved to localStorage.Intent Ingestion & PII Redaction: Before storing or querying context, the prompt is analyzed by a regex parser to redact credit card hashes, emails, and phone numbers.
Graph Database Memory: During each chat turn, the sanitized message is indexed into the Mem0 graph memory. When the user returns, the system automatically pulls the top-K relevant semantic nodes of historical context, prepending it to the Gemini system instructions.
3. 🎙️ Duplex Voice Pipeline: Real-Time Audio PCM
Duplex voice requires low latency to feel conversational. We avoided heavy WebRTC setups by building a direct, browser-native Audio Context stream:Buffer Ingestion: Capture microphone input in real-time at
16000Hz mono using a standard ScriptProcessorNode to ensure compatibility across Safari, Chrome, iOS, and Android.PCM to Base64 Conversion: Map the raw float32 array samples directly into 16-bit signed PCM integers, chunk them into base64 strings, and stream them via WebSockets to the Gemini Live gateway.
Duplex Interruption: The system constantly monitors outgoing client audio buffers. If incoming speech is detected while the model is playing synthesized text-to-speech, the browser context immediately clears the audio source nodes, allowing the user to naturally speak over the model's voice.
EOF