🎧 Kahani Magic Audio System

Complete audio architecture, issues, and improvement roadmap
Last Updated: January 21, 2026
Status: ✅ Word Timestamps Implemented (ElevenLabs + Deepgram)

---

📚 Documentation Cross-References

| Document | Audio Content | |----------|---------------| | ISSUES.md | Issues #1, #7, #8, #9, #14, #15 | | ASSETS.md | AUD, BGM, SFX asset definitions | | COMPONENTS.md | useAudioPlayer, useAudioCues hooks | | APIS.md | generate-audio, generate-bgm, generate-sfx APIs | | THIRD-PARTY-INTEGRATIONS.md | ElevenLabs, Deepgram, Beatoven docs | | Dev Docs | /dev-docs/studios/kahani-magic/audio-system | | TypeDoc | AudioCues Interface |

---

🎚️ Three Audio Layers

Layer 1: Narration (TTS)

Provider Chain (fallback order):

1. gemini-2.5-pro-preview-tts - Primary (best quality)
2. gemini-2.5-flash-preview-tts - Fallback (faster)
3. eleven_multilingual_v2 with timestamps - Final fallback ⭐ RETURNS WORD TIMING

Files: | File | Lines | Purpose | |------|-------|---------| | src/app/api/kahani-magic/generate-audio/route.ts | 265 | TTS API route | | src/lib/ai/emotion-service.ts | 316 | Emotion tagging | | src/lib/ai/elevenlabs-tts.ts | 485 | ElevenLabs + with_timestamps ⭐ |

Emotion Tags: [excited], [scared], [sad], [angry], [whisper], [mysterious], [happy], [curious]

Language Support:

• en - English (default)
• hi-en - Hinglish (Roman script, Hindi pronunciation)
• hi - Hindi
• ja-roman - Japanese Romaji

---

Layer 2: Background Music (BGM)

Provider Chain:

1. Beatoven.ai - AI music generation (50 free credits)
2. Freesound API - CC0 licensed search (unlimited)

Files: | File | Lines | Purpose | |------|-------|---------| | src/app/api/kahani-magic/generate-bgm/route.ts | 289 | BGM API route | | src/app/studio/kahani-magic/services/bgmService.ts | 496 | BGM controller |

8 Moods: peaceful, tense, joyful, sad, epic, mysterious, playful, romantic

Style Mapping:

indian-fables    → peaceful (indian-classical)
fairy-tales      → whimsical (gentle)
horror-mystery   → dark (mysterious)
sci-fi-adventure → epic (adventure)
comedy-humor     → playful (playful)

Auto-Ducking: Volume 15% → 3% during narration, smooth 0.3s transition

---

Layer 3: Sound Effects (SFX)

Provider Chain:

1. ElevenLabs Sound Generation - AI contextual sounds
2. Freesound API - CC0 search
3. Web Audio Synthesis - Client-side fallback

Files: | File | Lines | Purpose | |------|-------|---------| | src/app/api/kahani-magic/generate-sfx/route.ts | 259 | SFX API route | | src/app/studio/kahani-magic/services/sfxService.ts | 429 | SFX controller |

12 Predefined Types: pop, whoosh, swoosh, ding, rise, impact, success, chime, boing, tension, wind, laughter

Word Triggering: AI identifies trigger words like "BOOM!", "splash", "whispered" via /generate-audio-cues

---

📦 Playback Architecture

useAudioPlayer Hook (902 lines)

Location: src/app/studio/kahani-magic/components/BookReader/hooks/useAudioPlayer.ts

Current Responsibilities (⚠️ TOO MANY):

• Audio generation triggering
• Playback control (play/pause/stop)
• Karaoke time tracking
• WAV encoding
• Audio caching
• Autoplay sequencing
• BGM/SFX coordination

Proposed Modular Split: | New Hook | Responsibility | |----------|---------------| | useAudioPlayback.ts | Play/pause/stop, time tracking | | useAudioGeneration.ts | TTS generation, regeneration | | useAudioCache.ts | WAV URL caching, IndexedDB | | useAutoplay.ts | Autoplay sequence control | | useKaraoke.ts | Word highlighting sync |

---

🔴 Critical Issues (Audio-Related)

| # | Issue | Root Cause | Fix | Effort | |---|-------|------------|-----|--------| | 1 | No TTS timestamps | Gemini/ElevenLabs don't return timing | See Planned Approaches below | 2h | | 2 | Heuristic word timing | Equal division of audio duration | Remove after implementing timestamps | 0.5h | | 3 | languageCode not passed | UI→API flow broken | Trace and fix all paths | 0.5h | | 9 | Hinglish audio is English | pronunciationInstruction not applied | Fix pronoun/langCode flow | 1h |

✅ Planned Timestamp Approaches

Approach 1: ElevenLabs with_timestamps (Primary)

text → ElevenLabs TTS (with_timestamps: true)
     → audio + word-level timestamps (direct!)
     → No post-processing needed

Approach 2: Gemini + Deepgram (Fallback)

text → Gemini TTS → audioBase64
     → Deepgram STT (transcription)  
     → Forced alignment → word timestamps
     → Keep Gemini voices, add timing

Implementation Plan:

1. Primary: ElevenLabs with_timestamps (fastest, most accurate)
2. Fallback: Gemini + Deepgram (if ElevenLabs unavailable)
3. Remove heuristic /align-text completely after implementation

Ref: ISSUES.md

---

🟡 High/Medium Issues (Audio-Related)

| # | Issue | Root Cause | Fix | Effort | |---|-------|------------|-----|--------| | 7 | Caption timing drift | Uses heuristic timing | Link to TTS timestamps | 2h | | 8 | BGM not preloaded | Loads on first play | Preload on story load | 2h | | 14 | useAudioPlayer too large | 902 lines, mixed concerns | Split into 4-5 hooks | 4h | | 15 | No audio waveform | No visualization component | Add WaveformViz | 4h | | 18 | Limited BGM moods | Only 8 available | Expand to 15+ | 4h | | 19 | SFX caching basic | In-memory Map only | IndexedDB persistence | 2h |

Ref: ISSUES.md

---

✅ Recently Fixed (Audio-Related)

| Date | Issue | Fix | Files | |------|-------|-----|-------| | 2026-01-19 | Audio export corruption | Handle base64 in Remotion | remotion-factory/ | | 2026-01-18 | Caption sync drift | Added TTS_SPEED_FACTOR | CaptionTypography.tsx | | 2026-01-17 | SFX not playing | Fixed ElevenLabs soundPrompt | sfxService.ts |

---

📈 Improvement Roadmap

Short Term (1-2 weeks)

• [ ] Fix languageCode propagation in all API paths
• [ ] Preload BGM on story load (not first play)
• [ ] Persist SFX cache to IndexedDB
• [ ] Add audio waveform to player controls

Medium Term (1-2 months)

• [ ] Integrate Whisper/Deepgram for actual word timestamps
• [ ] Split useAudioPlayer into modular hooks
• [ ] Add streaming TTS for faster playback
• [ ] Expand BGM moods to 15+ options

Long Term (3+ months)

• [ ] Voice cloning for character consistency
• [ ] Manual timing adjustment UI
• [ ] Per-word animation editor
• [ ] Real-time audio effects processing

Architecture Goals

• [ ] Audio worklet for glitch-free playback
• [ ] Centralized audio state management (Zustand?)
• [ ] Offline-first audio caching (OPFS)
• [ ] Cross-tab audio synchronization

---

🔧 Environment Variables

# REQUIRED for TTS
GEMINI_API_KEY=...

# TTS Fallback + SFX
ELEVENLABS_API_KEY=...

# BGM Generation
BEATOVEN_API_KEY=...

# BGM/SFX Fallback
FREESOUND_API_KEY=...

---

🧪 Testing Checklist

After audio changes:

• [ ] English story plays with English audio
• [ ] Hindi story plays with Hindi pronunciation
• [ ] Hinglish story plays with Hindi pronunciation for Hindi words
• [ ] Word highlighting syncs with narration
• [ ] BGM starts when play button clicked
• [ ] BGM ducks during narration
• [ ] SFX triggers on power words
• [ ] Audio exports correctly in video

---

📁 Key Files Summary

| Category | Files | |----------|-------| | API Routes | generate-audio/, generate-bgm/, generate-sfx/, generate-audio-cues/, align-text/ | | Services | bgmService.ts, sfxService.ts, geminiService.ts | | Hooks | useAudioPlayer.ts, useAudioCues.ts | | Components | AudioControls.tsx, CaptionTypography.tsx, WordHighlighter.tsx | | Utils | wavEncoder.ts, audioUtils.ts, soundEffects.ts |

---

🌐 Multi-Language Audio Storage (V3)

V3 Architecture (Jan 2026)

Audio for translated languages is now stored in unified page.content[lang].audio:

page.content = {
  'hi': {
    text: 'Hindi translation...',
    audio: {
      base64: 'UklG...',     // Temp, deleted after upload
      url: 'https://...',     // Hostinger CDN
      duration: 12.5,
      status: 'ready'
    },
    wordTimings: [...],
    phrases: [...],
    kinetic: [...]
  }
}

Storage Flow

useKahaniAdapter (generation)
    → page.content[lang].audio.base64
    ↓
storageService.saveStoryToLibrary()
    → uploadAudioToHostinger()
    → page.content[lang].audio.url
    → delete base64 (save space)
    ↓
Firebase (only URL persisted)

Access via pageUtils

const audio = getPageAudio(page, langCode, originalLanguage);
// Returns { base64?, url?, duration?, status } or undefined

Edge Cases

• Original language: Falls back to page.audioBase64 / page.audioUrl
• Missing audio: Returns undefined, Monitor shows 🔴
• Auto-save: Triggers when page.content changes

---

Update this file when audio system changes