​

Speech-to-Speech AI Model Benchmarking Framework

​

Overview

​

Open-Source Benchmarks

​

Framework Architecture

├── 🔧 Boot Performance (Cold Start)
├── ⏱️ Latency (Sustained Performance)
├── 💻 Code Generation (HumanEval)
├── 🛠️ Function Calling (BFCL)
├── 😊 Emotional Intelligence
├── 🌍 Multilingual Naturalness
├── 🎭 Expressiveness & Prosody
├── 🤝 Ethic Reasoning (MM-NIAH)
└── 🎯 Helpful/Honest/Harmless (HLE)

​

Evaluation Pipeline

User Query → Text-to-Speech → Speech-to-Speech Model → Speech-to-Text → Evaluation Framework
     ↓              ↓                  ↓                      ↓              ↓
   Prompt      Vispark TTS       Aivoco Model            Vispark STT    Real Datasets

​

Vispark Models Integration

​

Text-to-Speech (TTS)

• Model: Advanced neural TTS with emotional expression
• Languages: 250+ languages including all major Indian languages
• Voices: Multiple voice options (male/female) with natural intonation
• Quality: 24kHz high-fidelity audio output
• Features: Emotion control, speed adjustment, pronunciation correction

​

Speech-to-Text (STT)

• Model: State-of-the-art automatic speech recognition
• Accuracy: 99%+ accuracy across supported languages
• Languages: 250+ languages with regional accent support
• Features: Real-time processing, noise cancellation, speaker diarization
• Output: Precise text transcription with punctuation

​

Vision (Multimodal AI)

• Model: Advanced multimodal understanding
• Capabilities: Text, image, audio, and video analysis
• Context Window: Up to 1 million tokens
• Features: Emotion analysis, content understanding, real-time streaming
• Use Cases: Judge for emotions, naturalness, and expressiveness benchmarks

​

Key Advantages

• SOTA Performance: Industry-leading accuracy and naturalness
• Real-time Processing: Low latency for interactive applications
• Multilingual Support: Native support for Indian regional languages
• API Integration: Simple REST API with comprehensive documentation
• Enterprise Ready: Production-grade reliability and security

​

Benchmark Results Summary

​

🏆 Model Performance Rankings (As of August 30, 2025)

​

Detailed Benchmark Results

​

Boot Time Performance

​

Latency Performance

​

Code Generation (HumanEval)

​

Function Calling (BFCL)

​

Emotional Intelligence

​

Multilingual Naturalness

​

Expressiveness & Prosody

​

Ethical Reasoning (MM-NIAH)

​

Helpful/Honest/Harmless (HLE)

​

Individual Benchmark Details

​

1. Boot Benchmark (boot/)

1. Initialize speech-to-speech connection
2. Send 100 sequential audio requests
3. Measure time to first response for each
4. Calculate average boot time

​

2. Latency Benchmark (latency/)

1. Start 30-minute conversation sessions
2. Use Vispark Vision to generate realistic scenarios
3. Send contextual audio messages throughout
4. Measure response latency for each interaction

​

3. HumanEval Benchmark (humaneval/)

1. Load HumanEval programming problems
2. Convert problems to speech using TTS
3. Send through speech-to-speech model
4. Convert responses back to text
5. Evaluate code correctness with official HumanEval framework

​

4. BFCL Benchmark (bfcl/)

1. Load BFCL function calling test cases
2. Convert function requests to speech
3. Process through speech-to-speech model
4. Parse function calls from voice responses
5. Evaluate accuracy against expected functions/parameters

​

5. Emotions Benchmark (emotions/)

1. Test 8 core emotions (joy, sadness, anger, fear, surprise, disgust, trust, anticipation)
2. Generate emotional scenarios using Vispark Vision
3. Send emotional prompts through speech-to-speech
4. Use multimodal AI to judge emotional authenticity

​

6. Naturalness Benchmark (naturalness/)

1. Test 15 languages (5 European + 10 Indian regional)
2. Generate conversations in native scripts
3. Process through speech-to-speech pipeline
4. Use multimodal AI to judge naturalness, fluency, and pronunciation

​

7. Expressiveness Benchmark (expressiveness/)

1. Test 8 expressive scenarios requiring different vocal styles
2. Send prompts requiring emotional expression
3. Use multimodal AI to analyze prosody, intonation, and pace
4. Score vocal expressiveness and variety

​

8. Ethic Benchmark (ethic/)

1. Load MM-NIAH ethical scenarios
2. Convert ethical dilemmas to speech
3. Process through speech-to-speech model
4. Evaluate responses using official MM-NIAH evaluation
5. Measure ethical reasoning quality

​

9. HLE Benchmark (hle/)

1. Load HLE test scenarios
2. Convert HLE prompts to speech
3. Process through speech-to-speech model
4. Evaluate responses using HLE framework
5. Score helpfulness, honesty, and harmlessness

​

Quick Start Guide

​

Prerequisites

pip install -r requirements.txt

​

API Key Setup

​

Vispark API Keys

1. Visit https://lab.vispark.in
2. Sign up for an account
3. Navigate to API Keys section
4. Generate your API key for:
   • Text-to-Speech (TTS) - Required for all benchmarks
   • Speech-to-Text (STT) - Required for most benchmarks
   • Vision (Multimodal) - Required for emotions/expressiveness/naturalness

​

Aivoco API Keys

1. Visit the Aivoco platform
2. Complete registration process
3. Generate API key for Speech-to-Speech model access

​

Environment Setup

# Create .env file with your API keys
cp .env.example .env

# Edit .env with your actual API keys:
# AIVOCO_API_KEY=your_aivoco_key
# VISPARK_API_KEY=your_vispark_key

​

Running Individual Benchmarks

# Run any benchmark
cd [benchmark_name]
python main.py

# Examples:
cd boot && python main.py
cd humaneval && python main.py
cd emotions && python main.py

​

Running All Benchmarks

# Run benchmarks sequentially
for dir in */; do
    if [ -f "$dir/main.py" ]; then
        echo "Running $dir"
        cd "$dir"
        python main.py
        cd ..
    fi
done

​

Performance Insights

​

Rose (Aivoco v1)

• Strengths: Best latency, excellent code generation, superior ethics
• Weaknesses: Higher boot time (acceptable for production)
• Best For: Production applications requiring reliability and ethics

​

OpenAI GPT Realtime

• Strengths: Fastest boot time, excellent expressiveness, good emotions
• Weaknesses: Moderate performance in some technical tasks
• Best For: Interactive applications needing quick responses

​

Google Gemini 2.5

• Strengths: Good expressiveness, solid emotional intelligence
• Weaknesses: Higher latency, moderate multilingual support
• Best For: General-purpose applications with balanced requirements

​

Sesame CSM 1

• Strengths: Good expressiveness and emotional intelligence
• Weaknesses: Poor multilingual support, limited technical capabilities
• Best For: Focused use cases with strong emotional requirements

​

Technical Specifications

​

Models Tested

• Rose: Aivoco v1 (Speech-to-Speech optimized)
• Google: Gemini 2.5 Native Live (latest)
• OpenAI: GPT Realtime (latest)
• Sesame: CSM 1 (latest)

​

Testing Environment

• Network: Same network for all tests
• Device: Consistent hardware across all benchmarks
• Date: August 30, 2025
• Sessions: 100 per benchmark (except where noted)

​

Evaluation Frameworks

• HumanEval: Official OpenAI code generation benchmark
• BFCL: Berkeley Function Calling Leaderboard
• MM-NIAH: Multimodal Non-Intrusive AI Helpfulness
• HLE: Helpful, Honest, and Harmless evaluation

​

Vispark Models Used

• TTS Model: /model/audio/text_to_speech - Neural TTS with emotion control
• STT Model: /model/audio/speech_to_text - Advanced ASR with 95%+ accuracy
• Vision Model: /model/text/vision - Multimodal AI with 1M token context
• Base URL: https://api.lab.vispark.in
• Authentication: X-API-Key header required
• Documentation: Available at https://lab.vispark.in

​

Vispark API Endpoints Used in Benchmarks

# Text-to-Speech
POST https://api.lab.vispark.in/model/audio/text_to_speech
Headers: X-API-Key: your_key
Body: {"size": "small/large", "text": "your_text", "voice": "boy/girl"}

# Speech-to-Text
POST https://api.lab.vispark.in/model/audio/speech_to_text
Headers: X-API-Key: your_key
Body: {"audio": "base64_encoded_audio"}

# Vision (Multimodal AI)
POST https://api.lab.vispark.in/model/text/vision
Headers: X-API-Key: your_key
Body: {"size": "small/medium/large", "content": [...], "system_message": "..."}

​

Support & Contributing

• Documentation: Each benchmark folder contains detailed README files
• Issues: Report bugs or request features
• Contributing: Pull requests welcome for new benchmarks or improvements

​

License & Attribution