Merge branch 'main' of github.com:GetStream/Vision-Agents into rag_and_sip

Author: tschellenbach

README.md

@@ -106,7 +106,7 @@ Get a free API key from [Stream](https://getstream.io/). Developers receive **33
 
 | **Plugin Name** | **Description** | **Docs Link** |
 |-------------|-------------|-----------|
-| AWS Polly | TTS plugin using Amazon's cloud-based service with natural-sounding voices and neural engine support | [AWS Polly](https://visionagents.ai/integrations/aws-polly) |
+| AWS | AWS (Bedrock) integration with support for standard LLM (Qwen, Claude with vision), realtime with Nova 2 Sonic, and TTS with AWS Polly | [AWS](https://visionagents.ai/integrations/aws) |
 | Cartesia | TTS plugin for realistic voice synthesis in real-time voice applications | [Cartesia](https://visionagents.ai/integrations/cartesia) |
 | Decart | Real-time video restyling capabilities using generative AI models | [Decart](https://visionagents.ai/integrations/decart) |
 | Deepgram | STT plugin for fast, accurate real-time transcription with speaker diarization | [Deepgram](https://visionagents.ai/integrations/deepgram) |
@@ -225,7 +225,7 @@ While building the integrations, here are the limitations we've noticed (Dec 202
 * Longer videos can cause the AI to lose context. For instance if it's watching a soccer match it will get confused after 30 seconds
 * Most applications require a combination of small specialized models like Yolo/Roboflow/Moondream, API calls to get more context and larger models like gemini/openAI
 * Image size & FPS need to stay relatively low due to performance constraints
-* Video doesn’t trigger responses in realtime models. You always need to send audio/text to trigger a response. 
+* Video doesn’t trigger responses in realtime models. You always need to send audio/text to trigger a response.
 
 ## Star History
 

agents-core/vision_agents/core/__init__.py

@@ -3,6 +3,6 @@
 from vision_agents.core.agents import Agent
 
 from vision_agents.core.cli.cli_runner import cli
+from vision_agents.core.agents.agent_launcher import AgentLauncher
 
-
-__all__ = ["Agent", "User", "cli"]
+__all__ = ["Agent", "User", "cli", "AgentLauncher"]

agents-core/vision_agents/core/agents/agents.py

@@ -99,6 +99,8 @@ class Agent:
     - Small methods so its easy to subclass/change behaviour
     """
 
+    options: AgentOptions
+
     def __init__(
         self,
         # edge network for video & audio

agents-core/vision_agents/core/vad/silero.py

@@ -38,6 +38,7 @@ def __init__(self, model_path: str, reset_interval_seconds: float = 5.0):
 
     def predict_speech(self, pcm: PcmData):
         # convert from pcm to the right format for silero
+
         chunks = pcm.resample(16000, 1).to_float32().chunks(SILERO_CHUNK, pad_last=True)
         scores = [self._predict_speech(c.samples) for c in chunks]
         return max(scores)

dev.py

@@ -77,7 +77,7 @@ def format():
 def lint():
     """Run ruff linting (check only)."""
     click.echo("Running ruff lint...")
-    run("uv run ruff check .")
+    run("uv run ruff format --check .")
 
 
 @cli.command()
@@ -103,7 +103,8 @@ def check():
 
     # Run ruff
     click.echo("\n=== 1. Ruff Linting ===")
-    run("uv run ruff check . --fix")
+    run("uv run ruff format")
+    run("uv run ruff format --check .")
 
     # Run mypy on main package
     click.echo("\n=== 2. MyPy Type Checking ===")

examples/01_simple_agent_example/README.md

@@ -19,12 +19,17 @@ This example shows you how to build a basic video AI agent using [Vision Agents]
 
 ## Installation
 
-1. Install dependencies using uv:
+1. Go to the example's directory
+    ```bash
+    cd examples/01_simple_agent_example
+    ```
+
+2. Install dependencies using uv:
    ```bash
    uv sync
    ```
 
-2. Create a `.env` file with your API keys:
+3. Create a `.env` file with your API keys:
    ```
    OPENAI_API_KEY=your_openai_key
    ELEVENLABS_API_KEY=your_11labs_key

examples/01_simple_agent_example/simple_agent_example.py

@@ -3,8 +3,7 @@
 
 from dotenv import load_dotenv
 
-from vision_agents.core import User, Agent, cli
-from vision_agents.core.agents import AgentLauncher
+from vision_agents.core import User, Agent, cli, AgentLauncher
 from vision_agents.core.utils.examples import get_weather_by_location
 from vision_agents.plugins import deepgram, getstream, gemini, elevenlabs
 

examples/02_golf_coach_example/README.md

@@ -21,12 +21,17 @@ This approach combines a fast object detection model (YOLO) with a full realtime
 
 ## Installation
 
-1. Install dependencies using uv:
+1. Go to the example's directory
+    ```bash
+    cd examples/02_golf_coach_example
+    ```
+   
+2. Install dependencies using uv:
    ```bash
    uv sync
    ```
 
-2. Create a `.env` file with your API keys:
+3. Create a `.env` file with your API keys:
    ```
    GEMINI_API_KEY=your_gemini_key
    STREAM_API_KEY=your_stream_key

examples/other_examples/09_github_mcp_demo/README.md

@@ -38,19 +38,40 @@ export GITHUB_PAT=your_github_personal_access_token_here
 ```
 
 ## Running the Demo
+
+1. Go to the example's directory
+    ```bash
+    cd examples/other_examples/09_github_mcp_demo
+    ```
+
+2. Install dependencies using uv:
+   ```bash
+   uv sync
+   ```
+
+3. Run the agent
 ```bash
-cd examples/09_github_mcp_demo
 uv run python github_mcp_demo.py
 ```
 
-### Gemini Realtime Version (New)
+### Gemini Realtime Version
 ```bash
-cd examples/09_github_mcp_demo
+cd examples/other_examples/09_github_mcp_demo
 uv run python gemini_realtime_github_mcp_demo.py
 ```
 
 **Note**: The Gemini Realtime version requires `GOOGLE_API_KEY` in your `.env` file in addition to `GITHUB_PAT`.
 
+
+### OpenAI Realtime Version
+
+```bash
+cd examples/other_examples/09_github_mcp_demo
+uv run python openai_realtime_github_mcp_demo.py
+```
+
+**Note**: The OpenAI Realtime version requires `OPENAI_API_KEY` in your `.env` file in addition to `GITHUB_PAT`.
+
 ## What the Demo Does
 
 

examples/other_examples/openai_realtime_webrtc/README.md

@@ -1,4 +1,4 @@
-# OpenAI Speech-to-Speech (STS) Example
+# OpenAI Realtime WebRTC Example
 
 This example demonstrates how to use OpenAI's Realtime API for speech-to-speech conversation through WebRTC.
 
@@ -18,24 +18,27 @@ The OpenAI Realtime API enables real-time, bidirectional audio conversations wit
 3. Python 3.12 or higher
 
 ## Setup
-
-1. Install dependencies:
-```bash
-cd examples/07_openai_sts_example
-uv sync
-```
-
-2. Create a `.env` file with your credentials:
-```env
-OPENAI_API_KEY=your_openai_api_key
-STREAM_API_KEY=your_stream_api_key
-STREAM_API_SECRET=your_stream_api_secret
-```
+1. Go to the example's directory
+    ```bash
+    cd examples/other_examples/openai_realtime_webrtc
+    ```
+
+2. Install dependencies:
+    ```bash
+    uv sync
+    ```
+
+3. Create a `.env` file with your credentials:
+    ```env
+    OPENAI_API_KEY=your_openai_api_key
+    STREAM_API_KEY=your_stream_api_key
+    STREAM_API_SECRET=your_stream_api_secret
+    ```
 
 ## Running the Example
 
 ```bash
-uv run python openai_sts_example.py
+uv run python openai_realtime_example.py
 ```
 
 The script will: