Skip to content

fix(closes OPEN-9306): oCI tracer issue serializing GenericChatResponse #608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
viniciusdsmello merged 1 commit into from
Mar 5, 2026
Merged

fix(closes OPEN-9306): oCI tracer issue serializing GenericChatResponse #608

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
255 changes: 85 additions & 170 deletions examples/tracing/oci/oci_genai_tracing.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,34 +8,11 @@
}
},
"source": [
"# Oracle OCI Generative AI Tracing with Openlayer\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openlayer-ai/openlayer-python/blob/main/examples/tracing/oci/oci_genai_tracing.ipynb)\n",
"\n",
"This notebook demonstrates how to use Openlayer tracing with Oracle Cloud Infrastructure (OCI) Generative AI service.\n",
"# <a id=\"top\">Oracle OCI Generative AI Tracing with Openlayer</a>\n",
"\n",
"## Setup\n",
"\n",
"Before running this notebook, ensure you have:\n",
"1. An OCI account with access to Generative AI service\n",
"2. OCI CLI configured or OCI config file set up\n",
"3. An Openlayer account with API key and inference pipeline ID\n",
"4. The required packages installed:\n",
" - `pip install oci`\n",
" - `pip install openlayer`\n",
"\n",
"## Configuration\n",
"\n",
"### Openlayer Setup\n",
"Set these environment variables before running:\n",
"```bash\n",
"export OPENLAYER_API_KEY=\"your-api-key\"\n",
"export OPENLAYER_INFERENCE_PIPELINE_ID=\"your-pipeline-id\"\n",
"```\n",
"\n",
"### OCI Setup\n",
"Make sure your OCI configuration is properly set up. You can either:\n",
"- Use the default OCI config file (`~/.oci/config`)\n",
"- Set up environment variables\n",
"- Use instance principal authentication (when running on OCI compute)\n"
"This notebook demonstrates how to use Openlayer tracing with Oracle Cloud Infrastructure (OCI) Generative AI service.\n"
]
},
{
Expand All @@ -44,44 +21,14 @@
"metadata": {},
"outputs": [],
"source": [
"# Install required packages (uncomment if needed)\n",
"# !pip install oci openlayer\n",
"\n",
"# Set up Openlayer environment variables\n",
"import os\n",
"\n",
"# Configure Openlayer API credentials\n",
"os.environ[\"OPENLAYER_API_KEY\"] = \"your-openlayer-api-key-here\"\n",
"os.environ[\"OPENLAYER_INFERENCE_PIPELINE_ID\"] = \"your-inference-pipeline-id-here\"\n",
"\n",
"# NOTE: Remember to set your actual Openlayer API key and inference pipeline ID!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import oci\n",
"from oci.generative_ai_inference import GenerativeAiInferenceClient\n",
"from oci.generative_ai_inference.models import Message, ChatDetails, GenericChatRequest\n",
"\n",
"# Import the Openlayer tracer\n",
"from openlayer.lib.integrations import trace_oci_genai"
"!pip install openlayer oci"
]
},
{
"cell_type": "markdown",
"metadata": {
"vscode": {
"languageId": "raw"
}
},
"metadata": {},
"source": [
"## Initialize OCI Client\n",
"\n",
"Set up the OCI Generative AI client with your configuration.\n"
"## 1. Set the environment variables"
]
},
{
Expand All @@ -90,36 +37,18 @@
"metadata": {},
"outputs": [],
"source": [
"# Configuration - Update these values for your environment\n",
"COMPARTMENT_ID = \"your-compartment-ocid-here\" # Replace with your compartment OCID\n",
"ENDPOINT = \"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\" # Replace with your region's endpoint\n",
"\n",
"# Load OCI configuration\n",
"config = oci.config.from_file() # Uses default config file location\n",
"# Alternatively, you can specify a custom config file:\n",
"# config = oci.config.from_file(\"~/.oci/config\", \"DEFAULT\")\n",
"import os\n",
"\n",
"# Create the OCI Generative AI client\n",
"client = GenerativeAiInferenceClient(config=config, service_endpoint=ENDPOINT)\n"
"# Configure Openlayer API credentials\n",
"os.environ[\"OPENLAYER_API_KEY\"] = \"YOUR_OPENLAYER_API_KEY_HERE\"\n",
"os.environ[\"OPENLAYER_INFERENCE_PIPELINE_ID\"] = \"YOUR_INFERENCE_PIPELINE_ID_HERE\""
]
},
{
"cell_type": "markdown",
"metadata": {
"vscode": {
"languageId": "raw"
}
},
"metadata": {},
"source": [
"## Apply Openlayer Tracing\n",
"\n",
"Wrap the OCI client with Openlayer tracing to automatically capture all interactions.\n",
"\n",
"The `trace_oci_genai()` function accepts an optional `estimate_tokens` parameter:\n",
"- `estimate_tokens=True` (default): Estimates token counts when not provided by OCI response\n",
"- `estimate_tokens=False`: Returns None for token fields when not available in the response\n",
"\n",
"OCI responses can be either CohereChatResponse or GenericChatResponse, both containing usage information when available.\n"
"## 2. Wrap the OCI Generative AI client with the `trace_oci_genai` function"
]
},
{
Expand All @@ -128,25 +57,28 @@
"metadata": {},
"outputs": [],
"source": [
"# Apply Openlayer tracing to the OCI client\n",
"# With token estimation enabled (default)\n",
"traced_client = trace_oci_genai(client, estimate_tokens=True)\n",
"import oci\n",
"from oci.generative_ai_inference import GenerativeAiInferenceClient\n",
"from oci.generative_ai_inference.models import (\n",
" Message,\n",
" ChatDetails,\n",
" TextContent,\n",
" BaseChatRequest,\n",
" GenericChatRequest,\n",
" OnDemandServingMode,\n",
")\n",
"\n",
"# Alternative: Disable token estimation to get None values when tokens are not available\n",
"# traced_client = trace_oci_genai(client, estimate_tokens=False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"vscode": {
"languageId": "raw"
}
},
"source": [
"## Example 1: Non-Streaming Chat Completion\n",
"# Point to your real files\n",
"OCI_CONFIG_PATH = \"YOUR_OCI_CONFIG_PATH_HERE\"\n",
"OCI_PROFILE = \"DEFAULT\"\n",
"\n",
"ENDPOINT = \"YOUR_ENDPOINT_HERE\"\n",
"COMPARTMENT_ID = \"YOUR_COMPARTMENT_OCID_HERE\"\n",
"MODEL_ID = \"YOUR_MODEL_OCID_HERE\"\n",
"\n",
"Simple chat completion without streaming.\n"
"\n",
"config = oci.config.from_file(OCI_CONFIG_PATH, OCI_PROFILE)\n",
"oci.config.validate_config(config)\n"
]
},
{
Expand All @@ -155,33 +87,24 @@
"metadata": {},
"outputs": [],
"source": [
"# Create a chat request\n",
"chat_request = GenericChatRequest(\n",
" messages=[Message(role=\"user\", content=\"Hello! Can you explain what Oracle Cloud Infrastructure is?\")],\n",
" model_id=\"cohere.command-r-plus\",\n",
" max_tokens=200,\n",
" temperature=0.7,\n",
" is_stream=False, # Non-streaming\n",
")\n",
"\n",
"chat_details = ChatDetails(compartment_id=COMPARTMENT_ID, chat_request=chat_request)\n",
"\n",
"# Make the request - the tracer will automatically capture it\n",
"response = traced_client.chat(chat_details)\n",
"response"
"# Import the Openlayer tracer\n",
"from openlayer.lib.integrations import trace_oci_genai\n",
"\n",
"client = trace_oci_genai(GenerativeAiInferenceClient(\n",
" config=config,\n",
" service_endpoint=ENDPOINT,\n",
" retry_strategy=oci.retry.NoneRetryStrategy(),\n",
" timeout=(10, 120),\n",
"), estimate_tokens=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"vscode": {
"languageId": "raw"
}
},
"metadata": {},
"source": [
"## Example 2: Streaming Chat Completion\n",
"## 3. Use the traced OCI Generative AI client normally\n",
"\n",
"Chat completion with streaming enabled to see tokens as they're generated.\n"
"That's it! Now you can continue using the traced OpenAI client normally. The data is automatically published to Openlayer and you can start creating tests around it!"
]
},
{
Expand All @@ -190,46 +113,40 @@
"metadata": {},
"outputs": [],
"source": [
"# Create a streaming chat request\n",
"streaming_chat_request = GenericChatRequest(\n",
"chat_request = GenericChatRequest(\n",
" api_format=BaseChatRequest.API_FORMAT_GENERIC,\n",
" messages=[\n",
" Message(role=\"system\", content=\"You are a helpful AI assistant that provides concise, informative answers.\"),\n",
" Message(role=\"user\", content=\"Tell me a short story about cloud computing and AI working together.\"),\n",
" Message(\n",
" role=\"USER\",\n",
" content=[TextContent(text=\"What is OCI in one sentence?\")]\n",
" )\n",
" ],\n",
" model_id=\"meta.llama-3.1-70b-instruct\",\n",
" max_tokens=300,\n",
" temperature=0.8,\n",
" is_stream=True, # Enable streaming\n",
" max_tokens=100,\n",
" temperature=0.2,\n",
" is_stream=False,\n",
")\n",
"\n",
"streaming_chat_details = ChatDetails(compartment_id=COMPARTMENT_ID, chat_request=streaming_chat_request)\n",
"\n",
"# Make the streaming request\n",
"streaming_response = traced_client.chat(streaming_chat_details)\n",
"\n",
"# Process the streaming response\n",
"full_content = \"\"\n",
"for chunk in streaming_response:\n",
" if hasattr(chunk, \"data\") and hastr(chunk.data, \"choices\"):\n",
" if chunk.data.choices and hasattr(chunk.data.choices[0], \"delta\"):\n",
" delta = chunk.data.choices[0].delta\n",
" if hasattr(delta, \"content\") and delta.content:\n",
" full_content += delta.content\n",
"\n",
"full_content"
"chat_details = ChatDetails(\n",
" serving_mode=OnDemandServingMode(model_id=MODEL_ID),\n",
" chat_request=chat_request,\n",
" compartment_id=COMPARTMENT_ID,\n",
")\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"vscode": {
"languageId": "raw"
}
},
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"## Example 3: Custom Parameters and Error Handling\n",
"resp = client.chat(chat_details)\n",
"\n",
"Demonstrate various model parameters and how tracing works with different scenarios.\n"
"answer = \"\"\n",
"choice = resp.data.chat_response.choices[0]\n",
"for part in choice.message.content:\n",
" if hasattr(part, \"text\"):\n",
" answer += part.text"
]
},
{
Expand All @@ -238,29 +155,27 @@
"metadata": {},
"outputs": [],
"source": [
"# Advanced parameters example\n",
"advanced_request = GenericChatRequest(\n",
" messages=[Message(role=\"user\", content=\"Write a creative haiku about artificial intelligence.\")],\n",
" model_id=\"meta.llama-3.1-70b-instruct\",\n",
" max_tokens=100,\n",
" temperature=0.9, # High creativity\n",
" top_p=0.8,\n",
" frequency_penalty=0.2, # Reduce repetition\n",
" presence_penalty=0.1,\n",
" stop=[\"\\n\\n\"], # Stop at double newline\n",
" is_stream=False,\n",
")\n",
"\n",
"advanced_details = ChatDetails(compartment_id=COMPARTMENT_ID, chat_request=advanced_request)\n",
"\n",
"response = traced_client.chat(advanced_details)\n",
"response"
"answer"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "oracle",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python"
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.12"
}
},
"nbformat": 4,
Expand Down
14 changes: 12 additions & 2 deletions src/openlayer/lib/integrations/oci_tracer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
from oci.generative_ai_inference import GenerativeAiInferenceClient

from ..tracing import tracer
from ..utils import safe_serialize_raw_output

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -126,8 +127,16 @@ def handle_streaming_chat(
Iterator[Any]
A generator that yields the chunks of the completion.
"""
# OCI SDK versions may expose stream events differently; support both wrapped
# responses (`response.data.events()`) and direct iterators.
chunks = response
if hasattr(response, "data") and hasattr(response.data, "events") and callable(response.data.events):
chunks = response.data.events()
elif hasattr(response, "events") and callable(response.events):
chunks = response.events()

return stream_chunks(
chunks=response.data.events(),
chunks=chunks,
chat_details=chat_details,
kwargs=kwargs,
start_time=start_time,
Expand Down Expand Up @@ -202,6 +211,7 @@ def stream_chunks(

except Exception as e:
logger.error("Failed yield chunk. %s", e)
raise
finally:
# Try to add step to the trace
try:
Expand Down Expand Up @@ -306,7 +316,7 @@ def handle_non_streaming_chat(
completion_tokens=tokens_info.get("output_tokens"),
model=model_id,
model_parameters=get_model_parameters(chat_details),
raw_output=response.data.__dict__ if hasattr(response, "data") else response.__dict__,
raw_output=safe_serialize_raw_output(response),
id=None,
metadata=additional_metadata,
)
Expand Down
Loading