-
Notifications
You must be signed in to change notification settings - Fork 164
Refactor guardrail metrics #1984
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
geoand
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💪🏽
Introduces GuardrailMetricsObserver to collect guardrail metrics with enhanced tags including aiservice name, operation, guardrail class, type (input/output), and outcome (success/failure/reprompt). Deprecates the previous annotation-based metrics collection in GuardrailObservabilityProcessor.
e4e57ff to
b79d501
Compare
|
I've reorganized the commits. |
Status for workflow
|
| Status | Name | Step | Failures | Logs | Raw logs |
|---|---|---|---|---|---|
| ❌ | JVM tests - agentic - Java 17 | Run tests of agentic with JDK 17 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - agentic - Java 21 | Run tests of agentic with JDK 21 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - agentic - Java 24 | Run tests of agentic with JDK 24 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - core - Java 17 | Run tests of core with JDK 17 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - core - Java 21 | Run tests of core with JDK 21 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - core - Java 24 | Run tests of core with JDK 24 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - docs - Java 17 | Run tests of docs with JDK 17 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - docs - Java 21 | Run tests of docs with JDK 21 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - docs - Java 24 | Run tests of docs with JDK 24 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - embedding-stores - Java 17 | Run tests of embedding-stores with JDK 17 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - embedding-stores - Java 21 | Run tests of embedding-stores with JDK 21 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - embedding-stores - Java 24 | Run tests of embedding-stores with JDK 24 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - integration-tests - Java 17 | Run tests of integration-tests with JDK 17 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - integration-tests - Java 21 | Run tests of integration-tests with JDK 21 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - integration-tests - Java 24 | Run tests of integration-tests with JDK 24 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - mcp - Java 17 | Run tests of mcp with JDK 17 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - mcp - Java 21 | Run tests of mcp with JDK 21 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - mcp - Java 24 | Run tests of mcp with JDK 24 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - memory-stores - Java 17 | Run tests of memory-stores with JDK 17 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - memory-stores - Java 21 | Run tests of memory-stores with JDK 21 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - memory-stores - Java 24 | Run tests of memory-stores with JDK 24 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - model-providers - Java 17 | Run tests of model-providers with JDK 17 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - model-providers - Java 21 | Run tests of model-providers with JDK 21 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - model-providers - Java 24 | Run tests of model-providers with JDK 24 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - rag - Java 17 | Run tests of rag with JDK 17 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - rag - Java 21 | Run tests of rag with JDK 21 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - rag - Java 24 | Run tests of rag with JDK 24 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - tools - Java 17 | Run tests of tools with JDK 17 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - tools - Java 21 | Run tests of tools with JDK 21 |
Failures | Logs | Raw logs |
| ❌ | JVM tests - tools - Java 24 | Run tests of tools with JDK 24 |
Failures | Logs | Raw logs |
| ❌ | Native tests - azure-openai | Run integration tests azure-openai |
Logs | Raw logs | |
| ❌ | Native tests - devui | Run integration tests devui |
Failures | Logs | Raw logs |
| ❌ | Native tests - devui-multiple-embedding-models | Run integration tests devui-multiple-embedding-models |
Failures | Logs | Raw logs |
| ❌ | Native tests - hugging-face | Run integration tests hugging-face |
Logs | Raw logs | |
| ❌ | Native tests - mcp | Run integration tests mcp |
Logs | Raw logs | |
| ❌ | Native tests - multiple-providers | Run integration tests multiple-providers |
Failures | Logs | Raw logs |
| ❌ | Native tests - ollama | Run integration tests ollama |
Logs | Raw logs | |
| ❌ | Native tests - rag | Run integration tests rag |
Logs | Raw logs | |
| ❌ | Native tests - rag-pgvector | Run integration tests rag-pgvector |
Logs | Raw logs | |
| ❌ | Native tests - rag-pgvector-flyway | Run integration tests rag-pgvector-flyway |
Logs | Raw logs | |
| ❌ | Native tests - secure-mcp | Run integration tests secure-mcp |
Logs | Raw logs | |
| ❌ | Native tests - simple-ollama | Run integration tests simple-ollama |
Logs | Raw logs | |
| ❌ | Native tests - tools | Run integration tests tools |
Logs | Raw logs | |
| ❌ | In process embedding model tests - embed-all-minilm-l6-v2 | Run integration tests embed-all-minilm-l6-v2 |
Logs | Raw logs | |
| ❌ | In process embedding model tests - embed-all-minilm-l6-v2-q | Run integration tests embed-all-minilm-l6-v2-q |
Logs | Raw logs | |
| ❌ | In process embedding model tests - embed-bge-small-en | Run integration tests embed-bge-small-en |
Logs | Raw logs | |
| ❌ | In process embedding model tests - embed-bge-small-en-q | Run integration tests embed-bge-small-en-q |
Logs | Raw logs | |
| ❌ | In process embedding model tests - embed-bge-small-en-v15 | Run integration tests embed-bge-small-en-v15 |
Logs | Raw logs | |
| ❌ | In process embedding model tests - embed-bge-small-en-v15-and-ollama | Run integration tests embed-bge-small-en-v15-and-ollama |
Failures | Logs | Raw logs |
| ❌ | In process embedding model tests - embed-bge-small-en-v15-q | Run integration tests embed-bge-small-en-v15-q |
Logs | Raw logs | |
| ❌ | In process embedding model tests - embed-e5-small-v2 | Run integration tests embed-e5-small-v2 |
Logs | Raw logs | |
| ❌ | In process embedding model tests - embed-e5-small-v2-q | Run integration tests embed-e5-small-v2-q |
Logs | Raw logs |
Full information is available in the Build summary check run.
Failures
⚙️ JVM tests - agentic - Java 17 #
📦 agentic/deployment
❌ io.quarkiverse.langchain4j.agentic.deployment.AgentBeanSmokeTest.
❌ io.quarkiverse.langchain4j.agentic.deployment.WorkflowTest.
❌ io.quarkiverse.langchain4j.agentic.deployment.validation.SupervisorAgentShouldWorkTest.
⚙️ JVM tests - agentic - Java 21 #
📦 agentic/deployment
❌ io.quarkiverse.langchain4j.agentic.deployment.AgentBeanSmokeTest.
❌ io.quarkiverse.langchain4j.agentic.deployment.WorkflowTest.
❌ io.quarkiverse.langchain4j.agentic.deployment.validation.SupervisorAgentShouldWorkTest.
⚙️ JVM tests - agentic - Java 24 #
📦 agentic/deployment
❌ io.quarkiverse.langchain4j.agentic.deployment.AgentBeanSmokeTest.
❌ io.quarkiverse.langchain4j.agentic.deployment.WorkflowTest.
❌ io.quarkiverse.langchain4j.agentic.deployment.validation.SupervisorAgentShouldWorkTest.
⚙️ JVM tests - core - Java 17 #
📦 core/opentelemetry-tests
❌ io.quarkiverse.langchain4j.opentelemetry.test.ListenersProcessorOnlySpanChatModelListenerTest.
❌ io.quarkiverse.langchain4j.opentelemetry.test.ListenersProcessorSingleChatModelSpanContributorTest.
❌ io.quarkiverse.langchain4j.opentelemetry.test.ListenersProcessorTwoChatModelSpanContributorsTest.
⚙️ JVM tests - core - Java 21 #
📦 core/opentelemetry-tests
❌ io.quarkiverse.langchain4j.opentelemetry.test.ListenersProcessorOnlySpanChatModelListenerTest.
❌ io.quarkiverse.langchain4j.opentelemetry.test.ListenersProcessorSingleChatModelSpanContributorTest.
❌ io.quarkiverse.langchain4j.opentelemetry.test.ListenersProcessorTwoChatModelSpanContributorsTest.
⚙️ JVM tests - core - Java 24 #
📦 core/opentelemetry-tests
❌ io.quarkiverse.langchain4j.opentelemetry.test.ListenersProcessorOnlySpanChatModelListenerTest.
❌ io.quarkiverse.langchain4j.opentelemetry.test.ListenersProcessorSingleChatModelSpanContributorTest.
❌ io.quarkiverse.langchain4j.opentelemetry.test.ListenersProcessorTwoChatModelSpanContributorsTest.
⚙️ JVM tests - docs - Java 17 #
📦 docs
❌ io.quarkiverse.langchain4j.docs.it.AntoraTest.antoraSite
⚙️ JVM tests - docs - Java 21 #
📦 docs
❌ io.quarkiverse.langchain4j.docs.it.AntoraTest.antoraSite
⚙️ JVM tests - docs - Java 24 #
📦 docs
❌ io.quarkiverse.langchain4j.docs.it.AntoraTest.antoraSite
⚙️ JVM tests - embedding-stores - Java 17 #
📦 embedding-stores/chroma/deployment
❌ io.quarkiverse.langchain4j.chroma.deployment.ChromaEmbeddingStoreV1Test.
❌ io.quarkiverse.langchain4j.chroma.deployment.ChromaEmbeddingStoreV2Test.
⚙️ JVM tests - embedding-stores - Java 21 #
📦 embedding-stores/chroma/deployment
❌ io.quarkiverse.langchain4j.chroma.deployment.ChromaEmbeddingStoreV1Test.
❌ io.quarkiverse.langchain4j.chroma.deployment.ChromaEmbeddingStoreV2Test.
⚙️ JVM tests - embedding-stores - Java 24 #
📦 embedding-stores/chroma/deployment
❌ io.quarkiverse.langchain4j.chroma.deployment.ChromaEmbeddingStoreV1Test.
❌ io.quarkiverse.langchain4j.chroma.deployment.ChromaEmbeddingStoreV2Test.
⚙️ JVM tests - integration-tests - Java 17 #
📦 integration-tests/openai
❌ org.acme.example.openai.aiservices.AssistantResourceWithGuardrailsAndObservabilityTest.guardrailMetricsAvailable line 49
⚙️ JVM tests - integration-tests - Java 21 #
📦 integration-tests/openai
❌ org.acme.example.openai.aiservices.AssistantResourceWithGuardrailsAndObservabilityTest.guardrailMetricsAvailable line 49
⚙️ JVM tests - integration-tests - Java 24 #
📦 integration-tests/openai
❌ org.acme.example.openai.aiservices.AssistantResourceWithGuardrailsAndObservabilityTest.guardrailMetricsAvailable line 49
⚙️ JVM tests - mcp - Java 17 #
📦 mcp/deployment
❌ io.quarkiverse.langchain4j.mcp.test.AgentMcpClientTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpAccessTokenTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpClaudeConfigTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpClientAndToolProviderCDITest.
❌ io.quarkiverse.langchain4j.mcp.test.McpHealthCheckTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpOverHttpTransportTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpWithResourcesAsToolsTest.
❌ io.quarkiverse.langchain4j.mcp.test.MultipleMcpClientsTest.
❌ io.quarkiverse.langchain4j.mcp.test.NoAutomaticToolProviderTest.
❌ io.quarkiverse.langchain4j.mcp.test.registryclient.McpRegistryClientTest.
⚙️ JVM tests - mcp - Java 21 #
📦 mcp/deployment
❌ io.quarkiverse.langchain4j.mcp.test.AgentMcpClientTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpAccessTokenTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpClaudeConfigTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpClientAndToolProviderCDITest.
❌ io.quarkiverse.langchain4j.mcp.test.McpHealthCheckTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpOverHttpTransportTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpWithResourcesAsToolsTest.
❌ io.quarkiverse.langchain4j.mcp.test.MultipleMcpClientsTest.
❌ io.quarkiverse.langchain4j.mcp.test.NoAutomaticToolProviderTest.
❌ io.quarkiverse.langchain4j.mcp.test.registryclient.McpRegistryClientTest.
⚙️ JVM tests - mcp - Java 24 #
📦 mcp/deployment
❌ io.quarkiverse.langchain4j.mcp.test.AgentMcpClientTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpAccessTokenTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpClaudeConfigTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpClientAndToolProviderCDITest.
❌ io.quarkiverse.langchain4j.mcp.test.McpHealthCheckTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpOverHttpTransportTest.
❌ io.quarkiverse.langchain4j.mcp.test.McpWithResourcesAsToolsTest.
❌ io.quarkiverse.langchain4j.mcp.test.MultipleMcpClientsTest.
❌ io.quarkiverse.langchain4j.mcp.test.NoAutomaticToolProviderTest.
❌ io.quarkiverse.langchain4j.mcp.test.registryclient.McpRegistryClientTest.
⚙️ JVM tests - memory-stores - Java 17 #
📦 memory-stores/memory-store-redis/deployment
❌ io.quarkiverse.langchain4j.memorystore.redis.test.RedisMemoryToolBoxTest.
⚙️ JVM tests - memory-stores - Java 21 #
📦 memory-stores/memory-store-redis/deployment
❌ io.quarkiverse.langchain4j.memorystore.redis.test.RedisMemoryToolBoxTest.
⚙️ JVM tests - memory-stores - Java 24 #
📦 memory-stores/memory-store-redis/deployment
❌ io.quarkiverse.langchain4j.memorystore.redis.test.RedisMemoryToolBoxTest.
⚙️ JVM tests - model-providers - Java 17 #
📦 model-providers/anthropic/deployment
❌ io.quarkiverse.langchain4j.anthropic.deployment.AnthropicChatLanguageModelSmokeTest.
❌ io.quarkiverse.langchain4j.anthropic.deployment.AnthropicDisabledLanguageModelSmokeTest.
❌ io.quarkiverse.langchain4j.anthropic.deployment.AnthropicStreamingChatLanguageModelSmokeTest.
⚙️ JVM tests - model-providers - Java 21 #
📦 model-providers/anthropic/deployment
❌ io.quarkiverse.langchain4j.anthropic.deployment.AnthropicChatLanguageModelSmokeTest.
❌ io.quarkiverse.langchain4j.anthropic.deployment.AnthropicDisabledLanguageModelSmokeTest.
❌ io.quarkiverse.langchain4j.anthropic.deployment.AnthropicStreamingChatLanguageModelSmokeTest.
⚙️ JVM tests - model-providers - Java 24 #
📦 model-providers/anthropic/deployment
❌ io.quarkiverse.langchain4j.anthropic.deployment.AnthropicChatLanguageModelSmokeTest.
❌ io.quarkiverse.langchain4j.anthropic.deployment.AnthropicDisabledLanguageModelSmokeTest.
❌ io.quarkiverse.langchain4j.anthropic.deployment.AnthropicStreamingChatLanguageModelSmokeTest.
⚙️ JVM tests - rag - Java 17 #
📦 rag/easy-rag/deployment
❌ io.quarkiverse.langchain4j.test.EasyRagManualTriggerOnClasspathTest.
❌ io.quarkiverse.langchain4j.test.EasyRagManualTriggerTest.
❌ io.quarkiverse.langchain4j.test.EasyRagNotRecursiveOnClasspathTest. line 50
❌ io.quarkiverse.langchain4j.test.EasyRagNotRecursiveTest. line 46
❌ io.quarkiverse.langchain4j.test.EasyRagOnClasspathTest. line 60
❌ io.quarkiverse.langchain4j.test.EasyRagPathMatcherOnClasspathTest. line 51
❌ io.quarkiverse.langchain4j.test.EasyRagPathMatcherTest. line 45
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsAlreadyExistTest. line 53
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsDontAlreadyExistTest.
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsFileNotSetTest.
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsOnClasspathAlreadyExistTest. line 55
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsOnClasspathDontAlreadyExistTest. line 38
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsOnClasspathFileNotSetTest. line 30
❌ io.quarkiverse.langchain4j.test.EasyRagTest. line 42
❌ io.quarkiverse.langchain4j.test.EasyRagWithExplicitAugmentorOnClasspathTest. line 50
❌ io.quarkiverse.langchain4j.test.EasyRagWithExplicitAugmentorTest. line 46
⚙️ JVM tests - rag - Java 21 #
📦 rag/easy-rag/deployment
❌ io.quarkiverse.langchain4j.test.EasyRagManualTriggerOnClasspathTest.
❌ io.quarkiverse.langchain4j.test.EasyRagManualTriggerTest.
❌ io.quarkiverse.langchain4j.test.EasyRagNotRecursiveOnClasspathTest. line 50
❌ io.quarkiverse.langchain4j.test.EasyRagNotRecursiveTest. line 46
❌ io.quarkiverse.langchain4j.test.EasyRagOnClasspathTest. line 60
❌ io.quarkiverse.langchain4j.test.EasyRagPathMatcherOnClasspathTest. line 51
❌ io.quarkiverse.langchain4j.test.EasyRagPathMatcherTest. line 45
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsAlreadyExistTest. line 53
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsDontAlreadyExistTest.
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsFileNotSetTest.
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsOnClasspathAlreadyExistTest. line 55
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsOnClasspathDontAlreadyExistTest. line 38
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsOnClasspathFileNotSetTest. line 30
❌ io.quarkiverse.langchain4j.test.EasyRagTest. line 42
❌ io.quarkiverse.langchain4j.test.EasyRagWithExplicitAugmentorOnClasspathTest. line 50
❌ io.quarkiverse.langchain4j.test.EasyRagWithExplicitAugmentorTest. line 46
⚙️ JVM tests - rag - Java 24 #
📦 rag/easy-rag/deployment
❌ io.quarkiverse.langchain4j.test.EasyRagManualTriggerOnClasspathTest.
❌ io.quarkiverse.langchain4j.test.EasyRagManualTriggerTest.
❌ io.quarkiverse.langchain4j.test.EasyRagNotRecursiveOnClasspathTest. line 50
❌ io.quarkiverse.langchain4j.test.EasyRagNotRecursiveTest. line 46
❌ io.quarkiverse.langchain4j.test.EasyRagOnClasspathTest. line 60
❌ io.quarkiverse.langchain4j.test.EasyRagPathMatcherOnClasspathTest. line 51
❌ io.quarkiverse.langchain4j.test.EasyRagPathMatcherTest. line 45
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsAlreadyExistTest. line 53
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsDontAlreadyExistTest.
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsFileNotSetTest.
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsOnClasspathAlreadyExistTest. line 55
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsOnClasspathDontAlreadyExistTest. line 38
❌ io.quarkiverse.langchain4j.test.EasyRagReuseEmbeddingsOnClasspathFileNotSetTest. line 30
❌ io.quarkiverse.langchain4j.test.EasyRagTest. line 42
❌ io.quarkiverse.langchain4j.test.EasyRagWithExplicitAugmentorOnClasspathTest. line 50
❌ io.quarkiverse.langchain4j.test.EasyRagWithExplicitAugmentorTest. line 46
⚙️ JVM tests - tools - Java 17 #
📦 tools/tavily/deployment
❌ io.quarkiverse.langchain4j.tavily.test.TavilyTest.
⚙️ JVM tests - tools - Java 21 #
📦 tools/tavily/deployment
❌ io.quarkiverse.langchain4j.tavily.test.TavilyTest.
⚙️ JVM tests - tools - Java 24 #
📦 tools/tavily/deployment
❌ io.quarkiverse.langchain4j.tavily.test.TavilyTest.
⚙️ Native tests - devui #
📦 integration-tests/devui
❌ devui.LangChain4jDevUIJsonRpcTest.testAddAndSearchEmbedding
⚙️ Native tests - devui-multiple-embedding-models #
📦 integration-tests/devui-multiple-embedding-models
❌ io.quarkiverse.langchain4j.test.devui.MultipleEmbeddingModelsDevUITest.test
⚙️ Native tests - multiple-providers #
📦 integration-tests/multiple-providers
❌ org.acme.example.multiple.MultipleChatProvidersTest.seventhNamedModel
⚙️ In process embedding model tests - embed-bge-small-en-v15-and-ollama #
📦 integration-tests/in-process-embedding-models/embed-bge-small-en-v15-and-ollama
❌ org.acme.test.ResourceTest.test
|
Hum, I need to implement some sort of conditional inclusion of the bean. |
Introduces GuardrailMetricsObserver to collect guardrail metrics with enhanced tags including aiservice name, operation, guardrail class, type (input/output), and outcome (success/failure/reprompt).
Deprecates the previous annotation-based metrics collection in GuardrailObservabilityProcessor.