[PR #851] [CLOSED] Add Monocle telemetry and LLM evaluation suite for earnings call analyst #7777

New Issue

GiteaMirror · 2026-06-03T21:58:28-05:00

GiteaMirror commented

2026-06-03 21:58:28 -05:00

📋 Pull Request Information

Original PR: https://github.com/Shubhamsaboo/awesome-llm-apps/pull/851
Author: @hocokahu
Created: 5/29/2026
Status: ❌ Closed

Base: main ← Head: main

📝 Commits (1)

41cdfc0 Add Monocle telemetry and LLM evaluation suite for earnings call analyst

📊 Changes

9 files changed (+486 additions, -7 deletions)

View changed files

📝 .gitignore (+9 -0)
📝 advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/README.md (+82 -7)
📝 advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/agent.py (+4 -0)
➕ advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/assets/okahu-earnings-call-analyst-trace.png (+0 -0)
📝 advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/live_demo/server.py (+4 -0)
📝 advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/requirements.txt (+2 -0)
➕ advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/telemetry.py (+24 -0)
➕ advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/tests/test_eval_with_real_llm.py (+204 -0)
➕ advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/tests/test_eval_with_real_youtube.py (+157 -0)

📄 Description

@Shubhamsaboo I ran through the example and thought it's a good idea for applying evals due to complexity of the agent flow. Hallucination and contextual are big deal. Let me know your thought. Monocle2AI is an open source under the Linux Foundation.

Summary

Adds production-grade observability and an LLM evaluation suite to the earnings call analyst demo:

Monocle auto-instrumentation — wires up OpenTelemetry-compatible spans across the FastAPI server, agent module, and eval tests via a single idempotent setup_telemetry() call. Spans capture the workflow tree workflow → agentic.turn → agentic.invocation → inference. Local JSON trace files land under ./.monocle/.
Evaluation suite under tests/:
- test_eval_with_real_llm.py — synthetic transcript with seeded facts for deterministic CI runs
- test_eval_with_real_youtube.py — pulls a real public earnings call (default: Alphabet Q1) through the same youtube_ingest pipeline the demo uses
- Both tests call Gemini for real (no mocks), capture the trace, and validate it via the monocle_test_tools fluent API: structural checks (called_agent, contains_any_output, under_token_limit, under_duration) and LLM-as-judge checks against four templates — hallucination, contextual_relevancy, summarization, pii_leakage. A failing label aborts the pytest run.
README updated with the run instructions, prereqs, and a screenshot of the captured trace.

No upstream behavior changes — all changes are additive to the existing demo. Telemetry is opt-out via MONOCLE_EXPORTER.

Test plan

pytest tests/test_eval_with_real_llm.py -v passes against real Gemini (synthetic transcript)
pytest tests/test_eval_with_real_youtube.py -v passes against the real Alphabet Q1 earnings call on YouTube
Live demo (http://127.0.0.1:4188) still serves submissions; trace appears under workflow earnings_call_analyst
No regressions in existing tests/test_core_contracts.py

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Shubhamsaboo/awesome-llm-apps/pull/851 **Author:** [@hocokahu](https://github.com/hocokahu) **Created:** 5/29/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (1) - [`41cdfc0`](https://github.com/Shubhamsaboo/awesome-llm-apps/commit/41cdfc0e6290836abb9dd64e5f61f437287a321b) Add Monocle telemetry and LLM evaluation suite for earnings call analyst ### 📊 Changes **9 files changed** (+486 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `.gitignore` (+9 -0) 📝 `advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/README.md` (+82 -7) 📝 `advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/agent.py` (+4 -0) ➕ `advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/assets/okahu-earnings-call-analyst-trace.png` (+0 -0) 📝 `advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/live_demo/server.py` (+4 -0) 📝 `advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/requirements.txt` (+2 -0) ➕ `advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/telemetry.py` (+24 -0) ➕ `advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/tests/test_eval_with_real_llm.py` (+204 -0) ➕ `advanced_ai_agents/single_agent_apps/earnings_call_analyst_agent/tests/test_eval_with_real_youtube.py` (+157 -0) </details> ### 📄 Description @Shubhamsaboo I ran through the example and thought it's a good idea for applying evals due to complexity of the agent flow. Hallucination and contextual are big deal. Let me know your thought. [Monocle2AI](https://github.com/monocle2ai/monocle) is an open source under the Linux Foundation. ## Summary Adds production-grade observability and an LLM evaluation suite to the earnings call analyst demo: - **Monocle auto-instrumentation** — wires up OpenTelemetry-compatible spans across the FastAPI server, agent module, and eval tests via a single idempotent `setup_telemetry()` call. Spans capture the workflow tree `workflow → agentic.turn → agentic.invocation → inference`. Local JSON trace files land under `./.monocle/`. - **Evaluation suite** under `tests/`: - `test_eval_with_real_llm.py` — synthetic transcript with seeded facts for deterministic CI runs - `test_eval_with_real_youtube.py` — pulls a real public earnings call (default: Alphabet Q1) through the same `youtube_ingest` pipeline the demo uses - Both tests call Gemini for real (no mocks), capture the trace, and validate it via the [`monocle_test_tools`](https://github.com/monocle2ai/monocle/tree/main/test_tools) fluent API: structural checks (`called_agent`, `contains_any_output`, `under_token_limit`, `under_duration`) and LLM-as-judge checks against four templates — `hallucination`, `contextual_relevancy`, `summarization`, `pii_leakage`. A failing label aborts the pytest run. - **README** updated with the run instructions, prereqs, and a screenshot of the captured trace. No upstream behavior changes — all changes are additive to the existing demo. Telemetry is opt-out via `MONOCLE_EXPORTER`. ## Test plan - [x] `pytest tests/test_eval_with_real_llm.py -v` passes against real Gemini (synthetic transcript) - [x] `pytest tests/test_eval_with_real_youtube.py -v` passes against the real Alphabet Q1 earnings call on YouTube - [x] Live demo (`http://127.0.0.1:4188`) still serves submissions; trace appears under workflow `earnings_call_analyst` - [x] No regressions in existing `tests/test_core_contracts.py` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-06-03 21:58:28 -05:00

GiteaMirror closed this issue

2026-06-03 21:58:31 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/awesome-llm-apps#7777