[PR #576] [CLOSED] Add tokenspy — LLM Cost Profiler & Observability (flame graph, tracing, evals) #2985

New Issue

GiteaMirror · 2026-04-26T22:00:09-05:00

GiteaMirror commented

2026-04-26 22:00:09 -05:00

📋 Pull Request Information

Original PR: https://github.com/Shubhamsaboo/awesome-llm-apps/pull/576
Author: @pinexai
Created: 3/10/2026
Status: ❌ Closed

Base: main ← Head: main

📝 Commits (1)

71e11c8 Add tokenspy — LLM Cost Profiler & Observability to LLM Optimization Tools

📊 Changes

1 file changed (+1 additions, -0 deletions)

View changed files

📝 README.md (+1 -0)

📄 Description

What is tokenspy?

tokenspy is a local-first LLM observability stack — the cProfile equivalent for AI apps. It tells you exactly which function is burning your budget, with zero cloud, zero signup, zero proxy.

One decorator. Instant flame graph:

@tokenspy.profile
def run_pipeline(query):
    docs = fetch_and_summarize(query)
    entities = extract_entities(docs)
    return generate_report(entities)

tokenspy.report()

fetch_and_summarize   $0.038  ████████████  73%  ← here's your $800/month
generate_report       $0.011  ████          21%
extract_entities      $0.003  █              6%

🔴 Switch fetch_and_summarize to gpt-4o-mini: 94% cheaper (~$540/month savings)

Why it fits 🎯 LLM Optimization Tools

Pinpoints which function/step is costing the most → actionable cost reduction
Built-in optimization hints (model switch recommendations with estimated savings)
v0.2.0 includes full observability: structured tracing, evaluations + datasets, prompt versioning, live dashboard (tokenspy serve), OpenTelemetry export
Zero dependencies (core), works offline, data never leaves the machine

Links

GitHub: https://github.com/pinakimishra95/tokenspy
PyPI: https://pypi.org/project/tokenspy/
Docs: https://pinakimishra95.github.io/tokenspy

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Shubhamsaboo/awesome-llm-apps/pull/576 **Author:** [@pinexai](https://github.com/pinexai) **Created:** 3/10/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (1) - [`71e11c8`](https://github.com/Shubhamsaboo/awesome-llm-apps/commit/71e11c813f095ecee25c3b8566d88f362280c987) Add tokenspy — LLM Cost Profiler & Observability to LLM Optimization Tools ### 📊 Changes **1 file changed** (+1 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+1 -0) </details> ### 📄 Description ## What is tokenspy? tokenspy is a local-first LLM observability stack — the **cProfile equivalent for AI apps**. It tells you exactly which function is burning your budget, with zero cloud, zero signup, zero proxy. **One decorator. Instant flame graph:** ```python @tokenspy.profile def run_pipeline(query): docs = fetch_and_summarize(query) entities = extract_entities(docs) return generate_report(entities) tokenspy.report() ``` ``` fetch_and_summarize $0.038 ████████████ 73% ← here's your $800/month generate_report $0.011 ████ 21% extract_entities $0.003 █ 6% 🔴 Switch fetch_and_summarize to gpt-4o-mini: 94% cheaper (~$540/month savings) ``` ## Why it fits 🎯 LLM Optimization Tools - Pinpoints which function/step is costing the most → actionable cost reduction - Built-in optimization hints (model switch recommendations with estimated savings) - v0.2.0 includes full observability: structured tracing, evaluations + datasets, prompt versioning, live dashboard (`tokenspy serve`), OpenTelemetry export - Zero dependencies (core), works offline, data never leaves the machine ## Links - GitHub: https://github.com/pinakimishra95/tokenspy - PyPI: https://pypi.org/project/tokenspy/ - Docs: https://pinakimishra95.github.io/tokenspy --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-04-26 22:00:09 -05:00

GiteaMirror closed this issue

2026-04-26 22:00:10 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/awesome-llm-apps#2985