Skip to main content
Section navigation

Analytics Service

Automatic tracking of AI costs, usage metrics, session engagement, content performance, and audit trails. Every request logged with full observability.

On this page

TL;DR

systemprompt.io captures every AI request, session, page view, and tool execution automatically. Token counts, costs, latencies, engagement depth, bot classification, and content performance are all recorded without instrumentation code. Query everything through the CLI, the API, or let agents analyze their own performance via MCP.

What It Does and Why It Matters

The analytics service provides full observability into your AI operations and web traffic. It answers questions that matter in production: How much did that agent cost this week? Which content is actually being read? Are sessions real humans or bots? What is the error rate on tool calls?

Data capture happens at multiple points in the request lifecycle through instrumented middleware. No application code changes are required. The system records timing, token counts, costs, engagement signals, and metadata transparently as requests flow through the stack.

The "closed loop" principle means agents can query their own performance data through the same interfaces humans use. An agent can ask "what was my average response time today?" and act on the answer. This self-awareness enables adaptive behavior in production.

Request --> Middleware (start timer) --> LLM / Tool Call --> Middleware (capture metrics) --> Response
                                              |
                                        Analytics Store (PostgreSQL)

Data Capture

The analytics system automatically records data across five domains. No manual instrumentation is needed.

AI Requests

Every LLM call is logged with:

  • Provider and model (Anthropic, OpenAI, Gemini)
  • Token counts (input and output)
  • Cost (calculated from model-specific pricing)
  • Latency (end-to-end response time)
  • Status (success, failure, error type)
  • Trace ID (links related events across the request chain)

Session and Engagement

The user_sessions and engagement_events tables capture visitor behavior:

  • Session classification -- known bots, scanners, behavioral bots, ghost sessions, clean humans
  • Page views with time-on-page and scroll depth
  • Click counts per page
  • Landing page and navigation path
  • Device type, browser, geographic region
  • Traffic source and referrer

MCP Tool Executions

Every tool call through MCP is tracked in mcp_tool_executions:

  • Tool name and server
  • User who triggered the call
  • Status (success or failed)
  • Timestamp for activity correlation

User Activity

The user_activity table logs operational events:

  • Logins and authentication events
  • Marketplace edits (plugin, skill, agent changes)
  • Category-based filtering for audit queries

Content Performance

The content_performance_metrics table aggregates engagement data per content item:

  • Total views and unique visitors
  • Average time on page (seconds)
  • Views over 7-day and 30-day windows
  • Trend direction (up, down, stable) -- computed by comparing recent week to prior average

This table is populated by the content_analytics_aggregation job, which is registered by the web extension. Run it manually with systemprompt infra jobs run content_analytics_aggregation, or add it to services/scheduler/config.yaml to run it on a cron schedule.

Cost Tracking

Cost tracking is automatic for all supported providers. The system knows token pricing for each model and calculates costs in real-time.

Supported Providers

Provider Models Cost Tracking
Anthropic Claude models (Opus, Sonnet, Haiku) Full support
OpenAI GPT models Full support
Gemini Gemini models Full support

Cost Calculation

Costs are calculated from token counts and model-specific pricing:

Cost = (input_tokens x input_price) + (output_tokens x output_price)

Per-model pricing lives in the provider model catalog, alongside the model's other capabilities. Inspect and manage catalog entries with:

systemprompt admin config catalog provider --help   # manage registry providers
systemprompt admin config catalog model --help      # manage the models a provider serves

Audit Trails

Every operation generates audit events. These create a complete record of who did what, when, and how.

Audit Event Types

Category Events
Authentication login, logout, token_issued, token_revoked
Authorization scope_granted, permission_denied
AI Operations request_started, request_completed, request_failed
Data Access file_accessed, content_created, content_deleted
Administration user_created, config_changed, agent_modified
MCP Tools tool_executed, tool_failed

Trace Correlation

Every request receives a trace ID that links all related events. Follow a single user action through authentication, authorization, AI calls, tool executions, and responses:

# Follow a trace end-to-end (accepts an AI request ID, task ID, or trace ID)
systemprompt infra logs audit <id>

Usage Metrics and Dashboards

Admin Dashboard

The admin dashboard at /admin shows real-time aggregate metrics:

  • Events today and this week (from user_activity)
  • Total edits and logins
  • MCP tool calls and MCP errors
  • Top users by activity (edits + MCP calls)
  • Popular skills by execution count
  • Hourly activity over the last 24 hours
  • Usage time-series bucketed by hour (tool uses, prompts, active users, sessions, errors)

Traffic Reports

Traffic data is available on demand through the CLI:

  • Session totals with bot/human breakdown (systemprompt analytics sessions stats)
  • Traffic sources and referrers (systemprompt analytics traffic sources)
  • Geographic distribution (systemprompt analytics traffic geo)
  • Device and browser breakdown (systemprompt analytics traffic devices)
  • Bot traffic analysis (systemprompt analytics traffic bots)
  • Real-time active sessions (systemprompt analytics sessions live)

Generated traffic reports are stored in the admin_traffic_reports table and surfaced on the admin dashboard.

CLI Reference

The systemprompt analytics command provides access to all analytics data.

Top-Level Commands

Command Description
systemprompt analytics overview Dashboard overview (supports --since, --until, --export)
systemprompt analytics conversations Conversation analytics
systemprompt analytics agents Agent performance analytics
systemprompt analytics tools Tool usage analytics
systemprompt analytics requests AI request analytics
systemprompt analytics sessions Session analytics
systemprompt analytics content Content performance analytics
systemprompt analytics traffic Traffic analytics
systemprompt analytics costs Cost analytics

Subcommand Details

Costs:

systemprompt analytics costs summary       # Cost summary
systemprompt analytics costs trends        # Cost trends over time
systemprompt analytics costs breakdown     # Cost breakdown by model/agent

Requests:

systemprompt analytics requests stats      # Aggregate AI request statistics
systemprompt analytics requests list       # List individual AI requests
systemprompt analytics requests trends     # AI request trends over time
systemprompt analytics requests models     # Model usage breakdown

Agents:

systemprompt analytics agents stats        # Aggregate agent statistics
systemprompt analytics agents list         # List agents with metrics
systemprompt analytics agents trends       # Agent usage trends over time
systemprompt analytics agents show <name>  # Deep dive into specific agent

Sessions:

systemprompt analytics sessions stats      # Session statistics
systemprompt analytics sessions trends     # Session trends over time
systemprompt analytics sessions live       # Real-time active sessions

Traffic:

systemprompt analytics traffic sources     # Traffic source breakdown
systemprompt analytics traffic geo         # Geographic distribution
systemprompt analytics traffic devices     # Device and browser breakdown
systemprompt analytics traffic bots        # Bot traffic analysis

Content:

systemprompt analytics content stats       # Content engagement statistics
systemprompt analytics content top         # Top performing content
systemprompt analytics content trends      # Content trends over time

Use systemprompt analytics <command> --help for detailed options on any subcommand.

Performance Monitoring

Background Jobs

The web extension registers an analytics aggregation job:

Job Purpose
content_analytics_aggregation Aggregates engagement events into content_performance_metrics

Run it manually with systemprompt infra jobs run content_analytics_aggregation, or schedule it by adding an entry to services/scheduler/config.yaml. Use systemprompt infra jobs list to see every registered job.

Agent Self-Analysis via MCP

Agents connected to the systemprompt MCP server can query their own analytics through its CLI passthrough tool, which runs any systemprompt command on the agent's behalf:

  • {"command": "analytics costs summary"} -- query cost data
  • {"command": "analytics requests stats"} -- query request metrics
  • {"command": "analytics agents show <name>"} -- deep dive into one agent
  • {"command": "infra logs audit <id>"} -- reconstruct a full audit chain

An agent noticing high latency might switch to a faster model. An agent seeing repeated errors can adjust its approach. This feedback loop is what makes the "closed loop" architecture practical.

Configuration

Analytics requires no configuration. Capture is always on: every AI request, session, page view, and tool execution is recorded to PostgreSQL as it happens, with no sampling and no opt-in flags.

The pieces you can configure live elsewhere:

Concern Where
Per-model pricing used for cost calculation Provider model catalog (systemprompt admin config catalog model)
Aggregation job scheduling services/scheduler/config.yaml
Data cleanup database_cleanup and session cleanup jobs in services/scheduler/config.yaml