Telemetry

About

Telemetry data such as logs, metrics, and traces will help developers understand the internal state of the system. This page walks though different types of telemetry and observability available in Toolbox.

Toolbox exports telemetry data of logs via standard out/err, and traces/metrics through OpenTelemetry. Additional flags can be passed to Toolbox to enable different logging behavior, or to export metrics through a specific exporter.

Logging

The following flags can be used to customize Toolbox logging:

Example:

./toolbox --tools-file "tools.yaml" --log-level warn --logging-format json

Level

Toolbox supports the following log levels, including:

Toolbox will only output logs that are equal or more severe to the level that it is set. Below are the log levels that Toolbox supports in the order of severity.

Format

Toolbox supports both standard and structured logging format.

The standard logging outputs log as string:

2024-11-12T15:08:11.451377-08:00 INFO "Initialized 0 sources.\n"

The structured logging outputs log as JSON:

{
  "timestamp":"2024-11-04T16:45:11.987299-08:00",
  "severity":"ERROR",
  "logging.googleapis.com/sourceLocation":{...},
  "message":"unable to parse tool file at \"tools.yaml\": \"cloud-sql-postgres1\" is not a valid type of data source"
}

Telemetry

Toolbox supports exporting metrics and traces to any OpenTelemetry compatible exporter.

Metrics

A metric is a measurement of a service captured at runtime. The collected data can be used to provide important insights into the service. Toolbox metrics follow the MCP Semantic Conventions where applicable, and include additional Toolbox-specific metrics for deeper observability.

Standard Metrics

Toolbox-specific Metrics

Duration histograms use the following bucket boundaries (in seconds), as defined by the MCP semantic conventions:

0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, 30, 60, 120, 300

Metric Attributes

The attributes recorded with each metric are listed below. Attributes marked optional are only included when applicable.

mcp.server.operation.duration

mcp.server.session.duration and toolbox.server.mcp.active_sessions

toolbox.tool.execution.duration

Traces

A trace is a tree of spans that shows the path that a request makes through an application.

Initialization Spans

When Toolbox starts, it generates a root span toolbox/server/init with child spans for each component initialized:

toolbox/server/init
├── toolbox/server/source/init         attr: source_type, source_name
│   └── toolbox/server/source/connect  attr: source_type, source_name
├── toolbox/server/auth/init           attr: auth_type, auth_name
├── toolbox/server/embeddingmodel/init attr: model_type, model_name
├── toolbox/server/tool/init           attr: tool_type, tool_name
├── toolbox/server/toolset/init        attr: toolset.name
└── toolbox/server/prompt/init         attr: prompt_type, prompt_name

Request Spans

Each incoming MCP request generates a transport span, with a child span for the MCP method being processed.

Toolbox connection spans

MCP method spans

Method-level spans follow the MCP Semantic Conventions. Each span represents the processing of a single MCP request or notification.

The span name follows the format {mcp.method.name} {target} where target is {gen_ai.tool.name} or {gen_ai.prompt.name} when applicable, otherwise just {mcp.method.name}. Span status is set to ERROR when an error occurs, with the status description set to the JSON-RPC error message.

All method spans include the following attributes:

Context Propagation

Toolbox supports distributed tracing via the W3C Trace Context standard. Incoming trace context is extracted in two ways:

• HTTP headers: The traceparent and tracestate headers are read from incoming HTTP requests.
• JSON-RPC _meta field: The params._meta.traceparent and params._meta.tracestate fields are read from the JSON-RPC message body. This allows trace context to propagate over stdio transport.

The examples below show how spans are connected across the client and Toolbox for each transport type.

STDIO Initialize

initialize (CLIENT, trace=t1, span=s1)                                   # FROM MCP Client
|
--- toolbox/server/mcp/stdio (SERVER, trace=t1, span=s2, parent=s1)      # IN TOOLBOX
    |
    --- initialize (SERVER, trace=t1, span=s3, parent=s2)                # IN TOOLBOX

STDIO Tool Call

tools/call get-weather (CLIENT, trace=t1, span=s1)                       # FROM MCP Client
|
--- toolbox/server/mcp/stdio (SERVER, trace=t1, span=s2, parent=s1)      # IN TOOLBOX
    |
    --- tools/call get-weather (SERVER, trace=t1, span=s3, parent=s2)    # IN TOOLBOX

SSE Connection

connection (CLIENT, trace=t1, span=s2, parent=s1)                        # FROM MCP Client
|
--- toolbox/server/mcp/sse (SERVER, trace=t1, span=s3, parent=s2)        # IN TOOLBOX

HTTP Initialize

initialize (CLIENT, trace=t1, span=s1)                                   # FROM MCP Client
|
--- toolbox/server/mcp/http (SERVER, trace=t1, span=s2, parent=s1)       # IN TOOLBOX
    |
    --- initialize (SERVER, trace=t1, span=s3, parent=s2)                # IN TOOLBOX

HTTP Tool Call

tools/call get-weather (CLIENT, trace=t1, span=s1)                       # FROM MCP Client
|
--- toolbox/server/mcp/http (SERVER, trace=t1, span=s2, parent=s1)       # IN TOOLBOX
    |
    --- tools/call get-weather (SERVER, trace=t1, span=s3, parent=s2)    # IN TOOLBOX

Resource Attributes

All metrics and traces generated within Toolbox will be associated with a unified resource. The list of resource attributes included are:

Exporter

An exporter is responsible for processing and exporting telemetry data. Toolbox generates telemetry data within the OpenTelemetry Protocol (OTLP), and user can choose to use exporters that are designed to support the OpenTelemetry Protocol. Within Toolbox, we provide two types of exporter implementation to choose from, either the Google Cloud Exporter that will send data directly to the backend, or the OTLP Exporter along with a Collector that will act as a proxy to collect and export data to the telemetry backend of user’s choice.

Google Cloud Exporter

The Google Cloud Exporter directly exports telemetry to Google Cloud Monitoring. It utilizes the GCP Metric Exporter and GCP Trace Exporter.

OTLP Exporter

This implementation uses the default OTLP Exporter over HTTP for metrics and traces. You can use this exporter if you choose to export your telemetry data to a Collector.

Collector

A collector acts as a proxy between the application and the telemetry backend. It receives telemetry data, transforms it, and then exports data to backends that can store it permanently. Toolbox provide an option to export telemetry data to user’s choice of backend(s) that are compatible with the Open Telemetry Protocol (OTLP). If you would like to use a collector, please refer to this Export Telemetry using the Otel Collector.

Flags

The following flags are used to determine Toolbox’s telemetry configuration:

In addition to the flags noted above, you can also make additional configuration for OpenTelemetry via the General SDK Configuration through environmental variables.

Examples:

To enable Google Cloud Exporter:

./toolbox --telemetry-gcp

To enable OTLP Exporter, provide Collector endpoint:

./toolbox --telemetry-otlp="127.0.0.1:4553"