Evaluation and Observability

Evaluation answers “did the result meet the expectation?” Observability answers “what ran, where did time go, and why did it fail?” KsADK provides local and cloud EvalSets, one evaluation report format, Studio evaluation and Trace Explorer, standard OTLP export, and RuntimeEvent replay. These features work independently and can also be correlated through TraceRef, run, and session identifiers.

Capability overview

Install and start

The complete installation includes evaluation, A2A, and OTLP support:

pip install -U "ksadk[all]"

Create a template, then inspect validation output and the evaluator plan:

agentengine evalset init \
  --template tool-routing \
  --output-file ./evals/tool-routing.yaml

agentengine eval \
  --evalset-file ./evals/tool-routing.yaml \
  --agent-dir ./my-agent \
  --validate-only \
  --format json

--validate-only does not invoke the Agent. The evaluationPlan in JSON output lists the evaluators that the EvalSet would use, making the result suitable for CI review before execution.

Write an EvalSet

Prefer the native ksadk.eval/v1 YAML format. A case can contain one input or an ordered turns list. Its final turn can set expectedOutput or reference_output.

schemaVersion: ksadk.eval/v1
name: agent-smoke
cases:
  - id: ping
    input: "Reply with PONG only"
    expectedOutput: "PONG"
    assertions:
      - type: response.equals
        value: "PONG"
      - type: runtime.maxLatencyMs
        value: 10000

  - id: weather
    input: "Check tomorrow's weather in Beijing and recommend what to do"
    reference_output: "Use the weather result to summarize tomorrow's conditions and give travel advice."
    assertions:
      - type: tool.succeeded
        value: weather_lookup
      - type: tool.sequence
        value: [weather_lookup]

KsADK also recognizes existing Studio EvaluationSuite and ADK eval_cases formats. It converts them to ksadk.eval/v1 and calculates a contentDigest. Case IDs must be unique.

Built-in templates

agentengine evalset init \
  --template structured-output \
  --output-file ./evals/structured-output.yaml

Supported assertions

When evidence is missing, the assertion is UNAVAILABLE; an unknown value is never treated as zero or as a pass. Tool assertions are usually UNAVAILABLE for an A2A target that does not expose a standardized tool trajectory. Local source and Studio Build targets project tool calls from RuntimeEvent evidence.

Publish and reuse a cloud EvalSet

preview is offline and prints the fixed-schema payload that would be published. push publishes the current workspace EvalSet. pull retrieves one fixed Dataset ID and version into a local file.

# Inspect the payload before publishing; cloud publishing requires full_trace
agentengine evalset preview \
  --evalset-file ./evals/tool-routing.yaml \
  --data-policy full_trace \
  --format json

# Publish a new immutable Dataset version
agentengine evalset push \
  --file ./evals/tool-routing.yaml \
  --dataset-id <dataset-id>

# Retrieve a fixed version for a reproducible run
agentengine evalset pull \
  --dataset-id <dataset-id> \
  --dataset-version 3 \
  --project-id <project-id> \
  --output-file ./evals/imported-v3.yaml

push and pull require access to the Agent Eval service. Never place a temporary download URL, account credential, or token in an EvalSet or commit it to the repository.

Run an evaluation

Each run must use either a local --evalset-file or an immutable cloud Dataset selected by --dataset-id --dataset-version. It must also select exactly one target.

Local source

The local target copies the project into an isolated snapshot, records its revision and Git state, and loads a supported ADK, LangGraph, LangChain, or DeepAgents entry point. Use --entrypoint to override detection.

agentengine eval \
  --evalset-file ./evals/tool-routing.yaml \
  --agent-dir ./my-agent \
  --timeout-seconds 120 \
  --report-dir ./.agentkit/evaluations \
  --format json

A2A Agent

Cases run sequentially in file order. A multi-turn case reuses one A2A context_id. Authentication accepts only an env:// credential reference, so the secret value is not written to the command line or report.

export A2A_EVAL_TOKEN="<your-token>"

agentengine eval \
  --evalset-file ./evals/tool-routing.yaml \
  --a2a-url https://agent.example.test/.well-known/agent-card.json \
  --credential-ref env://A2A_EVAL_TOKEN \
  --fail-fast

Cloud Dataset version

Use a fixed version instead of a moving active Dataset so later runs remain reproducible:

agentengine eval \
  --dataset-id <dataset-id> \
  --dataset-version 3 \
  --dataset-project-id <project-id> \
  --agent-dir ./my-agent

Common execution options

DataPolicy can be local_only, metadata_only, redacted_trace, or full_trace. metadata_only omits text and attributes, while redacted_trace retains redacted content. Selecting a policy does not upload a report or trace; remote trace export is controlled separately by OTLP environment variables.

Evaluators and automatic planning

Without --evaluator, KsADK derives a plan from the EvalSet. A reference answer uses a fully configured llm_judge@v1 when available and otherwise uses reference_match@v1. Response, runtime-budget, and tool assertions add their deterministic evaluators. A case with neither a reference nor a response assertion receives business_standard@v1 and an unavailable quality result, so “the Agent ran” is not mistaken for business success.

Explicit --evaluator options replace the automatic plan:

agentengine eval \
  --evalset-file ./evals/structured-output.yaml \
  --agent-dir ./my-agent \
  --evaluator response_contract@v1 \
  --evaluator runtime_budget@v1

The LLM judge requires ksadk[judge], a reference answer, full_trace, a model, an API base, and an environment variable that contains the key:

export KSADK_EVAL_JUDGE_API_KEY="<your-api-key>"

agentengine eval \
  --evalset-file ./evals/knowledge-qa.yaml \
  --agent-dir ./my-agent \
  --evaluator llm_judge@v1 \
  --judge-model <judge-model> \
  --judge-api-base https://judge.example.test/v1 \
  --data-policy full_trace

Read evaluation results

The default report path is:

.agentkit/evaluations/<eval-run-id>/report.json

The ksadk.eval.report/v1 report stores snapshots of the EvalSet, target, optional cloud Dataset, and evaluation configuration. Each case includes target status, latency, usage, metrics, TraceRef, and summary status. RuntimeEvent evidence for a local target is stored in evidence/ beside the report.

A successful target invocation is not an evaluation pass. Check both EvalRunReport.status and every required metric.

Use Studio

Start Studio and open Evaluations in the sidebar:

agentengine studio ./my-agent-workspace

Upload a YAML or JSON EvalSet, select an A2A Agent, local source, or Studio Build, configure timeout, fail-fast behavior, and evaluators, then start the background operation. The list shows status and summary; details show cases, metrics, target usage, and TraceRef. A running evaluation can be cancelled.

A Studio Build target must already be successful and carry an immutable digest. Studio does not treat an unfrozen Codex source tree as a reproducible build.

Open Observability for local traces, span parent/child trees, waterfalls, attributes, events, resources, instrumentation scope, raw OTLP JSON, and traceparent. Local OTLP files live in .agentkit/traces/ inside the workspace and are intended only for local diagnosis.

Export to an OTLP backend

Standard OTLP/HTTP settings work with Langfuse, an OTel Collector, and other compatible backends:

export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.example.test/otel"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20<your-token>"

agentengine run .

Traces-specific variables take precedence:

export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://otel.example.test/otel/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer%20<your-token>"

When only the general endpoint is set, KsADK derives /v1/traces. Header values use RFC 3986 encoding and multiple headers are comma-separated. To export to a second CloudMonitor backend, set CLOUD_MONITOR_OTLP_ENDPOINT or CLOUD_MONITOR_OTLP_TRACES_ENDPOINT and the corresponding headers. Both exporters receive the same spans with identical trace_id and span_id values. See Observability and tracing for the complete variable list.

Replay RuntimeEvent

OTel spans describe topology, timing, and diagnostics. Canonical RuntimeEvent schema v2 describes the semantic order of Agent execution and evaluation evidence. The two can be correlated, but one RuntimeEvent does not equal one span.

# Human-readable transcript
agentengine replay <session-id>

# Read a cursor window as JSON
agentengine replay <session-id> \
  --after-seq-id 120 \
  --before-seq-id 260 \
  --format json

Replay projects text, reasoning, tools, artifacts, and run status. It does not call the model, rerun tools, or repeat approvals. Only sessions persisted through the canonical RuntimeEvent v2 store are read; legacy SessionEvent rows are not silently converted into new canonical facts.

Choose the right tool

Troubleshooting