Getting Started

This guide covers installation and your first end-to-end evaluation run.

Prerequisites

• Python 3.11+
• pip
• Model credentials in environment variables (for example AZURE_API_KEY and AZURE_API_BASE for Azure OpenAI)

Install with a quickstart example: LangGraph travel planner

The flagship example evaluates a multi-tool LangGraph travel planner. The target is reached through target.callable — the same integration boundary you would use for any agent or multi-agent system — and Phoenix/OpenInference auto-instrumentation captures the agent's OpenTelemetry spans so the judge can cite tool calls and routing decisions. This is the recommended integration shape for any non-trivial agent.

Recommended install path

bash (macOS / Linux):

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[phoenix]"
python -m pip install -r examples/travel_planner_langgraph/requirements.txt
cp .env.example .env

The second install is required because the flagship target imports LangGraph, LangChain Core, and LangChain OpenAI directly. Those are dependencies of the example agent, not features of the assert-ai package.

Edit .env with credentials for your provider. Defaults match the example's azure/... model. Any LiteLLM provider (OpenAI, Anthropic, Bedrock, Vertex, Ollama, and others) works.

PowerShell (Windows):

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[phoenix]"
python -m pip install -r examples/travel_planner_langgraph/requirements.txt
Copy-Item .env.example .env

Run your first evaluation

The example's requirements install its LangChain OpenInference instrumentor. Its auto_trace.py then calls assert_ai.auto_trace.enable(), which discovers and activates installed instrumentors so the judge can cite tool calls, routing decisions, model calls, and latency as evidence. It does not install packages or start a Phoenix server.

phoenix serve is optional — only run it if you want a browser UI to inspect the traces visually. The eval runs and the judge see the same span data either way.

bash (macOS / Linux):

phoenix serve  # optional: trace UI on http://localhost:6006
assert-ai run --config examples/travel_planner_langgraph/evals/budget_overrun.yaml

PowerShell (Windows):

phoenix serve  # optional: trace UI on http://localhost:6006
assert-ai run --config examples/travel_planner_langgraph/evals/budget_overrun.yaml

Check run status:

PowerShell (Windows):

assert-ai results status travel-planner-langgraph-v1 demo-1

bash (macOS / Linux):

assert-ai results status travel-planner-langgraph-v1 demo-1

Artifacts are written under:

artifacts/results/travel-planner-langgraph-v1/demo-1/

Codespaces / VS Code Dev Containers

The repo includes a minimal dev container for the LangGraph quickstart. It installs the root development group plus examples/travel_planner_langgraph/requirements.txt because the container is also the golden-path example environment, copies .env.example to .env if needed, and forwards Phoenix on port 6006. After container setup, add your provider credentials to .env and run the same assert-ai run command.

PowerShell (Windows) — full sequence:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[phoenix]"
python -m pip install -r examples/travel_planner_langgraph/requirements.txt
Copy-Item .env.example .env

phoenix serve  # optional
assert-ai run --config examples/travel_planner_langgraph/evals/budget_overrun.yaml
assert-ai results status travel-planner-langgraph-v1 demo-1

What just happened

1. systematize expanded the behavior spec into behavior categories.
2. test_set generated prompt and scenario test cases.
3. inference executed the target for each case.
4. judge produced verdicts, evidence, and aggregate metrics.

What the quickstart does:

CLI helper assistant to create your own config

Don't want to write YAML by hand? assert-ai init starts a conversational LLM assistant that asks about your agent, eval goals, and constraints, then proposes a complete config YAML file to use for your evaluations.

assert-ai init needs an LLM to power the conversation. Pass --model with any LiteLLM model string and make sure the matching API key is set in your .env file (loaded by default) or environment:

assert-ai init --model azure/gpt-5.4
# or skip the first question:
assert-ai init --model azure/gpt-5.4 --describe "A customer-support chatbot with order-lookup and refund tools"
# or edit/extend an existing config:
assert-ai init --model azure/gpt-5.4 --from examples/travel_planner_langgraph/evals/budget_overrun.yaml

See CLI Commands for the full option reference.

• To learn the config format, see Config Overview.
• To inspect outputs in detail, see Results Guide.
• To use the local web viewer, see Run the Local UI Viewer Application.

Authenticating Azure OpenAI with Managed Identity

If you would rather not provision and rotate an AZURE_API_KEY, ASSERT can call Azure OpenAI using Entra ID (Microsoft Managed Identity / az login) instead. This works for any azure/* model string and uses LiteLLM's native azure_ad_token_provider hook under the hood — no other config changes required.

Install the optional dependency

python -m pip install -e ".[azure-auth]"

This pulls in azure-identity and lets ASSERT mint bearer tokens through DefaultAzureCredential.

Grant the caller the right RBAC role

On the target Azure OpenAI resource, give the caller identity (your user, a managed identity, or a service principal) the Cognitive Services OpenAI User role. Without this role every request will return 401.

Pick an auth mode

Auth resolution at process start follows a single precedence rule:

AZURE_API_BASE is still required so LiteLLM knows which Azure OpenAI endpoint to call.

The same auth mode also applies to azure_ai/* LiteLLM routes, including hosted Azure AI Foundry agents (azure_ai/agents/<AGENT_ID>). Those routes need AZURE_AI_API_BASE set to the Foundry project endpoint instead of AZURE_API_BASE. No extra setup beyond pip install -e ".[azure-auth]" and az login (or Service Principal env vars).

Local development with az login

az login
export ASSERT_AZURE_USE_AAD=1
unset AZURE_API_KEY  # optional — the flag wins regardless
assert-ai run --config examples/azure_managed_identity/eval_config.yaml

Running on Azure (App Service, AKS, Container Apps, VM)

Assign a managed identity to the workload, grant it the OpenAI User role, and set ASSERT_AZURE_USE_AAD=1. To pin a specific user-assigned identity when multiple are attached, set AZURE_CLIENT_ID to its client ID; DefaultAzureCredential will pick it up automatically.

Troubleshooting

• LLMAuthError: ... azure-identity package is not installed — run pip install -e ".[azure-auth]" (or assert-ai[azure-auth] if you installed from PyPI).
• 401 with a hint about Cognitive Services OpenAI User — the credential resolved, but the identity is missing the RBAC role on the resource.
• A 401 that mentions the install hint instead — you are in aad-fallback mode without azure-identity. Install the extra or set AZURE_API_KEY.