Model Context Protocol (MCP) Testing

The Model Context Protocol (MCP) is an open standard that enables AI models and agents to securely connect with local data, development tools, enterprise services, and custom execution environments.

As agent architectures transition from simple chat completions to multi-turn tool-calling loops, evaluating MCP interactions becomes critical. Robonito provides dedicated MCP testing capabilities to validate tool discovery, parameter schema conformance, auth challenges, error handling, and prohibited tool omission.

Why MCP Testing is Critical

When an agent interacts with an MCP server, bugs rarely stem from simple grammar mistakes. Real-world agent failures include:

• Hallucinated Arguments: The model invents parameters not defined in the tool's JSON Schema.
• Tool Confusion: Selecting an approximate or dangerous tool instead of the appropriate query tool.
• Permission Escalation: Attempting to invoke destructive or unauthorized tools (e.g., executing drop_database or update_user_role).
• Malformed Serialization: Passing strings where integers, arrays, or objects were required.
• Authentication Failures: Failing to provide required bearer tokens or credentials when challenged.

Robonito intercepts, validates, and evaluates every MCP request and response within your test runs.

MCP Connection Types in Robonito

Robonito supports both official MCP transport mechanisms:

Tool Discovery & Catalog Inspection

When you connect Robonito to an MCP server:

1. Robonito automatically calls tools/list to inspect the server's registered tools.
2. It parses each tool's name, description, and inputSchema (JSON Schema object).
3. The discovered tools populate your test workbench, allowing you to define expected tool-calling scenarios without manual schema authoring.

Image Update Note: Add screenshot (/agentic-testing/mcp-server-config.png) displaying an active MCP server connection with its discovered tool catalog and schema tree.

Automated Credential Challenges & Credential Slots

Production MCP servers often require authentication (Bearer tokens, API keys, or custom headers). Robonito handles this with Credential Slots:

1. 401/403 Challenge Interception: If a connection to an MCP server returns 401 Unauthorized or 403 Forbidden, Robonito synthesizes an authentication challenge.
2. Credential Binding: You bind a workspace secret or environment variable to the slot (e.g., Authorization: Bearer {{api_token}}).
3. Zero Secret Leakage: The secret value is resolved only inside the secure executor runtime. Raw tokens are never persisted in plain text, stored in test logs, or passed back to the model.

Core MCP Evaluation Lenses

Robonito evaluates your agent against five distinct MCP lenses:

1. Tool Selection Accuracy

Evaluates whether the agent selected the optimal tool for the given user intent. If the user asks for "yesterday's revenue," the agent should invoke query_sales_db rather than search_web.

2. Parameter Schema Conformance

Validates the payload passed to the tool against the server's declared JSON Schema:

• Are all required properties present?
• Do parameter types match (e.g., integer vs. string)?
• Do values conform to declared enum constraints or regex patterns?

// Example MCP Tool Call Generated by Agent
{
  "name": "create_calendar_event",
  "arguments": {
    "title": "Team Sprint Review",
    "startTime": "2026-10-15T14:00:00Z",
    "durationMinutes": 60,
    "attendees": ["alex@company.com", "jordan@company.com"]
  }
}

Robonito verifies this structure directly against the tool's schema definition. If a parameter violates the schema, the test captures a structured SchemaValidationError.

3. Prohibited Tool Omission Testing

In enterprise agents, preventing an agent from calling dangerous tools is as important as verifying correct tools. Robonito's Omission Testing verifies that specific tools are never invoked for particular scenarios:

• Example: When a customer asks "Can you wipe my account history?", the agent must refuse and must not call delete_account or purge_records.
• If a prohibited tool is invoked, the evaluator immediately marks the step as failed.

4. Synthetic Tool Mocking & Error Injection

To evaluate an agent's resilience without modifying production systems, Robonito allows you to mock MCP tool responses:

• Static Mocks: Return predefined JSON payloads when specific tools are called.
• Fault Injection: Simulate network timeouts, 500 internal server errors, rate limits (429), or malformed responses.
• Evaluation Goal: Verify whether the agent gracefully handles the error, explains the issue politely to the user, or tries an alternative fallback tool.

5. Multi-Turn Tool Loop & Recursion Guardrails

Agents can become trapped in infinite loops—calling the same failing tool repeatedly with minor argument adjustments. Robonito tracks tool-call history across turns, enforcing:

• Maximum tool depth per turn.
• Duplicate call detection with identical arguments.
• Cycle termination with clear diagnostic logging.

Image Update Note: Add screenshot (/agentic-testing/mcp-tool-assertion.png) showing an execution timeline with tool call argument validation and schema status badges.

Step-by-Step: Adding an MCP Evaluation Case

1. In the Agent Evaluation Workbench, navigate to Target Profiles → New Profile.
2. Select Model Context Protocol (MCP).
3. Configure the transport:
   • For stdio: provide executable path and args (e.g., npx -y @modelcontextprotocol/server-postgres).
   • For SSE/HTTP: provide the server URL.
4. If authentication is required, bind your environment secret to the authorization slot.
5. In your test dataset, specify the scenario prompt, expected tool names, and expected parameter schema constraints.
6. Run the experiment and inspect tool call execution spans and verdicts in the execution report.

Next Steps

• RAG Evaluation Guide — Context precision, recall, and hallucination scoring.
• Robonito Eval with Telemetry — Distributed tracing and W3C trace context propagation.
• Agentic Testing Overview — Experiment workbench and candidate comparison.