Connect to NeuralTrust

The TrustTest client provides a powerful interface for persisting and retrieving evaluation artifacts. It allows you to save and load any scenario, test set, and evaluation results through a consistent API.

Client Implementations

TrustTest offers multiple client implementations:

NeuralTrustClient

The NeuralTrustClient connects to the NeuralTrust API service and provides the following methods:

from trusttest.clients import NeuralTrustClient

# Initialize client with your API token and target ID
client = NeuralTrustClient(
    token="your_api_token",
    target_id="your_target_id",
)

Configuration

The token and target ID are defined in your app settings. You can either:

• Pass them directly when initializing the client
• Set them as environment variables: NEURALTRUST_TOKEN and NEURALTRUST_TARGET_ID

Evaluation Scenarios

# Save an evaluation scenario
client.save_evaluation_scenario(evaluation_scenario)

# Load an evaluation scenario by ID
scenario = client.get_evaluation_scenario("scenario_id")

Test Sets

# Save a test set for a specific scenario
client.save_evaluation_scenario_test_set("scenario_id", test_set)

# Update an existing test set or create if it doesn't exist
client.upsert_evaluation_scenario_test_set("scenario_id", test_set)

# Load a test set for a specific scenario
test_set = client.get_evaluation_scenario_test_set("scenario_id")

Evaluation Results

# Save evaluation run results
client.save_evaluation_scenario_run(evaluation_run)

# Load evaluation run results for a scenario
results = client.get_evaluation_scenario_run("scenario_id")

Evaluators

# Save a custom evaluator with optional name and description
client.save_evaluator(evaluator, name="my_evaluator", description="Custom evaluator")

# Load an evaluator by name
evaluator = client.get_evaluator("my_evaluator")

Remote execution and metrics

After you have saved a scenario (and usually a test set), you can run it on the platform and poll status:

run = client.run_evaluation_scenario(
    scenario_id,
    refresh_responses=True,
    notify=False,
)
# or a single case:
# client.run_evaluation_scenario_test_case(scenario_id, test_case_id)

status = client.get_evaluation_run_status(
    scenario_id,
    evaluation_run_id=run.id,
    timeout=10000,
)
test_set_status = client.get_evaluation_test_set_status(scenario_id, timeout=10000)

metrics = client.get_evaluation_scenario_run_metrics(
    scenario_id,
    from_date=None,
    to_date=None,
)
target_metrics = client.get_target_total_metrics()

run_evaluation_scenario / run_evaluation_scenario_test_case return a RunResult. Metrics types are ScenarioMetrics and TargetMetrics.

FileSystemClient

The FileSystemClient stores the same artifacts as JSON under path (default trusttest_db).

from trusttest.clients import FileSystemClient

client = FileSystemClient(path="trusttest_db")
overview = client.get_overview()
# RedTeamingOverview: overall, language[], category[] (by sub_category), frameworks[]
print(overview.overall.percentage_passed)
print(overview.frameworks)

get_overview() rolls up local red-team results by language, subcategory, and the framework tags attached by catalog builders (EU AI Act, OWASP, MITRE ATLAS, ISO/IEC 42001).

Key Capabilities

All TrustTest clients support these core operations:

• Evaluation Scenarios: Save and retrieve evaluation scenario definitions
• Test Sets: Manage test sets associated with evaluation scenarios
• Evaluation Results: Persist and load evaluation run results
• Evaluators: Store custom evaluator configurations

Example Workflow

Here’s how to use the client in a typical evaluation workflow:

from trusttest.clients import FileSystemClient
from trusttest.evaluation_scenarios import EvaluationScenario
from trusttest.probes import TestSet

# Initialize a client
client = FileSystemClient()

# Create and save an evaluation scenario
scenario = EvaluationScenario(name="My Test", description="Testing functionality")
client.save_evaluation_scenario(scenario)

# Save a test set for the scenario
test_set = TestSet(test_cases=[...])
client.save_evaluation_scenario_test_set(scenario.id, test_set)

# Later, retrieve the scenario and its test set
loaded_scenario = client.get_evaluation_scenario(scenario.id)
loaded_test_set = client.get_evaluation_scenario_test_set(scenario.id)

# After running an evaluation, save the results
client.save_evaluation_scenario_run(evaluation_run)

The client abstraction ensures your evaluation artifacts are consistently stored and retrieved regardless of the underlying storage mechanism you choose.