Test Reference

This page serves as a technical reference for creating tests with Catalyst.

TOML Syntax Guide

Catalyst uses TOML for test configuration. Understanding the TOML syntax is important for writing correct and maintainable tests.

Tables vs Arrays of Tables

TOML has two main ways to define structures:

1. Tables [table]: Define a single named table (object)
2. Arrays of Tables [[table]]: Define an element in an array of tables (array of objects)

When to Use Each Syntax

• Use [[tests]] for each test because a test file can contain multiple tests
• Use [tests.body], [tests.headers], etc. for single objects within a test
• Use [[tests.assertions]] for assertions because a test can have multiple assertions
• Use [tests.assertions.value] for the value of a specific assertion

Example

[[tests]]                # First test (element in an array)
name = "Example Test"

[tests.body]            # Request body (single object)
name = "Test User"

[[tests.assertions]]     # First assertion (element in an array)
type = "Contains"

[tests.assertions.value] # Value of this assertion (single object)
id = 123

This distinction is important for correctly representing data structures with different cardinalities in your tests.

Basic Structure

[config]
base_url = "https://api.example.com"
default_headers = { "Content-Type" = "application/json" }

[[tests]]
name = "Test Name"
method = "GET"
endpoint = "/path"
expected_status = 200

Global Configuration Options

[config]
base_url = "https://api.example.com"
default_headers = {
  "Content-Type" = "application/json",
  "Accept" = "application/json"
}
auth_method = "Bearer"
auth_token = "your-token-here"

Test Definition Options

[[tests]]
name = "Create User"
method = "POST"
endpoint = "/users"
query_params = { "source" = "api" }
headers = { "X-Custom-Header" = "value" }
body = {
  "name" = "John Doe",
  "email" = "[email protected]",
  "roles" = ["user", "admin"]
}
expected_status = 201
expected_body = { "success" = true }
expected_headers = [["Content-Type", "application/json"]]
max_response_time = 500

Response Validation

Using expected_body vs. assertions

expected_body Example

# Using expected_body for exact match validation
[[tests]]
name = "Get User"
method = "GET"
endpoint = "/users/1"
expected_status = 200
expected_body = {
  "id" = 1,
  "name" = "John Doe",
  "email" = "[email protected]",
  "created_at" = "2023-01-01T00:00:00Z"
}

assertions Example

# Using assertions for more flexible validation
[[tests]]
name = "Get User with Assertions"
method = "GET"
endpoint = "/users/1"
expected_status = 200

# Assertions using table array syntax for better readability
[[tests.assertions]]
type = "Contains"
value = { "id" = 1, "name" = "John Doe" }

[[tests.assertions]]
type = "PathRegex"
value = ["$.email", "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"]

Assertion Types

Assertion Examples

[[tests]]
name = "Advanced Assertions Example"
method = "GET"
endpoint = "/users/1"
expected_status = 200

# Example 1: Exact match (equivalent to expected_body)
[[tests.assertions]]
type = "Exact"

# Using inline table for simple values
[tests.assertions.value]
id = 1
name = "John Doe"
email = "[email protected]"

# Example 2: Contains match (partial validation)
[[tests.assertions]]
type = "Contains"

[tests.assertions.value]
id = 1
roles = ["user"]

# Example 3: Regex match on entire response
[[tests.assertions]]
type = "Regex"
value = ".*\"email\":\\s*\"[email protected]\".*"

# Example 4: PathRegex match on specific field
[[tests.assertions]]
type = "PathRegex"
value = ["$.email", ".*@example\\.com"]

# Example 5: PathRegex for numeric validation
[[tests.assertions]]
type = "PathRegex"
value = ["$.id", "^[0-9]+$"]

Using Variables in Assertions and Expected Body

As of v0.2, Catalyst supports using variables in both expected_body and assertions, allowing for more dynamic and powerful tests.

# Test 1: Create a user and store the ID
[[tests]]
name = "Create User"
method = "POST"
endpoint = "/users"
body = { "name" = "John Doe", "email" = "[email protected]" }
expected_status = 201
store = { "$.id" = "user_id" }

# Test 2: Verify the user details with the stored ID
[[tests]]
name = "Get User Details"
method = "GET"
endpoint = "/users/{{user_id}}"
expected_status = 200

# Use the stored ID in expected_body with inline table
[tests.expected_body]
id = "{{user_id}}"  # Variable in expected_body
name = "John Doe"

# Test 3: Alternative using assertions
[[tests]]
name = "Verify User with Assertions"
method = "GET"
endpoint = "/users/{{user_id}}"
expected_status = 200

# Use variable in Contains assertion
[[tests.assertions]]
type = "Contains"
value = { "id" = "{{user_id}}" }

# Use variable in PathRegex assertion
[[tests.assertions]]
type = "PathRegex"
value = ["$.id", "^{{user_id}}$"]

Variable Storage and Usage

Storing Variables

[[tests]]
name = "Extract and Store Values"
method = "POST"
endpoint = "/auth/login"
body = { "username" = "user", "password" = "pass" }
expected_status = 200

# Extract and store values from JSON body using inline table
[tests.store]
"$.token" = "auth_token"        # Stores value at $.token in auth_token
"$.user.id" = "user_id"         # Stores value at $.user.id in user_id
"$.expires_at" = "token_expiry"  # Stores value at $.expires_at in token_expiry

# Extract and store cookies using inline table
[tests.get_cookie]
"session" = "session_id"        # Stores session cookie value in session_id
"XSRF-TOKEN" = "csrf_token"      # Stores XSRF-TOKEN cookie value in csrf_token

Using Stored Variables

[[tests]]
name = "Use Stored Variables"
method = "GET"
endpoint = "/users/{{user_id}}"  # Uses user_id variable in URL
expected_status = 200

# Headers using inline table or subtable syntax
[tests.headers]
"Authorization" = "Bearer {{auth_token}}"  # Uses auth_token in header
"X-CSRF-Token" = "{{csrf_token}}"         # Uses csrf_token in header

# Variables can be used in any part of the test
[tests.body]
token = "{{auth_token}}"
session = "{{session_id}}"

Response Time Validation

Catalyst automatically measures the response time for each test and makes it available as a variable.

[[tests]]
name = "Response Time Validation"
method = "GET"
endpoint = "/fast-endpoint"
expected_status = 200
max_response_time = 100  # Response must be received in less than 100ms

# The response_time_ms variable is automatically created after each test
# and can be used in subsequent tests
[[tests]]
name = "Log Response Time"
method = "POST"
endpoint = "/metrics/log"
expected_status = 200

# Body using inline table for simple structure
body = { "previous_response_time" = "{{response_time_ms}}" }

Test Chaining Example

# Test 1: Create a resource
[[tests]]
name = "Create Resource"
method = "POST"
endpoint = "/resources"
body = { "name" = "New Resource" }
expected_status = 201
store = { "$.id" = "resource_id" }  # Store ID for next test

# Test 2: Get the created resource
[[tests]]
name = "Get Created Resource"
method = "GET"
endpoint = "/resources/{{resource_id}}"  # Use stored ID
expected_status = 200

[[tests.assertions]]
type = "Contains"
value = { "name" = "New Resource" }

# Test 3: Update the resource
[[tests]]
name = "Update Resource"
method = "PUT"
endpoint = "/resources/{{resource_id}}"
body = { "name" = "Updated Resource" }
expected_status = 200

# Test 4: Verify the update
[[tests]]
name = "Verify Update"
method = "GET"
endpoint = "/resources/{{resource_id}}"
expected_status = 200

[[tests.assertions]]
type = "Contains"
value = { "name" = "Updated Resource" }

# Test 5: Delete the resource
[[tests]]
name = "Delete Resource"
method = "DELETE"
endpoint = "/resources/{{resource_id}}"
expected_status = 204