Introduction to Catalyst

Catalyst is a lightweight API testing tool. It allows you to define and execute HTTP API tests through a TOML configuration file.

How Catalyst Works

Catalyst operates in a simple way:

1. You define your tests in a .catalyst/tests.toml file
2. You run the tests using the command-line interface
3. Catalyst executes the tests and reports the results

Current Features

Catalyst currently supports:

• HTTP Methods: GET, POST, PUT, DELETE, PATCH, HEAD, OPTIONS
• Request Configuration: Headers, query parameters, and JSON bodies
• Response Validation: Status code verification
• Variable Storage: Extract values from JSON responses using path notation
• Cookie Extraction: Store cookies from responses for use in subsequent requests
• Variable Substitution: Use stored variables in endpoints, headers, and request bodies
• Test Filtering: Run specific tests by name

Command-Line Interface

Catalyst provides a simple command-line interface with three main commands:

• catalyst run - Execute tests
• catalyst validate - Validate the test configuration file
• catalyst list - List available tests

Configuration File

The configuration file (.catalyst/tests.toml) consists of:

• A global [config] section with settings that apply to all tests
• Multiple [[tests]] sections, each defining a single test

In the following sections, we'll guide you through installing Catalyst and creating your first test configuration.

Installation

There are several ways to install Catalyst on your system.

Using Cargo

The recommended way to install Catalyst is through Cargo, Rust's package manager. If you already have Rust installed, you can install Catalyst with:

cargo install catalyst

This will download, compile, and install the latest version of Catalyst from crates.io.

Building from Source

If you prefer to build Catalyst from source, or if you want to use the latest development version, you can clone the repository and build it yourself:

# Clone the repository
git clone https://github.com/caffeidine/catalyst.git
cd catalyst

# Build and install
cargo install --path .

Verifying Installation

To verify that Catalyst has been installed correctly, run:

catalyst --version

You should see output showing the version number of Catalyst.

Next Steps

Now that you have Catalyst installed, you can proceed to the Getting Started section to learn how to create and run your first API tests.

Getting Started with Catalyst

This section will guide you through the basics of using Catalyst to test your APIs. We'll cover how to create a simple test configuration file, run your tests, and interpret the results.

Overview

Using Catalyst involves three main steps:

1. Create a test configuration file (.catalyst/tests.toml) that defines your API tests
2. Run the tests using the Catalyst command-line interface
3. Review the results to identify any issues with your API

Directory Structure

Catalyst expects your test configuration to be in a .catalyst directory in your project root:

your-project/
├── .catalyst/
│   └── tests.toml    # Your test configuration file
└── ...

Basic Example

Here's a simple example of a test configuration file that tests a REST API:

[config]
base_url = "https://api.example.com"
default_headers = { "Content-Type" = "application/json" }

[[tests]]
name = "Get Users"
method = "GET"
endpoint = "/users"
expected_status = 200

[[tests]]
name = "Create User"
method = "POST"
endpoint = "/users"
body = { "name" = "John Doe", "email" = "[email protected]" }
expected_status = 201

This configuration defines two tests:

1. A GET request to /users that should return a 200 status code
2. A POST request to /users with a JSON body that should return a 201 status code

Next Steps

In the following sections, we'll explore:

• Creating Your First Test - A step-by-step guide to creating your first test
• Running Tests - How to run your tests and interpret the results

Creating Your First Test

Creating your first test with Catalyst is straightforward. This guide will walk you through the process step by step.

1. Create the Directory Structure

First, create a .catalyst directory in your project root:

mkdir -p .catalyst

2. Create the Test File

Create a file named tests.toml inside the .catalyst directory:

touch .catalyst/tests.toml

3. Define the Global Configuration

Open the tests.toml file and add the global configuration:

[config]
base_url = "https://api.example.com"  # Replace with your API base URL
default_headers = { "Content-Type" = "application/json" }

4. Add Your First Test

Add a test definition to the file:

[[tests]]
name = "Simple GET Request"
method = "GET"
endpoint = "/status"
expected_status = 200

This test will make a GET request to https://api.example.com/status and expect a 200 status code in response.

5. Complete Example

Your complete tests.toml file should look like this:

[config]
base_url = "https://api.example.com"
default_headers = { "Content-Type" = "application/json" }

[[tests]]
name = "Simple GET Request"
method = "GET"
endpoint = "/status"
expected_status = 200

Next Steps

Now that you've created your first test, learn how to run it.

Running Tests

Once you've created your test configuration file, you can run your tests using the Catalyst command-line interface. This guide explains how to run tests and interpret the results.

Basic Test Execution

To run all tests defined in your .catalyst/tests.toml file, navigate to your project directory and run:

catalyst run

This will execute all tests in the order they are defined in the file.

CLI Options

Catalyst provides several command-line options to customize test execution:

Specifying a Custom Test File

By default, Catalyst looks for tests in .catalyst/tests.toml in your current directory. You can specify a different file using the --file option:

catalyst run --file /path/to/custom/tests.toml

Filtering Tests

You can run specific tests by using the --filter option:

catalyst run --filter "Login"

This will only run tests whose names contain the string "Login".

Verbose Output

For more detailed output, use the --verbose (or -v) flag:

catalyst run --verbose

This will show additional information such as response bodies and headers.

Disabling Colored Output

If you're running tests in an environment that doesn't support colored output, you can disable it:

catalyst run --disable-color

Complete CLI Reference

Here's a complete list of available commands and options:

CATALYST COMMANDS:
  run       Run API tests
    Options:
      -f, --filter <FILTER>    Filter by test name
      --disable-color          Disable colored output
      -v, --verbose            Enable verbose output
      --file <FILE>            Specify a custom test file path

  validate  Validate tests configuration
    Options:
      --file <FILE>            Specify a custom test file path

  list      List available tests
    Options:
      -v, --verbose            Enable detailed test information
      --file <FILE>            Specify a custom test file path

  help      Print this message or the help of the given subcommand(s)

Understanding Test Results

Catalyst provides clear feedback about test execution:

• [PASS] - The test succeeded (actual status code matches expected status code)
• [FAIL] - The test failed (actual status code differs from expected status code)

At the end of the test run, Catalyst will display a summary showing the total number of tests, how many passed, and how many failed.

Example Output

Running API tests...
[PASS] Simple GET Request      (200 Success)
[PASS] Create User             (201 Success)
[FAIL] Update User             (404 Not Found) (expected 200)

Failed tests:
- Update User

Test Summary:
Total: 3, Passed: 2, Failed: 1

Next Steps

Now that you know how to run tests, you can explore more configuration options in the Configuration section.

Configuration

Catalyst uses a TOML configuration file to define your API tests. This section explains the structure and options available in the configuration file.

Configuration File Location

By default, Catalyst looks for a configuration file at .catalyst/tests.toml in your project directory.

Configuration Structure

The configuration file has two main sections:

1. Global Configuration - Settings that apply to all tests
2. Test Definitions - Individual test cases

Global Configuration

The global configuration is defined in a [config] section at the top of the file:

[config]
base_url = "https://api.example.com"
auth_method = "Bearer"  # Optional
auth_token = "your-token"  # Optional
default_headers = { "Content-Type" = "application/json" }  # Optional

Available Options

Test Definitions

Test definitions are specified using [[tests]] sections:

[[tests]]
name = "Get Users"
method = "GET"
endpoint = "/users"
query_params = { "limit" = "10" }  # Optional
headers = { "X-Custom-Header" = "value" }  # Optional
body = { "key" = "value" }  # Optional
expected_status = 200
expected_body = { "success" = true }  # Optional
expected_headers = [["Content-Type", "application/json"]]  # Optional
store = { "$.token" = "auth_token" }  # Optional
get_cookie = { "session" = "session_cookie" }  # Optional
max_response_time = 500  # Maximum response time in milliseconds (Optional)

# Advanced assertions (Optional)
assertions = [
  # Exact match (same as expected_body)
  { type = "Exact", value = { "success" = true } },

  # Partial match - checks if response contains these fields
  { type = "Contains", value = { "data" = { "users" = [] } } },

  # Regex match on the entire response body
  { type = "Regex", value = "\\{.*\"success\"\\s*:\\s*true.*\\}" },

  # Regex match on a specific JSON path
  { type = "PathRegex", value = ["$.user.email", ".*@example\\.com"] }
]

Available Test Options

Advanced Assertions

Catalyst 0.2 introduces advanced assertions for more flexible response validation. You can use the assertions field to define multiple validation rules for a single test.

Types of Assertions

Exact Match

Validates that the response body exactly matches the expected value. This is equivalent to using expected_body.

assertions = [
  { type = "Exact", value = { "success" = true, "data" = { "id" = 1 } } }
]

Contains Match

Validates that the response body contains all the fields specified in the expected value, but may contain additional fields.

assertions = [
  { type = "Contains", value = { "success" = true } }
]

This will pass if the response contains {"success": true, "data": {...}} or any other JSON that includes the success field with a value of true.

Regex Match

Validates that the string representation of the response body matches the specified regular expression pattern.

assertions = [
  { type = "Regex", value = "\\{.*\"success\"\\s*:\\s*true.*\\}" }
]

Path Regex Match

Validates that a specific value in the response body, identified by a JSON path, matches the specified regular expression pattern.

assertions = [
  { type = "PathRegex", value = ["$.user.email", ".*@example\\.com"] }
]

This will pass if the value at $.user.email in the response matches the pattern .*@example\.com.

Combining Assertions

You can combine multiple assertions for a single test:

assertions = [
  { type = "Contains", value = { "success" = true } },
  { type = "PathRegex", value = ["$.data.id", "\\d+"] }
]

This test will pass only if both assertions are satisfied.

Response Time Validation

Catalyst 0.2 introduces response time validation and tracking. This feature allows you to:

1. Set maximum allowed response times for your API endpoints
2. Access and use the measured response times in subsequent tests

Setting Maximum Response Time

You can use the max_response_time field to specify the maximum allowed response time in milliseconds:

[[tests]]
name = "Fast API Response"
method = "GET"
endpoint = "/api/fast-endpoint"
expected_status = 200
max_response_time = 100  # Must respond within 100ms

If the API response takes longer than the specified time, the test will fail with a message indicating that the response time exceeded the maximum allowed time.

Automatic Response Time Tracking

Important: After each test execution, Catalyst automatically measures and stores the response time. This value is stored in a special variable named response_time_ms that becomes available to all subsequent tests.

This happens automatically for every test, whether or not you've specified a max_response_time value.

Using the Response Time Variable

You can reference the stored response time using the standard variable syntax {{response_time_ms}} in any subsequent test:

# First test - response time will be measured
[[tests]]
name = "Get User Profile"
method = "GET"
endpoint = "/users/profile"
expected_status = 200

# Second test - uses the response time from the first test
[[tests]]
name = "Log Response Time"
method = "POST"
endpoint = "/metrics/log"
body = {
  "endpoint" = "/users/profile",
  "response_time_ms" = "{{response_time_ms}}",
  "timestamp" = "{{current_timestamp}}"
}
expected_status = 200

This feature is particularly useful for:

• Performance logging and monitoring
• Creating tests that validate performance metrics
• Debugging performance issues across different API endpoints

Next Steps

For more detailed information about specific aspects of configuration, see:

• Test File Structure
• Security
• Complete Reference - Comprehensive reference with examples for all features

Test File Structure

Catalyst uses a TOML configuration file to define your API tests. This file should be located at .catalyst/tests.toml in your project directory.

Basic Structure

The test file has two main sections:

1. Global configuration ([config]) - Contains settings that apply to all tests
2. Test definitions ([[tests]]) - Contains individual test cases

Here's a simple example:

# Global settings
[config]
base_url = "https://api.example.com"
default_headers = { "Content-Type" = "application/json" }

# Test definitions
[[tests]]
name = "Get Users"
method = "GET"
endpoint = "/users"
expected_status = 200

[[tests]]
name = "Create User"
method = "POST"
endpoint = "/users"
body = { "name" = "John Doe", "email" = "[email protected]" }
expected_status = 201

File Organization

You can organize your tests in any order within the file. Catalyst will execute the tests in the order they are defined, which is important when tests depend on each other (for example, when one test stores a variable that another test uses).

Comments

You can add comments to your test file using the # character:

# This is a comment
[config]
base_url = "https://api.example.com"  # This is also a comment

Comments are useful for documenting your tests and explaining their purpose or any special considerations.

Multiple Test Files

Currently, Catalyst supports a single test file (.catalyst/tests.toml). If you need to organize your tests into multiple files, you might consider using symbolic links or a build process that combines multiple TOML files into a single test file.

Next Steps

For more information, see:

• Security - Details on authentication and security configuration
• Complete Reference - Comprehensive reference with examples for all features, including advanced assertions and response time validation

Security Configuration

This section explains how to configure authentication and security-related aspects when testing APIs with Catalyst.

Authentication Methods

Catalyst supports several authentication methods that can be configured in the global configuration section or per test.

Cookie-based Authentication

For cookie-based authentication, you can use the auth_method set to "Cookie":

[config]
base_url = "https://api.example.com"
auth_method = "Cookie"
default_headers = { "Content-Type" = "application/json" }

The typical workflow for cookie-based authentication is:

1. Perform a login request
2. Extract the session cookie
3. Use the cookie in subsequent requests

Example:

[[tests]]
name = "Login"
method = "POST"
endpoint = "/auth/login"
body = { "username" = "test", "password" = "password" }
expected_status = 200
get_cookie = { "session_id" = "session_cookie" }

[[tests]]
name = "Access Protected Resource"
method = "GET"
endpoint = "/protected"
headers = { "Cookie" = "session_id={{session_cookie}}" }
expected_status = 200

Bearer Token Authentication

To use Bearer token authentication:

[config]
base_url = "https://api.example.com"
auth_method = "Bearer"
auth_token = "your-jwt-token"

This will add an Authorization: Bearer your-jwt-token header to all requests.

Handling API Keys

You can include API keys in headers or query parameters:

[[tests]]
name = "API Key in Header"
method = "GET"
endpoint = "/protected"
headers = { "X-API-Key" = "your-api-key" }
expected_status = 200

[[tests]]
name = "API Key in Query Parameter"
method = "GET"
endpoint = "/protected"
query_params = { "api_key" = "your-api-key" }
expected_status = 200

Chaining Authentication

You can chain authentication by extracting tokens from responses:

[[tests]]
name = "Create User Token"
method = "POST"
endpoint = "/user/tokens"
headers = { "Cookie" = "{{session_cookie}}" }
expected_status = 200
store = { "data.token" = "token" }

[[tests]]
name = "Access Protected Resource"
method = "GET"
endpoint = "/protected"
headers = { "x-api-token" = "{{token}}" }
expected_status = 200

Next Steps

For more information about test configuration and advanced features, see:

• Test File Structure - Details on the overall structure of test files
• Complete Reference - Comprehensive reference with examples for all features, including advanced assertions and response time validation

Test Reference

This page serves as a technical reference for creating tests with Catalyst.

TOML Syntax Guide

Catalyst uses TOML for test configuration. Understanding the TOML syntax is important for writing correct and maintainable tests.

Tables vs Arrays of Tables

TOML has two main ways to define structures:

1. Tables [table]: Define a single named table (object)
2. Arrays of Tables [[table]]: Define an element in an array of tables (array of objects)

When to Use Each Syntax

• Use [[tests]] for each test because a test file can contain multiple tests
• Use [tests.body], [tests.headers], etc. for single objects within a test
• Use [[tests.assertions]] for assertions because a test can have multiple assertions
• Use [tests.assertions.value] for the value of a specific assertion

Example

[[tests]]                # First test (element in an array)
name = "Example Test"

[tests.body]            # Request body (single object)
name = "Test User"

[[tests.assertions]]     # First assertion (element in an array)
type = "Contains"

[tests.assertions.value] # Value of this assertion (single object)
id = 123

This distinction is important for correctly representing data structures with different cardinalities in your tests.

Basic Structure

[config]
base_url = "https://api.example.com"
default_headers = { "Content-Type" = "application/json" }

[[tests]]
name = "Test Name"
method = "GET"
endpoint = "/path"
expected_status = 200

Global Configuration Options

[config]
base_url = "https://api.example.com"
default_headers = {
  "Content-Type" = "application/json",
  "Accept" = "application/json"
}
auth_method = "Bearer"
auth_token = "your-token-here"

Test Definition Options

[[tests]]
name = "Create User"
method = "POST"
endpoint = "/users"
query_params = { "source" = "api" }
headers = { "X-Custom-Header" = "value" }
body = {
  "name" = "John Doe",
  "email" = "[email protected]",
  "roles" = ["user", "admin"]
}
expected_status = 201
expected_body = { "success" = true }
expected_headers = [["Content-Type", "application/json"]]
max_response_time = 500

Response Validation

Using expected_body vs. assertions

expected_body Example

# Using expected_body for exact match validation
[[tests]]
name = "Get User"
method = "GET"
endpoint = "/users/1"
expected_status = 200
expected_body = {
  "id" = 1,
  "name" = "John Doe",
  "email" = "[email protected]",
  "created_at" = "2023-01-01T00:00:00Z"
}

assertions Example

# Using assertions for more flexible validation
[[tests]]
name = "Get User with Assertions"
method = "GET"
endpoint = "/users/1"
expected_status = 200

# Assertions using table array syntax for better readability
[[tests.assertions]]
type = "Contains"
value = { "id" = 1, "name" = "John Doe" }

[[tests.assertions]]
type = "PathRegex"
value = ["$.email", "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"]

Assertion Types

Assertion Examples

[[tests]]
name = "Advanced Assertions Example"
method = "GET"
endpoint = "/users/1"
expected_status = 200

# Example 1: Exact match (equivalent to expected_body)
[[tests.assertions]]
type = "Exact"

# Using inline table for simple values
[tests.assertions.value]
id = 1
name = "John Doe"
email = "[email protected]"

# Example 2: Contains match (partial validation)
[[tests.assertions]]
type = "Contains"

[tests.assertions.value]
id = 1
roles = ["user"]

# Example 3: Regex match on entire response
[[tests.assertions]]
type = "Regex"
value = ".*\"email\":\\s*\"[email protected]\".*"

# Example 4: PathRegex match on specific field
[[tests.assertions]]
type = "PathRegex"
value = ["$.email", ".*@example\\.com"]

# Example 5: PathRegex for numeric validation
[[tests.assertions]]
type = "PathRegex"
value = ["$.id", "^[0-9]+$"]

Using Variables in Assertions and Expected Body

As of v0.2, Catalyst supports using variables in both expected_body and assertions, allowing for more dynamic and powerful tests.

# Test 1: Create a user and store the ID
[[tests]]
name = "Create User"
method = "POST"
endpoint = "/users"
body = { "name" = "John Doe", "email" = "[email protected]" }
expected_status = 201
store = { "$.id" = "user_id" }

# Test 2: Verify the user details with the stored ID
[[tests]]
name = "Get User Details"
method = "GET"
endpoint = "/users/{{user_id}}"
expected_status = 200

# Use the stored ID in expected_body with inline table
[tests.expected_body]
id = "{{user_id}}"  # Variable in expected_body
name = "John Doe"

# Test 3: Alternative using assertions
[[tests]]
name = "Verify User with Assertions"
method = "GET"
endpoint = "/users/{{user_id}}"
expected_status = 200

# Use variable in Contains assertion
[[tests.assertions]]
type = "Contains"
value = { "id" = "{{user_id}}" }

# Use variable in PathRegex assertion
[[tests.assertions]]
type = "PathRegex"
value = ["$.id", "^{{user_id}}$"]

Variable Storage and Usage

Storing Variables

[[tests]]
name = "Extract and Store Values"
method = "POST"
endpoint = "/auth/login"
body = { "username" = "user", "password" = "pass" }
expected_status = 200

# Extract and store values from JSON body using inline table
[tests.store]
"$.token" = "auth_token"        # Stores value at $.token in auth_token
"$.user.id" = "user_id"         # Stores value at $.user.id in user_id
"$.expires_at" = "token_expiry"  # Stores value at $.expires_at in token_expiry

# Extract and store cookies using inline table
[tests.get_cookie]
"session" = "session_id"        # Stores session cookie value in session_id
"XSRF-TOKEN" = "csrf_token"      # Stores XSRF-TOKEN cookie value in csrf_token

Using Stored Variables

[[tests]]
name = "Use Stored Variables"
method = "GET"
endpoint = "/users/{{user_id}}"  # Uses user_id variable in URL
expected_status = 200

# Headers using inline table or subtable syntax
[tests.headers]
"Authorization" = "Bearer {{auth_token}}"  # Uses auth_token in header
"X-CSRF-Token" = "{{csrf_token}}"         # Uses csrf_token in header

# Variables can be used in any part of the test
[tests.body]
token = "{{auth_token}}"
session = "{{session_id}}"

Response Time Validation

Catalyst automatically measures the response time for each test and makes it available as a variable.

[[tests]]
name = "Response Time Validation"
method = "GET"
endpoint = "/fast-endpoint"
expected_status = 200
max_response_time = 100  # Response must be received in less than 100ms

# The response_time_ms variable is automatically created after each test
# and can be used in subsequent tests
[[tests]]
name = "Log Response Time"
method = "POST"
endpoint = "/metrics/log"
expected_status = 200

# Body using inline table for simple structure
body = { "previous_response_time" = "{{response_time_ms}}" }

Test Chaining Example

# Test 1: Create a resource
[[tests]]
name = "Create Resource"
method = "POST"
endpoint = "/resources"
body = { "name" = "New Resource" }
expected_status = 201
store = { "$.id" = "resource_id" }  # Store ID for next test

# Test 2: Get the created resource
[[tests]]
name = "Get Created Resource"
method = "GET"
endpoint = "/resources/{{resource_id}}"  # Use stored ID
expected_status = 200

[[tests.assertions]]
type = "Contains"
value = { "name" = "New Resource" }

# Test 3: Update the resource
[[tests]]
name = "Update Resource"
method = "PUT"
endpoint = "/resources/{{resource_id}}"
body = { "name" = "Updated Resource" }
expected_status = 200

# Test 4: Verify the update
[[tests]]
name = "Verify Update"
method = "GET"
endpoint = "/resources/{{resource_id}}"
expected_status = 200

[[tests.assertions]]
type = "Contains"
value = { "name" = "Updated Resource" }

# Test 5: Delete the resource
[[tests]]
name = "Delete Resource"
method = "DELETE"
endpoint = "/resources/{{resource_id}}"
expected_status = 204