Skip to content
Talaia

Writing a suite

A suite is a directory under projects/<project>/<suite>/ (any nesting depth) containing a suite.yml and plain pytest test_*.py files. The suite id is the path relative to projects/ — e.g. bikecrm/backend.

Adding a suite

Section titled “Adding a suite”
  1. Copy projects/_template/ to projects/<project>/<suite>/.
  2. Edit suite.yml (name, schedule, required env vars) and set enabled: true.
  3. Write test_*.py pytest tests. Fast (<60s), independent, idempotent.
  4. Add the suite’s env vars to .env.example (documented, with placeholders) and the server’s .env (real values).
  5. uv run talaia run <project>/<suite> to verify, then redeploy.

suite.yml reference

Section titled “suite.yml reference”

All keys except schedule have defaults (shown below):

name: Template suite            # display name in the UI (default: suite id)
enabled: false                  # disabled suites are skipped by `run --all`, crontab and the UI
schedule: "*/15 * * * *"        # cron expression consumed by `talaia crontab` (supercronic)
timeout_seconds: 300            # hard kill for the whole pytest run
retries: 1                      # in-run retries before the run counts as failed
retry_delay_seconds: 30
alert:
  cooldown_minutes: 60          # min gap between repeat alerts while the suite stays down


env:
  required: []                  # env vars that must be set; missing -> run recorded as "error"
    # - MYPROJECT_BASE_URL
    # - MYPROJECT_USERNAME
    # - MYPROJECT_PASSWORD


# heartbeat_env: HC_URL_MYPROJECT   # env var holding a healthchecks.io ping URL (dead-man switch)
# version_url: https://api.example.com/api/version/   # GET'd after each run; tagged on runs/alerts

Two optional extras:

  • heartbeat_env names an env var holding a healthchecks.io ping URL. Talaia pings it after every run — the plain URL on a pass, the /fail endpoint on a failure — so healthchecks.io alerts you both when the suite goes red and when Talaia stops running at all.
  • version_url is fetched (GET) after each run; a version/release field from a JSON response (or the plain response body) is recorded on the run and included in alerts, so an alert can answer “which deploy caused this?”.

Conventions

Section titled “Conventions”
  • A suite is a leaf directory. Never put a suite.yml in an ancestor of another suite — the parent’s pytest run would collect the child’s tests.
  • Black box only. Suites run with cwd = the suite directory and must not import anything from the monitored project. API and browser access only.
  • Fast, independent, idempotent. Keep the whole suite under ~60 seconds; the entire run is killed at timeout_seconds.
  • Env var naming: <PROJECT>_<THING> (e.g. BIKECRM_BASE_URL). Declare them in env.required so a misconfigured server shows up as an explicit error run instead of a confusing failure.
  • Stagger schedules between suites of the same project (e.g. */10 vs 5-59/15) so they do not hammer the app at the same instant.

Safety rules (these tests hit production)

Section titled “Safety rules (these tests hit production)”
  • Test accounts only. Every suite authenticates against a dedicated test account/tenant. Never point a suite at a real customer account.
  • Prefix and clean up. Any entity a test creates must be named with a SMOKE- prefix and deleted by the same test when possible.
  • Gate every write flow. Before exercising any write against production, assert the expected sandbox conditions explicitly (for example: the test business slug is the expected one and irreversible modes are disabled). A suite must refuse to run against anything else.
  • Quarantine irreversible flows. Flows that leave undeletable data belong in low-frequency (e.g. daily) suites, never in the every-10-minutes ones.
  • No secrets in the repo. Credentials come from env (.env on the server, gitignored). Document every new var in .env.example with a placeholder.