testing orchestrator

textkit

A small text-analysis library, and the reason it exists: a pipeline that demonstrates what the CI/CD orchestrator actually does.

The code is deliberately modest — tokenise text, count words, render a report. The interesting part is the shape of .orchestrator/actions.yml.

What the pipeline demonstrates

prepare      quality        test              package     publish
┌────────┐   ┌──────────┐   ┌───────────────┐ ┌───────┐   ┌─────────┐
│ corpus │   │ syntax   │   │ test-tokenize │ │       │   │ smoke   │
│        │   │ style    │   │ test-stats    │ │package│   │ summary │
│        │   │ imports  │   │ test-report   │ │       │   │         │
└────────┘   └──────────┘   └───────────────┘ └───────┘   └─────────┘

Ten jobs, five stages:

1. Four jobs start at once. corpus and the three quality gates declare no needs, so the scheduler releases them together. With two workers online you can watch them claimed by different machines.

2. Three test suites run in parallel, on two machines. One per module, so they execute side by side rather than one machine three times. Each needs corpus.json — an artifact the other machine produced.

   The split is deliberate, and worth understanding. The scheduler prefers to run a job on whichever worker produced its dependencies, since that machine's files are already warm. Left alone, all three suites would follow corpus onto one worker and run one after another. So corpus is pinned to the server, and test-tokenize is tagged heavy to send it to the build machine. The other two follow the corpus. Two machines, both busy.

3. Every job reports its host. tools/machine.py runs first in each job and prints the hostname, architecture, and the branch and commit the coordinator injected. Reading three different hostnames in three parallel logs is the clearest evidence the system is distributed.

4. The results fan back in. Each suite writes results/<suite>.json naming the machine it ran on. summary reads all three — which only works because artifacts travel through the coordinator — and prints:

   suite      tests  fail    sec   ran on
   ------------------------------------------
   report         6     0    2.9   laptop-mac
   stats         10     0    2.9   nxtcloud-m
   tokenize      10     0    2.7   laptop-mac
   ------------------------------------------
   total         26     0
   
   3 suites across 2 machine(s): laptop-mac, nxtcloud-m
   

5. Nothing is packaged until everything passes. package needs all six upstream jobs; smoke then unpacks what was built and runs the CLI, which catches a package that builds but does not import.

What each machine needs

The build machine (a laptop) must advertise the heavy tag, and the server's worker must be named worker-1 to match the pin:

cargo run --release -- worker --name beefy-1 --tags heavy,docker \
  --coordinator https://ci.example.com

If no heavy worker is online, test-tokenize does not fail — it waits in the queue until one appears, and the run finishes when it does.

Running it

Register this repo in the dashboard, then push. Or trigger it directly:

curl -X POST https://ci.example.com/api/pipelines/trigger \
  -H 'Content-Type: application/json' -d '{"repo": "orchestrator-run-test"}'

Locally, without the orchestrator:

python3 tools/gen_corpus.py --out corpus.json --documents 1500
python3 tools/run_suite.py tokenize
python3 tools/style_check.py
PYTHONPATH=src python3 -m textkit corpus.json

Timing

Roughly 1–2 minutes end to end, most of it per-job overhead: each job gets a fresh workspace cloned from the commit, plus artifact transfer.

The test suites sweep the corpus DEMO_PASSES times (default 20) — about 3 seconds each on a laptop, longer on a small server. Repetition is not padding: a tokeniser holding state, or an unstable dict iteration, only shows up across repeated sweeps. It is also the dial for how long the demo takes:

DEMO_PASSES=60 python3 tools/run_suite.py stats     # slower, more visible
DEMO_PASSES=1  python3 tools/run_suite.py stats     # quick check

Corpus size is the other knob (--documents). At 1500 it is a ~5 MB artifact, big enough that moving it between machines is real work and small enough not to dominate the run.

No dependencies, on purpose

Standard library only — unittest, not pytest; a hand-written style checker, not a linter. A demo that pip-installs while it is being watched is a demo that can fail on somebody else's network. The gates are still real: style_check.py caught two over-length lines in this repo's own tooling the first time it ran.

Layout

src/textkit/      the library: tokenize, stats, report, cli
tests/            one suite per module, plus shared corpus loading
tools/            gen_corpus, run_suite, style_check, collect_results, machine
.orchestrator/    actions.yml — the pipeline