# RODMENA CI (the conductor) — pipeline authoring guide for LLMs

> CI for RODMENA repositories at https://ci.rodmena.co.uk. You are probably here
> because you were asked to "add CI" or "add a deployment pipeline" to a
> repository. This format is NOT GitHub Actions, and the two differences that
> matter most are the two that will silently break a deployment. Read the hard
> rules immediately below before writing any YAML; if you are adding
> deployment, also read "Deployment", which is where the answer to that request
> actually lives.

The pipeline lives at `.rodmena/ci.yml` in the repository being built, read at
the commit under test (for pull requests: at the merge result).

## Hard rules — if you read nothing else, read these

1. **Never put a deploy, publish, release or push-to-production job in this
   file.** It would run on every branch and every pull request. Deployment is
   done by an agent on the deploy host, not by CI. See "Deployment".
2. **There is no `if:`, no `on:`, no `branches:`, no expression language.**
   Branch selection is per-repository and set by an operator; pull requests
   always build.
3. **Unknown keys are a hard error.** `on`, `uses`, `runs-on`, `strategy`,
   `with`, `cache`, `if` — every Actions habit is rejected outright.
4. **Memory is capped at 2048 MB and 4 CPUs.** Asking for more is silently
   reduced, not refused, and your job dies of OOM with no explanation.
5. **Required keys:** `version: 1` at the top; `name`, `image` and `steps` on
   every job. Everything else is optional.
6. **A job is one container.** Steps share it; jobs share nothing. Use
   `artifacts` to pass files between jobs.
7. **If the user asked you to deploy**, your answer is a test-only pipeline
   *plus* an explanation that deployment needs a host-side bot and an operator.
   Do not silently drop the deployment half of their request.

---

## WARNING 1 — there is no branch filter and no `if:`

**Every job in the file runs on every build, including every pull request.**

There is no `on:`, no `if:`, no `branches:`, no `when:`, no expression language.
Branch selection is per-REPOSITORY, configured by an operator
(`repositories.branches`, default `["main"]`), not per-job. Pull requests on the
same repository always build, whatever that setting says.

So do NOT write this — it is the single most common wrong answer:

```yaml
# WRONG. `if` is not a key; unknown keys are a hard error, so this one at
# least fails loudly.
  deploy:
    name: "Deploy"
    if: github.ref == 'refs/heads/main'
```

And do NOT write this either — it is worse, because it *validates*:

```yaml
# WRONG, AND IT WILL RUN. This deploys from every branch and every pull
# request, with production credentials, to production.
  deploy:
    name: "Deploy"
    needs: [tests]
    image: containers.rodmena.co.uk/rodmena/ci-python:3.12
    secrets: [DEPLOY_SSH_KEY]
    steps:
      - name: Deploy
        run: ./deploy.sh
```

## WARNING 2 — the pipeline does not deploy. A bot does.

The supported deployment pattern is **poll-and-deploy**:

1. The pipeline builds and tests. That is all it does.
2. A small agent **on the deployment host** polls the conductor's read API for
   a pipeline that is `succeeded` on the deployment branch.
3. That agent runs the deploy, using credentials that live on the host and are
   never given to a CI job.

This is not a workaround, it is the design. The CI runs untrusted repository
code in a container; the deploy host holds the production credentials. Keeping
them apart means a compromised test dependency cannot reach production. It also
gives you the branch gate the format deliberately lacks — the bot decides.

See "Deployment" below for a working poller.

---

## Execution model

**One job = one container.** Steps run in sequence *inside that one container*,
so the workspace, installed dependencies and running services persist from step
to step **within a job**. Nothing persists **between jobs**: each starts from its
image plus a fresh checkout.

- `uv sync` in step 1 is still installed in step 5. This is the intended pattern.
- `needs` gives ordering, **not** shared state. To pass files between jobs, use
  `artifacts`.
- There is **no dependency cache** (a deliberate follow-on). Every job pays a
  checkout and an install, so prefer fewer, fatter jobs over many thin ones.

---

## Minimal valid pipeline

```yaml
version: 1

jobs:
  test:
    name: "Tests"
    image: containers.rodmena.co.uk/rodmena/ci-python:3.12
    steps:
      - name: Install
        run: uv sync --frozen
      - name: Tests
        run: uv run pytest -q
```

`version`, and for each job `name`, `image` and `steps`, are the only required
keys. **Unknown keys anywhere are a hard error**, not a warning — a mistyped key
that was silently ignored is how a job stops running without anyone noticing.

---

## Reference

### Top level

| Key | Required | Meaning |
|---|---|---|
| `version` | yes | must be `1` |
| `jobs` | yes | map of job id → job. Job ids match `[A-Za-z0-9._-]{1,64}`, max 50 jobs after matrix expansion |
| `concurrency` | no | `group` (required, ≤128 chars) and `cancel_in_progress` (default `false`) |

`concurrency.group` may use `${{ ref }}`, `${{ branch }}`, `${{ repo }}`.
`ci-${{ ref }}` gives each PR and branch its own group, which is almost always
what you want. Groups sharing a name never run concurrently; with
`cancel_in_progress: true` a new pipeline cancels the running one.

### Job

| Key | Required | Meaning |
|---|---|---|
| `name` | yes | **the check-run name**, verbatim. Branch protection matches this string. Must be unique after matrix expansion. `ci/definition` is reserved |
| `image` | yes | container image. Must provide `sh`, `git`, `base64`, plus `curl` if `artifacts` is used |
| `steps` | yes | ordered list, max 30 |
| `needs` | no | job ids that must succeed first. Acyclic; a failed or skipped dependency concludes this job `skipped` |
| `matrix` | no | map of name → list of values. Max 8 keys, max 64 combinations |
| `services` | no | `postgres` and/or `redis` only |
| `env` | no | map of string → string, max 100 keys |
| `secrets` | no | secret names, `[A-Z][A-Z0-9_]*`, must be scoped to this repository |
| `timeout_minutes` | no | default 30, wall clock for the whole job |
| `resources.cpu` | no | default 2, **max 4** |
| `resources.memory_mb` | no | default 2048, **max 2048** |
| `artifacts` | no | paths collected after the last step |

### Step

| Key | Required | Meaning |
|---|---|---|
| `run` | yes | shell command, executed with `sh` |
| `name` | no | shown in the summary; defaults to a truncated `run` |
| `continue_on_error` | no | default `false`. Records the failure and continues |

### Where `${{ }}` works — and nowhere else

| Context | Available |
|---|---|
| `concurrency.group` | `ref`, `branch`, `repo` |
| job `image`, `env` values, step `run` | `matrix.<key>` |

This is a substitution table, not an expression language. A `${{ }}` anywhere
else, or naming anything else, is a rejected definition — never an empty string.

---

## The memory ceiling: 2048 MB, and asking for more makes it worse

The execution layer caps a container at **2048 MB and 4 CPUs**. Requesting more
is **not an error and not a bigger job** — the request is *silently reduced*, and
your tests die of OOM with nothing in the output explaining why. The conductor
prefixes the check-run summary when this happens, but the ceiling itself only
moves by upgrading the execution tier.

`memory_mb` is therefore useful for declaring you need *less* than the default,
which helps the job schedule sooner on a busy host. It cannot buy you more.
`services: [postgres, redis]` share this budget with your build.

---

## Services

```yaml
    services: [postgres, redis]
```

Services run **inside the job's container**, not as sidecars. Injected
automatically:

| Service | Variable |
|---|---|
| `postgres` | `DATABASE_URL=postgresql://ci:ci@127.0.0.1:5432/ci` |
| `redis` | `REDIS_URL=redis://127.0.0.1:6379/0` |

Empty and identical on every job, persists across that job's steps, dies with
the container. Requires an image that ships them — the maintained
`containers.rodmena.co.uk/rodmena/ci-python:*` images do.

---

## Secrets

Named per job, injected as environment variables, scoped to one repository by an
operator. A secret not scoped to the repository **fails the job before any
container starts**, naming it — it never runs with the variable merely unset.
Values are redacted from logs, check-run output, API responses and artifacts.

**The job is the isolation boundary**: every step in a job can read every secret
that job names — on every branch and every pull request that builds. Deploy
credentials (`DEPLOY_SSH_KEY`, registry passwords, cloud tokens) therefore do
**not** belong here at all; they live on the deploy host. See "Deployment". Put credentials in their own job so that hundreds of lines of
test tooling never share a process tree with them.

Fork pull requests do not build at all in this phase, so secrets are never
exposed to them.

---

## Maintained images

| Image | Contains |
|---|---|
| `containers.rodmena.co.uk/rodmena/ci-python:3.12` | uv, git, curl, PostgreSQL 16, Redis |
| `containers.rodmena.co.uk/rodmena/ci-node:22` | Node 22, npm, git, curl |

Any public image works if it provides `sh`, `git`, `base64` (and `curl` for
artifacts), but only the maintained `containers.rodmena.co.uk/rodmena/ci-*` images get the
writable-rootfs sandbox that `services:` needs. Prefer them.

**Use the `containers.rodmena.co.uk` refs, not `rodmenalimited/`.** The maintained
images moved to the house registry and the execution host no longer holds a Docker
Hub credential, so an old `rodmenalimited/ci-*` ref does not get the writable-rootfs
sandbox. **The failure is silent**: the job runs hardened instead, and a `services:
[postgres]` job fails with

```
chown: changing ownership of '/var/lib/postgresql': Read-only file system
```

which reads as a permissions bug inside your container rather than as a sandbox
downgrade. If you see that, check your `image:` ref first.

---

## Deployment

The pipeline proves the commit is good. Something else acts on that.

### If you were asked to "deploy to production when main is green"

That is the most common request, and answering it with a `deploy` job is the
most common mistake. The request is reasonable; it just splits into two pieces
that live in different places. Answer with **both**, and say so plainly:

> Here is your `.rodmena/ci.yml`. It runs the tests — note it does **not**
> deploy, because this platform has no branch filter, so a deploy job would run
> on every pull request too. Deployment is done by a small poller on your deploy
> host, below, which watches for a green pipeline on `main`. You'll also need an
> operator to onboard the repository and set `main` as a build branch, since
> that isn't self-service.

Then give the pipeline (Step 1) and the poller (Step 2).

**Do not** put the deploy credential in the pipeline. If the user hands you a
`DEPLOY_SSH_KEY`, a registry password or a cloud token "for the deploy job",
that credential belongs on the deploy host, not in `secrets:`. A secret named in
a job is readable by every step of that job, on every branch and every pull
request that builds — which is exactly the exposure the split exists to prevent.
Say that rather than quietly omitting it, or the user will assume you forgot and
add it back.

### Step 1 — the pipeline (no deploy job)

```yaml
version: 1

concurrency:
  group: ci-${{ ref }}
  cancel_in_progress: true

jobs:

  backend:
    name: "Backend (lint + tests)"
    image: containers.rodmena.co.uk/rodmena/ci-python:3.12
    services: [postgres, redis]
    env:
      ENV: test
    timeout_minutes: 20
    steps:
      - name: Install
        run: cd backend && uv sync --all-groups
      - name: Lint
        run: cd backend && uv run ruff check .
      - name: Tests
        run: cd backend && uv run pytest -q

  frontend:
    name: "Frontend (lint + tests)"
    image: containers.rodmena.co.uk/rodmena/ci-node:22
    timeout_minutes: 15
    steps:
      - name: Install
        run: cd frontend && npm ci
      - name: Tests
        run: cd frontend && npx vitest run
```

### Step 2 — the deploy bot, on the deployment host

It polls the public read API, gates on branch and conclusion itself, and keeps a
marker so one green pipeline deploys exactly once.

```sh
#!/bin/sh
# Runs on the DEPLOY HOST (cron/systemd timer), not in CI.
set -eu
API=https://ci.rodmena.co.uk/api
REPO=your-org/your-repo
BRANCH=refs/heads/main
STATE=/var/db/deploy-last-sha

latest=$(curl -fsS "$API/pipelines" | python3 -c '
import json,sys
repo,branch = sys.argv[1], sys.argv[2]
for p in json.load(sys.stdin)["pipelines"]:          # newest first
    if p["repo_full_name"]==repo and p["ref"]==branch and p["state"]=="succeeded":
        print(p["build_sha"]); break
' "$REPO" "$BRANCH")

[ -n "${latest:-}" ] || exit 0
[ "$latest" = "$(cat "$STATE" 2>/dev/null || true)" ] && exit 0

/usr/local/bin/deploy.sh "$latest"     # your deploy, your credentials
echo "$latest" > "$STATE"
```

Gate on `state == "succeeded"`, never on "the newest pipeline exists". A
pipeline may be `queued`, `running`, `failed`, `cancelled`, `superseded`,
`timed_out` or `definition_error`; only `succeeded` means every job passed.

### Why not just give CI the credentials?

Because a CI job runs whatever the repository says, including whatever its
dependencies pull in, and every step in that job can read every secret it names.
The deploy host runs only your deploy script. That boundary is the whole point,
and it is also what gives you branch gating, one-deploy-per-commit, and a deploy
that does not rerun because someone re-ran a check.

---

## Public read API

Base `https://ci.rodmena.co.uk/api`. Read-only, unauthenticated, deny-by-default
— anything not listed here returns 404, deliberately.

| Endpoint | Returns |
|---|---|
| `GET /api/healthz` | `{status: healthy\|degraded\|unhealthy, ...}` |
| `GET /api/operational` | queue depth, capacity usage, breaker state |
| `GET /api/pipelines` | `{pipelines: [...]}`, newest first, max 100. Also `?delivery_id=` |
| `GET /api/jobs` | `{jobs: [...]}`, newest first, max 100. Also `?pipeline_id=` |
| `GET /api/jobs/{id}/steps` | `{steps: [...]}` per-step state, including `skipped` |
| `GET /api/jobs/{id}/logs` | `{lines, next_since, eof}`, secrets scrubbed. Also `?since=` |

Pipeline fields: `id`, `repo_full_name`, `report_sha`, `build_sha`, `ref`,
`state`, `conclusion`, `created_at`, `started_at`, `finished_at`, `deadline`.

Job fields: `id`, `pipeline_id`, `name`, `state`, `conclusion`, `exit_code`,
`failure_reason`, `image`, `needs`, `track_id`, `cpu_limit`, `memory_limit_mb`,
`timeout_seconds`, `started_at`, `finished_at`.

Job and pipeline states: `queued`, `running`, `succeeded`, `failed`,
`cancelled`, `timed_out`, `skipped` (jobs), plus `superseded` and
`definition_error` (pipelines).

---

## Onboarding a repository

Not self-service. An operator must, on the conductor host:

1. Add the repository and its build branches (`POST /admin/repos`, or the
   `ci-admin` CLI). Default `["main"]`. **Until this exists, pushes are recorded
   and nothing builds** — silently and by design, not an error.
2. Install the GitHub App on the repository.
3. Add any secrets, scoped to that repository (`POST /admin/secrets`).

`/admin/*` is internal-only and is not reachable from the public hostname.

---

## Validation

A rejected definition produces a failing `ci/definition` check run naming the
first error and its line. Rejections you are most likely to cause:

- any unknown key, anywhere (this is the one that catches Actions habits:
  `on`, `if`, `uses`, `runs-on`, `strategy`, `with`, `cache`)
- `needs` naming an unknown job, or a `needs` cycle
- duplicate expanded check-run names, or a job named `ci/definition`
- a `${{ }}` outside the two documented contexts
- matrix values outside `[A-Za-z0-9._-]`, 1–64 chars
- secret names not matching `[A-Z][A-Z0-9_]*`
- caps: file ≤ 256 KB, ≤ 50 jobs expanded, ≤ 64 matrix combinations, ≤ 30 steps
  per job, ≤ 100 env keys per job

Validate before pushing (internal network): `POST /admin/definitions/validate`
with the raw YAML body. It runs the identical code path as the pipeline, so the
two can never disagree.

---

## Not supported, and why

| Not supported | Why |
|---|---|
| `if:` / expression language | a small language becomes a large one. Use `continue_on_error`, split the job, or gate in the deploy bot |
| branch/tag filters in the file | branch selection is per-repository; PRs always build |
| `uses:` third-party actions | arbitrary third-party code with access to your secrets |
| per-step `secrets:` | steps share one container; per-step env is cosmetic isolation. Use a separate job |
| dependency caching | deliberate follow-on; correctness first |
| fork pull-request builds | requires a trust model that does not exist yet |
| self-hosted runner selection | there is one execution layer |
| deploying from a CI job | see WARNING 2 — use poll-and-deploy |

Additions require a spec change, not a pull request that adds a key.

---

## Checklist before you hand back a pipeline

- [ ] No `on:`, `if:`, `branches:`, `uses:`, `runs-on:`, `strategy:`, `cache:`
- [ ] No job that deploys, publishes or pushes anything to production
- [ ] Every job has `name`, `image`, `steps`
- [ ] `name` values unique after matrix expansion
- [ ] `needs` refers only to job ids in this file, acyclic
- [ ] `resources.memory_mb` ≤ 2048 and `resources.cpu` ≤ 4, or omitted
- [ ] `services:` only where the image ships them
- [ ] Credentials in their own job, or better, not in CI at all
- [ ] You told the user that deployment needs a host-side bot and an operator to
      onboard the repository — neither is something the YAML can do