Deployment & Infrastructure

The ship vs deploy asymmetry

Two entrypoints live at the root of the repository. They do not do the same thing and do not have the same owner. This asymmetry is a law of the Synedre, not a convenience.

	./deploy	./ship
Target	preprod (or the mothership's live runtime)	PRODUCTION
Owner	the intelligence (agent / worker / cascade), systematic	the Founder only, a manual gesture
Git	auto-commits the dirty tree, stays on the preprod branch	merges preprod → main branch, pushes, then returns to the preprod branch (guaranteed by an exit trap)
Dirty safeguard	silent auto-commit, unless on the main branch (hard refusal)	blocking: refuses if the working tree is dirty, unless an explicit flag is set
Drift check	a single preprod target is wired; any other tenant is explicitly skipped. A bypass exists (urgent hotfix)	mothership drift is strictly blocking (no bypass), plus the tenant's prod database when the target is known
Post-deploy smoke	a tenant smoke-test suite	a process-environment check (false-positive guard) then the smoke tests
Migrations	—	displays the pending .sql files for the scope, non-blocking and never auto-applied
Closure	—	offers to close chantiers in test status

The doctrine boils down to this:

• ./ship = PROD, never autonomous, exclusive to the Founder. The intelligence never types ./ship.
• ./deploy = always the intelligence, systematically and without asking — including on the mothership, where ./deploy rebuilds the runtime live (no preprod).

        ┌─────────────┐   ./deploy <tenant>   ┌──────────────┐
   AI → │  preprod     │ ───────────────────→ │  preprod /    │
        │  branch      │   (auto, N times)      │  live runtime │
        └─────────────┘                        └──────────────┘
                │
                │  the Founder reviews preprod, validates
                ▼
        ┌─────────────┐   ./ship <tenant>     ┌──────────────┐
 Found→ │ merge into   │ ───────────────────→ │  PRODUCTION   │
        │  main branch │   (manual only)        │              │
        └─────────────┘                        └──────────────┘

Founder exception. Some "founder" tenants have no preprod: their public site lives directly on a VPS. For those, the intelligence does trigger a deploy, but it points at a prod target. So the naive reading "deploy = never prod" does not hold universally — what matters is who holds the hands at shipping time.

The production pipeline

The production command runs, in order, a sequence of steps, each of which can block the rest:

1. Flag parsing: dirty-safeguard bypass, skipping the post-ship closure.
2. Blocking dirty safeguard: exits with an error if the working tree is dirty.
3. Exit trap: guaranteed return to the preprod branch even on error (scar: getting stuck on the main branch).
4. Cleanup of root-owned artefacts left by past deploys, which would otherwise block writes.
5. Mothership drift check (blocking, no bypass) and the tenant's prod database when the target is known.
6. Merge preprod → main branch, pull, merge, push.
7. Migrations: display only (see below).
8. Agent audit (non-blocking).
9. Prod deploy: calls the tenant's script.
10. Prod smoke: process-environment check before the smoke-test suite.
11. Post-ship closure: offers to close chantiers in test status.

Drift-bypass asymmetry. The deploy accepts a drift-check bypass (for an urgent hotfix). Production has none: you never ship to prod on a divergent database schema.

Migrations at production time

Production displays the pending .sql files but never applies them: no automatic migration runner is wired in. The block is purely informative (non-blocking); it prints the commands to run by hand. A tenant → scope mapping prevents mixing mothership and tenant migrations, and a special all case aggregates every target while excluding already-applied migrations.

The YAML-driven dispatcher

Beneath both entrypoints lives a single dispatcher, driven by a deploy.yaml file specific to each tenant. It is the standard pipeline for all remote tenants. The mothership follows its own self-deploy path (see below).

Configuration resolution

The dispatcher locates the deploy.yaml based on the tenant: the mothership root for the flagship, the tenant's folder otherwise; a target option lets you pick a variant (for instance a preprod).

Dispatcher steps

1. Parse options: target, "all" flag, optional clean.
2. Load + validate the YAML via a Python parser that emits shell-quoted variables; any error aborts the deploy.
3. Start banner and timing init.
4. Drift check (if declared).
5. Background hooks (agent audit).
6. Nuxt build.
7. i18n seed (if declared).
8. Source maps (if declared).
9. Pack into a compressed archive.
10. Git push in the background (if declared).
11. Archive upload to the target machine.
12. Remote reload dispatched on the variant: process manager (graceful reload or hard restart) or container (restart).
13. Wait for background hooks.
14. Health check.
15. End banner.

Configuration schema

The deploy.yaml is validated against a schema. Sections cover the name, the build (local client required, Node environment defaulting to production), SSH access (host required, default user, optional key), an optional drift, an optional i18n seed, the required remote reload (variant among process manager and container), a health check (URL required, max wait) and optional hooks. Consistency guards forbid mixing fields specific to the process manager with those specific to containers. Anonymised example:

name: example
build:
  local_client: codemyshop/tenants/example
  node_env: production
ssh:
  host: <VPS host>
remote:
  variant: pm2
  dir: <deploy directory>
  pm2_app: example-nuxt
health:
  url: https://example.com
hooks:
  git_push: example

Build on the host, never on the VPS

This is the central invariant for remote tenants: the VPS never builds. The Nuxt bundle is compiled on the mothership (the host), packed, uploaded, extracted and then reloaded on the target.

  ┌──────────────────── MOTHERSHIP (host) ────────────────────┐
  │ 1. install dependencies (monorepo root delta)              │
  │ 2. smoke invariants (URL-shape tests)                      │
  │ 3. Nuxt build → .output/                                   │
  │ 4. pack → compressed archive                               │
  └──────────────────────────┬──────────────────────────────────┘
                             │ upload (scp)
                             ▼
  ┌──────────────────────── VPS ──────────────────────────────┐
  │ 5. mv .output → .output_old   (rollback ready)            │
  │ 6. extract the archive                                     │
  │ 7. reload the process manager | restart the container      │
  │ 8. if online → purge .output_old   else → rollback          │
  └──────────────────────────┬──────────────────────────────────┘
                             │
                             ▼  poll HTTP 200 (up to max wait)
                       health check

Step by step:

• Nuxt build: purge .output (and the cache if a clean was requested), install dependencies, a blocking smoke test on the product-URL-shape invariant, then build with a clean-retry if the incremental build fails.
• Pack: archive .output with a parallel compressor if available, otherwise a systematic fallback to the standard compressor.
• Upload: copy the archive to the target machine, then delete the local copy.
• Container reload: SSH, rotate .output → .output_old, extract, purge an incompatible native module (tree-shaking scar), restart the container, verify running status or roll back.
• Process-manager reload: a graceful reload that keeps in-flight connections, with support for a dedicated user and an optional symlink for static files.
• Health check: an HTTP poll loop until a 200 code or the max wait.

Rollback is ensured by .output_old: kept during the reload, purged if the process comes back, restored otherwise.

Environment false-positive guard

An HTTP 200 health check is not enough to validate a deploy. A real scar: a tenant redeployed without its database variables served an empty-but-valid skeleton page — the HTTP smoke returned 200 for two days while every data-driven route threw a 500 server-side. That is why production runs, before the smoke tests, an SSH check for the presence of critical variables in the remote process's environment: OK, missing (blocking), or process unreachable (blocking). Without this guard, a systematic 500 hides behind an HTTP 200.

The mothership's self-deploy

The mothership does not go through the dispatcher or the build-host/upload pattern. It is a local self-deploy: the flagship builds inside its own Docker container.

• No preprod: deploying the mothership pushes straight to the cockpit's live runtime, without the git ceremony of production. This is what "./deploy = live rebuild" means.
• Local transfer by copying a minimal source archive into the container — no network transfer, since the mothership and its runtime share the same host. The archive excludes the build cache and installed dependencies; inside the container, the Nuxt cache is set aside then restored, making the build incremental rather than a cold build — the real performance lever.
• Build inside the container: the old server stays online during the build, hence a near-seamless swap (~7 s).
• Standalone: the mothership's Nuxt config no longer extends the core, which sharply reduces the archive size.
• Fast-path: a lockfile fingerprint lets the dependency install be skipped when it is unchanged.
• Accelerated variant, opt-in: a host-side build with an atomic swap, to gain even more time.

Putting the mothership into production adds the ceremony (merge, push, drift, audit, prod smoke, closure) but ultimately calls the same self-deploy script.

The VPS inventory: a single source of truth

The infrastructure topology (addresses, SSH users, database containers) never lives hard-coded in the code or the docs. It lives in a table of the mothership's database, the single source of truth. A deploy listing option dumps this table, formatted for the console.

Column	Role
tenant id	the tenant's logical key
purpose	production | staging | infra | legacy | audit
access	address, domain, SSH user, key path
associated database	container, database name, user, the variable name holding the password (never the value)
runtime	web container, stacks present
business	criticality, recurring revenue, offer
deploy codename	codename for the exact command
client federation	foreign key to the client table

Doctrine: before any deploy or production, open the hub's client record to copy the exact command. Markdown architecture docs can drift; the database is authoritative.

The secrets doctrine

Principle: each secret has a single canonical file; the same key in two places is a bug. The hierarchy runs from the most local to the most shared:

#	File	Scope	Git-tracked
1	tenant .env	a single tenant (read by its container and its deploy config)	no
2	mothership .env	flagship core (cockpit)	no
3	mothership .env.host	mothership + host scripts (cron, automates, SSH)	no
4	out-of-repo shared env file	cross-project (AI provider keys, encryption key, master SMTP)	no (out of repo)
5	.env.example templates	public OSS templates	yes

The load order runs from most shared to most local: on a duplicate, the most local level wins — all the more reason never to place the same key at two levels.

P0 anti-leak. No plaintext secret in any git-tracked file. When a tracked file must reference a secret, it cites only the variable name and the env file that carries it — never the value. The deploy config and the database inventory carry only variable names.

Commit before deploy

A P0 scar: a hand-applied fix that was never committed got overwritten by a sync between two actions, breaking prod for several hours. The lesson: an uncommitted local change survives only in the extracted build and is overwritten by the checkout the deploy performs. The hard rule that follows: always commit before deploying.

A target-scoped safeguard (a mothership file does not block a tenant ship, and vice versa) enforces two behaviours:

• Deploy: if the tree is dirty → silent auto-commit + background push, unless on the main branch → hard refusal.
• Production: if the tree is dirty → blocking, exit with an error, message "commit first then re-run". Bypass reserved for deliberate hotfixes.

An end-of-session hook also blocks closure while the working tree is not clean. Root rule: no finished work stays uncommitted — the intelligence commits, the Founder never types the git add or commit commands.