Automates, crons & runs

Agent (thinks) vs automate (executes)

The harness separates two execution registers. The agent thinks, decides and delegates through a ReAct loop; it is carried by an LLM persona and triggered by a spawn, by the Task tool or by a sy_task_run. The automate runs a deterministic routine; it is a Python script synedre/ac_*.py triggered by cron, by CLI or by the Nitro scheduler.

	Agent	Automate
Role	Thinks, decides, delegates (ReAct)	Runs a deterministic routine
Backing	LLM persona	Python script synedre/ac_*.py
Table	ps_ac_agents / sy_agents	sy_automates
Trigger	Spawn, Task tool, sy_task_run	cron, CLI, Nitro scheduledTasks

An automate does not think: it may call an LLM (content generation, classification) but its control flow is hard-coded. Thinking lives in the personas and in Atlas; repeatable execution lives in the automates. The link table sy_automate_agents ties each automate to its owning agent; conversely, an agent can spawn an automate or a sy_task_run.

The synedre/*.py façades

The synedre/ directory holds close to two hundred .py files. Not all are scheduled automates: most are façades — a single entry point for a given capability — alongside shared libraries, manual tools and one-shot scripts.

The canonical classification registry lives in the database in sy_automates, not in the files. Two axes structure it:

• kind — technical nature: recurring (scheduled, cron or Nitro), oneshot (on demand), tool (invoked by an agent or skill), lib (shared library, not runnable on its own) and meta (meta-tooling, such as the cron wrapper).
• caste — the owning agent group. Observed families: Vigies (audits/QA), Scribes (writing), Oracles (watch/reporting), Horlogers (infra/session/backup), Bâtisseurs (build/provisioning), Tisserands (SEO/linking).

Functional taxonomy of the ac_* families

Family	Role
Atlas (orchestrator)	Inbox → intent → spawn pipeline; Atlas health and monitoring.
Audits	Drift and findings detection. A non-zero exit means "findings", not "crash".
Backups	PG and file dumps to object storage, monthly restore test.
Brainstorm	Async jobs (looping worker, cron one-shot safety net).
Blog / SEO	Content generation and hygiene, technical SEO.
Email / inbox	Mail read and write. Client sending goes exclusively through the single façade.
Browser	Browser automation (residential headful worker).
Bank / invoicing	Bank synchronisation, recurring billing.
Brand / watch	Brand monitoring, technology watch, reviews sync.
Memory / learning	Vector RAG, consolidation, post-mortems.
SRE / safeguards	Cost and runaway monitoring, alerting, production-write safeguard.
Libs / infra	Shared building blocks (DB, logger, env, AI provider, log rotation, cron wrapper) — not scheduled.

The cron wrapper — a watchdog harness

Every automate scheduled via crontab is launched through a single wrapper (ac_cron_wrapper.py). This watchdog carries six responsibilities:

1. Environment loading — it loads the env files into the process without overwriting variables already present.
2. Circuit breaker — it reads the consecutive-failure counter in the database; past a threshold (ten failures in a row) the script is disabled, with a single line logged at the threshold to avoid spam.
3. Execution — it runs the script as a subprocess, with a default 300-second timeout, raised up to 1800 seconds for a few long-running scripts.
4. Self-repair — it classifies stderr then attempts a fix followed by a retry: missing import injected, non-writable log redirected, missing directory created (sandboxed under the root), module installed in the venv, network retry with backoff.
5. Soft-fail — for audit scripts, a non-zero exit counts as findings rather than a crash: the counter is reset and the exit code is propagated to downstream consumers.
6. DB logging — each crash is inserted into an append-only table with the error type, the traceback, the applied fix, the retry outcome and the deactivation state.

The per-script lock is implicit: there is no flock in the wrapper itself. A few crons add flock -n by hand, but most entries rely on the timeout and the frequency to avoid overlap.

The DB connection goes through the ac_db façade with a search_path on the mothership schema; some constants tied to a now-removed SQL engine remain as dead code.

The run system

sy_run — the "run" doctrine

A run is a scoped execution driven by Atlas over a perimeter (mothership or tenant), the perimeter automatically loading the context — machine, client, contact, mailbox — from the database. The sy_run table carries the source, trigger, scope, title, status and timestamps.

Trigger	Source	Entry point
email	atlas-inbox	forward classified by intent (run, question, chantier, noise).
chat	console	scoped console at the top of /hub/runs (scope selector + Atlas chat).

A third, scheduled trigger (cron) exists in doctrine but is not yet observed in use.

sy_task_run — the agent execution unit

A sy_task_run is not a run. It is the execution unit delegated to an agent, attached to a task or a chantier. A run can spawn a sy_task_run when Atlas delegates. The key columns carry the codename, the agent, the prompt, the template, the perimeter, the exit criteria, the status, the output log, the exit code, the consumed tokens and the cost.

The executor is ac_task_worker.py (cron every minute). Its logic: pick the oldest pending task, mark it running, spawn by template, stream stdout to the output log, then mark completed or failed. In --live mode it spawns the claude CLI with permissions scoped to the template: read-only for research, read plus report for audit, sandboxed write scoped to the perimeter for code.

sy_automate_logs — execution journal

The sy_automate_logs table is the verbose per-execution journal of automates (duration, steps, counters, errors, warnings, context), to be distinguished from the cron error table that only records wrapper crashes.

sy_run (Atlas, scoped)
   └─ spawns ─► sy_task_run (agent) ──log──► output_log
sy_automates (registry) ──scheduled──► sy_automate_logs (verbose journal)
                                    └─► cron error table (wrapper crashes)

Scheduling: two schedulers in parallel

Nitro scheduledTasks

The Nuxt/Nitro mothership exposes scheduledTasks (the audit:* tasks live in a dedicated layer, merged by Nuxt). Each task is protected by an environment guard that short-circuits execution outside the internal context.

Task	Cron
email:queue-process	*/2 * * * *
audit:uptime-monitor	*/15 * * * *
audit:dictionary-watch	*/30 * * * *
audit:deps-watch	0 2 * * *
audit:daily-meet	0 8 * * *
audit:ssl-watch	0 9 * * *
audit:brand-watch	0 12 * * *

Other audit files are present but intentionally not scheduled. IMAP sync and client sync are disabled (event-loop block on the mail provider's side).

Linux crontab safety net

The Nitro scheduler has been dead since a May 2026 regression (all scheduledTasks dark). The survival net is the host user's Linux crontab, which now carries the bulk of the load. A stale build risk runs alongside it: a ./deploy is required for the container to reflect the scheduledTasks from the config file.

The families of active entries:

• Through the wrapper: monitoring (every 2 min), nightly backup, automate and backup audits, weekly watch, bank sync, recurring billing, Atlas inbox poll, Atlas spawn, inbox sync, nightly dream consolidation.
• Outside the wrapper (python3 -m synedre.X): second-brain indexing, task worker, skill indexing, founder notification, unblock detector, SRE and cost alerts, runaway detector, negotiation event extraction (with flock), lead scan, scar-KPI refresh, reviews sync.
• Brainstorm worker: launched in a loop at reboot, with a net that restarts it if it has vanished and a one-shot catch-up mode.
• Shell / Node scripts: fleet and dependency scans, PG and file dumps to object storage (with a failure-notification fallback), monthly restore test, second-brain sync, chantier-lock cleanup, Atlas lock watchdog.
• Direct curl (bypassing the dead Nitro): mail queue drain and session-replay sync to the mothership's local endpoints.
• Memory consolidation: monthly consolidation (with flock).
• Self-improvement loop: session indexing, skill-proposal monitoring and detection, pattern detector, daily ReAct summary, memory metrics, log rotation. Several run outside the wrapper, hence without logging or self-repair.
• OSS quarantine + remote backups: OSS quarantine sweep and remote PG backups — one per tenant — with a failure-notification fallback.

Most entries are historically commented out with cutover or temporary-pause tags: the crontab is an archaeology log. Out of more than two hundred lines, a large majority are commented and only about fifty are actually active.

Registry ↔ scheduling reconciliation (to do). sy_automates declares about sixty recurring automates, but the crontab carries only about fifty active lines and Nitro seven tasks (dead, see above). No mapping today links each recurring automate to its crontab line or Nitro task: we cannot say how many are actually scheduled vs orphaned. This is the major operational gap of this layer.

The browser worker: a distributed off-VPS automate

The Browser family is the only subsystem where execution physically leaves the VPS. Two distinct reasons to leave the datacenter, hence two egress topologies not to be confused:

Topology	Where the browser runs	Defeats
Residential SOCKS5	Headless Chromium on the mothership VPS	the IP reputation
Remote headful worker	Headful Chrome on a residential machine	the browser fingerprint

Why residential headful rather than headless on the VPS

A SOCKS5 proxy routes the VPS traffic out through a residential IP via a reverse SSH tunnel from home. This is enough for sites that discriminate only on datacenter IP reputation. A safeguard refuses to launch if the proxy is down or if the resolved egress matches the VPS IP — zero accidental egress through the datacenter.

But a managed challenge such as Cloudflare Turnstile does not judge the IP: it judges the browser fingerprint (navigator.webdriver markers, plugins, WebRTC, headless signatures). A headless Chromium keeps a bot fingerprint even on a clean IP. The doctrine that follows: classify the protection before coding the flow. If it is a managed challenge, headful on a residential machine is mandatory; headless stealth does not get through. Hence the headful worker: a real Chrome, visible window, on an always-on residential machine with a residential IP. No proxy and no egress safeguard here: we are already on the residential IP.

The sy_browser_job queue + a job's lifecycle

The queue lives in the database (sy_browser_job) — an enqueue / claim / finish / get pattern. Columns: kind, payload (jsonb), status (queued|running|done|failed), result, error, host, attempts and timestamps, with indexes for ordered claiming.

The inversion of control is the key: the VPS has no inbound access to the residential machine (NAT, dynamic IP). It is the remote worker that polls the VPS over outbound SSH.

  [VPS]                              [Residential machine]
  Atlas / skill                      worker --loop (daemon)
   │ enqueue                           │ (1) auto-update when idle (git pull + re-exec)
   ▼                                   │ (2) SSH --claim ───────────────┐
  ac_browser_job_cli --enqueue        │                                │
   │  INSERT status=queued             ▼                                │
   ▼              FOR UPDATE SKIP LOCKED ── atomic claim running ◄───────┘
  sy_browser_job                       │ (3) dispatch by kind (strict whitelist)
                                       │     headful Chrome
                                       ▼
  sy_browser_job ◄── SSH --finish (status/result/error via STDIN JSON)

1. Enqueue (VPS) — refuses any kind outside the whitelist, validates the JSON, INSERTs as queued with randomised dollar-quoting (anti-injection).
2. Atomic claim — takes the oldest queued with FOR UPDATE SKIP LOCKED LIMIT 1, moves it to running, sets the host and increments the attempts. Two workers cannot steal a job from each other.
3. Dispatch (worker, headful) — routes by kind to the handler, never eval or exec of the payload (business parameters only). The kind whitelist must stay synchronised on both sides: any addition means editing both files.
4. Second factor — when a flow requires it, the 2FA code never reaches the residential machine by email: the worker calls the VPS back, which reads the mailbox on the VPS side and returns the code. The code is never logged in clear.
5. Finish — the worker passes {status, result, error} as a single JSON over STDIN (never as a shell argument, to avoid breaking on a newline or a quote).
6. Auto-update — in loop mode, only when idle, the worker does a throttled git pull --ff-only and re-executes itself if code has changed.

The SSH gate (defence if the key leaks)

The worker's SSH key is installed on the VPS with command=…,no-pty,no-agent-forwarding,no-X11-forwarding,no-port-forwarding in authorized_keys. If the residential machine is compromised, the key does not open a free VPS shell.

The gate reads the original SSH command, parses it into tokens with shlex.split (never bash -c), and executes only if: the prefix is exactly the expected CLI façade call; the sub-flag belongs to a closed set (--claim, --finish, --get, --enqueue, second factor); and each secondary flag matches a strict pattern. Any unknown token or non-conforming value triggers a refusal. This is also why the worker never uses a cd … && prefix: the gate requires a command beginning exactly with the prefix and handles the working directory itself. Worst case of a stolen key: pollute the queue, not run arbitrary code.

PII & screenshots. Results come back as result jsonb in the private database — never logged in clear by the worker (only a counter is). Debug screenshots are purged at the end of a run by the business module. A latent bug is noted: the job timeout is not enforced, so a browser job can hang indefinitely.

Safeguards & operations

• Circuit breaker: the cron wrapper disables an automate after ten consecutive crashes.
• Daily audit: an automate re-reads the cron error table and the execution journal every night.
• Cost / runaway: cost alerts, runaway detector and SRE alerts at short intervals.
• Backups + restore test: nightly dumps to object storage and a monthly restore test.
• Unblocking: a detector retries stuck sy_task_run entries (at most four attempts, with a kill-switch).

Known debt

• Silent broken cron: an active daily line points to a script that no longer exists, and the call is in direct python3 (outside the wrapper) — a silent daily failure, with no logging or self-repair. To be removed or rewired to the real façade through the wrapper.
• P0 anti-leak: some automates inherit a configuration-loading model that needs modernising; the clean pattern (reading from an environment variable) is already available.
• Dead Nitro scheduler → total dependence on the Linux crontab, an unsupervised single point of failure outside reboot launch.
• Case inconsistency in the classification columns of sy_automates.
• No generic lock in the cron wrapper; overlap is possible on slow jobs not protected by flock.