Automata, scheduling and runs

1. Mental model: an agent that thinks, an automaton that executes

Synedre orchestrates CodeMyShop: this chapter documents the Synedre harness, the internal orchestration layer — single-organization, centralized — that drives the product and operations. It does not describe CodeMyShop itself, the multi-tenant e-commerce PaaS that Synedre operates for its clients.

The harness keeps two distinct registers apart:

	Agent	Automaton
Role	Thinks, decides, delegates	Executes a deterministic routine
Backing	A language model embodying a persona	An automation script
Trigger	Delegation from the orchestrator, or an assigned task	Scheduling (time- or event-based), command line

An automaton does not "think": it may call a language model for a specific step (content generation, classification), but its control flow stays hard-coded. Decision-making lives on the agent side; repeatable execution lives on the automaton side.

  trigger (chat, email, scheduled task)
         │
         ▼
  orchestrator (Atlas)
         │ delegates
         ▼
  execution delegated to an agent
         │ spawn
         ▼
  language model  /  automation script

2. Automated entry points

The harness groups several hundred automation entry points ("facades"), each exposing a single invokable capability: sync, audit, content generation, backup, monitoring, and so on. A minority are scheduled; the majority are tools invoked on demand by an agent, or shared libraries not runnable on their own.

A central registry classifies each entry point along two axes:

• Technical nature — scheduled (recurring), triggered on demand (one-shot), a tool invoked by an agent, a shared library, or meta-tooling (the execution supervisor itself).
• Owning group — each entry point is attached to one of the agent families that conceptually maintains it: audits/QA, watch/reporting, writing, build/deployment, infrastructure/backup, SEO/internal linking, finance/treasury, and a dedicated strategic-arbitration family. This classification is still a work in progress: casing and naming are not yet fully consistent across the whole registry.

Functional families

Family	Role
Orchestration	Inbox → intent → delegation → shipping pipeline. A cleanup effort mistakenly removed two dedicated health/monitoring components for this family; generic monitoring (cost, alerting) still covers this need through another path.
Audits	Drift and finding detection (schema, backups, accessibility, security). A non-zero exit signals an audit finding, not a crash. A periodic multi-site security audit (weekly recon + alert on critical finding) replaced an older generic penetration-testing tool that was retired as technical debt.
Backups	Database and file extraction to object storage, with a monthly restore test.
Brainstorm	Asynchronous processing queue for ideation (promotion, challenge, narrative framing).
Blog / SEO	Content generation and hygiene, cannibalization detection, technical SEO.
Email / inbox	Reading/writing email — sending to a client always goes through a single entry point.
Automated browsing	Browser automation (see §6).
Bank / invoicing	Bank sync, recurring invoicing, reminders. A generic reminder module was absorbed into a larger package but is no longer wired to any scheduling — only a client-specific reminder actually runs today.
Brand watch	Brand monitoring, technology watch, customer reviews.
Memory / learning	Semantic search, consolidation, post-incident learnings.
Operational reliability	Cost monitoring, runaway detection, alerting, production write guardrails.
Documentation self-maintenance	Nightly loop detailed in §7.
Autonomy cadence	Bootstrapping the autonomous execution pipeline, bounded by a time window (see §8).

An unalerted regression is worth flagging here as an example: an automated cleanup effort (meant to purge dead code) mistakenly removed several scripts that were still being called by the nightly documentation-maintenance orchestrator (§7) and by an accessibility audit module. These calls have been failing silently ever since, absorbed by an existing guardrail — no dedicated alert fired. Detail and scope in §7 and §9.

3. The scheduled-execution supervisor

Every scheduled automaton runs through a common supervising wrapper, responsible for:

1. Configuration loading — loads the required environment variables without overwriting ones already set.
2. Circuit breaker — beyond a threshold of ten consecutive failures, the script is automatically disabled (a single alert fires at the threshold, no spam).
3. Bounded execution — runs in an isolated subprocess, with a default five-minute timeout, extended up to thirty minutes for known long-running tasks.
4. Self-repair — for a handful of identified error classes (missing import, missing dependency, missing log directory, local naming conflict, transient network error), the supervisor attempts an automatic fix and retries once. Network errors get a backoff-based retry cascade.
5. Finding tolerance — audit scripts that intentionally exit with an error (an audit that found a problem) do not trip the circuit breaker: that's a result, not a crash.
6. Logging — every failed run is logged (error type, traceback, fix attempted, retry success, consecutive-failure count) to feed the daily audit (§9).

Anti-overlap locking is not universal: the supervisor itself does not apply a generic per-script lock; a handful of overlap-sensitive tasks add one explicitly, while most rely on the timeout and scheduling frequency to avoid overlap.

4. The run system

4.1 The "run" doctrine

A run is a scoped execution driven by the orchestrator over a given perimeter (internal to the harness, or a client tenant) — the perimeter automatically loading its context (infrastructure, client, contact, mailbox) from the central registry. It is the harness's default execution path: a run chains edit → ship, with no ceremony, for any scoped and reversible piece of work.

Two main entry points feed the observed run volume: forwarding an incoming email classified by intent (run / question / larger initiative / noise), and a scoped chat console driven by the orchestrator. A third, scheduled entry point (cron) is envisioned in doctrine but still lightly used in practice.

4.2 The unit of execution delegated to an agent

A run is not itself a unit of agent execution: it is the orchestrator that, by delegating, spawns an execution delegated to an agent (comparable to a background task run by a sandboxed language model).

The most frequent creator of these delegated executions is the nightly autonomy cadence (§8), followed by the execution worker's internal auto-chaining, then the command-line interface and the automatic-shipping API calls.

The main executor runs at a tight cadence (about once a minute): it picks the oldest pending task, marks it running, spawns the corresponding process based on the task type (research, audit, or code), streams the output live, then marks the result done or failed. Granted permissions depend on the task type: a research task is limited to reading (no writes); a code task gets broader access but strictly confined to its declared perimeter.

The language model used for each task is now resolved dynamically from a recommendation computed upstream (for instance, a task judged complex is routed to a more capable model), with a safe fallback to a default model if that recommendation is missing or fails to read — a wiring that was recently fixed, after discovering the recommendation was being computed but never actually applied at launch time.

For tasks targeting a genuinely live environment (deployment, certificate, infrastructure), an additional visual check was added before marking a task done: a real browser verifies the target actually responds correctly, rather than relying solely on the agent's textual claim and a code read-through. If the target turns out broken despite a positive textual check, the verdict is downgraded to failed; if the visual check itself is unavailable, the task is still allowed through (fail-open), but the incident is logged and a production-shipping guardrail remains the final safety net.

4.3 The automaton execution log

Every scheduled automaton run is logged in detail (duration, step counters, errors, warnings, context) — distinct from the supervisor's error log (§3), which only traces the supervisor's own crashes.

5. Scheduling: two parallel schedulers

5.1 Application-level scheduler

The harness exposes its own application-level scheduled tasks, in addition to the system crontab net. These tasks notably cover: email send-queue processing, uptime monitoring, technical dictionary watch, dependency watch, a daily digest, certificate monitoring, and brand watch. Each sensitive task is protected by a guard that short-circuits it if it runs outside the expected internal context.

Two tasks remain intentionally disabled: mailbox sync and client-side email sync, because the mail protocol in use blocks the event loop and causes cascading slowdowns — to be re-enabled once the mail client is fixed for non-blocking mode.

5.2 System crontab safety net

The host machine's system crontab carries most of the scheduled load and acts as a net independent of the application-level scheduler. Out of several hundred total lines, a significant fraction (roughly 40%) is actually active; the rest is a log book of disabled tasks, kept as comments for historical memory.

Main families of active tasks:

• Through the execution supervisor — system monitoring, backup, daily audits, bank sync, recurring invoicing, inbox processing, memory-consolidation dreaming, documentation-maintenance loop, autonomy cadence (wired to real execution, running every two minutes, but bounded by a configurable daily time window).
• Outside the supervisor, direct Python modules — memory indexing, delegated-task executor (one-minute cadence), skill indexing, automatic stuck-task detection, reliability alerting, cost alerting, runaway detection, negotiation-event extraction, review watch.
• Outside the supervisor, no wrapper — documentation publishing, monthly restore test, log rotation, memory metrics, session indexing, proposal monitoring, daily digest. These scripts get neither centralized logging nor the supervisor's self-repair.
• Infrastructure scripts — fleet audit, dependency audit, backups to object storage (local and remote, per tenant), restore test, memory sync, orphan-lock cleanup.
• Direct HTTP calls — triggering the email send-queue processing, session-replay sync. A hardening debt is flagged here: one of these lines embeds a bearer token in plain text — to be migrated to a dedicated secrets store.

A reconciliation effort is still pending: the canonical registry of scheduled automata is not yet automatically cross-checked against the actual active crontab lines and application-level tasks — today there is no way to know how many automata registered as "recurring" are in fact orphaned (not scheduled anywhere).

6. The distributed browser automation

Browser automation is the only subsystem where execution physically leaves the server infrastructure. Two distinct reasons drive this, hence two different topologies that should not be confused:

Topology	Where the browser runs	Works around	Mode
Residential proxy	Headless browser on the server	IP address reputation	No visible window, with hardened stealth
Remote machine with a window	Visible-window browser on a residential machine	Browser fingerprint	Visible window, direct residential IP address

6.1 Why a residential browser with a visible window

A reverse network tunnel set up from a residential point lets traffic exit through a residential IP address rather than the server's own address. That's enough for sites that discriminate only on IP address reputation. A dedicated guardrail refuses to launch the automaton if the tunnel is unavailable or if the observed exit address matches the server's own address — no accidental exit through the wrong network path.

But some anti-bot protections don't judge the IP address: they judge the browser's own fingerprint (technical signals that reveal an automated browser). A headless browser, even through a clean residential tunnel, still carries a bot fingerprint. Hence the second mode: a real browser with a visible window, running on a residential machine that is genuinely powered on, with the native residential IP address — no tunnel needed, the anti-bot check passes naturally.

Operating rule: classify the type of protection encountered before coding the automation flow. A protection that judges the browser itself, not just the IP address, requires the visible-window mode; stealth-without-window is not enough.

6.2 Lifecycle of a browsing task

Tasks are queued centrally, with a status (queued / running / done / failed), an operation type restricted to an explicit allowlist, a structured result, and an attempt counter.

The key design point is inversion of control: the server has no inbound access to the residential machine (dynamic IP address, no open port). It is the residential machine that polls the server at a regular interval over an outbound secure connection.

  [Server]                                [Residential machine]
  enqueue                                  polling loop (~every 5s)
       │                                       │  (1) automatic code update if needed
       ▼                                       │  (2) atomic claim of the task
  central queue ◄────────────────────────────────┘
       │
       │  (3) execution: browser with a visible window,
       │      strict routing by operation type (allowlist)
       ▼
  queue ◄──── completion report (done/failed, structured result)

Mechanism details:

• Enqueue — refuses any operation type outside the allowlist, validates the payload before insertion.
• Atomic claim — the oldest pending task is claimed exclusively; multiple remote workers cannot steal a task from each other.
• Strict routing — no arbitrary code execution from the task payload, only predefined business parameters routed to a known handler.
• Two-factor authentication — for operations that need it, the verification code flows back through the server (which reads it from its own mailbox) rather than arriving directly on the remote machine, and is never logged in plain text.
• Completion report — the result is transmitted as a single structured block, never as a raw command-line argument, to avoid breaking on a special character.
• Automatic update — the remote machine updates itself only while idle (never mid-task).

6.3 The guardrail against a leaked access key

The remote machine's access key is installed server-side with strict restrictions: no interactive terminal, no port forwarding, no agent forwarding. If the residential machine is compromised, the stolen key does not open free server access.

A dedicated gate intercepts the received command, splits it into strict tokens (never free shell interpretation), and only executes if the command prefix matches exactly what's expected, the operation sub-type belongs to an explicit list, and every parameter matches a strict predefined format. Any unknown token or non-conforming value is refused and logged. Worst case of a stolen key: polluting the queue, never executing arbitrary code server-side.

Personal data returned by certain operations (names, message excerpts) is never logged in plain text — only an aggregate count is. Debug screenshots are automatically purged at the end of processing. A latent debt item is flagged: a per-task processing timeout exists in the code but is not actually enforced today, which can leave a browsing task hanging indefinitely.

7. The nightly documentation self-maintenance loop

The harness has a subsystem that measures the gap between its own technical documentation and its real code, and closes that gap largely autonomously, under explicit guardrails. It is the harness's newest and most intertwined automation layer.

7.1 Pipeline architecture

A single nightly orchestrator runs the following steps, in order:

1. Measure the gap — compares each documented chapter to its real linked code.
2. Measure coverage — detects blind spots (parts of the code never mentioned in the documentation).
3. Map components — checks consistency between a component's declared status and its real activation state.
4. Link components — derives factual relationships between documented components (what produces what, what monitors what).
5. Repair dead references — deterministically fixes pointers to code that has moved, never inventing anything.
6. Deep-regenerate — rewrites an entire chapter by re-reading the real code, under a bounded time budget.
7. Propose republishing — queues changed chapters for publication.
8. Publish documentation — pushes validated chapters to the public site, under an automated anti-leak gate.
9. Publish the component map — syncs the public status of components.
10. Re-measure the gap — re-measures after repair, so the final diagnosis reflects the real end-of-cycle state.
11. Produce the diagnosis — a summary report (see §7.2 below).
12. Publish the summary — pushes a sanitized snapshot of the diagnosis to the public site.

The final diagnosis is deliberately computed after the re-measure step, not right after the initial coverage measurement: otherwise the published summary would stay one cycle behind the actually-corrected state.

An unalerted incident worth flagging here as a transparency example: an automated dead-code cleanup effort mistakenly removed three scripts still being called by this nightly orchestrator (the "map components", "propose republishing", and "publish the component map" steps). Since that incident, these three steps have been failing every night — the failure is absorbed silently by the existing guardrail (only an outright crash of the orchestrator itself, not the failure of an isolated sub-step, surfaces as an alert), so no dedicated notification fired. Concrete consequence: component status/name no longer automatically re-syncs to the public site, and republish queuing now only flows through the direct publishing path (step 8). These scripts remain restorable within a grace window; failing that, the corresponding calls will be cleanly removed.

7.2 The fidelity mirror (measure the gap)

Runs strictly read-only: never writes either the code or the documentation, only records the observed gap. Three types of gap detected: a linked code file was modified after the documented chapter's last update; the chapter cites a code path that no longer exists; the chapter published on the site lags behind its internal version. Idempotent (one measurement per chapter per day), and called twice per cycle (at the start, and again at the end after repair) so the final diagnosis reflects the real state.

7.3 The coverage auditor (measure coverage)

Complements the fidelity mirror: where that one checks "is what I say true?", the coverage auditor checks "is there a part of the system I never mention?", via a set difference between the real documentable code and what is actually covered by at least one chapter. A single uncovered item isn't a problem; several uncovered items in the same family become a candidate for a new section or a new chapter. Strictly read-only.

7.4 The deterministic dead-reference repairer

Bridges the gap between diagnosis (which detects a dead path) and language-model rewriting (which fixes prose but doesn't know a file has moved): resolution here is deterministic, zero improvisation. For each dead reference, it searches for the possible new location by file name: exactly one candidate found → automatic fix; zero candidates → flags for human review; multiple ambiguous candidates → flags, never a blind automatic fix. Reversible (a single action undoes an entire repair cycle).

7.5 The deep regenerator

Under a bounded time budget, a headless language model re-reads a chapter's real code and (a) rewrites its internal version, (b) directly produces its sanitized public version, in both French and English. The internal version is validated before the public version goes through a dedicated anti-leak check.

7.6 The mechanical auto-repairer (independent cycle)

Runs on its own cadence, independent of the main nightly orchestrator. Touches only documentation files, never code. Strict safety rule: a dead reference is only fixed if its file name matches exactly one file tracked in the code repository — zero candidates or multiple ambiguous candidates, no automatic fix. Reversible, capped in volume per run, with a dedicated kill switch.

7.7 The end-of-cycle diagnosis

Computed at the very end of the cycle, after the gap re-measure. Aggregates five independent dimensions without modifying any of them: the doc-to-code gap, documentation coverage, open technical debt, the signal drawn from post-incident learnings, and the operational health of scheduled automata (recent error rate, scripts disabled by the circuit breaker). The result feeds a daily digest for the founder.

7.8 Publishing to the public site

Detects chapters whose internal version has diverged from its published version, queues them for republishing, with a multi-layer anti-leak check before any write. Actual publication to the visible public state happens either through a deliberate human action, or automatically for chapters that pass every machine check (anti-leak, plus a dedicated anti-drift register check — see §VII of the editorial charter); rejected chapters stay queued and generate an alert to the founder.

7.9 The external review process

Pulls reviews submitted from an outside perspective (feedback from another language model on a harness response), evaluates them via a sandboxed language model with strict anti-injection delimiters, and only draws a signal from it (no immediate regeneration) — actual regeneration remains consumed by the main nightly orchestrator, to avoid losing compute on an interruption.

8. The nightly autonomy cadence

8.1 Bootstrapping, time window, guardrails

The autonomous execution pipeline self-chains within an ongoing piece of work, but nothing bootstraps the very first step: a piece of work marked as autonomous, with pending tasks and no run in progress, would stay dormant without an external trigger. This cadence fills exactly that gap: it seeds a pending task for the next eligible step of every active piece of work that has no run in progress.

The cadence runs at a tight interval (every two minutes), but only actually seeds if the current time falls within a configurable time window for that day of the week. Outside the window, it falls back to silent observation mode (it notes, it does not act). An explicit manual override can bypass the window — a deliberate action by the founder, not a default behavior.

Guardrails in place:

• Cost cap — if an initiative's cumulative cost reaches its cap, it is frozen and no new task gets seeded.
• Kill switches — several independent switches allow cutting all or part of autonomy without touching the code.
• Autonomous shipping, fleet-governed — the cadence can run a preproduction deployment systematically, and an actual production deployment only if the target infrastructure explicitly allows it (a per-site flag, enabled by default, with explicit case-by-case exclusions). Unsupervised autonomous production shipping additionally requires a recent, positive quality-check proof — without that proof, the cadence signals without acting.
• Post-deploy rollback net — an HTTP health gate checks the target after deployment; on failure, an automatic rollback to the previous version is triggered. On autonomous production ships specifically, the previous version is kept one notch longer than on a manual deploy, to allow a manual after-the-fact rollback of a deployment that is "technically live but functionally wrong" — something the HTTP health gate alone cannot detect.
• Hard-coded, database-independent floor — for one site explicitly flagged as sensitive, a refusal of automatic production shipping is hard-coded, checked before even reading the authorization flag from the database. Reason: this cadence runs as a scheduled task, outside any protected interactive session — a guardrail that only protects interactive sessions would never see this call. A single corrupted configuration line must never, by itself, be enough to auto-ship this site to production. It mirrors, across two independent code surfaces, the same double-lock principle.

9. Guardrails & technical debt

• Circuit breaker — an automaton is disabled after ten consecutive failures.
• Daily audit — re-reads the error log and the automaton execution log every day.
• Cost / runaway — several independent monitors, at varying cadences (a few minutes to half an hour), cover cumulative cost and runaway detection.
• Backups + restore test — nightly dumps (database, files, remote client sites) and a real monthly restore test.
• Automatic unblocking — retries stuck runs, capped in attempt count, with a dedicated kill switch.
• Documentation loop — see §7.

Flagged debt

• A dead cron was cleaned up with no replacement: the URL-leak audit it used to run is no longer covered — to be recreated if the need resurfaces.
• An authentication token appears in plain text on a scheduled-task line — a hardening debt to fix (migration to a dedicated secrets store).
• An email-queue processing task runs twice (application scheduler + direct HTTP call) — redundant with no functional impact (the queue itself is idempotent) but worth cleaning up.
• No generic locking in the scheduled-execution supervisor: overlap is possible on slow tasks not explicitly protected.
• Unalerted regression (detailed in §7.1): three steps of the documentation loop have been failing every night since a dead-code cleanup incident, with no dedicated alert firing. A fourth component (accessibility audit) is affected by the same class of regression.
• Latent bug: a processing timeout for automated browsing tasks exists in the code but is not enforced in practice — a task can hang indefinitely.