Memory and learning of the Synedre OS harness

Memory & Learning

This page describes how the agentic harness retains information through a three-tier memory system, how it retrieves relevant information via semantic recall, and how it converts errors and successes into reusable rules.

1. Three-Tier Memory

The system distinguishes three memory layers of increasing granularity and write discipline.

┌──────────────────────────────────────────────────────────────────┐
│ TIER 1 — Reflex Memory                                           │
│   Loaded in EVERY session. Hard rules, feedback, references.     │
│   Strict and searchable by full-text; one entry per line in the  │
│   index (≤ 200 characters per entry).                            │
├──────────────────────────────────────────────────────────────────┤
│ TIER 2 — Doctrinal Knowledge Base (Zettelkasten)                 │
│   The "why": decisions, doctrine, concepts. Exploratory.         │
│   Draft zone = free-write depot; consolidated zone = validated   │
│   only on explicit instruction.                                  │
├──────────────────────────────────────────────────────────────────┤
│ TIER 3 — Vector Index (Semantic)                                 │
│   Semantic index of all of the above + DB tables.                │
│   Cosine HNSW vectors, 1024 dimensions. RAG recall.              │
└──────────────────────────────────────────────────────────────────┘

Tier 1 — Reflex Memory

• A central index file lists all entries; details are distributed across thematic files (feedback, user preferences, project references).
• These files are injected into the context of every session: this is the reflex memory, loaded without explicit recall.
• Reserved for genuinely impactful doctrines. Authoring rule: one index entry = one line, approximately 200 characters maximum.

Tier 2 — Doctrinal Knowledge Base (Zettelkasten)

• Organised into sub-folders: consolidated notes, draft inbox, thematic maps, daily journals, ephemeral notes.
• Inter-note links in wiki syntax. YAML front matter with type and status fields.
• Promotion rule: the inbox is the free-write zone (the agent writes here without restriction); the consolidated zone is never modified without explicit instruction. No automatic draft → consolidated merge is permitted — this is a governance safeguard.

Tier 3 — Semantic Vector Index

• Vector table with an HNSW index (cosine similarity) and a uniqueness constraint on the (source type, source identifier) pair.
• Vectors of dimension 1024, produced by the AI provider's embedding model.
• Covers all sources: reflex memory, doctrinal knowledge base, scars, doctrines, worksites, agents, drills, conduct rules, ReAct iterations.

2. Indexing

Two distinct automation pipelines feed tiers 2 and 3 respectively. It is important not to confuse them:

• The first populates the structural index of the knowledge base (full-text search and navigation within the hub interface).
• The second populates the vector index (RAG semantic recall).

2.1 Knowledge Base Structural Indexer

This pipeline traverses all sub-folders of the knowledge base and maintains a synchronised relational table.

• Parses the YAML front matter of each note and extracts inter-note links.
• Incremental synchronisation: compares the file modification date against the stored state; writes only if the note has changed. Orphaned entries (deleted files) are removed automatically.
• Indexed columns: relative path, type, title, tags, body, links, size, modification date, creation and update timestamps.
• Available options: --force (full re-indexing), --stats (statistics without writing).
• Schedule: every 15 minutes. Idempotent.

2.2 Vector Embedding Pipeline

This pipeline produces semantic vectors via the AI provider's embedding model (1024 dimensions) and inserts or updates them in the vector index.

Indexed sources and indicative volumes:

Source Type	Origin	Indicative Volume
Scar	Entries in the scar table that have a description	~740
Victory	Scars classified as victories	~4
Reflex Memory	Reflex memory files (excluding the main index)	~226
Doctrinal Knowledge Base	Knowledge base notes (all zones)	~14
Doctrine	Active doctrines	~25
Worksite	Worksite table	~10
Agent	Agent table	~30
Drill	Drill table	~31
Conduct Rule	Active conduct rules	~17
Iteration	ReAct triplets (reasoning / action / observation)	~16

• Deduplication: each chunk is fingerprinted by SHA-256 hash of its content. If the (source type, identifier) pair already carries the same hash, the chunk is skipped — no unnecessary call to the AI provider.
• Chunking: MVP strategy — one file = one chunk, truncated at 16,000 characters. Intelligent multi-chunk splitting is planned for a later phase.
• Batch processing: 50 inputs per embedding call.
• Insertion: UPSERT on the (source type, identifier) pair — the existing row is updated if the content has changed.
• Available options: --rebuild (full wipe then re-embedding), --source <type> (single source), --dry-run.

Immediate Re-indexing of a Scar

As soon as a scar is engraved during a ReAct loop, a unit re-indexing function immediately inserts or updates the corresponding vector — without waiting for the next scheduled cycle. The scar thus becomes searchable in the RAG as soon as the run completes.

Memory Automation Schedule

• Structural indexer: every 15 minutes.
• Vector embedding pipeline: full incremental sync daily (all sources, no rebuild). Real-time unit re-indexing (ReAct scars) remains active in parallel.
• Nightly loop (doctrinal consolidation): daily, at the start of the night.
• Pattern detector: weekly.
• Skill proposal detector: weekly.
• Skill proposal monitor: every hour.
• Memory metrics: daily, early morning.
• Memory consolidation: monthly (1st of the month).
• Nightly maintenance loop: daily, early morning.

Semantic and Lexical Recall Engine

The recall engine allows an agent to retrieve relevant memory fragments from a text query. It is accessible via the /memoire <query> skill and exposes three configurable search modes.

Search Pipeline

The incoming query is processed according to the selected mode, then results are re-ranked before being returned:

text query
   │
   ├─ mode 'sem'    (vector similarity)  ── 1024d embedding → sort by cosine distance
   ├─ mode 'lex'    (lexical matching)   ── words ≥ 3 chars → OR-tsquery → ts_rank
   └─ mode 'hybrid' (DEFAULT) ─────────── RRF k=60 : score = Σ 1/(60+rank_i)

[scope filter: scope]
LIMIT fetch_k (= k×3 if importance re-ranking is enabled)
   │
   ▼
re-ranking : score = similarity × (1 + importance/10) × exp(-age_days/365)
   │
   ▼
top-K final ──► recall counter increment

Mode Details

• Vector mode (sem): the query is transformed into a dense 1,024-dimensional vector via the AI provider's embedding service. The search relies on the cosine distance between this vector and those stored in the database. Score interpretation: above 0.75 the result is highly relevant; between 0.65 and 0.75 it is relevant; below 0.65 manual validation is recommended.
• Lexical mode (lex): words of at least three characters are extracted and combined into an OR query ("a | b | c") applied against a French full-text index. A result must match at least one term. This mode requires no external API call and reliably catches slugs and proper nouns that embeddings may dilute.
• Hybrid mode (hybrid, default): merges both rankings using Reciprocal Rank Fusion (RRF, k = 60). Final score: Σ 1/(60 + rank_i). A document absent from one mode receives 0 for that mode, without penalising the other. RRF scores are typically below 0.1: only the relative rank is meaningful, not the absolute values.

Importance and Recency Re-ranking

Enabled by default, this mechanism over-fetches k×3 candidates, then applies the following formula to the entries:

score = similarity × (1 + importance/10) × exp(-age_days/365)

Practical effect: a recent entry of maximum importance is boosted by a factor of ≈ 2; an entry one year old of minimum importance is brought down to ≈ 0.4 × its raw similarity.

Forgetting-Curve Countermeasure (Bump Recall)

Each entry served has its recall counter incremented in a non-blocking manner. The scheduled archival process only purges entries whose counter has remained at zero — ensuring that knowledge that has actually been reused is never prematurely archived.

Available Scopes

The --scope parameter filters the types of sources queried. The exposed scopes are: iterations, cicatrices, victoires, memory, brain, doctrine, chantiers, agents, drill, conduite, all. The brain scope aggregates several internal sub-types of structured knowledge.

Command-Line Options

• --k <integer>: number of results returned.
• --mode lex|sem|hybrid: search mode (default: hybrid).
• --scope <scope>: filters the type of sources.
• --json: machine-readable output.

Pre-Task Hook

A hook fires automatically before any task skeleton is created. It extracts the operation title, runs the recall engine in hybrid mode (k = 4) against that title, and injects the four most relevant results into the output stream before creation — without blocking the operation (exit 0 guaranteed). If the embedding service is unavailable (rate limit reached), the system automatically falls back to lexical mode. The objective is to avoid recreating an entry, a doctrine, or a task already on record, catching matches that the main linear memory index would have missed.

Recall Evaluation Harness

A non-regression harness continuously verifies that changes made to the recall engine do not degrade its performance. It operates in read-only mode: the recall counter is never modified during evaluations.

Computed Metrics

Metric	Definition
recall@k	Intersection between expected results and the top-k, divided by the number of expected results — averaged across all queries in the test set.
mrr@10	Reciprocal of the rank of the first relevant result in the top-10 (0 if absent) — averaged across all queries.
top1_accuracy	Proportion of queries for which the first result matches an expected identifier.

A result is considered matching if the expected identifier (case-insensitive) is a substring of the returned identifier — making the evaluation robust to path prefixes and composite formats.

Rate Control

In vector and hybrid modes, the harness enforces a fixed delay between each call to the embedding service, with exponential backoff when the rate limit is reached (up to five attempts). In lexical mode, no external call is made.

Flags and Exit Codes

Command / flag	Effect	Exit code
(none)	Gate mode: compares current metrics against the frozen baseline; fails if regression exceeds the tolerance threshold.	0 or 1
--baseline	Freezes a new baseline by running hybrid mode in production.	0
--mode-sweep	Compares all three modes (lex / sem / hybrid) — informational only, no gate.	0
--tol <float>	Acceptable regression tolerance (default: 0.05).	—
--json	Machine-readable output.	—
(baseline absent)	Warning and immediate halt.	2

Knowledge Graph (Bounded Experiment)

Status: time-bounded experiment, go/no-go decision scheduled for 2026. Reversible with no impact on existing components.

This experiment aims to determine whether structural proximity — "which hooks and entries are at most two hops away from the file the agent is about to modify?" — outperforms the recall engine's semantic similarity and the entropy detector's lexical matching for relevant context selection.

Data Model

Three tables make up the experiment:

• Node table: eight closed types (hook, event, cicatrice, doctrine, note, chantier, path, command), enforced by SQL check constraints.
• Edge table: eight closed relationship types; each edge carries a mandatory proof, a confidence level, and an origin (derive for computed edges, manuel for manually entered edges). Manual edges are never overwritten or pruned.
• Shadow log table: compares, on each invocation, what the graph would have suggested against the lexical matching results, without altering the actual behaviour in any way.

Graph Population

The graph is populated by strictly deterministic extractors: system parameters, hook headers, inter-document links, hook bodies, and protection relationships. No language model is involved in extraction: an LLM-generated edge would shift the hallucination risk into the graph itself.

Neighbourhood Query

A proximity query retrieves all nodes reachable within two hops from a given path or command, via an undirected recursive traversal. The confidence of a path is the product of the confidences of its edges.

Shadow Mode

When shadow mode is enabled in the local configuration, the entropy detector silently logs what the graph would have injected, without modifying production behaviour. The added latency is below 300 ms and any failure is silent.

Decision Criteria (Go/No-Go)

The graph will be adopted if its recall@5 exceeds that of vector mode by at least 15 points, or if the union of both methods exceeds vector mode alone by 20 points — and if the median latency remains below 150 ms.

Initial Measurement (Day 0)

Method	Recall@5	p50 Latency
Knowledge graph only	0.03	7.5 ms
Vector mode only	0.30	—
Graph + vector union	0.30	—

Day 0 verdict: no-go as-is. The cause is structural: the protection relationship only captures hooks that inspect the command chain; hooks that test a state (presence of a merge marker, file content) are invisible to the graph. Furthermore, only a minority of hooks cite an entry in their header. The final decision will be made after seven days of shadow-mode log analysis.

Learning Loop: Errors → Checks → Suggestions → Lessons

This is the core of knowledge capitalisation. An error (or a success) becomes a data point, then a concrete suggestion, then — after human validation — a rule loaded into all future sessions.

 Worksite / ReAct loop
      │ resolved error (or reproducible success)
      ▼
Scar registry  (kind=failure | victory ; associated lesson)
      │ immediate vector re-indexing
      │ importance scoring (LLM) ; learnable=true/false
      ▼
Learning engine  (LLM: learnable scar → suggestion)
      │ INSERT suggestion (status='pending')
      ▼
Validation hub  (HUMAN validation: feed / apply / [id])
      │ status → applied | refused | modified
      ▼
Audit log  +  actual application at the destination
      │ (orchestrator directive / config / memory / inbox / skill / tool)
      ▼
Closed loop: the rule is in memory or in the directive → reflex for the next session

4.1 The Scar Registry

The scar registry is the single source of lessons. Each record contains notably: an identifier, the codename of the agent involved, the error type, a description, the lesson learned, the root cause, the correcting agent, the associated tenant, the severity, the resolution state, the nature (kind), an importance score, a recall counter, the date of the last recall, a learnability flag, tags, a proposed category, the qualification confidence and reasoning, and the date added.

• kind separates failures and successes within the same table: failure (majority), cicatrice, victory, plus a few legacy values (bug, mistake, convention, pattern).
• The lesson (check_added): for a failure, what was added to prevent recurrence; for a victory, the pattern to reproduce.
• The importance score (integer) is computed by an LLM inline at record time, via the AI routing configured for the importance-scoring feature (defaulting to a lightweight Mistral model). Non-blocking: if the LLM is unavailable, the field remains null and the nightly loop catches up. This score weights recall (×2 bonus for maximum importance).
• The learnability flag (learnable) is the entry gate of the learning engine.
• Recording a victory: invoke the dedicated skill (/graver-victoire) by setting the kind field to "victory". Never raw SQL. Tags must be supplied as a literal PostgreSQL array (string "{a,b,c}"), not a Python list.

4.2 The Learning Engine — Learnable Scar → Suggestion

The learning engine follows a five-step pipeline:

1. Retrieval: scars flagged as learnable that have not yet generated a suggestion are fetched, sorted by descending importance then by date, with a per-run cap (50 by default).
2. PII scrubbing before sending to the LLM: a depersonalisation filter is applied to the scar, then re-applied to the LLM output before insertion — defence in depth.
3. LLM call: the model is called with a frozen prompt (JSON mode) via the configured AI provider. Note: for this specific engine, the provider and model are hard-coded (a lightweight Mistral model) and do not use the dynamic AI routing from the database.
4. LLM output format: the model returns a structured JSON object containing the target destination, the suggestion text, and a before/after illustrating the proposed change.
5. Insertion: the suggestion is inserted with status pending. The process is idempotent, with a courtesy delay between each LLM call.

The six possible destinations for a suggestion:

Destination	Meaning
claude_md	Rule to add or modify in the orchestrator directive
settings_json	Hook, permission, or environment variable configuration
memory	Persistent level-1 memory note (session reflex)
brain_inbox	New Zettelkasten note in the conceptual inbox (doctrine)
skill	Creation or improvement of an invocable skill
chantier_outil	New worksite or automated tool in the system core

4.3 Learning Tables and Human Validation

Three tables structure this pipeline:

• Suggestions table: identifier, reference to the source scar, target destination, suggestion text, before/after, status, creation date.
• Status cycle: pending → applied | refused | modified.
• Audit log: identifier, reference to the suggestion, applying author, timestamp, snapshot of the applied diff.

Validation is exclusively human, via the hub. Three routes expose the flow: reading the list of pending suggestions, applying a suggestion (transition to applied with an entry in the audit log), and individual update (refusal or modification). The engine only proposes — no self-application is possible.

4.4 Post-Worksite Lessons (Montessori Module)

A dedicated module generates synthetic lessons at the conclusion of a worksite. It is invocable via a specific skill and assigned to the @academy agent.

• It loads a worksite audit bundle: metadata, works, tasks, the last 30 task iterations, linked scars, and execution logs (truncated to ~6,000 characters per execution). Total volume is capped at ~45,000 tokens.
• ROI guardrail: rejects a worksite with fewer than 3 works (overridable with an explicit flag).
• Primary path: the Claude agent is invoked via a node-pty spawn bridge, with read-only restricted tools (Read, Grep, Glob). Two attempts are made, with a pause between them. This path does not use the generic AI client.
• Fallback path (only if the primary provider is absent or explicitly replaced): call to the generic AI client with a lightweight Mistral model. Three attempts with exponential backoff (2 s then 4 s).
• Frozen prompt in 5 sections: estimated vs actual token delta, top 3 recurring errors, top 3 patterns to standardise, 1 candidate doctrine, 1 tool to create.
• Output: a lesson note deposited in the Vault inbox, with frontmatter indicating the type, draft status, and the assigned reviewer.
• Promotion: never automatic. The founder reviews the note and manually moves it to permanent memory (RAG) or level-1 reflex memory. Three-level memory doctrine.

4.5 Patterns → Skills Pipeline (Operational)

Distinct from the scars → suggestions loop, a second operational pipeline transforms recurring patterns observed in ReAct loops into skill proposals. This is the self-improvement pattern via recurrence detection.

Task iterations (reason / action / observation, status done|pass)
      │ weekly scan
      ▼
Skill proposal detector  (LLM-first, via <spawn-bridge>)
      │ INSERT proposal (status='pending', source iterations, pattern evidence)
      ▼
Validation hub page  (Vue + API)  ── HUMAN validation (founder)
      │ list / validate-reject / stats
      ▼
Promotion → native skill  (becomes invocable /<slug>)

• The skill proposal detector (weekly cron, Sunday at 04:00 UTC, after the daily synthesis): loads a batch of recent task iterations with status done or pass, submits them to an LLM (invoked via subprocess with a 180 s timeout) which identifies patterns present in at least 3 tasks and worthy of a skill. For each proposal: if the proposed slug already exists (native skill or pending proposal), the entry is skipped; otherwise, a proposal is inserted with status pending, the source iteration identifiers, and the evidence text provided by the LLM. --dry-run and --limit flags available. LLM-first approach rather than statistical clustering (iteration volume still low).

• The proposal monitor (hourly cron): alerts the founder if the number of pending proposals exceeds a configured threshold (default: 10). One-hour cool-down via state file. Provides statistics: totals by status, average validation time, validations by agent, additions in the last hour. Notifications via the façade email client — never via direct SMTP.

• Human validation: a dedicated hub page exposes the proposal list, detail view, validate/reject actions, and statistics. A validated proposal is promoted to the native skills registry.

Not to be confused with: a second pattern-detection skeleton (Sunday cron at 03:00) is present in the code but remains inactive — its own docstring states this explicitly: it scans and logs, but does not yet create proposals (pattern-matching logic to come). It is the detector described above that actually feeds the proposals registry.

4.6 Other Related Automations in the Loop

4.6.1 The Nightly Maintenance Loop (daily cron, 04:00 UTC)

The nightly maintenance loop consolidates memory and produces insights across ten features (F1–F5, F7–F11 — F6 is reserved for inline importance scoring). Cross-cutting guardrail: never automatic merging from the inbox to permanent memory; dry-run mode (--dry-run) is the default.

The provider and model for each feature are resolved dynamically from the AI routing table in the database.

• F1 — Repeated scar detection: compares today's scars against the vector registry (cosine similarity ≥ 0.85, top 11 neighbours); if at least 2 similar past occurrences are found, a promotion draft is created in the inbox.
• F2 — Session transcript compaction: analyses the day's session logs (max 30 sessions; sessions containing secrets are filtered out by regex) via LLM; extracts 3 to 5 durable facts (decision, delivery, scar, finding) and deposits them in the Vault's daily log. In dry-run mode, no LLM call is made.
• F3 — Morning summary: a synthesis email sent to the founder via the email façade. Skipped if no draft or log was produced.
• F4 — Documentary divergence detection:
  • 4a: verifies 5 canonical tiers in the pricing file;
  • 4b: recursive scan for obsolete patterns defined in a dedicated YAML configuration file (thresholds: ≥ 3 occurrences × ≥ 2 files);
  • 4c: Markdown files with a table ≥ 6 rows and more than 50% numeric content → recommendation to migrate to a database table.
  Results feed drafts into the inbox.
• F5 — Nightly reflection: inspired by Generative Agents (Park et al., 2023). For each active agent, the last 20 resolved scars are submitted to the LLM, which generates 3 introspective questions and 3 abstractions, deposited in the inbox. In dry-run mode, only the count of eligible agents is computed.
• F7 — Forgetting curve: archives (state resolved=2) scars older than 180 days with no recall at all (recall_count=0); cap of 200 archivings per run. Never deletion — reversible by update. Archived scars become invisible to the semantic recall engine (which filters on resolved=1).
• F8 — Worksite ↔ email proactivity: scans worksites with non-empty external contacts (statuses paused, planning, discovery, dev), automatically links incoming/outgoing emails by sender address, reconstructs the full thread, and the LLM classifies the situation as: archive-refused, act-now, awaiting-them, or no-action-needed. (Marco Polo agent.)
• F9 — Worksite obsolescence audit: Marco Polo internal version (read-only). Scans paused or planning worksites inactive beyond a configurable day threshold, computes obsolescence signals, and classifies via LLM as still-relevant, maybe-obsolete, or archive-recommended. Non-still-relevant verdicts surface in the morning summary for the founder's decision.
• F10 — Persona drift audit: monthly, self-gated to the 1st of the month. Scans active agents, measures each persona's drift score, persists the history, and surfaces the top 3 drifting agents in the morning email.
• F11 — Daily cost tick: aggregates the previous day's costs by agent, compares against the 7-day average, and sends an email alert if the delta exceeds 20%.

4.6.2 The Scar Classifier

A dedicated automation classifies scars not yet qualified (with no proposed category). The LLM returns a structured object that is mapped column by column:

• The proposed category populates the error_type_proposed field;
• The reclassified severity updates severity if it differs from the current value;
• The confidence score populates qualify_confidence;
• Tags update the tags array;
• The reasoning populates qualify_reasoning.

Important note: the learnable verdict returned by the LLM is not written directly to the learnable column by this automation. It is serialised as a prefix in the qualify_reasoning field in the form learnable:true|false\n….

4.6.3 Level-1 Memory Consolidation Pipeline

A monthly maintenance pipeline sanitises level-1 memory. It is triggered on the 1st of each month at midnight UTC (with an overlap-prevention lock) and follows four sequential steps:

1. Environment validation: checks Python prerequisites and required paths. This step runs in an isolated subprocess to avoid import side effects. If it fails, the pipeline stops immediately.
2. Dead link detection: analyses the memory index file and identifies references pointing to non-existent files.
3. Obsolete file archiving: moves dismantled or obsolete notes to an off-repository archiving destination, and removes the corresponding references from the index.
4. Fingerprint-based deduplication: identifies and removes perfectly identical files (SHA-256 fingerprint).

A JSON report is generated at each run, indicating for each step: success, duration, and any error.

The pipeline exposes three flags:

Flag	Effect
(none)	Full pipeline with actual writes
--dry-run	Steps 3 and 4 are bypassed or simulated. Note: step 2 (dead link detection) always runs, even in dry-run mode, and may produce report files.
--validate-only	Stops after step 1 (environment validation only)

Exit codes are standardised:

Code	Meaning
0	Full pipeline OK (or --validate-only OK)
1	Environment validation failed — no further step executed
2	Fatal error in a step — pipeline halted
3	--dry-run mode completed (indicative, no changes persisted)

Two environment variables are mandatory (checked at step 1): the path to the level-1 memory directory and the archiving destination. If either is absent, the pipeline returns code 1 without executing any further step.

An executable non-regression quiz (16 checks across 4 modules, 100% score required) validates the actual system state after any modification to the pipeline:

• Environment / dependencies / cron: scripts present, Python dependencies installed, memory directory accessible, cron entry present;
• Execution: --validate-only returns exit 0; --dry-run returns exit 3 or 0; dead link detection functional; semantic consistency validation operational;
• Reports: report present, 4/4 steps OK, 0 dead links, memory index containing at least 10 links;
• Documentation / roles: complete training materials, exit codes documented, roles documented, frontmatter compliant.

Warning: the training materials (docs/consolidation_training.md) are hard-read by 4 of the 16 quiz checks. Deleting this file would drop the score below 100% and block the pipeline in non-regression. Any deletion of this file requires adapting those 4 checks beforehand.

A memory metrics automation (daily cron at 06:00 UTC) feeds a 90-day time series: number of memory files, size in KB, number of lines in the index, dead links, and orphans. Alerts are raised in the monitoring dashboard if thresholds are exceeded.

4.7 User Model Recall (Structured Channel, Distinct from Semantic RAG)

Semantic vector recall is not the only memory channel. A dedicated module loads the user model from a profile table (JSONB dimensions) to inject it into agent briefings — this is a structured and targeted recall, not vector-based.

• Direct slices (JSONB dimension keys): communication style, schedule, profile, skills, history, preferences, strictly private data.
• Composite slice core: communication style + cognitive preferences + appointment schedule. Calibrated for briefing injection (under 1,000 tokens) and never including strictly private data.
• API: get_profile(slice=…) returns a dictionary; format_for_briefing(...) returns ready-to-inject text. From the command line: python3 -m <user-profile-module> --slice core [--json|--format-briefing].

5. Nightly Maintenance Loop

This layer orchestrates documentation consistency and overall system health. A nightly orchestrator inspects documentation fidelity, measures overall health, mechanically repairs dead references, triggers gated regeneration of derived chapters, and pushes a sanitized snapshot to synedre.com.

The step sequence is as follows:

1. Sense — the drift detector measures the gap between documentation and code.
2. Coverage — the coverage scanner identifies blind spots.
3. Repair — the dead-reference fixer mechanically processes broken links.
4. Regenerate — the gated deep-rewrite recomposes derived chapters.
5. Propose — the staging cycle produces drafts pending validation.
6. Publish documentation — automated publication applies guardrails (leak, anti-guru, drift).
7. Re-sense — the drift detector runs a second time, after publication, to measure the actual final state.
8. Health check — the health assessment aggregates five dimensions and produces a care note.
9. Publish snapshot — the synchronizer pushes the sanitized snapshot to synedre.com (best-effort).

Scheduling: the loop runs daily at 05:00 UTC via the system task scheduler. Events are logged to a dedicated log file.

5.1 The Loop Orchestrator

The orchestrator is the single entry point for the nightly maintenance loop. It enforces several global guardrails:

• Database kill-switch: if the system's active configuration indicates that the loop is disabled, the orchestrator stops silently without executing any sub-step.
• Anti-runaway lock: a lock file carrying the current PID is created at startup. If a previous process is still alive, the loop stops immediately. The lock is removed in all cases at the end of execution, including when an exception occurs in a sub-step.
• Semantic return codes: codes 1 and 2 signal respectively a drift condition and a critical health state — these are informational statuses, not fatal failures. The loop only exits with a real error if a sub-step actually fails.
• Daily report: at the end of each execution, an idempotent summary (step : return code) is inserted into the morning report table. The operation is idempotent (DELETE + INSERT on the loop key).
• Double drift measurement: the drift detector runs once at the start of the night, then a second time after documentation publication. This ensures that the final health assessment reflects the actual post-regeneration state, not the state at the start of the night.

The flags available on the orchestrator are as follows:

Flag	Effect
(none)	Full execution in live mode.
--dry-run	All sub-steps receive --dry-run; the kill-switch is bypassed.
--no-deep	Skips the deep-rewrite step.
--deep-max N	Maximum number of chapters processed in the deep rewrite (default: 4). Actual convergence is bounded by --deep-budget-sec.
--deep-budget-sec N	Wall-clock budget in seconds for the deep rewrite (default: 4500 s = 75 min). The loop will not start a new chapter beyond this threshold.

5.2 The Health Assessment

The health assessment component aggregates five dimensions into an idempotent snapshot per execution date, and produces an item visible in the morning dashboard.

Dimension	Data source	Critical threshold
Proprioception	Documentation drift table	≥ 1 chapter with a dead reference
Coverage	Documentation coverage table	Always in warning if ≥ 1 uncovered unit
Debt	Task backlog	At least one P0-priority item pending
Learning	Aggregated scar view	Refresh absent for more than 3 days (refresh cron inactive)
Automations	Cron error log + automation registry	≥ 1 script disabled within the mothership perimeter

• Mothership perimeter: only automations belonging to the mothership perimeter and marked active are included in the health calculation. A client or voluntarily disabled automation does not degrade the metric.
• Care note (Nightingale agent): the Health Guardian agent reads the vital signs and writes a care note of three sentences maximum (in English). This output is stored in the JSON metrics field of the global assessment row. If the agent is unavailable, the value remains null without blocking the assessment.
• Flags: --dry-run (assessment displayed, no writes); --no-voice (metrics only, without care note).
• Exit codes: 0 = ok, 1 = warning, 2 = critical.

The health assessment writes to two destinations:

Destination	Role
Health table	Snapshot per execution date and dimension (idempotent UPSERT), score 0–100, JSON metrics, textual summary.
Daily report table	Morning report item, idempotent on key and date.

5.3 The Public Snapshot Synchronizer

This component reads the latest health and documentation drift snapshots from the mothership database (read-only), sanitizes the content (removal of IP addresses, internal paths, client identifiers), then pushes a JSON snapshot to the synedre.com database.

Guardrails:

• No writes are performed on the mothership side: access is strictly read-only.
• The target table migration is idempotent (CREATE TABLE IF NOT EXISTS) and runs automatically on the first run.
• A single snapshot per day is retained (UPSERT on date).
• In the event of an error during publication to synedre.com, the component returns an error code and logs to standard error, but does not interrupt the nightly loop (best-effort behavior).

The published snapshot exposes: overall and per-dimension health (scores, statuses, sanitized summaries), the list of drifting chapters (slug, type, score), and the Nightingale agent care note. This data feeds the maintenance dial visible at synedre.com/documentation.

Flags: --dry-run (displays the JSON, no writes to synedre.com).
Exit codes: 0 = ok, 1 = mothership read error, 2 = publication error.

6. Centralized AI Facade

Every call to an artificial intelligence provider within the system passes through a single facade. This doctrine guarantees traceability, provider substitutability, and cost control. The facade exposes two canonical functions:

embed(text, *, provider="auto", model=None) -> list[float]

complete(mission, *, context="", provider="auto", model=None,
         json_output=False, timeout=90,
         api_key=None, meter_tag=None) -> str | dict | None

Database-driven routing

When provider="auto", the facade reads the AI routing table from the database. This table exposes the columns feature, embed_provider, complete_provider, model, and extra (JSONB). The structure relies on a global fallback row and per-feature rows that merge with the fallback.

Operational advantage: switching AI provider amounts to updating a single database row. No redeployment is required; the nightly loop re-reads the configuration at each execution.

If the database is unavailable, the facade falls back to canonical defaults: Mistral for embedding, Claude for completion.

Providers and models

Usage	Valid providers	Default model
embed	Mistral, Voyage, OpenAI	mistral-embed / voyage-3 / text-embedding-3-small
complete	Claude (Anthropic), Mistral, OpenAI	claude-haiku-4-5 / mistral-large-latest / gpt-4o-mini

Per-provider behavior

• Embedding: each provider has its own HTTP adapter. Mistral reuses the internal vector callback module for embedding.
• Claude completion (Anthropic): two billing paths coexist.
  • Direct API path: if an Anthropic API key is provided (parameter or environment variable), the facade calls the Anthropic API directly. This path is reserved for usage billed to an Anthropic Console account (clients in Extended Agency mode). It includes retry logic on 429/5xx errors (3 attempts with backoff) and traces each completion in the AI usage metrics table (provider, model, billing tag, input/output tokens, estimated cost in USD). Short aliases (haiku, sonnet, opus) are resolved to full model identifiers.
  • CLI path: in the absence of an API key, the facade delegates to the Gauss agent (neutral persona), on the internal Max plan. This path is reserved for internal use.
• Mistral completion: direct HTTP call, temperature fixed at 0.0, JSON format activatable. 3 attempts (2 retries) on 429/5xx errors with backoff. Non-429 4xx errors are permanent and do not trigger a retry.
• OpenAI completion: direct HTTP call to the OpenAI API.

Traceability and robust parsing

• Usage metrics: each completion via the Anthropic direct API path is traced in the database (provider, model, billing tag, tokens, cost). This mechanism is fail-soft: a tracing error does not block the completion. It enables per-client rebilling via the meter_tag field.
• Robust JSON parsing: in json_output mode, the facade first attempts direct parsing, then code-tag stripping, then extraction by {…} pattern. This pipeline avoids failures on slightly malformed model responses.

Structural guardrails

• No module-level I/O: all logic is encapsulated in functions.
• Lazy provider imports: dependencies are loaded only when used.
• No shadowing of the log variable.

API keys

No key values appear in this documentation. Keys are read exclusively from the runtime environment via named variables. Bearer files are excluded from the Git repository (.gitignore).

Environment variable	Provider	Bearer file
Mistral API key	Mistral	Local environment file (excluded from repository)
Voyage API key	Voyage	Local environment file (excluded from repository)
OpenAI API key	OpenAI	Local environment file (excluded from repository)
Anthropic API key	Anthropic — direct API path (Extended Agency clients)	Local environment file (excluded from repository)
Database password	PostgreSQL — mothership database	Host environment file (excluded from repository)

Variable loading is performed at Python automation startup: environment files at the repository root are parsed and values are injected into the process environment via os.environ.setdefault (without overwriting variables already present in the system environment).

Sovereignty: Mistral is the canonical embedding provider, in line with the project's European digital sovereignty doctrine.

7. Database Connection

Python automations connect to the mothership database via the port exposed by the database container on the host machine. The connection specifies the target database name and the search schema dedicated to the mothership, in order to isolate system tables from public tables.

• The database container hosts a database whose name differs from the container name itself — an important distinction for connection scripts.
• The search schema is passed as a connection option (search_path) so that queries without a schema prefix automatically target the correct namespace.
• The connection user is dedicated to the automations and holds strictly necessary privileges.

Summary table of system components

The following table lists the main components of the system, organized by functional role. Each entry indicates the execution type and the conceptual responsibility of the component.

Component	Type	Role
Note indexer	Scheduled task (every 15 min)	Crawls the note base and indexes entries into the relational registry of structured memory.
Embedding synchronizer	Scheduled task (nightly) + inline call	Computes and stores semantic vectors (1024 dimensions) for all sources; supports targeted hot re-indexing during a ReAct cycle.
Retrieval engine	RAG	Hybrid search (lexical, semantic, score fusion) with re-ranking and a promotion mechanism for relevant results.
Retrieval evaluation harness	Quality assurance	Non-regression test suite measuring recall@k, MRR@10, and top-list precision against a golden dataset and a frozen baseline.
Pre-task retrieval hook	Automatic hook (before creation)	Triggers a hybrid search (top 4) before any structured creation, without blocking the workflow.
Nightly maintenance orchestrator	Scheduled task (nightly, 05:00)	Drives the full loop: perception → coverage → repair → regeneration → proposal → documentation publication → re-perception → care → final publication.
Health report	Multi-dimensional report	Assesses five system health dimensions and records results in the daily tracking registries.
Public health synchronizer	Synchronization	Exports a sanitized snapshot (free of sensitive data) to the public VPS, available on a day-by-day basis.
AI facade	Abstraction layer	Single entry point for embedding and completion operations; handles feature-based routing, dual-path access to the primary model, and per-request usage accounting.
Learning engine	Engine	Identifies lessons flagged as learnable and generates consolidation suggestions pending validation.
Post-task retrospective	Feedback loop	Produces a structured lesson at the end of each task; uses the primary model with automatic fallback to an alternative model if unavailable.
Feature consolidator	Scheduled task (every 4 h, simulation mode by default)	Consolidates up to ten feature axes in test mode; no production effect without explicit validation.
Skill proposal detector	Weekly task (LLM-first, functional)	Analyzes iteration patterns and inserts new skill proposals into the dedicated registry, ready for human validation.
Skill proposal monitor	Hourly task	Sends an alert to the founder when the number of pending proposals exceeds a configured threshold.
Pattern detector (skeleton)	Weekly task — inactive MVP	Analyzes and logs detected patterns but creates no proposals. Distinct from the functional detector above.
Lesson qualifier	LLM	Assigns each lesson an error type, severity level, confidence score, and reasoning; automatically flags lessons eligible for learning.
Memory hygiene	Monthly task	Cleans the note base: removes dead links, archives obsolete entries, deduplicates by content fingerprint; produces a monthly report.
Memory metrics	Scheduled task (every 6 h)	Computes memory volume and quality indicators, retains a 90-day history, and emits alerts when thresholds are crossed.
User recall	Contextual recall	Leverages the user model (preferences and behavioral dimensions in JSON) to produce personalized briefings for agents.
Knowledge graph	Experiment (go/no-go planned ~2026-09-01)	Builds a deterministic graph queryable in at most two hops, with built-in evaluation and a non-intrusive parallel report.

Data registries

Registry	Role
Lesson registry	Failure and success lessons, importance level, learning eligibility.
Vector registry	Semantic vectors (HNSW index, cosine similarity).
Structured note registry	Relational index of the Zettelkasten.
Skill proposal registry	Generated proposals pending human validation.
Skill repository	Validated and active skills (promotion target for proposals).
Learning suggestion registry	LLM suggestions pending validation.
Learning audit log	Immutable trace of every suggestion applied after validation.
Memory time series	Memory health indicators over a rolling 90-day window.
User model	Behavioral dimensions and preferences (JSON), used for agent briefings.
Health snapshots	Daily snapshots of the five health dimensions plus overall score.
Sanitized public snapshots	Snapshots exported to the public VPS, stripped of all sensitive data.
AI routing table	Single source of truth for feature-based routing: embedding provider, completion provider, target model, extended parameters.
AI usage log	Per-request record: provider, model, accounting label, input/output tokens, estimated cost — basis for per-client billing.

Interfaces and user entry points

• Skill validation dial — hub interface allowing the founder to approve or reject automatically generated skill proposals.
• Memory interface — allows the user to browse and enrich the system knowledge base.
• Learning interface — entry point for triggering or supervising learning cycles.
• Success recording interface — allows a notable success to be explicitly recorded as a lasting lesson.
• Golden dataset — set of reference queries used by the evaluation harness to measure retrieval quality.
• Frozen baseline — snapshot of recall@k, MRR, and top-1 metrics serving as the automatic non-regression threshold.