Skip to main content

​
Learnings — append-only memory of the docs-sync harness

  • 2026-08-06 · harness created; state seeded at mnfst/manifest PR #2691 (last merged before the harness existed).
  • 2026-08-06 · run 1 (message recording): the auditor inverted the REQUEST_RECORDING_STORAGE auto rule (claimed filesystem fallback on partial S3; code returns a null backend). The verifier’s code re-read killed it before publication. Lesson: auditors must quote the exact decision line, not summarize control flow.
  • 2026-08-06 · run 1: the platform-engineer’s report propagated the same misreading — verification must re-read code even when the report cites file:line.
  • 2026-08-06 · run 1: never name internal tools in the issue itself, even in the “clean” summary (public repo).
  • 2026-08-06 · run 1: keep the “Questions for Seb” section OUT of the public issue; deliver it privately.
  • 2026-08-06 · run 1 verdicts: findings must be written in PLAIN language — no UI-component jargon (no “drawer”, “Frame”, CSS/file names in the finding body; code refs stay in the evidence lines only). Seb could not understand the IMAGE findings as written. An IMAGE finding is one simple sentence: where in the doc, what photo to take.
  • 2026-08-06 · editorial: when history is convoluted (feature added/removed/re-added), docs say the simple useful thing, not the history. Recorded in AUDIENCE.md.
  • 2026-08-06 · terminology: “logs”, “request”, “attempt”; the platform will rename the /messages URL later.
  • 2026-08-06 · full audit (cron, autonomous, no human): 0 PRs merged on mnfst/manifest since baseline #2691 (max merged PR = #2691). Nothing to compare → no findings → no issue (silence is a valid outcome). lastAuditedPr left at 2691; lastRunAt bumped as a run heartbeat.
  • 2026-08-07 · full audit (cron, autonomous, no human): still 0 PRs merged since baseline #2691 (origin/main @ 47d35fa, 2026-08-05, contains merge of #2691; max merged PR = #2691). Nothing new → no findings → no issue. lastAuditedPr unchanged; lastRunAt bumped as heartbeat.
  • 2026-08-07 · issue format hardened: human-first (one plain sentence per finding, options one line each, ALL evidence collapsed in a details block). Seb could not tell two images were expected of him.
  • 2026-08-07 · questions to Seb must be complete and case-specific (state: existing agents vs newly created, cloud vs self-hosted, pre/post release). “Is recording live on Cloud?” was unanswerable as asked.
  • 2026-08-07 · Seb’s answer on cloud defaults: pre-existing agents have logs OFF; at agent creation the Connect Harness modal pre-checks Autofix and Logs (suggested defaults, like consent checkboxes) — the user confirms. Transition info that only concerns pre-release users is not worth documenting.
  • 2026-08-07 · internal tool names are never written anywhere in this public repo (including harness reference files); the engineer derives the list from code at each run.
  • 2026-08-07 · full audit (cron, autonomous, no human): first run WITH new PRs since baseline. 4 merged PRs #2693/2694/2696/2697 (#2695 not merged). 10 findings filed as issue #48; lastAuditedPr advanced 2691→2697. Categories: WRONG ×4, MISSING-CASE ×5, INTERNAL-LEAK ×1. Autofix-spelling and removed-Gemini-model leads were dropped by the auditor (docs already clean) — a lead from a PR title is not a finding until the docs actually contradict.
  • 2026-08-07 · a single “infra plumbing” PR (#2693, durable recordings across deploy templates) produced 8 of 10 findings across 7 deploy pages + request-logs. Lesson: template/IaC PRs that touch deploy/* fan out widely into the deploy guides; audit every touched deploy target, not just the feature’s own page. Each target falls in a different bucket (auto-provision vs mount-volume vs require-external-S3) and each needs its own case check.
  • 2026-08-07 · INTERNAL-LEAK finding on a public issue is doubly constrained: the leak is IN the docs, so the finding must point by file:line and describe the tool generically (the verifier grep-gates the draft for the codename/hostname). When the leak is the VARIABLE NAME itself (WINGMAN_CORS_ORIGINS), an interim doc reword cannot fully hide it — the real fix is a platform rename; offer both as options, don’t pretend docs alone close it.
  • 2026-08-07 · Railway recurs as an “unverified” edge: README claims durable S3 storage, but railway.toml (external template) has none in-repo. Kept as an OPEN QUESTION in decisions.md, out of the public issue (questions-to-Seb stay private). Resolve by confirming the actual Railway template before touching deploy/railway.mdx or request-logs.mdx line 118.
  • 2026-08-07 · run 2 (introduction): verifier killed two false evidence statements (“renamed everywhere but here” — 5 leftovers existed; “three times” — it was two). Lesson: universal quantifiers (everywhere, always, all) in evidence must be grep-verified before writing.
  • 2026-08-07 · run 2: the human-first issue format worked — findings are one plain sentence + collapsed proofs.
  • 2026-08-07 · run 2: issue #49 was first published in FRENCH — forbidden. Everything on GitHub is English, always; French stays in the private conversation. Rule added to ISSUE-FORMAT.md; the verifier must fail any non-English draft.
  • 2026-08-07 · issue rewrite #2 after Seb’s feedback: options were fake (restating the problem or citing context as if it were a choice) and recommendations repeated the title. New rules in ISSUE-FORMAT: single obvious fix = one “Fix:” line; options must be genuinely different outcomes, one line each; recommendation marked once; issue-writer voice rules apply.
  • 2026-08-07 · Seb’s idea adopted: every finding carries a visible “What actually happens” block — the user journey in 2-6 user-language steps, derived from code. It lets Seb verify the behavior before judging options and doubles as the spec for the fix. Verifier walks the code to confirm each step.
  • 2026-08-07 · Seb challenged finding 3 (“are you sure?”). Proof delivered by LIVE reproduction against Cloud (fake model → M302 verbatim; auto → routed). Rule: when a behavior finding is reproducible with one cheap request, reproduce it and put the transcript in the evidence — reproduction beats code citation.
  • 2026-08-07 · finding 5 was unreadable because the auditor hid the instance-mode variable’s name, over-applying the internal-names ban. Rule scoped: the ban covers names with no user-facing existence; anything already printed in public docs or product output may be named in findings when needed for readability. Also: when evidence makes one option clearly right, write a single deterministic Fix, not options — ten agents must produce the same issue.
  • 2026-08-07 · finding 5 fix over-documented the instance-mode env var: the variable is user-facing only for the value selfhosted; cloud (and the mode concept itself) is internal. Lesson: documenting a variable means documenting its USER-FACING surface, never the full internal value set.
  • 2026-08-07 · finding 5 finally REJECTED by Seb after two fix attempts. Lessons: (a) a Default column shows a value, never a described behavior (“auto-detect” was invented); (b) an escape-hatch variable fully explained where users meet it (guides, error message) needs NO reference row — in-place completeness beats reference completeness. Reference tables are for variables users must discover, not ones the product hands them.
  • 2026-08-07 · instance-mode saga, final lesson: I oscillated (documented too much → removed entirely → restored) instead of holding the invariant “never halfway: used in a guide = defined in the reference”. The halfway state I created by removing the row while guides kept the variable was exactly the original finding. Rule added to AUDIENCE.md; when Seb expresses doubt, re-derive from invariants, do not flip.
  • 2026-08-08 · scoped run (autofix, cron, autonomous, no human): CLEAN, 0 findings, no issue. A dense, current page (every status code, env default, guardrail constant, and data-flow claim correct) is a valid silence — do not manufacture a finding to have output. Ran the full loop anyway: platform-engineer state-of-truth → docs-auditor cross-check → issue-verifier as a FRESH-EYES ADVERSARY tasked to break the zero-findings conclusion. Repurposing the verifier to attack a silence (not just to kill a draft) is the right guard against the scoped audit’s worst failure mode, a false all-clear.
  • 2026-08-08 · autofix run, two near-findings that are NOT findings, recorded so they are not re-litigated: (a) “keeps its state” for pre-existing agents — the nullable-reset migration (1799000300000) resets explicit-false rows to NULL, so the internal mechanism differs, but the user-visible effect (cloud=on, self-hosted=enable-yourself) holds via resolveEnabled → NULL maps to !isSelfHosted(); user-visible truth is the bar, not the mechanism. (b) Anthropic subscription-exhaustion HTTP 400 is skipped by the heal path (autofix.service.ts:258-278) even though 400 is repairable; this aligns with the page’s own concept (“errors where the provider is telling you the request was wrong”), so enumerating it would add provider-specific noise, not fix a contradiction. Also: the observe feed excludes 401/402/403/429; the “When it runs” prose lists “401, 403, 429” as skipped (402 also not healed) — illustrative, not exhaustive, not worth a finding.
  • 2026-08-08 · internal names surfaced by the engineer this run (kept out of every published surface, descriptors only): the healing backend’s codename, an internal historical-scrape agent, an internal analytics-tier name, and the healer’s infra hostname. All absent from the two doc pages (grep-gated). The docs’ user-facing descriptor “the healing service” is the correct substitute.
  • 2026-08-08 · full-mode run over a 1-PR window (#2701, Bedrock GPT-5 responses-path fix): CLEAN, valid silence. An internal reverse-proxy buildPath change (upstream endpoint chosen server-side from the model id) has NO docs surface — the documented contract is the client-facing /v1/responses endpoint, not the internal Bedrock upstream /openai/v1/responses. Distinguishing the two same-named paths was the crux: reference/api.mdx:33’s /v1/responses is where users POST; the PR only changed what Manifest calls downstream. Ran the full loop anyway (truth → auditor → verifier-as-adversary); repurposing the verifier to attack the all-clear is the standing guard for clean runs. Also confirmed the discipline of NOT publishing regex-fixture model codenames (gpt-5.6-sol/terra/luna, gpt-5.99-future) as a supported-model list — they are test inputs, not a catalog.
  • 2026-08-09 · scoped run (llm gateway, cron rotation, autonomous, no human): 5 findings on a page that READ as dense/authoritative but carried three hard falsehoods about status codes. Lesson: exact HTTP-status claims are the highest-yield audit target on gateway pages — the doc confidently stated three wrong ones (424 on exhaustion, 429 for M200 blocks, /v1/messages force-routing). All three were falsifiable against a single predicate/constant in code (shouldTriggerFallback status >= 400; buildFriendlyResponse status: 200; the explicit-route branch running before any apiMode check). Verify every literal status code and endpoint claim against the exact line, never against the page’s own internal consistency.
  • 2026-08-09 · the “424 fallback exhausted” story was self-reinforcing across the page (frontmatter keyword :6, prose :97, prose :138) — a wrong concept propagates into keywords and multiple sentences. When killing a false concept, grep the whole page (incl. frontmatter) for every echo, or the fix leaves orphans. Filed as ONE finding with all three loci, not three.
  • 2026-08-09 · verifier caught a real self-contradiction the auditor+writer both missed: finding said exhaustion keeps “the last provider’s” status; code preserves the PRIMARY/original model’s status (proxy.service.ts:1258 status: primaryStatus), and the finding’s own evidence block already said “primary”. Lesson: primary-vs-last (and primary-vs-terminal) is a precise distinction on fallback chains; the visible journey step must match the evidence line verbatim. Fresh-eyes re-read of BOTH sides catches drift the drafting agents introduce.
  • 2026-08-09 · “up to 5 fallbacks” is NOT gateway-enforced (no numeric cap in proxy code) but the frontend copy (RoutingDefaultTierSection.tsx:164) says “up to 5” — so the doc matches the product the user sees. Lesson: a UI-copy convention is the right bar for a user-facing count, not a backend constant; don’t raise a finding when code lacks a cap the UI advertises. (welcome-helpers MAX_FALLBACKS=2 is only the auto-seeded clean-agent default, unrelated.)
  • 2026-08-09 · #2639 (forward uncatalogued models to providers) turns a former absolute (“unlisted model → M302”) into a conditional (M302 only when no connected route). Raised as low-severity MISSING-CASE with an honest “leave as intentional simplification” option 1, because the harm is success-in-the-user’s-favor, not a broken request. A false universal is still a finding, but severity tracks user harm and the option set must include leaving it.