The learning loop

This page is the repository's own document docs/learning.md, shown as it is, so that it never drifts from the product.

"Training" here is not fine-tuning. Three things are built: a corpus of public trials, patterns learned from it that the assistant reads, and a benchmark that scores the assistant against real SAPs on trials it never learned from.

The corpus (tools/corpus/build.ts)#

• Source: the ClinicalTrials.gov API v2 (no key, a generic User-Agent, nothing about the user sent). Interventional phase 2 or 3 trials, primary completion on or after 2021-01-01, results posted, and a protocol and a SAP as separate files (a combined file would put the SAP into the protocol the benchmark reads).
• Two strata, --per-stratum trials each (default 15): A a time-to-event primary endpoint (log-rank, Cox); B a continuous or binary primary endpoint whose protocol or SAP uses the words estimand and intercurrent. At most two trials per sponsor; the first pass takes one trial per first-named condition.
• Success is judged from the posted results (tools/corpus/lib.ts judgeSuccess): a primary outcome analysis with a p-value below 0.05 (the real threshold is not in the record; the rule is for the sorting, not a claim), the direction checked for a hazard ratio. A non-inferiority analysis, a mix of p-values, or no p-value makes it uncertain, and the uncertain trials are kept and marked in meta.json. A trial judged unsuccessful is left out.
• The split is made once, by a fixed seed (standards/exemplars/corpus.json): two thirds development, one third held out, per stratum. Patterns and prompts are made from the development set only; only the benchmark reads the held-out set. A trial that a guide of docs/guides/ already cites is marked inGuides and is not used by the benchmark.
• The sponsors' PDFs, their text and the records of the held-out set stay in .scratch/corpus/ (git-ignored). The repository has the tools, corpus.json (ids, sponsors, titles, dates, the split) and the records and patterns below, which are facts in the extractor's own words.

What is learned (standards/exemplars/)#

1. tools/corpus/extract.ts reads a SAP and writes one record (records/<NCT>.json): the estimand attributes and each intercurrent event with its strategy, the primary analysis, the missing-data handling, the multiplicity procedure, the populations, the time-to-event details, the sample size. Own words, no sentence copied, null for what the SAP does not say.
2. tools/corpus/patterns.ts counts over the development records and writes patterns/*.md and index.json: for a situation, what the SAPs do ("6 of 7: stratified log-rank"), what varies, what the statistician decides, the NCT ids. The counts are computed; the "varies" and "decide" sentences are written by hand in that file. A count tells what sponsors do, not that it is right.
3. src/exemplars.ts reads them (nothing happens when the folder is absent). The hook is exemplarsFor({ request, block, endpoint, limit }) -> { text, items }: text is a prompt section ("What recent public SAPs do"), items is what the interview box shows; exemplarsNote(request) is the same text with the blank lines before it, for a prompt that is built by appending. A pattern bears on a request when the endpoint type agrees and the kind of block or enough words match.

The benchmark (tools/sap_bench.ts)#

For a held-out trial: the record of the real SAP is extracted (apart, in .scratch/corpus/bench/records/); sapling new makes the skeleton from the protocol only; the assistant drafts the parts that carry the statistics (populations, analysis methods, interim analyses for time-to-event, estimands, multiplicity, missing data, additional analyses) with draft() of src/assistant.ts (its context, retries and compile check); the drafted plan is read into a record by the same extraction; tools/corpus/score.ts scores item by item (match 1, partial 0.5, miss 0, contradicts 0; the rules are in that file and tested in test/sap_bench.test.ts). Two arms: base and ex (the same plus the patterns of exemplarsFor for the part). Two scores: compiled (only what compiled in its place) and content (a part that did not compile is scored as the AI wrote it, apart from the compiler's friction). The compiler's own measures are in the report too: errors, warnings, review findings, citations and citation errors, the decisions left open.

Every AI answer is cached on disk and counted (tools/corpus/cached_agent.mjs; --max bounds the live requests), so a run is resumable and a rerun costs nothing. The AI is the claude CLI with the owner's login, model sonnet.

node tools/corpus/build.ts --per-stratum 15            # the corpus (cached)
node tools/corpus/extract.ts --set dev                 # the records of the development set
node tools/corpus/patterns.ts                          # the patterns
node tools/sap_bench.ts --per-stratum 2 --max 150      # the benchmark on held-out trials, both arms
node tools/sap_bench.ts --report                       # the report from what is on disk: .scratch/corpus/bench/report.md

What a difference means: a few trials are a few trials. The report gives the per-trial differences and their spread; a mean difference smaller than about two standard errors is not a finding. A published SAP is not a gold standard: the report lists the contradictions so that a person judges which side is right.