Neurotype
Neurotype lets you type fast without caring about spelling while an AI reconstructs what you meant. The typing tool is the working example. The real subject of this repository is managing an AI the way you would manage a person: define done, build an evaluation, coach from failure patterns, write an SOP, and let the AI improve itself while a human decides when to stop.
Define what done means
For an AI system, "correct" is a policy rather than a fact. Before measuring a model, a human has to decide what good output looks like wherever reasonable defaults conflict.
The ground truth is "tbh i dunno if we need another dashboard": fix the typos, keep the voice. The model's rationale said it fixed grammar and used standard written form. With the policy unstated, it applied its own default.
In another case, a bug report quotes a checkout page that literally says "Paymnet faild, plase try agian". Fixing those typos destroys the evidence, so the ground truth keeps the quote byte for byte. The ground truth in the dataset is where these decisions live.
Build an evaluation
An evaluation has three parts: a dataset, a metric, and a baseline. The dataset holds 82 short messages, each a pair of what was typed and what was meant, built by a seeded generator script that produces the same output every time. It covers six patterns, described below. A glossary file lists the people, projects and places the writer mentions often, such as Siobhan, Joaquin, the project Kestrl and the dashboard Bluefynn.
The metric is character error rate (CER): the edit distance between the output and the ground truth, divided by the length of the ground truth and capped at 1. Lower is better; 0 is perfect. Exact-match rate is a secondary metric.
The corrector runs on deepseek on AI Builder Space at temperature 0. The baseline, v0, returns the input unchanged and scores 0.109. Version v1 is the first prompt anyone would write: fix all spelling and typing errors, return JSON with the corrected text and a rationale. It scores 0.079 over three runs, with 34% of messages exactly right.
Know your noise floor
Even at temperature 0, the same unchanged version gets a different score every time it runs.
Across three runs per version, the largest range between the best and worst run is 0.007. That number becomes the noise floor in the SOP: on a single run, only a change larger than 0.007 counts, and a smaller gain has to be confirmed by comparing three-run means. A full benchmark run takes about 17 seconds (82 items, 32 parallel calls), so repeating it is cheap.
Make it observable
The dashboard has two layers. Layer 1 answers what: every version and every run, with the total CER, the exact-match rate and the CER of each pattern. Layer 2 answers why: each case with its input, ground truth, output, score, and the model's rationale, optionally side by side with another run. The rationale is what turns a score into a diagnosis. It shows whether a failure came from missing ability or from the model following a rule nobody gave it.
Open the dashboardCoach from failure patterns, not single cases
Patching one case at a time produces a prompt full of special cases. Coaching works on patterns: group the failures, decide what kind of failure each group is, and apply the fix that matches the kind. The kind also decides who can fix it.
| Pattern | Kind | What it looks like | Items | Fix | Pattern CER before → after |
|---|---|---|---|---|---|
| casual voice | Policy | Slang, chat abbreviations, informal grammar, all lowercase. tbh i dunno | 13 | v2: keep the writer's voice | 0.196 → 0.004 |
| glossary names | Context | Names typed by sound. shivon → Siobhan | 12 | v3: put the glossary in the prompt | 0.116 → 0.003 |
| pasted verbatim | Policy | Quoted UI strings, code, URLs, emails, hashtags that must not change. "Paymnet faild" | 18 | v4: never touch pasted text | 0.047 → 0.006 |
| shifted hands | Ability | Last words typed one key to the right. yrcy ,r ;syrt → text me later | 5 | v5: describe the drift and how to decode it | 0.30 → 0.16 |
| unknown names | Data | New people and projects missing from the glossary. keeva → Caoimhe | 6 | None in the prompt. A human updates the glossary. | stays near 0.08 |
| plain typos | none | Ordinary slips. Teh quarterly repport | 28 | Any prompt (v1) | 0.103 → 0.002 |
Policy failures happen when the model applies a defensible rule that is not ours. A human decides which rule is correct, and the prompt then states it. On both policy patterns, v1 was worse than doing nothing: casual voice went from 0.075 under the echo baseline to 0.196, pasted text from 0.042 to 0.063. Context failures happen when the answer depends on facts the input does not contain; the fix is to give the model those facts. Ability failures happen when the model knows what is wanted but does it unreliably; a described procedure helps, partly. Data failures happen when the answer exists nowhere in the system. No prompt can recover a name nobody wrote down.
The dataset was tuned together with the corrector over four drafts, until each fix moved the total by a visible step. Three patterns were dropped along the way: messages that look like requests to the AI (the model never fell for them), British spelling (the model already kept it), and fully garbled messages (too noisy, and they dominated the score).
The curve, and what the human decided at each step
Across five versions, mean CER fell from 0.079 to 0.021 while the exact-match rate rose from 34% to 75%. Each dot is one full run; the line connects the means.
Each step is one human decision. v1 is the naive prompt, and the human's job there is reading the rationales. For v2 the human decided on a policy, "keep the writer's voice", and casual voice fell from 0.196 to 0.004. For v3 the human recognized missing context and put the glossary in the prompt; glossary names fell from 0.116 to 0.003. For v4 the human decided that pasted text is evidence and must not change; that pattern fell from 0.047 to 0.006. For v5 the human judged the shifted-hands errors worth one attempt and described the keyboard drift; the pattern fell from 0.30 to about 0.16. Unknown names stayed near 0.08 in every version. That is the floor.
CER per pattern and version, mean of three runs. Darker means more error. Hover a cell for the individual runs.
The per-pattern view shows side effects that the total hides. In v2, "keep their capitalization" also kept typed names in lowercase, and glossary names went from 0.081 to 0.116 while the total improved. After v3 added the glossary, the model began seeing names everywhere: "me and hmi" became "me and Niamh", and "who aye my yogurt" became "who Aoife is". Casual voice rose from 0.004 to between 0.008 and 0.025. The glossary also pulls unknown names toward known ones: from v3 on, "osheen" (Oisin) becomes Siobhan in every run.
Write the coaching down as an SOP
A coaching conversation is lost when the session ends. Writing it down in SOP.md turns it into a procedure that a person or an AI can follow. It was written right after the first coaching session (v1 to v2) and records that session as problem, diagnosis, decision, fix and verification, followed by the procedure:
- Measure the current best version over three runs.
- Find one pattern by reading at least five failing cases with their rationales.
- Classify it: policy, context, ability or data.
- Change one thing in a new prompt version. Examples must be new, never copied from the dataset.
- Smoke test on the pattern's cases.
- Run the full benchmark. Keep the version only if the gain beats the 0.007 noise floor; if it is smaller, run twice more and compare three-run means.
- Record the result in the prompt's notes.
The SOP also says when to stop and hand back to a human: when two iterations bring no gain, when the remaining failures are data, when a policy question is not answered by the ground truth, or when fixes keep trading one pattern against another.
Let the AI improve itself
With the SOP in place, the loop runs from a one-line meta-prompt:
A coding agent can run it as is. The repository also has a script, auto_improve.py, that runs the same loop with gpt-5 on AI Builder Space as the coach. Every session below starts from v2 (CER 0.051).
| Session | Iterations | Time | What the coach chose, and the verdict | Best CER |
|---|---|---|---|---|
| A (script) | 2 | 108 s | glossary names, kept (0.034) · shifted hands, kept (0.024) | 0.024 |
| B (script) | 2 | 104 s | glossary names, kept (0.035) · shifted hands, kept (0.025) | 0.025 |
| E (script) | 2 | 103 s | glossary names, kept (0.036) · pasted text, kept (0.022) | 0.022 |
| C (script) | 4 | 305 s | glossary names, kept (0.035) · shifted hands, kept (0.026) · pasted text, kept on a three-run mean (0.019), flagged that shifted hands got worse · shifted hands again, kept on a three-run mean (0.013) | 0.013 |
| D (Codex CLI, one-line prompt) | 2 | 103 s | glossary names, kept (0.034) · shifted hands, kept on a narrow single-run margin (0.027) · also flagged one ambiguous label for human review | 0.027 |
In every session the coach chose the glossary first, the pattern with the largest share of the error, and named its kind correctly. For the second iteration it picked shifted hands in four sessions and pasted text in one. The Codex session also did something nobody asked for: it noticed that one expected output reads "nobody cares" where the typed text supports "nobody cared" just as well, and flagged the label for a human instead of tailoring the prompt to it. The prompts each session wrote, and their logs, are in docs/sessions/.
Earlier development sessions, run before the SOP had a rule against it, showed how an unguarded loop cheats. In four of them the coach wrote worked examples into the prompt that were copied straight from the benchmark, including the exact shifted-hands cases. In one it went further: to fix unknown names, it added Caoimhe, Eamonn and Oisin to the prompt, names it could only have learned from the expected outputs it was shown. Unknown names "improved" from 0.082 to 0.029. Nobody asked it to cheat; it was optimizing the number it was given. The fix came from a human: a rule in the SOP (never copy a case into the prompt, never add a fact you only know from the expected outputs) and a check in the script that rejects such prompts. None of the sessions since then has leaked.
Another development session compared a new version against a single lucky run of its parent and concluded that there was no gain. The SOP now requires a three-run baseline.
Where the human stays
After the reference v5, most of the remaining error comes from unknown names (data) and some from shifted hands (ability). This is where the SOP's stop conditions apply. Whether to continue, change the model, fix the data or stop is a human decision. That is the conclusion of the module: the human moves from doing each step to judging the system.
Run it yourself
The README explains how to run the benchmark, the app, the dashboard and the self-improvement loop. A full benchmark run takes about 17 to 25 seconds, 75,000 to 135,000 tokens and about $0.03 to $0.05 of platform credit; later prompts are longer because they carry the glossary and the rules, and the platform adds about 400 tokens of its own to every call. Two self-improvement iterations take under two minutes and about $0.32 with the script.