Research spike · finding
Can the wiki tag itself?
Before building a feature that reads every page and labels the concepts it mentions, a measurement test checked whether the approach could be accurate enough to trust. The test subject was biology.
What was being tested
The goal is a McBrain wiki that labels itself. Point it at a standard, published catalog of concepts for its subject — think of a shared, expert-agreed dictionary where every term has a fixed ID and sits in a family tree of broader and narrower ideas — and an AI reads every page and records which of those concepts each page mentions. Do that, and the wiki can answer "show everything about ion channels" with a complete, hierarchy-aware result instead of a keyword guess.
The catch is accuracy. If the auto-labeling is sloppy, users stop trusting it. The team set a bar before writing any real feature code: at least 85% of the labels it applies must be correct. This spike is a throwaway test rig built to answer one question — can the approach clear that bar? — using a published answer key instead of live wiki pages.
Why this matters
Better to run the experiment first, before writing any code for the actual app. Testing the idea against a known-correct answer key is quick — this spike took a couple of hours — and it settles the question cheaply. If the approach can't hit 85% on a dataset where the right answers are already written down, it won't on messy real pages, and it's far better to learn that from a throwaway test rig than from a half-built feature. And if it can hit the bar, the spike points to exactly which settings to ship.
The answer key was biology. The test used CRAFT — 97 full biomedical research articles that expert curators have already hand-labeled against standard biology catalogs. Because a human expert already decided the correct labels for every article, the machine can be scored against them precisely.
How the test worked
The machine labels a page in three passes, and the spike measured each one to reveal which pass was actually doing the work.
- Find candidates. Scan the text for words and phrases that match a concept in the catalog — including plurals and short forms. A second, "meaning-based" matcher was also tried, meant to catch concepts that are described but never named outright.
- Have an AI judge. An AI reads each candidate in context and accepts or rejects it. It can only say yes or no to a real catalog term — it can never invent one.
- Score it. Compare the machine's labels to the expert answer key on two measures: precision (of the labels it applied, how many were right) and recall (of the labels it should have found, how many it caught).
The pipeline ran several times, changing one thing at a time, and — importantly — the winning recipe was confirmed on a held-out slice of articles the tuning had never seen, so the result isn't just the settings memorizing the test.
What the spike found
The pipeline was built up one pass at a time, each version scored against the same 97 articles so the comparison is fair. Here is what each pass added.
| Pipeline | Precision | Recall |
|---|---|---|
| Word matching only | 60% | 74% |
| + meaning-based matcher | 56% | 75% |
| + AI judge | 85% | 49% |
The table settles two questions on its own. The meaning-based matcher backfired — adding it dropped precision (60% to 56%) while barely touching recall, because it flooded the results with noise. It was dropped. And the AI judge is what buys accuracy: it cut wrong labels from 444 down to 60 and lifted precision to the 85% bar.
But the judge also pulled recall down to 49%, and that turned out to be an artifact of how it was fed. Early versions only showed it the opening of each article, so it rejected concepts mentioned later for "no evidence." A follow-up experiment fixed this by having the AI read the whole document — recall recovered without costing precision.
Clears the bar. The final recipe — word matching plus the AI judge reading the full document, meaning-based matcher dropped — was then confirmed on 38 fresh articles it had never seen during tuning: 89% precision, 72% recall. Above the 85% target, on data that couldn't have been memorized.
An automated tool that rewrites the AI's instructions to squeeze out more accuracy was also tried. It didn't beat a ten-minute manual edit — deleting the single phrase "be conservative" from the instructions did more than the automated optimizer. Worth knowing the fancy tool isn't needed at this scale.
The honest failure — a different domain
Cell types worked. To see whether the same recipe carries to another domain, it was pointed, unchanged, at a catalog of chemicals — using the very same articles, just scored against their chemical labels instead of their cell-type labels. It fell apart.
| Catalog | Precision | Recall |
|---|---|---|
| Cell types | 89% | 72% |
| Chemicals | 46% | 61% |
Precision collapsed from 89% to 46%. The reasons are specific and fixable, but not free. A chemical like "glucose" maps to several near-identical entries in the catalog (glucose, D-glucose, alpha-D-glucose); the answer key picks exactly one, so the machine's other, defensible guesses are scored as wrong. And words like "water" and "oxygen" appear all over biomedical text, but the curators deliberately didn't tag most casual mentions — too trivial to index — so every one the machine tagged counted against it, several hundred per run.
Fixing this isn't a settings tweak. The candidate-finder has to be taught the chemistry-specific rules — collapse the duplicate IDs down to one, and ignore ubiquitous filler terms — which is real curation work, per catalog.
Bottom line
The approach is sound and it clears the accuracy bar — but only after per-catalog tuning, and that tuning ranges from "pick one setting" (cell types) to "roll up your sleeves and clean the catalog" (chemicals). The plan should ship the catalogs that tune cleanly first, and budget real curation time for the harder ones rather than assuming one recipe fits every domain.
Honest caveat
Two limits carry forward. First, catching 72% of the right concepts means the machine still misses about a quarter of them — the design closes that gap with light human review, which is real ongoing labor, not magic. Second, every number here is measured against a tidy biology answer key. Real wiki pages are messier, so these results have to be re-checked on actual vaults before they can be trusted in production.