Our New Proofreader Can't Write a Single Word
We had an AI that only answers yes or no reread every rule we publish. Here is how that went.
Last week TypeSafe released Jev, an AI model that can't write. You give it some text and a list of questions, and it answers with things like "yes, 0.96" or "option B." It won't summarize the text or explain itself, and an answer usually comes back in about a sixth of a second. TypeSafe calls it a System One model, after the fast, gut-reaction half of Kahneman's two kinds of thinking.
Their launch demos have it playing Doom and racing through Wikipedia links. We had a duller job in mind. Every rule on Court Rules comes with the judge's own words attached, so that a reader can check our summary against the sentence it came from. A model that does nothing except compare a claim to a piece of text is pretty much a proofreader, and we had about 64,000 rules that had never had a second pair of eyes on them all at once.
Twenty minutes and $1.34
Each rule we publish starts with the judge's sentence, quoted word for word. Under that is our plain-English summary, and under that a handful of structured fields (a page limit, say, or a number of days). The quote gets matched against the source document before a rule goes live. The summary and the fields are written by an extraction model, and until now the only way to double-check them was to pay a second large model to reread everything. That is slow and expensive enough that we had only ever done it in samples.
Jev read all of it in twenty minutes, and the bill was $1.34. For each rule we asked it two things: does the summary say what the quote says, and is each structured field something the quote states?
What it caught
About 95 percent of rules came back clean. In the rest, the quote and the citation were fine, and the trouble was a summary or a field that said slightly more than the judge did. One judge wrote that a party "may submit" two courtesy copies of a compendium of authorities, and our summary had turned that into a requirement. Another judge encourages junior attorneys to argue motions, and our structured data had attached a years-of-experience cutoff the judge never gave. A rule about tabbing exhibit binders had picked up page margins from somewhere.
A litigator mostly reads a rule to find out whether something is required or just allowed, so we took these seriously. We went through a sample by hand, agreed with Jev, and fixed the flagged rules the same day.
We also changed the form our extraction model fills in. It used to have required blanks for things like margins and experience cutoffs, and a model handed a required blank will find something to put in it. Those fields are optional now, and an empty one means the judge didn't say.
Where it was bad
Jev was worse at anything that wasn't a yes-or-no question. We gave it a menu of eleven document types and asked which one a page was, and it decided that nearly every blank template order was an order entered in one particular named case. We never figured out why. When we broke the menu into three separate yes-or-no questions, it agreed with a careful referee almost every time. Parallel ran into the same thing in their tests.
Its confidence scores only mean something at the ends. When it was 95 percent sure a summary contradicted its quote, it was usually right. At 60 percent it was right about one time in four, and once it insisted that "at least 21 days before" and "no later than 21 days before" were in conflict. Another time it pulled the correct number out of a 90-minute time limit and then told us the unit was hours.
And it has no ideas of its own. It will count any category you hand it across every rule we have, for pennies, but we had to bring the categories.
Questions we couldn't afford before
The cheapness changes what you bother to ask. We would never have paid a big model to find out how judges phrase deadlines, but at these prices we just ran the question across the nearly nine thousand rules that have one. More than a third just say "days" and leave you to work out which kind. About one in fourteen say "court days," which skip court holidays and are not the same thing as business days. We are reworking our deadline fields around what that survey found.
We also ran it over about four thousand source documents with a single question: does this page tell a litigant to do something? That took 70 seconds. It turned up several hundred pages we had set aside, most of them the blank template orders judges post for parties to fill in, which do carry requirements if you read them. A second model that had to quote its evidence agreed on 39 of the 40 we pulled at random. Those documents are in the extraction queue.
What it does now
Jev now gets the first say on whether a document we collect contains rules at all. The large model only weighs in when Jev is unsure. Next it gets the proofreading job for good, so every summary and field is checked against the judge's sentence before a rule shows up on the page you search. We will write again once that has been running for a while and we can say how much it catches.