It can't hallucinate. It scored a room nobody mentioned.

01 The problem

Restaurant reviews are the most honest messy text most businesses own. One sentence praises the food and complains about the wait. The stars routinely disagree with the words. Any owner with a few hundred reviews has more than they will ever sit down and read properly, and the interesting signal is buried in exactly the reviews nobody reads twice — the three-star ones.

This has been technically solvable for three years and economically stupid for most of them. Running a frontier model over every review in a population, rather than a sample, costs more than the answer is worth to a restaurant with eleven tables.

So when TypeSafe released a model that returns typed decisions with calibrated probabilities and no text at all, the question wasn’t is it smart. It was whether the price is low enough to change what you’re allowed to build.

And underneath that, a second problem I keep hitting: there’s no cheap way to answer that question on your own data. Vendor benchmarks are run on vendor workloads against vendor baselines. What I actually want is a bench where I can point the same pipeline at any model and read the numbers side by side.

So this spin is two things. The bench, and the first experiment on it.

---

02 The data

All live, all public, all reproducible with your own keys.

SOURCE	SCOPE	ROWS
Apify Google Maps scraper (compass/crawler-google-places)	“restaurant”, Vancouver BC, 50 places	varies by run
Supabase Postgres, restaurant_reviews	deduped, JSONB, timestamped	2 in this run
Jev jev-latest	7 typed questions per review	2,027 input tokens

Caveat stated plainly, because it’s the biggest one on this page: the published run covers two reviews. That is a smoke test. Every number below describes one run of a working pipeline, not a measurement of model quality, and I’d be annoyed at anyone who cited it as the latter.

---

03 What I built

The bench. SpinUp is a playground for running focused model experiments end to end: scraper in, Postgres in the middle, a classification hop, a dashboard out. The classification hop is the swappable part. Jev is one model in it, not the point of it — the same pipeline takes OpenAI, Gemini, Groq or whatever ships next month, and the results land side by side in one view. Latency and cost print next to every individual result rather than in a summary box, because the question is never “does this model work,” it’s “what does this cost me on my data against the thing I’m already paying for.”

The experiment. jev-reviews asks Jev seven questions about each review in a single call:

• one Choice — sentiment: positive, neutral, negative
• one Score — overall, 1 to 5
• five Scores — food, service, cleanliness, atmosphere, value

One judgment per question, and my code does the combining. Jev evaluates every question in parallel and in isolation against the same state, so seven questions cost barely more time than one.

That design constraint turns out to be the transferable lesson. Across TypeSafe’s own four workflows, every model tested was more accurate, cheaper and faster when a policy was decomposed into narrow typed questions with code deciding, than when the same policy was handed over as one prompt. Haiku went from 18.1% to 53.6%. That is not a fact about Jev. It’s a fact about formulation, and it applies to whatever you’re using.

---

04 Result

REVIEW	STARS	SENTIMENT	OVERALL	FOOD	SVC	ATMO	CONF	LATENCY	COST
A	★3	Positive	3.8	3/5	4/5	3/5	67%	365 ms	$0.000043
B	★5	Positive	4.8	5/5	5/5	1/5	100%	95 ms	$0.000042

Two things worth reading closely.

The 3.8 is not a rounding artifact. A Score returns a probability-weighted value that can land between levels, so you’re seeing the shape of a distribution rather than a bucket. That’s more information than a classifier normally gives you.

Confidence dropped exactly where the signal was contradictory. Review A carries three stars, but read the text and it’s a pleasant visit with a couple of reservations. The words and the rating disagree — and that’s the review where confidence fell to 67%. Review B, where text and stars agree completely, came back at the ceiling.

That is precisely the behaviour the whole routing pattern depends on: act on high confidence, escalate on low. On a sample of two it is encouraging and nothing more. The chart that would settle it — confidence buckets against observed accuracy, plotted against the diagonal — needs a few thousand reviews and is the next thing this bench is for.

---

05 Where it broke

1. It scored the atmosphere of a room nobody described. Review B never mentions the space; the only “welcoming” in it refers to a staff member. Atmosphere came back 1/5, confidently. Cleanliness and value correctly came back blank, so the pipeline can produce an empty pillar — atmosphere just didn’t. The cause is structural: a Score question always lands somewhere on the scale, and has no way to say the text doesn’t discuss this unless I build it one. Fix is a Noul gate in front of each pillar and a confidence floor below which the pillar stays empty. My bug, not the model’s — but it shipped precisely because the output was perfectly well-formed. Type safety guarantees the shape, not the sense. Those get collapsed into one claim constantly, including by the vendor, and they are not the same property.

2. My “100%” was the UI lying. Rounding hid the distribution, and nothing about a restaurant review is certain. Two decimals now, with the full distribution behind Inspect.

3. I got the cost comparison wrong in the write-up, and I’m correcting it here. I said the scraper costs about a hundred times more than the intelligence. That compared $4.00 per thousand places against $0.043 per thousand reviews — different units. A place yields many reviews, so the real ratio depends entirely on reviews-per-place and is nearer 5× than 100× at twenty reviews a place. The directional point survives: acquiring the text now costs more than understanding it. The multiple I published did not.

4. Atmosphere looks stingy even when it is mentioned. Reviewer A explicitly says they enjoyed it and got 3/5. Two reviews can’t separate criteria wording from model behaviour.

5. The app calls Jev an LLM. The stage is still labelled “LLM Classification Engine” in the screenshots, which is wrong in a way that undercuts the entire point. Fixed in the repo, preserved here because pretending otherwise would be the opposite of what this page is for.

---

06 Run it yourself

$ git clone https://github.com/halfspin-qc/SpinUp_Playground
$ cd SpinUp_Playground && cp .env.example .env
$ npm install && npm run dev

You’ll need three keys in .env: Apify, Supabase, TypeSafe. Jev is still waitlisted as of this writing, so that third one may take a few days. Point the scraper at your own city, hit Sync, then Run Classifier.

Swap the classification hop and you’ve got a bench for whatever you’re evaluating instead — that’s the part I’d actually like people to take. MIT.

---

07 Discuss

Every spin has a thread on GitHub Discussions. If the criteria wording is doing more work than the model, if my confidence read is wishful, or if you’ve run this at real volume and the calibration doesn’t hold — say so there. Corrections get credited in the spin itself.

Open GitHub Discussion →