Track 2 · Nebius Token Factory

Resource matching evaluation

Twenty hand-labelled scenarios, each isolating one decision the matcher has to get right — a dietary requirement against a shorter walk, a closed door against an open one, an appointment barrier against a walk-in. The label is the option a caseworker would send the person to.

Nebius Token Factoryopenai/gpt-oss-120brun 9/5/2026, 9:06:32 PM · scenarios scored as of 1:00 AM

Top-1 accuracy

95%

19/20 correct

Top-2 contains answer

100%

20/20

Avg inference

1.64s

p50 1590ms · p95 2368ms

Avg cost / request

$0.00027

23,326 tokens total

Built-in heuristic baseline on the same set: 17/20 (85%) — the model earns its place on the cases where rules alone are too blunt.

Where it fails (1/20)

The most instructive miss: Not urgent — best programme wins. The label says Weekly Grocery Program because with a week of flexibility, a full grocery pickup beats one bowl of soup. The model chose Tonight Only Soup Line, reasoning: “Open now (6‑7 PM), only 1.1 mi away, walk‑in, so you can get dinner tonight”

Why this model — same set, 3 models

Model	Top-1	p50	p95	$ / req
→openai/gpt-oss-120b	19/20	1.59s	2.37s	$0.00027
Qwen/Qwen3-30B-A3B-Instruct-2507	18/20	3.16s	5.12s	$0.00015
meta-llama/Llama-3.3-70B-Instruct	18/20	2.92s	17.72s	$0.00014

Accuracy is a tie — at this sample size the intervals overlap almost entirely, so none of these is measurably more accurate than another. Tail latency is not a tie: p95 spans 7× across the set. Someone who has not eaten is waiting on this call, so the model was chosen on the tail, not the average — and Llama-3.3-70B-Instruct was dropped for a 18s worst case despite matching on quality.

What this measurement does not tell you

20 scenarios is too few to rank two models. At 95% the 95% confidence interval runs 76%–99%. A run scoring 5 points higher has not been shown to be better. Treat this as a regression check that catches gross failures, not a leaderboard.
The labels are our judgment, not outcomes. Each answer is what we believe a caseworker would choose. Nobody was actually sent to these places and asked whether they got fed.
The candidate lists are synthetic. Each scenario is hand-built to isolate one decision. Real Linkup results are messier — missing fields, near-duplicate organizations, stale hours — and the ranking is only ever as good as the research underneath it.
Latency is single-request, from one machine. No batching, no concurrency, one network path. It says what a user waits, not what the endpoint sustains under load.

All 20 scenarios

✓
01Vegetarian, no car, needs food tonight
Does a hard dietary requirement outrank a slightly shorter walk?
Community Kitchen 95Regional Food Bank 78Corner Pantry 10
Model: Only 1.2 mi away, open until 8 PM, confirmed vegetarian meals, walk‑in, and solid confidence.
2703ms
✓
02Closest option already closed
Is a closed door correctly disqualifying at 6pm?
Evening Meal Program 95Midday Kitchen 8Dawn Pantry 5
Model: Open now (5‑8 PM) and within walking distance (1.1 mi), so you can get dinner tonight.
1581ms
✓
03Appointment-only vs. walk-in tonight
Does an access barrier outweigh a half-mile of distance?
Open Door Meals 92Referral Pantry 68
Model: Only 0.9 mi away, open now, walk‑in allowed, no referral needed
1014ms
✓
04Has a car, best option is further out
Does transport correctly relax the distance penalty?
Masjid Community Meal 92Nearby Pantry 18
Model: Your car can easily reach the Masjid Community Meal (6.5 mi) before it closes at 9 PM, it explicitly offers halal hot meals, and has no appointment barriers.
1590ms
✓
05Same pair, but no car
The mirror of s04 — unreachable beats well-matched.
Nearby Pantry 92Masjid Community Meal 15
Model: Only 0.7 mi away, open until 7 PM, halal meals, walk‑in, solid confidence
1361ms
✓
06Open, but closing in 20 minutes
Is a closing-soon window discounted against a comfortable one?
Late Kitchen 88Last Call Pantry 72
Model: Within walking distance and open until 9 PM, giving you plenty of time to arrive and get dinner tonight.
1542ms
✓
07Non-student sent to a campus pantry
Is an eligibility rule the person fails treated as blocking?
Neighborhood Meal 92Campus Pantry 22
Model: Only 1.4 mi away, open now (5 pm‑8 pm) and no ID needed, so you can walk there and get dinner tonight.
1997ms
✓
08Verified nearby vs. unverified nearer
Does low research confidence cost a resource the top slot?
Confirmed Kitchen 92Rumoured Fridge 45
Model: Open now until 8 PM, only 1 mile away, walk‑in allowed and high confidence data
1350ms
✓
09Unresolved conflict about access
Is an open contradiction penalised versus a clean record?
Steady Meals 92Disputed Pantry 78
Model: Within walking distance, open now, walk‑in allowed and high confidence info
1005ms
✓
10Vegan requirement, mixed boxes elsewhere
Is 'not guaranteed' treated as not meeting a strict requirement?
Plant Plate Kitchen 78Mixed Grocery Box 73
Model: Vegan meals confirmed and still open, but 1.6 mi walk is a bit beyond comfortable distance for walking only.
2368ms
✓
11Bus available, mid-distance best fit
Does transit correctly sit between walking and driving?
Transit Line Kitchen 84Corner Store Pantry 57
Model: Transit can reach the 3.2‑mile site, it’s open until 9 PM, and it explicitly offers vegetarian meals.
1669ms
✓
12Two near-identical options
Does a longer open window break an otherwise even tie?
South Meals 85North Meals 78
Model: South Meals is a 1‑mile walk, open until 8:30 PM giving you plenty of time to arrive tonight
1418ms
✗
13Not urgent — best programme wins
When timing is loose, does quality of fit take over?
Tonight Only Soup Line 95Weekly Grocery Program 10
Model: Open now (6‑7 PM), only 1.1 mi away, walk‑in, so you can get dinner tonight
Expected Weekly Grocery Program: With a week of flexibility, a full grocery pickup beats one bowl of soup.
1277ms
✓
14Coeliac, needs certainty
Is an explicit gluten-free provision preferred over proximity?
Allergy Aware Pantry 92Bread Line 31
Model: Within walking distance, open until 7:30 PM, confirmed gluten‑free options, walk‑in access, and decent confidence level.
1674ms
✓
15Every option is flawed
Can it pick the least-bad option instead of refusing?
Near Appointment Pantry 88Far Vegetarian Kitchen 45Near Closed Pantry 5
Model: Only 0.7 mi away (walkable) and still open until 8 PM, so you can get there tonight; the only downside is the appointment requirement.
1902ms
✓
16Bike widens the range
Does a bike unlock a 3-mile option a walker could not reach?
Crosstown Veg Kitchen 92Local Canned Goods 78
Model: 3 mi away, reachable by bike; open until 9 PM so you have time; vegetarian meals confirmed; walk‑in, no appointments.
1804ms
✓
17Unknown hours vs. known hours
Is a documented open window preferred over an unknown one?
Published Hours Kitchen 86Unlisted Hours Pantry 44
Model: Open until 8 PM, 1.2 mi walkable distance, walk‑in, high confidence data
1565ms
✓
18Free-text note changes the answer
Are the person's own words used, not just the structured fields?
Family Dinner Program 92Single Adult Meal 68
Model: Only 1.3 mi away, open until 8 PM, walk‑in and families welcome with no ID needed
1316ms
✓
19Kosher requirement
Is a named religious dietary requirement honoured?
Kosher Food Pantry 88General Soup Kitchen 15
Model: Within transit range (2.6 mi), open until 7:30 PM, confirmed kosher and walk‑in, so fits timing and diet.
1860ms
✓
20Late night, everything closed but one
Is a 24-hour resource recognised as the only live option?
Community Fridge 95Evening Meal 60Daytime Pantry 5
Model: Only 1.1 mi away, open 24 h, walk‑in with no eligibility barriers, so you can get food right now.
1722ms