Auto-curation: agents that fill out forms

Every day, thousands of jobs arrive in Trusted’s platform from vendor management systems. Each one is a raw snapshot: codes, abbreviations, fragments, attached files. Before any clinician sees it, someone has to figure out what unit it’s really for, whether the rate is plausible, what credentials are required, whether the posting should even exist. We call that curation, and for most of Trusted’s history, a human did it.

Last week, every incoming job from one of our largest vendor relationships went live without a human touching it. The auto-curation rate is roughly 96%, and the system has been live in production since the end of March with no regressions against the human-curated baseline. The cost per curated job is about thirteen cents.

We expect to reuse this design for other work in the platform.

What curation actually is

Curation involves at least four tasks.

Concern	Tool count	What the agent owns	One example
Decision	2	Whether this job proceeds to the marketplace, is rejected, or escalates to a human.	Reject obvious junk; escalate a posting with conflicting bill-rate fields.
Clinical Unit	4	Mapping the raw unit string from the VMS onto a canonical unit in our taxonomy.	VMS says ICU-MED-3W; the agent pins it to Medical-Surgical ICU.
Job Details	17	Normalizing shift type, hours per week, bill rate, number of positions, start date, end date, location, and the rest of the structured record.	VMS provides fragments; the agent fills out the structured marketplace record.
Job Rules	24	Resolving certifications, years of experience, EMR proficiency, charge-nurse status, floating policies, holiday coverage---every constraint on who can accept the shift.	Required BLS plus two years of recent unit experience plus a specific EMR.

Each task requires looking things up, reading other parts of the job, making a judgment, and recording the result in a form. Human curators do this well, but each new vendor adds more jobs to their queue.

Why the first attempt didn’t work

Our first version, built in Q4 of last year, was the obvious move: send the LLM the raw job, hand it a giant JSON schema covering every curation output, ask for the fully curated record in one pass.

It worked often enough to be promising, but the failures were hard to diagnose.

The schema was large enough that the API would sometimes reject the request outright. The model returned valid JSON that didn’t match reality, with no clear path to interrogate why it had landed there---all the reasoning was inside one opaque call. Extending the system meant editing one shared schema, so every change risked breaking every other field. Debugging meant reading raw JSON outputs and guessing at the model’s thought process.

We rebuilt it several times. The largest agent, which handles job rules, went through four named iterations before I was comfortable with production volume. We adjusted the boundary between rules and job details, the size of individual tool actions, and when to ask a human rather than guess.

Separating those responsibilities made failures easier to locate. With one large agent, a wrong unit, rate, or rule sent us back through the same prompt. With separate agents, we could usually identify the relevant prompt and tools, then test a smaller change.

v2: four agents, one form

The new architecture replaces the single one-shot call with four specialized agents, chained.

Each agent has one concern. Each has its own system prompt, its own tool set, and its own success criterion. Each runs the same loop:

think → call tools → observe → submit

The four agents:

A curation decision agent (two tools) that decides whether the job proceeds, gets rejected, or is escalated to a human.

A clinical unit mapping agent (four tools) that pins the job to a canonical unit in our taxonomy.

A job details agent (seventeen tools) that normalizes shift type, rate, hours, count, and the rest of the structured record.

A job rules agent (twenty-four tools) that resolves every requirement and constraint on the shift.

The tool counts aren’t arbitrary. Each tool corresponds to a specific atomic action---looking up a certification, searching the unit catalog, setting a single form field, validating a date range, submitting the final state. The smallest agent has the simplest job. The largest agent (job rules) handles enough complexity that we’re already considering splitting it into more specialists.

Underneath, all four use the same reasoning model, configured at medium effort. Same model, four different sets of context, four different goals.

That separation helped immediately with debugging. In v1, a unit-mapping bug meant searching a 20-page prompt. In v2, that agent’s prompt fits on a screen and has an engineer responsible for it.

The Form pattern

The agents don’t talk to the database. They don’t issue queries. They don’t even produce the final job record directly. What they do is fill out a form---the same form a human curator fills out in our internal Manage app. We call this the Form pattern.

The Form is a virtual representation of the curator UI, exposed to the agent as tools.

form.set_shift_type(value, reason="job description says '3x12 nights'")
form.set_bill_rate(value, reason="VMS rate field is 78.50/hr")
form.add_certification(code="BLS", reason="required in original posting")
form.submit()

The Form is seeded from the existing job snapshot, so the agent starts where a human curator would start, with whatever fields are already populated. From there, the agent can search (looking up canonical units, certifications, EMR codes), mutate (writing to specific form fields), and at the end, call one terminal tool: submit.

Submit is where the design earns its keep. The Form submits to the same validation layer the human curator UI uses. If the model wrote an invalid combination of fields, the validation error comes back to the agent on the next turn---the same error message a human curator would see in the UI. Wrong nurse-to-patient ratio for that unit type? Same error. The agent gets to correct itself, exactly the way a human curator clicks Save, sees a red error, and tries again.

Five things follow from it.

The agent never persists a half-correct state. Mutations accumulate in the Form until submit. A failed validation rolls back the entire submission attempt. We don’t end up with jobs in production that are almost-right-but-wrong-in-one-field.

The validation contract is shared with the human UI. There is no parallel set of rules for “what AI is allowed to do.” By construction, the agent can only do what the curator UI lets a human do. Constraints live in one place.

Every mutation comes paired with a model-supplied reason. When an auditor later asks why the system pinned a job to one unit instead of another, the audit log has the agent’s answer in its own words, attached to the specific tool call that set the field. An actual entry, lightly cleaned:

form.set_clinical_unit(
  "Medical-Surgical ICU",
  reason="VMS unit string 'ICU-MED-3W' matched canonical unit via taxonomy lookup; confirmed by shift description mentioning med-surg patients and 1:3 nurse-to-patient ratio."
)

That explanation gives an auditor something specific to inspect when a result is wrong.

Extensibility is local. Adding a new field is the same work as adding it to the human UI plus exposing one tool. Adding a new agent for a new concern doesn’t touch the other agents.

The pattern is portable. Anywhere a human currently fills out a structured form in our app is a place where this same architecture can run. Job curation is the first place we’ve put it; it will not be the last.

The loop that keeps it honest

Once the agents were in production, we needed a way to learn from the jobs they got wrong.

Every auto-curated job is logged turn by turn: model, tokens, tool calls, results, reasons. Auditors review the auto-curated job alongside the original VMS snapshot and either approve it (thumbs up, sometimes with a small note) or correct it (thumbs down, with a note explaining what was wrong). Negative feedback represents a real production error. Positive feedback with a note usually represents minor drift---something a human noticed that wasn’t worth blocking the job, but was worth recording.

Auditors’ notes are especially useful because they explain what should have happened on a real job. We can compare each note with the original input and the agent’s recorded actions.

The feedback flows into evals. Every eval case is a real production job, frozen at the input/output boundary: input is the raw VMS snapshot, expected output is the curated state a human eventually approved. The agents run against the frozen input. The output gets compared against the expected via three kinds of assertions: exact match where it should be exact, structural match where shape matters more than literal values, and LLM-judged equivalence where the output is correct as long as it’s semantically right.

Evals live in the repo. They run locally on every prompt change. They’re part of the dev loop, not a separate QA stage. Four suites, one per agent. When a feedback note flags a regression, the regression becomes a new eval case before the prompt change goes back out.

One early case showed how the loop is supposed to feel. An auditor flagged a job that the unit-mapping agent had pinned to Telemetry when the right answer was Step-Down---a borderline case where two adjacent unit types share most of their criteria. The note was three sentences, but it was specific: the agent had over-weighted the presence of a cardiac-monitoring requirement and under-weighted the nurse-to-patient ratio in the original posting. We froze that job as a new eval case, sharpened the unit-mapping prompt to weigh ratio language explicitly, and re-ran the four suites. The new case passed; nothing else regressed; the change went out the same day. Multiply that pattern by a few hundred over six months and you get a system measurably better than it was at launch, by a path you can read in the eval history.

Where we are

Our first vendor is live---one of the largest in our footprint. Every job from that relationship is being auto-curated end-to-end, with no manual intervention. The success rate over the last seven days is roughly 96%, measured against auditor thumbs-up versus thumbs-down on the resulting job.

There are a few qualifications to that number.

Not every negative is critical. Some of the failures are minor drift---fields a human auditor would have caught and corrected, but that don’t affect whether the right clinician finds the right shift. The real range across the last few weeks has been 85 to 96%, depending on which slice you look at and how generously you score borderline cases.

There are no regressions against v1. Since the end of March, the system has improved monotonically against the same human-curated baseline that ran the queue before it.

Each curated job costs roughly thirteen cents in model spend and platform overhead.

This is the first multi-turn reasoning agent we’ve deployed for marketplace operations in a health system relationship.

What the Form pattern tells us about agents in production

Decompose by concern, not by step. v1 tried to produce the entire curation result in one call. v2 has one agent per concern, and each agent sees only the prompt and tools relevant to its own concern. The decomposition reduces prompt surface area, makes failures localizable, and lets different engineers own different agents without conflicts.

Don’t let the agent touch the system of record directly. Give it a Form. The Form mediates between the agent and the database. Validation lives on the Form. Auditing lives on the Form. The agent acts; the Form enforces.

Share validation with the human UI. The strongest constraint on agent behavior is the constraint you already wrote for humans. Don’t build a second one. An agent that can do everything a curator can do---and only what a curator can do---is the agent you want in production.

Make every mutation explain itself. The reason field on every tool call has paid for itself ten times over. It’s the audit trail, the training signal for evals, and the diagnostic surface when something goes wrong.

Keep improving the evaluations. Frozen job examples and auditor notes let us test whether a prompt change fixes the reported problem without bringing back an old one.

What’s next

One vendor is fully on autopilot. A second is close behind, already producing strong eval results from a smaller starting point. A third is queued. After that, the rest of our vendor footprint.

The same Form-pattern infrastructure runs each one. The work specific to each new vendor is mostly in the prompts, the unit-mapping data, and the rule taxonomies. The architecture stays.