A parcel agent that knows when to hand over
A customer agent on a delivery network's app and website that answers from the live parcel record, investigates "delivered but not received" the way operations does, and passes the right conversations to a person.
Context
Parcel recipients ask a delivery network the same things at scale: where is it, why is it late, what happened at the failed attempt, where is the pickup point, what is this customs charge, and the hardest one, “it says delivered and I don’t have it”. The network put an agent on its app and website that answers from the live parcel record, follows the same rules its people follow, and knows when to stop and hand over.
Constraints
- The agent speaks only about the parcel in front of it. The parcel comes from the request, never from the model’s choice, and fields the customer has no business seeing are stripped before anything reaches the prompt.
- Some things are never said: where a wrongly delivered parcel went, a driver’s surname, a GPS position, a promise of compensation.
- Disputes are decided the way operations decides them, by a documented flow that checks the evidence, not by the model’s impression.
- A person has to be one message away: complaints, suspected fraud, damage, and anyone who asks three times. Out of hours the agent says when to come back rather than offering what it can’t deliver.
- Every turn traced, so a reviewer can see what the agent knew when it answered.
What we built
A LangGraph agent served from Cloud Functions in the client’s region, conversation state in Cloud SQL, Gemini on Vertex AI as the model, every turn traced in LangSmith.
Each customer turn starts with a classifier: is this a dispute about a delivery or an attempt? If not, one call answers from the parcel’s status and event history and the knowledge base, 16,000 words of business and response rules plus 223 books, one for every parcel status and event code. It returns a structured answer with flags the front end acts on: hand over to a person, outside hours, a message for the driver, options to offer while the parcel is in flight. If it is a dispute, an investigation loop follows the operations flowchart: check the driver’s photo for compliance, measure the GPS distance from the delivery address, ask for what is missing, and finish with one of five outcomes, from deemed delivered to referred to a live agent.
The agent reads and never writes. The one action it takes is passing a message to the driver while the parcel is out for delivery, and even that goes through the front end. Changes to a delivery stay in the channel the customer already uses.
Models change faster than the rules do, so the agent has a harness for swapping them. Finished real conversations are the baseline. A candidate model replays each one from the first message, on a frozen parcel snapshot and clock, building its own history. Every turn is scored three ways: whether the five flags match the baseline, how much of the wording overlaps in both directions, and a judge model’s 0 to 100 on whether the customer gets the same outcome, same facts, same next step, same tone, with unusable replies scored zero. That is how candidate models, including small open-weight ones, are compared against production before any customer sees them.
Outcome
In production on the network’s app and website since 2025: 10,000 to 15,000 conversations a day, 73% of them resolved without a person, with replies usually under 20 seconds. Designed and led by Jakub. The evaluation harness is in daily use to vet the next model before it ships.
Stack
Python, LangGraph, Gemini on Vertex AI, Cloud Functions, Cloud SQL, LangSmith.
What an agent run looks like
A replay of one production request, step by step, as our tracing records it.
- 0 msuserCustomer: It says delivered at 14:12 but there's nothing at my door.
- +410 mstoolclassify_turn({ "parcel": "<from request>" })dispute: yes, type: delivered_not_received
- +180 mstoolget_parcel_events({ "parcel": "<from request>" })status: delivered 14:12, location: front door, driver photo: available, GPS: recorded
- +640 mstoolcheck_evidence({ "photo": true, "gps": true })photo: shows a parcel at a door, compliant. distance from delivery address: 11 m
- +1900 msassistantI can see the driver photographed your parcel at your front door at 14:12, 11 metres from your address. Could you check with anyone else at home and around the porch? If it is still missing, say so and I'll pass this to a colleague to open an investigation. 118 tokens