When One Silent Error Breaks Three Things

Two students paid two hundred and fifty dollars each for an advanced course. Both received a confirmation email. Neither appeared on the instructor's calendar, both were still recorded as unpaid, and both were queued to receive a discount code chasing them to complete a booking they had already paid for in full.

Nothing in the system reported a problem. Every automation run was marked successful.

What actually happened

One step in the confirmation flow writes the payment back onto the booking record. It marks the booking paid, stamps the payment id, and sets the flag that stops the reminder emails.

That write was being rejected. The records in question carried a few extra fields that our database did not formally recognise, and the database validates the whole record on every update, so any change to those records failed. Not some changes. Every change.

The step that made it invisible was the last one. The tool making that request does not treat a rejection as a failure unless you tell it to. So the run continued, sent the confirmation email, created the calendar event, and finished with a green tick.

The automation was not broken. It was succeeding at everything except the one step that mattered.

One fault, three symptoms

What made this hard to see is that it looked like three unrelated problems, and each had a plausible innocent explanation.

• Students missing from the calendar. The calendar only displays confirmed bookings, and the write that confirms them never landed. Easy to blame on the instructor's view or a sync delay.
• Paid students still counted as unpaid. They sat in the same bucket as genuine abandoned checkouts, quietly inflating that number and making the reporting look worse than reality.
• Discount emails to people who had paid. The flag that suppresses those emails is set by the same failed write, so the reminder ladder never learned the sale had happened.

Chased separately, each of these is a small mystery you might shrug at. Together they are one line of code. That is the argument for looking at the finished result rather than at the pieces: the pieces all looked fine.

The check that would have caught it

Not better error handling. The obvious fix is to make that step fail loudly, and we did that, but it only catches the fault you already know about.

The check that generalises is a reconciliation. Once a night, list every payment the processor actually took, and for each one confirm the booking system agrees a sale happened. Anything the processor collected that your system does not know about is the alert.

That check does not care why the mismatch exists. A rejected write, a webhook that never arrived, a payment on a checkout link nobody remembers creating, a customer whose card name did not match their booking. All of it surfaces the same way, because the question is not whether the software ran. The question is whether the money and the records agree.

Where to point it

Start with the processor, not your database. Your database can only tell you about bookings it knows about, and the whole failure mode here is bookings it does not. The payment processor is the only system that cannot be wrong about whether money moved.

Then decide what the job is allowed to fix on its own. Ours repairs a payment that matches exactly one open booking, and reports anything ambiguous instead of guessing. Attaching a payment to the wrong student is worse than leaving it unattached, and a repair job that guesses will eventually produce a mess nobody can unpick.

The reason to build this even if you think your setup is fine: we thought ours was fine. The confirmations were going out. The customers were happy. The only visible symptom was an instructor asking why his class list looked short.