The demo fix that stayed: a hundred transactions lost their detail

Executive Summary

A self-order feature taught our POS a new shape: one request, one order. When a fix for that feature had to ship fast while a client demo was waiting, the handler was copied into the counter flow — where one transaction carries many orders. The copy kept the last order and dropped the rest. Totals were summed before the insert, so every transaction looked correct at the top, and nothing threw an error.

At 21:30 a client opened a transaction and found the order detail missing. The code fix was ready an hour later. Deploying it would restart the service and wipe the request logs — the only complete copy of the orders that never reached the detail table. I held the deploy until those logs were extracted, then rebuilt about a hundred transactions that night. The last eight, from the minutes before an earlier restart, came from printed receipts. Nothing stayed lost.

The Problem & Context

Our POS takes orders two ways.

• Self-order: the customer scans a table QR, picks their items and submits. One submission is one order — a single record.
• Counter: the cashier builds a transaction that can carry several orders before it is paid.

The self-order work added columns to the order-detail table. That change reached the shared handler both flows used, and the fix that made self-order demo-ready went in fast. The shortcut it left was small: in self-order, a request always carries exactly one order, so the code never needed to collect more than one.

That was true of one caller. The other caller had been sending batches all along.

The Change

The copied handler kept the self-order shape:

Simplified, sanitized mock of the loop and the insert. Not production code.

// setupOrder parses one request item into a GORM-ready row.
// The counter flow: one transaction can carry many orders.
for _, item := range req.Orders {
    order = setupOrder(item) // overwrites; only the last order survives
}
db.Create(&order)

setupOrder itself was fine. The missing line was the append: each pass replaced the previous order, and GORM inserted one row where the counter flow expected one row per order.

The transaction header was built from the full request before this point. Every total — item count, subtotal, tax, service — was summed across all orders and written correctly. So the failure had a very quiet shape:

A wrong total gets caught by reconciliation. A wrong detail row under a correct total gets caught when a human opens the screen — five hours later, in this case.

sequenceDiagram
    participant C as Counter app
    participant A as Go API
    participant D as Database
    C->>A: One transaction, five orders
    loop each order
        A->>A: order = setupOrder(item)
    end
    A->>D: INSERT transaction — totals for five orders
    A->>D: INSERT order detail — last order only
    D-->>A: Success
    Note over A,D: No error, no warning. Header correct, detail incomplete.

Detection & Fix

Detection came from the client. Anything that read the transaction header stayed healthy; the only broken view was the one a person had to open. By the time the fix went live, about a hundred transactions carried incomplete detail.

The fix was the append the copied version had lost:

Simplified, sanitized mock. Not production code.

// What it needed: one row per order, collected before the insert.
orders := make([]Order, 0, len(req.Orders))
for _, item := range req.Orders {
    orders = append(orders, setupOrder(item))
}
db.Create(&orders)

The code fix took about an hour. The deploy waited nine more.

The Trade-off

At 22:30 the fix was ready, and deploying it was the obvious move: every hour the old handler ran, more counter transactions could lose their detail. But the service wrote its request logs locally, and a deploy restarted it. Those logs held the only complete copy of the orders that never reached the detail table — the transactions had the totals, not the items.

Deploying at 22:30 would end the bad writes and destroy the evidence needed to repair the ones already written. I chose the logs. Devops held the deploy; I pulled the request bodies from the surviving log files and finished the extraction before the fix went out at 07:40.

The bad path stayed live overnight. The alternative was a shorter incident with permanently missing data.

Recovery

At 23:00 I pulled the request JSON for the affected window and turned it into INSERT statements. I used Claude for the conversion: the log format was consistent, the transformation was mechanical, and doing it by hand for a hundred transactions invited transcription errors. Then I checked the output the same way I would check any migration — by the numbers.

I inserted ten rows by hand first, and compared each transaction’s rebuilt detail total against the total already stored in its header. All ten matched. With that, the rest went in as generated, directly against production. By 03:00 the table was whole again — except for eight transactions.

Those eight fell in the first 20–30 minutes after the deploy. The service had restarted for another push in that window, and the logs from before the restart were gone. The transactions still existed; their items no longer had a digital source. The clients had printed receipts from the day, so we took the eight orders from paper, re-entered the detail, and patched it onto the existing transactions. No transaction was recreated and no total changed.

With the data safe, the fix deployed at 07:40.

The Gate That Failed

QC tested the part that was fixed: the self-order flow that had been demoed. The counter flow, which shared the same handler, was never opened. There was no peer review before merge at the time — QC was the only gate, and its scope was the change, not the blast radius.

That scope is the whole lesson. A shared handler has as many contracts as it has callers, and the two callers here had opposite shapes: one order per request, and many. The review tested the new shape. The old shape was the one that broke.

The system had no test that would have caught it either. Nothing asserted that a transaction with five orders must end with five detail rows. The bug was invisible to every check we had because every check asked whether the request succeeded, not whether the result was complete.

Results & Impact

All affected transactions were restored, patched in place, with no permanent loss. The self-order feature stayed live, and the counter flow has run on the corrected handler since.

The lasting changes were peer review before merge and a QC pass that covers the existing flows a shared change touches — not only the screen that changed.

The recovery also left a useful fact behind: the request log held enough to rebuild the data, which is more than most incident stories can say. It just was not kept beyond the process that wrote it.

What I’d do differently

• One test on the contract. A transaction with N orders must produce N detail rows. That single assertion would have failed on the first counter sale and caught every later refactor of the same handler.
• Review the caller, not the diff. A copied function looks correct in the shape it came from. The review has to ask what the new caller sends, not whether the old path still works.
• Treat a shared handler as public API. Its callers are a list, and every change is tested against the whole list. QC’s scope should be that list, not the changed screen.
• Verify all rows, not a sample. Ten matching rows gave me confidence; a count query per transaction would have given me proof. I would run the full check before applying the rest.
• Ship logs off the box. Holding the deploy was the only reason the data could be rebuilt — too much weight on one decision. Request logs with retention outside the service lifetime turn a near-miss into a routine repair.
• Deploy with room for a smoke pass. The change went out at 16:30 and a second push followed within half an hour. Both flows deserved one manual transaction before either was called done.