Your pipeline succeeded and lost 38% of the rows

The run finished in four minutes and fifty-one seconds. Exit code zero. The scheduler logged success, no alert fired, and the on-call engineer slept through the night.

It had loaded 744,118 of an expected 1,200,000 rows.

This is the failure mode that costs the most and gets caught the latest, because every tool in the stack is looking for the wrong thing. Monitors watch for errors. This run had none.

Where the rows actually went

The pipeline parses point-of-sale transaction files and maps each tender code — Cash, CreditCard, Voucher — to an internal identifier before loading. The mapping is a lookup table. The code that uses it looks roughly like this:

if (_mapping.TryGetValue(tx.PaymentType, out var tenderId))
{
    return MappedTender.Ok(tenderId, tx);
}

// Unknown tender code: reject the row and keep going.
_log.Warn($"unmapped PaymentType '{tx.PaymentType}' — routing to RejectedTransactions");
_rejects.Add(tx);
return MappedTender.Rejected(tx);

That else branch is the whole story. An unrecognised value doesn’t fail the run — it writes a warning and moves the row to a side table. Do that 4,821 times and the run still reports success, because from the pipeline’s point of view nothing went wrong. It handled every record it was given. It just didn’t load them all.

Two days earlier, the POS team had shipped a release introducing a new tender type called VoucherX. Nobody told the data team, because from their side it wasn’t a data change.

Why nothing caught it

Walk through the tools that should have.

The scheduler saw exit code zero. Row-count monitoring, if it existed, would have compared against a static threshold that a 38% drop on a variable-volume feed may well sit inside. Data quality tests ran against the target table and passed — every row in dbo.Transactions was valid. The invalid ones were never there to test. The reject table had no monitoring at all, because it appears in no pipeline definition, no configuration file, and no runbook. It was created years ago by a developer who has left.

Each tool did exactly what it was configured to do. The failure fell through the gaps between them.

The tell is concentration

When you do find a shortfall, the fastest signal is not the total. It’s the distribution.

SELECT PaymentType, COUNT(*) AS Rejected
FROM   dbo.RejectedTransactions
WHERE  RunId = 18291
GROUP  BY PaymentType
ORDER  BY Rejected DESC;

PaymentType	Rejected
VoucherX	4,821
GiftCardLegacy	118
CashOnDelivery	51
(null)	24

Random corruption spreads across values. A mapping gap concentrates on one. When 96% of rejections carry a single discriminator, you are not looking at data quality — you are looking at a value the system has never seen before.

The next question answers itself: when did it first appear? If the answer is “shortly after a deployment,” you have both the cause and the change that introduced it.

What to check on your own pipelines

Four questions, and you can answer them this afternoon:

1. Does any step in your pipeline swallow rows and continue? Search for catch, continue, and anything writing to a table with reject, error, quarantine, or invalid in the name.
2. Is that table monitored? Almost always no. Undocumented targets are undocumented precisely because nobody decided to create them; they accreted.
3. Does your run status distinguish “completed” from “completed correctly”? If a run can report success while diverting rows, your success signal means less than you think.
4. Do you reconcile counts across stages, or only at the end? Rows in versus rows out, per step. The step where they stop matching is the answer, and it’s usually one query away.

The general shape

Absence is evidence, and almost nothing alerts on it.

A step declared in config that hasn’t run in thirty days. A table written on every execution that appears in no definition. A quality gate that exists in the repository and has never once fired. A run that completed in three seconds when it normally takes four minutes.

None of these throw. All of them are findable — but only if something is comparing what your pipeline is supposed to do against what it actually did, rather than waiting for an exception that is never coming.