Your Test Suite Passed Because It Asserted Nothing

For its first eight months, our new Laravel API had a test suite that was always green and a CI pipeline that never ran it. Merging to the development branch deployed straight to our alpha environment with the tests unexecuted — and when developers did run them locally, they ran against whatever shared development database happened to be in that developer's .env. This is the story of the day the suite ran honestly, and what it said.

The anatomy of a vacuous green

The API sits on top of a twenty-year-old database it doesn't own — 149 tables and 68 views accreted across four backend generations. You can't migrate:fresh a schema like that, so the tests had grown a coping mechanism: reach for real rows (SaleBarn::first()) and call markTestSkipped when nothing came back. By July there were 412 markTestSkipped calls across 56 of 124 test files. On an empty database, the suite "passed" having asserted almost nothing. A skipped test is a lie with good manners: it shows up green-adjacent in the summary and nobody reads the skip column.

Making the database reproducible

The fix started with the schema. We derived a committed, 256 KB schema-only artifact from a 96 MB production dump, with three transforms:

1. Strip every row — no customer data in the repo, and no 96 MB seed in CI.
2. Drop DEFINER clauses from views — they referenced a database user that doesn't exist on test servers, which fails view creation with a cryptic error.
3. Convert 59 legacy MyISAM tables to InnoDB. This is the transform that unlocked everything: MyISAM is non-transactional, so Laravel's DatabaseTransactions trait had been silently committing on those tables the whole time, and tests hand-deleted their own rows to compensate. Transactional test isolation is a storage-engine property, not a framework feature — the trait can only roll back what the engine can.

With that artifact, any CI runner can stand up a byte-faithful copy of a two-decade-old schema in seconds. The database stopped being an environmental hazard and became a build input.

What the first honest run said

The first full run with skips converted to real executions produced 194 failures from a single cause — and it's my favorite bug of the year. Each test file invented its own JWT signing secret, and our JWT library enforces a 256-bit minimum key length. So 'test-secret-for-my-tests' (24 bytes) threw, while 'test-secret-for-calendar-session-tests' (38 bytes) passed. Whether a test file passed depended on how long its own name was. The fix — one shared, sufficiently long test secret — collapsed all 194.

Another 27 failures revealed something more uncomfortable: four tables that the application's own models require had never been created by any migration. The repository could not rebuild the schema its own code depends on — production had the tables, so nobody had noticed. Now migrations exist and the schema artifact proves it.

Three tests remained that genuinely disagreed with the application about intended behavior. Those went into a KNOWN-FAILING.md under a shrink-only policy: the list may only get shorter, and no assertion gets weakened to make it so. Documented disagreement beats deleted assertions.

The result

The first honest suite: 1,341 passing, 0 failing, skips down from 83 to 54. Three weeks later, with the suite finally cheap to run, it had grown to 1,480 passing — and, at last, a CI gate: no merge deploys without the suite running against the committed schema.

Three lessons I'd carry anywhere:

• A test suite you don't run in CI is a rumor. The deploy pipeline is the only opinion that counts.
• Reproducibility is a schema problem before it's a test problem. Distill the legacy database into a committed artifact; everything downstream gets easier.
• When you turn honest testing on, budget for what it finds. 194 + 27 failures sounds like a disaster; it was actually eight months of deferred information arriving in one afternoon — including at least one bug class (filename-length-dependent tests) nobody would ever have found by staring at code.