Postgres was hiding it, not Django

This is a post about a bug we misdiagnosed. We are publishing it because the misdiagnosis is the interesting part, and because the thing we found while correcting it was live in production the whole time.

The setup

In our ticketing system every tenant-owned table carries a tenant identifier, and isolation is enforced twice over. There is application-level scoping that returns nothing when no tenant context is set, and underneath it PostgreSQL row-level security, enforced by a database role that is not permitted to bypass it. Two independent layers, both failing closed.

The symptom

Deleting a tenant from outside a tenant context failed instead of cascading. The child rows — users, tickets, teams — were plainly in the table, and yet the delete tripped a foreign key constraint from those very rows.

The diagnosis we wrote down

Django’s deletion collector gathers cascade targets through each related model’s base manager. We concluded that the base manager was our tenant-scoped one, that it saw no rows without a context, and that the fix was to declare an unscoped base manager on every model — a model-by-model audit across the entire codebase.

That explanation sat in our README for two months. It is wrong.

What is actually true

Django auto-creates a plain, unfiltered base manager when none is declared, rather than falling back to the first declared manager. So the base manager was never our scoped one. The ORM layer was always perfectly willing to see the child rows, and the sweeping audit that fix implied was never needed.

Postgres was hiding them

With no tenant context set, the row-level security policy matches nothing. The collector queries for child rows and gets none back — not because Django filtered them, but because the database did. Finding nothing to cascade, it issues the delete against the tenant row alone, and the foreign keys from the rows it could not see reject it.

Both layers behaved exactly as designed. The bug was that we had built a cascade which requires seeing rows, and run it in a context deliberately constructed to see none.

The fix is small. Tenant deletion now enters its own tenant context rather than trusting every future caller to remember — plus the same at queryset level, because bulk deletes and the admin’s “delete selected” action never call the model’s delete method at all, and would have silently kept the bug.

Chasing it surfaced a live one underneath

Reproducing the failure as the restricted role gave the wrong error: a data error about invalid UUID syntax, not the integrity error we had documented. That is the signature of an older policy.

A custom Postgres setting that has been set once and then reset does not come back as null. It comes back as an empty string. Our policies had been fixed to handle that. So we checked what was actually in the database.

Twelve of twenty-three tables still carried the old policy.

The fix had been made to the migration operation — the code that creates a policy. That only affects policies created from then onward. Nothing ever rewrote the ones already in the database.

On those twelve tables, a query with no tenant context, on a connection that had previously had one, crashed instead of failing closed. Not a data leak — and we want to be precise about that, because it would be easy to tell this story as a near-miss it was not. But an isolation control whose failure mode had quietly changed, on more than half the tables, for months, without anything reporting it.

Why the test suite could never have caught it

This is the part worth taking away, and it generalises far past Django.

The test database is built by running migrations against the current operation code. Its policies are therefore always the fixed ones. Every test passed, and would have kept passing forever, while production carried a different schema.

A test suite validates the code that generates your schema. It does not validate the schema you have.

Those are the same thing only on a database that has never been migrated by an older version of the code — which is to say, never in production. We now have a check that reads the policies out of a migrated database and asserts their shape, rather than trusting that the code which writes them is correct. It is the only kind of test that could have found this.

A smaller one, from the same area

An earlier version of the tenant middleware set the session variable and relied on Django’s per-request transaction wrapping to keep it alive. It does not: that setting wraps the resolved view, not middleware running ahead of it. The statement was executing in its own throwaway autocommitted transaction and vanishing before any view query ran, and every login returned a server error. The middleware now opens its own transaction around the downstream chain.

The general lesson

Defence in depth works. Neither layer here ever leaked data, and that is not luck — it is what the second layer is for.

But it also means a broken layer can hide behind a working one indefinitely, because nothing visible goes wrong. If you have two independent controls, you need a test that exercises each one alone, as the real restricted role, against the real schema. We verify row-level security over a second live database connection for exactly this reason: the test runner’s own connection is privileged, and under it every policy passes whether it exists or not.