Why Deslopping never scores engineers

Any product that asks engineers questions can grade their answers, and any graded answer can be averaged per person. One AVG() grouped by reviewer and there is a number for every engineer on the team, sorted, on what would quickly become the most-viewed screen in the product. Deslopping has no such screen, and it is built so that one cannot be added in a hurry: its dashboards read through a database role that is granted nothing outside the aggregates.

The data for a score exists

It would be dishonest to say there is nothing to score. When a reviewer answers a question, whether the answer covered it is stored under their name. The product needs it: an answer that misses brings another question on the same lines, and the reviewer can read their own record.

That record lives in a schema the dashboards’ role cannot read, and it is deleted on the organisation’s retention schedule: after 90 days by default, settable from 14, and up to 365 on Enterprise. No per-person score, ranking, streak or league table is computed from it, anywhere. So the claim is not that the data does not exist. It is that nobody who manages the reviewer can reach it.

Why a score on the person would break the product

The most useful thing Deslopping records is an uncomfortable one: a reviewer looked at a hunk the AI wrote and could not explain it. Recorded against the code, that is a finding about the AI. It says this module, or this tool, produces code people cannot follow, which is something a team can act on.

Recorded against the person, the same event becomes a mark against them, and people would stop producing it. They would steer clear of the risky pull requests, answer from the description instead of the code, or quietly ask the author. The data would get worse exactly where it was meant to be useful.

It would also measure the wrong thing. Whether an answer covers its question depends on the code as much as on the reader, so the reviewers who take the riskiest pull requests would score worst, and the score would teach everyone else to leave those pull requests alone.

That is Goodhart’s law in its usual form. The authors of a study of a company that made merged pull requests per engineer its target, first published in July and revised in October, put it in one line when they advise pairing such a target with other signals: “a count that becomes a target stops measuring what it was meant to”.

There is a second reason, and engineers raise it first. The two things that reliably turn a team against a tool like this are individual scores and answers a manager can read. Neither can be reached from any view a manager can open.

What we measure instead

Every rating in Deslopping has code as its subject. After answering, the reviewer rates how the AI’s code reads: good, mixed or poor. A question nobody can answer is treated as evidence about the AI’s code rather than about the person.

Because every rating is about a specific hunk, the ratings add up in the two directions a team can act on: by module, to where the AI writes code people cannot follow, and by tool, with Claude Code, Cursor, Copilot and the rest on the same scale. A lead who sees one module keep reading poorly can change something about the module. A lead who saw one engineer keep missing questions could only change something about the engineer.

How it is enforced, not promised

A policy can be relaxed by whoever is under pressure that week. A database grant has to be changed in a migration somebody reviews. These are the mechanisms, in the order a query for one person’s data would meet them.

A separate database role

Every dashboard reads through a database role, deslopping_analytics_ro, that is granted nothing outside the aggregates, so a query for one person’s words fails at the database. This is the header of the first migration in our repository, as it stands today:

-- The privacy promise is enforced by which role can read which schema:
--
--   app        product data                    web + worker read and write
--   private    per-person data: attempts,      web reads its own rows under
--              answers, escalations, disputes  RLS; worker reads and writes
--   analytics  suppressed rollups              worker writes; analytics_ro
--                                              reads, and reads nothing else
--
-- deslopping_analytics_ro is deliberately granted nothing in app or private.
-- Analytics views are owned by the migrating role and run with the owner's
-- privileges, so they can read the underlying tables while the role calling
-- them cannot (PRIV-001, PRIV-002).

Nothing below five reviewers

Any figure built from fewer than five reviewers shows “Held back this month” on every view a lead can open, including when the lead knows the team has four people. It is tested rather than trusted: every change to Deslopping runs a test that seeds a module with four reviewers and checks that every read a manager view can make holds it back, along with the usual ways around a floor, such as overlapping periods or a total minus the parts that are shown.

Answers belong to the person who wrote them

An engineer can read everything they have written, export it, or delete it early. Answers are encrypted, and deleted after the retention period anyway. A lead sees module-level numbers, and the answers are not in their product at all. A note left with a rating can reach a lead only inside a pattern built from five or more reviewers, with no name on it.

What it costs

Some buyers want a per-engineer number. If you are accountable for a team’s output it is a reasonable thing to want, and it is the one thing we will not build. If a per-person report is what you need, this is the wrong product, and we would rather say so on the first call than in month three.

What a lead gets instead is the map: modules and tools, held to the floor, and a record that each risky change was read by someone who could explain it.

Questions to ask any tool that questions your engineers

• Which database role do the manager views read through, and what is it granted? Ask to see the grant, not the policy.
• What does a manager see for a module with four people in it?
• Can an engineer export and delete their own answers, and when are they deleted regardless?
• Is there a per-person score anywhere, including one that is “only used internally”?

What a comprehension check is explains the questions themselves, and the last chapter of How it works is about what nobody sees.

Lazar Stanković

Founder

Founder of Deslopping, which asks engineers about the AI-written code in their pull requests before it merges. Writes here about how it works, and about what AI is changing in the work.