Good Tests Are Hard to Vary

A criterion borrowed from David Deutsch, applied to two versions of the same Spock specification: which details is the test entitled to be strict about?

A test can pass because the rule holds, or because a name, a call count or a message string happened not to change. The build is green either way.

David Deutsch gives me a useful way to see the difference. The glossary to The Beginning of Infinity defines a good explanation as one that is “hard to vary while still accounting for what it purports to account for.”

Bad explanations make testable predictions all the time. In his 2009 TED talk Deutsch takes the Greek myth of Persephone as his example: in its simple form it cannot account for the seasons being opposite in the two hemispheres, and the reason that failure does not settle anything is that the myth can absorb it. Change who is angry, or why, or what was promised, and the story survives intact. The axial-tilt explanation has no such slack. Every part of it is doing work, so varying the tilt changes the predictions.

Deutsch is writing about explanations of physical reality. What I am borrowing is the shape of the criterion:

A good test is hard to vary in the places that matter.

Its setup, its input and its assertions are there because they help detect a specific kind of defect. Change a part that is doing work and the test should either fail or stop testing the same thing.

A test that explains too little

Take a Spring Boot service deciding whether a taxpayer qualifies for a relocation deduction.

Spock, briefly

Spock is a Groovy test framework. A feature method splits into labelled blocks; the labels are structural, not decoration.

given:: set-up
when:: the call under test
then:: assertions, where a bare expression is an assertion
expect:: the two combined, for a one-liner
where:: the data block: here a table, one iteration per row
Stub(): answers a call and forgets it
Mock(): answers a call and remembers it, so you can assert it happened
1 * mock.call(): an interaction assertion: this happened exactly once
0 * _: no other interaction with any mock happened

The rule is simple:

the taxpayer must have moved to another municipality
the taxpayer must meet the income floor of 200,000
the decision must carry a machine-readable reason code
the service must record a successful evaluation

Here is a test that passes and contains far too much accidental detail:

Groovy / SpockRelocationDeductionServiceSpec, first version36 lines

@SpringBootTest
class RelocationDeductionServiceSpec extends Specification {
@SpringBean
CitizenRegistryClient citizenRegistryClient = Stub()
@SpringBean
IncomeRepository incomeRepository = Mock()
@SpringBean
AuditLog auditLog = Mock()

@Autowired
RelocationDeductionService service

def "approves relocation deduction"() {
given:
citizenRegistryClient.find("p-123") >> new CitizenProfile(
cpr: "0000000000",
firstName: "Anna",
lastName: "Jensen",
previousMunicipality: "0851",
currentMunicipality: "0101"
)

when:
def decision = service.evaluate("p-123", 2025)

then:
decision.approved
decision.reason == Reason.ELIGIBLE
decision.message == "Citizen Anna Jensen qualifies for relocation deduction"
decision.evaluatedAt != null

1 * incomeRepository.totalEarnedIncome("p-123", 2025) >> 350_000.00G
1 * auditLog.record("p-123", "RELOCATION_DEDUCTION_EVALUATED", { it.success })
0 * _
}
}

The test checks real behaviour. It does not distinguish the rule from the scenery.

The CPR number, the names and the exact message text belong here only if the business contract depends on them. Otherwise they pin the test to incidental facts. The repository call count is the same problem: it verifies how the service currently gets its answer rather than the rule the service has to enforce. And 0 * _ sharpens the problem, because it fails the test for any unrelated change in collaboration.

So the test is strict about a great deal that nobody promised. The harder thing to see is that it is not strict about the rule either. Nothing here forces the service to check that the municipality changed, and nothing forces it to compare the income with the floor. Substitute Aarhus for Aalborg, or 400,000 for 350,000, and the test still passes.

The same test, harder to vary

Now remove everything that does not explain approval, and add a rejection case:

Groovy / SpockRelocationDeductionServiceSpec, second version50 lines

class RelocationDeductionServiceSpec extends Specification {
CitizenRegistryClient citizenRegistryClient = Stub()
IncomeRepository incomeRepository = Stub()
AuditLog auditLog = Mock()

def service = new RelocationDeductionService(
citizenRegistryClient,
incomeRepository,
auditLog
)

def "approves deduction when taxpayer has moved to another municipality and meets the income floor"() {
given:
citizenRegistryClient.find("p-123") >> new CitizenProfile(
previousMunicipality: "0851",
currentMunicipality: "0101"
)
incomeRepository.totalEarnedIncome("p-123", 2025) >> 350_000.00G

when:
def decision = service.evaluate("p-123", 2025)

then:
decision.approved
decision.reason == Reason.ELIGIBLE

1 * auditLog.record("p-123", "RELOCATION_DEDUCTION_EVALUATED", { it.success })
}

def "rejects deduction when the taxpayer did not move, or earns below the floor"() {
given:
citizenRegistryClient.find("p-123") >> new CitizenProfile(
previousMunicipality: previous,
currentMunicipality: current
)
incomeRepository.totalEarnedIncome("p-123", 2025) >> income

when:
def decision = service.evaluate("p-123", 2025)

then:
!decision.approved
decision.reason == expectedReason

where:
previous | current | income      || expectedReason
"0101"   | "0101"  | 350_000.00G || Reason.NOT_MOVED
"0851"   | "0101"  | 120_000.00G || Reason.BELOW_FLOOR
}
}

what changed

@SpringBootTest: Gone, with @SpringBean and @Autowired. Booting the whole context to exercise one decision, with every collaborator doubled, is the largest piece of scenery here and the likeliest to fail for a reason unrelated to the rule.
cpr, firstName, lastName: Gone. Nothing asserted them, and a fixture nobody reads is noise sitting in a public example.
decision.message: Gone. Presentation detail, unless someone has promised the wording is stable.
decision.evaluatedAt: Gone. It asserted only that a timestamp is not null.
incomeRepository: Mock() became Stub(), and the 1 * interaction became a plain return. This one costs something; see below.
0 * _: Gone.
the test name: Now names the conditions rather than the outcome.
a rejection case: New. A taxpayer who did not move, and one below the floor. Without it the service could stop checking either and stay green.

Dropping the 1 * costs something. It was the only line in the approval case forcing the service to look at income, and in exchange the test no longer breaks if the service starts caching that lookup. The rejection case takes most of the cost back: a taxpayer below the floor is refused only if the service looks.

The name alone carries weight it did not carry before: it states the conditions that make approval correct. The old name only announced that approval happens.

The municipality values are doing work, because moving between municipalities is part of the rule. The income value places the taxpayer above the floor. Approval and the reason code are both in the rule, so they stay.

The audit interaction is the one that needs an argument. I stipulated it at the top: the service must record a successful evaluation. That is a contract, and a test is entitled to assert one. The message text has no such promise behind it, which is why it went.

One example, and the rule behind it

This test may be correct:

Groovy / SpockOne example2 lines

expect:
lateFilingPenaltyPolicy.penalty(19) == 2_850.00G

It records one fact and leaves the shape of the rule invisible.

If the rule is 150 a day until the penalty reaches 3,000, a better test captures the relationship:

Groovy / SpockLateFilingPenaltyPolicySpec, a data table20 lines

class LateFilingPenaltyPolicySpec extends Specification {
def policy = new LateFilingPenaltyPolicy(
dailyPenalty: 150.00G,
cap: 3_000.00G
)

def "penalty increases by the daily rate until the cap"() {
expect:
policy.penalty(daysLate) == expectedPenalty

where:
daysLate || expectedPenalty
0        || 0.00G
1        || 150.00G
19       || 2_850.00G
20       || 3_000.00G
21       || 3_000.00G
60       || 3_000.00G
}
}

This is harder to vary because the examples are not arbitrary. They mark the structure of the rule: before the cap, at the cap, and past it. Adding rows does nothing by itself; a table of arbitrary examples is still arbitrary, and the value comes from the reason each row is there.

Three questions for every test

Three questions do most of the work.

What is doing work?

Which details are necessary for the behaviour under test? If a specific value appears in given, it should have a visible reason in then, or in the path to the outcome. Keep those. Remove, default, or soften the rest.

What defect would this test catch?

If I cannot answer that in a sentence, the test probably does not explain its own purpose clearly enough. Often the rule it protects is not the rule I thought it was.

What should be allowed to change?

A good test should fail when the rule breaks. It should keep quiet when harmless implementation detail changes.

“Hard to vary” is a discipline for choosing which assertions to make, and most of that work is deciding which details the test is entitled to be strict about.

It also cuts the other way. A test can be far too strict about the scenery and still be too loose about the rule, and the second failure is the quiet one.