Watch
RubyConf 2015 - How to Stop Hating your Test Suite by Justin Searls
A language-agnostic tour of why suites become slow, inconsistent and falsely green, covering how to pick test levels deliberately and when mocking turns a test into a lie.
Contents Section 2, Verification
Engineering Robust Tests
Mistakes to catch in review
Everything mocked, so the tests never touch a real database and miss every query bug.
Tests that depend on the current time, the order tests run in or the network, and fail at random.
Only happy-path tests, with no empty, maximum, negative, non-ASCII or concurrent inputs.
Fixed sleeps in end-to-end tests that are either needlessly slow or too short to be reliable.
Choosing the right level of test, controlling time and randomness, and building suites that stay fast, deterministic and trustworthy.
Topics
- Unit, Integration and End-to-End Tests
- What each level catches, what it costs, and how to balance a suite across them.
- Test Doubles
- Fakes, stubs and mocks, and when replacing a dependency makes a test lie.
- Equivalence Classes and Boundary Values
- Choosing a few inputs that stand for many, with extra attention at the edges.
- Deterministic Tests
- Controlling clocks, random seeds, network access and ordering so a test gives the same answer on every run.
- Test Data and Fixtures
- Building realistic synthetic data, never real customer data, and keeping test setup readable.
- Flaky Tests
- Finding, quarantining and fixing nondeterministic tests before people learn to ignore failures.
You understand it when you can
- Pick the right test level (unit, integration or end-to-end) for five given behaviors and justify each choice.
- Make a flaky test deterministic by controlling time, randomness and ordering.
- Derive boundary-value test cases for a function from its specification alone.
Drill
An agent's suite for a subscription renewal job mocks the database, the clock and the payment client, and one test computes the expected renewal date from the real current date plus 30 days. Find the bugs this suite cannot catch, and explain why the date test passes or fails depending on the day it runs.
Start here
Watch
Testing Without Mocks - James Shore | Craft Conference 2024
Presents 'nullables', production code with a built-in off switch for I/O, as an alternative to mock-heavy suites that are slow, fail at random and break on refactoring.
"Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson
Shows how FoundationDB controlled the clock, random seeds, network and scheduling so that every failure replays exactly, the strongest form of the deterministic-test idea.
Read
Software Engineering at Google: Lessons Learned from Programming Over Time
Chapters 11 to 14 cover test sizes versus scopes, why Google prefers real implementations and fakes over mocks, and how it handles flakiness and larger tests at scale.
Effective Software Testing: A developer's guide
Teaches specification-based testing and boundary testing step by step, so you derive equivalence classes and edge cases from the requirement instead of from the code.
xUnit Test Patterns: Refactoring Test Code
The source of the standard vocabulary for test doubles (dummy, stub, spy, mock, fake), plus a catalog of test smells such as erratic and fragile tests with their causes.
Primary sources
Reference
Mocks Aren't Stubs (Martin Fowler)
Separates state verification from behavior verification and explains how mock-based tests couple to implementation, which is the root of the 'everything mocked' failure.
Reference
Eradicating Non-Determinism in Tests (Martin Fowler)
Lists the concrete causes of flaky tests, including shared state, asynchrony, remote services and time, with the fix for each: quarantine, clock injection, and polling instead of fixed sleeps.