Flaky Tests: Seven Bad Runs, Sorted by Class

You open the release gate's output and a stage is red. The reflex is one of two sentences: "that test is flaky, rerun it", or "the last commit broke something". Both can be wrong in the same way, because the run may never have produced a test result at all. I keep a log of every run of my Android release gate, and in it, flaky tests are only one of several reasons a stage goes red. This post sorts seven cases from that log by what they actually were, and says plainly which ones I never explained.

Updated 2026-10-26. Source for the definition: Google's Flaky Tests at Google and How We Mitigate Them, John Micco, 2016-05-27, fetched 2026-10-26.

What counts as a flaky test

Google's post defines it this way: "We define a 'flaky' test result as a test that exhibits both a passing and a failing result with the same code." It lists causes including "concurrency, relying on non-deterministic or undefined behaviors, flaky third party code, infrastructure problems". It also warns about the failure on the other side: "In some cases, developers dismiss a failing result as flaky only to later realize that it was a legitimate failure caused by the code."

That definition needs a result. A test has to produce both a pass and a fail on the same code before it is a flaky test. Several of my bad runs fail that bar, because the stage never produced a complete set of results. I treat those as a separate class and call them no verdict: not red, not green, and not a reason to touch product code. A stage that stops partway through is in that class too, even if the tests it did run passed.

The seven cases

Six of the seven come from the release gate of one Android app, a ten-stage run (static checks, JVM unit tests, screenshot tests, a Maestro suite, instrumented tests on a leased phone, and more). Six are from one release prep in September 2026, which ran the gate at least 20 times. The seventh is a later full unit-test run in the same app.

Four of the seven never produced a complete result, and only one has a mechanism I reproduced on purpose.

#	Stage	What the log showed	Class	Cause known?
1	JVM unit tests	1 failure in 9461 tests; a count of 2 where 1 was expected	Test harness	Yes
2	JVM unit tests	6 failures, one root exception thrown inside Robolectric's own setup (the library that runs Android unit tests on the JVM), no app frames	Unknown	No
3	Static checks	Run killed under host memory pressure; no results	No verdict	Log says host memory
4	Instrumented tests	"Expected 426 tests, received 424"; the phone had left the USB bus	No verdict	Yes, device
5	Instrumented tests	Gradle daemon disappeared 9 seconds in; zero tests started	No verdict	No
6	Screenshot tests	Test executor exited with SIGABRT, zero reported test failures	No verdict	No
7	JVM unit tests (later run)	1 log-capture assertion failed in one full run; passed alone and in two reruns	Unknown, order-dependent suspected	No

Case 1: the test that stopped testing what it said

The one failure in 9461 asserted a count of "marker" calls (a record the send engine writes at the start of a send) and got 2 instead of 1. The two assertions that protect a user from a double send, one engine call and one saved attempt, both passed. The same code had passed this stage about 40 minutes earlier, and eight reruns of the test class on an idle machine produced 0 failures in 8.

That is the flaky-test pattern. What I did next was the part that mattered: I read the test's own precondition. The test is meant to hold a first request inside the engine while a duplicate arrives. The hold was a 5-second wait, and the code threw away the boolean that says whether the wait timed out. If the machine starves long enough, the hold expires, the first request finishes, and the test silently becomes two sequential requests, where a second marker call is correct. I reproduced the exact failure signature by forcing the hold to expire. I did not observe the machine starving at that moment, so "the host stalled for over 5 seconds" is the explanation that fits, not one I watched.

The fix was to make the test record its own expiry and fail with a message naming a lost precondition before reading any count, to lengthen the hold to 30 seconds, and to print the observed event list in every count assertion. No product code changed.

Cases 3 to 6: runs that never produced a complete result

Case 3: killed inside the first stage under host memory pressure.
Case 4: the leased phone left the bus after test 425 of 426. Gradle reported "There were failing tests", but that was the count mismatch, not an assertion. I confirmed the device was absent from both adb devices and the Mac's USB listing, so it left the bus rather than just going offline to adb.
Case 5: the daemon was 4 hours 14 minutes old, on its 33rd build, and vanished before a test started.
Case 6: an executor died with exit value 134 and zero failing tests reported.

My rule for all four is the same: a stage that never produced its full set of results carries no verdict, and it gets relaunched, not investigated as a regression. If the same stage dies the same way again, that is a pattern, and I investigate. For cases 3 and 4 the log names a cause. In case 5 it records only that the daemon disappeared, and for case 6 I did not look further, so I claim no cause for either.

Cases I never explained

Cases 2 and 7 have no cause. In case 2 the root exception was thrown inside Robolectric's native-runtime setup, before any test code ran, and the other five failures were collateral from it. The log says what the stack proves, where the exception was thrown and that no app code is in it, and that the exact interleaving "was not reproduced, so no cause is written down". Case 7 failed once in a full unit-test run on a machine under load (load average about 14 on 12 cores), then passed in a clean rerun of the full suite (10288 passed, 0 failed) and in a run alongside the classes that touch the same global logger. My note suspects shared global logging state across tests, and my own status line for it is "not reproduced", with an instruction to capture state if it recurs.

Those two are the textbook flaky-test cases, and I handled them the way Google's post describes the cheap option: a rerun. Google also says what that costs, that marking tests flaky "encourages developers to ignore flakiness in their own tests". I have one recorded entry waiting for a recurrence, and that is the whole defence.

The check I run before calling anything flaky

Did the stage finish? Compare the count that ran against the count expected. A mismatch or zero is an incomplete result, not a flaky test.
Is the failure in app code or around it? Frames from the app, a test, a runner, a daemon or a device bridge are different problems.
Does a failing test depend on a hold or a wait? Read what its precondition is, and whether the test notices when it is lost.
Does it reproduce on an idle machine? Mine did not, 0 in 8. That is non-reproduction, not proof; what explained it was reading the precondition and forcing it to expire.
Is a second tool colliding with the first? Two Maestro runs at once produced a connection refusal on a local port in a different incident, and the cause there was the tool, which I covered in Running Multiple Claude Code Agents.

When an agent hands you a green run, the same questions apply to its claim; see Don't Trust an Agent That Says the Tests Passed. And a fix that makes a test pass can still break the feature next to it: A Correct Fix That Quietly Broke the Feature Next to It.

What I did not measure

I have no rate of flaky tests for this app, only the counts above, and I have no data on how many reruns I did that hid a real fault. These seven cases are the ones I could document; they are not a sample. I did not compare flaky-test detection tools.