pest/PLAN_PHASE_SIX.md

TIA deep audit — phase six

Your task

Phase five went looking rather than fixing what was reported, and found nine defects in the read/write path and the state dir. All nine are fixed and pinned by rows in the repo. It stopped there deliberately: roughly half of TIA was never opened. Phase six is that other half.

Two deliverables, both required:

1. A ranked findings list. Every finding needs a reproduction, not a reading of the code. A finding you cannot reproduce is a hypothesis — say so and rank it separately.
2. New scenario tests in tests/Features/Tia/*. Rows that pass go in the repo (they are the regression net). Rows that fail stay in your scratchpad until the fix lands — never commit a red test.

Work in passes, and report between passes rather than at the very end:

• Pass A — reproduce the leads in Part 3. Report which are real.
• Pass B — your own hunt: the invariants in Part 2, attacked with inputs nobody tried.
• Pass C — fixes, smallest first, each with the row that pins it.

Fix what is clearly a defect with an obvious correct answer. Stop and ask when the fix is a behaviour choice — those are Nuno's calls, and a reproduction with a crisp yes/no question is worth more than a guessed fix.

---

Part 1 — Where things stand

1.1 What TIA is

Test Impact Analysis. pest --tia records a dependency graph (test file → source files it touched) plus a per-branch baseline of results, then on later runs replays the tests whose dependencies did not change. State lives in a per-project dir under ~/.pest/tia/; graph.json is the whole thing.

Source of truth: src/Plugins/Tia.php (the plugin) and src/Plugins/Tia/Graph.php (the graph model + read/write API). Supporting: ChangedFiles.php (git), Fingerprint.php (environment/structure hashing), Storage.php / FileState.php (the state dir), ResultCollector.php (what a run observed), Recorder.php + CoverageCollector.php (how edges are recorded).

Read PLAN.md, then PLAN_PHASE_TWO.md → THREE → FOUR → FIVE for the history and the tier contract. Struck-through rows are fixed; do not re-report them. Section 1.2 below supersedes any phase-five row that contradicts it.

1.2 What phase five settled — do not re-report these

Nine defects, all fixed, each with a row that fails if it comes back:

#	Defect	Fix	Pinned by
1	pruneStaleResults() unset an entry; the next read layered the default branch's entry for the same test id back in, so a renamed/removed test stayed "previously unsuccessful" on a feature branch forever	Graph::baselineFor() layers per file once a branch has had a complete run (new complete flag on the baseline); a key minted by a narrowed run keeps the per-test-id merge	StateReclamation → a pruned result does not come back from the fallback, the fallback still reaches a branch that has never run a test file
2	A cached failure whose test file was deleted was never reclaimed, so --filtered degraded to a full replay on every later run	hasUnlocatedTestsToRerun() widens only for a path it cannot address; Graph::pruneResultsForMissingFiles() + pruneMissingTests() run on every complete write	StateReclamation → a cached failure whose test file was deleted stops widening later runs, a complete run reclaims the entry and the edge of a deleted test file
3	A detached HEAD is read-only for writes, but three paths still deleted: structural drift, --fresh (Storage::purge), and the corrupt-graph discard. The checkout that wiped the baseline could never rebuild it	Tia::deleteState() no-ops when detached; Storage::purge guarded	StateReclamation → the three a detached HEAD does not purge… rows
4	A status int outside 0–8 became TestStatus::unknown(), which ReplayType folded into Failure — a green test went red, exit 1	Graph::getResult() returns null for an unknown status (re-run, don't replay); shouldRerunStatus() treats unknown as re-run	HostileState → a cached status this build cannot interpret is re-run, not replayed
5	Notice/deprecation/warning (3/4/6) decode fine but ReplayType had no case, so they also folded into Failure	Explicit Pass cases — those statuses only reach replay when the configured failOn* / displayDetailsOn* policies say they do not matter	HostileState → a cached status with no replay of its own does not fail the run
6	Graph::decode() took baselines, edges and files verbatim; one malformed entry raised a TypeError inside a test	decodeBaselines() / decodeResults() / decodeEdges() / decodeFiles() validate every field. Numeric-looking keys are cast, not filtered — a branch named 12345 decodes as an int	HostileState → a graph whose shape is wrong everywhere is repaired rather than trusted
7	Baseline keys grew forever — one full copy of the suite per branch ever created	ChangedFiles::branchNames() (local + remote refs) + Graph::pruneMissingBranches(), on complete writes only, and only when the fallback branch is visible in the refs	BranchShapes → deleting many branches reclaims every one of their baselines and the four rows around it
8	A test that triggered a deprecation was recorded as status=0, because PHPUnit emits Passed for it and TIA had no issue subscribers. A fresh --fail-on-deprecation run exited 1; the replayed one exited 0	Six subscribers (Notice/PhpNotice/Deprecation/PhpDeprecation/Warning/PhpWarning) feed ResultCollector; most-important-status-wins; a plain Passed does not downgrade a triggered issue; @-suppressed issues are ignored	IssueStatuses → a triggered issue is recorded as itself, not as a pass, a cached deprecation still fails the run that asked to fail on one
9	A replay wrote back the status it looked like from outside, so a cached deprecation replaying as a pass was persisted as 0 — defect 8's fix eroded after one run	Tia::replayedAsRecorded() writes back the cached status and message for replayed tests, in the sequential path and in the worker flush	IssueStatuses → replaying a cached issue does not downgrade it to a pass
10	A run torn down mid-file (an exit() inside a test) still flushed what it had, and the parent read that as licence to prune the siblings it never reached. Sequential and parallel disagreed	ResultCollector::hasUnfinishedTest() demotes such a run to results-only, in terminate() (the shutdown path) and in addOutput()	StateReclamation → a run torn down mid-file does not prune the tests it never reached

One existing assertion was changed. DefaultBranchWriteTier > filtered mode falls back to a full replay when a cached failure cannot be located used tests/Unit/DeletedTest.php — a path that resolves but does not exist, which is precisely the shape behind defect 2. It now points at /build/agent/…, so it still pins the widening safety net for the case where widening can help. If you disagree with that reading, that is a finding, not a licence to change it back quietly.

1.3 Attacked in phase five and found solid — do not spend time here again

• Parity. Sequential vs --processes=1/2/8 across replay-with-edit, --filtered with a cached failure, a first run on a feature branch, --bail, --stop-on-failure, --compact: identical tally and identical graph delta every time.
• Hostile state dir. Empty / truncated / not-JSON / JSON scalar / JSON list / null / {} / NUL bytes; graph.json as a directory; a read-only state dir; dangling and negative edge ids; a result file pointing outside the project; schema: 2. All degrade to "run the tests", exit 0.
• Status int mapping. ResultCollector (asInt()), Graph::getResult() and TestStatus::from() agree exactly on 0–8. No off-by-one.
• FileState::write is tmp + rename, so concurrent runs cannot tear a file. Racing runs are last-writer-wins by design — treat as a decision, not a bug, unless you can show data loss worse than that.
• Branch names. Slashes, dots, unicode, digits-only, 180 characters, case-only differences, remote-only branches, worktree branches: all keyed and reclaimed correctly.
• sha/tree layering. No shape found that under-reports changed files. The fallback tree only drops a file whose current content hashes identically to what the fallback ran with, which is sound.

1.4 Known and deliberately unfixed

Item	Why it is still open
Reclamation is skipped when a complete run executes zero tests	Stale edges linger until a run that executes at least one test. Writing the graph from a run that produced nothing costs more than it buys. Revisit if you find a real project stuck there.
SIGINT never reaches the suite (PcovRestarter re-exec)	Real, but signal handling outside the TIA read/write path. Needs its own pass.
Orphaned state dirs when the origin URL changes	Storage::projectKey() keys on the origin identity so clones share a graph. Changing the remote silently moves TIA to a fresh dir and nothing reclaims the old one. Unbounded growth in ~/.pest/tia.
environmental fingerprint holds only php_minor	Nuno declined. The clearResults() drift path is therefore latent — it becomes live the moment that bucket grows. Do not re-flag the PHP_MAJOR_VERSION line itself.
PLAN.md §3 (raw coverage flags leave filtered mode on)	No longer reproduces — coverageReportActive() consults COVERAGE_REPORT_FLAGS now. Struck.

---

Part 2 — The invariants to attack

Same list as phase five; 1, 2, 4, 5, 7 and 8 are now well covered, so weight your effort toward 3 and 6 and toward the recording half of the system, which nothing below the plugin has ever probed.

1. Parity. pest <args> and pest --parallel --processes=N <args> must leave the same graph and reach the same tally. Hard rule. Project::SEQUENTIAL_AND_PARALLEL encodes it; use it on every new write-path row.
2. The tiers hold. COMPLETE / RESULTS-ONLY / HARD-SUPPRESSED, exactly one each, no leaking.
3. Replay is faithful. ← weak spot. A replayed test reports the same status, message, time and assertion count as the recorded run. Phase five fixed statuses; edges are unproven.
4. Reads never write.
5. A branch never corrupts another branch's baseline.
6. Nothing is unbounded. ← weak spot. Baseline keys are reclaimed now; files, edges, worker partials, coverage caches and orphaned state dirs are not.
7. A hostile state dir cannot break a run.
8. Git shapes are all handled.

---

Part 3 — Leads to reproduce first (Pass A)

Ranked by expected value. Everything here is unexplored, not merely unfixed — phase five never opened these files.

M1 — the recording path with a real coverage driver · highest value

Everything phase five did was driver-independent by design (seed a graph, never record one). Nothing verified that a recorded graph is correct. This is the biggest blind spot in the audit.

• Do the recorded edges match what the test actually touched? Record with pcov, then hand-check edges against the source files each fixture test uses.
• PLAN.md §4: pest --tia --coverage narrows edges — Feature/ExampleTest recorded with 2 files instead of 16, dropping self-edges. Same observable shape as the parallel bug G12 that phase four closed, likely a different mechanism (the piggyback collector is scoped by phpunit.xml <include>, the pcov-restarted recorder is not). Confirm whether these are one fix or two.
• Recorder::activateLinkTracking() (piggyback) vs activate() (pcov restart) must produce the same edge set for the same suite. Compare them directly.
• keepExisting: $this->piggybackCoverage in replaceEdges() — what happens to a test whose edges genuinely shrank while piggybacking?

Rows for this must be ->skipOnPhpVersionsWithoutCoverage()-style guarded, or seeded, or CI goes red on php84. That constraint is why phase five skipped it; solve it deliberately rather than by accident. Adding a coverage-driver guard helper to the fixture is a legitimate deliverable.

M2 — BaselineSync (621 lines, never opened)

The remote-baseline fetch is the only path where a graph arrives from another machine, which is exactly where the hostile-state work matters most and where none of it has been exercised.

• A fetched baseline whose fingerprint matches but whose files/edges describe a different tree.
• A fetched baseline recorded on a branch this checkout does not have.
• fetchIfAvailable() under a broken network, a 404, a truncated download, a non-gzip body.
• KEY_FETCH_COOLDOWN — does it bound retries, and does a corrupt cooldown file break a run?
• Interaction with defect 7's branch GC: a fetched baseline carries branch keys this clone has never heard of. They will be pruned on the next complete write. Is that right, or must fetched keys be exempt? This is a real question, answer it.

M3 — selection paths nobody has probed

Graph::affected() is ~600 lines and phase five only exercised the plain PHP-edge path.

• Migrations → TableExtractor → testTables intersection. What happens with an unparseable migration, a migration that drops a table, a squashed schema dump?
• Blade — bladeAncestorsFor() walks @include/@extends/<x-*> transitively. Cycles? Depth? A component referenced only dynamically?
• Inertia — componentForInertiaPage(), jsFileToComponents, JsModuleGraph::buildStrict(). What if vite is missing, or the resolver returns garbage?
• usesSiblingHeuristicForUnknownPhp() — a hard-coded list of Laravel directories. A changed file in app/Providers/ widens to every test whose deps share that directory. Measure how much that over-selects on a real tree.
• Arch tests — testSourceDeclaresArchGroup() greps the source with three regexes. False positives (the string arch( in a comment) select the file on every PHP source change.

M4 — git shapes phase five left alone

• A repo with no commits yet (currentSha() returns null / git fails).
• Submodules — a changed file inside one; git status --porcelain reports the submodule path.
• A repo whose root is above the pest project — TiaRequiresRepositoryRoot panics deliberately; confirm it still panics and writes nothing.
• Shallow / single-branch CI checkouts. Defect 7's guard (fallbackBranch must be visible in the refs) was reasoned about, not measured. Build one and check nothing is over-pruned.
• A branch behind the recorded sha, so merge-base --is-ancestor fails and the graph is declared unreachable. Does it recover, or thrash?

M5 — a real concurrent race

Phase five established FileState::write is atomic and called it last-writer-wins by design. Nobody launched two runs. Launch them: a watcher plus a manual run, two --tia processes on one project, --parallel where the parent writes while a straggler worker flushes. Look for loss worse than last-writer-wins — a partially-merged baseline, a pruned entry from a run that never saw the file, worker partials from run A consumed by run B (KEY_WORKER_* are not namespaced per run).

That last one is the sharpest: purgeWorkerPartials() deletes all partials by prefix, so two concurrent parallel runs will eat each other's. Probe it.

M6 — the unbounded remainder

Defect 7 reclaimed baseline keys. These still grow:

• files and edges — a deleted source file's entry is never removed (pruneMissingTests() only covers test files). Every rename leaves an orphan id forever.
• Orphaned state dirs under ~/.pest/tia (see 1.4).
• KEY_COVERAGE_CACHE / KEY_COVERAGE_MARKER — who deletes them, and when?
• Worker partials when a worker dies before terminate().

Measure the growth on a realistic tree before proposing anything. Then ask — a cap or a GC is a behaviour choice.

---

Part 4 — The harness

4.1 The scenario suite

tests/Features/Tia/* scaffold a throwaway git project into a temp dir, run a real pest subprocess against it, and diff the graph it wrote. Everything lives in tests/Fixtures/Tia/:

Class	What it gives you
Project	make(branch, overlay:), withoutGit(), seed(branch, sentinel:, failing:), seedFor($root, …), pest(...$args), pestWithEnvironment($dir, $env, ...$args), pestIn($dir, ...), write($rel, $contents), path($rel), graph(), branchKeys(), graphDir(), graphExists(), snapshot(), delta(), mutateGraph(fn), addBaseline($branch), worktree($branch), destroy(), destroyAll(), SEQUENTIAL_AND_PARALLEL
GitRepo ($project->git())	switchTo($b, new:), rename($from, $to), detach(), commit($msg), config($k, $v), addOrigin(), removeOrigin(), setOriginHead($b), unsetOriginHead(), worktree(), sha(), branchNames(), run([...]) for anything else
GraphDelta ($project->delta())	writtenCount(), added(), removed(), branchKeys(), baselineUntouched($b), shaMoved(), treeMoved(), edgesMoved(), filesMoved(), fingerprintMoved(), structureMoved(), isResultsOnly(), isHardSuppressed(), summary()
PestResult (returned by pest())	replayed(), uncached(), affected(), tally(), output, exitCode, describe()

The fixture app is 3 test files / 6 tests (Project::TOTAL_TESTS); Project::EDGES and Project::TESTS describe the graph seed() writes. Project::testId($file, $description) builds a result key. Overlays in tests/Fixtures/Tia/overlays/<name>/ supply a different tests/Pest.php — that is how you configure pest()->tia()->… for a scenario.

If the fixture cannot express your case, extend the fixture. A new GitRepo verb, a new overlay, or a coverage-driver guard is a legitimate part of the deliverable. Do not water down a scenario to fit the current helpers. Phase five added GitRepo::commit() usage, numeric-key handling in Project::branchKeys() and GraphDelta, and used $project->write() to author test files inline — follow that pattern.

4.2 The sentinel discriminator — read before writing any assertion

seed() writes a graph and then rewrites every cached result to time=9.999, assertions=42 (only where assertions are non-zero — risky/skipped/incomplete statuses are derived from "performed no assertions", so falsifying those would rewrite the status on replay), then snapshots. Therefore:

• $delta->writtenCount() is the only reliable way to tell "replayed" from "executed and wrote back the same values". 0 means nothing was written.
• isResultsOnly() = no structure moved, no sha/tree movement, nothing added or removed.
• isHardSuppressed() = the graph is byte-identical.
• A replayed suite reports inflated assertion counts (6 tests × 42). That is the sentinel, not a bug.

The three write tiers, unchanged since phase two:

• COMPLETE — may change everything.
• RESULTS-ONLY — may change only baselines[<branch>].results for tests that ran; never removes an entry, never adds a result for a test file absent from edges, never touches sha/tree/edges/files/fingerprint.
• HARD-SUPPRESSED — may change nothing.

One addition from phase five: a complete run on a non-default branch also sets baselines[<branch>].complete = true. It is deliberately not set on the default branch, so a clean green run there stays byte-identical.

4.3 Running them

They are in the integration group (tests/Pest.php), so composer test:unit skips them. A directory argument finds nothing — pass files, space-separated, as separate argv entries:

PAO_DISABLE=1 php84 bin/pest tests/Features/Tia/BranchShapes.php tests/Features/Tia/CompleteRunWriteTier.php \
  tests/Features/Tia/DefaultBranchReplay.php tests/Features/Tia/DefaultBranchResolution.php \
  tests/Features/Tia/DefaultBranchWriteTier.php tests/Features/Tia/FilteredMode.php \
  tests/Features/Tia/HostileState.php tests/Features/Tia/IssueStatuses.php \
  tests/Features/Tia/PartialRunWriteTier.php tests/Features/Tia/StateReclamation.php

PAO_DISABLE=1 php   bin/pest <same list>

Both interpreters must be green. php84 is 8.4.x with no pcov — that is what CI has (.github/workflows/tests.yml sets coverage: none). php is 8.5.x with pcov — that is the dev machine. Any assertion that depends on a coverage driver passes locally and fails in CI. Concretely:

• A cold recording run writes no graph at all without pcov/xdebug (it prints Running in TIA mode, however TIA is skipped as it needs ext-pcov or Xdebug). Seed a graph, never record one, unless the driver is the point of the row — see M1, which has to solve this properly.
• A PHP source file edit driverless triggers Detected PHP source changes but no coverage driver is available → full suite, affected=0. Edit test files, not app/ files.
• The terminate() path differs by driver: with pcov the plugin reaches the complete write through the shutdown handler, without it the run exits earlier. Defect 10 only reproduced on php. Run every probe on both before believing it.

PAO_DISABLE=1 is mandatory on every pest invocation and every probe: laravel/pao emits JSON under agents and corrupts the captured output.

Baseline: 161 scenario rows green on both interpreters, at b49ba062. Per file:

File	Rows
StateReclamation.php	37
HostileState.php	25
CompleteRunWriteTier.php	17
DefaultBranchResolution.php	14
BranchShapes.php	14
DefaultBranchReplay.php	13
IssueStatuses.php	13
PartialRunWriteTier.php	12
DefaultBranchWriteTier.php	10
FilteredMode.php	6

Plus 80 unit/arch rows (tests/Unit/Plugins/Tia/*, tests/Arch.php). If those numbers do not reproduce, stop and say so.

4.4 Measure before you assert

Never guess an expectation from reading the code. Probe first, in your scratchpad:

<?php
require '/Users/nunomaduro/Work/projects/pestphp/pest/vendor/autoload.php';
use Tests\Fixtures\Tia\Project;

$p = Project::make('master');
$p->seed('master');
$p->git()->switchTo('feature-x', new: true);
$r = $p->pest('--tia');
printf("tally=[%s] keys=[%s] %s\n", $r->tally(), implode(',', $p->branchKeys()), $p->delta()->summary());
$p->destroy();

PAO_DISABLE=1 php probe.php. One project per scenario; always destroy(); Project::destroyAll() at the end. Run every probe on both interpreters before believing it.

---

Part 5 — Ground rules

• Do not run composer test. It takes minutes and you do not need it.
• tests/.snapshots/success.txt and the tally in tests/Visual/Parallel.php are stale right now — phase five added 89 rows and did not regenerate them. Report that they need regenerating and let Nuno run composer update:snapshots. Do not run it yourself.
• Do not commit. Leave the tree dirty.
• Do not touch the playground's vendor/. If a finding truly needs the playground (25 real tests, real timings), say so and ask — syncing it is a manual step Nuno owns.
• Run vendor/bin/phpstan analyse <the files you touched> --memory-limit=-1 --no-progress and vendor/bin/pint <the files you touched> before reporting. Scope both to files you changed — Nuno edits src/ live, and a repo-wide fixer will revert his work.
• The fixture projects hardlink src/. If you edit src/ while a scenario run is in flight, those rows go red for no reason. Finish the edit, then run.
• Keep scratch probes out of the repo.
• Never weaken an existing assertion to make something pass. If an existing row contradicts your fix, that is a finding: report the contradiction and ask. Phase five hit this once (see 1.2) and rewrote the row to pin the narrower contract rather than deleting it — that is the bar.

---

Part 6 — Reporting

Between passes, not just at the end. Per finding:

• Reproduced (yes / no / hypothesis only), with the exact command and the measured tally + delta()->summary() on both interpreters.
• Severity, and say why in one line. The scale that matters here: a test wrongly replayed as passing (worst) > cache silently useless, full suite forever > graph grows / stale data > cosmetic.
• Fixed / deferred / needs-a-decision, and the row that pins it.
• For anything needing a decision: one yes/no question, no essay.

Close with:

1. The count of tests/Features/Tia/* green on both php84 (no pcov) and php (pcov), against the 161 baseline.
2. Every row you added, and what invariant from Part 2 it defends.
3. Which findings are still open, as yes/no questions.
4. That the snapshots need regenerating — do not regenerate them.
5. What you looked at and found solid. A list of attacks that did not break anything is a real result: it tells the next phase where not to spend its time.

---

Part 7 — Open questions carried into this phase

Answer these before or during Pass C; they change what the fixes should be.

1. A fetched remote baseline carries branch keys this clone has never heard of, and the branch GC will prune them on the next complete write. Should fetched keys be exempt?
2. Two concurrent parallel runs share the worker-edges-* / worker-results-* prefixes and purgeWorkerPartials() deletes by prefix. Should partials be namespaced per run, or is "do not run two TIA suites at once" the contract?
3. files and edges never lose a deleted source file. Cap, GC, or accept?
4. Orphaned state dirs accumulate under ~/.pest/tia whenever a project's origin URL changes. Reclaim them, or accept?