pest/PLAN_PHASE_FIVE.md

TIA deep audit — phase five

Your task

Phases one through four fixed the defects that were reported. This phase is the opposite shape: go looking. Read TIA's read/write path adversarially, find what is wrong or fragile, and leave behind a scenario suite that covers the edges nobody has exercised yet.

Two deliverables, both required:

1. A ranked findings list. Every finding needs a reproduction, not a reading of the code. A finding you cannot reproduce is a hypothesis — say so and rank it separately.
2. New scenario tests in tests/Features/Tia/*, covering the edges you probed. Rows that pass go in the repo (they are the regression net). Rows that fail stay in your scratchpad until the fix lands — never commit a red test.

Work in passes, and report between passes rather than at the very end:

• Pass A — reproduce the leads in Part 3 below. Report which are real.
• Pass B — your own hunt: the invariants in Part 2, attacked with inputs nobody tried.
• Pass C — fixes, smallest first, each with the row that pins it.

Fix what is clearly a defect with an obvious correct answer. Stop and ask when the fix is a behaviour choice (what should TIA do when two runs race for one graph?) — those are Nuno's calls, and a reproduction with a crisp yes/no question is worth more than a guessed fix.

---

Part 1 — The harness

1.1 What TIA is

Test Impact Analysis. pest --tia records a dependency graph (test file → source files it touched) plus a per-branch baseline of results, then on later runs replays the tests whose dependencies did not change. State lives in a per-project dir under ~/.pest/tia/; graph.json is the whole thing.

Source of truth: src/Plugins/Tia.php (the plugin, ~1900 lines) and src/Plugins/Tia/Graph.php (the graph model + read/write API). Supporting: src/Plugins/Tia/ChangedFiles.php (git), Fingerprint.php (environment/structure hashing), State.php (the state dir).

Read PLAN.md, PLAN_PHASE_TWO.md, PLAN_PHASE_THREE.md, PLAN_PHASE_FOUR.md first — in that order. They carry the history, the tier contract, and the decisions already made. Struck-through rows are fixed; do not re-report them.

1.2 The scenario harness

tests/Features/Tia/* scaffold a throwaway git project into a temp dir, run a real pest subprocess against it, and diff the graph it wrote. Everything lives in tests/Fixtures/Tia/:

Class	What it gives you
Project	make(branch, overlay:), withoutGit(), seed(branch, sentinel:, failing:), seedFor($root, …), pest(...$args), pestWithEnvironment($dir, $env, ...$args), pestIn($dir, ...), write($rel, $contents), path($rel), graph(), branchKeys(), graphDir(), graphExists(), snapshot(), delta(), mutateGraph(fn), addBaseline($branch), worktree($branch), destroy(), destroyAll(), SEQUENTIAL_AND_PARALLEL
GitRepo ($project->git())	switchTo($b, new:), rename($from, $to), detach(), config($k, $v), addOrigin(), removeOrigin(), setOriginHead($b), unsetOriginHead(), worktree(), sha(), branchNames()
GraphDelta ($project->delta())	writtenCount(), added(), removed(), branchKeys(), baselineUntouched($b), shaMoved(), treeMoved(), edgesMoved(), filesMoved(), fingerprintMoved(), structureMoved(), isResultsOnly(), isHardSuppressed(), summary()
PestResult (returned by pest())	replayed(), uncached(), affected(), tally(), output, exitCode, describe()

The fixture app is 3 test files / 6 tests (Project::TOTAL_TESTS); Project::EDGES and Project::TESTS describe the graph seed() writes. Project::testId($file, $description) builds a result key. Overlays in tests/Fixtures/Tia/overlays/<name>/ supply a different tests/Pest.php — that is how you configure pest()->tia()->… for a scenario.

If the fixture cannot express your case, extend the fixture. A new GitRepo verb or a new overlay is a legitimate part of the deliverable — several findings below need one. Do not water down a scenario to fit the current helpers.

1.3 The sentinel discriminator — read before writing any assertion

seed() writes a graph and then rewrites every cached result to time=9.999, assertions=42 (only where assertions are non-zero — risky/skipped/incomplete statuses are derived from "performed no assertions", so falsifying those would rewrite the status on replay), then snapshots. Therefore:

• $delta->writtenCount() is the only reliable way to tell "replayed" from "executed and wrote back the same values". 0 means nothing was written.
• isResultsOnly() = no structure moved, no sha/tree movement, nothing added or removed.
• isHardSuppressed() = the graph is byte-identical.
• A replayed suite reports inflated assertion counts (6 tests × 42). That is the sentinel, not a bug.

The three write tiers, unchanged since phase two:

• COMPLETE — may change everything.
• RESULTS-ONLY — may change only baselines[<branch>].results for tests that ran; never removes an entry, never adds a result for a test file absent from edges, never touches sha/tree/edges/files/fingerprint.
• HARD-SUPPRESSED — may change nothing.

1.4 Running them

They are in the integration group (tests/Pest.php), so composer test:unit skips them. A directory argument finds nothing — pass files, space-separated, as separate argv entries:

PAO_DISABLE=1 php84 bin/pest tests/Features/Tia/DefaultBranchReplay.php tests/Features/Tia/DefaultBranchResolution.php \
  tests/Features/Tia/DefaultBranchWriteTier.php tests/Features/Tia/PartialRunWriteTier.php \
  tests/Features/Tia/CompleteRunWriteTier.php tests/Features/Tia/FilteredMode.php

PAO_DISABLE=1 php   bin/pest <same list>

Both interpreters must be green. php84 is 8.4.x with no pcov — that is what CI has (.github/workflows/tests.yml sets coverage: none). php is 8.5.x with pcov — that is the dev machine. Any assertion that depends on a coverage driver passes locally and fails in CI. Concretely:

• A cold recording run writes no graph at all without pcov/xdebug (it prints Running in TIA mode, however TIA is skipped as it needs ext-pcov or Xdebug). Seed a graph, never record one, unless the driver is the point of the row.
• A PHP source file edit driverless triggers Detected PHP source changes but no coverage driver is available → full suite, affected=0. Edit test files, not app/ files.

PAO_DISABLE=1 is mandatory on every pest invocation and every probe: laravel/pao emits JSON under agents and corrupts the captured output.

Baseline before you touch anything: 72 passed on both interpreters, at HEAD plus the phase-four working-tree changes. If that number does not reproduce, stop and say so.

1.5 Measure before you assert

Never guess an expectation from reading the code. Probe first, in your scratchpad:

<?php
require '/Users/nunomaduro/Work/projects/pestphp/pest/vendor/autoload.php';
use Tests\Fixtures\Tia\Project;

$p = Project::make('master');
$p->seed('master');
$p->git()->switchTo('feature-x', new: true);
$r = $p->pest('--tia');
printf("tally=[%s] keys=[%s] %s\n", $r->tally(), implode(',', $p->branchKeys()), $p->delta()->summary());
$p->destroy();

PAO_DISABLE=1 php probe.php. One project per scenario; always destroy(); Project::destroyAll() at the end. Run every probe on both interpreters before believing it.

1.6 Ground rules

• Do not run composer test. It takes minutes and you do not need it.
• Do not run composer update:snapshots. tests/.snapshots/success.txt and the tally in tests/Visual/Parallel.php encode the whole suite's result, so every row you add breaks them. That is expected — report that they need regenerating and let Nuno run it.
• Do not commit. Leave the tree dirty.
• Do not touch the playground's vendor/. If a finding truly needs the playground (25 real tests, real timings), say so and ask — syncing it is a manual step Nuno owns.
• Run vendor/bin/phpstan analyse <the files you touched> --memory-limit=-1 --no-progress and vendor/bin/pint <the files you touched> before reporting. Scope both to files you changed — Nuno edits src/ live, and a repo-wide fixer will revert his work.
• The fixture projects hardlink src/. If you edit src/ while a scenario run is in flight, those rows go red for no reason. Finish the edit, then run.
• Keep scratch probes out of the repo.
• Never weaken an existing assertion to make something pass. If an existing row contradicts your fix, that is a finding: report the contradiction and ask.

---

Part 2 — The invariants to attack

These are the properties TIA is supposed to have. Each one is a place to hunt: construct the input that breaks it.

1. Parity. pest <args> and pest --parallel --processes=N <args> must leave the same graph and reach the same tally. This is a hard rule from Nuno — sequential and parallel must always agree. Project::SEQUENTIAL_AND_PARALLEL is the dataset that encodes it; consider making every new write-path row use it. Vary --processes (1, 2, 8 — more processes than test files).
2. The tiers hold. Every command lands in exactly one of COMPLETE / RESULTS-ONLY / HARD-SUPPRESSED, and stays inside it. Combinations are where this frays: --fresh --parallel --filter, --bail --shard, --filtered plus an explicit path, --tia --no-tia, --retry.
3. Replay is faithful. A replayed test reports the same status, message, time and assertion count as the recorded run — and replay itself writes nothing. Statuses beyond pass/fail are the soft spot: skipped, incomplete, risky, notice, deprecation, warning, todo, and a test that failed with a multi-line message.
4. Reads never write. No read path may mint a baseline key, move a sha, or create the state dir. A project that has never run TIA must gain nothing from a plain pest run.
5. A branch never corrupts another branch's baseline. Writes land on the branch that ran, and only there. Reads may layer the default branch under the current one (phase four, B1) — but that layering must not leak into a write.
6. Nothing is unbounded. Baseline keys, files, edges, worker partials, state files: something must eventually reclaim them, or the graph grows forever.
7. A hostile state dir cannot break a run. Corrupt, truncated, empty, wrong-schema, read-only, absent, or someone else's graph.json — the suite still runs and exits on the tests' merit.
8. Git shapes are all handled. Detached HEAD (read-only, per phase four B2), worktrees, no commits yet, no origin, no origin/HEAD, submodules, a repo whose root is above the pest project (that one panics deliberately — TiaRequiresRepositoryRoot), renamed branches, deleted branches.

---

Part 3 — Leads to reproduce first (Pass A)

These came out of reading the phase-four diff. Each is a hypothesis, not a finding — several may turn out to be fine. Reproduce or refute each, in order, and report the measured delta either way.

L1 — the per-entry fallback may resurrect a pruned or deleted test · highest value

Phase four made Graph::baselineFor() layer the branch's results over the default branch's. Two consequences worth probing:

• pruneStaleResults() unsets an entry from baselines[branch].results. The very next read layers the fallback's entry for that same test id back in. So a delete may not stick from a branch's point of view.
• hasUnlocatedTestsToRerun() returns true when a failing cached result names a file that no longer exists on disk — and that forces a full suite. If a feature branch deletes a test file that fails on the default branch, the merged read still carries master's entry pointing at the now-absent file. Suspected symptom: that branch runs the full suite forever.

Probe: seed master with a failing test (seed('master', failing: [...])), branch off, delete the test file that holds it, run --tia, and compare replayed/uncached/affected against the same shape where the failure is on the branch instead. Then the mirror case with a green deleted test.

L2 — environment drift clears one branch's results and the fallback serves them right back

reconcileFingerprint() on environmental drift calls $graph->clearResults($this->branch) and warns results dropped, edges reused. On a feature branch that clears only the branch's results — the layered read then re-serves the default branch's results, which were recorded under the old environment. Suspected symptom: the drop is a no-op on any branch that is not the default one.

Probe: pestWithEnvironment() to shift whatever Fingerprint reads as environmental (check Fingerprint::environmentalDrift() for the exact keys), on the default branch vs a feature branch, and compare what survives.

L3 — sha/tree may be taken from a different commit than the results

baselineFor() takes sha from the branch when non-null and otherwise from the fallback; tree likewise when the branch's is empty. So a branch can end up computing "what changed since" against the default branch's recorded sha while reading its own results — or vice versa. Is there a shape where that under-reports changed files (a test replays that should have run)? That is the dangerous direction: a false replay is a lie about a passing test.

Probe: seed master, commit a test edit on the branch so the shas genuinely differ, and check whether the edited test is treated as affected.

L4 — pruning from merged worker partials

Phase four made a complete --parallel run write and prune from merged worker results. The stated safety net is that pruneStaleResults() only prunes files it saw results for, and that a truncated worker sets results-only. Try to defeat it: a worker that reports results for a test file it did not finish. Candidate shapes — a fatal error mid-file (not an assertion failure), exit() inside a test, an uncaught error in an afterEach, a test that kills its own process, --stop-on-failure variants, --processes greater than the number of test files.

L5 — two runs racing for one graph.json

State::write() has no locking. Two pest processes on one project (a watcher plus a manual run, two CI jobs sharing a cache dir, --parallel where the parent writes while a straggler worker flushes) can lose an update or interleave. Probe by launching two pest --tia subprocesses concurrently against one project and diffing. This is likely a design decision, not a bug — if you reproduce a lost update, report it as a question (accept last-writer-wins, or lock?), do not invent a locking scheme.

L6 — a detached HEAD still purges on structural drift

Phase four made a detached HEAD read-only for writes (saveGraph() refuses). But reconcileFingerprint() deletes the whole graph on structural drift (Tia.php, the state->delete(KEY_GRAPH) in the structural branch) before any write happens. So --tia from a detached checkout with a changed composer.lock can still wipe the default branch's baseline. Confirm it, then ask: should the detached-HEAD guard cover the purge too?

L7 — statuses that may not round-trip

PLAN.md §5 claims warnings and deprecations record as status=0, and that codes 6 and 4 look unreachable. Graph::getResult() maps 0–8 to TestStatus. Verify each status end to end: record it, replay it, and check the replayed run reports the same thing — including the message, the exit code, and whether shouldRerunStatus() decides to re-execute it. failOnRisky / failOnSkipped / displayDetailsOn* change that decision, so an overlay that flips those config flags is part of this. A status that replays as a pass would be the most serious class of bug in TIA.

L8 — branch-key hygiene

Nothing appears to reclaim baseline keys. Probe: create and delete 5 branches, rename one (GitRepo::rename()), and check what branchKeys() holds afterwards. Also try names that stress the JSON keying: feature/x/y (already covered), a name differing from another only in case (macOS is case-insensitive — does the key match the ref?), a name with a space or a unicode character, a branch literally called HEAD, and a very long name. Then: is unbounded growth acceptable, or does this need a cap / GC? Ask rather than build.

L9 — soleRecordedBranch() as a fallback source

When config, CI env and git all fail to name a default branch, resolution falls back to "the only branch in the graph". If that sole key was minted by a narrowed run on a feature branch (which phase four's B1 made a live possibility), the fallback now names a feature branch, and every other branch layers its results underneath. Probe: withoutGit() or removeOrigin() + no config, with a graph whose only key is feature-x.

L10 — the state dir as an adversary

Beyond corrupt JSON (fixed in phase four by deleting it): a valid-JSON graph with schema: 2; a graph whose files and edges disagree; results entries with a file pointing outside the project root or at an absolute path from another machine; a graph.json that is a directory; a state dir with no write permission; $HOME unset. Each should degrade to "run the tests", never crash and never write garbage.

---

Part 4 — Reporting

Between passes, not just at the end. Per finding:

• Reproduced (yes / no / hypothesis only), with the exact command and the measured tally + delta()->summary() on both interpreters.
• Severity, and say why in one line. The scale that matters here: a test wrongly replayed as passing (worst) > cache silently useless, full suite forever > graph grows / stale data > cosmetic.
• Fixed / deferred / needs-a-decision, and the row that pins it.
• For anything needing a decision: one yes/no question, no essay.

Close with:

1. The count of tests/Features/Tia/* green on both php84 (no pcov) and php85 (pcov), against the 72 baseline.
2. Every row you added, and what invariant from Part 2 it defends.
3. Which findings are still open, as yes/no questions.
4. That tests/.snapshots/success.txt and the tests/Visual/Parallel.php tally need regenerating — do not regenerate them.
5. What you looked at and found solid. A list of attacks that did not break anything is a real result: it tells the next phase where not to spend its time.