diff --git a/PLAN_PHASE_FIVE.md b/PLAN_PHASE_FIVE.md deleted file mode 100644 index ad75908a..00000000 --- a/PLAN_PHASE_FIVE.md +++ /dev/null @@ -1,312 +0,0 @@ -# TIA deep audit — phase five - -## Your task - -Phases one through four fixed the defects that were *reported*. This phase is the opposite shape: go -looking. Read TIA's read/write path adversarially, find what is wrong or fragile, and leave behind a -scenario suite that covers the edges nobody has exercised yet. - -Two deliverables, both required: - -1. **A ranked findings list.** Every finding needs a *reproduction*, not a reading of the code. A - finding you cannot reproduce is a hypothesis — say so and rank it separately. -2. **New scenario tests** in `tests/Features/Tia/*`, covering the edges you probed. Rows that pass go - in the repo (they are the regression net). Rows that fail stay in your scratchpad until the fix - lands — **never commit a red test.** - -Work in passes, and **report between passes** rather than at the very end: - -- **Pass A** — reproduce the leads in Part 3 below. Report which are real. -- **Pass B** — your own hunt: the invariants in Part 2, attacked with inputs nobody tried. -- **Pass C** — fixes, smallest first, each with the row that pins it. - -Fix what is clearly a defect with an obvious correct answer. **Stop and ask** when the fix is a -behaviour *choice* (what should TIA do when two runs race for one graph?) — those are Nuno's calls, -and a reproduction with a crisp yes/no question is worth more than a guessed fix. - ---- - -## Part 1 — The harness - -### 1.1 What TIA is - -Test Impact Analysis. `pest --tia` records a dependency graph (test file → source files it touched) -plus a per-branch baseline of results, then on later runs replays the tests whose dependencies did not -change. State lives in a per-project dir under `~/.pest/tia/`; `graph.json` is the whole thing. - -Source of truth: `src/Plugins/Tia.php` (the plugin, ~1900 lines) and `src/Plugins/Tia/Graph.php` (the -graph model + read/write API). Supporting: `src/Plugins/Tia/ChangedFiles.php` (git), `Fingerprint.php` -(environment/structure hashing), `State.php` (the state dir). - -Read `PLAN.md`, `PLAN_PHASE_TWO.md`, `PLAN_PHASE_THREE.md`, `PLAN_PHASE_FOUR.md` first — in that -order. They carry the history, the tier contract, and the decisions already made. Struck-through rows -are fixed; do not re-report them. - -### 1.2 The scenario harness - -`tests/Features/Tia/*` scaffold a throwaway git project into a temp dir, run a **real `pest` -subprocess** against it, and diff the graph it wrote. Everything lives in `tests/Fixtures/Tia/`: - -| Class | What it gives you | -|---|---| -| `Project` | `make(branch, overlay:)`, `withoutGit()`, `seed(branch, sentinel:, failing:)`, `seedFor($root, …)`, `pest(...$args)`, `pestWithEnvironment($dir, $env, ...$args)`, `pestIn($dir, ...)`, `write($rel, $contents)`, `path($rel)`, `graph()`, `branchKeys()`, `graphDir()`, `graphExists()`, `snapshot()`, `delta()`, `mutateGraph(fn)`, `addBaseline($branch)`, `worktree($branch)`, `destroy()`, `destroyAll()`, `SEQUENTIAL_AND_PARALLEL` | -| `GitRepo` (`$project->git()`) | `switchTo($b, new:)`, `rename($from, $to)`, `detach()`, `config($k, $v)`, `addOrigin()`, `removeOrigin()`, `setOriginHead($b)`, `unsetOriginHead()`, `worktree()`, `sha()`, `branchNames()` | -| `GraphDelta` (`$project->delta()`) | `writtenCount()`, `added()`, `removed()`, `branchKeys()`, `baselineUntouched($b)`, `shaMoved()`, `treeMoved()`, `edgesMoved()`, `filesMoved()`, `fingerprintMoved()`, `structureMoved()`, `isResultsOnly()`, `isHardSuppressed()`, `summary()` | -| `PestResult` (returned by `pest()`) | `replayed()`, `uncached()`, `affected()`, `tally()`, `output`, `exitCode`, `describe()` | - -The fixture app is 3 test files / **6 tests** (`Project::TOTAL_TESTS`); `Project::EDGES` and -`Project::TESTS` describe the graph `seed()` writes. `Project::testId($file, $description)` builds a -result key. Overlays in `tests/Fixtures/Tia/overlays//` supply a different `tests/Pest.php` — -that is how you configure `pest()->tia()->…` for a scenario. - -**If the fixture cannot express your case, extend the fixture.** A new `GitRepo` verb or a new overlay -is a legitimate part of the deliverable — several findings below need one. Do not water down a -scenario to fit the current helpers. - -### 1.3 The sentinel discriminator — read before writing any assertion - -`seed()` writes a graph **and then rewrites every cached result to `time=9.999`, `assertions=42`** -(only where assertions are non-zero — risky/skipped/incomplete statuses are *derived* from "performed -no assertions", so falsifying those would rewrite the status on replay), then snapshots. Therefore: - -- `$delta->writtenCount()` is the only reliable way to tell **"replayed"** from **"executed and wrote - back the same values"**. `0` means nothing was written. -- `isResultsOnly()` = no structure moved, no `sha`/`tree` movement, nothing added or removed. -- `isHardSuppressed()` = the graph is byte-identical. -- A replayed suite reports inflated assertion counts (6 tests × 42). That is the sentinel, not a bug. - -The three write tiers, unchanged since phase two: - -- **COMPLETE** — may change everything. -- **RESULTS-ONLY** — may change only `baselines[].results` for tests that ran; never removes - an entry, never adds a result for a test file absent from `edges`, never touches - `sha`/`tree`/`edges`/`files`/`fingerprint`. -- **HARD-SUPPRESSED** — may change nothing. - -### 1.4 Running them - -They are in the `integration` group (`tests/Pest.php`), so `composer test:unit` skips them. **A -directory argument finds nothing** — pass files, space-separated, as separate argv entries: - -```bash -PAO_DISABLE=1 php84 bin/pest tests/Features/Tia/DefaultBranchReplay.php tests/Features/Tia/DefaultBranchResolution.php \ - tests/Features/Tia/DefaultBranchWriteTier.php tests/Features/Tia/PartialRunWriteTier.php \ - tests/Features/Tia/CompleteRunWriteTier.php tests/Features/Tia/FilteredMode.php - -PAO_DISABLE=1 php bin/pest -``` - -**Both interpreters must be green.** `php84` is 8.4.x with **no pcov** — that is what CI has -(`.github/workflows/tests.yml` sets `coverage: none`). `php` is 8.5.x with pcov — that is the dev -machine. Any assertion that depends on a coverage driver passes locally and fails in CI. Concretely: - -- A **cold recording run writes no graph at all** without pcov/xdebug (it prints `Running in TIA mode, - however TIA is skipped as it needs ext-pcov or Xdebug`). **Seed a graph, never record one**, unless - the driver *is* the point of the row. -- A **PHP source file edit** driverless triggers `Detected PHP source changes but no coverage driver - is available` → full suite, `affected=0`. Edit *test* files, not `app/` files. - -`PAO_DISABLE=1` is mandatory on every pest invocation and every probe: `laravel/pao` emits JSON under -agents and corrupts the captured output. - -Baseline before you touch anything: **72 passed** on both interpreters, at `HEAD` plus the phase-four -working-tree changes. If that number does not reproduce, stop and say so. - -### 1.5 Measure before you assert - -Never guess an expectation from reading the code. Probe first, in your scratchpad: - -```php -seed('master'); -$p->git()->switchTo('feature-x', new: true); -$r = $p->pest('--tia'); -printf("tally=[%s] keys=[%s] %s\n", $r->tally(), implode(',', $p->branchKeys()), $p->delta()->summary()); -$p->destroy(); -``` - -`PAO_DISABLE=1 php probe.php`. One project per scenario; always `destroy()`; `Project::destroyAll()` -at the end. Run every probe on **both** interpreters before believing it. - -### 1.6 Ground rules - -- **Do not** run `composer test`. It takes minutes and you do not need it. -- **Do not** run `composer update:snapshots`. `tests/.snapshots/success.txt` and the tally in - `tests/Visual/Parallel.php` encode the whole suite's result, so every row you add breaks them. - That is expected — **report that they need regenerating and let Nuno run it.** -- **Do not commit.** Leave the tree dirty. -- **Do not touch the playground's `vendor/`.** If a finding truly needs the playground (25 real tests, - real timings), say so and ask — syncing it is a manual step Nuno owns. -- Run `vendor/bin/phpstan analyse --memory-limit=-1 --no-progress` and - `vendor/bin/pint ` before reporting. **Scope both to files you changed** — - Nuno edits `src/` live, and a repo-wide fixer will revert his work. -- The fixture projects **hardlink `src/`**. If you edit `src/` while a scenario run is in flight, - those rows go red for no reason. Finish the edit, then run. -- Keep scratch probes out of the repo. -- **Never weaken an existing assertion to make something pass.** If an existing row contradicts your - fix, that is a finding: report the contradiction and ask. - ---- - -## Part 2 — The invariants to attack - -These are the properties TIA is supposed to have. Each one is a place to hunt: construct the input -that breaks it. - -1. **Parity.** `pest ` and `pest --parallel --processes=N ` must leave the **same graph** - and reach the same tally. This is a hard rule from Nuno — sequential and parallel must *always* - agree. `Project::SEQUENTIAL_AND_PARALLEL` is the dataset that encodes it; consider making every - new write-path row use it. Vary `--processes` (1, 2, 8 — more processes than test files). -2. **The tiers hold.** Every command lands in exactly one of COMPLETE / RESULTS-ONLY / - HARD-SUPPRESSED, and stays inside it. Combinations are where this frays: `--fresh --parallel - --filter`, `--bail --shard`, `--filtered` plus an explicit path, `--tia --no-tia`, `--retry`. -3. **Replay is faithful.** A replayed test reports the same status, message, time and assertion count - as the recorded run — and replay itself writes nothing. Statuses beyond pass/fail are the soft - spot: skipped, incomplete, risky, notice, deprecation, warning, todo, and a test that failed with a - multi-line message. -4. **Reads never write.** No read path may mint a baseline key, move a `sha`, or create the state dir. - A project that has never run TIA must gain nothing from a plain `pest` run. -5. **A branch never corrupts another branch's baseline.** Writes land on the branch that ran, and - only there. Reads may *layer* the default branch under the current one (phase four, B1) — but that - layering must not leak into a write. -6. **Nothing is unbounded.** Baseline keys, `files`, `edges`, worker partials, state files: something - must eventually reclaim them, or the graph grows forever. -7. **A hostile state dir cannot break a run.** Corrupt, truncated, empty, wrong-schema, read-only, - absent, or *someone else's* `graph.json` — the suite still runs and exits on the tests' merit. -8. **Git shapes are all handled.** Detached HEAD (read-only, per phase four B2), worktrees, no commits - yet, no `origin`, no `origin/HEAD`, submodules, a repo whose root is above the pest project (that - one panics deliberately — `TiaRequiresRepositoryRoot`), renamed branches, deleted branches. - ---- - -## Part 3 — Leads to reproduce first (Pass A) - -These came out of reading the phase-four diff. **Each is a hypothesis, not a finding** — several may -turn out to be fine. Reproduce or refute each, in order, and report the measured delta either way. - -### L1 — the per-entry fallback may resurrect a pruned or deleted test · **highest value** - -Phase four made `Graph::baselineFor()` layer the branch's results **over** the default branch's. Two -consequences worth probing: - -- `pruneStaleResults()` unsets an entry from `baselines[branch].results`. The very next read layers the - **fallback's** entry for that same test id back in. So a delete may not stick from a branch's point - of view. -- `hasUnlocatedTestsToRerun()` returns true when a *failing* cached result names a file that no longer - exists on disk — and that forces a **full suite**. If a feature branch deletes a test file that - fails on the default branch, the merged read still carries master's entry pointing at the now-absent - file. Suspected symptom: **that branch runs the full suite forever.** - -Probe: seed master with a failing test (`seed('master', failing: [...])`), branch off, delete the test -file that holds it, run `--tia`, and compare `replayed`/`uncached`/`affected` against the same shape -where the failure is on the branch instead. Then the mirror case with a green deleted test. - -### L2 — environment drift clears one branch's results and the fallback serves them right back - -`reconcileFingerprint()` on environmental drift calls `$graph->clearResults($this->branch)` and warns -`results dropped, edges reused`. On a feature branch that clears only the *branch's* results — the -layered read then re-serves the default branch's results, which were recorded under the **old** -environment. Suspected symptom: the drop is a no-op on any branch that is not the default one. - -Probe: `pestWithEnvironment()` to shift whatever `Fingerprint` reads as environmental (check -`Fingerprint::environmentalDrift()` for the exact keys), on the default branch vs a feature branch, -and compare what survives. - -### L3 — `sha`/`tree` may be taken from a different commit than the results - -`baselineFor()` takes `sha` from the branch when non-null and otherwise from the fallback; `tree` -likewise when the branch's is empty. So a branch can end up computing "what changed since" against the -**default branch's** recorded sha while reading its own results — or vice versa. Is there a shape where -that under-reports changed files (a test replays that should have run)? That is the dangerous -direction: a false replay is a lie about a passing test. - -Probe: seed master, commit a test edit on the branch so the shas genuinely differ, and check whether -the edited test is treated as affected. - -### L4 — pruning from merged worker partials - -Phase four made a complete `--parallel` run write and prune from merged worker results. The stated -safety net is that `pruneStaleResults()` only prunes files it saw results for, and that a truncated -worker sets results-only. Try to defeat it: a worker that reports results for a test file it did not -finish. Candidate shapes — a fatal error mid-file (not an assertion failure), `exit()` inside a test, -an uncaught error in an `afterEach`, a test that kills its own process, `--stop-on-failure` variants, -`--processes` greater than the number of test files. - -### L5 — two runs racing for one `graph.json` - -`State::write()` has no locking. Two pest processes on one project (a watcher plus a manual run, two CI -jobs sharing a cache dir, `--parallel` where the parent writes while a straggler worker flushes) can -lose an update or interleave. Probe by launching two `pest --tia` subprocesses concurrently against one -project and diffing. **This is likely a design decision, not a bug** — if you reproduce a lost update, -report it as a question (accept last-writer-wins, or lock?), do not invent a locking scheme. - -### L6 — a detached HEAD still purges on structural drift - -Phase four made a detached HEAD read-only *for writes* (`saveGraph()` refuses). But -`reconcileFingerprint()` deletes the whole graph on structural drift (`Tia.php`, the -`state->delete(KEY_GRAPH)` in the structural branch) before any write happens. So `--tia` from a -detached checkout with a changed `composer.lock` can still wipe the default branch's baseline. Confirm -it, then ask: should the detached-HEAD guard cover the purge too? - -### L7 — statuses that may not round-trip - -`PLAN.md` §5 claims warnings and deprecations record as `status=0`, and that codes `6` and `4` look -unreachable. `Graph::getResult()` maps 0–8 to `TestStatus`. Verify each status end to end: record it, -replay it, and check the replayed run reports the same thing — including the message, the exit code, -and whether `shouldRerunStatus()` decides to re-execute it. `failOnRisky` / `failOnSkipped` / -`displayDetailsOn*` change that decision, so an overlay that flips those config flags is part of this. -A status that replays as a pass would be the most serious class of bug in TIA. - -### L8 — branch-key hygiene - -Nothing appears to reclaim baseline keys. Probe: create and delete 5 branches, rename one -(`GitRepo::rename()`), and check what `branchKeys()` holds afterwards. Also try names that stress the -JSON keying: `feature/x/y` (already covered), a name differing from another only in case (macOS is -case-insensitive — does the key match the ref?), a name with a space or a unicode character, a branch -literally called `HEAD`, and a very long name. Then: is unbounded growth acceptable, or does this need -a cap / GC? Ask rather than build. - -### L9 — `soleRecordedBranch()` as a fallback source - -When config, CI env and git all fail to name a default branch, resolution falls back to "the only -branch in the graph". If that sole key was minted by a *narrowed* run on a feature branch (which phase -four's B1 made a live possibility), the fallback now names a feature branch, and every other branch -layers **its** results underneath. Probe: `withoutGit()` or `removeOrigin()` + no config, with a graph -whose only key is `feature-x`. - -### L10 — the state dir as an adversary - -Beyond corrupt JSON (fixed in phase four by deleting it): a valid-JSON graph with `schema: 2`; a graph -whose `files` and `edges` disagree; `results` entries with a `file` pointing outside the project root -or at an absolute path from another machine; a `graph.json` that is a directory; a state dir with no -write permission; `$HOME` unset. Each should degrade to "run the tests", never crash and never write -garbage. - ---- - -## Part 4 — Reporting - -**Between passes**, not just at the end. Per finding: - -- **Reproduced (yes / no / hypothesis only)**, with the exact command and the measured - `tally` + `delta()->summary()` on **both** interpreters. -- **Severity**, and say why in one line. The scale that matters here: *a test wrongly replayed as - passing* (worst) > *cache silently useless, full suite forever* > *graph grows / stale data* > - *cosmetic*. -- **Fixed / deferred / needs-a-decision**, and the row that pins it. -- For anything needing a decision: **one yes/no question**, no essay. - -Close with: - -1. The count of `tests/Features/Tia/*` green on **both** `php84` (no pcov) and `php85` (pcov), against - the 72 baseline. -2. Every row you added, and what invariant from Part 2 it defends. -3. Which findings are still open, as yes/no questions. -4. That `tests/.snapshots/success.txt` and the `tests/Visual/Parallel.php` tally need regenerating — - **do not regenerate them.** -5. **What you looked at and found solid.** A list of attacks that did not break anything is a real - result: it tells the next phase where not to spend its time. diff --git a/PLAN_PHASE_FOUR.md b/PLAN_PHASE_FOUR.md deleted file mode 100644 index 84d0c338..00000000 --- a/PLAN_PHASE_FOUR.md +++ /dev/null @@ -1,351 +0,0 @@ -# TIA defect sweep — phase four - -## Your task - -Five defects in TIA's read/write path, found while building the repo's TIA scenario suite. One is -confirmed and load-bearing (**B1**); four need a decision before a fix (**B2**–**B5**). - -Work **one bug at a time, in order**, and for each: - -1. **Reproduce it as a repo test first.** The reproduction is the deliverable even when the fix is - deferred — a red test that pins the exact symptom is worth more than a prose report. Do not commit - a red test to the suite; keep it in a scratch file until the fix lands (see Part 1.4). -2. Confirm the measured numbers in this file still hold. They were taken at commit `4d3d0105` plus - the two uncommitted changes described in Part 2. If a number has moved, say so and stop. -3. Fix, then re-run **the whole `tests/Features/Tia/*` set on two interpreters** (Part 1.3). -4. **HARD STOP after B1.** Report the diff and wait — B1's fix changes `Graph`'s read semantics for - every caller, and Nuno wants to see it before B2–B5 pile on top. - -Per `CLAUDE.md`: do not run `composer test`, and do not regenerate snapshots unless told. Do not -commit. Do not touch the playground's `vendor/`. - ---- - -## Part 1 — The harness - -### 1.1 What exists - -`tests/Features/Tia/*` scaffold a throwaway git project into a temp dir, run a **real `pest` -subprocess** against it, and diff the TIA graph it wrote. Everything lives in -`tests/Fixtures/Tia/`: - -| Class | What it gives you | -|---|---| -| `Project` | `make(branch, overlay:)`, `withoutGit()`, `seed(branch, sentinel:, failing:)`, `pest(...$args)`, `pestWithEnvironment($dir, $env, ...$args)`, `pestIn($dir, ...)`, `write($rel, $contents)`, `graph()`, `branchKeys()`, `graphDir()`, `graphExists()`, `snapshot()`, `delta()`, `mutateGraph(fn)`, `addBaseline($branch)`, `worktree($branch)`, `destroyAll()` | -| `GitRepo` (`$project->git()`) | `switchTo($b, new:)`, `rename($from, $to)`, `detach()`, `config($k, $v)`, `addOrigin()`, `removeOrigin()`, `setOriginHead($b)`, `unsetOriginHead()`, `worktree()`, `sha()`, `branchNames()` | -| `GraphDelta` (`$project->delta()`) | `writtenCount()`, `added()`, `removed()`, `branchKeys()`, `baselineUntouched($b)`, `shaMoved()`, `treeMoved()`, `edgesMoved()`, `filesMoved()`, `fingerprintMoved()`, `structureMoved()`, `isResultsOnly()`, `isHardSuppressed()`, `summary()` | -| `PestResult` (returned by `pest()`) | `replayed()`, `uncached()`, `affected()`, `tally()`, `output`, `exitCode`, `describe()` | - -The fixture app is 3 test files / **6 tests** (`Project::TOTAL_TESTS`), with `Project::EDGES` and -`Project::TESTS` describing the graph `seed()` writes. `Project::testId($file, $description)` builds -a result key. - -### 1.2 The sentinel discriminator — read this before writing an assertion - -`seed()` writes a graph **and then rewrites every cached result to `time=9.999`, -`assertions=42`** (non-zero assertion counts only — risky/skipped/incomplete statuses are *derived* -from "performed no assertions", so falsifying those would rewrite the status on replay), and takes a -snapshot. Therefore: - -- `$delta->writtenCount()` is the only reliable way to tell **"replayed"** from **"executed and wrote - back the same values"**. `0` means nothing was written. -- `isResultsOnly()` = no structure moved, no `sha`/`tree` movement, nothing added or removed. -- `isHardSuppressed()` = the graph is byte-identical. - -Tiers, unchanged since phase two: **COMPLETE** may change everything · **RESULTS-ONLY** may change -only `baselines[].results` for tests that ran, never removing an entry, never adding a result -for a test file absent from `edges`, never touching `sha`/`tree`/`edges`/`files`/`fingerprint` · -**HARD-SUPPRESSED** may change nothing. - -### 1.3 Running them - -They are in the `integration` group (`tests/Pest.php:22`), so `composer test:unit` skips them. A -**directory argument finds nothing** — pass files: - -```bash -F="tests/Features/Tia/DefaultBranchReplay.php tests/Features/Tia/DefaultBranchResolution.php \ - tests/Features/Tia/DefaultBranchWriteTier.php tests/Features/Tia/PartialRunWriteTier.php \ - tests/Features/Tia/CompleteRunWriteTier.php tests/Features/Tia/FilteredMode.php" - -PAO_DISABLE=1 php84 bin/pest $F # 8.4.23, NO pcov — this is what CI has -PAO_DISABLE=1 php bin/pest $F # 8.5.8, pcov — this is what your machine has -``` - -**Both must be green.** `.github/workflows/tests.yml` sets `coverage: none`, so any assertion that -depends on a coverage driver fails in CI while passing locally. Two tests were already caught by -this. Concretely: a **cold recording run writes no graph at all** without pcov/xdebug (it prints -`Running in TIA mode, however TIA is skipped as it needs ext-pcov or Xdebug`), and a **PHP source -file edit** triggers `Detected PHP source changes but no coverage driver is available` → full suite, -`affected=0`. Seed a graph instead of recording one, and edit *test* files rather than `app/` files, -unless the point of the row is the driver itself. - -### 1.4 Measure before you assert - -Do not guess expectations from reading the code — every number in Part 3 came from a scratch probe. -The pattern (put it in your scratchpad, not in the repo): - -```php -seed('master'); -$p->git()->switchTo('feature-x', new: true); -$r = $p->pest('--tia'); -printf("tally=[%s] keys=[%s] %s\n", $r->tally(), implode(',', $p->branchKeys()), $p->delta()->summary()); -$p->destroy(); -``` - -`PAO_DISABLE=1 php probe.php`. One project per scenario; always `destroy()`. - ---- - -## Part 2 — What the code looks like right now - -Phase three landed default-branch resolution: `ChangedFiles::defaultBranch()`, the -`pest()->tia()->defaultBranch()` config surface, `Graph::setFallbackBranch()` + `?string -$fallbackBranch = null` on the seven read methods, and `Tia::resolveFallbackBranch()` -(`Tia.php:~1776`) resolving **config → CI env (`CiDefaultBranch`) → git (`origin/HEAD`, then -`init.defaultBranch` if the branch exists) → `soleRecordedBranch()`**, failing loudly when nothing -can name it. - -On top of that, **two uncommitted changes** you will see in `git diff`: - -1. `TIA_RESULTS_ONLY` global — a *partial* parallel run with an existing graph now purges stale - worker partials, sets the global, and workers flush their results through the existing - `flushWorkerReplay()` / `mergeWorkerReplayPartials()` path; the parent writes them with - `complete: false`. Gated on a graph already existing, so a TIA-less project still creates no - baseline dir. This gave `--parallel --filter` parity with sequential — **and, per B1, handed it - the shadowing bug too.** -2. `loadGraph()` emits `WARN The dependency graph could not be read — it will be rebuilt.` once per - parent process when `graph.json` exists but will not decode. Previously silent. - -62 scenario tests cover this and pass on both interpreters. - ---- - -## Part 3 — The defects - -### B1 — a thin baseline key permanently shadows the default-branch fallback · **confirmed, priority 1** - -**Symptom.** Any *narrowed* run on a new branch (`--filter`, `--group`, a path, `--bail`, `--shard`, -and now `--parallel --filter`) writes a baseline key holding only the tests that ran. From then on -`--tia` on that branch reads that thin key instead of falling back to the default branch, so -everything else is uncached — **one full suite per branch, forever**, which is the exact cost -issue [#1823](https://github.com/pestphp/pest/issues/1823) was about, re-entering through a side door. - -**Measured** (fixture: 6 tests, graph seeded on `master`): - -``` -switch -c feature-x; pest --filter="adds two numbers" - → keys=[master,feature-x] (feature-x holds 1 result) -pest --tia - → 6 passed (6 assertions, 5 uncached, 1 replayed) ← want: 6 replayed, 0 uncached - -same via: pest --parallel --processes=2 --filter="adds two numbers" → identical -``` - -**Where.** `src/Plugins/Tia/Graph.php::baselineFor()` (~line 814): - -```php -if (isset($this->baselines[$branch])) return $this->baselines[$branch]; -if ($branch !== $fallbackBranch && isset($this->baselines[$fallbackBranch])) return $this->baselines[$fallbackBranch]; -``` - -The fallback is all-or-nothing: it fires only when the branch has **no** key at all. The key itself -is minted by `Graph::setResult()` → `ensureBaseline($branch)` (~599 / ~829), reached from -`Tia::snapshotTestResults()` on partial runs. - -**Reproduction to add** (`tests/Features/Tia/DefaultBranchReplay.php`): - -```php -test('a narrowed run on a new branch does not cost the fallback', function (): void { - $project = Project::make('master'); - $project->seed('master'); - - $project->git()->switchTo('feature-x', new: true); - $project->pest('--filter=adds two numbers'); - - $result = $project->pest('--tia'); - - expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe()) - ->and($result->uncached())->toBe(0, $result->describe()); -})->skipOnWindows(); -``` - -Add the `--parallel --processes=2 --filter=…` variant as a second row (dataset), since the two write -paths are different code. - -**Fix direction.** Make the fallback **per entry** rather than per baseline: in `baselineFor()`, -when the branch has its own baseline *and* a distinct fallback baseline exists, return -`results` = the branch's results **layered over** the fallback's (branch wins per test id), and take -`sha`/`tree` from the fallback when the branch's are `null`/empty. `baselineFor()` is the single -funnel for `recordedAtSha()`, `lastRunTree()`, `getResult()`, `getTime()`, `getAssertions()`, -`testFilesToRerun()` and `hasUnlocatedTestsToRerun()`, so one change covers every reader. An -alternative — never mint a key from a partial run — is smaller but loses the executed result -entirely, which regresses the parity just gained. - -**Done when.** Both reproduction rows are green on both interpreters, and none of these move: - -- `the branch that ran gets its own key and the default branch keeps its baseline` — writes stay on - the real branch; the merge must be **read-only** and must not leak into `ensureBaseline()`/`setResult()`. -- `a declared default branch that does not exist degrades to a full run` — with a fallback that names - nothing, a branch's own thin results must still be all you get. -- `a detached HEAD replays without minting a branch key`, `writes nothing on a second run on the same - branch`, `filtered mode finds nothing to do…` (both rows) — a merged read must not make a clean - replay start writing. -- The whole `PartialRunWriteTier.php` / `CompleteRunWriteTier.php` set — tier semantics are unchanged - by this fix. - ---- - -### B2 — a partial run on detached HEAD writes into the default branch's baseline · **needs a decision** - -**Symptom.** With `HEAD` detached, `Tia::resolveBranch()` (`Tia.php:~1756`) sets -`$this->branch = $changedFiles->currentBranch() ?? $this->fallbackBranch` — and that branch is used -for **writes**. A `--tia` run in this state happens to be harmless (a clean replay writes nothing), -but any run that *executes* tests writes their results into the default branch's baseline. - -**Measured.** - -``` -seed on master; git checkout --detach; pest --filter="adds two numbers" - → keys=[master] w=1 struct:ok ← master's baseline rewritten from a detached checkout -seed on master; git checkout --detach; pest --tia - → keys=[master] w=0 ← read-only, as intended -``` - -**The decision.** `PLAN_PHASE_THREE.md` §2.3 **D3** recommended detached HEAD be *read-only*. If that -still stands, suppress writes when `currentBranch()` is `null` (a dedicated flag — note -`resultsOnlyWrites` is **not** enough, it still writes results). If Nuno prefers the current -behaviour, add a test pinning it and close this out. - -**Reproduction** (`tests/Features/Tia/DefaultBranchWriteTier.php`), written for the read-only answer: - -```php -test('a detached HEAD does not write into the default branch baseline', function (): void { - $project = Project::make('master'); - $project->seed('master'); - - $project->git()->detach(); - $project->pest('--filter=adds two numbers'); - - $delta = $project->delta(); - - expect($delta->baselineUntouched('master'))->toBeTrue($delta->summary()) - ->and($project->branchKeys())->toBe(['master']); -})->skipOnWindows(); -``` - ---- - -### B3 — an unreadable graph is never repaired on a machine with no coverage driver · **needs a decision** - -**Symptom.** A corrupt `graph.json` is now *reported* (Part 2, change 2) but only *rebuilt* when a -coverage driver is present, because rebuilding means recording. Driverless, the file stays corrupt -run after run and TIA is silently inert until someone deletes it by hand — while the WARN claims -`it will be rebuilt`. - -**Measured** (`php84`, no pcov): - -``` -overwrite graph.json with '{not json' -run 1: exit=0, 6 passed, file still '{not json' -run 2: exit=0, 6 passed, file still '{not json' -headline: "Running in TIA mode, however TIA is skipped as it needs ext-pcov or Xdebug" -``` - -**Options.** (a) delete the file when it cannot be decoded, so the next drivered run starts clean and -the state dir does not carry a permanent landmine; (b) keep the file but reword the WARN when no -driver is available. (a) is the honest one and costs one `State::delete()`. - -**Reproduction** (`tests/Features/Tia/FilteredMode.php`, extending the existing corrupt-graph row): - -```php -expect($result->output)->toContain('The dependency graph could not be read') - ->and(file_get_contents($project->graphDir().'/graph.json'))->not->toBe('{not json'); -``` - -Must pass on **both** interpreters — that is the whole point of the row. - ---- - -### B4 — a complete `--parallel` run writes nothing and prunes nothing · **needs a decision** - -**Symptom.** With a graph present and TIA not flagged, a sequential run refreshes results and applies -the prune; the same run under `--parallel` does neither, because the parent's `ResultCollector` is -empty (results live in the workers) and workers only flush when they were told to record, replay, or -— since Part 2's change — results-only for a *partial* run. So parallel CI contributes nothing to the -cache, and a deleted test's entry survives forever. - -**Measured** (graph seeded on `master`, sentinelled): - -``` -pest → w=6 +0 -0 struct:ok (sequential baseline) -pest --parallel --processes=2 → w=0 +0 -0 struct:ok ← writes nothing -delete a test, then: -pest --parallel --processes=2 → w=0 +0 -0 struct:ok ← and does not prune (sequential gives -1) -pest --tia --parallel --processes=2 → w=0 ← correct: everything replayed -``` - -**The decision.** Extending the `TIA_RESULTS_ONLY` mechanism to complete parallel runs is -mechanically easy, but a *complete* run also prunes, and pruning from merged worker partials is the -risky half: a worker that dies, or a shard that never ran, would look like "these tests no longer -exist". If it is done, the prune must key off "every worker reported" and fall back to -results-only when it cannot prove that. `PLAN.md` §5 lists this as a known gap, not a regression. - -**Reproduction** (`tests/Features/Tia/CompleteRunWriteTier.php`) — mirror the two sequential rows -that already exist (`a complete run prunes a deleted test`, `--no-tia refreshes results…`) with -`--parallel --processes=2` added, and assert the same deltas. - ---- - -### B5 — G4 ("parallel replay clobbers cached `time`") no longer reproduces · **verify, then correct the record** - -**Symptom.** `PLAN_PHASE_THREE.md` §4.6 lists as still-present: *"parallel replay clobbers cached -`time` on all non-executed tests — `mergeWorkerReplayPartials()` takes `$result['time']` verbatim, -never routing through `resultTime()`"*. The repo fixture disagrees: after a parallel replay the -sentinelled `time=9.999` / `assertions=42` survive on every non-executed test. - -**Measured.** `a parallel run merges worker results into the parent baseline` edits one test file, -then runs `--tia --parallel --processes=2`: `2 affected, 4 replayed, w=2`. If replayed times were -being clobbered, `w` would be `6`. - -**Why the record may be stale.** `flushWorkerReplay()` (`Tia.php:~1286`) already applies -`resultTime()` **worker-side** before writing the partial, so the parent's verbatim read is reading -values that were already corrected. - -**What to do.** Either find a shape where it still reproduces (the playground has 25 tests and real -timings; the fixture has 6 and may be too small), or confirm it is fixed and strike it from §4.6. -Add a direct row either way: - -```php -test('a parallel replay keeps the recorded time of tests that did not run', function (): void { - $project = Project::make('master'); - $project->seed('master'); - - $project->pest('--tia', '--parallel', '--processes=2'); - - $delta = $project->delta(); - - expect($delta->writtenCount())->toBe(0, $delta->summary()) - ->and($delta->isResultsOnly())->toBeTrue($delta->summary()); -})->skipOnWindows(); -``` - ---- - -## Part 4 — Reporting - -Per bug: **reproduced (yes/no)** with the measured delta, **fixed (yes/no/deferred)**, and the test -that now pins it. Close with: - -1. Whether all `tests/Features/Tia/*` are green on **both** `php84` (no pcov) and `php85` (pcov). -2. Which of B2–B5 still need Nuno's decision, phrased as a yes/no question each. -3. Whether `tests/.snapshots/success.txt` and the `tests/Visual/Parallel.php` tally now need - regenerating (they will, if you added rows) — **do not run `composer update:snapshots` unless - asked.** -4. Anything you found that is not in this file. - -Leave the tree uncommitted and the scratch probes out of the repo. diff --git a/PLAN_PHASE_SIX.md b/PLAN_PHASE_SIX.md deleted file mode 100644 index 1be4ea69..00000000 --- a/PLAN_PHASE_SIX.md +++ /dev/null @@ -1,386 +0,0 @@ -# TIA deep audit — phase six - -## Your task - -Phase five went looking rather than fixing what was reported, and found nine defects in the -**read/write path and the state dir**. All nine are fixed and pinned by rows in the repo. It stopped -there deliberately: roughly half of TIA was never opened. Phase six is that other half. - -Two deliverables, both required: - -1. **A ranked findings list.** Every finding needs a *reproduction*, not a reading of the code. A - finding you cannot reproduce is a hypothesis — say so and rank it separately. -2. **New scenario tests** in `tests/Features/Tia/*`. Rows that pass go in the repo (they are the - regression net). Rows that fail stay in your scratchpad until the fix lands — **never commit a red - test.** - -Work in passes, and **report between passes** rather than at the very end: - -- **Pass A** — reproduce the leads in Part 3. Report which are real. -- **Pass B** — your own hunt: the invariants in Part 2, attacked with inputs nobody tried. -- **Pass C** — fixes, smallest first, each with the row that pins it. - -Fix what is clearly a defect with an obvious correct answer. **Stop and ask** when the fix is a -behaviour *choice* — those are Nuno's calls, and a reproduction with a crisp yes/no question is worth -more than a guessed fix. - ---- - -## Part 1 — Where things stand - -### 1.1 What TIA is - -Test Impact Analysis. `pest --tia` records a dependency graph (test file → source files it touched) -plus a per-branch baseline of results, then on later runs replays the tests whose dependencies did not -change. State lives in a per-project dir under `~/.pest/tia/`; `graph.json` is the whole thing. - -Source of truth: `src/Plugins/Tia.php` (the plugin) and `src/Plugins/Tia/Graph.php` (the graph model + -read/write API). Supporting: `ChangedFiles.php` (git), `Fingerprint.php` (environment/structure -hashing), `Storage.php` / `FileState.php` (the state dir), `ResultCollector.php` (what a run observed), -`Recorder.php` + `CoverageCollector.php` (how edges are recorded). - -Read `PLAN.md`, then `PLAN_PHASE_TWO.md` → `THREE` → `FOUR` → `FIVE` for the history and the tier -contract. Struck-through rows are fixed; do not re-report them. **Section 1.2 below supersedes any -phase-five row that contradicts it.** - -### 1.2 What phase five settled — do not re-report these - -Nine defects, all fixed, each with a row that fails if it comes back: - -| # | Defect | Fix | Pinned by | -|---|---|---|---| -| 1 | `pruneStaleResults()` unset an entry; the next read layered the default branch's entry for the same test id back in, so a renamed/removed test stayed "previously unsuccessful" on a feature branch forever | `Graph::baselineFor()` layers per *file* once a branch has had a complete run (new `complete` flag on the baseline); a key minted by a narrowed run keeps the per-test-id merge | `StateReclamation` → *a pruned result does not come back from the fallback*, *the fallback still reaches a branch that has never run a test file* | -| 2 | A cached failure whose test file was deleted was never reclaimed, so `--filtered` degraded to a full replay on every later run | `hasUnlocatedTestsToRerun()` widens only for a path it cannot *address*; `Graph::pruneResultsForMissingFiles()` + `pruneMissingTests()` run on every complete write | `StateReclamation` → *a cached failure whose test file was deleted stops widening later runs*, *a complete run reclaims the entry and the edge of a deleted test file* | -| 3 | A detached HEAD is read-only for writes, but three paths still *deleted*: structural drift, `--fresh` (`Storage::purge`), and the corrupt-graph discard. The checkout that wiped the baseline could never rebuild it | `Tia::deleteState()` no-ops when detached; `Storage::purge` guarded | `StateReclamation` → the three *a detached HEAD does not purge…* rows | -| 4 | A status int outside 0–8 became `TestStatus::unknown()`, which `ReplayType` folded into `Failure` — a green test went red, exit 1 | `Graph::getResult()` returns `null` for an unknown status (re-run, don't replay); `shouldRerunStatus()` treats unknown as re-run | `HostileState` → *a cached status this build cannot interpret is re-run, not replayed* | -| 5 | Notice/deprecation/warning (3/4/6) decode fine but `ReplayType` had no case, so they also folded into `Failure` | Explicit `Pass` cases — those statuses only reach replay when the configured `failOn*` / `displayDetailsOn*` policies say they do not matter | `HostileState` → *a cached status with no replay of its own does not fail the run* | -| 6 | `Graph::decode()` took `baselines`, `edges` and `files` verbatim; one malformed entry raised a `TypeError` inside a test | `decodeBaselines()` / `decodeResults()` / `decodeEdges()` / `decodeFiles()` validate every field. Numeric-looking keys are cast, not filtered — a branch named `12345` decodes as an `int` | `HostileState` → *a graph whose shape is wrong everywhere is repaired rather than trusted* | -| 7 | Baseline keys grew forever — one full copy of the suite per branch ever created | `ChangedFiles::branchNames()` (local + remote refs) + `Graph::pruneMissingBranches()`, on complete writes only, and only when the fallback branch is visible in the refs | `BranchShapes` → *deleting many branches reclaims every one of their baselines* and the four rows around it | -| 8 | A test that triggered a deprecation was recorded as `status=0`, because PHPUnit emits `Passed` for it and TIA had no issue subscribers. A fresh `--fail-on-deprecation` run exited 1; the replayed one exited 0 | Six subscribers (`Notice`/`PhpNotice`/`Deprecation`/`PhpDeprecation`/`Warning`/`PhpWarning`) feed `ResultCollector`; most-important-status-wins; a plain `Passed` does not downgrade a triggered issue; `@`-suppressed issues are ignored | `IssueStatuses` → *a triggered issue is recorded as itself, not as a pass*, *a cached deprecation still fails the run that asked to fail on one* | -| 9 | A replay wrote back the status it *looked* like from outside, so a cached deprecation replaying as a pass was persisted as `0` — defect 8's fix eroded after one run | `Tia::replayedAsRecorded()` writes back the cached status and message for replayed tests, in the sequential path and in the worker flush | `IssueStatuses` → *replaying a cached issue does not downgrade it to a pass* | -| 10 | A run torn down mid-file (an `exit()` inside a test) still flushed what it had, and the parent read that as licence to prune the siblings it never reached. Sequential and parallel disagreed | `ResultCollector::hasUnfinishedTest()` demotes such a run to results-only, in `terminate()` (the shutdown path) and in `addOutput()` | `StateReclamation` → *a run torn down mid-file does not prune the tests it never reached* | - -**One existing assertion was changed.** `DefaultBranchWriteTier > filtered mode falls back to a full -replay when a cached failure cannot be located` used `tests/Unit/DeletedTest.php` — a path that -resolves but does not exist, which is precisely the shape behind defect 2. It now points at -`/build/agent/…`, so it still pins the widening safety net for the case where widening can help. If -you disagree with that reading, that is a finding, not a licence to change it back quietly. - -### 1.3 Attacked in phase five and found solid — do not spend time here again - -- **Parity.** Sequential vs `--processes=1/2/8` across replay-with-edit, `--filtered` with a cached - failure, a first run on a feature branch, `--bail`, `--stop-on-failure`, `--compact`: identical - tally *and* identical graph delta every time. -- **Hostile state dir.** Empty / truncated / not-JSON / JSON scalar / JSON list / `null` / `{}` / NUL - bytes; `graph.json` as a directory; a read-only state dir; dangling and negative edge ids; a result - `file` pointing outside the project; `schema: 2`. All degrade to "run the tests", exit 0. -- **Status int mapping.** `ResultCollector` (`asInt()`), `Graph::getResult()` and `TestStatus::from()` - agree exactly on 0–8. No off-by-one. -- **`FileState::write`** is tmp + rename, so concurrent runs cannot tear a file. Racing runs are - last-writer-wins **by design** — treat as a decision, not a bug, unless you can show data loss - worse than that. -- **Branch names.** Slashes, dots, unicode, digits-only, 180 characters, case-only differences, - remote-only branches, worktree branches: all keyed and reclaimed correctly. -- **`sha`/`tree` layering.** No shape found that under-reports changed files. The fallback `tree` only - drops a file whose current content hashes identically to what the fallback ran with, which is sound. - -### 1.4 Known and deliberately unfixed - -| Item | Why it is still open | -|---|---| -| Reclamation is skipped when a complete run executes **zero** tests | Stale edges linger until a run that executes at least one test. Writing the graph from a run that produced nothing costs more than it buys. Revisit if you find a real project stuck there. | -| SIGINT never reaches the suite (`PcovRestarter` re-exec) | Real, but signal handling outside the TIA read/write path. Needs its own pass. | -| Orphaned state dirs when the origin URL changes | `Storage::projectKey()` keys on the origin identity so clones share a graph. Changing the remote silently moves TIA to a fresh dir and nothing reclaims the old one. Unbounded growth in `~/.pest/tia`. | -| `environmental` fingerprint holds only `php_minor` | Nuno declined. The `clearResults()` drift path is therefore latent — it becomes live the moment that bucket grows. **Do not re-flag the `PHP_MAJOR_VERSION` line itself.** | -| PLAN.md §3 (raw coverage flags leave filtered mode on) | **No longer reproduces** — `coverageReportActive()` consults `COVERAGE_REPORT_FLAGS` now. Struck. | - ---- - -## Part 2 — The invariants to attack - -Same list as phase five; 1, 2, 4, 5, 7 and 8 are now well covered, so weight your effort toward 3 and -6 and toward the *recording* half of the system, which nothing below the plugin has ever probed. - -1. **Parity.** `pest ` and `pest --parallel --processes=N ` must leave the same graph and - reach the same tally. Hard rule. `Project::SEQUENTIAL_AND_PARALLEL` encodes it; use it on every new - write-path row. -2. **The tiers hold.** COMPLETE / RESULTS-ONLY / HARD-SUPPRESSED, exactly one each, no leaking. -3. **Replay is faithful.** ← *weak spot.* A replayed test reports the same status, message, time and - assertion count as the recorded run. Phase five fixed statuses; **edges** are unproven. -4. **Reads never write.** -5. **A branch never corrupts another branch's baseline.** -6. **Nothing is unbounded.** ← *weak spot.* Baseline keys are reclaimed now; `files`, `edges`, worker - partials, coverage caches and orphaned state dirs are not. -7. **A hostile state dir cannot break a run.** -8. **Git shapes are all handled.** - ---- - -## Part 3 — Leads to reproduce first (Pass A) - -Ranked by expected value. Everything here is **unexplored**, not merely unfixed — phase five never -opened these files. - -### M1 — the recording path with a real coverage driver · **highest value** - -Everything phase five did was driver-independent by design (seed a graph, never record one). Nothing -verified that a *recorded* graph is correct. This is the biggest blind spot in the audit. - -- Do the recorded edges match what the test actually touched? Record with pcov, then hand-check - `edges` against the source files each fixture test uses. -- `PLAN.md` §4: **`pest --tia --coverage` narrows edges** — `Feature/ExampleTest` recorded with 2 - files instead of 16, dropping self-edges. Same observable shape as the parallel bug G12 that phase - four closed, likely a different mechanism (the piggyback collector is scoped by `phpunit.xml - `, the pcov-restarted recorder is not). **Confirm whether these are one fix or two.** -- `Recorder::activateLinkTracking()` (piggyback) vs `activate()` (pcov restart) must produce the same - edge set for the same suite. Compare them directly. -- `keepExisting: $this->piggybackCoverage` in `replaceEdges()` — what happens to a test whose edges - genuinely shrank while piggybacking? - -Rows for this **must** be `->skipOnPhpVersionsWithoutCoverage()`-style guarded, or seeded, or CI goes -red on `php84`. That constraint is why phase five skipped it; solve it deliberately rather than by -accident. Adding a coverage-driver guard helper to the fixture is a legitimate deliverable. - -### M2 — `BaselineSync` (621 lines, never opened) - -The remote-baseline fetch is the only path where a graph arrives from **another machine**, which is -exactly where the hostile-state work matters most and where none of it has been exercised. - -- A fetched baseline whose `fingerprint` matches but whose `files`/`edges` describe a different tree. -- A fetched baseline recorded on a branch this checkout does not have. -- `fetchIfAvailable()` under a broken network, a 404, a truncated download, a non-gzip body. -- `KEY_FETCH_COOLDOWN` — does it bound retries, and does a corrupt cooldown file break a run? -- Interaction with defect 7's branch GC: a fetched baseline carries branch keys this clone has never - heard of. **They will be pruned on the next complete write.** Is that right, or must fetched keys be - exempt? This is a real question, answer it. - -### M3 — selection paths nobody has probed - -`Graph::affected()` is ~600 lines and phase five only exercised the plain PHP-edge path. - -- **Migrations** → `TableExtractor` → `testTables` intersection. What happens with an unparseable - migration, a migration that drops a table, a squashed schema dump? -- **Blade** — `bladeAncestorsFor()` walks `@include`/`@extends`/`` transitively. Cycles? - Depth? A component referenced only dynamically? -- **Inertia** — `componentForInertiaPage()`, `jsFileToComponents`, `JsModuleGraph::buildStrict()`. - What if `vite` is missing, or the resolver returns garbage? -- **`usesSiblingHeuristicForUnknownPhp()`** — a hard-coded list of Laravel directories. A changed file - in `app/Providers/` widens to every test whose deps share that directory. Measure how much that - over-selects on a real tree. -- **Arch tests** — `testSourceDeclaresArchGroup()` greps the source with three regexes. False - positives (the string `arch(` in a comment) select the file on *every* PHP source change. - -### M4 — git shapes phase five left alone - -- A repo with **no commits yet** (`currentSha()` returns null / git fails). -- **Submodules** — a changed file inside one; `git status --porcelain` reports the submodule path. -- A repo whose root is **above** the pest project — `TiaRequiresRepositoryRoot` panics deliberately; - confirm it still panics and writes nothing. -- **Shallow / single-branch CI checkouts.** Defect 7's guard (`fallbackBranch` must be visible in the - refs) was reasoned about, not measured. Build one and check nothing is over-pruned. -- A branch **behind** the recorded sha, so `merge-base --is-ancestor` fails and the graph is declared - unreachable. Does it recover, or thrash? - -### M5 — a real concurrent race - -Phase five established `FileState::write` is atomic and called it last-writer-wins by design. Nobody -launched two runs. Launch them: a watcher plus a manual run, two `--tia` processes on one project, -`--parallel` where the parent writes while a straggler worker flushes. Look for **loss worse than -last-writer-wins** — a partially-merged baseline, a pruned entry from a run that never saw the file, -worker partials from run A consumed by run B (`KEY_WORKER_*` are not namespaced per run). - -That last one is the sharpest: `purgeWorkerPartials()` deletes *all* partials by prefix, so two -concurrent parallel runs will eat each other's. Probe it. - -### M6 — the unbounded remainder - -Defect 7 reclaimed baseline keys. These still grow: - -- `files` and `edges` — a deleted *source* file's entry is never removed (`pruneMissingTests()` only - covers test files). Every rename leaves an orphan id forever. -- Orphaned state dirs under `~/.pest/tia` (see 1.4). -- `KEY_COVERAGE_CACHE` / `KEY_COVERAGE_MARKER` — who deletes them, and when? -- Worker partials when a worker dies before `terminate()`. - -Measure the growth on a realistic tree before proposing anything. Then **ask** — a cap or a GC is a -behaviour choice. - ---- - -## Part 4 — The harness - -### 4.1 The scenario suite - -`tests/Features/Tia/*` scaffold a throwaway git project into a temp dir, run a **real `pest` -subprocess** against it, and diff the graph it wrote. Everything lives in `tests/Fixtures/Tia/`: - -| Class | What it gives you | -|---|---| -| `Project` | `make(branch, overlay:)`, `withoutGit()`, `seed(branch, sentinel:, failing:)`, `seedFor($root, …)`, `pest(...$args)`, `pestWithEnvironment($dir, $env, ...$args)`, `pestIn($dir, ...)`, `write($rel, $contents)`, `path($rel)`, `graph()`, `branchKeys()`, `graphDir()`, `graphExists()`, `snapshot()`, `delta()`, `mutateGraph(fn)`, `addBaseline($branch)`, `worktree($branch)`, `destroy()`, `destroyAll()`, `SEQUENTIAL_AND_PARALLEL` | -| `GitRepo` (`$project->git()`) | `switchTo($b, new:)`, `rename($from, $to)`, `detach()`, `commit($msg)`, `config($k, $v)`, `addOrigin()`, `removeOrigin()`, `setOriginHead($b)`, `unsetOriginHead()`, `worktree()`, `sha()`, `branchNames()`, `run([...])` for anything else | -| `GraphDelta` (`$project->delta()`) | `writtenCount()`, `added()`, `removed()`, `branchKeys()`, `baselineUntouched($b)`, `shaMoved()`, `treeMoved()`, `edgesMoved()`, `filesMoved()`, `fingerprintMoved()`, `structureMoved()`, `isResultsOnly()`, `isHardSuppressed()`, `summary()` | -| `PestResult` (returned by `pest()`) | `replayed()`, `uncached()`, `affected()`, `tally()`, `output`, `exitCode`, `describe()` | - -The fixture app is 3 test files / **6 tests** (`Project::TOTAL_TESTS`); `Project::EDGES` and -`Project::TESTS` describe the graph `seed()` writes. `Project::testId($file, $description)` builds a -result key. Overlays in `tests/Fixtures/Tia/overlays//` supply a different `tests/Pest.php` — -that is how you configure `pest()->tia()->…` for a scenario. - -**If the fixture cannot express your case, extend the fixture.** A new `GitRepo` verb, a new overlay, -or a coverage-driver guard is a legitimate part of the deliverable. Do not water down a scenario to -fit the current helpers. Phase five added `GitRepo::commit()` usage, numeric-key handling in -`Project::branchKeys()` and `GraphDelta`, and used `$project->write()` to author test files inline — -follow that pattern. - -### 4.2 The sentinel discriminator — read before writing any assertion - -`seed()` writes a graph **and then rewrites every cached result to `time=9.999`, `assertions=42`** -(only where assertions are non-zero — risky/skipped/incomplete statuses are *derived* from "performed -no assertions", so falsifying those would rewrite the status on replay), then snapshots. Therefore: - -- `$delta->writtenCount()` is the only reliable way to tell **"replayed"** from **"executed and wrote - back the same values"**. `0` means nothing was written. -- `isResultsOnly()` = no structure moved, no `sha`/`tree` movement, nothing added or removed. -- `isHardSuppressed()` = the graph is byte-identical. -- A replayed suite reports inflated assertion counts (6 tests × 42). That is the sentinel, not a bug. - -The three write tiers, unchanged since phase two: - -- **COMPLETE** — may change everything. -- **RESULTS-ONLY** — may change only `baselines[].results` for tests that ran; never removes - an entry, never adds a result for a test file absent from `edges`, never touches - `sha`/`tree`/`edges`/`files`/`fingerprint`. -- **HARD-SUPPRESSED** — may change nothing. - -One addition from phase five: a complete run on a **non-default** branch also sets -`baselines[].complete = true`. It is deliberately *not* set on the default branch, so a clean -green run there stays byte-identical. - -### 4.3 Running them - -They are in the `integration` group (`tests/Pest.php`), so `composer test:unit` skips them. **A -directory argument finds nothing** — pass files, space-separated, as separate argv entries: - -```bash -PAO_DISABLE=1 php84 bin/pest tests/Features/Tia/BranchShapes.php tests/Features/Tia/CompleteRunWriteTier.php \ - tests/Features/Tia/DefaultBranchReplay.php tests/Features/Tia/DefaultBranchResolution.php \ - tests/Features/Tia/DefaultBranchWriteTier.php tests/Features/Tia/FilteredMode.php \ - tests/Features/Tia/HostileState.php tests/Features/Tia/IssueStatuses.php \ - tests/Features/Tia/PartialRunWriteTier.php tests/Features/Tia/StateReclamation.php - -PAO_DISABLE=1 php bin/pest -``` - -**Both interpreters must be green.** `php84` is 8.4.x with **no pcov** — that is what CI has -(`.github/workflows/tests.yml` sets `coverage: none`). `php` is 8.5.x with pcov — that is the dev -machine. Any assertion that depends on a coverage driver passes locally and fails in CI. Concretely: - -- A **cold recording run writes no graph at all** without pcov/xdebug (it prints `Running in TIA mode, - however TIA is skipped as it needs ext-pcov or Xdebug`). **Seed a graph, never record one**, unless - the driver *is* the point of the row — see M1, which has to solve this properly. -- A **PHP source file edit** driverless triggers `Detected PHP source changes but no coverage driver - is available` → full suite, `affected=0`. Edit *test* files, not `app/` files. -- The `terminate()` path differs by driver: with pcov the plugin reaches the complete write through - the shutdown handler, without it the run exits earlier. Defect 10 only reproduced on `php`. **Run - every probe on both before believing it.** - -`PAO_DISABLE=1` is mandatory on every pest invocation and every probe: `laravel/pao` emits JSON under -agents and corrupts the captured output. - -**Baseline: 161 scenario rows green on both interpreters**, at `b49ba062`. Per file: - -| File | Rows | -|---|---| -| `StateReclamation.php` | 37 | -| `HostileState.php` | 25 | -| `CompleteRunWriteTier.php` | 17 | -| `DefaultBranchResolution.php` | 14 | -| `BranchShapes.php` | 14 | -| `DefaultBranchReplay.php` | 13 | -| `IssueStatuses.php` | 13 | -| `PartialRunWriteTier.php` | 12 | -| `DefaultBranchWriteTier.php` | 10 | -| `FilteredMode.php` | 6 | - -Plus 80 unit/arch rows (`tests/Unit/Plugins/Tia/*`, `tests/Arch.php`). If those numbers do not -reproduce, stop and say so. - -### 4.4 Measure before you assert - -Never guess an expectation from reading the code. Probe first, in your scratchpad: - -```php -seed('master'); -$p->git()->switchTo('feature-x', new: true); -$r = $p->pest('--tia'); -printf("tally=[%s] keys=[%s] %s\n", $r->tally(), implode(',', $p->branchKeys()), $p->delta()->summary()); -$p->destroy(); -``` - -`PAO_DISABLE=1 php probe.php`. One project per scenario; always `destroy()`; `Project::destroyAll()` -at the end. Run every probe on **both** interpreters before believing it. - ---- - -## Part 5 — Ground rules - -- **Do not** run `composer test`. It takes minutes and you do not need it. -- **`tests/.snapshots/success.txt` and the tally in `tests/Visual/Parallel.php` are stale right now** — - phase five added 89 rows and did not regenerate them. **Report that they need regenerating and let - Nuno run `composer update:snapshots`.** Do not run it yourself. -- **Do not commit.** Leave the tree dirty. -- **Do not touch the playground's `vendor/`.** If a finding truly needs the playground (25 real tests, - real timings), say so and ask — syncing it is a manual step Nuno owns. -- Run `vendor/bin/phpstan analyse --memory-limit=-1 --no-progress` and - `vendor/bin/pint ` before reporting. **Scope both to files you changed** — - Nuno edits `src/` live, and a repo-wide fixer will revert his work. -- The fixture projects **hardlink `src/`**. If you edit `src/` while a scenario run is in flight, - those rows go red for no reason. Finish the edit, then run. -- Keep scratch probes out of the repo. -- **Never weaken an existing assertion to make something pass.** If an existing row contradicts your - fix, that is a finding: report the contradiction and ask. Phase five hit this once (see 1.2) and - rewrote the row to pin the *narrower* contract rather than deleting it — that is the bar. - ---- - -## Part 6 — Reporting - -**Between passes**, not just at the end. Per finding: - -- **Reproduced (yes / no / hypothesis only)**, with the exact command and the measured - `tally` + `delta()->summary()` on **both** interpreters. -- **Severity**, and say why in one line. The scale that matters here: *a test wrongly replayed as - passing* (worst) > *cache silently useless, full suite forever* > *graph grows / stale data* > - *cosmetic*. -- **Fixed / deferred / needs-a-decision**, and the row that pins it. -- For anything needing a decision: **one yes/no question**, no essay. - -Close with: - -1. The count of `tests/Features/Tia/*` green on **both** `php84` (no pcov) and `php` (pcov), against - the 161 baseline. -2. Every row you added, and what invariant from Part 2 it defends. -3. Which findings are still open, as yes/no questions. -4. That the snapshots need regenerating — **do not regenerate them.** -5. **What you looked at and found solid.** A list of attacks that did not break anything is a real - result: it tells the next phase where not to spend its time. - ---- - -## Part 7 — Open questions carried into this phase - -Answer these before or during Pass C; they change what the fixes should be. - -1. A fetched remote baseline carries branch keys this clone has never heard of, and the branch GC will - prune them on the next complete write. **Should fetched keys be exempt?** -2. Two concurrent parallel runs share the `worker-edges-*` / `worker-results-*` prefixes and - `purgeWorkerPartials()` deletes by prefix. **Should partials be namespaced per run, or is "do not - run two TIA suites at once" the contract?** -3. `files` and `edges` never lose a deleted *source* file. **Cap, GC, or accept?** -4. Orphaned state dirs accumulate under `~/.pest/tia` whenever a project's origin URL changes. - **Reclaim them, or accept?** diff --git a/PLAN_PHASE_THREE.md b/PLAN_PHASE_THREE.md deleted file mode 100644 index e607b211..00000000 --- a/PLAN_PHASE_THREE.md +++ /dev/null @@ -1,455 +0,0 @@ -# TIA default-branch fallback — phase three - -## Your task - -Fix [pestphp/pest#1823](https://github.com/pestphp/pest/issues/1823) — TIA's cached results never -hit for repos whose default branch is not literally `main` — then re-run the **whole** conformance -matrix against the playground app, plus the new section **L** that covers the fix. - -Work in this order, and **stop where the plan says stop**: - -1. **Part 1** — read the diagnosis. It is already measured; do not re-derive it, but do re-confirm - the two reproductions in Part 1.3 take ~2 minutes and prove your environment is sane. -2. **Part 2** — implement the fix in the `pestphp/pest` repo. **Do not commit. Do not touch the - playground's `vendor/`.** -3. **HARD STOP → Part 3.** Report the diff to Nuno and wait. He validates, commits, and syncs it - into the playground himself. You must not proceed until he confirms. -4. **Part 4** — verify the sync landed, then run section **L** (new) and the full phase-two matrix - (**A–K**, all 156 rows) against the playground. -5. **Part 5** — report in the given format. - -Per `CLAUDE.md`: **do not write new `pestphp/pest` unit tests and do not run `composer test`.** Make -the change, report it, and ask whether repo tests should be added. The section-L rows are *playground -invocations*, not repo tests — those are the deliverable and are always in scope. - ---- - -## Part 1 — The diagnosis (already measured; commit `bfd5b756`) - -### 1.1 Root cause — two independent hardcoded `'main'` literals - -**(a) The read fallback.** `src/Plugins/Tia/Graph.php` — seven methods default the fallback to the -literal `'main'`: - -| line | method | -|---|---| -| 579 | `recordedAtSha(string $branch, string $fallbackBranch = 'main')` | -| 614 | `getAssertions(…, string $fallbackBranch = 'main')` | -| 625 | `getTime(…, string $fallbackBranch = 'main')` | -| 636 | `getResult(…, string $fallbackBranch = 'main')` | -| 663 | `testFilesToRerun(string $branch, string $fallbackBranch = 'main')` | -| 700 | `hasUnlocatedTestsToRerun(string $branch, string $fallbackBranch = 'main')` | -| 811 | `lastRunTree(string $branch, string $fallbackBranch = 'main')` | - -They all funnel into `Graph::baselineFor()` (line 819), which *does* implement a real cross-branch -fallback: - -```php -if (isset($this->baselines[$branch])) return $this->baselines[$branch]; -if ($branch !== $fallbackBranch && isset($this->baselines[$fallbackBranch])) return $this->baselines[$fallbackBranch]; -return ['sha' => null, 'tree' => [], 'results' => []]; -``` - -The mechanism is deliberate. The problem is that **no caller ever passes `$fallbackBranch`** — all -nine call sites in `src/Plugins/Tia.php` (lines 419, 429, 432, 785, 857, 1026, 1031, 1048, 1056) pass -only `$this->branch`. So on a `master`-named repo the second branch of `baselineFor()` can never -fire, and the whole mechanism is dead code. - -`grep -rniE 'defaultBranch|symbolic-ref|init\.defaultBranch|origin/HEAD' src/` returns **nothing** — -no default-branch resolution exists anywhere. - -**(b) The detached-HEAD default.** `src/Plugins/Tia.php:206` declares `private string $branch = 'main';` -and `ChangedFiles::currentBranch()` (`ChangedFiles.php:208`) returns `null` for detached HEAD. So on -detached HEAD `$this->branch` stays the literal `'main'` and is used for **both reads and writes** — -minting a baseline key for a branch that does not exist. This is a *write*-side bug and a separate fix -from (a). - -### 1.2 Not a regression - -`git log -S"fallbackBranch = 'main'"` bottoms out at `c7e32f5d feat(tia): continues to work on poc`. -This is original PoC code, untouched by phase one. Phase one's change 1 modified -`hasUnlocatedTestsToRerun()`'s file-existence check — one of the seven methods — without going near -the fallback. Do not report it as a phase-one regression. - -No test anywhere exercises the mechanism: `tests/Unit/Plugins/Tia/Graph.php` uses `'main'` as the -*actual* branch name, so those assertions pass whether or not the fallback exists. A branch-name -mismatch is never tested. - -### 1.3 The two reproductions (re-confirm these before you start) - -Both on the playground, `master`-named default branch, zero local changes: - -``` -default = master default = main -1. record on default → full run (cold) → full run (cold) -2. 1st run on feature-x → 25 UNCACHED → 25 replayed ← (a) -3. 2nd run on feature-x → 25 replayed → 25 replayed -4. back on default → 25 replayed → 25 replayed -5. 1st run on feature-y → 25 UNCACHED → 25 replayed -``` - -``` -master-only graph, then `git checkout --detach`, then `pest --tia`: - → 25 uncached, and keys become [master,main] ← (b) spurious key -``` - -The cost is **one full run per new branch, forever**, with no output explaining why — the only clue -is the `N uncached` count; the headline is just `─ Experimental TIA mode enabled.` - -### 1.4 Correction to phase two - -Phase two reported **H6 and H7 as passing. Both were false passes.** They ran after H5, which had -renamed `master`→`main` and left a `main` key in the graph, so the hardcoded fallback resolved by -accident. Re-measured against a clean `master`-only graph, both fail. Section L replaces them as the -load-bearing rows; H6/H7 must be re-run **from a cold graph** this time (see Part 4.2). - ---- - -## Part 2 — The fix to implement - -### 2.1 Recommended design - -**Step 1 — add a non-throwing resolver** to `src/Plugins/Tia/ChangedFiles.php`, next to -`currentBranch()`: - -```php -public function defaultBranch(): ?string -``` - -Resolution order, each step failing soft to the next: - -1. `git symbolic-ref --short refs/remotes/origin/HEAD` → strip a leading `origin/` -2. `git config --get init.defaultBranch` -3. `null` - -Unlike `currentBranch()`, this must **never throw** `MissingDependency` — it is advisory. Return -`null` on any non-zero exit or empty output. - -**Step 2 — add a config surface.** `src/Plugins/Tia/Configuration.php` already exposes `always()`, -`locally()`, `filtered()`, `baselined()`, `watch()`. Add: - -```php -public function defaultBranch(string $branch): self -``` - -so `pest()->tia()->defaultBranch('master')` works in `tests/Pest.php`. Explicit config **always -wins** over autodetection — that is the escape hatch when `origin/HEAD` is unset. - -**Step 3 — resolve once, in `Tia.php`.** The read path is hot (`getResult()` is called per test at -line 419), so resolution must not shell out per call. Resolve alongside `$this->branch` at -`Tia.php:1910`, under the existing `$branchResolved` guard: - -```php -$this->fallbackBranch = $configuredDefaultBranch - ?? $changedFiles->defaultBranch() - ?? 'main'; -``` - -**Step 4 — thread it into `Graph`.** Prefer a `Graph`-level property over editing nine call sites: -add `Graph::setFallbackBranch(string $branch)`, change the seven signatures to -`?string $fallbackBranch = null`, and resolve inside each with -`$fallbackBranch ??= $this->fallbackBranch;`. `baselineFor()` itself needs no change. This keeps the -public signatures backward-compatible and minimises blast radius. - -**Step 5 — fix the detached-HEAD write.** `Tia.php:206`'s `= 'main'` default must become the -resolved default branch, so detached HEAD stops minting a phantom key. - -### 2.2 Invariants the fix must not break - -These are all covered by existing matrix rows — the fix is wrong if any of them moves: - -- **Read-only.** The fallback must affect *reads* only. Writes go through `ensureBaseline($branch)` - and must keep using the real current branch. Otherwise H1–H4/H8 ("no baseline key other than the - real branch") break. -- **H9** — a non-git dir with `--tia` must still raise - `MissingDependency: The feature "Tia mode" requires "git".` Adding a soft resolver must not - swallow that. -- **H10** — plain `pest` in a non-git dir must still run and create no baseline dir. -- **A2/A3** — cold-graph recording unchanged. -- **I1/E3** — clean+green `--tia --filtered` must still be a true zero-delta run. -- Filtered mode reads `testFilesToRerun()` and `hasUnlocatedTestsToRerun()`, so the fallback must - reach those two as well, not just `getResult()`. - -### 2.3 Decisions for Nuno (raise these at the Part 3 stop) - -- **D1** — Consult `origin/HEAD` at all? It requires `git remote set-head` and is absent in many CI - checkouts and all remote-less repos. Config + `init.defaultBranch` only is simpler but helps fewer - people out of the box. *Recommendation: keep it, first in the chain, since it fails soft.* -- **D2** — Single-key heuristic: if the graph holds exactly one baseline key, use it as the fallback? - Fixes the issue with zero git calls, but is implicit and surprising when several keys exist. - *Recommendation: no.* -- **D3** — Should detached HEAD write a baseline at all, or be read-only? Current behaviour mints a - key. *Recommendation: read-only.* -- **D4** — Should `pest()->tia()->defaultBranch()` validate that the branch exists, or accept any - string? *Recommendation: accept any string; a nonexistent name degrades to a full run, which is - safe.* - ---- - -## Part 3 — HARD STOP - -When the code is written: - -1. Show Nuno the diff (`git -C /Users/nunomaduro/Work/projects/pestphp/pest diff`) and a one-paragraph - summary of each file's change. -2. Answer/raise the D1–D4 decisions. -3. State explicitly that you have **not** committed and have **not** synced `vendor/`. -4. Ask whether repo unit tests should be added (per `CLAUDE.md`), describing the tests you have in - mind — do not write them yet. -5. **Wait.** Nuno commits and applies the change to the playground. - -**Never sync the playground's `vendor/` yourself.** `vendor/pestphp/pest` there is a dist copy, not a -symlink (composer installed `dev-fix/tia-filtered as 5.2.0`), so pest-repo edits do not reach it. Say -what is stale and wait. This applies to before/after contrasts too. - -Once he confirms, verify the sync actually landed before measuring anything: - -```bash -V=/Users/nunomaduro/Work/projects/playground/laravel/vendor/pestphp/pest -grep -c 'defaultBranch' "$V/src/Plugins/Tia/ChangedFiles.php" # must be ≥ 1 -for f in src/Plugins/Tia.php src/Plugins/Tia/Graph.php src/Plugins/Tia/ChangedFiles.php \ - src/Plugins/Tia/Configuration.php; do - diff -q "/Users/nunomaduro/Work/projects/pestphp/pest/$f" "$V/$f" >/dev/null \ - && echo "SAME $f" || echo "STALE $f" -done -``` - -If anything reports `STALE`, stop and tell him. State in your final report which pest commit produced -the playground numbers. - ---- - -## Part 4 — Measurement - -### 4.1 Environment and traps - -**Playground:** `/Users/nunomaduro/Work/projects/playground/laravel` (branch `master`). - -1. **Every** pest invocation needs `PAO_DISABLE=1` — the app has `laravel/pao`, which emits JSON when - it detects an agent. -2. **Pin the interpreter to `php85`.** The playground requires PHP `>= 8.4.1` *and* pcov. Locally only - `php85` (8.5.8) has both — `php84` (8.4.23) has no pcov, and the bare Herd `php` shim has been - observed drifting to 8.3.32 mid-session, which kills every run in - `vendor/composer/platform_check.php`. Also put a `php` → `php85` symlink first on `PATH`: the - `--shard` list-tests probe spawns a subprocess via bare `php`, not `PHP_BINARY`. -3. **Never run `git checkout .` or `git checkout -- composer.lock` in the playground.** It has four - pre-existing user-modified files — `AGENTS.md`, `CLAUDE.md`, `composer.json`, `composer.lock`. Scope - resets to `git checkout -- tests app` / `git clean -fd tests app`. For `phpunit.xml` and - `composer.lock`, copy aside and copy back, verifying with `shasum`. -4. **zsh does not word-split unquoted parameters.** A `$PEST` string containing a space becomes one - command name. Route every invocation through `eval` (the `pest()` helper below does this). -5. Comment-only edits to a test file are **not** changes (AST-level hashing). Use semantic edits. -6. **The sentinel technique must not falsify `assertions` on zero-assertion tests.** A - risky/skipped/incomplete status is *derived* from "performed no assertions", so patching those to - `42` rewrites the status on replay and destroys the discriminator — it shows up as a phantom - `status 5→0` defect. The `sentinel.php` below only patches `assertions` where it is already - non-zero. With that, a full replay gives a clean `rewritten=0`. -7. Restore branch state after every L row. Several rows rename or detach; leaving a stray `main` key - in the graph is exactly what produced phase two's false H6/H7 passes. - -### 4.2 Reference numbers - -The playground already carries the phase-two fixtures at commit `fb2e77e` — **do not rebuild them.** -A healthy sequential `--tia` graph is: - -> **`files=27`, 10 edge keys (one per test file), self-edge on all 10, `n=25` results.** - -`sha` will differ once Nuno commits the vendor sync — re-derive it once and use it throughout. The -`files`/`edges`/`n` numbers hold as long as no test fixture changes. Suite shape: 10 test files, 25 -tests, including six deliberate status fixtures (skipped, todo, incomplete, risky, warning, -deprecation), a 3-row dataset, a `smoke` group, an env-driven flaky test (green unless -`FLAKY_FAIL=1`), and the annotation set (`covers`/`note`/`flaky`/`issue`/`pr`/`ticket`/`assignee`). - -**Re-run H6 and H7 from a cold graph** (`rm -rf` the graph dir, record on `master` only, *then* -branch/detach). Their phase-two results are void. - -### 4.3 Harness - -Write these to your scratchpad. `$SP` is your own scratchpad dir. - -
-lib.sh - -```bash -#!/bin/zsh -export PAO_DISABLE=1 -PG=/Users/nunomaduro/Work/projects/playground/laravel -SP="" -PHPBIN="php85" -PEST="$PHPBIN $PG/vendor/bin/pest" -cd "$PG" || exit 1 -pest() { eval "$PEST $*"; } # zsh: no word-splitting, must eval -GRAPHDIR="$(pest --baseline)" -GRAPH="$GRAPHDIR/graph.json" -mkdir -p "$SP/bin" && ln -sf "$(command -v php85)" "$SP/bin/php" -export PATH="$SP/bin:$PATH" # --shard spawns bare `php` -reset_tree() { git checkout -- tests app 2>/dev/null; git clean -qfd tests app 2>/dev/null; } -seed() { rm -rf "$GRAPHDIR"; pest --tia >/dev/null 2>&1; $PHPBIN "$SP/sentinel.php" "$GRAPH" >/dev/null; cp "$GRAPH" "$SP/before.json"; } -snap() { cp "$GRAPH" "$SP/before.json"; } -delta() { $PHPBIN "$SP/cmp.php" "$SP/before.json" "$GRAPH"; } -keys() { $PHPBIN -r '$g=json_decode(file_get_contents($argv[1]),true);echo "[".implode(",",array_keys($g["baselines"]??[]))."]";' "$GRAPH"; } -tally() { sed -E $'s/\x1b\\[[0-9;]*[a-zA-Z]//g' <"$SP/out.txt" | grep -E 'Tests:' | sed -E 's/^ +//;s/Tests: +//'; } -``` -
- -
-sentinel.php — the write discriminator - -```php - -$p = $argv[1]; -$g = json_decode((string) file_get_contents($p), true, 512, JSON_THROW_ON_ERROR); -$n = 0; -foreach ($g['baselines'] ?? [] as $br => $b) { - foreach (array_keys($b['results'] ?? []) as $id) { - $g['baselines'][$br]['results'][$id]['time'] = 9.999; - // Only falsify a non-zero assertion count: risky/skipped/incomplete are - // DERIVED from "performed no assertions", so patching those to 42 would - // rewrite the status on replay and destroy the discriminator. - if ((int) ($b['results'][$id]['assertions'] ?? 0) > 0) { - $g['baselines'][$br]['results'][$id]['assertions'] = 42; - } - $n++; - } -} -file_put_contents($p, json_encode($g, JSON_THROW_ON_ERROR)); -echo "sentinelled $n results\n"; -``` -
- -
-oneline.php — one compact tier verdict per row - -```php - -function load(string $p): ?array { - return is_file($p) ? json_decode((string) file_get_contents($p), true, 512, JSON_THROW_ON_ERROR) : null; -} -$a = load($argv[1]); $b = load($argv[2]); -if ($a === null || $b === null) { echo 'GRAPH '.($b === null ? 'DELETED' : 'CREATED'); exit; } -function edgeSets(array $g): array { - $out = []; - foreach ($g['edges'] ?? [] as $t => $ids) { $s = array_map(fn($i) => $g['files'][$i] ?? "?$i", (array) $ids); sort($s); $out[$t] = $s; } - ksort($out); return $out; -} -$moved = []; -if (edgeSets($a) !== edgeSets($b)) $moved[] = 'edges'; -if (($a['files'] ?? []) !== ($b['files'] ?? [])) $moved[] = 'files'; -if (($a['fingerprint'] ?? null) !== ($b['fingerprint'] ?? null)) $moved[] = 'fingerprint'; -$brA = array_keys($a['baselines'] ?? []); $brB = array_keys($b['baselines'] ?? []); -if ($brA !== $brB) $moved[] = 'branchkeys('.implode('|', $brA).'->'.implode('|', $brB).')'; -$add = $rem = $wr = 0; $shaMoved = $treeMoved = false; -foreach ($brB as $br) { - $ra = $a['baselines'][$br]['results'] ?? []; $rb = $b['baselines'][$br]['results'] ?? []; - if (($a['baselines'][$br]['sha'] ?? null) !== ($b['baselines'][$br]['sha'] ?? null)) $shaMoved = true; - if (($a['baselines'][$br]['tree'] ?? null) !== ($b['baselines'][$br]['tree'] ?? null)) $treeMoved = true; - $add += count(array_diff(array_keys($rb), array_keys($ra))); - $rem += count(array_diff(array_keys($ra), array_keys($rb))); - foreach ($ra as $id => $x) { - if (! isset($rb[$id])) continue; - foreach (['status','time','assertions','message'] as $f) { - if (($x[$f] ?? null) !== ($rb[$id][$f] ?? null)) { $wr++; break; } - } - } -} -$n = 0; foreach ($brB as $br) $n = max($n, count($b['baselines'][$br]['results'] ?? [])); -printf('n=%d w=%-2d +%d -%d %s%s%s', $n, $wr, $add, $rem, - $moved === [] ? 'struct:ok' : 'STRUCT:'.implode(',', $moved), - $shaMoved ? ' sha:CHANGED' : '', $treeMoved ? ' tree:chg' : ''); -``` -
- -`cmp.php` (verbose per-entry version of the same) and `summarise.php` / `edgediff.php` are in -`PLAN_PHASE_TWO.md` §"Graph summariser" — reuse them for drill-downs. Reading the verdict: - -- `w=` — entries actually **written**. Under sentinel patching this is the only reliable way to tell - "wrote identical values" from "wrote nothing". A full replay must give `w=0`. -- `struct:ok` + `+0 -0` — no prune, no edges/files/fingerprint movement. Required by RESULTS-ONLY. -- `STRUCT:branchkeys(...)` — a new baseline key appeared. For section L this is the headline signal. - -### 4.4 Section L — new rows for this fix - -Tiers, unchanged from phase two: **COMPLETE** may change everything · **RESULTS-ONLY (RO)** may -change only `baselines[].results` for tests that ran, and must never remove an entry, add a -result for a test file absent from `edges`, or alter `sha`/`tree`/`edges`/`files`/`fingerprint` · -**HARD-SUPPRESSED** may change nothing. - -Every L row starts from a **cold graph recorded on the named default branch only** — verify -`keys=[]` before branching. Restore branch state afterwards. - -| # | Case | Target outcome | -|---|---|---| -| L1 | default `master`, record, `git switch -c feature-x`, `pest --tia`, zero changes | **all 25 replayed** (`w=0`), not `25 uncached`. The headline fix. | -| L2 | as L1 but default `main` | still all replayed — regression guard, this already worked | -| L3 | default `trunk`, then `develop` | replayed for both; the fix must not special-case two names | -| L4 | L1, then a *second* new branch `feature-y` | replayed too — the toll must not return per branch | -| L5 | L1 then edit `app/Services/Calculator.php` on `feature-x` | narrows to the 2 affected files (`CalculatorTest` + `AnnotationsTest`, which `covers` it); the other 17 replay | -| L6 | L1 with `--tia --filtered` | filtered mode reads the fallback too (`testFilesToRerun`, `hasUnlocatedTestsToRerun`) → `No affected tests found`, zero delta | -| L7 | L1 with `--tia --parallel` | fallback works in workers as well as the parent | -| L8 | L1, then `pest --tia` twice on `feature-x` | idempotent; second run also `w=0` | -| L9 | detached HEAD on a `master`-only graph | replays, and **no `main` key minted** — `keys` stays `[master]`. Bug (b). | -| L10 | `pest()->tia()->defaultBranch('master')` in `tests/Pest.php`, repo default renamed away | config wins over autodetect | -| L11 | config set to a nonexistent branch (`defaultBranch('nope')`) | degrades to a full run; no crash, no phantom key | -| L12 | no remote at all (`git remote remove origin` if present) | still resolves (via `init.defaultBranch`) or degrades safely — must not throw | -| L13 | branch name with a slash (`feature/x/y`) | replayed; no key-splitting bugs | -| L14 | L1, then confirm writes | `feature-x` gets its **own** key; the `master` key is **not** written to (fallback is read-only) | -| L15 | git worktree on a new branch (the issue's scenario) | replays from the default-branch baseline | -| L16 | non-git dir, `pest --tia` | still `MissingDependency: The feature "Tia mode" requires "git".` — the soft resolver must not swallow it | -| L17 | non-git dir, plain `pest` | runs normally; no baseline dir created | -| L18 | count `git` subprocesses during one `--tia` run | default-branch resolution is cached, not one call per test. Probe by shimming `git` on `PATH` to a logging wrapper. | - -L10–L11 need a `tests/Pest.php` edit — that file is tracked and **outside** the `tests app` reset -scope in practice (it lives in `tests/`, so `git checkout -- tests` does restore it; verify with -`git status` after). - -For L16/L17, build a throwaway non-git project — and note the trap that burned phase two: a -**symlinked** `vendor` makes Pest resolve the project root back to the playground (identical baseline -hash), silently invalidating the test. Use a hardlinked copy: - -```bash -NG="$SP/nogit"; rm -rf "$NG"; mkdir -p "$NG" -cp -R composer.json composer.lock phpunit.xml artisan tests app bootstrap config routes resources storage "$NG/" -[ -f .env ] && cp .env "$NG/" -cp -Rl vendor "$NG/vendor" || cp -R vendor "$NG/vendor" -``` - -Confirm the baseline path differs (`nogit-`, not `laravel-4a455a95622ac0ec`), and delete both -the temp project and its `~/.pest/tia/nogit-*` dir afterwards. - -### 4.5 Re-run the phase-two matrix (A–K, 156 rows) - -Re-run every row of `PLAN_PHASE_TWO.md` Part 2 against the fixed build. No row's status is trusted -until re-measured — the fix touches `Graph`'s read path, which nearly every row exercises. Sections -**A, B, H, I** are the load-bearing ones here (H is branch-key resolution; I is filtered mode; both -consume the changed methods directly). **H6 and H7 must be re-derived from a cold graph** (Part 1.4). - -Most rows batch cheaply — phase two ran C1–C20 in one call at roughly one line of output each. Use -`oneline.php` for the sweep and `cmp.php` only to drill into anomalies. - -### 4.6 Known pre-existing failures — do not report as regressions - -| item | status | -|---|---| -| ~~**G4 / G4b** — parallel replay clobbers cached `time` on all non-executed tests.~~ | **Struck in phase four — does not reproduce.** `flushWorkerReplay()` applies `resultTime()` worker-side before writing the partial, so the parent's verbatim read of `$result['time']` is reading values that were already corrected. Pinned by `a parallel replay keeps the recorded time of tests that did not run` (`tests/Features/Tia/CompleteRunWriteTier.php`), which sentinels every cached `time` and asserts a parallel replay writes nothing. | -| **C19** — `--tia --uses=…` cannot be fixtured. TIA hard-errors on PHPUnit classes (`EnsureTiaIsRunningPestTestsOnly`), and Pest has no chainable `->uses()`. | **Expected behaviour per Nuno.** Verify the tier (`w=0`, RO, notice) and move on. Not a defect. | -| **J11** — `--repeat` is not a Pest option (`Unknown option "--repeat"`). | **Don't care per Nuno.** Mark SKIP. | -| **J10** — `--random-order-seed` alone exits 1 with a WARN. Identical without `--tia`. | Pre-existing Pest behaviour, unrelated. Tier still holds. | - ---- - -## Part 5 — Reporting - -Per row: **tier respected (yes/no)**, the **graph delta** under sentinel patching, and for any failure -the **pre-fix contrast** so a regression is told apart from a pre-existing defect. For a pre-fix -contrast you need `db70017c` (or `bfd5b756` for "before phase three") files swapped into the -playground's `vendor/` — that is a sync, so **ask Nuno first** and always restore afterwards. - -State which pest commit produced the numbers. Close with: - -1. Whether L1–L18 all pass (the fix works). -2. Whether A–K regressed anywhere relative to phase two's 154/156. -3. The D1–D4 decisions as implemented. -4. Anything still open — including G4, which will still be failing. - -Leave the playground on `master` with only the four user-modified files dirty, no stray branches, and -no leftover `~/.pest/tia/*` dirs beyond `laravel-4a455a95622ac0ec`. diff --git a/PLAN_PHASE_TWO.md b/PLAN_PHASE_TWO.md deleted file mode 100644 index b952d220..00000000 --- a/PLAN_PHASE_TWO.md +++ /dev/null @@ -1,466 +0,0 @@ -# TIA write-tier conformance — phase two - -## Your task - -**Re-run all 156 rows of the matrix in Part 2 from scratch, against the playground app** at -`/Users/nunomaduro/Work/projects/playground/laravel`. Every row, including the ones already marked -VERIFIED or PASS — the point of phase two is that no row's status is trusted until it has been -re-measured against the current code. The `Phase 1` column is prior evidence and a hint at what to -watch, never a reason to skip a row. - -These are **not** the `pestphp/pest` repo's own tests (`composer test`) — do not run those. Each row is -a `pest` invocation against the playground's suite, followed by a diff of the TIA graph it wrote. - -Report, per row: tier respected (yes/no), the graph delta under sentinel patching, and — for any -failure — the pre-fix contrast so a regression is told apart from a pre-existing defect. Do **not** -stop at reading this file or summarising it; the deliverable is executed results. - -Work in this order: **Part 1 (environment traps) → Part 1b (build the fixtures — the playground has -none of them) → Part 2 (the matrix) → Part 3 (priorities)**. The matrix is too large for one context; -take it one lettered section at a time and report as you finish each. Sections A, B, G, I and K are -the load-bearing ones — do those first if you run short. - -Phase one implemented `PLAN.md` Part 1 items 1–4 plus three §5 items. This file turns the matrix into -a conformance check rather than a bug list. - -Code under test: `pestphp/pest` at `/Users/nunomaduro/Work/projects/pestphp/pest`, branch -`fix/tia-filtered`, commit **`bfd5b756`** or later. Verify with -`grep -c 'recordsEdgesInWorkers\|recordsEdges' src/Plugins/Tia.php` → at least 3 hits. Pre-fix -baseline for every contrast is **`db70017c`**. - ---- - -## Part 0 — What phase one changed - -| # | Change | Files | -|---|---|---| -| 1 | `hasUnlocatedTestsToRerun()` stats the file, so a deleted test file is "unlocated" | `src/Plugins/Tia/Graph.php` | -| 2 | `enterReplayMode()` uses `activateLinkTracking()` under piggyback coverage | `src/Plugins/Tia.php` | -| 3 | `enterReplayMode()` stamps `TIA_PIGGYBACK_COVERAGE` for workers | `src/Plugins/Tia.php` | -| 4 | `replaceEdges(…, keepExisting:)` — piggyback edges seed empty sets, never overwrite populated ones | `Graph.php`, `Tia.php` | -| 5 | `renderFreshGraph()` stops claiming "fresh graph" when the graph is kept; reason reworded to `recording a coverage baseline` | `Tia.php` | -| 6 | `COVERAGE_REPORT_FLAGS` + `coverageReportActive()` union over `originalArguments`; new `pestCoverageActive()` keeps the coverage-cache marker/hijack on Pest's own `--coverage` | `Tia.php` | -| 7 | `Tia::recordsEdgesInWorkers()` + `WrapperRunner::handleTia()` inject `-d pcov.directory=` into worker argv | `Tia.php`, `src/Plugins/Parallel/Paratest/WrapperRunner.php` | -| 8 | Sequential record runs announce structural drift via `renderFreshGraph()` | `Tia.php` | -| 9 | `Graph::getTime()` + `cachedTimeByTestId` + `resultTime()` preserve replayed durations; edge-less write guard is now `$recordsEdges = $complete && ($markKnownTestFiles \|\| $this->recordingActive)` | `Graph.php`, `Tia.php` | - -Deliberately **not** done: `PLAN.md` §5 SIGINT propagation, §5 warning/deprecation `status=0` -mapping, and all of §6 (playground annotation fixtures). The vacuous C rows below stay vacuous. - -### Target-outcome changes this forces - -Two rows in the original matrix asserted the **old**, buggy behaviour. Their targets are updated -below — do not report them as regressions: - -- **B10** was "`time` differs; statuses stable". Change 9 means replayed entries now **keep** their - recorded `time`. New target: `time` differs only for tests that actually executed. -- **B12** gains an edge-preservation assertion it never had (see K1). - -### One known-failing repo test - -`tests/Unit/Plugins/Tia/Graph.php:69-76` asserts `hasUnlocatedTestsToRerun('main')` is `false` for -`tests/Feature/FooTest.php` under `new Graph(sys_get_temp_dir())` — a path that does not exist. That -assertion encodes the I9 bug and **will fail** under change 1. It needs re-pointing at a root/file -that exists (e.g. `dirname(__DIR__, 4)` + `'tests/Unit/Plugins/Tia/Graph.php'`). Left untouched by -request; it is a repo-test matter, not a playground one. - ---- - -## Part 1 — Environment, and the traps in it - -**Playground:** `/Users/nunomaduro/Work/projects/playground/laravel` - -```bash -cd /Users/nunomaduro/Work/projects/playground/laravel -GRAPH="$(PAO_DISABLE=1 ./vendor/bin/pest --baseline)/graph.json" # ~/.pest/tia/laravel-4a455a95622ac0ec -``` - -1. **Every** pest invocation needs `PAO_DISABLE=1` — the app has `laravel/pao`, which emits JSON - when it detects an agent. -2. **Never run `git checkout .` or `git checkout -- composer.lock` in the playground.** It has four - pre-existing user-modified files — `AGENTS.md`, `CLAUDE.md`, `composer.json`, `composer.lock` — - that a blanket reset would destroy. `PLAN.md`'s "reset ritual" is unsafe as written. Scope resets - to what you touched: `git checkout -- tests app` / `git clean -fd tests app`, and for - `composer.lock` (needed for the drift rows) **copy it aside and copy it back**, verifying with - `shasum`. -3. **`vendor/pestphp/pest` is a dist copy, not a symlink.** Composer installed - `dev-fix/tia-filtered as 5.2.0`, so edits in the pest repo do **not** reach the playground. - - **Tell Nuno whenever a sync is needed to move forward — do not sync silently.** Say what is stale - and what the sync would be, then wait. This includes temporarily swapping in `db70017c` files for a - before/after contrast. When you report any playground result, state which commit produced it. - The sync itself, once he agrees: - ```bash - PEST=/Users/nunomaduro/Work/projects/pestphp/pest - V=/Users/nunomaduro/Work/projects/playground/laravel/vendor/pestphp/pest - for f in src/Plugins/Tia.php src/Plugins/Tia/Graph.php \ - src/Plugins/Parallel/Paratest/WrapperRunner.php; do cp "$PEST/$f" "$V/$f"; done - ``` - Verify with `grep -c recordsEdgesInWorkers "$V/src/Plugins/Tia.php"` → `1`. **Always restore the - current version before continuing** after a pre-fix contrast. -4. **Coverage driver:** pcov only, no xdebug. `ini_get('pcov.directory')` is `''` by default — that - emptiness is the entire mechanism behind G12. -5. **Suite shape as found:** 7 tests in 5 files — `tests/Unit/{CalculatorTest,ExampleTest,GreeterTest, - TrioTest}.php`, `tests/Feature/ExampleTest.php`. A healthy sequential graph is **`files=22`, - 5 edge keys, self-edge on every test** (`PLAN.md`'s `files=25` is stale). Pre-fix parallel gives - `files=4` with zero self-edges. **These numbers shift the moment you add the Part 1b fixtures** — - re-derive them once, after the fixtures land, and use the new numbers throughout. The invariants - that do *not* shift: sequential and parallel must agree, and every test must have a self-edge. -6. **Sentinel patching is the only reliable discriminator** between "wrote identical values" and - "wrote nothing": rewrite every cached result to `time=9.999 assertions=42`, snapshot, run the - case, diff. A canary test absent from `edges` is unreliable under `--filter` because it never - matches the filter and so never runs. -7. Seeding a cached failure needs `--fresh` (or an env-driven flaky fixture): `--tia` on a clean - green tree replays rather than executes, so it can never cache a failure. Working recipe — break - an assertion, `pest --tia --fresh`, then restore the source. -8. Comment-only edits to a test file are **not** changes (AST-level hashing). Use semantic edits. -9. The shell is zsh — build commands with arrays or `eval`; unquoted `$args` does not word-split. - -### Graph summariser - -Write this to a scratch path and use it for every diff. - -```php - [label] -$g = json_decode((string) file_get_contents($argv[1]), true, 512, JSON_THROW_ON_ERROR); -$files = $g['files'] ?? []; $edges = $g['edges'] ?? []; -echo ($argv[2] ?? $argv[1])."\n files=".count($files).' edges keys='.count($edges)."\n"; -$dirs = []; foreach ($files as $f) { $dirs[explode('/', (string) $f)[0]] = true; } -ksort($dirs); echo ' file dirs: '.implode(', ', array_keys($dirs))."\n"; -ksort($edges); -foreach ($edges as $test => $ids) { - $self = 'no'; - foreach ((array) $ids as $id) { if (($files[$id] ?? null) === $test) { $self = 'YES'; break; } } - echo sprintf(" %-42s n=%-3d self=%s\n", $test, count((array) $ids), $self); -} -foreach ($g['baselines'] ?? [] as $branch => $b) { - $r = $b['results'] ?? []; - echo " baseline[$branch]: n=".count($r).' sha='.substr((string) ($b['sha'] ?? '-'), 0, 7)."\n"; - foreach ($r as $id => $x) { - echo sprintf(" %-58s status=%d time=%s asserts=%d file=%s\n", substr((string) $id, -58), - $x['status'], $x['time'], $x['assertions'], $x['file'] ?? '-'); - } -} -``` - -Normalised edge-set equality (for G12 / I5 / K1): - -```php - -function edges(string $p): array { - $g = json_decode((string) file_get_contents($p), true, 512, JSON_THROW_ON_ERROR); $out = []; - foreach ($g['edges'] as $t => $ids) { $s = array_map(fn ($i) => $g['files'][$i], $ids); sort($s); $out[$t] = $s; } - ksort($out); return $out; -} -$a = edges($argv[1]); $b = edges($argv[2]); -echo $a === $b ? "IDENTICAL edge sets\n" : "DIFFER\n"; -foreach ($a as $t => $s) { - $m = array_diff($s, $b[$t] ?? []); $e = array_diff($b[$t] ?? [], $s); - if ($m || $e) printf(" %s: -%d +%d\n", $t, count($m), count($e)); -} -``` - -### Per-case loop - -```bash -git checkout -- tests app 2>/dev/null; git clean -qfd tests app -rm -rf "$(PAO_DISABLE=1 ./vendor/bin/pest --baseline)" -PAO_DISABLE=1 ./vendor/bin/pest --tia >/dev/null 2>&1 # seed a healthy graph -# sentinel-patch every result to time=9.999 assertions=42, snapshot, run the case, diff -``` - ---- - -## Part 1b — Fixtures you must build first - -**The playground has none of the fixtures the matrix depends on.** Verified inventory: the only test -files are `tests/Unit/{CalculatorTest,ExampleTest,GreeterTest,TrioTest}.php` and -`tests/Feature/ExampleTest.php` — 7 plain tests, zero occurrences of `->group()`, `->only()`, -`->skip()`, `->todo()`, `->note()`, `->flaky()`, `->covers()`, `->uses()`, `->issue()`, `->pr()`, -`->ticket()`, `->assignee()`, datasets, or any `FLAKY_OK`-style env hook. `phpunit.xml` defines only -the `Unit` and `Feature` testsuites. - -So roughly 30 rows below cannot run as written until you create the fixtures. Build them **all up -front, in one commit**, then re-derive the baseline graph shape once and use those numbers for the -whole sweep — every fixture you add changes `files`, the `edges` key count and `n`, so adding them -piecemeal invalidates earlier rows. - -| Fixture to create | Rows that need it | -|---|---| -| a test with `->group('smoke')` | C7, C8, C9, C10 | -| an env-driven flaky test (passes iff `FLAKY_OK=1`) | E1, E2, E3, E4 | -| a `->skip()`ed test | E9, D8 | -| a `->todo()` test | E10, C23, C39 | -| a risky test (no assertions; pair with `--disallow-test-output`) | E11, D7 | -| a test that triggers a PHPUnit warning | D6 | -| a test calling `markTestIncomplete()` | D9 | -| a test that triggers a deprecation | D10 | -| a dataset test with ≥3 rows | E13, E14 | -| `->only()` — added and removed per case, not left in | C31, C32, G7 | -| `->covers(App\Services\Calculator::class)` | C18 | -| `->uses(...)` / `UsesClass` annotation | C19 | -| `->note(...)` | C24 | -| `->flaky()` **annotation** (distinct from the env-driven flaky test above) | C25 | -| `->issue(123)`, `->pr(1)`, `->ticket('X')`, `->assignee('X')` | C26, C27, C28, C37, C38 | - -This is `PLAN.md` §6's "test-harness gaps to close before re-running", now itemised: without these, -the listed rows' graph-invariant assertions match zero tests and **prove nothing** — they pass -vacuously. Any row still marked "(vacuous)" in Part 2 is vacuous *only because* its fixture is -missing; once you add the fixture, treat the row as unverified and make it load-bearing. - -Rows needing an *action* rather than a fixture — breaking `trio one` for the D rows, an uncommitted -test edit for `--dirty`, branch renames and a non-git dir for H5–H10, `--mutate` against -`app/Services` for the F rows — are fine as written; `pestphp/pest-plugin-mutate` is installed. - ---- - -## Part 2 — Full case matrix - -Tiers: **COMPLETE** may change everything · **RESULTS-ONLY** (RO) may change only -`baselines[].results` for tests that ran, and must never remove an entry, add a result for a -test file absent from `edges`, or alter `sha`/`tree`/`edges`/`files`/`fingerprint` · -**HARD-SUPPRESSED** may change nothing. - -Phase-1 column: **VERIFIED** = re-run against the fixed code in the phase-one session, pre-fix -contrast captured · **PASS (sweep)** = passed in the original `db70017` sweep and *not* re-checked -since the changes — these are the bulk of phase two's work · **SKIP** = not runnable as written. - -### A — Setup & sanity - -| # | Case | Target outcome | Phase 1 | -|---|---|---|---| -| A1 | `composer show pestphp/pest` | version `dev-fix/tia-filtered` | PASS (sweep) | -| A2 | `pest --baseline` | prints an existing dir; exit 0 | VERIFIED | -| A3 | delete graph, `pest --tia` | graph.json created; one `edges` key per test file; one result per test | VERIFIED | -| A4 | `git status` after reset ritual | clean *except the four user-modified files* (see trap 2) | VERIFIED | -| A5 | `extension_loaded("pcov")` | `true` | VERIFIED | -| A6 | delete graph, plain `pest` | no graph created | PASS (sweep) | -| A7 | delete graph, `pest --filter=adds` | no graph created | PASS (sweep) | -| A8 | two consecutive `pest --tia` | second replays everything; `sha` unchanged | VERIFIED | - -### B — COMPLETE runs still write - -| # | Case | Target outcome | Phase 1 | -|---|---|---|---| -| B1 | `pest --tia` (clean) | replays; graph written; `sha` = `git rev-parse HEAD` | VERIFIED | -| B2 | `pest --no-tia` | results refreshed; prune applied; `sha`/`tree`/`edges` unchanged | PASS (sweep) — recheck under change 9 | -| B3 | `pest` (plain) | same as B2 | PASS (sweep) — recheck under change 9 | -| B4 | `pest --tia --fresh` | graph purged and rebuilt; `edges` rebuilt; `files` repopulated | VERIFIED | -| B5 | `pest --bail`, all green | COMPLETE — graph written, prune applied | PASS (sweep) | -| B6 | edit `Calculator.php`, `pest --tia` | `CalculatorTest` re-runs, others replay; `tree` updated | PASS (sweep) | -| B7 | delete `trio three`, `pest --tia` | its result pruned; `trio one`/`two` remain | PASS (sweep) | -| B8 | delete `GreeterTest.php`, `pest --tia` | result and edges survive (missing-file prune is `--fresh`-only) | PASS (sweep) | -| B9 | B8 then `pest --tia --fresh` | entry gone; `files` drops `Greeter.php` too | PASS (sweep) | -| B10 | `pest --tia` twice | **UPDATED TARGET:** `time` differs only for tests that executed; replayed entries keep their recorded `time`; statuses stable | VERIFIED | -| B11 | add a test file, `pest --tia` | new `edges` key **and** new result appear in the **same** run | VERIFIED (regression guard for change 9) | -| B12 | `pest --tia --coverage` | completes; graph written; coverage report prints | VERIFIED | - -### C — Selection narrowing → RESULTS-ONLY - -All rows: RO invariants hold. "Notice" = `TIA does not apply to partial runs — running the selected tests directly.` - -| # | Case | Target outcome | Phase 1 | -|---|---|---|---| -| C1 | `pest --filter="adds numbers"` | RO; no notice | PASS (sweep) | -| C2 | `pest --tia --filter="adds numbers"` | RO; notice | PASS (sweep) | -| C3 | `pest --filter="trio one"` | RO; `trio two`/`three` byte-identical | PASS (sweep) | -| C4 | `pest --filter="trio"` | RO; all three update; nothing else does | PASS (sweep) | -| C5 | `pest --exclude-filter="Feature"` | RO; no notice; no prune | PASS (sweep) | -| C6 | `pest --tia --exclude-filter="Feature"` | RO; notice | PASS (sweep) | -| C7 | `pest --group=smoke` | RO; no notice | PASS (sweep) | -| C8 | `pest --tia --group=smoke` | RO; notice | PASS (sweep) | -| C9 | `pest --exclude-group=smoke` | RO; no notice; no prune | PASS (sweep) | -| C10 | `pest --tia --exclude-group=smoke` | RO; notice | PASS (sweep) | -| C11 | `pest tests/Unit` | RO; no notice; Feature entry not pruned | PASS (sweep) | -| C12 | `pest --tia tests/Unit` | RO; notice | PASS (sweep) | -| C13 | `pest tests/Unit/TrioTest.php` | RO; no notice | PASS (sweep) | -| C14 | `pest --tia tests/Unit/TrioTest.php` | RO; notice | PASS (sweep) | -| C15 | `pest --testsuite=Unit` | RO; no notice | PASS (sweep) | -| C16 | `pest --tia --testsuite=Unit` | RO; notice | PASS (sweep) | -| C17 | `pest --tia --exclude-testsuite=Feature` | RO; notice | PASS (sweep) | -| C18 | `pest --tia --covers='App\Services\Calculator'` | RO; notice | PASS (sweep) | -| C19 | `pest --tia --uses='App\Services\Calculator'` | RO; notice | PASS (vacuous — needs fixture) | -| C20 | `pest --tia --test-suffix=Test.php` | RO even though all tests run; `edges`/`files`/`n` unchanged | PASS (sweep) | -| C21 | `pest --dirty` with an uncommitted test edit | RO; no notice | PASS (sweep) | -| C22 | `pest --tia --dirty` | RO; notice | PASS (sweep) | -| C23 | `pest --tia --todos` | RO; notice | PASS (vacuous) | -| C24 | `pest --tia --notes` | RO; notice | PASS (vacuous) | -| C25 | `pest --tia --flaky` | RO; notice | PASS (vacuous) | -| C26 | `pest --tia --issue=123` | RO; notice | PASS (vacuous) | -| C27 | `pest --tia --pr=1` | RO; notice | PASS (vacuous) | -| C28 | `pest --tia --pull-request=1` | RO; notice | PASS (vacuous) | -| C29 | `pest --tia --shard=1/2` | RO; notice; no prune | PASS (sweep) | -| C30 | `pest --tia --shard=2/2` | RO; notice; C29+C30 covers the suite; neither prunes | PASS (sweep) | -| C31 | `->only()` on `trio one`, `pest --tia` | RO; no notice; siblings untouched | PASS (sweep) | -| C32 | `->only()` on `trio one`, plain `pest` | RO; no notice | PASS (sweep) | -| C33 | `pest --tia --filtered --filter=adds` | RO; notice; filtered yields to narrowing | PASS (sweep) | -| C34 | `PEST_TIA=1 pest --filter=adds` | RO; notice | PASS (sweep) | -| C35 | `PEST_TIA_FILTERED=1 pest --filter=adds` | RO; notice (regression guard) | PASS (sweep) | -| C36 | `pest --filtered --filter=adds` | RO; notice | PASS (sweep) | -| C37 | `pest --tia --ticket=X` | RO; notice. `--ticket` **is** a real option (`src/Plugins/Snapshot.php:83`) — it narrows legitimately | PASS (vacuous) | -| C38 | `pest --tia --assignee=X` | RO; notice (`src/Plugins/Snapshot.php:81`) | PASS (vacuous) | -| C39 | `pest --tia --todo` | RO; notice; matches nothing | PASS (vacuous) | - -### D — Truncation → RESULTS-ONLY - -D1–D5, D11–D13, D16 precondition: `trio one` broken. D6–D10 run against a green suite so the `--stop-on-*` trigger is the only narrowing. - -| # | Case | Target outcome | Phase 1 | -|---|---|---|---| -| D1 | `pest --bail` | RO; `trio one` = `status=7`; siblings survive unchanged | PASS (sweep) | -| D2 | `pest --retry` | RO; siblings survive | PASS (sweep) | -| D3 | `pest --stop-on-failure` | RO | PASS (sweep) | -| D4 | `pest --stop-on-defect` | RO | PASS (sweep) | -| D5 | `pest --stop-on-error` (no error occurs) | COMPLETE — nothing stopped it | PASS (sweep) | -| D6 | `pest --stop-on-warning` | RO when it fires. Warning stored as `status=0`, not `6` | PASS (sweep) | -| D7 | `pest --stop-on-risky --disallow-test-output` | RO; `status=5` | PASS (sweep) | -| D8 | `pest --stop-on-skipped` | RO; `status=1` | PASS (sweep) | -| D9 | `pest --stop-on-incomplete` | RO; `status=2` | PASS (sweep) | -| D10 | `pest --stop-on-deprecation` | RO. Deprecation stored as `status=0`, not `4` | PASS (sweep) | -| D11 | `pest --tia --bail` | RO | PASS (sweep) | -| D12 | `pest --bail --filter=trio` | RO (narrowed *and* truncated) | PASS (sweep) | -| D13 | `stopOnFailure="true"` in `phpunit.xml`, plain `pest` | RO — caught via `stoppedEarly()`, not flag matching | PASS (sweep) | -| D14 | D13 config, green suite | COMPLETE | PASS (sweep) | -| D15 | `pest --bail`, last test in run order fails | RO — over-conservative but safe; nothing was skipped | PASS (sweep) | -| D16 | `pest --tia --filtered --bail`, broken affected test | RO | PASS (sweep) | - -### E — Result merge semantics - -| # | Case | Target outcome | Phase 1 | -|---|---|---|---| -| E1 | cached `flaky`=7, `FLAKY_OK=1 pest --filter=flaky` | flips to `status=0`; all other results byte-identical | PASS (sweep) | -| E2 | cached `flaky`=0, `pest --filter=flaky` (env unset) | flips to `status=7` | PASS (sweep) | -| E3 | after E1, `pest --tia --filtered` clean | `No affected tests found`; zero graph delta | PASS (sweep) | -| E4 | after E2, `pest --tia --filtered` | re-runs, `from 1 previously unsuccessful test` | PASS (sweep) | -| E5 | any partial run | `assertions` and `time` update for the test that ran (it executed, so change 9 does not apply) | PASS (sweep) | -| E6 | any partial run | `message` of untouched tests unchanged | PASS (sweep) | -| E7 | two sequential partial runs on different tests | each updates only its own entry; both persist | PASS (sweep) | -| E8 | `pest --filter="trio one"` | siblings keep exact `status`/`time`/`assertions`/`message` | PASS (sweep) | -| E9 | partial run of a `->skip()`ed test | `status=1` | PASS (sweep) | -| E10 | partial run of a `->todo()` test | `status=1`, `assertions=0`, `message="__TODO__"` | PASS (sweep) | -| E11 | partial run of a risky test | `status=5` | PASS (sweep) | -| E12 | any partial run | `fingerprint` byte-identical | PASS (sweep) | -| E13 | dataset test, `--filter` matching one row | that row updates; other rows survive (the old prune bug) | PASS (sweep) | -| E14 | dataset test, full `--tia` after deleting a row | the deleted row is pruned | PASS (sweep) | - -### F — Guard rails - -| # | Case | Target outcome | Phase 1 | -|---|---|---|---| -| F1 | test absent from `edges`, `pest --filter="brand new"` | no result recorded | VERIFIED | -| F2 | complete `pest --tia` first, then the same filter | result is recorded | VERIFIED | -| F3 | `pest --mutate --path=app/Services --covered-only` | statuses/`edges`/`sha`/`tree`/`files` identical; only `time` may change | PASS (sweep) | -| F4 | `pest --mutate --parallel --path=… --covered-only` | same (observed: zero delta) | PASS (sweep) — recheck under change 7 | -| F5 | weakened test so a mutant survives, `pest --mutate` | no status change in the baseline | PASS (sweep) | -| F6 | `pest --mutate` with the graph deleted | no graph created | PASS (sweep) | -| F7 | mutation subprocess env | carries `PEST_MUTATION_TESTING`; `--mutate` stripped from argv | PASS (sweep) | -| F8 | grep baseline after `--mutate` | no mutation-flavoured messages | PASS (sweep) | - -### G — Parallel - -| # | Case | Target outcome | Phase 1 | -|---|---|---|---| -| G1 | `pest --tia --parallel` (clean) | COMPLETE — graph written; all 7 results present | VERIFIED | -| G2 | `pest --tia --parallel --filter=adds` | RO | PASS (sweep) | -| G3 | `pest --parallel --bail`, `trio one` broken | RO; siblings survive | PASS (sweep) | -| G4 | `pest --tia --parallel --bail` | RO | PASS (sweep) | -| G5 | `pest --tia --filtered --parallel` after a source edit | narrows to affected; writes | VERIFIED (see I5) | -| G6 | `pest --tia --parallel` | worker results reach the parent baseline | VERIFIED | -| G7 | `->only()` + `pest --tia --parallel` | RO | PASS (sweep) | -| G8 | `pest --tia --parallel --shard=1/2` | RO | PASS (sweep) | -| G9 | broken test in one worker, `pest --parallel --bail` | whole run is RO, not just that worker's slice | PASS (sweep) | -| G10 | `pest --parallel --retry` | `InvalidOption`; graph untouched | PASS (sweep) | -| G11 | `pest --tia --parallel --fresh` | graph rebuilt | VERIFIED | -| G12 | `edges` after `pest --tia` vs `pest --tia --parallel --fresh` | **equivalent edge sets** — `files=22`, self-edge on all 5 tests, both | **VERIFIED (fixed)** — pre-fix: `files=4`, 0 self-edges | - -### H — Baseline key / branch resolution - -| # | Case | Target outcome | Phase 1 | -|---|---|---|---| -| H1 | `pest --tia` on `master` | only a `master` key | VERIFIED | -| H2 | plain `pest` on `master` | writes `master`, not `main` | PASS (sweep) | -| H3 | `pest --bail` truncated on `master` | no `main` key minted | PASS (sweep) | -| H4 | `pest --filter=adds` on `master` | no `main` key minted | PASS (sweep) | -| H5 | `git branch -m main`, full `pest --tia` | single `main` key | PASS (sweep) | -| H6 | `git checkout -b feature/x`, `pest --tia` | `feature/x` key appears; reads fall back to the existing baseline | PASS (sweep) | -| H7 | detached HEAD, `pest --tia` | falls back to the `main` key for both reads and writes; no `HEAD` key | PASS (sweep) | -| H8 | after every C and D case | no baseline key other than the real branch | PASS (sweep) | -| H9 | non-git dir, `pest --tia` | `MissingDependency` — `The feature "Tia mode" requires "git".` | PASS (sweep) | -| H10 | non-git dir, plain `pest` | runs normally; no baseline dir created | PASS (sweep) | - -### I — Filtered mode - -| # | Case | Target outcome | Phase 1 | -|---|---|---|---| -| I1 | clean + green, `pest --tia --filtered` | `No affected tests found`; zero graph delta | PASS (sweep) | -| I2 | edit `Calculator.php` | only `CalculatorTest` runs | VERIFIED | -| I3 | cached failure, clean tree | re-runs it, `from 1 previously unsuccessful test` | PASS (sweep) | -| I4 | I2 with sentinels | `tree` updated; exactly one entry rewritten; all others retained | PASS (sweep) | -| I5 | `pest --tia --filtered --parallel` | same narrowing as I2 **and `edges` unchanged** | **VERIFIED (fixed)** — 1 affected test, edges identical | -| I6 | `PEST_TIA_FILTERED=1 pest --tia` | identical delta to I4 | PASS (sweep) | -| I7 | `pest --tia --filtered tests/Unit` | explicit path wins; filtered off; RO; notice | PASS (sweep) | -| I8 | `pest --tia --filtered --coverage-text` | filtered mode disabled by an active coverage report → full suite runs | **VERIFIED (fixed)** — pre-fix: `No affected tests found` | -| I9 | cached failure whose test file was deleted | WARN `Some cached tests due a re-run could not be located on disk` + `Running the full suite with replay instead of a filtered run` | **VERIFIED (fixed)** — pre-fix: `No tests found`, exit 0 green | -| I10 | `pest --tia --filtered` with no baseline yet | records a baseline instead of filtering | PASS (sweep) | -| I11 | edit a Blade view a Feature test renders | the Feature test is selected | PASS (sweep) | -| I12 | edit `composer.lock` | fingerprint drift → full rebuild, **and the reason is printed sequentially**: `fresh graph (composer.lock changed)` | **VERIFIED (fixed)** — pre-fix: bare `Running in TIA mode.` | - -### J — Interactions & regressions - -| # | Case | Target outcome | Phase 1 | -|---|---|---|---| -| J1 | `pest --tia --fresh --filter=adds` | graph **not** purged; RO | PASS (sweep) | -| J2 | `pest --tia --refetch --filter=adds` | refetch not performed; RO | PASS (sweep) | -| J3 | `pest --no-tia --filter=adds` | RO — narrowing wins over `--no-tia`; no notice | PASS (sweep) | -| J4 | `pest --tia --no-tia` | TIA disabled; COMPLETE | PASS (sweep) | -| J5 | `integration`, `pest --tia` | COMPLETE, no notice — an always-in-force filter applies to baseline runs too | PASS (sweep) | -| J6 | `pest --tia --compact` | COMPLETE | PASS (sweep) | -| J7 | `pest --tia -v` | COMPLETE | PASS (sweep) | -| J8 | `pest --tia --profile` | COMPLETE | PASS (sweep) | -| J9 | `pest --tia --order-by=random` | COMPLETE | PASS (sweep) | -| J10 | `pest --tia --random-order-seed=1234` | COMPLETE | PASS (sweep) | -| J11 | `pest --tia --repeat=2` | COMPLETE — repetition is not narrowing | **SKIP** — `--repeat` is not a Pest option | -| J12 | `pest --filter=adds` twice | only the executed test's `time` differs; rest byte-identical | PASS (sweep) | -| J13 | `pest --tia --min=50` | COMPLETE (silent no-op without `--coverage`) | PASS (sweep) | -| J14 | SIGINT mid-suite | RO — an interrupted run is truncated | PASS (sweep) — signal still does not reach the re-exec'd child (not fixed) | -| J15 | delete the graph, `pest --tia --filtered` | records a baseline rather than erroring | PASS (sweep) | -| J16 | corrupt `graph.json`, `pest --tia` | recovers by rebuilding; no crash | PASS (sweep) | - -### K — New rows for the phase-one fixes - -These assert behaviour no original row covered. All were verified in phase one; re-run them as -regression guards. - -| # | Case | Target outcome | Phase 1 | -|---|---|---|---| -| K1 | healthy `pest --tia` graph, then `pest --tia --coverage` | `edges` **byte-identical** — piggyback data may seed empty sets, never narrow populated ones; `files=22` stays `22` | **VERIFIED (fixed)** — pre-fix: `ExampleTest` 16→2 edges, self-edges lost | -| K2 | same run's headline | `Experimental TIA mode enabled / recording a coverage baseline.` — no false `fresh graph` | **VERIFIED (fixed)** | -| K3 | `pest --tia` ×3 on a clean tree | replayed entries keep their recorded `time` across all three | **VERIFIED (fixed)** — pre-fix: `0.053 → 0.001 → 0.001 → 0.001` | -| K4 | add a test file, one `pest --tia` (graph exists, fingerprint matches → replay+refresh) | result **and** edges appear in that same run; `n` grows by 1 | **VERIFIED** — regression guard for change 9 | -| K5 | add a test file, plain `pest --no-tia` | no result written for it; `n` unchanged; no edges | **VERIFIED (fixed)** — pre-fix: edge-less result inflated `n` | -| K6 | `pest --tia --parallel` worker argv | carries `-d pcov.directory=`; `PEST_TIA` unset; not re-exec'd | **VERIFIED** — probe `bin/worker.php` | -| K7 | second `pest --tia --coverage` (cache primed → replay) | recorder uses link tracking only, does not clear PHPUnit's data mid-collection; coverage report intact | Not yet measured — **new work** | -| K8 | `pest --tia --parallel --coverage` | workers read `TIA_PIGGYBACK_COVERAGE`; no widened pcov scope; report intact | Not yet measured — **new work** | -| K9 | `pest --tia --filtered --coverage-html=` / `--coverage-clover=` | filtered mode off, same as I8, for every flag in `COVERAGE_REPORT_FLAGS` | Not yet measured — **new work** | - ---- - -## Part 3 — Priorities for phase two - -0. **Part 1b fixtures** — nothing in C, D6–D10, or E9–E14 means anything until they exist. -1. **K7, K8, K9** — the only rows never measured. K7/K8 exercise changes 2 and 3, which were - reasoned about but not observed; K9 covers the seven `COVERAGE_REPORT_FLAGS` beyond - `--coverage-text`. -2. **B2, B3, F4** and all of **D** and **E** — change 9 touched the shared result-write path, and - these are the rows that exercise it hardest. `$recordsEdges` is the thing to falsify: it must be - false for every partial and every non-recording run. -3. **F3–F8** — change 7 injects a `-d` into worker argv; `--mutate --parallel` is the one place that - both spawns workers and must write nothing. -4. **H1–H10** — untouched by these changes; cheapest bulk confirmation. -5. Raw PHPUnit coverage flags print **no report at all** in Pest, with or without TIA - (`--no-tia --coverage-text` is equally silent). Pre-existing, unrelated to change 6 — do not - chase it as a regression, but it means I8/K9 can only assert the selection half. - -Report per row: tier respected (yes/no), the graph delta under sentinel patching, and for any -failure the pre-fix contrast (write `git show db70017c:` into vendor, re-run, restore) so a -regression is told apart from a pre-existing defect.