diff --git a/.gitignore b/.gitignore index 5302996a..f9e47d6d 100644 --- a/.gitignore +++ b/.gitignore @@ -1,7 +1,7 @@ .idea/* .idea/codeStyleSettings.xml .temp/* -composer.lock +/composer.lock /vendor/ coverage.xml .phpunit.result.cache diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 00000000..f8aa3f1d --- /dev/null +++ b/PLAN.md @@ -0,0 +1,342 @@ +# TIA write-tier conformance — plan + +Baseline sweep: **147 cases, 144 PASS / 2 FAIL / 1 SKIP** against `dev-fix/tia-filtered` (`db70017`) +on `playground/laravel` with pcov enabled. + +This file records (1) what is left to fix, and (2) the full case matrix with the target +outcome for every row, so the sweep can be re-run as a conformance check after the fixes land. + +--- + +## Part 1 — What's left + +### 1. `--parallel` silently under-records `edges` (G12, I5) — **high** + +Any `--tia --parallel` complete run writes `files=4` where sequential writes `files=25`. Only +`app/**` and blade views survive; every `config/`, `routes/`, `bootstrap/` edge and every test +self-edge is lost. A `routes/web.php` edit that sequential TIA catches, parallel TIA reports as +`No affected tests found`. + +Root cause — `src/Restarters/PcovRestarter.php:33`: + +```php +if (! Tia::isEnabledForRun($arguments)) { + return; +} +``` + +`Tia::isEnabledForRun()` (`src/Plugins/Tia.php:275-288`) returns true only for `--tia` in argv or +the `PEST_TIA` env flag. A paratest worker's argv carries neither, so workers never re-exec with +`-d pcov.directory=` and record under pcov's empty default scope. + +Proven both directions: +- `PEST_PCOV_RESTARTER_RESTARTED=1 pest --tia` (sequential, restart suppressed) reproduces + `files=4` with edge sets identical to the parallel graph. +- `PEST_TIA=1 pest --parallel` restores the full `files=25` graph. **Working workaround today.** + +Coverage config is irrelevant: widening `phpunit.xml` `` to `app|config|routes|bootstrap|tests` +still yields `files=4`. + +Sticky, not just per-run: `pest --tia --filtered --parallel` re-records only the affected test +through a worker and strips that entry's edges from an otherwise-healthy sequential graph, so one +parallel run degrades a good graph incrementally. + +**Fix direction:** propagate the TIA-enabled decision into workers (e.g. set `PEST_TIA` in the +worker env, or have the parent stamp a recording global the restarter also consults) so +`PcovRestarter` restarts them. Then assert `edges` equivalence between sequential and parallel. + +### 2. Deleted test file with a cached failure wedges `--filtered` forever (I9) — **high** + +Expected a WARN and a full-suite fallback. Actual: `--filtered` selects the phantom file, runs +nothing, exits **0 green**, on every subsequent invocation. + +Root cause — `src/Plugins/Tia/Graph.php:1523-1525`: + +```php +if ($real === false) { + $real = $path; +} +``` + +`relative()` falls back to the raw path when `realpath()` fails, so a deleted-but-in-project test +file is never "unlocated", `Graph::hasUnlocatedTestsToRerun()` never fires, and the WARN branch at +`src/Plugins/Tia.php:982-986` is unreachable for the deletion case. `pruneMissingTests()` only runs +under `--fresh`, so nothing clears it. + +**Fix direction:** distinguish "path could not be resolved" from "path resolved outside the root" in +`relative()`, or have `hasUnlocatedTestsToRerun()` stat the file directly. + +### 3. `--coverage-text` does not disable filtered mode (I8) — **low** + +`coverageReportActive()` (`src/Plugins/Tia.php:1705-1710`) reads only Pest's own `--coverage`, so +raw PHPUnit coverage flags (`--coverage-text`, `--coverage-html`, `--coverage-clover`, …) leave +filtered mode on. `Tia::COVERAGE_FLAGS` at `:113-115` already enumerates them — that list is not +consulted here. + +### 4. `--coverage` narrows edges like parallel does — **medium, needs triage** + +`pest --tia --coverage` prints `fresh graph (recording coverage baseline)` even when a graph +exists, and writes `Feature/ExampleTest` with 2 files instead of 16, dropping self-edges. Same +observable shape as G12, likely a different mechanism (piggyback collector scoped by +`phpunit.xml ` rather than the pcov-restarted recorder). Theme: **edges silently narrow to +`app/**` whenever recording does not go through the pcov-restarted TIA recorder.** Confirm whether +these are one fix or two. + +### 5. Smaller items + +| Item | Where | Note | +|---|---|---| +| Warning/deprecation recorded as `status=0` | write path | Codes `6` and `4` appear unreachable; risky/skipped/incomplete record faithfully | +| Complete non-TIA runs write edge-less results | `src/Plugins/Tia.php:1645`, `:638` | Guard is `! $complete && ! knowsTest()`; complete bypasses it and `markKnownTestFiles` stays false. Inert — replay guards on the same predicate at `:360` — but inflates `n` | +| SIGINT never reaches the suite | `PcovRestarter` re-exec | Parent ignores it; the child holds PHPUnit's handler. CI `timeout`/Ctrl-C will not stop a run | +| Structural drift never announced sequentially | `src/Plugins/Tia.php:1178-1181` | `enterRecordMode()` prints a bare `Running in TIA mode.`; `renderFreshGraph()` (`fresh graph (composer.lock changed)`) only runs under `--parallel` or coverage piggyback | +| Replay clobbers cached `time` | write path | `0.058` → `0.001`; timing data degrades toward zero across runs | +| `pest --parallel` without `--tia` writes nothing | G3, G6 | Sequential `pest --filter=…` does record. Parallel CI contributes nothing to the cached-failure replay path | +| `--tia --parallel --filter`/`--shard` record nothing | G2, G8 | More conservative than the results-only contract requires | +| `--min=50` without `--coverage` is a silent no-op | — | — | +| `pest --repeat=2` is not a Pest option | J11 | Case unrunnable as written; drop it or add the option | + +### 6. Test-harness gaps to close before re-running + +- The playground has no `UsesClass`, `->note()`, `->flaky()`, `->issue()`, `->pr()`, `->ticket()` or + `->assignee()` annotations, so **C19, C23–C28, C37–C39 matched zero tests** and their + graph-invariant assertions proved nothing. Add annotated fixtures to make those rows load-bearing. +- Use **sentinel patching** (rewrite every cached entry to `time=9.999 assertions=42`, then see which + entries get overwritten) as the discriminator. It is the only way to tell "wrote identical values" + from "wrote nothing". A canary test absent from `edges` is unreliable under `--filter` because it + never matches the filter and so never runs. +- `--tia` on a clean green tree can never cache a failure (unchanged tests replay rather than + execute); seeding one requires `--fresh` or an env-driven flaky fixture. +- Comment-only edits to a test file are **not** changes (AST-level hashing). Use semantic edits. + +--- + +## Part 2 — Full case matrix (target outcomes) + +Tiers: **COMPLETE** may change everything · **RESULTS-ONLY** (RO) may change only +`baselines[].results` for tests that ran, and must never remove an entry, add a result for a +test file absent from `edges`, or alter `sha`/`tree`/`edges`/`files`/`fingerprint` · +**HARD-SUPPRESSED** may change nothing. + +Status column: `PASS` = conforming today · `FIX` = must pass once the item above lands · +`SKIP` = case not runnable as written. + +### A — Setup & sanity + +| # | Case | Target outcome | Status | +|---|---|---|---| +| A1 | `composer show pestphp/pest` | version `dev-fix/tia-filtered` | PASS | +| A2 | `pest --baseline` | prints an existing dir; exit 0 | PASS | +| A3 | delete graph, `pest --tia` | graph.json created; one `edges` key per test file; one result per test | PASS | +| A4 | `git status` after reset ritual | clean | PASS | +| A5 | `extension_loaded("pcov")` | `true` | PASS | +| A6 | delete graph, plain `pest` | no graph created | PASS | +| A7 | delete graph, `pest --filter=adds` | no graph created | PASS | +| A8 | two consecutive `pest --tia` | second replays everything; `sha` unchanged | PASS | + +### B — COMPLETE runs still write + +| # | Case | Target outcome | Status | +|---|---|---|---| +| B1 | `pest --tia` (clean) | replays; graph written; `sha` = `git rev-parse HEAD` | PASS | +| B2 | `pest --no-tia` | results refreshed; prune applied; `sha`/`tree`/`edges` unchanged | PASS | +| B3 | `pest` (plain) | same as B2 | PASS | +| B4 | `pest --tia --fresh` | graph purged and rebuilt; `edges` rebuilt; `files` repopulated | PASS | +| B5 | `pest --bail`, all green | COMPLETE — graph written, prune applied | PASS | +| B6 | edit `Calculator.php`, `pest --tia` | `CalculatorTest` re-runs, others replay; `tree` updated | PASS | +| B7 | delete `trio three`, `pest --tia` | its result pruned; `trio one`/`two` remain | PASS | +| B8 | delete `GreeterTest.php`, `pest --tia` | result and edges survive (missing-file prune is `--fresh`-only) | PASS | +| B9 | B8 then `pest --tia --fresh` | entry gone; `files` drops `Greeter.php` too | PASS | +| B10 | `pest --tia` twice | `time` differs; statuses stable | PASS | +| B11 | add a test file, `pest --tia` | new `edges` key + new result appear | PASS | +| B12 | `pest --tia --coverage` | completes; graph written | PASS | + +### C — Selection narrowing → RESULTS-ONLY + +All rows: RO invariants hold. "Notice" = `TIA does not apply to partial runs — running the selected tests directly.` + +| # | Case | Target outcome | Status | +|---|---|---|---| +| C1 | `pest --filter="adds numbers"` | RO; no notice | PASS | +| C2 | `pest --tia --filter="adds numbers"` | RO; notice | PASS | +| C3 | `pest --filter="trio one"` | RO; `trio two`/`three` byte-identical | PASS | +| C4 | `pest --filter="trio"` | RO; all three update; nothing else does | PASS | +| C5 | `pest --exclude-filter="Feature"` | RO; no notice; no prune | PASS | +| C6 | `pest --tia --exclude-filter="Feature"` | RO; notice | PASS | +| C7 | `pest --group=smoke` | RO; no notice | PASS | +| C8 | `pest --tia --group=smoke` | RO; notice | PASS | +| C9 | `pest --exclude-group=smoke` | RO; no notice; no prune | PASS | +| C10 | `pest --tia --exclude-group=smoke` | RO; notice | PASS | +| C11 | `pest tests/Unit` | RO; no notice; Feature entry not pruned | PASS | +| C12 | `pest --tia tests/Unit` | RO; notice | PASS | +| C13 | `pest tests/Unit/TrioTest.php` | RO; no notice | PASS | +| C14 | `pest --tia tests/Unit/TrioTest.php` | RO; notice | PASS | +| C15 | `pest --testsuite=Unit` | RO; no notice | PASS | +| C16 | `pest --tia --testsuite=Unit` | RO; notice | PASS | +| C17 | `pest --tia --exclude-testsuite=Feature` | RO; notice | PASS | +| C18 | `pest --tia --covers='App\Services\Calculator'` | RO; notice | PASS | +| C19 | `pest --tia --uses='App\Services\Calculator'` | RO; notice | PASS (vacuous — needs fixture) | +| C20 | `pest --tia --test-suffix=Test.php` | RO even though all tests run; `edges`/`files`/`n` unchanged | PASS | +| C21 | `pest --dirty` with an uncommitted test edit | RO; no notice | PASS | +| C22 | `pest --tia --dirty` | RO; notice | PASS | +| C23 | `pest --tia --todos` | RO; notice | PASS (vacuous) | +| C24 | `pest --tia --notes` | RO; notice | PASS (vacuous) | +| C25 | `pest --tia --flaky` | RO; notice | PASS (vacuous) | +| C26 | `pest --tia --issue=123` | RO; notice | PASS (vacuous) | +| C27 | `pest --tia --pr=1` | RO; notice | PASS (vacuous) | +| C28 | `pest --tia --pull-request=1` | RO; notice | PASS (vacuous) | +| C29 | `pest --tia --shard=1/2` | RO; notice; no prune | PASS | +| C30 | `pest --tia --shard=2/2` | RO; notice; C29+C30 covers the suite; neither prunes | PASS | +| C31 | `->only()` on `trio one`, `pest --tia` | RO; no notice; siblings untouched | PASS | +| C32 | `->only()` on `trio one`, plain `pest` | RO; no notice | PASS | +| C33 | `pest --tia --filtered --filter=adds` | RO; notice; filtered yields to narrowing | PASS | +| C34 | `PEST_TIA=1 pest --filter=adds` | RO; notice | PASS | +| C35 | `PEST_TIA_FILTERED=1 pest --filter=adds` | RO; notice (regression guard) | PASS | +| C36 | `pest --filtered --filter=adds` | RO; notice | PASS | +| C37 | `pest --tia --ticket=X` | RO; notice. `--ticket` **is** a real option (`src/Plugins/Snapshot.php:83`) — it narrows legitimately | PASS (vacuous) | +| C38 | `pest --tia --assignee=X` | RO; notice (`src/Plugins/Snapshot.php:81`) | PASS (vacuous) | +| C39 | `pest --tia --todo` | RO; notice; matches nothing | PASS (vacuous) | + +### D — Truncation → RESULTS-ONLY + +D1–D5, D11–D13, D16 precondition: `trio one` broken. D6–D10 run against a green suite so the `--stop-on-*` trigger is the only narrowing. + +| # | Case | Target outcome | Status | +|---|---|---|---| +| D1 | `pest --bail` | RO; `trio one` = `status=7`; siblings survive unchanged | PASS | +| D2 | `pest --retry` | RO; siblings survive | PASS | +| D3 | `pest --stop-on-failure` | RO | PASS | +| D4 | `pest --stop-on-defect` | RO | PASS | +| D5 | `pest --stop-on-error` (no error occurs) | COMPLETE — nothing stopped it | PASS | +| D6 | `pest --stop-on-warning` | RO when it fires. Warning stored as `status=0`, not `6` | PASS | +| D7 | `pest --stop-on-risky --disallow-test-output` | RO; `status=5` | PASS | +| D8 | `pest --stop-on-skipped` | RO; `status=1` | PASS | +| D9 | `pest --stop-on-incomplete` | RO; `status=2` | PASS | +| D10 | `pest --stop-on-deprecation` | RO. Deprecation stored as `status=0`, not `4` | PASS | +| D11 | `pest --tia --bail` | RO | PASS | +| D12 | `pest --bail --filter=trio` | RO (narrowed *and* truncated) | PASS | +| D13 | `stopOnFailure="true"` in `phpunit.xml`, plain `pest` | RO — caught via `stoppedEarly()`, not flag matching | PASS | +| D14 | D13 config, green suite | COMPLETE | PASS | +| D15 | `pest --bail`, last test in run order fails | RO — over-conservative but safe; nothing was skipped | PASS | +| D16 | `pest --tia --filtered --bail`, broken affected test | RO | PASS | + +### E — Result merge semantics + +| # | Case | Target outcome | Status | +|---|---|---|---| +| E1 | cached `flaky`=7, `FLAKY_OK=1 pest --filter=flaky` | flips to `status=0`; all other results byte-identical | PASS | +| E2 | cached `flaky`=0, `pest --filter=flaky` (env unset) | flips to `status=7` | PASS | +| E3 | after E1, `pest --tia --filtered` clean | `No affected tests found`; zero graph delta | PASS | +| E4 | after E2, `pest --tia --filtered` | re-runs, `from 1 previously unsuccessful test` | PASS | +| E5 | any partial run | `assertions` and `time` update for the test that ran | PASS | +| E6 | any partial run | `message` of untouched tests unchanged | PASS | +| E7 | two sequential partial runs on different tests | each updates only its own entry; both persist | PASS | +| E8 | `pest --filter="trio one"` | siblings keep exact `status`/`time`/`assertions`/`message` | PASS | +| E9 | partial run of a `->skip()`ed test | `status=1` | PASS | +| E10 | partial run of a `->todo()` test | `status=1`, `assertions=0`, `message="__TODO__"` | PASS | +| E11 | partial run of a risky test | `status=5` | PASS | +| E12 | any partial run | `fingerprint` byte-identical | PASS | +| E13 | dataset test, `--filter` matching one row | that row updates; other rows survive (the old prune bug) | PASS | +| E14 | dataset test, full `--tia` after deleting a row | the deleted row is pruned | PASS | + +### F — Guard rails + +| # | Case | Target outcome | Status | +|---|---|---|---| +| F1 | test absent from `edges`, `pest --filter="brand new"` | no result recorded | PASS | +| F2 | complete `pest --tia` first, then the same filter | result is recorded | PASS | +| F3 | `pest --mutate --path=app/Services --covered-only` | statuses/`edges`/`sha`/`tree`/`files` identical; only `time` may change | PASS | +| F4 | `pest --mutate --parallel --path=… --covered-only` | same (observed: zero delta) | PASS | +| F5 | weakened test so a mutant survives, `pest --mutate` | no status change in the baseline | PASS | +| F6 | `pest --mutate` with the graph deleted | no graph created | PASS | +| F7 | mutation subprocess env | carries `PEST_MUTATION_TESTING`; `--mutate` stripped from argv | PASS | +| F8 | grep baseline after `--mutate` | no mutation-flavoured messages | PASS | + +### G — Parallel + +| # | Case | Target outcome | Status | +|---|---|---|---| +| G1 | `pest --tia --parallel` (clean) | COMPLETE — graph written | PASS | +| G2 | `pest --tia --parallel --filter=adds` | RO | PASS | +| G3 | `pest --parallel --bail`, `trio one` broken | RO; siblings survive | PASS | +| G4 | `pest --tia --parallel --bail` | RO | PASS | +| G5 | `pest --tia --filtered --parallel` after a source edit | narrows to affected; writes | PASS (but see G12 — it strips edges) | +| G6 | `pest --tia --parallel` | worker results reach the parent baseline | PASS | +| G7 | `->only()` + `pest --tia --parallel` | RO | PASS | +| G8 | `pest --tia --parallel --shard=1/2` | RO | PASS | +| G9 | broken test in one worker, `pest --parallel --bail` | whole run is RO, not just that worker's slice | PASS | +| G10 | `pest --parallel --retry` | `InvalidOption`; graph untouched | PASS | +| G11 | `pest --tia --parallel --fresh` | graph rebuilt | PASS | +| G12 | `edges` after `pest --tia` vs `pest --tia --parallel --fresh` | **equivalent edge sets** (`files=25` both) | **FIX (item 1)** | + +### H — Baseline key / branch resolution + +| # | Case | Target outcome | Status | +|---|---|---|---| +| H1 | `pest --tia` on `master` | only a `master` key | PASS | +| H2 | plain `pest` on `master` | writes `master`, not `main` | PASS | +| H3 | `pest --bail` truncated on `master` | no `main` key minted | PASS | +| H4 | `pest --filter=adds` on `master` | no `main` key minted | PASS | +| H5 | `git branch -m main`, full `pest --tia` | single `main` key | PASS | +| H6 | `git checkout -b feature/x`, `pest --tia` | `feature/x` key appears; reads fall back to the existing baseline | PASS | +| H7 | detached HEAD, `pest --tia` | falls back to the `main` key for both reads and writes; no `HEAD` key | PASS | +| H8 | after every C and D case | no baseline key other than the real branch | PASS | +| H9 | non-git dir, `pest --tia` | `MissingDependency` — `The feature "Tia mode" requires "git".` | PASS | +| H10 | non-git dir, plain `pest` | runs normally; no baseline dir created | PASS | + +### I — Filtered mode + +| # | Case | Target outcome | Status | +|---|---|---|---| +| I1 | clean + green, `pest --tia --filtered` | `No affected tests found`; zero graph delta | PASS | +| I2 | edit `Calculator.php` | only `CalculatorTest` runs | PASS | +| I3 | cached failure, clean tree | re-runs it, `from 1 previously unsuccessful test` | PASS | +| I4 | I2 with sentinels | `tree` updated; exactly one entry rewritten; all others retained | PASS | +| I5 | `pest --tia --filtered --parallel` | same narrowing as I2 **and `edges` unchanged** | **FIX (item 1)** | +| I6 | `PEST_TIA_FILTERED=1 pest --tia` | identical delta to I4 | PASS | +| I7 | `pest --tia --filtered tests/Unit` | explicit path wins; filtered off; RO; notice | PASS | +| I8 | `pest --tia --filtered --coverage-text` | filtered mode disabled by an active coverage report | **FIX (item 3)** | +| I9 | cached failure whose test file was deleted | WARN `could not be located on disk`; falls back to the full suite with replay | **FIX (item 2)** | +| I10 | `pest --tia --filtered` with no baseline yet | records a baseline instead of filtering | PASS | +| I11 | edit a Blade view a Feature test renders | the Feature test is selected | PASS | +| I12 | edit `composer.lock` | fingerprint drift → full rebuild (and the drift reason should be printed — item 5) | PASS | + +### J — Interactions & regressions + +| # | Case | Target outcome | Status | +|---|---|---|---| +| J1 | `pest --tia --fresh --filter=adds` | graph **not** purged; RO | PASS | +| J2 | `pest --tia --refetch --filter=adds` | refetch not performed; RO | PASS | +| J3 | `pest --no-tia --filter=adds` | RO — narrowing wins over `--no-tia`; no notice | PASS | +| J4 | `pest --tia --no-tia` | TIA disabled; COMPLETE | PASS | +| J5 | `integration`, `pest --tia` | COMPLETE, no notice — an always-in-force filter applies to baseline runs too | PASS | +| J6 | `pest --tia --compact` | COMPLETE | PASS | +| J7 | `pest --tia -v` | COMPLETE | PASS | +| J8 | `pest --tia --profile` | COMPLETE | PASS | +| J9 | `pest --tia --order-by=random` | COMPLETE | PASS | +| J10 | `pest --tia --random-order-seed=1234` | COMPLETE | PASS | +| J11 | `pest --tia --repeat=2` | COMPLETE — repetition is not narrowing | **SKIP** — `--repeat` is not a Pest option | +| J12 | `pest --filter=adds` twice | only the executed test's `time` differs; rest byte-identical | PASS | +| J13 | `pest --tia --min=50` | COMPLETE | PASS | +| J14 | SIGINT mid-suite | RO — an interrupted run is truncated | PASS (signal must go to the re-exec'd child — item 5) | +| J15 | delete the graph, `pest --tia --filtered` | records a baseline rather than erroring | PASS | +| J16 | corrupt `graph.json`, `pest --tia` | recovers by rebuilding; no crash | PASS | + +--- + +## Re-run procedure + +```bash +cd /Users/nunomaduro/Work/projects/playground/laravel +GRAPH="$(PAO_DISABLE=1 ./vendor/bin/pest --baseline)/graph.json" + +# per case +git checkout -q . && git clean -qfd tests app +rm -rf "$(PAO_DISABLE=1 ./vendor/bin/pest --baseline)" +PAO_DISABLE=1 ./vendor/bin/pest --tia >/dev/null 2>&1 +# sentinel-patch every result to time=9.999 assertions=42, snapshot, run the case, diff +``` + +Every pest invocation must be prefixed with `PAO_DISABLE=1` (the app has `laravel/pao`, which emits +JSON when it detects an agent). The shell is zsh — build commands with arrays or `eval`; unquoted +`$args` does not word-split. diff --git a/PLAN_PHASE_THREE.md b/PLAN_PHASE_THREE.md new file mode 100644 index 00000000..4d2f8fc6 --- /dev/null +++ b/PLAN_PHASE_THREE.md @@ -0,0 +1,455 @@ +# TIA default-branch fallback — phase three + +## Your task + +Fix [pestphp/pest#1823](https://github.com/pestphp/pest/issues/1823) — TIA's cached results never +hit for repos whose default branch is not literally `main` — then re-run the **whole** conformance +matrix against the playground app, plus the new section **L** that covers the fix. + +Work in this order, and **stop where the plan says stop**: + +1. **Part 1** — read the diagnosis. It is already measured; do not re-derive it, but do re-confirm + the two reproductions in Part 1.3 take ~2 minutes and prove your environment is sane. +2. **Part 2** — implement the fix in the `pestphp/pest` repo. **Do not commit. Do not touch the + playground's `vendor/`.** +3. **HARD STOP → Part 3.** Report the diff to Nuno and wait. He validates, commits, and syncs it + into the playground himself. You must not proceed until he confirms. +4. **Part 4** — verify the sync landed, then run section **L** (new) and the full phase-two matrix + (**A–K**, all 156 rows) against the playground. +5. **Part 5** — report in the given format. + +Per `CLAUDE.md`: **do not write new `pestphp/pest` unit tests and do not run `composer test`.** Make +the change, report it, and ask whether repo tests should be added. The section-L rows are *playground +invocations*, not repo tests — those are the deliverable and are always in scope. + +--- + +## Part 1 — The diagnosis (already measured; commit `bfd5b756`) + +### 1.1 Root cause — two independent hardcoded `'main'` literals + +**(a) The read fallback.** `src/Plugins/Tia/Graph.php` — seven methods default the fallback to the +literal `'main'`: + +| line | method | +|---|---| +| 579 | `recordedAtSha(string $branch, string $fallbackBranch = 'main')` | +| 614 | `getAssertions(…, string $fallbackBranch = 'main')` | +| 625 | `getTime(…, string $fallbackBranch = 'main')` | +| 636 | `getResult(…, string $fallbackBranch = 'main')` | +| 663 | `testFilesToRerun(string $branch, string $fallbackBranch = 'main')` | +| 700 | `hasUnlocatedTestsToRerun(string $branch, string $fallbackBranch = 'main')` | +| 811 | `lastRunTree(string $branch, string $fallbackBranch = 'main')` | + +They all funnel into `Graph::baselineFor()` (line 819), which *does* implement a real cross-branch +fallback: + +```php +if (isset($this->baselines[$branch])) return $this->baselines[$branch]; +if ($branch !== $fallbackBranch && isset($this->baselines[$fallbackBranch])) return $this->baselines[$fallbackBranch]; +return ['sha' => null, 'tree' => [], 'results' => []]; +``` + +The mechanism is deliberate. The problem is that **no caller ever passes `$fallbackBranch`** — all +nine call sites in `src/Plugins/Tia.php` (lines 419, 429, 432, 785, 857, 1026, 1031, 1048, 1056) pass +only `$this->branch`. So on a `master`-named repo the second branch of `baselineFor()` can never +fire, and the whole mechanism is dead code. + +`grep -rniE 'defaultBranch|symbolic-ref|init\.defaultBranch|origin/HEAD' src/` returns **nothing** — +no default-branch resolution exists anywhere. + +**(b) The detached-HEAD default.** `src/Plugins/Tia.php:206` declares `private string $branch = 'main';` +and `ChangedFiles::currentBranch()` (`ChangedFiles.php:208`) returns `null` for detached HEAD. So on +detached HEAD `$this->branch` stays the literal `'main'` and is used for **both reads and writes** — +minting a baseline key for a branch that does not exist. This is a *write*-side bug and a separate fix +from (a). + +### 1.2 Not a regression + +`git log -S"fallbackBranch = 'main'"` bottoms out at `c7e32f5d feat(tia): continues to work on poc`. +This is original PoC code, untouched by phase one. Phase one's change 1 modified +`hasUnlocatedTestsToRerun()`'s file-existence check — one of the seven methods — without going near +the fallback. Do not report it as a phase-one regression. + +No test anywhere exercises the mechanism: `tests/Unit/Plugins/Tia/Graph.php` uses `'main'` as the +*actual* branch name, so those assertions pass whether or not the fallback exists. A branch-name +mismatch is never tested. + +### 1.3 The two reproductions (re-confirm these before you start) + +Both on the playground, `master`-named default branch, zero local changes: + +``` +default = master default = main +1. record on default → full run (cold) → full run (cold) +2. 1st run on feature-x → 25 UNCACHED → 25 replayed ← (a) +3. 2nd run on feature-x → 25 replayed → 25 replayed +4. back on default → 25 replayed → 25 replayed +5. 1st run on feature-y → 25 UNCACHED → 25 replayed +``` + +``` +master-only graph, then `git checkout --detach`, then `pest --tia`: + → 25 uncached, and keys become [master,main] ← (b) spurious key +``` + +The cost is **one full run per new branch, forever**, with no output explaining why — the only clue +is the `N uncached` count; the headline is just `─ Experimental TIA mode enabled.` + +### 1.4 Correction to phase two + +Phase two reported **H6 and H7 as passing. Both were false passes.** They ran after H5, which had +renamed `master`→`main` and left a `main` key in the graph, so the hardcoded fallback resolved by +accident. Re-measured against a clean `master`-only graph, both fail. Section L replaces them as the +load-bearing rows; H6/H7 must be re-run **from a cold graph** this time (see Part 4.2). + +--- + +## Part 2 — The fix to implement + +### 2.1 Recommended design + +**Step 1 — add a non-throwing resolver** to `src/Plugins/Tia/ChangedFiles.php`, next to +`currentBranch()`: + +```php +public function defaultBranch(): ?string +``` + +Resolution order, each step failing soft to the next: + +1. `git symbolic-ref --short refs/remotes/origin/HEAD` → strip a leading `origin/` +2. `git config --get init.defaultBranch` +3. `null` + +Unlike `currentBranch()`, this must **never throw** `MissingDependency` — it is advisory. Return +`null` on any non-zero exit or empty output. + +**Step 2 — add a config surface.** `src/Plugins/Tia/Configuration.php` already exposes `always()`, +`locally()`, `filtered()`, `baselined()`, `watch()`. Add: + +```php +public function defaultBranch(string $branch): self +``` + +so `pest()->tia()->defaultBranch('master')` works in `tests/Pest.php`. Explicit config **always +wins** over autodetection — that is the escape hatch when `origin/HEAD` is unset. + +**Step 3 — resolve once, in `Tia.php`.** The read path is hot (`getResult()` is called per test at +line 419), so resolution must not shell out per call. Resolve alongside `$this->branch` at +`Tia.php:1910`, under the existing `$branchResolved` guard: + +```php +$this->fallbackBranch = $configuredDefaultBranch + ?? $changedFiles->defaultBranch() + ?? 'main'; +``` + +**Step 4 — thread it into `Graph`.** Prefer a `Graph`-level property over editing nine call sites: +add `Graph::setFallbackBranch(string $branch)`, change the seven signatures to +`?string $fallbackBranch = null`, and resolve inside each with +`$fallbackBranch ??= $this->fallbackBranch;`. `baselineFor()` itself needs no change. This keeps the +public signatures backward-compatible and minimises blast radius. + +**Step 5 — fix the detached-HEAD write.** `Tia.php:206`'s `= 'main'` default must become the +resolved default branch, so detached HEAD stops minting a phantom key. + +### 2.2 Invariants the fix must not break + +These are all covered by existing matrix rows — the fix is wrong if any of them moves: + +- **Read-only.** The fallback must affect *reads* only. Writes go through `ensureBaseline($branch)` + and must keep using the real current branch. Otherwise H1–H4/H8 ("no baseline key other than the + real branch") break. +- **H9** — a non-git dir with `--tia` must still raise + `MissingDependency: The feature "Tia mode" requires "git".` Adding a soft resolver must not + swallow that. +- **H10** — plain `pest` in a non-git dir must still run and create no baseline dir. +- **A2/A3** — cold-graph recording unchanged. +- **I1/E3** — clean+green `--tia --filtered` must still be a true zero-delta run. +- Filtered mode reads `testFilesToRerun()` and `hasUnlocatedTestsToRerun()`, so the fallback must + reach those two as well, not just `getResult()`. + +### 2.3 Decisions for Nuno (raise these at the Part 3 stop) + +- **D1** — Consult `origin/HEAD` at all? It requires `git remote set-head` and is absent in many CI + checkouts and all remote-less repos. Config + `init.defaultBranch` only is simpler but helps fewer + people out of the box. *Recommendation: keep it, first in the chain, since it fails soft.* +- **D2** — Single-key heuristic: if the graph holds exactly one baseline key, use it as the fallback? + Fixes the issue with zero git calls, but is implicit and surprising when several keys exist. + *Recommendation: no.* +- **D3** — Should detached HEAD write a baseline at all, or be read-only? Current behaviour mints a + key. *Recommendation: read-only.* +- **D4** — Should `pest()->tia()->defaultBranch()` validate that the branch exists, or accept any + string? *Recommendation: accept any string; a nonexistent name degrades to a full run, which is + safe.* + +--- + +## Part 3 — HARD STOP + +When the code is written: + +1. Show Nuno the diff (`git -C /Users/nunomaduro/Work/projects/pestphp/pest diff`) and a one-paragraph + summary of each file's change. +2. Answer/raise the D1–D4 decisions. +3. State explicitly that you have **not** committed and have **not** synced `vendor/`. +4. Ask whether repo unit tests should be added (per `CLAUDE.md`), describing the tests you have in + mind — do not write them yet. +5. **Wait.** Nuno commits and applies the change to the playground. + +**Never sync the playground's `vendor/` yourself.** `vendor/pestphp/pest` there is a dist copy, not a +symlink (composer installed `dev-fix/tia-filtered as 5.2.0`), so pest-repo edits do not reach it. Say +what is stale and wait. This applies to before/after contrasts too. + +Once he confirms, verify the sync actually landed before measuring anything: + +```bash +V=/Users/nunomaduro/Work/projects/playground/laravel/vendor/pestphp/pest +grep -c 'defaultBranch' "$V/src/Plugins/Tia/ChangedFiles.php" # must be ≥ 1 +for f in src/Plugins/Tia.php src/Plugins/Tia/Graph.php src/Plugins/Tia/ChangedFiles.php \ + src/Plugins/Tia/Configuration.php; do + diff -q "/Users/nunomaduro/Work/projects/pestphp/pest/$f" "$V/$f" >/dev/null \ + && echo "SAME $f" || echo "STALE $f" +done +``` + +If anything reports `STALE`, stop and tell him. State in your final report which pest commit produced +the playground numbers. + +--- + +## Part 4 — Measurement + +### 4.1 Environment and traps + +**Playground:** `/Users/nunomaduro/Work/projects/playground/laravel` (branch `master`). + +1. **Every** pest invocation needs `PAO_DISABLE=1` — the app has `laravel/pao`, which emits JSON when + it detects an agent. +2. **Pin the interpreter to `php85`.** The playground requires PHP `>= 8.4.1` *and* pcov. Locally only + `php85` (8.5.8) has both — `php84` (8.4.23) has no pcov, and the bare Herd `php` shim has been + observed drifting to 8.3.32 mid-session, which kills every run in + `vendor/composer/platform_check.php`. Also put a `php` → `php85` symlink first on `PATH`: the + `--shard` list-tests probe spawns a subprocess via bare `php`, not `PHP_BINARY`. +3. **Never run `git checkout .` or `git checkout -- composer.lock` in the playground.** It has four + pre-existing user-modified files — `AGENTS.md`, `CLAUDE.md`, `composer.json`, `composer.lock`. Scope + resets to `git checkout -- tests app` / `git clean -fd tests app`. For `phpunit.xml` and + `composer.lock`, copy aside and copy back, verifying with `shasum`. +4. **zsh does not word-split unquoted parameters.** A `$PEST` string containing a space becomes one + command name. Route every invocation through `eval` (the `pest()` helper below does this). +5. Comment-only edits to a test file are **not** changes (AST-level hashing). Use semantic edits. +6. **The sentinel technique must not falsify `assertions` on zero-assertion tests.** A + risky/skipped/incomplete status is *derived* from "performed no assertions", so patching those to + `42` rewrites the status on replay and destroys the discriminator — it shows up as a phantom + `status 5→0` defect. The `sentinel.php` below only patches `assertions` where it is already + non-zero. With that, a full replay gives a clean `rewritten=0`. +7. Restore branch state after every L row. Several rows rename or detach; leaving a stray `main` key + in the graph is exactly what produced phase two's false H6/H7 passes. + +### 4.2 Reference numbers + +The playground already carries the phase-two fixtures at commit `fb2e77e` — **do not rebuild them.** +A healthy sequential `--tia` graph is: + +> **`files=27`, 10 edge keys (one per test file), self-edge on all 10, `n=25` results.** + +`sha` will differ once Nuno commits the vendor sync — re-derive it once and use it throughout. The +`files`/`edges`/`n` numbers hold as long as no test fixture changes. Suite shape: 10 test files, 25 +tests, including six deliberate status fixtures (skipped, todo, incomplete, risky, warning, +deprecation), a 3-row dataset, a `smoke` group, an env-driven flaky test (green unless +`FLAKY_FAIL=1`), and the annotation set (`covers`/`note`/`flaky`/`issue`/`pr`/`ticket`/`assignee`). + +**Re-run H6 and H7 from a cold graph** (`rm -rf` the graph dir, record on `master` only, *then* +branch/detach). Their phase-two results are void. + +### 4.3 Harness + +Write these to your scratchpad. `$SP` is your own scratchpad dir. + +
+lib.sh + +```bash +#!/bin/zsh +export PAO_DISABLE=1 +PG=/Users/nunomaduro/Work/projects/playground/laravel +SP="" +PHPBIN="php85" +PEST="$PHPBIN $PG/vendor/bin/pest" +cd "$PG" || exit 1 +pest() { eval "$PEST $*"; } # zsh: no word-splitting, must eval +GRAPHDIR="$(pest --baseline)" +GRAPH="$GRAPHDIR/graph.json" +mkdir -p "$SP/bin" && ln -sf "$(command -v php85)" "$SP/bin/php" +export PATH="$SP/bin:$PATH" # --shard spawns bare `php` +reset_tree() { git checkout -- tests app 2>/dev/null; git clean -qfd tests app 2>/dev/null; } +seed() { rm -rf "$GRAPHDIR"; pest --tia >/dev/null 2>&1; $PHPBIN "$SP/sentinel.php" "$GRAPH" >/dev/null; cp "$GRAPH" "$SP/before.json"; } +snap() { cp "$GRAPH" "$SP/before.json"; } +delta() { $PHPBIN "$SP/cmp.php" "$SP/before.json" "$GRAPH"; } +keys() { $PHPBIN -r '$g=json_decode(file_get_contents($argv[1]),true);echo "[".implode(",",array_keys($g["baselines"]??[]))."]";' "$GRAPH"; } +tally() { sed -E $'s/\x1b\\[[0-9;]*[a-zA-Z]//g' <"$SP/out.txt" | grep -E 'Tests:' | sed -E 's/^ +//;s/Tests: +//'; } +``` +
+ +
+sentinel.php — the write discriminator + +```php + +$p = $argv[1]; +$g = json_decode((string) file_get_contents($p), true, 512, JSON_THROW_ON_ERROR); +$n = 0; +foreach ($g['baselines'] ?? [] as $br => $b) { + foreach (array_keys($b['results'] ?? []) as $id) { + $g['baselines'][$br]['results'][$id]['time'] = 9.999; + // Only falsify a non-zero assertion count: risky/skipped/incomplete are + // DERIVED from "performed no assertions", so patching those to 42 would + // rewrite the status on replay and destroy the discriminator. + if ((int) ($b['results'][$id]['assertions'] ?? 0) > 0) { + $g['baselines'][$br]['results'][$id]['assertions'] = 42; + } + $n++; + } +} +file_put_contents($p, json_encode($g, JSON_THROW_ON_ERROR)); +echo "sentinelled $n results\n"; +``` +
+ +
+oneline.php — one compact tier verdict per row + +```php + +function load(string $p): ?array { + return is_file($p) ? json_decode((string) file_get_contents($p), true, 512, JSON_THROW_ON_ERROR) : null; +} +$a = load($argv[1]); $b = load($argv[2]); +if ($a === null || $b === null) { echo 'GRAPH '.($b === null ? 'DELETED' : 'CREATED'); exit; } +function edgeSets(array $g): array { + $out = []; + foreach ($g['edges'] ?? [] as $t => $ids) { $s = array_map(fn($i) => $g['files'][$i] ?? "?$i", (array) $ids); sort($s); $out[$t] = $s; } + ksort($out); return $out; +} +$moved = []; +if (edgeSets($a) !== edgeSets($b)) $moved[] = 'edges'; +if (($a['files'] ?? []) !== ($b['files'] ?? [])) $moved[] = 'files'; +if (($a['fingerprint'] ?? null) !== ($b['fingerprint'] ?? null)) $moved[] = 'fingerprint'; +$brA = array_keys($a['baselines'] ?? []); $brB = array_keys($b['baselines'] ?? []); +if ($brA !== $brB) $moved[] = 'branchkeys('.implode('|', $brA).'->'.implode('|', $brB).')'; +$add = $rem = $wr = 0; $shaMoved = $treeMoved = false; +foreach ($brB as $br) { + $ra = $a['baselines'][$br]['results'] ?? []; $rb = $b['baselines'][$br]['results'] ?? []; + if (($a['baselines'][$br]['sha'] ?? null) !== ($b['baselines'][$br]['sha'] ?? null)) $shaMoved = true; + if (($a['baselines'][$br]['tree'] ?? null) !== ($b['baselines'][$br]['tree'] ?? null)) $treeMoved = true; + $add += count(array_diff(array_keys($rb), array_keys($ra))); + $rem += count(array_diff(array_keys($ra), array_keys($rb))); + foreach ($ra as $id => $x) { + if (! isset($rb[$id])) continue; + foreach (['status','time','assertions','message'] as $f) { + if (($x[$f] ?? null) !== ($rb[$id][$f] ?? null)) { $wr++; break; } + } + } +} +$n = 0; foreach ($brB as $br) $n = max($n, count($b['baselines'][$br]['results'] ?? [])); +printf('n=%d w=%-2d +%d -%d %s%s%s', $n, $wr, $add, $rem, + $moved === [] ? 'struct:ok' : 'STRUCT:'.implode(',', $moved), + $shaMoved ? ' sha:CHANGED' : '', $treeMoved ? ' tree:chg' : ''); +``` +
+ +`cmp.php` (verbose per-entry version of the same) and `summarise.php` / `edgediff.php` are in +`PLAN_PHASE_TWO.md` §"Graph summariser" — reuse them for drill-downs. Reading the verdict: + +- `w=` — entries actually **written**. Under sentinel patching this is the only reliable way to tell + "wrote identical values" from "wrote nothing". A full replay must give `w=0`. +- `struct:ok` + `+0 -0` — no prune, no edges/files/fingerprint movement. Required by RESULTS-ONLY. +- `STRUCT:branchkeys(...)` — a new baseline key appeared. For section L this is the headline signal. + +### 4.4 Section L — new rows for this fix + +Tiers, unchanged from phase two: **COMPLETE** may change everything · **RESULTS-ONLY (RO)** may +change only `baselines[].results` for tests that ran, and must never remove an entry, add a +result for a test file absent from `edges`, or alter `sha`/`tree`/`edges`/`files`/`fingerprint` · +**HARD-SUPPRESSED** may change nothing. + +Every L row starts from a **cold graph recorded on the named default branch only** — verify +`keys=[]` before branching. Restore branch state afterwards. + +| # | Case | Target outcome | +|---|---|---| +| L1 | default `master`, record, `git switch -c feature-x`, `pest --tia`, zero changes | **all 25 replayed** (`w=0`), not `25 uncached`. The headline fix. | +| L2 | as L1 but default `main` | still all replayed — regression guard, this already worked | +| L3 | default `trunk`, then `develop` | replayed for both; the fix must not special-case two names | +| L4 | L1, then a *second* new branch `feature-y` | replayed too — the toll must not return per branch | +| L5 | L1 then edit `app/Services/Calculator.php` on `feature-x` | narrows to the 2 affected files (`CalculatorTest` + `AnnotationsTest`, which `covers` it); the other 17 replay | +| L6 | L1 with `--tia --filtered` | filtered mode reads the fallback too (`testFilesToRerun`, `hasUnlocatedTestsToRerun`) → `No affected tests found`, zero delta | +| L7 | L1 with `--tia --parallel` | fallback works in workers as well as the parent | +| L8 | L1, then `pest --tia` twice on `feature-x` | idempotent; second run also `w=0` | +| L9 | detached HEAD on a `master`-only graph | replays, and **no `main` key minted** — `keys` stays `[master]`. Bug (b). | +| L10 | `pest()->tia()->defaultBranch('master')` in `tests/Pest.php`, repo default renamed away | config wins over autodetect | +| L11 | config set to a nonexistent branch (`defaultBranch('nope')`) | degrades to a full run; no crash, no phantom key | +| L12 | no remote at all (`git remote remove origin` if present) | still resolves (via `init.defaultBranch`) or degrades safely — must not throw | +| L13 | branch name with a slash (`feature/x/y`) | replayed; no key-splitting bugs | +| L14 | L1, then confirm writes | `feature-x` gets its **own** key; the `master` key is **not** written to (fallback is read-only) | +| L15 | git worktree on a new branch (the issue's scenario) | replays from the default-branch baseline | +| L16 | non-git dir, `pest --tia` | still `MissingDependency: The feature "Tia mode" requires "git".` — the soft resolver must not swallow it | +| L17 | non-git dir, plain `pest` | runs normally; no baseline dir created | +| L18 | count `git` subprocesses during one `--tia` run | default-branch resolution is cached, not one call per test. Probe by shimming `git` on `PATH` to a logging wrapper. | + +L10–L11 need a `tests/Pest.php` edit — that file is tracked and **outside** the `tests app` reset +scope in practice (it lives in `tests/`, so `git checkout -- tests` does restore it; verify with +`git status` after). + +For L16/L17, build a throwaway non-git project — and note the trap that burned phase two: a +**symlinked** `vendor` makes Pest resolve the project root back to the playground (identical baseline +hash), silently invalidating the test. Use a hardlinked copy: + +```bash +NG="$SP/nogit"; rm -rf "$NG"; mkdir -p "$NG" +cp -R composer.json composer.lock phpunit.xml artisan tests app bootstrap config routes resources storage "$NG/" +[ -f .env ] && cp .env "$NG/" +cp -Rl vendor "$NG/vendor" || cp -R vendor "$NG/vendor" +``` + +Confirm the baseline path differs (`nogit-`, not `laravel-4a455a95622ac0ec`), and delete both +the temp project and its `~/.pest/tia/nogit-*` dir afterwards. + +### 4.5 Re-run the phase-two matrix (A–K, 156 rows) + +Re-run every row of `PLAN_PHASE_TWO.md` Part 2 against the fixed build. No row's status is trusted +until re-measured — the fix touches `Graph`'s read path, which nearly every row exercises. Sections +**A, B, H, I** are the load-bearing ones here (H is branch-key resolution; I is filtered mode; both +consume the changed methods directly). **H6 and H7 must be re-derived from a cold graph** (Part 1.4). + +Most rows batch cheaply — phase two ran C1–C20 in one call at roughly one line of output each. Use +`oneline.php` for the sweep and `cmp.php` only to drill into anomalies. + +### 4.6 Known pre-existing failures — do not report as regressions + +| item | status | +|---|---| +| **G4 / G4b** — parallel replay clobbers cached `time` on all non-executed tests. `mergeWorkerReplayPartials()` takes `$result['time']` verbatim at `Tia.php:1451`, never routing through `resultTime()` as the sequential sites (1681, 1762) do. Assertions survive because workers replay those themselves. | Pre-existing (pre-fix had no preservation at all), **out of scope for phase three**. Report it as still-present; do not fix it unless Nuno asks. | +| **C19** — `--tia --uses=…` cannot be fixtured. TIA hard-errors on PHPUnit classes (`EnsureTiaIsRunningPestTestsOnly`), and Pest has no chainable `->uses()`. | **Expected behaviour per Nuno.** Verify the tier (`w=0`, RO, notice) and move on. Not a defect. | +| **J11** — `--repeat` is not a Pest option (`Unknown option "--repeat"`). | **Don't care per Nuno.** Mark SKIP. | +| **J10** — `--random-order-seed` alone exits 1 with a WARN. Identical without `--tia`. | Pre-existing Pest behaviour, unrelated. Tier still holds. | + +--- + +## Part 5 — Reporting + +Per row: **tier respected (yes/no)**, the **graph delta** under sentinel patching, and for any failure +the **pre-fix contrast** so a regression is told apart from a pre-existing defect. For a pre-fix +contrast you need `db70017c` (or `bfd5b756` for "before phase three") files swapped into the +playground's `vendor/` — that is a sync, so **ask Nuno first** and always restore afterwards. + +State which pest commit produced the numbers. Close with: + +1. Whether L1–L18 all pass (the fix works). +2. Whether A–K regressed anywhere relative to phase two's 154/156. +3. The D1–D4 decisions as implemented. +4. Anything still open — including G4, which will still be failing. + +Leave the playground on `master` with only the four user-modified files dirty, no stray branches, and +no leftover `~/.pest/tia/*` dirs beyond `laravel-4a455a95622ac0ec`. diff --git a/PLAN_PHASE_TWO.md b/PLAN_PHASE_TWO.md new file mode 100644 index 00000000..b952d220 --- /dev/null +++ b/PLAN_PHASE_TWO.md @@ -0,0 +1,466 @@ +# TIA write-tier conformance — phase two + +## Your task + +**Re-run all 156 rows of the matrix in Part 2 from scratch, against the playground app** at +`/Users/nunomaduro/Work/projects/playground/laravel`. Every row, including the ones already marked +VERIFIED or PASS — the point of phase two is that no row's status is trusted until it has been +re-measured against the current code. The `Phase 1` column is prior evidence and a hint at what to +watch, never a reason to skip a row. + +These are **not** the `pestphp/pest` repo's own tests (`composer test`) — do not run those. Each row is +a `pest` invocation against the playground's suite, followed by a diff of the TIA graph it wrote. + +Report, per row: tier respected (yes/no), the graph delta under sentinel patching, and — for any +failure — the pre-fix contrast so a regression is told apart from a pre-existing defect. Do **not** +stop at reading this file or summarising it; the deliverable is executed results. + +Work in this order: **Part 1 (environment traps) → Part 1b (build the fixtures — the playground has +none of them) → Part 2 (the matrix) → Part 3 (priorities)**. The matrix is too large for one context; +take it one lettered section at a time and report as you finish each. Sections A, B, G, I and K are +the load-bearing ones — do those first if you run short. + +Phase one implemented `PLAN.md` Part 1 items 1–4 plus three §5 items. This file turns the matrix into +a conformance check rather than a bug list. + +Code under test: `pestphp/pest` at `/Users/nunomaduro/Work/projects/pestphp/pest`, branch +`fix/tia-filtered`, commit **`bfd5b756`** or later. Verify with +`grep -c 'recordsEdgesInWorkers\|recordsEdges' src/Plugins/Tia.php` → at least 3 hits. Pre-fix +baseline for every contrast is **`db70017c`**. + +--- + +## Part 0 — What phase one changed + +| # | Change | Files | +|---|---|---| +| 1 | `hasUnlocatedTestsToRerun()` stats the file, so a deleted test file is "unlocated" | `src/Plugins/Tia/Graph.php` | +| 2 | `enterReplayMode()` uses `activateLinkTracking()` under piggyback coverage | `src/Plugins/Tia.php` | +| 3 | `enterReplayMode()` stamps `TIA_PIGGYBACK_COVERAGE` for workers | `src/Plugins/Tia.php` | +| 4 | `replaceEdges(…, keepExisting:)` — piggyback edges seed empty sets, never overwrite populated ones | `Graph.php`, `Tia.php` | +| 5 | `renderFreshGraph()` stops claiming "fresh graph" when the graph is kept; reason reworded to `recording a coverage baseline` | `Tia.php` | +| 6 | `COVERAGE_REPORT_FLAGS` + `coverageReportActive()` union over `originalArguments`; new `pestCoverageActive()` keeps the coverage-cache marker/hijack on Pest's own `--coverage` | `Tia.php` | +| 7 | `Tia::recordsEdgesInWorkers()` + `WrapperRunner::handleTia()` inject `-d pcov.directory=` into worker argv | `Tia.php`, `src/Plugins/Parallel/Paratest/WrapperRunner.php` | +| 8 | Sequential record runs announce structural drift via `renderFreshGraph()` | `Tia.php` | +| 9 | `Graph::getTime()` + `cachedTimeByTestId` + `resultTime()` preserve replayed durations; edge-less write guard is now `$recordsEdges = $complete && ($markKnownTestFiles \|\| $this->recordingActive)` | `Graph.php`, `Tia.php` | + +Deliberately **not** done: `PLAN.md` §5 SIGINT propagation, §5 warning/deprecation `status=0` +mapping, and all of §6 (playground annotation fixtures). The vacuous C rows below stay vacuous. + +### Target-outcome changes this forces + +Two rows in the original matrix asserted the **old**, buggy behaviour. Their targets are updated +below — do not report them as regressions: + +- **B10** was "`time` differs; statuses stable". Change 9 means replayed entries now **keep** their + recorded `time`. New target: `time` differs only for tests that actually executed. +- **B12** gains an edge-preservation assertion it never had (see K1). + +### One known-failing repo test + +`tests/Unit/Plugins/Tia/Graph.php:69-76` asserts `hasUnlocatedTestsToRerun('main')` is `false` for +`tests/Feature/FooTest.php` under `new Graph(sys_get_temp_dir())` — a path that does not exist. That +assertion encodes the I9 bug and **will fail** under change 1. It needs re-pointing at a root/file +that exists (e.g. `dirname(__DIR__, 4)` + `'tests/Unit/Plugins/Tia/Graph.php'`). Left untouched by +request; it is a repo-test matter, not a playground one. + +--- + +## Part 1 — Environment, and the traps in it + +**Playground:** `/Users/nunomaduro/Work/projects/playground/laravel` + +```bash +cd /Users/nunomaduro/Work/projects/playground/laravel +GRAPH="$(PAO_DISABLE=1 ./vendor/bin/pest --baseline)/graph.json" # ~/.pest/tia/laravel-4a455a95622ac0ec +``` + +1. **Every** pest invocation needs `PAO_DISABLE=1` — the app has `laravel/pao`, which emits JSON + when it detects an agent. +2. **Never run `git checkout .` or `git checkout -- composer.lock` in the playground.** It has four + pre-existing user-modified files — `AGENTS.md`, `CLAUDE.md`, `composer.json`, `composer.lock` — + that a blanket reset would destroy. `PLAN.md`'s "reset ritual" is unsafe as written. Scope resets + to what you touched: `git checkout -- tests app` / `git clean -fd tests app`, and for + `composer.lock` (needed for the drift rows) **copy it aside and copy it back**, verifying with + `shasum`. +3. **`vendor/pestphp/pest` is a dist copy, not a symlink.** Composer installed + `dev-fix/tia-filtered as 5.2.0`, so edits in the pest repo do **not** reach the playground. + + **Tell Nuno whenever a sync is needed to move forward — do not sync silently.** Say what is stale + and what the sync would be, then wait. This includes temporarily swapping in `db70017c` files for a + before/after contrast. When you report any playground result, state which commit produced it. + The sync itself, once he agrees: + ```bash + PEST=/Users/nunomaduro/Work/projects/pestphp/pest + V=/Users/nunomaduro/Work/projects/playground/laravel/vendor/pestphp/pest + for f in src/Plugins/Tia.php src/Plugins/Tia/Graph.php \ + src/Plugins/Parallel/Paratest/WrapperRunner.php; do cp "$PEST/$f" "$V/$f"; done + ``` + Verify with `grep -c recordsEdgesInWorkers "$V/src/Plugins/Tia.php"` → `1`. **Always restore the + current version before continuing** after a pre-fix contrast. +4. **Coverage driver:** pcov only, no xdebug. `ini_get('pcov.directory')` is `''` by default — that + emptiness is the entire mechanism behind G12. +5. **Suite shape as found:** 7 tests in 5 files — `tests/Unit/{CalculatorTest,ExampleTest,GreeterTest, + TrioTest}.php`, `tests/Feature/ExampleTest.php`. A healthy sequential graph is **`files=22`, + 5 edge keys, self-edge on every test** (`PLAN.md`'s `files=25` is stale). Pre-fix parallel gives + `files=4` with zero self-edges. **These numbers shift the moment you add the Part 1b fixtures** — + re-derive them once, after the fixtures land, and use the new numbers throughout. The invariants + that do *not* shift: sequential and parallel must agree, and every test must have a self-edge. +6. **Sentinel patching is the only reliable discriminator** between "wrote identical values" and + "wrote nothing": rewrite every cached result to `time=9.999 assertions=42`, snapshot, run the + case, diff. A canary test absent from `edges` is unreliable under `--filter` because it never + matches the filter and so never runs. +7. Seeding a cached failure needs `--fresh` (or an env-driven flaky fixture): `--tia` on a clean + green tree replays rather than executes, so it can never cache a failure. Working recipe — break + an assertion, `pest --tia --fresh`, then restore the source. +8. Comment-only edits to a test file are **not** changes (AST-level hashing). Use semantic edits. +9. The shell is zsh — build commands with arrays or `eval`; unquoted `$args` does not word-split. + +### Graph summariser + +Write this to a scratch path and use it for every diff. + +```php + [label] +$g = json_decode((string) file_get_contents($argv[1]), true, 512, JSON_THROW_ON_ERROR); +$files = $g['files'] ?? []; $edges = $g['edges'] ?? []; +echo ($argv[2] ?? $argv[1])."\n files=".count($files).' edges keys='.count($edges)."\n"; +$dirs = []; foreach ($files as $f) { $dirs[explode('/', (string) $f)[0]] = true; } +ksort($dirs); echo ' file dirs: '.implode(', ', array_keys($dirs))."\n"; +ksort($edges); +foreach ($edges as $test => $ids) { + $self = 'no'; + foreach ((array) $ids as $id) { if (($files[$id] ?? null) === $test) { $self = 'YES'; break; } } + echo sprintf(" %-42s n=%-3d self=%s\n", $test, count((array) $ids), $self); +} +foreach ($g['baselines'] ?? [] as $branch => $b) { + $r = $b['results'] ?? []; + echo " baseline[$branch]: n=".count($r).' sha='.substr((string) ($b['sha'] ?? '-'), 0, 7)."\n"; + foreach ($r as $id => $x) { + echo sprintf(" %-58s status=%d time=%s asserts=%d file=%s\n", substr((string) $id, -58), + $x['status'], $x['time'], $x['assertions'], $x['file'] ?? '-'); + } +} +``` + +Normalised edge-set equality (for G12 / I5 / K1): + +```php + +function edges(string $p): array { + $g = json_decode((string) file_get_contents($p), true, 512, JSON_THROW_ON_ERROR); $out = []; + foreach ($g['edges'] as $t => $ids) { $s = array_map(fn ($i) => $g['files'][$i], $ids); sort($s); $out[$t] = $s; } + ksort($out); return $out; +} +$a = edges($argv[1]); $b = edges($argv[2]); +echo $a === $b ? "IDENTICAL edge sets\n" : "DIFFER\n"; +foreach ($a as $t => $s) { + $m = array_diff($s, $b[$t] ?? []); $e = array_diff($b[$t] ?? [], $s); + if ($m || $e) printf(" %s: -%d +%d\n", $t, count($m), count($e)); +} +``` + +### Per-case loop + +```bash +git checkout -- tests app 2>/dev/null; git clean -qfd tests app +rm -rf "$(PAO_DISABLE=1 ./vendor/bin/pest --baseline)" +PAO_DISABLE=1 ./vendor/bin/pest --tia >/dev/null 2>&1 # seed a healthy graph +# sentinel-patch every result to time=9.999 assertions=42, snapshot, run the case, diff +``` + +--- + +## Part 1b — Fixtures you must build first + +**The playground has none of the fixtures the matrix depends on.** Verified inventory: the only test +files are `tests/Unit/{CalculatorTest,ExampleTest,GreeterTest,TrioTest}.php` and +`tests/Feature/ExampleTest.php` — 7 plain tests, zero occurrences of `->group()`, `->only()`, +`->skip()`, `->todo()`, `->note()`, `->flaky()`, `->covers()`, `->uses()`, `->issue()`, `->pr()`, +`->ticket()`, `->assignee()`, datasets, or any `FLAKY_OK`-style env hook. `phpunit.xml` defines only +the `Unit` and `Feature` testsuites. + +So roughly 30 rows below cannot run as written until you create the fixtures. Build them **all up +front, in one commit**, then re-derive the baseline graph shape once and use those numbers for the +whole sweep — every fixture you add changes `files`, the `edges` key count and `n`, so adding them +piecemeal invalidates earlier rows. + +| Fixture to create | Rows that need it | +|---|---| +| a test with `->group('smoke')` | C7, C8, C9, C10 | +| an env-driven flaky test (passes iff `FLAKY_OK=1`) | E1, E2, E3, E4 | +| a `->skip()`ed test | E9, D8 | +| a `->todo()` test | E10, C23, C39 | +| a risky test (no assertions; pair with `--disallow-test-output`) | E11, D7 | +| a test that triggers a PHPUnit warning | D6 | +| a test calling `markTestIncomplete()` | D9 | +| a test that triggers a deprecation | D10 | +| a dataset test with ≥3 rows | E13, E14 | +| `->only()` — added and removed per case, not left in | C31, C32, G7 | +| `->covers(App\Services\Calculator::class)` | C18 | +| `->uses(...)` / `UsesClass` annotation | C19 | +| `->note(...)` | C24 | +| `->flaky()` **annotation** (distinct from the env-driven flaky test above) | C25 | +| `->issue(123)`, `->pr(1)`, `->ticket('X')`, `->assignee('X')` | C26, C27, C28, C37, C38 | + +This is `PLAN.md` §6's "test-harness gaps to close before re-running", now itemised: without these, +the listed rows' graph-invariant assertions match zero tests and **prove nothing** — they pass +vacuously. Any row still marked "(vacuous)" in Part 2 is vacuous *only because* its fixture is +missing; once you add the fixture, treat the row as unverified and make it load-bearing. + +Rows needing an *action* rather than a fixture — breaking `trio one` for the D rows, an uncommitted +test edit for `--dirty`, branch renames and a non-git dir for H5–H10, `--mutate` against +`app/Services` for the F rows — are fine as written; `pestphp/pest-plugin-mutate` is installed. + +--- + +## Part 2 — Full case matrix + +Tiers: **COMPLETE** may change everything · **RESULTS-ONLY** (RO) may change only +`baselines[].results` for tests that ran, and must never remove an entry, add a result for a +test file absent from `edges`, or alter `sha`/`tree`/`edges`/`files`/`fingerprint` · +**HARD-SUPPRESSED** may change nothing. + +Phase-1 column: **VERIFIED** = re-run against the fixed code in the phase-one session, pre-fix +contrast captured · **PASS (sweep)** = passed in the original `db70017` sweep and *not* re-checked +since the changes — these are the bulk of phase two's work · **SKIP** = not runnable as written. + +### A — Setup & sanity + +| # | Case | Target outcome | Phase 1 | +|---|---|---|---| +| A1 | `composer show pestphp/pest` | version `dev-fix/tia-filtered` | PASS (sweep) | +| A2 | `pest --baseline` | prints an existing dir; exit 0 | VERIFIED | +| A3 | delete graph, `pest --tia` | graph.json created; one `edges` key per test file; one result per test | VERIFIED | +| A4 | `git status` after reset ritual | clean *except the four user-modified files* (see trap 2) | VERIFIED | +| A5 | `extension_loaded("pcov")` | `true` | VERIFIED | +| A6 | delete graph, plain `pest` | no graph created | PASS (sweep) | +| A7 | delete graph, `pest --filter=adds` | no graph created | PASS (sweep) | +| A8 | two consecutive `pest --tia` | second replays everything; `sha` unchanged | VERIFIED | + +### B — COMPLETE runs still write + +| # | Case | Target outcome | Phase 1 | +|---|---|---|---| +| B1 | `pest --tia` (clean) | replays; graph written; `sha` = `git rev-parse HEAD` | VERIFIED | +| B2 | `pest --no-tia` | results refreshed; prune applied; `sha`/`tree`/`edges` unchanged | PASS (sweep) — recheck under change 9 | +| B3 | `pest` (plain) | same as B2 | PASS (sweep) — recheck under change 9 | +| B4 | `pest --tia --fresh` | graph purged and rebuilt; `edges` rebuilt; `files` repopulated | VERIFIED | +| B5 | `pest --bail`, all green | COMPLETE — graph written, prune applied | PASS (sweep) | +| B6 | edit `Calculator.php`, `pest --tia` | `CalculatorTest` re-runs, others replay; `tree` updated | PASS (sweep) | +| B7 | delete `trio three`, `pest --tia` | its result pruned; `trio one`/`two` remain | PASS (sweep) | +| B8 | delete `GreeterTest.php`, `pest --tia` | result and edges survive (missing-file prune is `--fresh`-only) | PASS (sweep) | +| B9 | B8 then `pest --tia --fresh` | entry gone; `files` drops `Greeter.php` too | PASS (sweep) | +| B10 | `pest --tia` twice | **UPDATED TARGET:** `time` differs only for tests that executed; replayed entries keep their recorded `time`; statuses stable | VERIFIED | +| B11 | add a test file, `pest --tia` | new `edges` key **and** new result appear in the **same** run | VERIFIED (regression guard for change 9) | +| B12 | `pest --tia --coverage` | completes; graph written; coverage report prints | VERIFIED | + +### C — Selection narrowing → RESULTS-ONLY + +All rows: RO invariants hold. "Notice" = `TIA does not apply to partial runs — running the selected tests directly.` + +| # | Case | Target outcome | Phase 1 | +|---|---|---|---| +| C1 | `pest --filter="adds numbers"` | RO; no notice | PASS (sweep) | +| C2 | `pest --tia --filter="adds numbers"` | RO; notice | PASS (sweep) | +| C3 | `pest --filter="trio one"` | RO; `trio two`/`three` byte-identical | PASS (sweep) | +| C4 | `pest --filter="trio"` | RO; all three update; nothing else does | PASS (sweep) | +| C5 | `pest --exclude-filter="Feature"` | RO; no notice; no prune | PASS (sweep) | +| C6 | `pest --tia --exclude-filter="Feature"` | RO; notice | PASS (sweep) | +| C7 | `pest --group=smoke` | RO; no notice | PASS (sweep) | +| C8 | `pest --tia --group=smoke` | RO; notice | PASS (sweep) | +| C9 | `pest --exclude-group=smoke` | RO; no notice; no prune | PASS (sweep) | +| C10 | `pest --tia --exclude-group=smoke` | RO; notice | PASS (sweep) | +| C11 | `pest tests/Unit` | RO; no notice; Feature entry not pruned | PASS (sweep) | +| C12 | `pest --tia tests/Unit` | RO; notice | PASS (sweep) | +| C13 | `pest tests/Unit/TrioTest.php` | RO; no notice | PASS (sweep) | +| C14 | `pest --tia tests/Unit/TrioTest.php` | RO; notice | PASS (sweep) | +| C15 | `pest --testsuite=Unit` | RO; no notice | PASS (sweep) | +| C16 | `pest --tia --testsuite=Unit` | RO; notice | PASS (sweep) | +| C17 | `pest --tia --exclude-testsuite=Feature` | RO; notice | PASS (sweep) | +| C18 | `pest --tia --covers='App\Services\Calculator'` | RO; notice | PASS (sweep) | +| C19 | `pest --tia --uses='App\Services\Calculator'` | RO; notice | PASS (vacuous — needs fixture) | +| C20 | `pest --tia --test-suffix=Test.php` | RO even though all tests run; `edges`/`files`/`n` unchanged | PASS (sweep) | +| C21 | `pest --dirty` with an uncommitted test edit | RO; no notice | PASS (sweep) | +| C22 | `pest --tia --dirty` | RO; notice | PASS (sweep) | +| C23 | `pest --tia --todos` | RO; notice | PASS (vacuous) | +| C24 | `pest --tia --notes` | RO; notice | PASS (vacuous) | +| C25 | `pest --tia --flaky` | RO; notice | PASS (vacuous) | +| C26 | `pest --tia --issue=123` | RO; notice | PASS (vacuous) | +| C27 | `pest --tia --pr=1` | RO; notice | PASS (vacuous) | +| C28 | `pest --tia --pull-request=1` | RO; notice | PASS (vacuous) | +| C29 | `pest --tia --shard=1/2` | RO; notice; no prune | PASS (sweep) | +| C30 | `pest --tia --shard=2/2` | RO; notice; C29+C30 covers the suite; neither prunes | PASS (sweep) | +| C31 | `->only()` on `trio one`, `pest --tia` | RO; no notice; siblings untouched | PASS (sweep) | +| C32 | `->only()` on `trio one`, plain `pest` | RO; no notice | PASS (sweep) | +| C33 | `pest --tia --filtered --filter=adds` | RO; notice; filtered yields to narrowing | PASS (sweep) | +| C34 | `PEST_TIA=1 pest --filter=adds` | RO; notice | PASS (sweep) | +| C35 | `PEST_TIA_FILTERED=1 pest --filter=adds` | RO; notice (regression guard) | PASS (sweep) | +| C36 | `pest --filtered --filter=adds` | RO; notice | PASS (sweep) | +| C37 | `pest --tia --ticket=X` | RO; notice. `--ticket` **is** a real option (`src/Plugins/Snapshot.php:83`) — it narrows legitimately | PASS (vacuous) | +| C38 | `pest --tia --assignee=X` | RO; notice (`src/Plugins/Snapshot.php:81`) | PASS (vacuous) | +| C39 | `pest --tia --todo` | RO; notice; matches nothing | PASS (vacuous) | + +### D — Truncation → RESULTS-ONLY + +D1–D5, D11–D13, D16 precondition: `trio one` broken. D6–D10 run against a green suite so the `--stop-on-*` trigger is the only narrowing. + +| # | Case | Target outcome | Phase 1 | +|---|---|---|---| +| D1 | `pest --bail` | RO; `trio one` = `status=7`; siblings survive unchanged | PASS (sweep) | +| D2 | `pest --retry` | RO; siblings survive | PASS (sweep) | +| D3 | `pest --stop-on-failure` | RO | PASS (sweep) | +| D4 | `pest --stop-on-defect` | RO | PASS (sweep) | +| D5 | `pest --stop-on-error` (no error occurs) | COMPLETE — nothing stopped it | PASS (sweep) | +| D6 | `pest --stop-on-warning` | RO when it fires. Warning stored as `status=0`, not `6` | PASS (sweep) | +| D7 | `pest --stop-on-risky --disallow-test-output` | RO; `status=5` | PASS (sweep) | +| D8 | `pest --stop-on-skipped` | RO; `status=1` | PASS (sweep) | +| D9 | `pest --stop-on-incomplete` | RO; `status=2` | PASS (sweep) | +| D10 | `pest --stop-on-deprecation` | RO. Deprecation stored as `status=0`, not `4` | PASS (sweep) | +| D11 | `pest --tia --bail` | RO | PASS (sweep) | +| D12 | `pest --bail --filter=trio` | RO (narrowed *and* truncated) | PASS (sweep) | +| D13 | `stopOnFailure="true"` in `phpunit.xml`, plain `pest` | RO — caught via `stoppedEarly()`, not flag matching | PASS (sweep) | +| D14 | D13 config, green suite | COMPLETE | PASS (sweep) | +| D15 | `pest --bail`, last test in run order fails | RO — over-conservative but safe; nothing was skipped | PASS (sweep) | +| D16 | `pest --tia --filtered --bail`, broken affected test | RO | PASS (sweep) | + +### E — Result merge semantics + +| # | Case | Target outcome | Phase 1 | +|---|---|---|---| +| E1 | cached `flaky`=7, `FLAKY_OK=1 pest --filter=flaky` | flips to `status=0`; all other results byte-identical | PASS (sweep) | +| E2 | cached `flaky`=0, `pest --filter=flaky` (env unset) | flips to `status=7` | PASS (sweep) | +| E3 | after E1, `pest --tia --filtered` clean | `No affected tests found`; zero graph delta | PASS (sweep) | +| E4 | after E2, `pest --tia --filtered` | re-runs, `from 1 previously unsuccessful test` | PASS (sweep) | +| E5 | any partial run | `assertions` and `time` update for the test that ran (it executed, so change 9 does not apply) | PASS (sweep) | +| E6 | any partial run | `message` of untouched tests unchanged | PASS (sweep) | +| E7 | two sequential partial runs on different tests | each updates only its own entry; both persist | PASS (sweep) | +| E8 | `pest --filter="trio one"` | siblings keep exact `status`/`time`/`assertions`/`message` | PASS (sweep) | +| E9 | partial run of a `->skip()`ed test | `status=1` | PASS (sweep) | +| E10 | partial run of a `->todo()` test | `status=1`, `assertions=0`, `message="__TODO__"` | PASS (sweep) | +| E11 | partial run of a risky test | `status=5` | PASS (sweep) | +| E12 | any partial run | `fingerprint` byte-identical | PASS (sweep) | +| E13 | dataset test, `--filter` matching one row | that row updates; other rows survive (the old prune bug) | PASS (sweep) | +| E14 | dataset test, full `--tia` after deleting a row | the deleted row is pruned | PASS (sweep) | + +### F — Guard rails + +| # | Case | Target outcome | Phase 1 | +|---|---|---|---| +| F1 | test absent from `edges`, `pest --filter="brand new"` | no result recorded | VERIFIED | +| F2 | complete `pest --tia` first, then the same filter | result is recorded | VERIFIED | +| F3 | `pest --mutate --path=app/Services --covered-only` | statuses/`edges`/`sha`/`tree`/`files` identical; only `time` may change | PASS (sweep) | +| F4 | `pest --mutate --parallel --path=… --covered-only` | same (observed: zero delta) | PASS (sweep) — recheck under change 7 | +| F5 | weakened test so a mutant survives, `pest --mutate` | no status change in the baseline | PASS (sweep) | +| F6 | `pest --mutate` with the graph deleted | no graph created | PASS (sweep) | +| F7 | mutation subprocess env | carries `PEST_MUTATION_TESTING`; `--mutate` stripped from argv | PASS (sweep) | +| F8 | grep baseline after `--mutate` | no mutation-flavoured messages | PASS (sweep) | + +### G — Parallel + +| # | Case | Target outcome | Phase 1 | +|---|---|---|---| +| G1 | `pest --tia --parallel` (clean) | COMPLETE — graph written; all 7 results present | VERIFIED | +| G2 | `pest --tia --parallel --filter=adds` | RO | PASS (sweep) | +| G3 | `pest --parallel --bail`, `trio one` broken | RO; siblings survive | PASS (sweep) | +| G4 | `pest --tia --parallel --bail` | RO | PASS (sweep) | +| G5 | `pest --tia --filtered --parallel` after a source edit | narrows to affected; writes | VERIFIED (see I5) | +| G6 | `pest --tia --parallel` | worker results reach the parent baseline | VERIFIED | +| G7 | `->only()` + `pest --tia --parallel` | RO | PASS (sweep) | +| G8 | `pest --tia --parallel --shard=1/2` | RO | PASS (sweep) | +| G9 | broken test in one worker, `pest --parallel --bail` | whole run is RO, not just that worker's slice | PASS (sweep) | +| G10 | `pest --parallel --retry` | `InvalidOption`; graph untouched | PASS (sweep) | +| G11 | `pest --tia --parallel --fresh` | graph rebuilt | VERIFIED | +| G12 | `edges` after `pest --tia` vs `pest --tia --parallel --fresh` | **equivalent edge sets** — `files=22`, self-edge on all 5 tests, both | **VERIFIED (fixed)** — pre-fix: `files=4`, 0 self-edges | + +### H — Baseline key / branch resolution + +| # | Case | Target outcome | Phase 1 | +|---|---|---|---| +| H1 | `pest --tia` on `master` | only a `master` key | VERIFIED | +| H2 | plain `pest` on `master` | writes `master`, not `main` | PASS (sweep) | +| H3 | `pest --bail` truncated on `master` | no `main` key minted | PASS (sweep) | +| H4 | `pest --filter=adds` on `master` | no `main` key minted | PASS (sweep) | +| H5 | `git branch -m main`, full `pest --tia` | single `main` key | PASS (sweep) | +| H6 | `git checkout -b feature/x`, `pest --tia` | `feature/x` key appears; reads fall back to the existing baseline | PASS (sweep) | +| H7 | detached HEAD, `pest --tia` | falls back to the `main` key for both reads and writes; no `HEAD` key | PASS (sweep) | +| H8 | after every C and D case | no baseline key other than the real branch | PASS (sweep) | +| H9 | non-git dir, `pest --tia` | `MissingDependency` — `The feature "Tia mode" requires "git".` | PASS (sweep) | +| H10 | non-git dir, plain `pest` | runs normally; no baseline dir created | PASS (sweep) | + +### I — Filtered mode + +| # | Case | Target outcome | Phase 1 | +|---|---|---|---| +| I1 | clean + green, `pest --tia --filtered` | `No affected tests found`; zero graph delta | PASS (sweep) | +| I2 | edit `Calculator.php` | only `CalculatorTest` runs | VERIFIED | +| I3 | cached failure, clean tree | re-runs it, `from 1 previously unsuccessful test` | PASS (sweep) | +| I4 | I2 with sentinels | `tree` updated; exactly one entry rewritten; all others retained | PASS (sweep) | +| I5 | `pest --tia --filtered --parallel` | same narrowing as I2 **and `edges` unchanged** | **VERIFIED (fixed)** — 1 affected test, edges identical | +| I6 | `PEST_TIA_FILTERED=1 pest --tia` | identical delta to I4 | PASS (sweep) | +| I7 | `pest --tia --filtered tests/Unit` | explicit path wins; filtered off; RO; notice | PASS (sweep) | +| I8 | `pest --tia --filtered --coverage-text` | filtered mode disabled by an active coverage report → full suite runs | **VERIFIED (fixed)** — pre-fix: `No affected tests found` | +| I9 | cached failure whose test file was deleted | WARN `Some cached tests due a re-run could not be located on disk` + `Running the full suite with replay instead of a filtered run` | **VERIFIED (fixed)** — pre-fix: `No tests found`, exit 0 green | +| I10 | `pest --tia --filtered` with no baseline yet | records a baseline instead of filtering | PASS (sweep) | +| I11 | edit a Blade view a Feature test renders | the Feature test is selected | PASS (sweep) | +| I12 | edit `composer.lock` | fingerprint drift → full rebuild, **and the reason is printed sequentially**: `fresh graph (composer.lock changed)` | **VERIFIED (fixed)** — pre-fix: bare `Running in TIA mode.` | + +### J — Interactions & regressions + +| # | Case | Target outcome | Phase 1 | +|---|---|---|---| +| J1 | `pest --tia --fresh --filter=adds` | graph **not** purged; RO | PASS (sweep) | +| J2 | `pest --tia --refetch --filter=adds` | refetch not performed; RO | PASS (sweep) | +| J3 | `pest --no-tia --filter=adds` | RO — narrowing wins over `--no-tia`; no notice | PASS (sweep) | +| J4 | `pest --tia --no-tia` | TIA disabled; COMPLETE | PASS (sweep) | +| J5 | `integration`, `pest --tia` | COMPLETE, no notice — an always-in-force filter applies to baseline runs too | PASS (sweep) | +| J6 | `pest --tia --compact` | COMPLETE | PASS (sweep) | +| J7 | `pest --tia -v` | COMPLETE | PASS (sweep) | +| J8 | `pest --tia --profile` | COMPLETE | PASS (sweep) | +| J9 | `pest --tia --order-by=random` | COMPLETE | PASS (sweep) | +| J10 | `pest --tia --random-order-seed=1234` | COMPLETE | PASS (sweep) | +| J11 | `pest --tia --repeat=2` | COMPLETE — repetition is not narrowing | **SKIP** — `--repeat` is not a Pest option | +| J12 | `pest --filter=adds` twice | only the executed test's `time` differs; rest byte-identical | PASS (sweep) | +| J13 | `pest --tia --min=50` | COMPLETE (silent no-op without `--coverage`) | PASS (sweep) | +| J14 | SIGINT mid-suite | RO — an interrupted run is truncated | PASS (sweep) — signal still does not reach the re-exec'd child (not fixed) | +| J15 | delete the graph, `pest --tia --filtered` | records a baseline rather than erroring | PASS (sweep) | +| J16 | corrupt `graph.json`, `pest --tia` | recovers by rebuilding; no crash | PASS (sweep) | + +### K — New rows for the phase-one fixes + +These assert behaviour no original row covered. All were verified in phase one; re-run them as +regression guards. + +| # | Case | Target outcome | Phase 1 | +|---|---|---|---| +| K1 | healthy `pest --tia` graph, then `pest --tia --coverage` | `edges` **byte-identical** — piggyback data may seed empty sets, never narrow populated ones; `files=22` stays `22` | **VERIFIED (fixed)** — pre-fix: `ExampleTest` 16→2 edges, self-edges lost | +| K2 | same run's headline | `Experimental TIA mode enabled / recording a coverage baseline.` — no false `fresh graph` | **VERIFIED (fixed)** | +| K3 | `pest --tia` ×3 on a clean tree | replayed entries keep their recorded `time` across all three | **VERIFIED (fixed)** — pre-fix: `0.053 → 0.001 → 0.001 → 0.001` | +| K4 | add a test file, one `pest --tia` (graph exists, fingerprint matches → replay+refresh) | result **and** edges appear in that same run; `n` grows by 1 | **VERIFIED** — regression guard for change 9 | +| K5 | add a test file, plain `pest --no-tia` | no result written for it; `n` unchanged; no edges | **VERIFIED (fixed)** — pre-fix: edge-less result inflated `n` | +| K6 | `pest --tia --parallel` worker argv | carries `-d pcov.directory=`; `PEST_TIA` unset; not re-exec'd | **VERIFIED** — probe `bin/worker.php` | +| K7 | second `pest --tia --coverage` (cache primed → replay) | recorder uses link tracking only, does not clear PHPUnit's data mid-collection; coverage report intact | Not yet measured — **new work** | +| K8 | `pest --tia --parallel --coverage` | workers read `TIA_PIGGYBACK_COVERAGE`; no widened pcov scope; report intact | Not yet measured — **new work** | +| K9 | `pest --tia --filtered --coverage-html=` / `--coverage-clover=` | filtered mode off, same as I8, for every flag in `COVERAGE_REPORT_FLAGS` | Not yet measured — **new work** | + +--- + +## Part 3 — Priorities for phase two + +0. **Part 1b fixtures** — nothing in C, D6–D10, or E9–E14 means anything until they exist. +1. **K7, K8, K9** — the only rows never measured. K7/K8 exercise changes 2 and 3, which were + reasoned about but not observed; K9 covers the seven `COVERAGE_REPORT_FLAGS` beyond + `--coverage-text`. +2. **B2, B3, F4** and all of **D** and **E** — change 9 touched the shared result-write path, and + these are the rows that exercise it hardest. `$recordsEdges` is the thing to falsify: it must be + false for every partial and every non-recording run. +3. **F3–F8** — change 7 injects a `-d` into worker argv; `--mutate --parallel` is the one place that + both spawns workers and must write nothing. +4. **H1–H10** — untouched by these changes; cheapest bulk confirmation. +5. Raw PHPUnit coverage flags print **no report at all** in Pest, with or without TIA + (`--no-tia --coverage-text` is equally silent). Pre-existing, unrelated to change 6 — do not + chase it as a regression, but it means I8/K9 can only assert the selection half. + +Report per row: tier respected (yes/no), the graph delta under sentinel patching, and for any +failure the pre-fix contrast (write `git show db70017c:` into vendor, re-run, restore) so a +regression is told apart from a pre-existing defect. diff --git a/composer.json b/composer.json index edfc4805..1da65caa 100644 --- a/composer.json +++ b/composer.json @@ -48,6 +48,7 @@ "Tests\\Fixtures\\Covers\\": "tests/Fixtures/Covers", "Tests\\Fixtures\\Inheritance\\": "tests/Fixtures/Inheritance", "Tests\\Fixtures\\Arch\\": "tests/Fixtures/Arch", + "Tests\\Fixtures\\Tia\\": "tests/Fixtures/Tia", "Tests\\": "tests/PHPUnit/" }, "classmap": [ diff --git a/phpunit.xml b/phpunit.xml index caca9cf4..864fc579 100644 --- a/phpunit.xml +++ b/phpunit.xml @@ -20,6 +20,7 @@ ./tests/.snapshots ./tests/Fixtures/Inheritance ./tests/Fixtures/Suites + ./tests/Fixtures/Tia diff --git a/src/Exceptions/TiaRequiresRemote.php b/src/Exceptions/TiaRequiresRemote.php new file mode 100644 index 00000000..91ad1f40 --- /dev/null +++ b/src/Exceptions/TiaRequiresRemote.php @@ -0,0 +1,47 @@ +writeln([ + '', + ' ERROR Tia mode requires a repository with a remote.', + '', + ' Without one there is no way to tell which branch is the default — the branch', + ' whose baseline every other branch falls back to reading — so the first run on', + ' each new branch would silently re-run the whole suite.', + '', + ' Add a remote, or name the branch yourself in tests/Pest.php:', + '', + ' pest()->tia()->defaultBranch(\'master\');', + '', + ]); + } + + public function exitCode(): int + { + return 1; + } +} diff --git a/src/Plugins/Tia.php b/src/Plugins/Tia.php index c37898ff..97e5e791 100644 --- a/src/Plugins/Tia.php +++ b/src/Plugins/Tia.php @@ -9,8 +9,10 @@ use Pest\Contracts\Plugins\AddsOutput; use Pest\Contracts\Plugins\HandlesArguments; use Pest\Contracts\Plugins\HandlesOriginalArguments; use Pest\Contracts\Plugins\Terminable; +use Pest\Exceptions\InvalidOption; use Pest\Exceptions\MissingDependency; use Pest\Exceptions\NoAffectedTestsFound; +use Pest\Exceptions\TiaRequiresRemote; use Pest\Exceptions\TiaRequiresRepositoryRoot; use Pest\Panic; use Pest\Plugins\Concerns\HandleArguments; @@ -183,6 +185,19 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument '--assignee', '--issue', '--ticket', '--pr', '--pull-request', ]; + /** + * Options that cannot be combined with Tia mode. + * + * `--covers` and `--uses` select on coverage metadata Tia does not model, + * so they resolve to no tests at all rather than to the ones the user meant. + * `--random-order-seed` exits non-zero on its own, with or without Tia. + * Either way the run cannot honour both things it was asked for, so it says + * so instead of silently dropping Tia and running something else. + */ + private const array UNSUPPORTED_OPTIONS = [ + '--covers', '--uses', '--random-order-seed', + ]; + private bool $graphWritten = false; private bool $replayRan = false; @@ -506,6 +521,10 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument $cliEnabled = $this->hasArgument(self::OPTION, $arguments) || self::envFlagEnabled(self::ENV_TIA); $alwaysEnabled = $watchPatterns->isEnabled() && (! $watchPatterns->isLocally() || Environment::name() === Environment::LOCAL); + if (! $isWorker && ! $disabled && ($cliEnabled || $alwaysEnabled)) { + $this->guardUnsupportedOptions($arguments); + } + $hasExplicitPath = $this->hasExplicitPathArgument($arguments); $partial = ! $isWorker && ($hasExplicitPath || $this->hasPartialSelection($arguments)); $disabled = $disabled || $partial; @@ -869,6 +888,14 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument $this->resolveBranch($projectRoot); + // After resolveBranch(), so a directory that is no repository at all + // still reports the missing git dependency rather than a missing remote. + // Skipped once the default branch is configured by hand: there is then + // nothing left for a remote to answer. + if ($this->watchPatterns->defaultBranch() === null && ! new ChangedFiles($projectRoot)->hasRemote()) { + Panic::with(new TiaRequiresRemote); + } + $fingerprint = Fingerprint::compute($projectRoot); $this->startFingerprint = $fingerprint; @@ -1870,13 +1897,7 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument return true; } - foreach (self::COVERAGE_REPORT_FLAGS as $flag) { - if ($this->hasArgument($flag, $this->originalArguments)) { - return true; - } - } - - return false; + return array_any(self::COVERAGE_REPORT_FLAGS, fn (string $flag): bool => $this->hasArgument($flag, $this->originalArguments)); } /** @@ -1891,6 +1912,29 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument return $coverage->coverage; } + /** + * Panics when the run asks for Tia alongside an option Tia cannot honour. + * + * Checked against the original argv as well, because `bin/pest` consumes + * some of these itself before PHPUnit ever sees them. + * + * @param array $arguments + */ + private function guardUnsupportedOptions(array $arguments): void + { + foreach (self::UNSUPPORTED_OPTIONS as $option) { + if (! $this->hasArgument($option, $arguments) && ! $this->hasArgument($option, $this->originalArguments)) { + continue; + } + + Panic::with(new InvalidOption(sprintf( + 'The [%s] option cannot be combined with [%s].', + $option, + self::OPTION, + ))); + } + } + /** * Whether a selection-narrowing flag was given, either among the arguments * PHPUnit receives or — for the flags `bin/pest` consumes itself — among diff --git a/src/Plugins/Tia/ChangedFiles.php b/src/Plugins/Tia/ChangedFiles.php index d39da1d8..979eba48 100644 --- a/src/Plugins/Tia/ChangedFiles.php +++ b/src/Plugins/Tia/ChangedFiles.php @@ -244,6 +244,17 @@ final readonly class ChangedFiles return $this->gitOutput(['git', 'config', '--get', 'init.defaultBranch']); } + /** + * Whether the repository has any remote configured. + * + * Advisory like {@see self::defaultBranch()} — a `git` that cannot answer + * is reported as "no remote", and the caller decides what that means. + */ + public function hasRemote(): bool + { + return $this->gitOutput(['git', 'remote']) !== null; + } + /** * @param array $command */ diff --git a/tests/.snapshots/success.txt b/tests/.snapshots/success.txt index 60139720..fd30a45f 100644 --- a/tests/.snapshots/success.txt +++ b/tests/.snapshots/success.txt @@ -1568,6 +1568,34 @@ PASS Tests\Features\Tia ✓ it does not run user hooks when replaying cached skipped and incomplete results + PASS Tests\Features\Tia\DefaultBranchReplay + ✓ replays the default branch baseline on a new branch + ✓ replays whatever the default branch is called with ('main') + ✓ replays whatever the default branch is called with ('master') + ✓ replays whatever the default branch is called with ('trunk') + ✓ replays whatever the default branch is called with ('develop') + ✓ replays on a second new branch too + ✓ writes nothing on a second run on the same branch + ✓ replays on a branch whose name contains slashes + ✓ replays inside a worktree on a new branch + + PASS Tests\Features\Tia\DefaultBranchResolution + ✓ a declared default branch beats autodetection + ✓ a declared default branch that does not exist degrades to a full run + ✓ a repository with no remote is refused rather than silently re-run + ✓ a declared default branch stands in for a missing remote + ✓ tia still requires git + ✓ a plain run outside a repository creates no baseline + ✓ the default branch is resolved once per run, not once per test + + PASS Tests\Features\Tia\DefaultBranchWriteTier + ✓ narrows to the affected tests on a new branch + ✓ filtered mode reads the fallback too + ✓ filtered mode finds nothing to do on a clean green feature branch + ✓ a detached HEAD replays without minting a branch key + ✓ the branch that ran gets its own key and the default branch keeps its baseline + ✓ the fallback reaches parallel workers + PASS Tests\Features\Ticket ✓ it may be associated with an ticket #1, #2 ✓ nested → it may be associated with an ticket #1, #4, #5, #6, #3 @@ -2197,4 +2225,4 @@ ✓ pass with dataset with ('my-datas-set-value') ✓ within describe → pass with dataset with ('my-datas-set-value') - Tests: 1 deprecated, 4 warnings, 5 incomplete, 2 notices, 40 todos, 35 skipped, 1549 passed (3389 assertions) \ No newline at end of file + Tests: 1 deprecated, 4 warnings, 5 incomplete, 2 notices, 40 todos, 35 skipped, 1571 passed (3446 assertions) \ No newline at end of file diff --git a/tests/Features/Tia/DefaultBranchReplay.php b/tests/Features/Tia/DefaultBranchReplay.php new file mode 100644 index 00000000..e31447a1 --- /dev/null +++ b/tests/Features/Tia/DefaultBranchReplay.php @@ -0,0 +1,108 @@ +seed('master'); + + $project->git()->switchTo('feature-x', new: true); + + $result = $project->pest('--tia'); + + expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe()) + ->and($result->uncached())->toBe(0, $result->describe()) + ->and($result->exitCode)->toBe(0); +})->skipOnWindows(); + +test('replays whatever the default branch is called', function (string $defaultBranch): void { + $project = Project::make($defaultBranch); + $project->seed($defaultBranch); + + $project->git()->switchTo('feature-x', new: true); + + $result = $project->pest('--tia'); + + expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe()) + ->and($result->uncached())->toBe(0, $result->describe()); +})->with(['main', 'master', 'trunk', 'develop'])->skipOnWindows(); + +test('replays on a second new branch too', function (): void { + $project = Project::make('master'); + $project->seed('master'); + + $project->git()->switchTo('feature-x', new: true); + $project->pest('--tia'); + + $project->git()->switchTo('master'); + $project->git()->switchTo('feature-y', new: true); + + $result = $project->pest('--tia'); + + // The toll is one full run per new branch. It must not come back for the + // second branch either. + expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe()) + ->and($result->uncached())->toBe(0, $result->describe()); +})->skipOnWindows(); + +test('writes nothing on a second run on the same branch', function (): void { + $project = Project::make('master'); + $project->seed('master'); + + $project->git()->switchTo('feature-x', new: true); + $project->pest('--tia'); + + $project->snapshot(); + $result = $project->pest('--tia'); + $delta = $project->delta(); + + expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe()) + ->and($delta->writtenCount())->toBe(0, $delta->summary()) + ->and($delta->isResultsOnly())->toBeTrue($delta->summary()); +})->skipOnWindows(); + +test('replays on a branch whose name contains slashes', function (): void { + $project = Project::make('master'); + $project->seed('master'); + + $project->git()->switchTo('feature/x/y', new: true); + + $result = $project->pest('--tia'); + + expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe()) + ->and($result->uncached())->toBe(0, $result->describe()) + ->and($project->branchKeys())->toContain('feature/x/y'); +})->skipOnWindows(); + +test('replays inside a worktree on a new branch', function (): void { + $project = Project::make('master'); + $worktree = $project->worktree('feature-worktree'); + + // Seeded against the worktree rather than the main checkout: a worktree's + // `.git` is a file, so `Storage::originIdentity()` cannot read the remote + // from it and the worktree resolves a storage key of its own. That gap is + // separate from the branch fallback, and it is the fallback this row is + // about — the worktree is checked out on a branch the baseline does not + // name, which is the scenario from the issue. + $project->seedFor($worktree, 'master'); + + $result = $project->pestIn($worktree, '--tia'); + + expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe()) + ->and($result->uncached())->toBe(0, $result->describe()); +})->skipOnWindows(); diff --git a/tests/Features/Tia/DefaultBranchResolution.php b/tests/Features/Tia/DefaultBranchResolution.php new file mode 100644 index 00000000..e80fd29d --- /dev/null +++ b/tests/Features/Tia/DefaultBranchResolution.php @@ -0,0 +1,131 @@ +seed('master'); + + $project->git()->switchTo('feature-x', new: true); + + $result = $project->pest('--tia'); + + expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe()) + ->and($result->uncached())->toBe(0, $result->describe()); +})->skipOnWindows(); + +test('a declared default branch that does not exist degrades to a full run', function (): void { + $project = Project::make('master', overlay: 'unknown-default-branch'); + $project->seed('master'); + + $project->git()->switchTo('feature-x', new: true); + + $result = $project->pest('--tia'); + + // Nothing to fall back to, so everything runs — and no baseline is minted + // under the name that resolved to nothing. + expect($result->uncached())->toBe(Project::TOTAL_TESTS, $result->describe()) + ->and($result->exitCode)->toBe(0, $result->describe()) + ->and($project->branchKeys())->toBe(['master', 'feature-x']); +})->skipOnWindows(); + +test('a repository with no remote is refused rather than silently re-run', function (): void { + $project = Project::make('master'); + + $project->git()->removeOrigin(); + $project->seed('master'); + + $project->git()->switchTo('feature-x', new: true); + + $result = $project->pest('--tia'); + + // Nothing can name the default branch, so every new branch would re-run the + // whole suite with no explanation. Saying so beats doing that quietly. + expect($result->output)->toContain('Tia mode requires a repository with a remote.') + ->and($result->exitCode)->toBe(1, $result->describe()); +})->skipOnWindows(); + +test('a declared default branch stands in for a missing remote', function (): void { + // The escape hatch the refusal above points at: with the branch named by + // hand there is nothing left for a remote to answer. + $project = Project::make('master', overlay: 'configured-default-branch'); + + $project->git()->removeOrigin(); + $project->seed('master'); + + $project->git()->switchTo('feature-x', new: true); + + $result = $project->pest('--tia'); + + expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe()) + ->and($result->uncached())->toBe(0, $result->describe()); +})->skipOnWindows(); + +test('tia still requires git', function (): void { + $project = Project::withoutGit(); + + $result = $project->pest('--tia'); + + // The soft default-branch resolver runs before the branch is named, and it + // must not swallow this. + expect($result->output)->toContain('The feature "Tia mode" requires "git".') + ->and($result->exitCode)->not->toBe(0); +})->skipOnWindows(); + +test('a plain run outside a repository creates no baseline', function (): void { + $project = Project::withoutGit(); + + $result = $project->pest(); + + expect($result->exitCode)->toBe(0, $result->describe()) + ->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed') + ->and($project->graphExists())->toBeFalse() + ->and($project->graphDir())->not->toBeDirectory(); +})->skipOnWindows(); + +test('the default branch is resolved once per run, not once per test', function (): void { + $project = Project::make('master'); + $project->seed('master'); + + $project->git()->switchTo('feature-x', new: true); + + $git = (new ExecutableFinder)->find('git'); + + expect($git)->not->toBeNull(); + + // A `git` first on `PATH` that records what it was asked before handing over + // to the real one. + $log = $project->path('git-calls.log'); + $project->write('shim/git', implode("\n", [ + '#!/bin/sh', + 'echo "$@" >> '.escapeshellarg($log), + 'exec '.escapeshellarg((string) $git).' "$@"', + '', + ])); + chmod($project->path('shim/git'), 0755); + + $result = $project->pestWithEnvironment($project->path(), [ + 'PATH' => $project->path('shim').':'.getenv('PATH'), + ], '--tia'); + + $calls = file_exists($log) ? explode("\n", trim((string) file_get_contents($log))) : []; + $resolutions = array_filter($calls, fn (string $call): bool => str_contains($call, 'symbolic-ref') + || str_contains($call, 'init.defaultBranch')); + + expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe()) + ->and($calls)->not->toBeEmpty('the git shim was never reached') + ->and($resolutions)->toHaveCount(1, implode("\n", $calls)); +})->skipOnWindows(); diff --git a/tests/Features/Tia/DefaultBranchWriteTier.php b/tests/Features/Tia/DefaultBranchWriteTier.php new file mode 100644 index 00000000..caa4c825 --- /dev/null +++ b/tests/Features/Tia/DefaultBranchWriteTier.php @@ -0,0 +1,133 @@ +seed('master'); + + $project->git()->switchTo('feature-x', new: true); + + // Semantic, not cosmetic: PHP is hashed at the AST level, so a comment + // would not register as a change at all. + $project->write('app/Calculator.php', <<<'PHP' + pest('--tia'); + + // Two of the three test files cover `Calculator`; the third replays from the + // default branch's baseline. + expect($result->affected())->toBe(4, $result->describe()) + ->and($result->replayed())->toBe(2, $result->describe()) + ->and($result->exitCode)->toBe(0, $result->describe()); +})->skipOnWindows(); + +test('filtered mode reads the fallback too', function (): void { + $project = Project::make('master'); + + // A failure cached on the default branch. Filtered mode asks the graph which + // test files are due a re-run, and that read has to reach the fallback as + // well: without it a new branch sees nothing to do, reports green, and never + // re-runs the failure — on every subsequent invocation. + $project->seed('master', failing: ['adds two numbers']); + + $project->git()->switchTo('feature-x', new: true); + + $result = $project->pest('--tia', '--filtered'); + + // Selection is by file, so the failure's two siblings in `CalculatorTest` + // come along; the other two test files stay out of the run entirely. + expect($result->output)->toContain('from 1 previously unsuccessful test') + ->and($result->output)->not->toContain('No affected tests found') + ->and($result->tally())->toContain('2 passed'); +})->skipOnWindows(); + +test('filtered mode finds nothing to do on a clean green feature branch', function (): void { + $project = Project::make('master'); + $project->seed('master'); + + $project->git()->switchTo('feature-x', new: true); + + $result = $project->pest('--tia', '--filtered'); + $delta = $project->delta(); + + expect($result->output)->toContain('No affected tests found') + ->and($result->exitCode)->toBe(0, $result->describe()) + ->and($delta->isHardSuppressed())->toBeTrue($delta->summary()); +})->skipOnWindows(); + +test('a detached HEAD replays without minting a branch key', function (): void { + $project = Project::make('master'); + $project->seed('master'); + + $project->git()->detach(); + + $result = $project->pest('--tia'); + + // A detached HEAD has no branch of its own. The default branch is the + // honest key, and no phantom one appears beside it. + expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe()) + ->and($result->uncached())->toBe(0, $result->describe()) + ->and($project->branchKeys())->toBe(['master']); +})->skipOnWindows(); + +test('the branch that ran gets its own key and the default branch keeps its baseline', function (): void { + $project = Project::make('master'); + $project->seed('master'); + + $project->git()->switchTo('feature-x', new: true); + $project->pest('--tia'); + + $delta = $project->delta(); + + expect($project->branchKeys())->toBe(['master', 'feature-x']) + ->and($delta->baselineUntouched('master'))->toBeTrue($delta->summary()) + ->and($delta->writtenCount())->toBe(0, $delta->summary()); +})->skipOnWindows(); + +test('the fallback reaches parallel workers', function (): void { + $project = Project::make('master'); + $project->seed('master'); + + $project->git()->switchTo('feature-x', new: true); + + $result = $project->pest('--tia', '--parallel', '--processes=2'); + + expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe()) + ->and($result->uncached())->toBe(0, $result->describe()); +})->skipOnWindows(); diff --git a/tests/Fixtures/Tia/GitRepo.php b/tests/Fixtures/Tia/GitRepo.php new file mode 100644 index 00000000..18c4689e --- /dev/null +++ b/tests/Fixtures/Tia/GitRepo.php @@ -0,0 +1,170 @@ + + */ + public const array ENV = [ + 'GIT_CONFIG_GLOBAL' => '/dev/null', + 'GIT_CONFIG_SYSTEM' => '/dev/null', + // `GIT_CONFIG_SYSTEM` does not cover every system-level file git reads: + // Apple's git also loads one from inside Xcode, and it sets + // `init.defaultBranch`. Only this suppresses all of them. + 'GIT_CONFIG_NOSYSTEM' => '1', + 'GIT_AUTHOR_NAME' => 'Pest Fixture', + 'GIT_AUTHOR_EMAIL' => 'fixture@pestphp.io', + 'GIT_COMMITTER_NAME' => 'Pest Fixture', + 'GIT_COMMITTER_EMAIL' => 'fixture@pestphp.io', + ]; + + public function __construct(public string $path) {} + + /** + * Initialises the repository on `$branch` and commits everything in it. + */ + public function init(string $branch = 'master'): void + { + $this->run(['init', '--quiet']); + $this->run(['checkout', '--quiet', '-b', $branch]); + $this->commit('Initial commit'); + } + + public function commit(string $message): void + { + $this->run(['add', '-A']); + $this->run(['commit', '--quiet', '--allow-empty', '-m', $message]); + } + + public function switchTo(string $branch, bool $new = false): void + { + $this->run($new ? ['checkout', '--quiet', '-b', $branch] : ['checkout', '--quiet', $branch]); + } + + public function rename(string $from, string $to): void + { + $this->run(['branch', '-m', $from, $to]); + } + + public function detach(): void + { + $this->run(['checkout', '--quiet', '--detach']); + } + + /** + * Registers an `origin`, which also decides the graph's storage key: + * {@see Storage::projectKey()} prefers the remote's + * identity over the path, so two checkouts of one repository — a worktree, + * say — share a single graph. + */ + public function addOrigin(string $url = 'git@github.com:pestphp/tia-fixture.git'): void + { + $this->run(['remote', 'add', 'origin', $url]); + } + + public function removeOrigin(): void + { + $this->run(['remote', 'remove', 'origin']); + } + + /** + * Points `refs/remotes/origin/HEAD` at a local branch — what + * `git remote set-head` would write, without a remote to talk to. + */ + public function setOriginHead(string $branch): void + { + $this->run(['update-ref', 'refs/remotes/origin/'.$branch, 'HEAD']); + $this->run(['symbolic-ref', 'refs/remotes/origin/HEAD', 'refs/remotes/origin/'.$branch]); + } + + public function config(string $key, string $value): void + { + $this->run(['config', '--local', $key, $value]); + } + + /** + * Adds a worktree for a new branch and returns its path. + */ + public function worktree(string $path, string $branch): string + { + $this->run(['worktree', 'add', '--quiet', '-b', $branch, $path]); + + return $path; + } + + public function sha(): string + { + return $this->output(['rev-parse', 'HEAD']); + } + + public function currentBranch(): string + { + return $this->output(['rev-parse', '--abbrev-ref', 'HEAD']); + } + + /** + * @return array + */ + public function branchNames(): array + { + $names = explode("\n", $this->output(['for-each-ref', '--format=%(refname:short)', 'refs/heads'])); + + return array_values(array_filter($names, fn (string $name): bool => $name !== '')); + } + + /** + * @param array $arguments + */ + public function output(array $arguments): string + { + return trim($this->process($arguments, mustSucceed: true)->getOutput()); + } + + /** + * @param array $arguments + */ + public function run(array $arguments): void + { + $this->process($arguments, mustSucceed: true); + } + + /** + * @param array $arguments + */ + private function process(array $arguments, bool $mustSucceed): Process + { + $process = new Process(['git', ...$arguments], $this->path, self::ENV); + $process->setTimeout(30.0); + $process->run(); + + if ($mustSucceed && ! $process->isSuccessful()) { + throw new RuntimeException(sprintf( + "git %s failed in [%s]:\n%s", + implode(' ', $arguments), + $this->path, + $process->getErrorOutput().$process->getOutput(), + )); + } + + return $process; + } +} diff --git a/tests/Fixtures/Tia/GraphDelta.php b/tests/Fixtures/Tia/GraphDelta.php new file mode 100644 index 00000000..54ba5abd --- /dev/null +++ b/tests/Fixtures/Tia/GraphDelta.php @@ -0,0 +1,339 @@ +].results` for tests that + * actually ran. It may never remove an entry, nor move `sha`, `tree`, + * `edges`, `files` or `fingerprint`. + * - HARD-SUPPRESSED — may change nothing at all. + * + * {@see self::writtenCount()} is the load-bearing measurement, and the reason + * {@see Project::sentinel()} exists: without falsified cached values there is no + * way to tell "wrote the same values back" from "wrote nothing". + * + * @internal + */ +final readonly class GraphDelta +{ + /** + * @param array|null $before + * @param array|null $after + */ + public function __construct( + private ?array $before, + private ?array $after, + ) {} + + public function graphWasCreated(): bool + { + return $this->before === null && $this->after !== null; + } + + public function graphWasDeleted(): bool + { + return $this->before !== null && $this->after === null; + } + + /** + * Result entries whose stored values actually moved. + */ + public function writtenCount(): int + { + $written = 0; + + foreach ($this->branchKeys() as $branch) { + $before = $this->results($this->before, $branch); + $after = $this->results($this->after, $branch); + + foreach ($before as $testId => $entry) { + if (! isset($after[$testId])) { + continue; + } + + foreach (['status', 'time', 'assertions', 'message'] as $field) { + if (($entry[$field] ?? null) !== ($after[$testId][$field] ?? null)) { + $written++; + + break; + } + } + } + } + + return $written; + } + + /** + * Result entries that appeared. + */ + public function added(): int + { + $added = 0; + + foreach ($this->branchKeys() as $branch) { + $added += count(array_diff( + array_keys($this->results($this->after, $branch)), + array_keys($this->results($this->before, $branch)), + )); + } + + return $added; + } + + /** + * Result entries that were pruned. + */ + public function removed(): int + { + $removed = 0; + + foreach ($this->branchKeys() as $branch) { + $removed += count(array_diff( + array_keys($this->results($this->before, $branch)), + array_keys($this->results($this->after, $branch)), + )); + } + + return $removed; + } + + /** + * The baseline keys after the run — the headline signal for the + * default-branch rows, where a phantom key is the defect. + * + * @return array + */ + public function branchKeys(): array + { + return array_keys($this->baselines($this->after)); + } + + /** + * @return array + */ + public function branchKeysBefore(): array + { + return array_keys($this->baselines($this->before)); + } + + public function branchKeysMoved(): bool + { + return $this->branchKeysBefore() !== $this->branchKeys(); + } + + /** + * Whether the named branch's baseline is byte-identical — how a row proves + * the fallback is read-only. + */ + public function baselineUntouched(string $branch): bool + { + return ($this->baselines($this->before)[$branch] ?? null) + === ($this->baselines($this->after)[$branch] ?? null); + } + + public function shaMoved(): bool + { + return array_any($this->branchKeys(), fn (string $branch) => $this->baselineField($branch, 'sha', $this->before) !== $this->baselineField($branch, 'sha', $this->after)); + } + + public function treeMoved(): bool + { + return array_any($this->branchKeys(), fn (string $branch) => $this->baselineField($branch, 'tree', $this->before) !== $this->baselineField($branch, 'tree', $this->after)); + } + + /** + * Compares edges by the file paths they resolve to, not by file id: ids are + * an implementation detail that shifts whenever `files` is rebuilt in a + * different order. + */ + public function edgesMoved(): bool + { + return $this->edgeSets($this->before) !== $this->edgeSets($this->after); + } + + public function filesMoved(): bool + { + return $this->section($this->before, 'files') !== $this->section($this->after, 'files'); + } + + public function fingerprintMoved(): bool + { + return $this->section($this->before, 'fingerprint') !== $this->section($this->after, 'fingerprint'); + } + + public function structureMoved(): bool + { + if ($this->edgesMoved()) { + return true; + } + if ($this->filesMoved()) { + return true; + } + if ($this->fingerprintMoved()) { + return true; + } + + return $this->branchKeysMoved(); + } + + /** + * Nothing moved at all. + */ + public function isHardSuppressed(): bool + { + return $this->before === $this->after; + } + + /** + * Results may have moved for tests that ran; nothing structural did. + */ + public function isResultsOnly(): bool + { + return ! $this->graphWasCreated() + && ! $this->graphWasDeleted() + && ! $this->structureMoved() + && ! $this->shaMoved() + && ! $this->treeMoved() + && $this->removed() === 0 + && $this->added() === 0; + } + + /** + * A one-line verdict, for failure messages. + */ + public function summary(): string + { + if ($this->graphWasCreated()) { + return 'graph created'; + } + + if ($this->graphWasDeleted()) { + return 'graph deleted'; + } + + $moved = []; + + foreach (['edges', 'files', 'fingerprint'] as $section) { + if ($this->{$section.'Moved'}()) { + $moved[] = $section; + } + } + + if ($this->branchKeysMoved()) { + $moved[] = sprintf( + 'branchkeys(%s->%s)', + implode('|', $this->branchKeysBefore()), + implode('|', $this->branchKeys()), + ); + } + + return sprintf( + 'w=%d +%d -%d %s%s%s', + $this->writtenCount(), + $this->added(), + $this->removed(), + $moved === [] ? 'struct:ok' : 'STRUCT:'.implode(',', $moved), + $this->shaMoved() ? ' sha:changed' : '', + $this->treeMoved() ? ' tree:changed' : '', + ); + } + + /** + * @param array|null $graph + * @return array> + */ + private function edgeSets(?array $graph): array + { + $files = $this->section($graph, 'files'); + $sets = []; + + foreach ($this->section($graph, 'edges') as $test => $ids) { + if (! is_string($test)) { + continue; + } + if (! is_array($ids)) { + continue; + } + $paths = array_map( + fn (mixed $id): string => is_int($id) && isset($files[$id]) && is_string($files[$id]) + ? $files[$id] + : '?'.json_encode($id), + $ids, + ); + + sort($paths); + $sets[$test] = $paths; + } + + ksort($sets); + + return $sets; + } + + /** + * @param array|null $graph + * @return array + */ + private function section(?array $graph, string $key): array + { + $section = $graph[$key] ?? null; + + return is_array($section) ? $section : []; + } + + /** + * @param array|null $graph + * @return array + */ + private function baselines(?array $graph): array + { + $baselines = []; + + foreach ($this->section($graph, 'baselines') as $branch => $baseline) { + if (is_string($branch)) { + $baselines[$branch] = $baseline; + } + } + + return $baselines; + } + + /** + * @param array|null $graph + * @return array> + */ + private function results(?array $graph, string $branch): array + { + $results = $this->baselines($graph)[$branch]['results'] ?? null; + + if (! is_array($results)) { + return []; + } + + $entries = []; + + foreach ($results as $testId => $entry) { + if (is_string($testId) && is_array($entry)) { + $entries[$testId] = $entry; + } + } + + return $entries; + } + + /** + * @param array|null $graph + */ + private function baselineField(string $branch, string $field, ?array $graph): mixed + { + return $this->baselines($graph)[$branch][$field] ?? null; + } +} diff --git a/tests/Fixtures/Tia/PestResult.php b/tests/Fixtures/Tia/PestResult.php new file mode 100644 index 00000000..16533be7 --- /dev/null +++ b/tests/Fixtures/Tia/PestResult.php @@ -0,0 +1,102 @@ + $arguments + */ + public function __construct( + public array $arguments, + string $output, + public int $exitCode, + ) { + $this->output = (string) preg_replace([ + '#\x1b[[][^A-Za-z]*[A-Za-z]#', // colours, cursor moves + '#\x1b\]8;[^\x1b\x07]*(?:\x1b\\\\|\x07)#', // hyperlinks + ], '', $output); + } + + /** + * Tests whose cached result was replayed instead of executed. + */ + public function replayed(): int + { + return $this->recapFragment('replayed'); + } + + /** + * Tests that ran because the graph held nothing for them — the count that + * betrays a fallback which never resolved. + */ + public function uncached(): int + { + return $this->recapFragment('uncached'); + } + + /** + * Tests that ran because a file they depend on changed. + */ + public function affected(): int + { + return $this->recapFragment('affected'); + } + + /** + * The `Tests:` summary line, without its label or leading whitespace. + */ + public function tally(): string + { + if (preg_match('/^\s*Tests:\s+(.+)$/m', $this->output, $matches) !== 1) { + return ''; + } + + return trim($matches[1]); + } + + public function contains(string $needle): bool + { + return str_contains($this->output, $needle); + } + + /** + * A description of the run, for failure messages that would otherwise say + * only that 0 !== 6. + */ + public function describe(): string + { + return sprintf( + "pest %s exited %d:\n%s", + implode(' ', $this->arguments), + $this->exitCode, + $this->output, + ); + } + + /** + * Read off the `Tests:` line rather than the whole output: the TIA headline + * counts affected *files*, and matching that instead would be a quietly + * wrong number. + */ + private function recapFragment(string $label): int + { + if (preg_match('/(\d+) '.preg_quote($label, '/').'/', $this->tally(), $matches) !== 1) { + return 0; + } + + return (int) $matches[1]; + } +} diff --git a/tests/Fixtures/Tia/Project.php b/tests/Fixtures/Tia/Project.php new file mode 100644 index 00000000..67443f43 --- /dev/null +++ b/tests/Fixtures/Tia/Project.php @@ -0,0 +1,638 @@ +> + */ + public const array EDGES = [ + 'tests/Unit/CalculatorTest.php' => ['app/Calculator.php'], + 'tests/Unit/GreeterTest.php' => ['app/Greeter.php'], + 'tests/Feature/CoversCalculatorTest.php' => ['app/Calculator.php'], + ]; + + /** + * Test file → the descriptions it declares, in declaration order. + * + * @var array> + */ + public const array TESTS = [ + 'tests/Unit/CalculatorTest.php' => ['adds two numbers', 'subtracts two numbers'], + 'tests/Unit/GreeterTest.php' => ['greets a person', 'greets the world'], + 'tests/Feature/CoversCalculatorTest.php' => ['adds within a feature test', 'subtracts within a feature test'], + ]; + + /** + * Every test in the fixture suite. + */ + public const int TOTAL_TESTS = 6; + + /** + * Every project scaffolded so far, so a row cannot leak one by failing + * before its own cleanup. + * + * @var array + */ + private static array $created = []; + + private ?GitRepo $repo = null; + + /** + * @var array|null + */ + private ?array $snapshot = null; + + /** + * @var array + */ + private array $extraPaths = []; + + /** + * The root whose graph this project reads and writes. + * + * Its own, except where a row seeds a worktree: a worktree's `.git` is a + * file rather than a directory, so {@see Storage} cannot read the remote + * from it and resolves a storage key of its own. + */ + private string $graphRoot; + + private function __construct(public readonly string $path) + { + $this->graphRoot = $path; + } + + /** + * Scaffolds a project whose default branch is `$branch`, and hands it back + * checked out there. + * + * The repository is realistic on purpose: it has an `origin`, and an + * `origin/HEAD` naming `$branch`, which is what a checkout of a real project + * looks like and what the default branch is autodetected from. + */ + public static function make(string $branch = 'master', ?string $overlay = null): self + { + $project = self::scaffold($overlay); + + $project->git()->init($branch); + $project->git()->addOrigin(); + $project->git()->setOriginHead($branch); + + return $project; + } + + /** + * Destroys every project scaffolded so far. Belongs in an `afterEach`. + */ + public static function destroyAll(): void + { + while (self::$created !== []) { + array_pop(self::$created)->destroy(); + } + } + + /** + * A project that is not a git repository at all — for the rows that assert + * TIA still demands git, and that a plain run does not care. + * + * Lives in the system temp directory, so it is outside any enclosing + * repository, and owns a real `vendor` rather than a symlinked one, so its + * baseline key cannot collide with another fixture's. + */ + public static function withoutGit(?string $overlay = null): self + { + return self::scaffold($overlay); + } + + public function git(): GitRepo + { + return $this->repo ??= new GitRepo($this->path); + } + + public function path(string $relative = ''): string + { + return $relative === '' ? $this->path : $this->path.DIRECTORY_SEPARATOR.$relative; + } + + /** + * Overwrites a file in the project. + * + * Edits must be semantic: TIA hashes PHP at the AST level, so a + * comment-only change is not a change at all. + */ + public function write(string $relative, string $contents): void + { + $path = $this->path($relative); + $directory = dirname($path); + + if (! is_dir($directory) && ! @mkdir($directory, 0755, true) && ! is_dir($directory)) { + throw new RuntimeException(sprintf('Unable to create [%s].', $directory)); + } + + if (file_put_contents($path, $contents) === false) { + throw new RuntimeException(sprintf('Unable to write [%s].', $path)); + } + } + + /** + * Adds a worktree for a new branch, scaffolded so `pest` can run in it, and + * returns its path. + * + * The graph is shared with the main checkout, which is the whole point: both + * resolve the same storage key, because {@see Storage} prefers the `origin` + * identity over the path. + */ + public function worktree(string $branch): string + { + $path = $this->path.'-worktree-'.preg_replace('/[^a-z0-9]+/i', '-', $branch); + + $this->git()->worktree($path, $branch); + $this->scaffoldVendor($path); + $this->extraPaths[] = $path; + + return $path; + } + + /** + * Runs `pest` in the project and returns what happened. + */ + public function pest(string ...$arguments): PestResult + { + return $this->pestIn($this->path, ...$arguments); + } + + /** + * Runs `pest` in `$directory` — a worktree, say — against this project's + * graph. + */ + public function pestIn(string $directory, string ...$arguments): PestResult + { + return $this->pestWithEnvironment($directory, [], ...$arguments); + } + + /** + * @param array $environment + */ + public function pestWithEnvironment(string $directory, array $environment, string ...$arguments): PestResult + { + $process = new Process( + [PHP_BINARY, $directory.'/vendor/pestphp/pest/bin/pest', ...$arguments], + $directory, + [ + ...GitRepo::ENV, + 'COLLISION_PRINTER' => 'DefaultPrinter', + 'COLLISION_IGNORE_DURATION' => 'true', + 'PARATEST' => '0', + 'PAO_DISABLE' => '1', + 'HOME' => $this->home(), + ...$environment, + ], + ); + + $process->setTimeout(180.0); + $process->run(); + + return new PestResult( + array_values($arguments), + $process->getOutput().$process->getErrorOutput(), + (int) $process->getExitCode(), + ); + } + + /** + * Writes the graph a clean, green recording run on `$branch` would have + * written, and remembers it as the snapshot the next {@see self::delta()} + * compares against. + * + * Sentinelled by default: a row almost always wants to know which entries + * were written, and only a real recording run's values can answer that. + * + * `$failing` names descriptions to record as failures — a clean green run + * can never cache one, so a row that needs a cached failure to re-run has to + * be handed it. + * + * @param array $failing + */ + public function seed(string $branch, bool $sentinel = true, array $failing = []): void + { + $this->seedFor($this->path, $branch, $sentinel, $failing); + } + + /** + * Seeds the graph belonging to `$root` — a worktree, say, which resolves a + * storage key of its own. + * + * @param array $failing + */ + public function seedFor(string $root, string $branch, bool $sentinel = true, array $failing = []): void + { + $this->graphRoot = $root; + + $changedFiles = new ChangedFiles($root); + $sha = new GitRepo($root)->sha(); + + $graph = new Graph($root); + $graph->setFingerprint(Fingerprint::compute($root)); + $graph->setRecordedAtSha($branch, $sha); + + // Hashes the tree as it stands, so the run under test sees nothing as + // changed — the same call the recording path makes. + $graph->setLastRunTree($branch, $changedFiles->snapshotTree($changedFiles->since($sha) ?? [])); + + $graph->markKnownTestFiles(array_keys(self::EDGES)); + + foreach (self::EDGES as $testFile => $sourceFiles) { + $graph->link($testFile, $testFile); + + foreach ($sourceFiles as $sourceFile) { + $graph->link($testFile, $sourceFile); + } + } + + foreach (self::TESTS as $testFile => $descriptions) { + foreach ($descriptions as $description) { + $failed = in_array($description, $failing, true); + + $graph->setResult( + $branch, + self::testId($testFile, $description), + $failed ? 7 : 0, + $failed ? 'cached failure' : '', + 0.05, + 1, + $testFile, + ); + } + } + + $json = $graph->encode(); + + if ($json === null) { + throw new RuntimeException('Unable to encode the seeded graph.'); + } + + if (! $this->state()->write(Tia::KEY_GRAPH, $json)) { + throw new RuntimeException('Unable to persist the seeded graph.'); + } + + $sentinel ? $this->sentinel() : $this->snapshot(); + } + + /** + * The id PHPUnit reports for a test in the fixture suite. + * + * Mirrors how Pest names generated test classes + * ({@see TestCaseFactory}): a wrong id here shows up as + * `0 replayed`, which every scenario asserts against. + */ + public static function testId(string $testFile, string $description): string + { + $basename = basename($testFile, '.php'); + $dotPosition = strpos($basename, '.'); + + if ($dotPosition !== false) { + $basename = substr($basename, 0, $dotPosition); + } + + $relative = dirname(ucfirst($testFile)).DIRECTORY_SEPARATOR.$basename; + + return 'P\\'.str_replace(DIRECTORY_SEPARATOR, '\\', $relative).'::'.Str::evaluable($description); + } + + /** + * Falsifies every cached value, so the next {@see self::delta()} can tell + * "wrote the same values back" from "wrote nothing". + * + * `assertions` is only ever falsified where it is already non-zero: risky, + * skipped and incomplete statuses are *derived* from "performed no + * assertions", so patching those would rewrite the status on replay and + * destroy the very discriminator this exists to provide. + */ + public function sentinel(): void + { + $graph = $this->graph(); + + if ($graph === null) { + throw new RuntimeException('There is no graph to sentinel.'); + } + + foreach ($graph['baselines'] ?? [] as $branch => $baseline) { + foreach (array_keys($baseline['results'] ?? []) as $testId) { + $graph['baselines'][$branch]['results'][$testId]['time'] = 9.999; + + if ((int) ($baseline['results'][$testId]['assertions'] ?? 0) > 0) { + $graph['baselines'][$branch]['results'][$testId]['assertions'] = 42; + } + } + } + + $this->state()->write(Tia::KEY_GRAPH, (string) json_encode($graph, JSON_UNESCAPED_SLASHES)); + + $this->snapshot(); + } + + /** + * The decoded graph, or `null` when there is none. + * + * @return array|null + */ + public function graph(): ?array + { + $json = $this->state()->read(Tia::KEY_GRAPH); + + if ($json === null) { + return null; + } + + $graph = json_decode($json, true); + + return is_array($graph) ? $graph : null; + } + + /** + * @return array + */ + public function branchKeys(): array + { + $baselines = $this->graph()['baselines'] ?? []; + + return is_array($baselines) ? array_keys($baselines) : []; + } + + public function graphDir(): string + { + return $this->withHome(fn (): string => Storage::tempDir($this->graphRoot)); + } + + public function graphExists(): bool + { + return is_file($this->graphDir().DIRECTORY_SEPARATOR.Tia::KEY_GRAPH); + } + + /** + * Remembers the graph as it stands now. + */ + public function snapshot(): void + { + $this->snapshot = $this->graph(); + } + + /** + * What has happened to the graph since the last snapshot. + */ + public function delta(): GraphDelta + { + return new GraphDelta($this->snapshot, $this->graph()); + } + + /** + * Writes the `vendor` a composer install would have produced. + * + * Pest is mirrored in at `vendor/pestphp/pest` rather than pointed at, + * because Pest locates things from where its own files sit: + * `bin/pest` finds the project root by walking up from the autoloader it + * loads, and a parallel run picks its worker binary — and with it the + * worker's project root — from the directory the runner class was loaded + * from. Deferring to the repository's copy would resolve both back to the + * Pest repository, and every scenario would quietly measure that instead. + * + * Hardlinked where the filesystem allows it, so the mirror costs almost + * nothing and can never drift from the working tree. + */ + public function scaffoldVendor(string $directory): void + { + $pestRoot = dirname(__DIR__, 3); + $pest = $directory.'/vendor/pestphp/pest'; + + // `overrides`, `resources` and `stubs` come along because Pest loads + // them relative to `src` — the same list `BootExcludeList` walks. + foreach (['src', 'overrides', 'resources', 'stubs'] as $tree) { + self::mirror($pestRoot.'/'.$tree, $pest.'/'.$tree); + } + + foreach (['pest', 'worker.php'] as $binary) { + self::mirror($pestRoot.'/bin/'.$binary, $pest.'/bin/'.$binary); + } + + self::mirror($pestRoot.'/composer.json', $pest.'/composer.json'); + + // Pest's own autoloader, with the mirrored copy taking precedence: the + // repository's `vendor` supplies PHPUnit, Symfony and the plugin + // packages, none of which care where they are loaded from. + file_put_contents($directory.'/vendor/autoload.php', sprintf( + "addPsr4('Pest\\\\', __DIR__.'/pestphp/pest/src', true);\n\nreturn \$loader;\n", + var_export($pestRoot.'/vendor/autoload.php', true), + )); + + // Invoking the binary directly skips composer's bin proxy, which is + // what would otherwise define `$GLOBALS['_composer_bin_dir']`. Without + // it `Pest\Plugin\Loader` looks for `vendor/bin/../pest-plugins.json` + // relative to the working directory — so that is where the plugin list + // goes, and mirroring the repository's keeps it in step with + // composer.json. `vendor/bin` has to exist for the `..` in that path to + // resolve, empty though it is. + if (! is_dir($directory.'/vendor/bin') && ! @mkdir($directory.'/vendor/bin', 0755, true)) { + throw new RuntimeException(sprintf('Unable to create [%s].', $directory.'/vendor/bin')); + } + + self::mirror($pestRoot.'/vendor/pest-plugins.json', $directory.'/vendor/pest-plugins.json'); + } + + public function destroy(): void + { + foreach ([...$this->extraPaths, $this->path] as $path) { + $this->remove($path); + } + } + + private function home(): string + { + return $this->path.DIRECTORY_SEPARATOR.'.home'; + } + + private function state(): FileState + { + return new FileState($this->graphDir()); + } + + /** + * `Storage` reads `HOME` from the environment, so the graph lands inside the + * throwaway project instead of the developer's real `~/.pest`. + * + * @template TReturn + * + * @param callable(): TReturn $callback + * @return TReturn + */ + private function withHome(callable $callback): mixed + { + $original = getenv('HOME'); + + putenv('HOME='.$this->home()); + + try { + return $callback(); + } finally { + putenv($original === false ? 'HOME' : 'HOME='.$original); + } + } + + private static function scaffold(?string $overlay): self + { + $path = sys_get_temp_dir().DIRECTORY_SEPARATOR.'pest-tia-'.bin2hex(random_bytes(8)); + + if (! @mkdir($path, 0755, true) && ! is_dir($path)) { + throw new RuntimeException(sprintf('Unable to create [%s].', $path)); + } + + // Realpathed because Pest realpaths its own project root, and macOS + // hands out a symlinked temp directory. + $real = realpath($path); + + $project = new self($real === false ? $path : $real); + + self::$created[] = $project; + + self::copy(__DIR__.'/app', $project->path); + + if ($overlay !== null) { + self::copy(__DIR__.'/overlays/'.$overlay, $project->path); + } + + // `ChangedFiles` asks git what changed, so anything the scaffold writes + // but the project does not own has to be invisible to it. + $project->write('.gitignore', implode("\n", ['/vendor/', '/.home/', '/.phpunit.cache/', ''])); + + $project->scaffoldVendor($project->path); + @mkdir($project->home(), 0755, true); + + return $project; + } + + /** + * Mirrors a file or directory, hardlinking where the filesystem allows it + * and copying where it does not. + */ + private static function mirror(string $from, string $to): void + { + if (is_dir($from)) { + foreach (self::contentsOf($from) as $path) { + self::mirror($path->getPathname(), $to.DIRECTORY_SEPARATOR.substr($path->getPathname(), strlen($from) + 1)); + } + + return; + } + + $directory = dirname($to); + + if (! is_dir($directory) && ! @mkdir($directory, 0755, true) && ! is_dir($directory)) { + throw new RuntimeException(sprintf('Unable to create [%s].', $directory)); + } + + if (@link($from, $to) || @copy($from, $to)) { + return; + } + + throw new RuntimeException(sprintf('Unable to mirror [%s] into [%s].', $from, $to)); + } + + /** + * @return iterable<\SplFileInfo> + */ + private static function contentsOf(string $directory): iterable + { + $paths = new RecursiveIteratorIterator( + new RecursiveDirectoryIterator($directory, FilesystemIterator::SKIP_DOTS), + RecursiveIteratorIterator::SELF_FIRST, + ); + + foreach ($paths as $path) { + if (! $path->isDir()) { + yield $path; + } + } + } + + private static function copy(string $from, string $to): void + { + $paths = new RecursiveIteratorIterator( + new RecursiveDirectoryIterator($from, FilesystemIterator::SKIP_DOTS), + RecursiveIteratorIterator::SELF_FIRST, + ); + + foreach ($paths as $path) { + $target = $to.DIRECTORY_SEPARATOR.substr($path->getPathname(), strlen($from) + 1); + + if ($path->isDir()) { + if (! is_dir($target) && ! @mkdir($target, 0755, true) && ! is_dir($target)) { + throw new RuntimeException(sprintf('Unable to create [%s].', $target)); + } + + continue; + } + + if (! is_dir(dirname($target)) && ! @mkdir(dirname($target), 0755, true) && ! is_dir(dirname($target))) { + throw new RuntimeException(sprintf('Unable to create [%s].', dirname($target))); + } + + if (! @copy($path->getPathname(), $target)) { + throw new RuntimeException(sprintf('Unable to copy [%s].', $path->getPathname())); + } + } + } + + private function remove(string $path): void + { + if (! is_dir($path)) { + return; + } + + $paths = new RecursiveIteratorIterator( + new RecursiveDirectoryIterator($path, FilesystemIterator::SKIP_DOTS), + RecursiveIteratorIterator::CHILD_FIRST, + ); + + foreach ($paths as $entry) { + $entry->isDir() && ! $entry->isLink() ? @rmdir($entry->getPathname()) : @unlink($entry->getPathname()); + } + + @rmdir($path); + } +} diff --git a/tests/Fixtures/Tia/app/app/Calculator.php b/tests/Fixtures/Tia/app/app/Calculator.php new file mode 100644 index 00000000..6598f171 --- /dev/null +++ b/tests/Fixtures/Tia/app/app/Calculator.php @@ -0,0 +1,18 @@ + + + + + ./tests + + + + + ./app + + + diff --git a/tests/Fixtures/Tia/app/tests/Feature/CoversCalculatorTest.php b/tests/Fixtures/Tia/app/tests/Feature/CoversCalculatorTest.php new file mode 100644 index 00000000..bebd43ac --- /dev/null +++ b/tests/Fixtures/Tia/app/tests/Feature/CoversCalculatorTest.php @@ -0,0 +1,15 @@ +add(10, 5))->toBe(15); +}); + +test('subtracts within a feature test', function (): void { + expect((new Calculator)->subtract(10, 5))->toBe(5); +}); diff --git a/tests/Fixtures/Tia/app/tests/Pest.php b/tests/Fixtures/Tia/app/tests/Pest.php new file mode 100644 index 00000000..7a4c6ed1 --- /dev/null +++ b/tests/Fixtures/Tia/app/tests/Pest.php @@ -0,0 +1,9 @@ +add(1, 2))->toBe(3); +}); + +test('subtracts two numbers', function (): void { + expect((new Calculator)->subtract(3, 1))->toBe(2); +}); diff --git a/tests/Fixtures/Tia/app/tests/Unit/GreeterTest.php b/tests/Fixtures/Tia/app/tests/Unit/GreeterTest.php new file mode 100644 index 00000000..f4f10dbc --- /dev/null +++ b/tests/Fixtures/Tia/app/tests/Unit/GreeterTest.php @@ -0,0 +1,13 @@ +greet('Nuno'))->toBe('Hello, Nuno!'); +}); + +test('greets the world', function (): void { + expect((new Greeter)->greet('world'))->toBe('Hello, world!'); +}); diff --git a/tests/Fixtures/Tia/overlays/configured-default-branch/tests/Pest.php b/tests/Fixtures/Tia/overlays/configured-default-branch/tests/Pest.php new file mode 100644 index 00000000..37d0447c --- /dev/null +++ b/tests/Fixtures/Tia/overlays/configured-default-branch/tests/Pest.php @@ -0,0 +1,11 @@ +tia()->defaultBranch('master'); diff --git a/tests/Fixtures/Tia/overlays/unknown-default-branch/tests/Pest.php b/tests/Fixtures/Tia/overlays/unknown-default-branch/tests/Pest.php new file mode 100644 index 00000000..55c99b94 --- /dev/null +++ b/tests/Fixtures/Tia/overlays/unknown-default-branch/tests/Pest.php @@ -0,0 +1,10 @@ +tia()->defaultBranch('nope'); diff --git a/tests/Pest.php b/tests/Pest.php index db291c04..6a0988ff 100644 --- a/tests/Pest.php +++ b/tests/Pest.php @@ -19,6 +19,10 @@ pest()->in('PHPUnit/GlobPatternTests/SubFolder2/*AsPattern.php')->use(CustomTest pest()->in('Visual')->group('integration'); +// Every row scaffolds a throwaway project and runs `pest` in it as a +// subprocess, which is far too slow for the unit suite. +pest()->in('Features/Tia')->group('integration'); + // NOTE: global test value container to be mutated and checked across files, as needed $_SERVER['globalHook'] = (object) ['calls' => (object) ['beforeAll' => 0, 'afterAll' => 0]]; diff --git a/tests/Unit/Plugins/Tia/Graph.php b/tests/Unit/Plugins/Tia/Graph.php index 443f19cc..db18f612 100644 --- a/tests/Unit/Plugins/Tia/Graph.php +++ b/tests/Unit/Plugins/Tia/Graph.php @@ -66,8 +66,26 @@ describe('applyMigrationChanges()', function (): void { }); describe('rerun tracking', function (): void { + beforeEach(function (): void { + // `hasUnlocatedTestsToRerun()` stats each recorded file to tell a + // deleted test apart from a live one, so the files have to exist. + $this->projectRoot = sys_get_temp_dir().'/pest-tia-rerun-'.bin2hex(random_bytes(4)); + mkdir($this->projectRoot.'/tests/Feature', 0755, true); + + touch($this->projectRoot.'/tests/Feature/FooTest.php'); + touch($this->projectRoot.'/tests/Feature/BarTest.php'); + }); + + afterEach(function (): void { + @unlink($this->projectRoot.'/tests/Feature/FooTest.php'); + @unlink($this->projectRoot.'/tests/Feature/BarTest.php'); + @rmdir($this->projectRoot.'/tests/Feature'); + @rmdir($this->projectRoot.'/tests'); + @rmdir($this->projectRoot); + }); + it('reruns cached failures via their file', function (): void { - $graph = new Graph(sys_get_temp_dir()); + $graph = new Graph($this->projectRoot); $graph->setResult('main', 'Tests\FooTest::it fails', 7, 'boom', 0.1, 1, 'tests/Feature/FooTest.php'); $graph->setResult('main', 'Tests\BarTest::it passes', 0, '', 0.1, 1, 'tests/Feature/BarTest.php'); @@ -76,7 +94,7 @@ describe('rerun tracking', function (): void { }); it('flags cached failures whose file is unknown', function (): void { - $graph = new Graph(sys_get_temp_dir()); + $graph = new Graph($this->projectRoot); $graph->setResult('main', 'Tests\EvalTest::it fails', 7, 'boom', 0.1, 1); expect($graph->testFilesToRerun('main'))->toBeEmpty()