SADM/pest
1
0
Fork 0
mirror of https://github.com/pestphp/pest.git synced 2026-09-05 06:13:35 +02:00
Code Issues Packages Projects Releases Wiki Activity

qwd

Browse Source
This commit is contained in:
nuno maduro
2026-08-07 02:00:58 +01:00
parent 16d1208344
commit c3ceb7d0c7
1 changed files with 0 additions and 342 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PLAN.md
-342
View File
@@ -1,342 +0,0 @@
# TIA write-tier conformance — plan
Baseline sweep: **147 cases, 144 PASS / 2 FAIL / 1 SKIP** against `dev-fix/tia-filtered` (`db70017`)
on `playground/laravel` with pcov enabled.
This file records (1) what is left to fix, and (2) the full case matrix with the target
outcome for every row, so the sweep can be re-run as a conformance check after the fixes land.
---
## Part 1 — What's left
### 1. `--parallel` silently under-records `edges` (G12, I5) — **high**
Any `--tia --parallel` complete run writes `files=4` where sequential writes `files=25`. Only
`app/**` and blade views survive; every `config/`, `routes/`, `bootstrap/` edge and every test
self-edge is lost. A `routes/web.php` edit that sequential TIA catches, parallel TIA reports as
`No affected tests found`.
Root cause — `src/Restarters/PcovRestarter.php:33`:
```php
if (! Tia::isEnabledForRun($arguments)) {
return;
}
```
`Tia::isEnabledForRun()` (`src/Plugins/Tia.php:275-288`) returns true only for `--tia` in argv or
the `PEST_TIA` env flag. A paratest worker's argv carries neither, so workers never re-exec with
`-d pcov.directory=<projectRoot>` and record under pcov's empty default scope.
Proven both directions:
- `PEST_PCOV_RESTARTER_RESTARTED=1 pest --tia` (sequential, restart suppressed) reproduces
`files=4` with edge sets identical to the parallel graph.
- `PEST_TIA=1 pest --parallel` restores the full `files=25` graph. **Working workaround today.**
Coverage config is irrelevant: widening `phpunit.xml` `<include>` to `app|config|routes|bootstrap|tests`
still yields `files=4`.
Sticky, not just per-run: `pest --tia --filtered --parallel` re-records only the affected test
through a worker and strips that entry's edges from an otherwise-healthy sequential graph, so one
parallel run degrades a good graph incrementally.
**Fix direction:** propagate the TIA-enabled decision into workers (e.g. set `PEST_TIA` in the
worker env, or have the parent stamp a recording global the restarter also consults) so
`PcovRestarter` restarts them. Then assert `edges` equivalence between sequential and parallel.
### 2. Deleted test file with a cached failure wedges `--filtered` forever (I9) — **high**
Expected a WARN and a full-suite fallback. Actual: `--filtered` selects the phantom file, runs
nothing, exits **0 green**, on every subsequent invocation.
Root cause — `src/Plugins/Tia/Graph.php:1523-1525`:
```php
if ($real === false) {
$real = $path;
}
```
`relative()` falls back to the raw path when `realpath()` fails, so a deleted-but-in-project test
file is never "unlocated", `Graph::hasUnlocatedTestsToRerun()` never fires, and the WARN branch at
`src/Plugins/Tia.php:982-986` is unreachable for the deletion case. `pruneMissingTests()` only runs
under `--fresh`, so nothing clears it.
**Fix direction:** distinguish "path could not be resolved" from "path resolved outside the root" in
`relative()`, or have `hasUnlocatedTestsToRerun()` stat the file directly.
### 3. `--coverage-text` does not disable filtered mode (I8) — **low**
`coverageReportActive()` (`src/Plugins/Tia.php:1705-1710`) reads only Pest's own `--coverage`, so
raw PHPUnit coverage flags (`--coverage-text`, `--coverage-html`, `--coverage-clover`, …) leave
filtered mode on. `Tia::COVERAGE_FLAGS` at `:113-115` already enumerates them — that list is not
consulted here.
### 4. `--coverage` narrows edges like parallel does — **medium, needs triage**
`pest --tia --coverage` prints `fresh graph (recording coverage baseline)` even when a graph
exists, and writes `Feature/ExampleTest` with 2 files instead of 16, dropping self-edges. Same
observable shape as G12, likely a different mechanism (piggyback collector scoped by
`phpunit.xml <include>` rather than the pcov-restarted recorder). Theme: **edges silently narrow to
`app/**` whenever recording does not go through the pcov-restarted TIA recorder.** Confirm whether
these are one fix or two.
### 5. Smaller items
| Item | Where | Note |
|---|---|---|
| Warning/deprecation recorded as `status=0` | write path | Codes `6` and `4` appear unreachable; risky/skipped/incomplete record faithfully |
| Complete non-TIA runs write edge-less results | `src/Plugins/Tia.php:1645`, `:638` | Guard is `! $complete && ! knowsTest()`; complete bypasses it and `markKnownTestFiles` stays false. Inert — replay guards on the same predicate at `:360` — but inflates `n` |
| SIGINT never reaches the suite | `PcovRestarter` re-exec | Parent ignores it; the child holds PHPUnit's handler. CI `timeout`/Ctrl-C will not stop a run |
| Structural drift never announced sequentially | `src/Plugins/Tia.php:1178-1181` | `enterRecordMode()` prints a bare `Running in TIA mode.`; `renderFreshGraph()` (`fresh graph (composer.lock changed)`) only runs under `--parallel` or coverage piggyback |
| ~~Replay clobbers cached `time`~~ | write path | **Struck in phase four — does not reproduce**, sequentially or in parallel. Both write paths route through `resultTime()`; a replay with every cached `time` sentinelled writes nothing |
| ~~`pest --parallel` without `--tia` writes nothing~~ | G3, G6 | **Closed in phase four.** Workers flush their results through `requestWorkerResults()`, so a parallel run refreshes and prunes exactly like the sequential run of the same command |
| ~~`--tia --parallel --filter`/`--shard` record nothing~~ | G2, G8 | **Closed in phase four** (partial parity landed first, complete parity with the row above) |
| `--min=50` without `--coverage` is a silent no-op | — | — |
| `pest --repeat=2` is not a Pest option | J11 | Case unrunnable as written; drop it or add the option |
### 6. Test-harness gaps to close before re-running
- The playground has no `UsesClass`, `->note()`, `->flaky()`, `->issue()`, `->pr()`, `->ticket()` or
`->assignee()` annotations, so **C19, C23–C28, C37–C39 matched zero tests** and their
graph-invariant assertions proved nothing. Add annotated fixtures to make those rows load-bearing.
- Use **sentinel patching** (rewrite every cached entry to `time=9.999 assertions=42`, then see which
entries get overwritten) as the discriminator. It is the only way to tell "wrote identical values"
from "wrote nothing". A canary test absent from `edges` is unreliable under `--filter` because it
never matches the filter and so never runs.
- `--tia` on a clean green tree can never cache a failure (unchanged tests replay rather than
execute); seeding one requires `--fresh` or an env-driven flaky fixture.
- Comment-only edits to a test file are **not** changes (AST-level hashing). Use semantic edits.
---
## Part 2 — Full case matrix (target outcomes)
Tiers: **COMPLETE** may change everything · **RESULTS-ONLY** (RO) may change only
`baselines[<branch>].results` for tests that ran, and must never remove an entry, add a result for a
test file absent from `edges`, or alter `sha`/`tree`/`edges`/`files`/`fingerprint` ·
**HARD-SUPPRESSED** may change nothing.
Status column: `PASS` = conforming today · `FIX` = must pass once the item above lands ·
`SKIP` = case not runnable as written.
### A — Setup & sanity
| # | Case | Target outcome | Status |
|---|---|---|---|
| A1 | `composer show pestphp/pest` | version `dev-fix/tia-filtered` | PASS |
| A2 | `pest --baseline` | prints an existing dir; exit 0 | PASS |
| A3 | delete graph, `pest --tia` | graph.json created; one `edges` key per test file; one result per test | PASS |
| A4 | `git status` after reset ritual | clean | PASS |
| A5 | `extension_loaded("pcov")` | `true` | PASS |
| A6 | delete graph, plain `pest` | no graph created | PASS |
| A7 | delete graph, `pest --filter=adds` | no graph created | PASS |
| A8 | two consecutive `pest --tia` | second replays everything; `sha` unchanged | PASS |
### B — COMPLETE runs still write
| # | Case | Target outcome | Status |
|---|---|---|---|
| B1 | `pest --tia` (clean) | replays; graph written; `sha` = `git rev-parse HEAD` | PASS |
| B2 | `pest --no-tia` | results refreshed; prune applied; `sha`/`tree`/`edges` unchanged | PASS |
| B3 | `pest` (plain) | same as B2 | PASS |
| B4 | `pest --tia --fresh` | graph purged and rebuilt; `edges` rebuilt; `files` repopulated | PASS |
| B5 | `pest --bail`, all green | COMPLETE — graph written, prune applied | PASS |
| B6 | edit `Calculator.php`, `pest --tia` | `CalculatorTest` re-runs, others replay; `tree` updated | PASS |
| B7 | delete `trio three`, `pest --tia` | its result pruned; `trio one`/`two` remain | PASS |
| B8 | delete `GreeterTest.php`, `pest --tia` | result and edges survive (missing-file prune is `--fresh`-only) | PASS |
| B9 | B8 then `pest --tia --fresh` | entry gone; `files` drops `Greeter.php` too | PASS |
| B10 | `pest --tia` twice | `time` differs; statuses stable | PASS |
| B11 | add a test file, `pest --tia` | new `edges` key + new result appear | PASS |
| B12 | `pest --tia --coverage` | completes; graph written | PASS |
### C — Selection narrowing → RESULTS-ONLY
All rows: RO invariants hold. "Notice" = `TIA does not apply to partial runs — running the selected tests directly.`
| # | Case | Target outcome | Status |
|---|---|---|---|
| C1 | `pest --filter="adds numbers"` | RO; no notice | PASS |
| C2 | `pest --tia --filter="adds numbers"` | RO; notice | PASS |
| C3 | `pest --filter="trio one"` | RO; `trio two`/`three` byte-identical | PASS |
| C4 | `pest --filter="trio"` | RO; all three update; nothing else does | PASS |
| C5 | `pest --exclude-filter="Feature"` | RO; no notice; no prune | PASS |
| C6 | `pest --tia --exclude-filter="Feature"` | RO; notice | PASS |
| C7 | `pest --group=smoke` | RO; no notice | PASS |
| C8 | `pest --tia --group=smoke` | RO; notice | PASS |
| C9 | `pest --exclude-group=smoke` | RO; no notice; no prune | PASS |
| C10 | `pest --tia --exclude-group=smoke` | RO; notice | PASS |
| C11 | `pest tests/Unit` | RO; no notice; Feature entry not pruned | PASS |
| C12 | `pest --tia tests/Unit` | RO; notice | PASS |
| C13 | `pest tests/Unit/TrioTest.php` | RO; no notice | PASS |
| C14 | `pest --tia tests/Unit/TrioTest.php` | RO; notice | PASS |
| C15 | `pest --testsuite=Unit` | RO; no notice | PASS |
| C16 | `pest --tia --testsuite=Unit` | RO; notice | PASS |
| C17 | `pest --tia --exclude-testsuite=Feature` | RO; notice | PASS |
| C18 | `pest --tia --covers='App\Services\Calculator'` | RO; notice | PASS |
| C19 | `pest --tia --uses='App\Services\Calculator'` | RO; notice | PASS (vacuous — needs fixture) |
| C20 | `pest --tia --test-suffix=Test.php` | RO even though all tests run; `edges`/`files`/`n` unchanged | PASS |
| C21 | `pest --dirty` with an uncommitted test edit | RO; no notice | PASS |
| C22 | `pest --tia --dirty` | RO; notice | PASS |
| C23 | `pest --tia --todos` | RO; notice | PASS (vacuous) |
| C24 | `pest --tia --notes` | RO; notice | PASS (vacuous) |
| C25 | `pest --tia --flaky` | RO; notice | PASS (vacuous) |
| C26 | `pest --tia --issue=123` | RO; notice | PASS (vacuous) |
| C27 | `pest --tia --pr=1` | RO; notice | PASS (vacuous) |
| C28 | `pest --tia --pull-request=1` | RO; notice | PASS (vacuous) |
| C29 | `pest --tia --shard=1/2` | RO; notice; no prune | PASS |
| C30 | `pest --tia --shard=2/2` | RO; notice; C29+C30 covers the suite; neither prunes | PASS |
| C31 | `->only()` on `trio one`, `pest --tia` | RO; no notice; siblings untouched | PASS |
| C32 | `->only()` on `trio one`, plain `pest` | RO; no notice | PASS |
| C33 | `pest --tia --filtered --filter=adds` | RO; notice; filtered yields to narrowing | PASS |
| C34 | `PEST_TIA=1 pest --filter=adds` | RO; notice | PASS |
| C35 | `PEST_TIA_FILTERED=1 pest --filter=adds` | RO; notice (regression guard) | PASS |
| C36 | `pest --filtered --filter=adds` | RO; notice | PASS |
| C37 | `pest --tia --ticket=X` | RO; notice. `--ticket` **is** a real option (`src/Plugins/Snapshot.php:83`) — it narrows legitimately | PASS (vacuous) |
| C38 | `pest --tia --assignee=X` | RO; notice (`src/Plugins/Snapshot.php:81`) | PASS (vacuous) |
| C39 | `pest --tia --todo` | RO; notice; matches nothing | PASS (vacuous) |
### D — Truncation → RESULTS-ONLY
D1–D5, D11–D13, D16 precondition: `trio one` broken. D6–D10 run against a green suite so the `--stop-on-*` trigger is the only narrowing.
| # | Case | Target outcome | Status |
|---|---|---|---|
| D1 | `pest --bail` | RO; `trio one` = `status=7`; siblings survive unchanged | PASS |
| D2 | `pest --retry` | RO; siblings survive | PASS |
| D3 | `pest --stop-on-failure` | RO | PASS |
| D4 | `pest --stop-on-defect` | RO | PASS |
| D5 | `pest --stop-on-error` (no error occurs) | COMPLETE — nothing stopped it | PASS |
| D6 | `pest --stop-on-warning` | RO when it fires. Warning stored as `status=0`, not `6` | PASS |
| D7 | `pest --stop-on-risky --disallow-test-output` | RO; `status=5` | PASS |
| D8 | `pest --stop-on-skipped` | RO; `status=1` | PASS |
| D9 | `pest --stop-on-incomplete` | RO; `status=2` | PASS |
| D10 | `pest --stop-on-deprecation` | RO. Deprecation stored as `status=0`, not `4` | PASS |
| D11 | `pest --tia --bail` | RO | PASS |
| D12 | `pest --bail --filter=trio` | RO (narrowed *and* truncated) | PASS |
| D13 | `stopOnFailure="true"` in `phpunit.xml`, plain `pest` | RO — caught via `stoppedEarly()`, not flag matching | PASS |
| D14 | D13 config, green suite | COMPLETE | PASS |
| D15 | `pest --bail`, last test in run order fails | RO — over-conservative but safe; nothing was skipped | PASS |
| D16 | `pest --tia --filtered --bail`, broken affected test | RO | PASS |
### E — Result merge semantics
| # | Case | Target outcome | Status |
|---|---|---|---|
| E1 | cached `flaky`=7, `FLAKY_OK=1 pest --filter=flaky` | flips to `status=0`; all other results byte-identical | PASS |
| E2 | cached `flaky`=0, `pest --filter=flaky` (env unset) | flips to `status=7` | PASS |
| E3 | after E1, `pest --tia --filtered` clean | `No affected tests found`; zero graph delta | PASS |
| E4 | after E2, `pest --tia --filtered` | re-runs, `from 1 previously unsuccessful test` | PASS |
| E5 | any partial run | `assertions` and `time` update for the test that ran | PASS |
| E6 | any partial run | `message` of untouched tests unchanged | PASS |
| E7 | two sequential partial runs on different tests | each updates only its own entry; both persist | PASS |
| E8 | `pest --filter="trio one"` | siblings keep exact `status`/`time`/`assertions`/`message` | PASS |
| E9 | partial run of a `->skip()`ed test | `status=1` | PASS |
| E10 | partial run of a `->todo()` test | `status=1`, `assertions=0`, `message="__TODO__"` | PASS |
| E11 | partial run of a risky test | `status=5` | PASS |
| E12 | any partial run | `fingerprint` byte-identical | PASS |
| E13 | dataset test, `--filter` matching one row | that row updates; other rows survive (the old prune bug) | PASS |
| E14 | dataset test, full `--tia` after deleting a row | the deleted row is pruned | PASS |
### F — Guard rails
| # | Case | Target outcome | Status |
|---|---|---|---|
| F1 | test absent from `edges`, `pest --filter="brand new"` | no result recorded | PASS |
| F2 | complete `pest --tia` first, then the same filter | result is recorded | PASS |
| F3 | `pest --mutate --path=app/Services --covered-only` | statuses/`edges`/`sha`/`tree`/`files` identical; only `time` may change | PASS |
| F4 | `pest --mutate --parallel --path=… --covered-only` | same (observed: zero delta) | PASS |
| F5 | weakened test so a mutant survives, `pest --mutate` | no status change in the baseline | PASS |
| F6 | `pest --mutate` with the graph deleted | no graph created | PASS |
| F7 | mutation subprocess env | carries `PEST_MUTATION_TESTING`; `--mutate` stripped from argv | PASS |
| F8 | grep baseline after `--mutate` | no mutation-flavoured messages | PASS |
### G — Parallel
| # | Case | Target outcome | Status |
|---|---|---|---|
| G1 | `pest --tia --parallel` (clean) | COMPLETE — graph written | PASS |
| G2 | `pest --tia --parallel --filter=adds` | RO | PASS |
| G3 | `pest --parallel --bail`, `trio one` broken | RO; siblings survive | PASS |
| G4 | `pest --tia --parallel --bail` | RO | PASS |
| G5 | `pest --tia --filtered --parallel` after a source edit | narrows to affected; writes | PASS (but see G12 — it strips edges) |
| G6 | `pest --tia --parallel` | worker results reach the parent baseline | PASS |
| G7 | `->only()` + `pest --tia --parallel` | RO | PASS |
| G8 | `pest --tia --parallel --shard=1/2` | RO | PASS |
| G9 | broken test in one worker, `pest --parallel --bail` | whole run is RO, not just that worker's slice | PASS |
| G10 | `pest --parallel --retry` | `InvalidOption`; graph untouched | PASS |
| G11 | `pest --tia --parallel --fresh` | graph rebuilt | PASS |
| G12 | `edges` after `pest --tia` vs `pest --tia --parallel --fresh` | **equivalent edge sets** (`files=25` both) | **FIX (item 1)** |
### H — Baseline key / branch resolution
| # | Case | Target outcome | Status |
|---|---|---|---|
| H1 | `pest --tia` on `master` | only a `master` key | PASS |
| H2 | plain `pest` on `master` | writes `master`, not `main` | PASS |
| H3 | `pest --bail` truncated on `master` | no `main` key minted | PASS |
| H4 | `pest --filter=adds` on `master` | no `main` key minted | PASS |
| H5 | `git branch -m main`, full `pest --tia` | single `main` key | PASS |
| H6 | `git checkout -b feature/x`, `pest --tia` | `feature/x` key appears; reads fall back to the existing baseline | PASS |
| H7 | detached HEAD, `pest --tia` | falls back to the `main` key for both reads and writes; no `HEAD` key | PASS |
| H8 | after every C and D case | no baseline key other than the real branch | PASS |
| H9 | non-git dir, `pest --tia` | `MissingDependency` — `The feature "Tia mode" requires "git".` | PASS |
| H10 | non-git dir, plain `pest` | runs normally; no baseline dir created | PASS |
### I — Filtered mode
| # | Case | Target outcome | Status |
|---|---|---|---|
| I1 | clean + green, `pest --tia --filtered` | `No affected tests found`; zero graph delta | PASS |
| I2 | edit `Calculator.php` | only `CalculatorTest` runs | PASS |
| I3 | cached failure, clean tree | re-runs it, `from 1 previously unsuccessful test` | PASS |
| I4 | I2 with sentinels | `tree` updated; exactly one entry rewritten; all others retained | PASS |
| I5 | `pest --tia --filtered --parallel` | same narrowing as I2 **and `edges` unchanged** | **FIX (item 1)** |
| I6 | `PEST_TIA_FILTERED=1 pest --tia` | identical delta to I4 | PASS |
| I7 | `pest --tia --filtered tests/Unit` | explicit path wins; filtered off; RO; notice | PASS |
| I8 | `pest --tia --filtered --coverage-text` | filtered mode disabled by an active coverage report | **FIX (item 3)** |
| I9 | cached failure whose test file was deleted | WARN `could not be located on disk`; falls back to the full suite with replay | **FIX (item 2)** |
| I10 | `pest --tia --filtered` with no baseline yet | records a baseline instead of filtering | PASS |
| I11 | edit a Blade view a Feature test renders | the Feature test is selected | PASS |
| I12 | edit `composer.lock` | fingerprint drift → full rebuild (and the drift reason should be printed — item 5) | PASS |
### J — Interactions & regressions
| # | Case | Target outcome | Status |
|---|---|---|---|
| J1 | `pest --tia --fresh --filter=adds` | graph **not** purged; RO | PASS |
| J2 | `pest --tia --refetch --filter=adds` | refetch not performed; RO | PASS |
| J3 | `pest --no-tia --filter=adds` | RO — narrowing wins over `--no-tia`; no notice | PASS |
| J4 | `pest --tia --no-tia` | TIA disabled; COMPLETE | PASS |
| J5 | `<groups><exclude><group>integration</group></exclude></groups>`, `pest --tia` | COMPLETE, no notice — an always-in-force filter applies to baseline runs too | PASS |
| J6 | `pest --tia --compact` | COMPLETE | PASS |
| J7 | `pest --tia -v` | COMPLETE | PASS |
| J8 | `pest --tia --profile` | COMPLETE | PASS |
| J9 | `pest --tia --order-by=random` | COMPLETE | PASS |
| J10 | `pest --tia --random-order-seed=1234` | COMPLETE | PASS |
| J11 | `pest --tia --repeat=2` | COMPLETE — repetition is not narrowing | **SKIP** — `--repeat` is not a Pest option |
| J12 | `pest --filter=adds` twice | only the executed test's `time` differs; rest byte-identical | PASS |
| J13 | `pest --tia --min=50` | COMPLETE | PASS |
| J14 | SIGINT mid-suite | RO — an interrupted run is truncated | PASS (signal must go to the re-exec'd child — item 5) |
| J15 | delete the graph, `pest --tia --filtered` | records a baseline rather than erroring | PASS |
| J16 | corrupt `graph.json`, `pest --tia` | recovers by rebuilding; no crash | PASS |
---
## Re-run procedure
```bash
cd /Users/nunomaduro/Work/projects/playground/laravel
GRAPH="$(PAO_DISABLE=1 ./vendor/bin/pest --baseline)/graph.json"
# per case
git checkout -q . && git clean -qfd tests app
rm -rf "$(PAO_DISABLE=1 ./vendor/bin/pest --baseline)"
PAO_DISABLE=1 ./vendor/bin/pest --tia >/dev/null 2>&1
# sentinel-patch every result to time=9.999 assertions=42, snapshot, run the case, diff
```
Every pest invocation must be prefixed with `PAO_DISABLE=1` (the app has `laravel/pao`, which emits
JSON when it detects an agent). The shell is zsh — build commands with arrays or `eval`; unquoted
`$args` does not word-split.