mirror of
https://github.com/pestphp/pest.git
synced 2026-09-05 14:23:34 +02:00
Compare commits
9 Commits
7bfb2a6185
...
d0572ce138
| Author | SHA1 | Date | |
|---|---|---|---|
 nuno maduro
|
d0572ce138 | ||
|
nuno maduro
|
b0d1fc589d | ||
|
nuno maduro
|
5a4f124199 | ||
|
nuno maduro
|
c3ceb7d0c7 | ||
|
nuno maduro
|
16d1208344 | ||
|
nuno maduro
|
1cebc84c2d | ||
|
nuno maduro
|
1f79660add | ||
|
nuno maduro
|
37bdf60a1c | ||
|
nuno maduro
|
cc7acfe485 |
@@ -1,342 +0,0 @@
|
||||
# TIA write-tier conformance — plan
|
||||
|
||||
Baseline sweep: **147 cases, 144 PASS / 2 FAIL / 1 SKIP** against `dev-fix/tia-filtered` (`db70017`)
|
||||
on `playground/laravel` with pcov enabled.
|
||||
|
||||
This file records (1) what is left to fix, and (2) the full case matrix with the target
|
||||
outcome for every row, so the sweep can be re-run as a conformance check after the fixes land.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — What's left
|
||||
|
||||
### 1. `--parallel` silently under-records `edges` (G12, I5) — **high**
|
||||
|
||||
Any `--tia --parallel` complete run writes `files=4` where sequential writes `files=25`. Only
|
||||
`app/**` and blade views survive; every `config/`, `routes/`, `bootstrap/` edge and every test
|
||||
self-edge is lost. A `routes/web.php` edit that sequential TIA catches, parallel TIA reports as
|
||||
`No affected tests found`.
|
||||
|
||||
Root cause — `src/Restarters/PcovRestarter.php:33`:
|
||||
|
||||
```php
|
||||
if (! Tia::isEnabledForRun($arguments)) {
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
`Tia::isEnabledForRun()` (`src/Plugins/Tia.php:275-288`) returns true only for `--tia` in argv or
|
||||
the `PEST_TIA` env flag. A paratest worker's argv carries neither, so workers never re-exec with
|
||||
`-d pcov.directory=<projectRoot>` and record under pcov's empty default scope.
|
||||
|
||||
Proven both directions:
|
||||
- `PEST_PCOV_RESTARTER_RESTARTED=1 pest --tia` (sequential, restart suppressed) reproduces
|
||||
`files=4` with edge sets identical to the parallel graph.
|
||||
- `PEST_TIA=1 pest --parallel` restores the full `files=25` graph. **Working workaround today.**
|
||||
|
||||
Coverage config is irrelevant: widening `phpunit.xml` `<include>` to `app|config|routes|bootstrap|tests`
|
||||
still yields `files=4`.
|
||||
|
||||
Sticky, not just per-run: `pest --tia --filtered --parallel` re-records only the affected test
|
||||
through a worker and strips that entry's edges from an otherwise-healthy sequential graph, so one
|
||||
parallel run degrades a good graph incrementally.
|
||||
|
||||
**Fix direction:** propagate the TIA-enabled decision into workers (e.g. set `PEST_TIA` in the
|
||||
worker env, or have the parent stamp a recording global the restarter also consults) so
|
||||
`PcovRestarter` restarts them. Then assert `edges` equivalence between sequential and parallel.
|
||||
|
||||
### 2. Deleted test file with a cached failure wedges `--filtered` forever (I9) — **high**
|
||||
|
||||
Expected a WARN and a full-suite fallback. Actual: `--filtered` selects the phantom file, runs
|
||||
nothing, exits **0 green**, on every subsequent invocation.
|
||||
|
||||
Root cause — `src/Plugins/Tia/Graph.php:1523-1525`:
|
||||
|
||||
```php
|
||||
if ($real === false) {
|
||||
$real = $path;
|
||||
}
|
||||
```
|
||||
|
||||
`relative()` falls back to the raw path when `realpath()` fails, so a deleted-but-in-project test
|
||||
file is never "unlocated", `Graph::hasUnlocatedTestsToRerun()` never fires, and the WARN branch at
|
||||
`src/Plugins/Tia.php:982-986` is unreachable for the deletion case. `pruneMissingTests()` only runs
|
||||
under `--fresh`, so nothing clears it.
|
||||
|
||||
**Fix direction:** distinguish "path could not be resolved" from "path resolved outside the root" in
|
||||
`relative()`, or have `hasUnlocatedTestsToRerun()` stat the file directly.
|
||||
|
||||
### 3. `--coverage-text` does not disable filtered mode (I8) — **low**
|
||||
|
||||
`coverageReportActive()` (`src/Plugins/Tia.php:1705-1710`) reads only Pest's own `--coverage`, so
|
||||
raw PHPUnit coverage flags (`--coverage-text`, `--coverage-html`, `--coverage-clover`, …) leave
|
||||
filtered mode on. `Tia::COVERAGE_FLAGS` at `:113-115` already enumerates them — that list is not
|
||||
consulted here.
|
||||
|
||||
### 4. `--coverage` narrows edges like parallel does — **medium, needs triage**
|
||||
|
||||
`pest --tia --coverage` prints `fresh graph (recording coverage baseline)` even when a graph
|
||||
exists, and writes `Feature/ExampleTest` with 2 files instead of 16, dropping self-edges. Same
|
||||
observable shape as G12, likely a different mechanism (piggyback collector scoped by
|
||||
`phpunit.xml <include>` rather than the pcov-restarted recorder). Theme: **edges silently narrow to
|
||||
`app/**` whenever recording does not go through the pcov-restarted TIA recorder.** Confirm whether
|
||||
these are one fix or two.
|
||||
|
||||
### 5. Smaller items
|
||||
|
||||
| Item | Where | Note |
|
||||
|---|---|---|
|
||||
| Warning/deprecation recorded as `status=0` | write path | Codes `6` and `4` appear unreachable; risky/skipped/incomplete record faithfully |
|
||||
| Complete non-TIA runs write edge-less results | `src/Plugins/Tia.php:1645`, `:638` | Guard is `! $complete && ! knowsTest()`; complete bypasses it and `markKnownTestFiles` stays false. Inert — replay guards on the same predicate at `:360` — but inflates `n` |
|
||||
| SIGINT never reaches the suite | `PcovRestarter` re-exec | Parent ignores it; the child holds PHPUnit's handler. CI `timeout`/Ctrl-C will not stop a run |
|
||||
| Structural drift never announced sequentially | `src/Plugins/Tia.php:1178-1181` | `enterRecordMode()` prints a bare `Running in TIA mode.`; `renderFreshGraph()` (`fresh graph (composer.lock changed)`) only runs under `--parallel` or coverage piggyback |
|
||||
| ~~Replay clobbers cached `time`~~ | write path | **Struck in phase four — does not reproduce**, sequentially or in parallel. Both write paths route through `resultTime()`; a replay with every cached `time` sentinelled writes nothing |
|
||||
| ~~`pest --parallel` without `--tia` writes nothing~~ | G3, G6 | **Closed in phase four.** Workers flush their results through `requestWorkerResults()`, so a parallel run refreshes and prunes exactly like the sequential run of the same command |
|
||||
| ~~`--tia --parallel --filter`/`--shard` record nothing~~ | G2, G8 | **Closed in phase four** (partial parity landed first, complete parity with the row above) |
|
||||
| `--min=50` without `--coverage` is a silent no-op | — | — |
|
||||
| `pest --repeat=2` is not a Pest option | J11 | Case unrunnable as written; drop it or add the option |
|
||||
|
||||
### 6. Test-harness gaps to close before re-running
|
||||
|
||||
- The playground has no `UsesClass`, `->note()`, `->flaky()`, `->issue()`, `->pr()`, `->ticket()` or
|
||||
`->assignee()` annotations, so **C19, C23–C28, C37–C39 matched zero tests** and their
|
||||
graph-invariant assertions proved nothing. Add annotated fixtures to make those rows load-bearing.
|
||||
- Use **sentinel patching** (rewrite every cached entry to `time=9.999 assertions=42`, then see which
|
||||
entries get overwritten) as the discriminator. It is the only way to tell "wrote identical values"
|
||||
from "wrote nothing". A canary test absent from `edges` is unreliable under `--filter` because it
|
||||
never matches the filter and so never runs.
|
||||
- `--tia` on a clean green tree can never cache a failure (unchanged tests replay rather than
|
||||
execute); seeding one requires `--fresh` or an env-driven flaky fixture.
|
||||
- Comment-only edits to a test file are **not** changes (AST-level hashing). Use semantic edits.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — Full case matrix (target outcomes)
|
||||
|
||||
Tiers: **COMPLETE** may change everything · **RESULTS-ONLY** (RO) may change only
|
||||
`baselines[<branch>].results` for tests that ran, and must never remove an entry, add a result for a
|
||||
test file absent from `edges`, or alter `sha`/`tree`/`edges`/`files`/`fingerprint` ·
|
||||
**HARD-SUPPRESSED** may change nothing.
|
||||
|
||||
Status column: `PASS` = conforming today · `FIX` = must pass once the item above lands ·
|
||||
`SKIP` = case not runnable as written.
|
||||
|
||||
### A — Setup & sanity
|
||||
|
||||
| # | Case | Target outcome | Status |
|
||||
|---|---|---|---|
|
||||
| A1 | `composer show pestphp/pest` | version `dev-fix/tia-filtered` | PASS |
|
||||
| A2 | `pest --baseline` | prints an existing dir; exit 0 | PASS |
|
||||
| A3 | delete graph, `pest --tia` | graph.json created; one `edges` key per test file; one result per test | PASS |
|
||||
| A4 | `git status` after reset ritual | clean | PASS |
|
||||
| A5 | `extension_loaded("pcov")` | `true` | PASS |
|
||||
| A6 | delete graph, plain `pest` | no graph created | PASS |
|
||||
| A7 | delete graph, `pest --filter=adds` | no graph created | PASS |
|
||||
| A8 | two consecutive `pest --tia` | second replays everything; `sha` unchanged | PASS |
|
||||
|
||||
### B — COMPLETE runs still write
|
||||
|
||||
| # | Case | Target outcome | Status |
|
||||
|---|---|---|---|
|
||||
| B1 | `pest --tia` (clean) | replays; graph written; `sha` = `git rev-parse HEAD` | PASS |
|
||||
| B2 | `pest --no-tia` | results refreshed; prune applied; `sha`/`tree`/`edges` unchanged | PASS |
|
||||
| B3 | `pest` (plain) | same as B2 | PASS |
|
||||
| B4 | `pest --tia --fresh` | graph purged and rebuilt; `edges` rebuilt; `files` repopulated | PASS |
|
||||
| B5 | `pest --bail`, all green | COMPLETE — graph written, prune applied | PASS |
|
||||
| B6 | edit `Calculator.php`, `pest --tia` | `CalculatorTest` re-runs, others replay; `tree` updated | PASS |
|
||||
| B7 | delete `trio three`, `pest --tia` | its result pruned; `trio one`/`two` remain | PASS |
|
||||
| B8 | delete `GreeterTest.php`, `pest --tia` | result and edges survive (missing-file prune is `--fresh`-only) | PASS |
|
||||
| B9 | B8 then `pest --tia --fresh` | entry gone; `files` drops `Greeter.php` too | PASS |
|
||||
| B10 | `pest --tia` twice | `time` differs; statuses stable | PASS |
|
||||
| B11 | add a test file, `pest --tia` | new `edges` key + new result appear | PASS |
|
||||
| B12 | `pest --tia --coverage` | completes; graph written | PASS |
|
||||
|
||||
### C — Selection narrowing → RESULTS-ONLY
|
||||
|
||||
All rows: RO invariants hold. "Notice" = `TIA does not apply to partial runs — running the selected tests directly.`
|
||||
|
||||
| # | Case | Target outcome | Status |
|
||||
|---|---|---|---|
|
||||
| C1 | `pest --filter="adds numbers"` | RO; no notice | PASS |
|
||||
| C2 | `pest --tia --filter="adds numbers"` | RO; notice | PASS |
|
||||
| C3 | `pest --filter="trio one"` | RO; `trio two`/`three` byte-identical | PASS |
|
||||
| C4 | `pest --filter="trio"` | RO; all three update; nothing else does | PASS |
|
||||
| C5 | `pest --exclude-filter="Feature"` | RO; no notice; no prune | PASS |
|
||||
| C6 | `pest --tia --exclude-filter="Feature"` | RO; notice | PASS |
|
||||
| C7 | `pest --group=smoke` | RO; no notice | PASS |
|
||||
| C8 | `pest --tia --group=smoke` | RO; notice | PASS |
|
||||
| C9 | `pest --exclude-group=smoke` | RO; no notice; no prune | PASS |
|
||||
| C10 | `pest --tia --exclude-group=smoke` | RO; notice | PASS |
|
||||
| C11 | `pest tests/Unit` | RO; no notice; Feature entry not pruned | PASS |
|
||||
| C12 | `pest --tia tests/Unit` | RO; notice | PASS |
|
||||
| C13 | `pest tests/Unit/TrioTest.php` | RO; no notice | PASS |
|
||||
| C14 | `pest --tia tests/Unit/TrioTest.php` | RO; notice | PASS |
|
||||
| C15 | `pest --testsuite=Unit` | RO; no notice | PASS |
|
||||
| C16 | `pest --tia --testsuite=Unit` | RO; notice | PASS |
|
||||
| C17 | `pest --tia --exclude-testsuite=Feature` | RO; notice | PASS |
|
||||
| C18 | `pest --tia --covers='App\Services\Calculator'` | RO; notice | PASS |
|
||||
| C19 | `pest --tia --uses='App\Services\Calculator'` | RO; notice | PASS (vacuous — needs fixture) |
|
||||
| C20 | `pest --tia --test-suffix=Test.php` | RO even though all tests run; `edges`/`files`/`n` unchanged | PASS |
|
||||
| C21 | `pest --dirty` with an uncommitted test edit | RO; no notice | PASS |
|
||||
| C22 | `pest --tia --dirty` | RO; notice | PASS |
|
||||
| C23 | `pest --tia --todos` | RO; notice | PASS (vacuous) |
|
||||
| C24 | `pest --tia --notes` | RO; notice | PASS (vacuous) |
|
||||
| C25 | `pest --tia --flaky` | RO; notice | PASS (vacuous) |
|
||||
| C26 | `pest --tia --issue=123` | RO; notice | PASS (vacuous) |
|
||||
| C27 | `pest --tia --pr=1` | RO; notice | PASS (vacuous) |
|
||||
| C28 | `pest --tia --pull-request=1` | RO; notice | PASS (vacuous) |
|
||||
| C29 | `pest --tia --shard=1/2` | RO; notice; no prune | PASS |
|
||||
| C30 | `pest --tia --shard=2/2` | RO; notice; C29+C30 covers the suite; neither prunes | PASS |
|
||||
| C31 | `->only()` on `trio one`, `pest --tia` | RO; no notice; siblings untouched | PASS |
|
||||
| C32 | `->only()` on `trio one`, plain `pest` | RO; no notice | PASS |
|
||||
| C33 | `pest --tia --filtered --filter=adds` | RO; notice; filtered yields to narrowing | PASS |
|
||||
| C34 | `PEST_TIA=1 pest --filter=adds` | RO; notice | PASS |
|
||||
| C35 | `PEST_TIA_FILTERED=1 pest --filter=adds` | RO; notice (regression guard) | PASS |
|
||||
| C36 | `pest --filtered --filter=adds` | RO; notice | PASS |
|
||||
| C37 | `pest --tia --ticket=X` | RO; notice. `--ticket` **is** a real option (`src/Plugins/Snapshot.php:83`) — it narrows legitimately | PASS (vacuous) |
|
||||
| C38 | `pest --tia --assignee=X` | RO; notice (`src/Plugins/Snapshot.php:81`) | PASS (vacuous) |
|
||||
| C39 | `pest --tia --todo` | RO; notice; matches nothing | PASS (vacuous) |
|
||||
|
||||
### D — Truncation → RESULTS-ONLY
|
||||
|
||||
D1–D5, D11–D13, D16 precondition: `trio one` broken. D6–D10 run against a green suite so the `--stop-on-*` trigger is the only narrowing.
|
||||
|
||||
| # | Case | Target outcome | Status |
|
||||
|---|---|---|---|
|
||||
| D1 | `pest --bail` | RO; `trio one` = `status=7`; siblings survive unchanged | PASS |
|
||||
| D2 | `pest --retry` | RO; siblings survive | PASS |
|
||||
| D3 | `pest --stop-on-failure` | RO | PASS |
|
||||
| D4 | `pest --stop-on-defect` | RO | PASS |
|
||||
| D5 | `pest --stop-on-error` (no error occurs) | COMPLETE — nothing stopped it | PASS |
|
||||
| D6 | `pest --stop-on-warning` | RO when it fires. Warning stored as `status=0`, not `6` | PASS |
|
||||
| D7 | `pest --stop-on-risky --disallow-test-output` | RO; `status=5` | PASS |
|
||||
| D8 | `pest --stop-on-skipped` | RO; `status=1` | PASS |
|
||||
| D9 | `pest --stop-on-incomplete` | RO; `status=2` | PASS |
|
||||
| D10 | `pest --stop-on-deprecation` | RO. Deprecation stored as `status=0`, not `4` | PASS |
|
||||
| D11 | `pest --tia --bail` | RO | PASS |
|
||||
| D12 | `pest --bail --filter=trio` | RO (narrowed *and* truncated) | PASS |
|
||||
| D13 | `stopOnFailure="true"` in `phpunit.xml`, plain `pest` | RO — caught via `stoppedEarly()`, not flag matching | PASS |
|
||||
| D14 | D13 config, green suite | COMPLETE | PASS |
|
||||
| D15 | `pest --bail`, last test in run order fails | RO — over-conservative but safe; nothing was skipped | PASS |
|
||||
| D16 | `pest --tia --filtered --bail`, broken affected test | RO | PASS |
|
||||
|
||||
### E — Result merge semantics
|
||||
|
||||
| # | Case | Target outcome | Status |
|
||||
|---|---|---|---|
|
||||
| E1 | cached `flaky`=7, `FLAKY_OK=1 pest --filter=flaky` | flips to `status=0`; all other results byte-identical | PASS |
|
||||
| E2 | cached `flaky`=0, `pest --filter=flaky` (env unset) | flips to `status=7` | PASS |
|
||||
| E3 | after E1, `pest --tia --filtered` clean | `No affected tests found`; zero graph delta | PASS |
|
||||
| E4 | after E2, `pest --tia --filtered` | re-runs, `from 1 previously unsuccessful test` | PASS |
|
||||
| E5 | any partial run | `assertions` and `time` update for the test that ran | PASS |
|
||||
| E6 | any partial run | `message` of untouched tests unchanged | PASS |
|
||||
| E7 | two sequential partial runs on different tests | each updates only its own entry; both persist | PASS |
|
||||
| E8 | `pest --filter="trio one"` | siblings keep exact `status`/`time`/`assertions`/`message` | PASS |
|
||||
| E9 | partial run of a `->skip()`ed test | `status=1` | PASS |
|
||||
| E10 | partial run of a `->todo()` test | `status=1`, `assertions=0`, `message="__TODO__"` | PASS |
|
||||
| E11 | partial run of a risky test | `status=5` | PASS |
|
||||
| E12 | any partial run | `fingerprint` byte-identical | PASS |
|
||||
| E13 | dataset test, `--filter` matching one row | that row updates; other rows survive (the old prune bug) | PASS |
|
||||
| E14 | dataset test, full `--tia` after deleting a row | the deleted row is pruned | PASS |
|
||||
|
||||
### F — Guard rails
|
||||
|
||||
| # | Case | Target outcome | Status |
|
||||
|---|---|---|---|
|
||||
| F1 | test absent from `edges`, `pest --filter="brand new"` | no result recorded | PASS |
|
||||
| F2 | complete `pest --tia` first, then the same filter | result is recorded | PASS |
|
||||
| F3 | `pest --mutate --path=app/Services --covered-only` | statuses/`edges`/`sha`/`tree`/`files` identical; only `time` may change | PASS |
|
||||
| F4 | `pest --mutate --parallel --path=… --covered-only` | same (observed: zero delta) | PASS |
|
||||
| F5 | weakened test so a mutant survives, `pest --mutate` | no status change in the baseline | PASS |
|
||||
| F6 | `pest --mutate` with the graph deleted | no graph created | PASS |
|
||||
| F7 | mutation subprocess env | carries `PEST_MUTATION_TESTING`; `--mutate` stripped from argv | PASS |
|
||||
| F8 | grep baseline after `--mutate` | no mutation-flavoured messages | PASS |
|
||||
|
||||
### G — Parallel
|
||||
|
||||
| # | Case | Target outcome | Status |
|
||||
|---|---|---|---|
|
||||
| G1 | `pest --tia --parallel` (clean) | COMPLETE — graph written | PASS |
|
||||
| G2 | `pest --tia --parallel --filter=adds` | RO | PASS |
|
||||
| G3 | `pest --parallel --bail`, `trio one` broken | RO; siblings survive | PASS |
|
||||
| G4 | `pest --tia --parallel --bail` | RO | PASS |
|
||||
| G5 | `pest --tia --filtered --parallel` after a source edit | narrows to affected; writes | PASS (but see G12 — it strips edges) |
|
||||
| G6 | `pest --tia --parallel` | worker results reach the parent baseline | PASS |
|
||||
| G7 | `->only()` + `pest --tia --parallel` | RO | PASS |
|
||||
| G8 | `pest --tia --parallel --shard=1/2` | RO | PASS |
|
||||
| G9 | broken test in one worker, `pest --parallel --bail` | whole run is RO, not just that worker's slice | PASS |
|
||||
| G10 | `pest --parallel --retry` | `InvalidOption`; graph untouched | PASS |
|
||||
| G11 | `pest --tia --parallel --fresh` | graph rebuilt | PASS |
|
||||
| G12 | `edges` after `pest --tia` vs `pest --tia --parallel --fresh` | **equivalent edge sets** (`files=25` both) | **FIX (item 1)** |
|
||||
|
||||
### H — Baseline key / branch resolution
|
||||
|
||||
| # | Case | Target outcome | Status |
|
||||
|---|---|---|---|
|
||||
| H1 | `pest --tia` on `master` | only a `master` key | PASS |
|
||||
| H2 | plain `pest` on `master` | writes `master`, not `main` | PASS |
|
||||
| H3 | `pest --bail` truncated on `master` | no `main` key minted | PASS |
|
||||
| H4 | `pest --filter=adds` on `master` | no `main` key minted | PASS |
|
||||
| H5 | `git branch -m main`, full `pest --tia` | single `main` key | PASS |
|
||||
| H6 | `git checkout -b feature/x`, `pest --tia` | `feature/x` key appears; reads fall back to the existing baseline | PASS |
|
||||
| H7 | detached HEAD, `pest --tia` | falls back to the `main` key for both reads and writes; no `HEAD` key | PASS |
|
||||
| H8 | after every C and D case | no baseline key other than the real branch | PASS |
|
||||
| H9 | non-git dir, `pest --tia` | `MissingDependency` — `The feature "Tia mode" requires "git".` | PASS |
|
||||
| H10 | non-git dir, plain `pest` | runs normally; no baseline dir created | PASS |
|
||||
|
||||
### I — Filtered mode
|
||||
|
||||
| # | Case | Target outcome | Status |
|
||||
|---|---|---|---|
|
||||
| I1 | clean + green, `pest --tia --filtered` | `No affected tests found`; zero graph delta | PASS |
|
||||
| I2 | edit `Calculator.php` | only `CalculatorTest` runs | PASS |
|
||||
| I3 | cached failure, clean tree | re-runs it, `from 1 previously unsuccessful test` | PASS |
|
||||
| I4 | I2 with sentinels | `tree` updated; exactly one entry rewritten; all others retained | PASS |
|
||||
| I5 | `pest --tia --filtered --parallel` | same narrowing as I2 **and `edges` unchanged** | **FIX (item 1)** |
|
||||
| I6 | `PEST_TIA_FILTERED=1 pest --tia` | identical delta to I4 | PASS |
|
||||
| I7 | `pest --tia --filtered tests/Unit` | explicit path wins; filtered off; RO; notice | PASS |
|
||||
| I8 | `pest --tia --filtered --coverage-text` | filtered mode disabled by an active coverage report | **FIX (item 3)** |
|
||||
| I9 | cached failure whose test file was deleted | WARN `could not be located on disk`; falls back to the full suite with replay | **FIX (item 2)** |
|
||||
| I10 | `pest --tia --filtered` with no baseline yet | records a baseline instead of filtering | PASS |
|
||||
| I11 | edit a Blade view a Feature test renders | the Feature test is selected | PASS |
|
||||
| I12 | edit `composer.lock` | fingerprint drift → full rebuild (and the drift reason should be printed — item 5) | PASS |
|
||||
|
||||
### J — Interactions & regressions
|
||||
|
||||
| # | Case | Target outcome | Status |
|
||||
|---|---|---|---|
|
||||
| J1 | `pest --tia --fresh --filter=adds` | graph **not** purged; RO | PASS |
|
||||
| J2 | `pest --tia --refetch --filter=adds` | refetch not performed; RO | PASS |
|
||||
| J3 | `pest --no-tia --filter=adds` | RO — narrowing wins over `--no-tia`; no notice | PASS |
|
||||
| J4 | `pest --tia --no-tia` | TIA disabled; COMPLETE | PASS |
|
||||
| J5 | `<groups><exclude><group>integration</group></exclude></groups>`, `pest --tia` | COMPLETE, no notice — an always-in-force filter applies to baseline runs too | PASS |
|
||||
| J6 | `pest --tia --compact` | COMPLETE | PASS |
|
||||
| J7 | `pest --tia -v` | COMPLETE | PASS |
|
||||
| J8 | `pest --tia --profile` | COMPLETE | PASS |
|
||||
| J9 | `pest --tia --order-by=random` | COMPLETE | PASS |
|
||||
| J10 | `pest --tia --random-order-seed=1234` | COMPLETE | PASS |
|
||||
| J11 | `pest --tia --repeat=2` | COMPLETE — repetition is not narrowing | **SKIP** — `--repeat` is not a Pest option |
|
||||
| J12 | `pest --filter=adds` twice | only the executed test's `time` differs; rest byte-identical | PASS |
|
||||
| J13 | `pest --tia --min=50` | COMPLETE | PASS |
|
||||
| J14 | SIGINT mid-suite | RO — an interrupted run is truncated | PASS (signal must go to the re-exec'd child — item 5) |
|
||||
| J15 | delete the graph, `pest --tia --filtered` | records a baseline rather than erroring | PASS |
|
||||
| J16 | corrupt `graph.json`, `pest --tia` | recovers by rebuilding; no crash | PASS |
|
||||
|
||||
---
|
||||
|
||||
## Re-run procedure
|
||||
|
||||
```bash
|
||||
cd /Users/nunomaduro/Work/projects/playground/laravel
|
||||
GRAPH="$(PAO_DISABLE=1 ./vendor/bin/pest --baseline)/graph.json"
|
||||
|
||||
# per case
|
||||
git checkout -q . && git clean -qfd tests app
|
||||
rm -rf "$(PAO_DISABLE=1 ./vendor/bin/pest --baseline)"
|
||||
PAO_DISABLE=1 ./vendor/bin/pest --tia >/dev/null 2>&1
|
||||
# sentinel-patch every result to time=9.999 assertions=42, snapshot, run the case, diff
|
||||
```
|
||||
|
||||
Every pest invocation must be prefixed with `PAO_DISABLE=1` (the app has `laravel/pao`, which emits
|
||||
JSON when it detects an agent). The shell is zsh — build commands with arrays or `eval`; unquoted
|
||||
`$args` does not word-split.
|
||||
@@ -1,312 +0,0 @@
|
||||
# TIA deep audit — phase five
|
||||
|
||||
## Your task
|
||||
|
||||
Phases one through four fixed the defects that were *reported*. This phase is the opposite shape: go
|
||||
looking. Read TIA's read/write path adversarially, find what is wrong or fragile, and leave behind a
|
||||
scenario suite that covers the edges nobody has exercised yet.
|
||||
|
||||
Two deliverables, both required:
|
||||
|
||||
1. **A ranked findings list.** Every finding needs a *reproduction*, not a reading of the code. A
|
||||
finding you cannot reproduce is a hypothesis — say so and rank it separately.
|
||||
2. **New scenario tests** in `tests/Features/Tia/*`, covering the edges you probed. Rows that pass go
|
||||
in the repo (they are the regression net). Rows that fail stay in your scratchpad until the fix
|
||||
lands — **never commit a red test.**
|
||||
|
||||
Work in passes, and **report between passes** rather than at the very end:
|
||||
|
||||
- **Pass A** — reproduce the leads in Part 3 below. Report which are real.
|
||||
- **Pass B** — your own hunt: the invariants in Part 2, attacked with inputs nobody tried.
|
||||
- **Pass C** — fixes, smallest first, each with the row that pins it.
|
||||
|
||||
Fix what is clearly a defect with an obvious correct answer. **Stop and ask** when the fix is a
|
||||
behaviour *choice* (what should TIA do when two runs race for one graph?) — those are Nuno's calls,
|
||||
and a reproduction with a crisp yes/no question is worth more than a guessed fix.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — The harness
|
||||
|
||||
### 1.1 What TIA is
|
||||
|
||||
Test Impact Analysis. `pest --tia` records a dependency graph (test file → source files it touched)
|
||||
plus a per-branch baseline of results, then on later runs replays the tests whose dependencies did not
|
||||
change. State lives in a per-project dir under `~/.pest/tia/`; `graph.json` is the whole thing.
|
||||
|
||||
Source of truth: `src/Plugins/Tia.php` (the plugin, ~1900 lines) and `src/Plugins/Tia/Graph.php` (the
|
||||
graph model + read/write API). Supporting: `src/Plugins/Tia/ChangedFiles.php` (git), `Fingerprint.php`
|
||||
(environment/structure hashing), `State.php` (the state dir).
|
||||
|
||||
Read `PLAN.md`, `PLAN_PHASE_TWO.md`, `PLAN_PHASE_THREE.md`, `PLAN_PHASE_FOUR.md` first — in that
|
||||
order. They carry the history, the tier contract, and the decisions already made. Struck-through rows
|
||||
are fixed; do not re-report them.
|
||||
|
||||
### 1.2 The scenario harness
|
||||
|
||||
`tests/Features/Tia/*` scaffold a throwaway git project into a temp dir, run a **real `pest`
|
||||
subprocess** against it, and diff the graph it wrote. Everything lives in `tests/Fixtures/Tia/`:
|
||||
|
||||
| Class | What it gives you |
|
||||
|---|---|
|
||||
| `Project` | `make(branch, overlay:)`, `withoutGit()`, `seed(branch, sentinel:, failing:)`, `seedFor($root, …)`, `pest(...$args)`, `pestWithEnvironment($dir, $env, ...$args)`, `pestIn($dir, ...)`, `write($rel, $contents)`, `path($rel)`, `graph()`, `branchKeys()`, `graphDir()`, `graphExists()`, `snapshot()`, `delta()`, `mutateGraph(fn)`, `addBaseline($branch)`, `worktree($branch)`, `destroy()`, `destroyAll()`, `SEQUENTIAL_AND_PARALLEL` |
|
||||
| `GitRepo` (`$project->git()`) | `switchTo($b, new:)`, `rename($from, $to)`, `detach()`, `config($k, $v)`, `addOrigin()`, `removeOrigin()`, `setOriginHead($b)`, `unsetOriginHead()`, `worktree()`, `sha()`, `branchNames()` |
|
||||
| `GraphDelta` (`$project->delta()`) | `writtenCount()`, `added()`, `removed()`, `branchKeys()`, `baselineUntouched($b)`, `shaMoved()`, `treeMoved()`, `edgesMoved()`, `filesMoved()`, `fingerprintMoved()`, `structureMoved()`, `isResultsOnly()`, `isHardSuppressed()`, `summary()` |
|
||||
| `PestResult` (returned by `pest()`) | `replayed()`, `uncached()`, `affected()`, `tally()`, `output`, `exitCode`, `describe()` |
|
||||
|
||||
The fixture app is 3 test files / **6 tests** (`Project::TOTAL_TESTS`); `Project::EDGES` and
|
||||
`Project::TESTS` describe the graph `seed()` writes. `Project::testId($file, $description)` builds a
|
||||
result key. Overlays in `tests/Fixtures/Tia/overlays/<name>/` supply a different `tests/Pest.php` —
|
||||
that is how you configure `pest()->tia()->…` for a scenario.
|
||||
|
||||
**If the fixture cannot express your case, extend the fixture.** A new `GitRepo` verb or a new overlay
|
||||
is a legitimate part of the deliverable — several findings below need one. Do not water down a
|
||||
scenario to fit the current helpers.
|
||||
|
||||
### 1.3 The sentinel discriminator — read before writing any assertion
|
||||
|
||||
`seed()` writes a graph **and then rewrites every cached result to `time=9.999`, `assertions=42`**
|
||||
(only where assertions are non-zero — risky/skipped/incomplete statuses are *derived* from "performed
|
||||
no assertions", so falsifying those would rewrite the status on replay), then snapshots. Therefore:
|
||||
|
||||
- `$delta->writtenCount()` is the only reliable way to tell **"replayed"** from **"executed and wrote
|
||||
back the same values"**. `0` means nothing was written.
|
||||
- `isResultsOnly()` = no structure moved, no `sha`/`tree` movement, nothing added or removed.
|
||||
- `isHardSuppressed()` = the graph is byte-identical.
|
||||
- A replayed suite reports inflated assertion counts (6 tests × 42). That is the sentinel, not a bug.
|
||||
|
||||
The three write tiers, unchanged since phase two:
|
||||
|
||||
- **COMPLETE** — may change everything.
|
||||
- **RESULTS-ONLY** — may change only `baselines[<branch>].results` for tests that ran; never removes
|
||||
an entry, never adds a result for a test file absent from `edges`, never touches
|
||||
`sha`/`tree`/`edges`/`files`/`fingerprint`.
|
||||
- **HARD-SUPPRESSED** — may change nothing.
|
||||
|
||||
### 1.4 Running them
|
||||
|
||||
They are in the `integration` group (`tests/Pest.php`), so `composer test:unit` skips them. **A
|
||||
directory argument finds nothing** — pass files, space-separated, as separate argv entries:
|
||||
|
||||
```bash
|
||||
PAO_DISABLE=1 php84 bin/pest tests/Features/Tia/DefaultBranchReplay.php tests/Features/Tia/DefaultBranchResolution.php \
|
||||
tests/Features/Tia/DefaultBranchWriteTier.php tests/Features/Tia/PartialRunWriteTier.php \
|
||||
tests/Features/Tia/CompleteRunWriteTier.php tests/Features/Tia/FilteredMode.php
|
||||
|
||||
PAO_DISABLE=1 php bin/pest <same list>
|
||||
```
|
||||
|
||||
**Both interpreters must be green.** `php84` is 8.4.x with **no pcov** — that is what CI has
|
||||
(`.github/workflows/tests.yml` sets `coverage: none`). `php` is 8.5.x with pcov — that is the dev
|
||||
machine. Any assertion that depends on a coverage driver passes locally and fails in CI. Concretely:
|
||||
|
||||
- A **cold recording run writes no graph at all** without pcov/xdebug (it prints `Running in TIA mode,
|
||||
however TIA is skipped as it needs ext-pcov or Xdebug`). **Seed a graph, never record one**, unless
|
||||
the driver *is* the point of the row.
|
||||
- A **PHP source file edit** driverless triggers `Detected PHP source changes but no coverage driver
|
||||
is available` → full suite, `affected=0`. Edit *test* files, not `app/` files.
|
||||
|
||||
`PAO_DISABLE=1` is mandatory on every pest invocation and every probe: `laravel/pao` emits JSON under
|
||||
agents and corrupts the captured output.
|
||||
|
||||
Baseline before you touch anything: **72 passed** on both interpreters, at `HEAD` plus the phase-four
|
||||
working-tree changes. If that number does not reproduce, stop and say so.
|
||||
|
||||
### 1.5 Measure before you assert
|
||||
|
||||
Never guess an expectation from reading the code. Probe first, in your scratchpad:
|
||||
|
||||
```php
|
||||
<?php
|
||||
require '/Users/nunomaduro/Work/projects/pestphp/pest/vendor/autoload.php';
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
$p = Project::make('master');
|
||||
$p->seed('master');
|
||||
$p->git()->switchTo('feature-x', new: true);
|
||||
$r = $p->pest('--tia');
|
||||
printf("tally=[%s] keys=[%s] %s\n", $r->tally(), implode(',', $p->branchKeys()), $p->delta()->summary());
|
||||
$p->destroy();
|
||||
```
|
||||
|
||||
`PAO_DISABLE=1 php probe.php`. One project per scenario; always `destroy()`; `Project::destroyAll()`
|
||||
at the end. Run every probe on **both** interpreters before believing it.
|
||||
|
||||
### 1.6 Ground rules
|
||||
|
||||
- **Do not** run `composer test`. It takes minutes and you do not need it.
|
||||
- **Do not** run `composer update:snapshots`. `tests/.snapshots/success.txt` and the tally in
|
||||
`tests/Visual/Parallel.php` encode the whole suite's result, so every row you add breaks them.
|
||||
That is expected — **report that they need regenerating and let Nuno run it.**
|
||||
- **Do not commit.** Leave the tree dirty.
|
||||
- **Do not touch the playground's `vendor/`.** If a finding truly needs the playground (25 real tests,
|
||||
real timings), say so and ask — syncing it is a manual step Nuno owns.
|
||||
- Run `vendor/bin/phpstan analyse <the files you touched> --memory-limit=-1 --no-progress` and
|
||||
`vendor/bin/pint <the files you touched>` before reporting. **Scope both to files you changed** —
|
||||
Nuno edits `src/` live, and a repo-wide fixer will revert his work.
|
||||
- The fixture projects **hardlink `src/`**. If you edit `src/` while a scenario run is in flight,
|
||||
those rows go red for no reason. Finish the edit, then run.
|
||||
- Keep scratch probes out of the repo.
|
||||
- **Never weaken an existing assertion to make something pass.** If an existing row contradicts your
|
||||
fix, that is a finding: report the contradiction and ask.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — The invariants to attack
|
||||
|
||||
These are the properties TIA is supposed to have. Each one is a place to hunt: construct the input
|
||||
that breaks it.
|
||||
|
||||
1. **Parity.** `pest <args>` and `pest --parallel --processes=N <args>` must leave the **same graph**
|
||||
and reach the same tally. This is a hard rule from Nuno — sequential and parallel must *always*
|
||||
agree. `Project::SEQUENTIAL_AND_PARALLEL` is the dataset that encodes it; consider making every
|
||||
new write-path row use it. Vary `--processes` (1, 2, 8 — more processes than test files).
|
||||
2. **The tiers hold.** Every command lands in exactly one of COMPLETE / RESULTS-ONLY /
|
||||
HARD-SUPPRESSED, and stays inside it. Combinations are where this frays: `--fresh --parallel
|
||||
--filter`, `--bail --shard`, `--filtered` plus an explicit path, `--tia --no-tia`, `--retry`.
|
||||
3. **Replay is faithful.** A replayed test reports the same status, message, time and assertion count
|
||||
as the recorded run — and replay itself writes nothing. Statuses beyond pass/fail are the soft
|
||||
spot: skipped, incomplete, risky, notice, deprecation, warning, todo, and a test that failed with a
|
||||
multi-line message.
|
||||
4. **Reads never write.** No read path may mint a baseline key, move a `sha`, or create the state dir.
|
||||
A project that has never run TIA must gain nothing from a plain `pest` run.
|
||||
5. **A branch never corrupts another branch's baseline.** Writes land on the branch that ran, and
|
||||
only there. Reads may *layer* the default branch under the current one (phase four, B1) — but that
|
||||
layering must not leak into a write.
|
||||
6. **Nothing is unbounded.** Baseline keys, `files`, `edges`, worker partials, state files: something
|
||||
must eventually reclaim them, or the graph grows forever.
|
||||
7. **A hostile state dir cannot break a run.** Corrupt, truncated, empty, wrong-schema, read-only,
|
||||
absent, or *someone else's* `graph.json` — the suite still runs and exits on the tests' merit.
|
||||
8. **Git shapes are all handled.** Detached HEAD (read-only, per phase four B2), worktrees, no commits
|
||||
yet, no `origin`, no `origin/HEAD`, submodules, a repo whose root is above the pest project (that
|
||||
one panics deliberately — `TiaRequiresRepositoryRoot`), renamed branches, deleted branches.
|
||||
|
||||
---
|
||||
|
||||
## Part 3 — Leads to reproduce first (Pass A)
|
||||
|
||||
These came out of reading the phase-four diff. **Each is a hypothesis, not a finding** — several may
|
||||
turn out to be fine. Reproduce or refute each, in order, and report the measured delta either way.
|
||||
|
||||
### L1 — the per-entry fallback may resurrect a pruned or deleted test · **highest value**
|
||||
|
||||
Phase four made `Graph::baselineFor()` layer the branch's results **over** the default branch's. Two
|
||||
consequences worth probing:
|
||||
|
||||
- `pruneStaleResults()` unsets an entry from `baselines[branch].results`. The very next read layers the
|
||||
**fallback's** entry for that same test id back in. So a delete may not stick from a branch's point
|
||||
of view.
|
||||
- `hasUnlocatedTestsToRerun()` returns true when a *failing* cached result names a file that no longer
|
||||
exists on disk — and that forces a **full suite**. If a feature branch deletes a test file that
|
||||
fails on the default branch, the merged read still carries master's entry pointing at the now-absent
|
||||
file. Suspected symptom: **that branch runs the full suite forever.**
|
||||
|
||||
Probe: seed master with a failing test (`seed('master', failing: [...])`), branch off, delete the test
|
||||
file that holds it, run `--tia`, and compare `replayed`/`uncached`/`affected` against the same shape
|
||||
where the failure is on the branch instead. Then the mirror case with a green deleted test.
|
||||
|
||||
### L2 — environment drift clears one branch's results and the fallback serves them right back
|
||||
|
||||
`reconcileFingerprint()` on environmental drift calls `$graph->clearResults($this->branch)` and warns
|
||||
`results dropped, edges reused`. On a feature branch that clears only the *branch's* results — the
|
||||
layered read then re-serves the default branch's results, which were recorded under the **old**
|
||||
environment. Suspected symptom: the drop is a no-op on any branch that is not the default one.
|
||||
|
||||
Probe: `pestWithEnvironment()` to shift whatever `Fingerprint` reads as environmental (check
|
||||
`Fingerprint::environmentalDrift()` for the exact keys), on the default branch vs a feature branch,
|
||||
and compare what survives.
|
||||
|
||||
### L3 — `sha`/`tree` may be taken from a different commit than the results
|
||||
|
||||
`baselineFor()` takes `sha` from the branch when non-null and otherwise from the fallback; `tree`
|
||||
likewise when the branch's is empty. So a branch can end up computing "what changed since" against the
|
||||
**default branch's** recorded sha while reading its own results — or vice versa. Is there a shape where
|
||||
that under-reports changed files (a test replays that should have run)? That is the dangerous
|
||||
direction: a false replay is a lie about a passing test.
|
||||
|
||||
Probe: seed master, commit a test edit on the branch so the shas genuinely differ, and check whether
|
||||
the edited test is treated as affected.
|
||||
|
||||
### L4 — pruning from merged worker partials
|
||||
|
||||
Phase four made a complete `--parallel` run write and prune from merged worker results. The stated
|
||||
safety net is that `pruneStaleResults()` only prunes files it saw results for, and that a truncated
|
||||
worker sets results-only. Try to defeat it: a worker that reports results for a test file it did not
|
||||
finish. Candidate shapes — a fatal error mid-file (not an assertion failure), `exit()` inside a test,
|
||||
an uncaught error in an `afterEach`, a test that kills its own process, `--stop-on-failure` variants,
|
||||
`--processes` greater than the number of test files.
|
||||
|
||||
### L5 — two runs racing for one `graph.json`
|
||||
|
||||
`State::write()` has no locking. Two pest processes on one project (a watcher plus a manual run, two CI
|
||||
jobs sharing a cache dir, `--parallel` where the parent writes while a straggler worker flushes) can
|
||||
lose an update or interleave. Probe by launching two `pest --tia` subprocesses concurrently against one
|
||||
project and diffing. **This is likely a design decision, not a bug** — if you reproduce a lost update,
|
||||
report it as a question (accept last-writer-wins, or lock?), do not invent a locking scheme.
|
||||
|
||||
### L6 — a detached HEAD still purges on structural drift
|
||||
|
||||
Phase four made a detached HEAD read-only *for writes* (`saveGraph()` refuses). But
|
||||
`reconcileFingerprint()` deletes the whole graph on structural drift (`Tia.php`, the
|
||||
`state->delete(KEY_GRAPH)` in the structural branch) before any write happens. So `--tia` from a
|
||||
detached checkout with a changed `composer.lock` can still wipe the default branch's baseline. Confirm
|
||||
it, then ask: should the detached-HEAD guard cover the purge too?
|
||||
|
||||
### L7 — statuses that may not round-trip
|
||||
|
||||
`PLAN.md` §5 claims warnings and deprecations record as `status=0`, and that codes `6` and `4` look
|
||||
unreachable. `Graph::getResult()` maps 0–8 to `TestStatus`. Verify each status end to end: record it,
|
||||
replay it, and check the replayed run reports the same thing — including the message, the exit code,
|
||||
and whether `shouldRerunStatus()` decides to re-execute it. `failOnRisky` / `failOnSkipped` /
|
||||
`displayDetailsOn*` change that decision, so an overlay that flips those config flags is part of this.
|
||||
A status that replays as a pass would be the most serious class of bug in TIA.
|
||||
|
||||
### L8 — branch-key hygiene
|
||||
|
||||
Nothing appears to reclaim baseline keys. Probe: create and delete 5 branches, rename one
|
||||
(`GitRepo::rename()`), and check what `branchKeys()` holds afterwards. Also try names that stress the
|
||||
JSON keying: `feature/x/y` (already covered), a name differing from another only in case (macOS is
|
||||
case-insensitive — does the key match the ref?), a name with a space or a unicode character, a branch
|
||||
literally called `HEAD`, and a very long name. Then: is unbounded growth acceptable, or does this need
|
||||
a cap / GC? Ask rather than build.
|
||||
|
||||
### L9 — `soleRecordedBranch()` as a fallback source
|
||||
|
||||
When config, CI env and git all fail to name a default branch, resolution falls back to "the only
|
||||
branch in the graph". If that sole key was minted by a *narrowed* run on a feature branch (which phase
|
||||
four's B1 made a live possibility), the fallback now names a feature branch, and every other branch
|
||||
layers **its** results underneath. Probe: `withoutGit()` or `removeOrigin()` + no config, with a graph
|
||||
whose only key is `feature-x`.
|
||||
|
||||
### L10 — the state dir as an adversary
|
||||
|
||||
Beyond corrupt JSON (fixed in phase four by deleting it): a valid-JSON graph with `schema: 2`; a graph
|
||||
whose `files` and `edges` disagree; `results` entries with a `file` pointing outside the project root
|
||||
or at an absolute path from another machine; a `graph.json` that is a directory; a state dir with no
|
||||
write permission; `$HOME` unset. Each should degrade to "run the tests", never crash and never write
|
||||
garbage.
|
||||
|
||||
---
|
||||
|
||||
## Part 4 — Reporting
|
||||
|
||||
**Between passes**, not just at the end. Per finding:
|
||||
|
||||
- **Reproduced (yes / no / hypothesis only)**, with the exact command and the measured
|
||||
`tally` + `delta()->summary()` on **both** interpreters.
|
||||
- **Severity**, and say why in one line. The scale that matters here: *a test wrongly replayed as
|
||||
passing* (worst) > *cache silently useless, full suite forever* > *graph grows / stale data* >
|
||||
*cosmetic*.
|
||||
- **Fixed / deferred / needs-a-decision**, and the row that pins it.
|
||||
- For anything needing a decision: **one yes/no question**, no essay.
|
||||
|
||||
Close with:
|
||||
|
||||
1. The count of `tests/Features/Tia/*` green on **both** `php84` (no pcov) and `php85` (pcov), against
|
||||
the 72 baseline.
|
||||
2. Every row you added, and what invariant from Part 2 it defends.
|
||||
3. Which findings are still open, as yes/no questions.
|
||||
4. That `tests/.snapshots/success.txt` and the `tests/Visual/Parallel.php` tally need regenerating —
|
||||
**do not regenerate them.**
|
||||
5. **What you looked at and found solid.** A list of attacks that did not break anything is a real
|
||||
result: it tells the next phase where not to spend its time.
|
||||
@@ -1,351 +0,0 @@
|
||||
# TIA defect sweep — phase four
|
||||
|
||||
## Your task
|
||||
|
||||
Five defects in TIA's read/write path, found while building the repo's TIA scenario suite. One is
|
||||
confirmed and load-bearing (**B1**); four need a decision before a fix (**B2**–**B5**).
|
||||
|
||||
Work **one bug at a time, in order**, and for each:
|
||||
|
||||
1. **Reproduce it as a repo test first.** The reproduction is the deliverable even when the fix is
|
||||
deferred — a red test that pins the exact symptom is worth more than a prose report. Do not commit
|
||||
a red test to the suite; keep it in a scratch file until the fix lands (see Part 1.4).
|
||||
2. Confirm the measured numbers in this file still hold. They were taken at commit `4d3d0105` plus
|
||||
the two uncommitted changes described in Part 2. If a number has moved, say so and stop.
|
||||
3. Fix, then re-run **the whole `tests/Features/Tia/*` set on two interpreters** (Part 1.3).
|
||||
4. **HARD STOP after B1.** Report the diff and wait — B1's fix changes `Graph`'s read semantics for
|
||||
every caller, and Nuno wants to see it before B2–B5 pile on top.
|
||||
|
||||
Per `CLAUDE.md`: do not run `composer test`, and do not regenerate snapshots unless told. Do not
|
||||
commit. Do not touch the playground's `vendor/`.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — The harness
|
||||
|
||||
### 1.1 What exists
|
||||
|
||||
`tests/Features/Tia/*` scaffold a throwaway git project into a temp dir, run a **real `pest`
|
||||
subprocess** against it, and diff the TIA graph it wrote. Everything lives in
|
||||
`tests/Fixtures/Tia/`:
|
||||
|
||||
| Class | What it gives you |
|
||||
|---|---|
|
||||
| `Project` | `make(branch, overlay:)`, `withoutGit()`, `seed(branch, sentinel:, failing:)`, `pest(...$args)`, `pestWithEnvironment($dir, $env, ...$args)`, `pestIn($dir, ...)`, `write($rel, $contents)`, `graph()`, `branchKeys()`, `graphDir()`, `graphExists()`, `snapshot()`, `delta()`, `mutateGraph(fn)`, `addBaseline($branch)`, `worktree($branch)`, `destroyAll()` |
|
||||
| `GitRepo` (`$project->git()`) | `switchTo($b, new:)`, `rename($from, $to)`, `detach()`, `config($k, $v)`, `addOrigin()`, `removeOrigin()`, `setOriginHead($b)`, `unsetOriginHead()`, `worktree()`, `sha()`, `branchNames()` |
|
||||
| `GraphDelta` (`$project->delta()`) | `writtenCount()`, `added()`, `removed()`, `branchKeys()`, `baselineUntouched($b)`, `shaMoved()`, `treeMoved()`, `edgesMoved()`, `filesMoved()`, `fingerprintMoved()`, `structureMoved()`, `isResultsOnly()`, `isHardSuppressed()`, `summary()` |
|
||||
| `PestResult` (returned by `pest()`) | `replayed()`, `uncached()`, `affected()`, `tally()`, `output`, `exitCode`, `describe()` |
|
||||
|
||||
The fixture app is 3 test files / **6 tests** (`Project::TOTAL_TESTS`), with `Project::EDGES` and
|
||||
`Project::TESTS` describing the graph `seed()` writes. `Project::testId($file, $description)` builds
|
||||
a result key.
|
||||
|
||||
### 1.2 The sentinel discriminator — read this before writing an assertion
|
||||
|
||||
`seed()` writes a graph **and then rewrites every cached result to `time=9.999`,
|
||||
`assertions=42`** (non-zero assertion counts only — risky/skipped/incomplete statuses are *derived*
|
||||
from "performed no assertions", so falsifying those would rewrite the status on replay), and takes a
|
||||
snapshot. Therefore:
|
||||
|
||||
- `$delta->writtenCount()` is the only reliable way to tell **"replayed"** from **"executed and wrote
|
||||
back the same values"**. `0` means nothing was written.
|
||||
- `isResultsOnly()` = no structure moved, no `sha`/`tree` movement, nothing added or removed.
|
||||
- `isHardSuppressed()` = the graph is byte-identical.
|
||||
|
||||
Tiers, unchanged since phase two: **COMPLETE** may change everything · **RESULTS-ONLY** may change
|
||||
only `baselines[<branch>].results` for tests that ran, never removing an entry, never adding a result
|
||||
for a test file absent from `edges`, never touching `sha`/`tree`/`edges`/`files`/`fingerprint` ·
|
||||
**HARD-SUPPRESSED** may change nothing.
|
||||
|
||||
### 1.3 Running them
|
||||
|
||||
They are in the `integration` group (`tests/Pest.php:22`), so `composer test:unit` skips them. A
|
||||
**directory argument finds nothing** — pass files:
|
||||
|
||||
```bash
|
||||
F="tests/Features/Tia/DefaultBranchReplay.php tests/Features/Tia/DefaultBranchResolution.php \
|
||||
tests/Features/Tia/DefaultBranchWriteTier.php tests/Features/Tia/PartialRunWriteTier.php \
|
||||
tests/Features/Tia/CompleteRunWriteTier.php tests/Features/Tia/FilteredMode.php"
|
||||
|
||||
PAO_DISABLE=1 php84 bin/pest $F # 8.4.23, NO pcov — this is what CI has
|
||||
PAO_DISABLE=1 php bin/pest $F # 8.5.8, pcov — this is what your machine has
|
||||
```
|
||||
|
||||
**Both must be green.** `.github/workflows/tests.yml` sets `coverage: none`, so any assertion that
|
||||
depends on a coverage driver fails in CI while passing locally. Two tests were already caught by
|
||||
this. Concretely: a **cold recording run writes no graph at all** without pcov/xdebug (it prints
|
||||
`Running in TIA mode, however TIA is skipped as it needs ext-pcov or Xdebug`), and a **PHP source
|
||||
file edit** triggers `Detected PHP source changes but no coverage driver is available` → full suite,
|
||||
`affected=0`. Seed a graph instead of recording one, and edit *test* files rather than `app/` files,
|
||||
unless the point of the row is the driver itself.
|
||||
|
||||
### 1.4 Measure before you assert
|
||||
|
||||
Do not guess expectations from reading the code — every number in Part 3 came from a scratch probe.
|
||||
The pattern (put it in your scratchpad, not in the repo):
|
||||
|
||||
```php
|
||||
<?php
|
||||
require '/Users/nunomaduro/Work/projects/pestphp/pest/vendor/autoload.php';
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
$p = Project::make('master');
|
||||
$p->seed('master');
|
||||
$p->git()->switchTo('feature-x', new: true);
|
||||
$r = $p->pest('--tia');
|
||||
printf("tally=[%s] keys=[%s] %s\n", $r->tally(), implode(',', $p->branchKeys()), $p->delta()->summary());
|
||||
$p->destroy();
|
||||
```
|
||||
|
||||
`PAO_DISABLE=1 php probe.php`. One project per scenario; always `destroy()`.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — What the code looks like right now
|
||||
|
||||
Phase three landed default-branch resolution: `ChangedFiles::defaultBranch()`, the
|
||||
`pest()->tia()->defaultBranch()` config surface, `Graph::setFallbackBranch()` + `?string
|
||||
$fallbackBranch = null` on the seven read methods, and `Tia::resolveFallbackBranch()`
|
||||
(`Tia.php:~1776`) resolving **config → CI env (`CiDefaultBranch`) → git (`origin/HEAD`, then
|
||||
`init.defaultBranch` if the branch exists) → `soleRecordedBranch()`**, failing loudly when nothing
|
||||
can name it.
|
||||
|
||||
On top of that, **two uncommitted changes** you will see in `git diff`:
|
||||
|
||||
1. `TIA_RESULTS_ONLY` global — a *partial* parallel run with an existing graph now purges stale
|
||||
worker partials, sets the global, and workers flush their results through the existing
|
||||
`flushWorkerReplay()` / `mergeWorkerReplayPartials()` path; the parent writes them with
|
||||
`complete: false`. Gated on a graph already existing, so a TIA-less project still creates no
|
||||
baseline dir. This gave `--parallel --filter` parity with sequential — **and, per B1, handed it
|
||||
the shadowing bug too.**
|
||||
2. `loadGraph()` emits `WARN The dependency graph could not be read — it will be rebuilt.` once per
|
||||
parent process when `graph.json` exists but will not decode. Previously silent.
|
||||
|
||||
62 scenario tests cover this and pass on both interpreters.
|
||||
|
||||
---
|
||||
|
||||
## Part 3 — The defects
|
||||
|
||||
### B1 — a thin baseline key permanently shadows the default-branch fallback · **confirmed, priority 1**
|
||||
|
||||
**Symptom.** Any *narrowed* run on a new branch (`--filter`, `--group`, a path, `--bail`, `--shard`,
|
||||
and now `--parallel --filter`) writes a baseline key holding only the tests that ran. From then on
|
||||
`--tia` on that branch reads that thin key instead of falling back to the default branch, so
|
||||
everything else is uncached — **one full suite per branch, forever**, which is the exact cost
|
||||
issue [#1823](https://github.com/pestphp/pest/issues/1823) was about, re-entering through a side door.
|
||||
|
||||
**Measured** (fixture: 6 tests, graph seeded on `master`):
|
||||
|
||||
```
|
||||
switch -c feature-x; pest --filter="adds two numbers"
|
||||
→ keys=[master,feature-x] (feature-x holds 1 result)
|
||||
pest --tia
|
||||
→ 6 passed (6 assertions, 5 uncached, 1 replayed) ← want: 6 replayed, 0 uncached
|
||||
|
||||
same via: pest --parallel --processes=2 --filter="adds two numbers" → identical
|
||||
```
|
||||
|
||||
**Where.** `src/Plugins/Tia/Graph.php::baselineFor()` (~line 814):
|
||||
|
||||
```php
|
||||
if (isset($this->baselines[$branch])) return $this->baselines[$branch];
|
||||
if ($branch !== $fallbackBranch && isset($this->baselines[$fallbackBranch])) return $this->baselines[$fallbackBranch];
|
||||
```
|
||||
|
||||
The fallback is all-or-nothing: it fires only when the branch has **no** key at all. The key itself
|
||||
is minted by `Graph::setResult()` → `ensureBaseline($branch)` (~599 / ~829), reached from
|
||||
`Tia::snapshotTestResults()` on partial runs.
|
||||
|
||||
**Reproduction to add** (`tests/Features/Tia/DefaultBranchReplay.php`):
|
||||
|
||||
```php
|
||||
test('a narrowed run on a new branch does not cost the fallback', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
$project->pest('--filter=adds two numbers');
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($result->uncached())->toBe(0, $result->describe());
|
||||
})->skipOnWindows();
|
||||
```
|
||||
|
||||
Add the `--parallel --processes=2 --filter=…` variant as a second row (dataset), since the two write
|
||||
paths are different code.
|
||||
|
||||
**Fix direction.** Make the fallback **per entry** rather than per baseline: in `baselineFor()`,
|
||||
when the branch has its own baseline *and* a distinct fallback baseline exists, return
|
||||
`results` = the branch's results **layered over** the fallback's (branch wins per test id), and take
|
||||
`sha`/`tree` from the fallback when the branch's are `null`/empty. `baselineFor()` is the single
|
||||
funnel for `recordedAtSha()`, `lastRunTree()`, `getResult()`, `getTime()`, `getAssertions()`,
|
||||
`testFilesToRerun()` and `hasUnlocatedTestsToRerun()`, so one change covers every reader. An
|
||||
alternative — never mint a key from a partial run — is smaller but loses the executed result
|
||||
entirely, which regresses the parity just gained.
|
||||
|
||||
**Done when.** Both reproduction rows are green on both interpreters, and none of these move:
|
||||
|
||||
- `the branch that ran gets its own key and the default branch keeps its baseline` — writes stay on
|
||||
the real branch; the merge must be **read-only** and must not leak into `ensureBaseline()`/`setResult()`.
|
||||
- `a declared default branch that does not exist degrades to a full run` — with a fallback that names
|
||||
nothing, a branch's own thin results must still be all you get.
|
||||
- `a detached HEAD replays without minting a branch key`, `writes nothing on a second run on the same
|
||||
branch`, `filtered mode finds nothing to do…` (both rows) — a merged read must not make a clean
|
||||
replay start writing.
|
||||
- The whole `PartialRunWriteTier.php` / `CompleteRunWriteTier.php` set — tier semantics are unchanged
|
||||
by this fix.
|
||||
|
||||
---
|
||||
|
||||
### B2 — a partial run on detached HEAD writes into the default branch's baseline · **needs a decision**
|
||||
|
||||
**Symptom.** With `HEAD` detached, `Tia::resolveBranch()` (`Tia.php:~1756`) sets
|
||||
`$this->branch = $changedFiles->currentBranch() ?? $this->fallbackBranch` — and that branch is used
|
||||
for **writes**. A `--tia` run in this state happens to be harmless (a clean replay writes nothing),
|
||||
but any run that *executes* tests writes their results into the default branch's baseline.
|
||||
|
||||
**Measured.**
|
||||
|
||||
```
|
||||
seed on master; git checkout --detach; pest --filter="adds two numbers"
|
||||
→ keys=[master] w=1 struct:ok ← master's baseline rewritten from a detached checkout
|
||||
seed on master; git checkout --detach; pest --tia
|
||||
→ keys=[master] w=0 ← read-only, as intended
|
||||
```
|
||||
|
||||
**The decision.** `PLAN_PHASE_THREE.md` §2.3 **D3** recommended detached HEAD be *read-only*. If that
|
||||
still stands, suppress writes when `currentBranch()` is `null` (a dedicated flag — note
|
||||
`resultsOnlyWrites` is **not** enough, it still writes results). If Nuno prefers the current
|
||||
behaviour, add a test pinning it and close this out.
|
||||
|
||||
**Reproduction** (`tests/Features/Tia/DefaultBranchWriteTier.php`), written for the read-only answer:
|
||||
|
||||
```php
|
||||
test('a detached HEAD does not write into the default branch baseline', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->detach();
|
||||
$project->pest('--filter=adds two numbers');
|
||||
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($delta->baselineUntouched('master'))->toBeTrue($delta->summary())
|
||||
->and($project->branchKeys())->toBe(['master']);
|
||||
})->skipOnWindows();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### B3 — an unreadable graph is never repaired on a machine with no coverage driver · **needs a decision**
|
||||
|
||||
**Symptom.** A corrupt `graph.json` is now *reported* (Part 2, change 2) but only *rebuilt* when a
|
||||
coverage driver is present, because rebuilding means recording. Driverless, the file stays corrupt
|
||||
run after run and TIA is silently inert until someone deletes it by hand — while the WARN claims
|
||||
`it will be rebuilt`.
|
||||
|
||||
**Measured** (`php84`, no pcov):
|
||||
|
||||
```
|
||||
overwrite graph.json with '{not json'
|
||||
run 1: exit=0, 6 passed, file still '{not json'
|
||||
run 2: exit=0, 6 passed, file still '{not json'
|
||||
headline: "Running in TIA mode, however TIA is skipped as it needs ext-pcov or Xdebug"
|
||||
```
|
||||
|
||||
**Options.** (a) delete the file when it cannot be decoded, so the next drivered run starts clean and
|
||||
the state dir does not carry a permanent landmine; (b) keep the file but reword the WARN when no
|
||||
driver is available. (a) is the honest one and costs one `State::delete()`.
|
||||
|
||||
**Reproduction** (`tests/Features/Tia/FilteredMode.php`, extending the existing corrupt-graph row):
|
||||
|
||||
```php
|
||||
expect($result->output)->toContain('The dependency graph could not be read')
|
||||
->and(file_get_contents($project->graphDir().'/graph.json'))->not->toBe('{not json');
|
||||
```
|
||||
|
||||
Must pass on **both** interpreters — that is the whole point of the row.
|
||||
|
||||
---
|
||||
|
||||
### B4 — a complete `--parallel` run writes nothing and prunes nothing · **needs a decision**
|
||||
|
||||
**Symptom.** With a graph present and TIA not flagged, a sequential run refreshes results and applies
|
||||
the prune; the same run under `--parallel` does neither, because the parent's `ResultCollector` is
|
||||
empty (results live in the workers) and workers only flush when they were told to record, replay, or
|
||||
— since Part 2's change — results-only for a *partial* run. So parallel CI contributes nothing to the
|
||||
cache, and a deleted test's entry survives forever.
|
||||
|
||||
**Measured** (graph seeded on `master`, sentinelled):
|
||||
|
||||
```
|
||||
pest → w=6 +0 -0 struct:ok (sequential baseline)
|
||||
pest --parallel --processes=2 → w=0 +0 -0 struct:ok ← writes nothing
|
||||
delete a test, then:
|
||||
pest --parallel --processes=2 → w=0 +0 -0 struct:ok ← and does not prune (sequential gives -1)
|
||||
pest --tia --parallel --processes=2 → w=0 ← correct: everything replayed
|
||||
```
|
||||
|
||||
**The decision.** Extending the `TIA_RESULTS_ONLY` mechanism to complete parallel runs is
|
||||
mechanically easy, but a *complete* run also prunes, and pruning from merged worker partials is the
|
||||
risky half: a worker that dies, or a shard that never ran, would look like "these tests no longer
|
||||
exist". If it is done, the prune must key off "every worker reported" and fall back to
|
||||
results-only when it cannot prove that. `PLAN.md` §5 lists this as a known gap, not a regression.
|
||||
|
||||
**Reproduction** (`tests/Features/Tia/CompleteRunWriteTier.php`) — mirror the two sequential rows
|
||||
that already exist (`a complete run prunes a deleted test`, `--no-tia refreshes results…`) with
|
||||
`--parallel --processes=2` added, and assert the same deltas.
|
||||
|
||||
---
|
||||
|
||||
### B5 — G4 ("parallel replay clobbers cached `time`") no longer reproduces · **verify, then correct the record**
|
||||
|
||||
**Symptom.** `PLAN_PHASE_THREE.md` §4.6 lists as still-present: *"parallel replay clobbers cached
|
||||
`time` on all non-executed tests — `mergeWorkerReplayPartials()` takes `$result['time']` verbatim,
|
||||
never routing through `resultTime()`"*. The repo fixture disagrees: after a parallel replay the
|
||||
sentinelled `time=9.999` / `assertions=42` survive on every non-executed test.
|
||||
|
||||
**Measured.** `a parallel run merges worker results into the parent baseline` edits one test file,
|
||||
then runs `--tia --parallel --processes=2`: `2 affected, 4 replayed, w=2`. If replayed times were
|
||||
being clobbered, `w` would be `6`.
|
||||
|
||||
**Why the record may be stale.** `flushWorkerReplay()` (`Tia.php:~1286`) already applies
|
||||
`resultTime()` **worker-side** before writing the partial, so the parent's verbatim read is reading
|
||||
values that were already corrected.
|
||||
|
||||
**What to do.** Either find a shape where it still reproduces (the playground has 25 tests and real
|
||||
timings; the fixture has 6 and may be too small), or confirm it is fixed and strike it from §4.6.
|
||||
Add a direct row either way:
|
||||
|
||||
```php
|
||||
test('a parallel replay keeps the recorded time of tests that did not run', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->pest('--tia', '--parallel', '--processes=2');
|
||||
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($delta->writtenCount())->toBe(0, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 4 — Reporting
|
||||
|
||||
Per bug: **reproduced (yes/no)** with the measured delta, **fixed (yes/no/deferred)**, and the test
|
||||
that now pins it. Close with:
|
||||
|
||||
1. Whether all `tests/Features/Tia/*` are green on **both** `php84` (no pcov) and `php85` (pcov).
|
||||
2. Which of B2–B5 still need Nuno's decision, phrased as a yes/no question each.
|
||||
3. Whether `tests/.snapshots/success.txt` and the `tests/Visual/Parallel.php` tally now need
|
||||
regenerating (they will, if you added rows) — **do not run `composer update:snapshots` unless
|
||||
asked.**
|
||||
4. Anything you found that is not in this file.
|
||||
|
||||
Leave the tree uncommitted and the scratch probes out of the repo.
|
||||
@@ -1,386 +0,0 @@
|
||||
# TIA deep audit — phase six
|
||||
|
||||
## Your task
|
||||
|
||||
Phase five went looking rather than fixing what was reported, and found nine defects in the
|
||||
**read/write path and the state dir**. All nine are fixed and pinned by rows in the repo. It stopped
|
||||
there deliberately: roughly half of TIA was never opened. Phase six is that other half.
|
||||
|
||||
Two deliverables, both required:
|
||||
|
||||
1. **A ranked findings list.** Every finding needs a *reproduction*, not a reading of the code. A
|
||||
finding you cannot reproduce is a hypothesis — say so and rank it separately.
|
||||
2. **New scenario tests** in `tests/Features/Tia/*`. Rows that pass go in the repo (they are the
|
||||
regression net). Rows that fail stay in your scratchpad until the fix lands — **never commit a red
|
||||
test.**
|
||||
|
||||
Work in passes, and **report between passes** rather than at the very end:
|
||||
|
||||
- **Pass A** — reproduce the leads in Part 3. Report which are real.
|
||||
- **Pass B** — your own hunt: the invariants in Part 2, attacked with inputs nobody tried.
|
||||
- **Pass C** — fixes, smallest first, each with the row that pins it.
|
||||
|
||||
Fix what is clearly a defect with an obvious correct answer. **Stop and ask** when the fix is a
|
||||
behaviour *choice* — those are Nuno's calls, and a reproduction with a crisp yes/no question is worth
|
||||
more than a guessed fix.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — Where things stand
|
||||
|
||||
### 1.1 What TIA is
|
||||
|
||||
Test Impact Analysis. `pest --tia` records a dependency graph (test file → source files it touched)
|
||||
plus a per-branch baseline of results, then on later runs replays the tests whose dependencies did not
|
||||
change. State lives in a per-project dir under `~/.pest/tia/`; `graph.json` is the whole thing.
|
||||
|
||||
Source of truth: `src/Plugins/Tia.php` (the plugin) and `src/Plugins/Tia/Graph.php` (the graph model +
|
||||
read/write API). Supporting: `ChangedFiles.php` (git), `Fingerprint.php` (environment/structure
|
||||
hashing), `Storage.php` / `FileState.php` (the state dir), `ResultCollector.php` (what a run observed),
|
||||
`Recorder.php` + `CoverageCollector.php` (how edges are recorded).
|
||||
|
||||
Read `PLAN.md`, then `PLAN_PHASE_TWO.md` → `THREE` → `FOUR` → `FIVE` for the history and the tier
|
||||
contract. Struck-through rows are fixed; do not re-report them. **Section 1.2 below supersedes any
|
||||
phase-five row that contradicts it.**
|
||||
|
||||
### 1.2 What phase five settled — do not re-report these
|
||||
|
||||
Nine defects, all fixed, each with a row that fails if it comes back:
|
||||
|
||||
| # | Defect | Fix | Pinned by |
|
||||
|---|---|---|---|
|
||||
| 1 | `pruneStaleResults()` unset an entry; the next read layered the default branch's entry for the same test id back in, so a renamed/removed test stayed "previously unsuccessful" on a feature branch forever | `Graph::baselineFor()` layers per *file* once a branch has had a complete run (new `complete` flag on the baseline); a key minted by a narrowed run keeps the per-test-id merge | `StateReclamation` → *a pruned result does not come back from the fallback*, *the fallback still reaches a branch that has never run a test file* |
|
||||
| 2 | A cached failure whose test file was deleted was never reclaimed, so `--filtered` degraded to a full replay on every later run | `hasUnlocatedTestsToRerun()` widens only for a path it cannot *address*; `Graph::pruneResultsForMissingFiles()` + `pruneMissingTests()` run on every complete write | `StateReclamation` → *a cached failure whose test file was deleted stops widening later runs*, *a complete run reclaims the entry and the edge of a deleted test file* |
|
||||
| 3 | A detached HEAD is read-only for writes, but three paths still *deleted*: structural drift, `--fresh` (`Storage::purge`), and the corrupt-graph discard. The checkout that wiped the baseline could never rebuild it | `Tia::deleteState()` no-ops when detached; `Storage::purge` guarded | `StateReclamation` → the three *a detached HEAD does not purge…* rows |
|
||||
| 4 | A status int outside 0–8 became `TestStatus::unknown()`, which `ReplayType` folded into `Failure` — a green test went red, exit 1 | `Graph::getResult()` returns `null` for an unknown status (re-run, don't replay); `shouldRerunStatus()` treats unknown as re-run | `HostileState` → *a cached status this build cannot interpret is re-run, not replayed* |
|
||||
| 5 | Notice/deprecation/warning (3/4/6) decode fine but `ReplayType` had no case, so they also folded into `Failure` | Explicit `Pass` cases — those statuses only reach replay when the configured `failOn*` / `displayDetailsOn*` policies say they do not matter | `HostileState` → *a cached status with no replay of its own does not fail the run* |
|
||||
| 6 | `Graph::decode()` took `baselines`, `edges` and `files` verbatim; one malformed entry raised a `TypeError` inside a test | `decodeBaselines()` / `decodeResults()` / `decodeEdges()` / `decodeFiles()` validate every field. Numeric-looking keys are cast, not filtered — a branch named `12345` decodes as an `int` | `HostileState` → *a graph whose shape is wrong everywhere is repaired rather than trusted* |
|
||||
| 7 | Baseline keys grew forever — one full copy of the suite per branch ever created | `ChangedFiles::branchNames()` (local + remote refs) + `Graph::pruneMissingBranches()`, on complete writes only, and only when the fallback branch is visible in the refs | `BranchShapes` → *deleting many branches reclaims every one of their baselines* and the four rows around it |
|
||||
| 8 | A test that triggered a deprecation was recorded as `status=0`, because PHPUnit emits `Passed` for it and TIA had no issue subscribers. A fresh `--fail-on-deprecation` run exited 1; the replayed one exited 0 | Six subscribers (`Notice`/`PhpNotice`/`Deprecation`/`PhpDeprecation`/`Warning`/`PhpWarning`) feed `ResultCollector`; most-important-status-wins; a plain `Passed` does not downgrade a triggered issue; `@`-suppressed issues are ignored | `IssueStatuses` → *a triggered issue is recorded as itself, not as a pass*, *a cached deprecation still fails the run that asked to fail on one* |
|
||||
| 9 | A replay wrote back the status it *looked* like from outside, so a cached deprecation replaying as a pass was persisted as `0` — defect 8's fix eroded after one run | `Tia::replayedAsRecorded()` writes back the cached status and message for replayed tests, in the sequential path and in the worker flush | `IssueStatuses` → *replaying a cached issue does not downgrade it to a pass* |
|
||||
| 10 | A run torn down mid-file (an `exit()` inside a test) still flushed what it had, and the parent read that as licence to prune the siblings it never reached. Sequential and parallel disagreed | `ResultCollector::hasUnfinishedTest()` demotes such a run to results-only, in `terminate()` (the shutdown path) and in `addOutput()` | `StateReclamation` → *a run torn down mid-file does not prune the tests it never reached* |
|
||||
|
||||
**One existing assertion was changed.** `DefaultBranchWriteTier > filtered mode falls back to a full
|
||||
replay when a cached failure cannot be located` used `tests/Unit/DeletedTest.php` — a path that
|
||||
resolves but does not exist, which is precisely the shape behind defect 2. It now points at
|
||||
`/build/agent/…`, so it still pins the widening safety net for the case where widening can help. If
|
||||
you disagree with that reading, that is a finding, not a licence to change it back quietly.
|
||||
|
||||
### 1.3 Attacked in phase five and found solid — do not spend time here again
|
||||
|
||||
- **Parity.** Sequential vs `--processes=1/2/8` across replay-with-edit, `--filtered` with a cached
|
||||
failure, a first run on a feature branch, `--bail`, `--stop-on-failure`, `--compact`: identical
|
||||
tally *and* identical graph delta every time.
|
||||
- **Hostile state dir.** Empty / truncated / not-JSON / JSON scalar / JSON list / `null` / `{}` / NUL
|
||||
bytes; `graph.json` as a directory; a read-only state dir; dangling and negative edge ids; a result
|
||||
`file` pointing outside the project; `schema: 2`. All degrade to "run the tests", exit 0.
|
||||
- **Status int mapping.** `ResultCollector` (`asInt()`), `Graph::getResult()` and `TestStatus::from()`
|
||||
agree exactly on 0–8. No off-by-one.
|
||||
- **`FileState::write`** is tmp + rename, so concurrent runs cannot tear a file. Racing runs are
|
||||
last-writer-wins **by design** — treat as a decision, not a bug, unless you can show data loss
|
||||
worse than that.
|
||||
- **Branch names.** Slashes, dots, unicode, digits-only, 180 characters, case-only differences,
|
||||
remote-only branches, worktree branches: all keyed and reclaimed correctly.
|
||||
- **`sha`/`tree` layering.** No shape found that under-reports changed files. The fallback `tree` only
|
||||
drops a file whose current content hashes identically to what the fallback ran with, which is sound.
|
||||
|
||||
### 1.4 Known and deliberately unfixed
|
||||
|
||||
| Item | Why it is still open |
|
||||
|---|---|
|
||||
| Reclamation is skipped when a complete run executes **zero** tests | Stale edges linger until a run that executes at least one test. Writing the graph from a run that produced nothing costs more than it buys. Revisit if you find a real project stuck there. |
|
||||
| SIGINT never reaches the suite (`PcovRestarter` re-exec) | Real, but signal handling outside the TIA read/write path. Needs its own pass. |
|
||||
| Orphaned state dirs when the origin URL changes | `Storage::projectKey()` keys on the origin identity so clones share a graph. Changing the remote silently moves TIA to a fresh dir and nothing reclaims the old one. Unbounded growth in `~/.pest/tia`. |
|
||||
| `environmental` fingerprint holds only `php_minor` | Nuno declined. The `clearResults()` drift path is therefore latent — it becomes live the moment that bucket grows. **Do not re-flag the `PHP_MAJOR_VERSION` line itself.** |
|
||||
| PLAN.md §3 (raw coverage flags leave filtered mode on) | **No longer reproduces** — `coverageReportActive()` consults `COVERAGE_REPORT_FLAGS` now. Struck. |
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — The invariants to attack
|
||||
|
||||
Same list as phase five; 1, 2, 4, 5, 7 and 8 are now well covered, so weight your effort toward 3 and
|
||||
6 and toward the *recording* half of the system, which nothing below the plugin has ever probed.
|
||||
|
||||
1. **Parity.** `pest <args>` and `pest --parallel --processes=N <args>` must leave the same graph and
|
||||
reach the same tally. Hard rule. `Project::SEQUENTIAL_AND_PARALLEL` encodes it; use it on every new
|
||||
write-path row.
|
||||
2. **The tiers hold.** COMPLETE / RESULTS-ONLY / HARD-SUPPRESSED, exactly one each, no leaking.
|
||||
3. **Replay is faithful.** ← *weak spot.* A replayed test reports the same status, message, time and
|
||||
assertion count as the recorded run. Phase five fixed statuses; **edges** are unproven.
|
||||
4. **Reads never write.**
|
||||
5. **A branch never corrupts another branch's baseline.**
|
||||
6. **Nothing is unbounded.** ← *weak spot.* Baseline keys are reclaimed now; `files`, `edges`, worker
|
||||
partials, coverage caches and orphaned state dirs are not.
|
||||
7. **A hostile state dir cannot break a run.**
|
||||
8. **Git shapes are all handled.**
|
||||
|
||||
---
|
||||
|
||||
## Part 3 — Leads to reproduce first (Pass A)
|
||||
|
||||
Ranked by expected value. Everything here is **unexplored**, not merely unfixed — phase five never
|
||||
opened these files.
|
||||
|
||||
### M1 — the recording path with a real coverage driver · **highest value**
|
||||
|
||||
Everything phase five did was driver-independent by design (seed a graph, never record one). Nothing
|
||||
verified that a *recorded* graph is correct. This is the biggest blind spot in the audit.
|
||||
|
||||
- Do the recorded edges match what the test actually touched? Record with pcov, then hand-check
|
||||
`edges` against the source files each fixture test uses.
|
||||
- `PLAN.md` §4: **`pest --tia --coverage` narrows edges** — `Feature/ExampleTest` recorded with 2
|
||||
files instead of 16, dropping self-edges. Same observable shape as the parallel bug G12 that phase
|
||||
four closed, likely a different mechanism (the piggyback collector is scoped by `phpunit.xml
|
||||
<include>`, the pcov-restarted recorder is not). **Confirm whether these are one fix or two.**
|
||||
- `Recorder::activateLinkTracking()` (piggyback) vs `activate()` (pcov restart) must produce the same
|
||||
edge set for the same suite. Compare them directly.
|
||||
- `keepExisting: $this->piggybackCoverage` in `replaceEdges()` — what happens to a test whose edges
|
||||
genuinely shrank while piggybacking?
|
||||
|
||||
Rows for this **must** be `->skipOnPhpVersionsWithoutCoverage()`-style guarded, or seeded, or CI goes
|
||||
red on `php84`. That constraint is why phase five skipped it; solve it deliberately rather than by
|
||||
accident. Adding a coverage-driver guard helper to the fixture is a legitimate deliverable.
|
||||
|
||||
### M2 — `BaselineSync` (621 lines, never opened)
|
||||
|
||||
The remote-baseline fetch is the only path where a graph arrives from **another machine**, which is
|
||||
exactly where the hostile-state work matters most and where none of it has been exercised.
|
||||
|
||||
- A fetched baseline whose `fingerprint` matches but whose `files`/`edges` describe a different tree.
|
||||
- A fetched baseline recorded on a branch this checkout does not have.
|
||||
- `fetchIfAvailable()` under a broken network, a 404, a truncated download, a non-gzip body.
|
||||
- `KEY_FETCH_COOLDOWN` — does it bound retries, and does a corrupt cooldown file break a run?
|
||||
- Interaction with defect 7's branch GC: a fetched baseline carries branch keys this clone has never
|
||||
heard of. **They will be pruned on the next complete write.** Is that right, or must fetched keys be
|
||||
exempt? This is a real question, answer it.
|
||||
|
||||
### M3 — selection paths nobody has probed
|
||||
|
||||
`Graph::affected()` is ~600 lines and phase five only exercised the plain PHP-edge path.
|
||||
|
||||
- **Migrations** → `TableExtractor` → `testTables` intersection. What happens with an unparseable
|
||||
migration, a migration that drops a table, a squashed schema dump?
|
||||
- **Blade** — `bladeAncestorsFor()` walks `@include`/`@extends`/`<x-*>` transitively. Cycles?
|
||||
Depth? A component referenced only dynamically?
|
||||
- **Inertia** — `componentForInertiaPage()`, `jsFileToComponents`, `JsModuleGraph::buildStrict()`.
|
||||
What if `vite` is missing, or the resolver returns garbage?
|
||||
- **`usesSiblingHeuristicForUnknownPhp()`** — a hard-coded list of Laravel directories. A changed file
|
||||
in `app/Providers/` widens to every test whose deps share that directory. Measure how much that
|
||||
over-selects on a real tree.
|
||||
- **Arch tests** — `testSourceDeclaresArchGroup()` greps the source with three regexes. False
|
||||
positives (the string `arch(` in a comment) select the file on *every* PHP source change.
|
||||
|
||||
### M4 — git shapes phase five left alone
|
||||
|
||||
- A repo with **no commits yet** (`currentSha()` returns null / git fails).
|
||||
- **Submodules** — a changed file inside one; `git status --porcelain` reports the submodule path.
|
||||
- A repo whose root is **above** the pest project — `TiaRequiresRepositoryRoot` panics deliberately;
|
||||
confirm it still panics and writes nothing.
|
||||
- **Shallow / single-branch CI checkouts.** Defect 7's guard (`fallbackBranch` must be visible in the
|
||||
refs) was reasoned about, not measured. Build one and check nothing is over-pruned.
|
||||
- A branch **behind** the recorded sha, so `merge-base --is-ancestor` fails and the graph is declared
|
||||
unreachable. Does it recover, or thrash?
|
||||
|
||||
### M5 — a real concurrent race
|
||||
|
||||
Phase five established `FileState::write` is atomic and called it last-writer-wins by design. Nobody
|
||||
launched two runs. Launch them: a watcher plus a manual run, two `--tia` processes on one project,
|
||||
`--parallel` where the parent writes while a straggler worker flushes. Look for **loss worse than
|
||||
last-writer-wins** — a partially-merged baseline, a pruned entry from a run that never saw the file,
|
||||
worker partials from run A consumed by run B (`KEY_WORKER_*` are not namespaced per run).
|
||||
|
||||
That last one is the sharpest: `purgeWorkerPartials()` deletes *all* partials by prefix, so two
|
||||
concurrent parallel runs will eat each other's. Probe it.
|
||||
|
||||
### M6 — the unbounded remainder
|
||||
|
||||
Defect 7 reclaimed baseline keys. These still grow:
|
||||
|
||||
- `files` and `edges` — a deleted *source* file's entry is never removed (`pruneMissingTests()` only
|
||||
covers test files). Every rename leaves an orphan id forever.
|
||||
- Orphaned state dirs under `~/.pest/tia` (see 1.4).
|
||||
- `KEY_COVERAGE_CACHE` / `KEY_COVERAGE_MARKER` — who deletes them, and when?
|
||||
- Worker partials when a worker dies before `terminate()`.
|
||||
|
||||
Measure the growth on a realistic tree before proposing anything. Then **ask** — a cap or a GC is a
|
||||
behaviour choice.
|
||||
|
||||
---
|
||||
|
||||
## Part 4 — The harness
|
||||
|
||||
### 4.1 The scenario suite
|
||||
|
||||
`tests/Features/Tia/*` scaffold a throwaway git project into a temp dir, run a **real `pest`
|
||||
subprocess** against it, and diff the graph it wrote. Everything lives in `tests/Fixtures/Tia/`:
|
||||
|
||||
| Class | What it gives you |
|
||||
|---|---|
|
||||
| `Project` | `make(branch, overlay:)`, `withoutGit()`, `seed(branch, sentinel:, failing:)`, `seedFor($root, …)`, `pest(...$args)`, `pestWithEnvironment($dir, $env, ...$args)`, `pestIn($dir, ...)`, `write($rel, $contents)`, `path($rel)`, `graph()`, `branchKeys()`, `graphDir()`, `graphExists()`, `snapshot()`, `delta()`, `mutateGraph(fn)`, `addBaseline($branch)`, `worktree($branch)`, `destroy()`, `destroyAll()`, `SEQUENTIAL_AND_PARALLEL` |
|
||||
| `GitRepo` (`$project->git()`) | `switchTo($b, new:)`, `rename($from, $to)`, `detach()`, `commit($msg)`, `config($k, $v)`, `addOrigin()`, `removeOrigin()`, `setOriginHead($b)`, `unsetOriginHead()`, `worktree()`, `sha()`, `branchNames()`, `run([...])` for anything else |
|
||||
| `GraphDelta` (`$project->delta()`) | `writtenCount()`, `added()`, `removed()`, `branchKeys()`, `baselineUntouched($b)`, `shaMoved()`, `treeMoved()`, `edgesMoved()`, `filesMoved()`, `fingerprintMoved()`, `structureMoved()`, `isResultsOnly()`, `isHardSuppressed()`, `summary()` |
|
||||
| `PestResult` (returned by `pest()`) | `replayed()`, `uncached()`, `affected()`, `tally()`, `output`, `exitCode`, `describe()` |
|
||||
|
||||
The fixture app is 3 test files / **6 tests** (`Project::TOTAL_TESTS`); `Project::EDGES` and
|
||||
`Project::TESTS` describe the graph `seed()` writes. `Project::testId($file, $description)` builds a
|
||||
result key. Overlays in `tests/Fixtures/Tia/overlays/<name>/` supply a different `tests/Pest.php` —
|
||||
that is how you configure `pest()->tia()->…` for a scenario.
|
||||
|
||||
**If the fixture cannot express your case, extend the fixture.** A new `GitRepo` verb, a new overlay,
|
||||
or a coverage-driver guard is a legitimate part of the deliverable. Do not water down a scenario to
|
||||
fit the current helpers. Phase five added `GitRepo::commit()` usage, numeric-key handling in
|
||||
`Project::branchKeys()` and `GraphDelta`, and used `$project->write()` to author test files inline —
|
||||
follow that pattern.
|
||||
|
||||
### 4.2 The sentinel discriminator — read before writing any assertion
|
||||
|
||||
`seed()` writes a graph **and then rewrites every cached result to `time=9.999`, `assertions=42`**
|
||||
(only where assertions are non-zero — risky/skipped/incomplete statuses are *derived* from "performed
|
||||
no assertions", so falsifying those would rewrite the status on replay), then snapshots. Therefore:
|
||||
|
||||
- `$delta->writtenCount()` is the only reliable way to tell **"replayed"** from **"executed and wrote
|
||||
back the same values"**. `0` means nothing was written.
|
||||
- `isResultsOnly()` = no structure moved, no `sha`/`tree` movement, nothing added or removed.
|
||||
- `isHardSuppressed()` = the graph is byte-identical.
|
||||
- A replayed suite reports inflated assertion counts (6 tests × 42). That is the sentinel, not a bug.
|
||||
|
||||
The three write tiers, unchanged since phase two:
|
||||
|
||||
- **COMPLETE** — may change everything.
|
||||
- **RESULTS-ONLY** — may change only `baselines[<branch>].results` for tests that ran; never removes
|
||||
an entry, never adds a result for a test file absent from `edges`, never touches
|
||||
`sha`/`tree`/`edges`/`files`/`fingerprint`.
|
||||
- **HARD-SUPPRESSED** — may change nothing.
|
||||
|
||||
One addition from phase five: a complete run on a **non-default** branch also sets
|
||||
`baselines[<branch>].complete = true`. It is deliberately *not* set on the default branch, so a clean
|
||||
green run there stays byte-identical.
|
||||
|
||||
### 4.3 Running them
|
||||
|
||||
They are in the `integration` group (`tests/Pest.php`), so `composer test:unit` skips them. **A
|
||||
directory argument finds nothing** — pass files, space-separated, as separate argv entries:
|
||||
|
||||
```bash
|
||||
PAO_DISABLE=1 php84 bin/pest tests/Features/Tia/BranchShapes.php tests/Features/Tia/CompleteRunWriteTier.php \
|
||||
tests/Features/Tia/DefaultBranchReplay.php tests/Features/Tia/DefaultBranchResolution.php \
|
||||
tests/Features/Tia/DefaultBranchWriteTier.php tests/Features/Tia/FilteredMode.php \
|
||||
tests/Features/Tia/HostileState.php tests/Features/Tia/IssueStatuses.php \
|
||||
tests/Features/Tia/PartialRunWriteTier.php tests/Features/Tia/StateReclamation.php
|
||||
|
||||
PAO_DISABLE=1 php bin/pest <same list>
|
||||
```
|
||||
|
||||
**Both interpreters must be green.** `php84` is 8.4.x with **no pcov** — that is what CI has
|
||||
(`.github/workflows/tests.yml` sets `coverage: none`). `php` is 8.5.x with pcov — that is the dev
|
||||
machine. Any assertion that depends on a coverage driver passes locally and fails in CI. Concretely:
|
||||
|
||||
- A **cold recording run writes no graph at all** without pcov/xdebug (it prints `Running in TIA mode,
|
||||
however TIA is skipped as it needs ext-pcov or Xdebug`). **Seed a graph, never record one**, unless
|
||||
the driver *is* the point of the row — see M1, which has to solve this properly.
|
||||
- A **PHP source file edit** driverless triggers `Detected PHP source changes but no coverage driver
|
||||
is available` → full suite, `affected=0`. Edit *test* files, not `app/` files.
|
||||
- The `terminate()` path differs by driver: with pcov the plugin reaches the complete write through
|
||||
the shutdown handler, without it the run exits earlier. Defect 10 only reproduced on `php`. **Run
|
||||
every probe on both before believing it.**
|
||||
|
||||
`PAO_DISABLE=1` is mandatory on every pest invocation and every probe: `laravel/pao` emits JSON under
|
||||
agents and corrupts the captured output.
|
||||
|
||||
**Baseline: 161 scenario rows green on both interpreters**, at `b49ba062`. Per file:
|
||||
|
||||
| File | Rows |
|
||||
|---|---|
|
||||
| `StateReclamation.php` | 37 |
|
||||
| `HostileState.php` | 25 |
|
||||
| `CompleteRunWriteTier.php` | 17 |
|
||||
| `DefaultBranchResolution.php` | 14 |
|
||||
| `BranchShapes.php` | 14 |
|
||||
| `DefaultBranchReplay.php` | 13 |
|
||||
| `IssueStatuses.php` | 13 |
|
||||
| `PartialRunWriteTier.php` | 12 |
|
||||
| `DefaultBranchWriteTier.php` | 10 |
|
||||
| `FilteredMode.php` | 6 |
|
||||
|
||||
Plus 80 unit/arch rows (`tests/Unit/Plugins/Tia/*`, `tests/Arch.php`). If those numbers do not
|
||||
reproduce, stop and say so.
|
||||
|
||||
### 4.4 Measure before you assert
|
||||
|
||||
Never guess an expectation from reading the code. Probe first, in your scratchpad:
|
||||
|
||||
```php
|
||||
<?php
|
||||
require '/Users/nunomaduro/Work/projects/pestphp/pest/vendor/autoload.php';
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
$p = Project::make('master');
|
||||
$p->seed('master');
|
||||
$p->git()->switchTo('feature-x', new: true);
|
||||
$r = $p->pest('--tia');
|
||||
printf("tally=[%s] keys=[%s] %s\n", $r->tally(), implode(',', $p->branchKeys()), $p->delta()->summary());
|
||||
$p->destroy();
|
||||
```
|
||||
|
||||
`PAO_DISABLE=1 php probe.php`. One project per scenario; always `destroy()`; `Project::destroyAll()`
|
||||
at the end. Run every probe on **both** interpreters before believing it.
|
||||
|
||||
---
|
||||
|
||||
## Part 5 — Ground rules
|
||||
|
||||
- **Do not** run `composer test`. It takes minutes and you do not need it.
|
||||
- **`tests/.snapshots/success.txt` and the tally in `tests/Visual/Parallel.php` are stale right now** —
|
||||
phase five added 89 rows and did not regenerate them. **Report that they need regenerating and let
|
||||
Nuno run `composer update:snapshots`.** Do not run it yourself.
|
||||
- **Do not commit.** Leave the tree dirty.
|
||||
- **Do not touch the playground's `vendor/`.** If a finding truly needs the playground (25 real tests,
|
||||
real timings), say so and ask — syncing it is a manual step Nuno owns.
|
||||
- Run `vendor/bin/phpstan analyse <the files you touched> --memory-limit=-1 --no-progress` and
|
||||
`vendor/bin/pint <the files you touched>` before reporting. **Scope both to files you changed** —
|
||||
Nuno edits `src/` live, and a repo-wide fixer will revert his work.
|
||||
- The fixture projects **hardlink `src/`**. If you edit `src/` while a scenario run is in flight,
|
||||
those rows go red for no reason. Finish the edit, then run.
|
||||
- Keep scratch probes out of the repo.
|
||||
- **Never weaken an existing assertion to make something pass.** If an existing row contradicts your
|
||||
fix, that is a finding: report the contradiction and ask. Phase five hit this once (see 1.2) and
|
||||
rewrote the row to pin the *narrower* contract rather than deleting it — that is the bar.
|
||||
|
||||
---
|
||||
|
||||
## Part 6 — Reporting
|
||||
|
||||
**Between passes**, not just at the end. Per finding:
|
||||
|
||||
- **Reproduced (yes / no / hypothesis only)**, with the exact command and the measured
|
||||
`tally` + `delta()->summary()` on **both** interpreters.
|
||||
- **Severity**, and say why in one line. The scale that matters here: *a test wrongly replayed as
|
||||
passing* (worst) > *cache silently useless, full suite forever* > *graph grows / stale data* >
|
||||
*cosmetic*.
|
||||
- **Fixed / deferred / needs-a-decision**, and the row that pins it.
|
||||
- For anything needing a decision: **one yes/no question**, no essay.
|
||||
|
||||
Close with:
|
||||
|
||||
1. The count of `tests/Features/Tia/*` green on **both** `php84` (no pcov) and `php` (pcov), against
|
||||
the 161 baseline.
|
||||
2. Every row you added, and what invariant from Part 2 it defends.
|
||||
3. Which findings are still open, as yes/no questions.
|
||||
4. That the snapshots need regenerating — **do not regenerate them.**
|
||||
5. **What you looked at and found solid.** A list of attacks that did not break anything is a real
|
||||
result: it tells the next phase where not to spend its time.
|
||||
|
||||
---
|
||||
|
||||
## Part 7 — Open questions carried into this phase
|
||||
|
||||
Answer these before or during Pass C; they change what the fixes should be.
|
||||
|
||||
1. A fetched remote baseline carries branch keys this clone has never heard of, and the branch GC will
|
||||
prune them on the next complete write. **Should fetched keys be exempt?**
|
||||
2. Two concurrent parallel runs share the `worker-edges-*` / `worker-results-*` prefixes and
|
||||
`purgeWorkerPartials()` deletes by prefix. **Should partials be namespaced per run, or is "do not
|
||||
run two TIA suites at once" the contract?**
|
||||
3. `files` and `edges` never lose a deleted *source* file. **Cap, GC, or accept?**
|
||||
4. Orphaned state dirs accumulate under `~/.pest/tia` whenever a project's origin URL changes.
|
||||
**Reclaim them, or accept?**
|
||||
@@ -1,455 +0,0 @@
|
||||
# TIA default-branch fallback — phase three
|
||||
|
||||
## Your task
|
||||
|
||||
Fix [pestphp/pest#1823](https://github.com/pestphp/pest/issues/1823) — TIA's cached results never
|
||||
hit for repos whose default branch is not literally `main` — then re-run the **whole** conformance
|
||||
matrix against the playground app, plus the new section **L** that covers the fix.
|
||||
|
||||
Work in this order, and **stop where the plan says stop**:
|
||||
|
||||
1. **Part 1** — read the diagnosis. It is already measured; do not re-derive it, but do re-confirm
|
||||
the two reproductions in Part 1.3 take ~2 minutes and prove your environment is sane.
|
||||
2. **Part 2** — implement the fix in the `pestphp/pest` repo. **Do not commit. Do not touch the
|
||||
playground's `vendor/`.**
|
||||
3. **HARD STOP → Part 3.** Report the diff to Nuno and wait. He validates, commits, and syncs it
|
||||
into the playground himself. You must not proceed until he confirms.
|
||||
4. **Part 4** — verify the sync landed, then run section **L** (new) and the full phase-two matrix
|
||||
(**A–K**, all 156 rows) against the playground.
|
||||
5. **Part 5** — report in the given format.
|
||||
|
||||
Per `CLAUDE.md`: **do not write new `pestphp/pest` unit tests and do not run `composer test`.** Make
|
||||
the change, report it, and ask whether repo tests should be added. The section-L rows are *playground
|
||||
invocations*, not repo tests — those are the deliverable and are always in scope.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — The diagnosis (already measured; commit `bfd5b756`)
|
||||
|
||||
### 1.1 Root cause — two independent hardcoded `'main'` literals
|
||||
|
||||
**(a) The read fallback.** `src/Plugins/Tia/Graph.php` — seven methods default the fallback to the
|
||||
literal `'main'`:
|
||||
|
||||
| line | method |
|
||||
|---|---|
|
||||
| 579 | `recordedAtSha(string $branch, string $fallbackBranch = 'main')` |
|
||||
| 614 | `getAssertions(…, string $fallbackBranch = 'main')` |
|
||||
| 625 | `getTime(…, string $fallbackBranch = 'main')` |
|
||||
| 636 | `getResult(…, string $fallbackBranch = 'main')` |
|
||||
| 663 | `testFilesToRerun(string $branch, string $fallbackBranch = 'main')` |
|
||||
| 700 | `hasUnlocatedTestsToRerun(string $branch, string $fallbackBranch = 'main')` |
|
||||
| 811 | `lastRunTree(string $branch, string $fallbackBranch = 'main')` |
|
||||
|
||||
They all funnel into `Graph::baselineFor()` (line 819), which *does* implement a real cross-branch
|
||||
fallback:
|
||||
|
||||
```php
|
||||
if (isset($this->baselines[$branch])) return $this->baselines[$branch];
|
||||
if ($branch !== $fallbackBranch && isset($this->baselines[$fallbackBranch])) return $this->baselines[$fallbackBranch];
|
||||
return ['sha' => null, 'tree' => [], 'results' => []];
|
||||
```
|
||||
|
||||
The mechanism is deliberate. The problem is that **no caller ever passes `$fallbackBranch`** — all
|
||||
nine call sites in `src/Plugins/Tia.php` (lines 419, 429, 432, 785, 857, 1026, 1031, 1048, 1056) pass
|
||||
only `$this->branch`. So on a `master`-named repo the second branch of `baselineFor()` can never
|
||||
fire, and the whole mechanism is dead code.
|
||||
|
||||
`grep -rniE 'defaultBranch|symbolic-ref|init\.defaultBranch|origin/HEAD' src/` returns **nothing** —
|
||||
no default-branch resolution exists anywhere.
|
||||
|
||||
**(b) The detached-HEAD default.** `src/Plugins/Tia.php:206` declares `private string $branch = 'main';`
|
||||
and `ChangedFiles::currentBranch()` (`ChangedFiles.php:208`) returns `null` for detached HEAD. So on
|
||||
detached HEAD `$this->branch` stays the literal `'main'` and is used for **both reads and writes** —
|
||||
minting a baseline key for a branch that does not exist. This is a *write*-side bug and a separate fix
|
||||
from (a).
|
||||
|
||||
### 1.2 Not a regression
|
||||
|
||||
`git log -S"fallbackBranch = 'main'"` bottoms out at `c7e32f5d feat(tia): continues to work on poc`.
|
||||
This is original PoC code, untouched by phase one. Phase one's change 1 modified
|
||||
`hasUnlocatedTestsToRerun()`'s file-existence check — one of the seven methods — without going near
|
||||
the fallback. Do not report it as a phase-one regression.
|
||||
|
||||
No test anywhere exercises the mechanism: `tests/Unit/Plugins/Tia/Graph.php` uses `'main'` as the
|
||||
*actual* branch name, so those assertions pass whether or not the fallback exists. A branch-name
|
||||
mismatch is never tested.
|
||||
|
||||
### 1.3 The two reproductions (re-confirm these before you start)
|
||||
|
||||
Both on the playground, `master`-named default branch, zero local changes:
|
||||
|
||||
```
|
||||
default = master default = main
|
||||
1. record on default → full run (cold) → full run (cold)
|
||||
2. 1st run on feature-x → 25 UNCACHED → 25 replayed ← (a)
|
||||
3. 2nd run on feature-x → 25 replayed → 25 replayed
|
||||
4. back on default → 25 replayed → 25 replayed
|
||||
5. 1st run on feature-y → 25 UNCACHED → 25 replayed
|
||||
```
|
||||
|
||||
```
|
||||
master-only graph, then `git checkout --detach`, then `pest --tia`:
|
||||
→ 25 uncached, and keys become [master,main] ← (b) spurious key
|
||||
```
|
||||
|
||||
The cost is **one full run per new branch, forever**, with no output explaining why — the only clue
|
||||
is the `N uncached` count; the headline is just `─ Experimental TIA mode enabled.`
|
||||
|
||||
### 1.4 Correction to phase two
|
||||
|
||||
Phase two reported **H6 and H7 as passing. Both were false passes.** They ran after H5, which had
|
||||
renamed `master`→`main` and left a `main` key in the graph, so the hardcoded fallback resolved by
|
||||
accident. Re-measured against a clean `master`-only graph, both fail. Section L replaces them as the
|
||||
load-bearing rows; H6/H7 must be re-run **from a cold graph** this time (see Part 4.2).
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — The fix to implement
|
||||
|
||||
### 2.1 Recommended design
|
||||
|
||||
**Step 1 — add a non-throwing resolver** to `src/Plugins/Tia/ChangedFiles.php`, next to
|
||||
`currentBranch()`:
|
||||
|
||||
```php
|
||||
public function defaultBranch(): ?string
|
||||
```
|
||||
|
||||
Resolution order, each step failing soft to the next:
|
||||
|
||||
1. `git symbolic-ref --short refs/remotes/origin/HEAD` → strip a leading `origin/`
|
||||
2. `git config --get init.defaultBranch`
|
||||
3. `null`
|
||||
|
||||
Unlike `currentBranch()`, this must **never throw** `MissingDependency` — it is advisory. Return
|
||||
`null` on any non-zero exit or empty output.
|
||||
|
||||
**Step 2 — add a config surface.** `src/Plugins/Tia/Configuration.php` already exposes `always()`,
|
||||
`locally()`, `filtered()`, `baselined()`, `watch()`. Add:
|
||||
|
||||
```php
|
||||
public function defaultBranch(string $branch): self
|
||||
```
|
||||
|
||||
so `pest()->tia()->defaultBranch('master')` works in `tests/Pest.php`. Explicit config **always
|
||||
wins** over autodetection — that is the escape hatch when `origin/HEAD` is unset.
|
||||
|
||||
**Step 3 — resolve once, in `Tia.php`.** The read path is hot (`getResult()` is called per test at
|
||||
line 419), so resolution must not shell out per call. Resolve alongside `$this->branch` at
|
||||
`Tia.php:1910`, under the existing `$branchResolved` guard:
|
||||
|
||||
```php
|
||||
$this->fallbackBranch = $configuredDefaultBranch
|
||||
?? $changedFiles->defaultBranch()
|
||||
?? 'main';
|
||||
```
|
||||
|
||||
**Step 4 — thread it into `Graph`.** Prefer a `Graph`-level property over editing nine call sites:
|
||||
add `Graph::setFallbackBranch(string $branch)`, change the seven signatures to
|
||||
`?string $fallbackBranch = null`, and resolve inside each with
|
||||
`$fallbackBranch ??= $this->fallbackBranch;`. `baselineFor()` itself needs no change. This keeps the
|
||||
public signatures backward-compatible and minimises blast radius.
|
||||
|
||||
**Step 5 — fix the detached-HEAD write.** `Tia.php:206`'s `= 'main'` default must become the
|
||||
resolved default branch, so detached HEAD stops minting a phantom key.
|
||||
|
||||
### 2.2 Invariants the fix must not break
|
||||
|
||||
These are all covered by existing matrix rows — the fix is wrong if any of them moves:
|
||||
|
||||
- **Read-only.** The fallback must affect *reads* only. Writes go through `ensureBaseline($branch)`
|
||||
and must keep using the real current branch. Otherwise H1–H4/H8 ("no baseline key other than the
|
||||
real branch") break.
|
||||
- **H9** — a non-git dir with `--tia` must still raise
|
||||
`MissingDependency: The feature "Tia mode" requires "git".` Adding a soft resolver must not
|
||||
swallow that.
|
||||
- **H10** — plain `pest` in a non-git dir must still run and create no baseline dir.
|
||||
- **A2/A3** — cold-graph recording unchanged.
|
||||
- **I1/E3** — clean+green `--tia --filtered` must still be a true zero-delta run.
|
||||
- Filtered mode reads `testFilesToRerun()` and `hasUnlocatedTestsToRerun()`, so the fallback must
|
||||
reach those two as well, not just `getResult()`.
|
||||
|
||||
### 2.3 Decisions for Nuno (raise these at the Part 3 stop)
|
||||
|
||||
- **D1** — Consult `origin/HEAD` at all? It requires `git remote set-head` and is absent in many CI
|
||||
checkouts and all remote-less repos. Config + `init.defaultBranch` only is simpler but helps fewer
|
||||
people out of the box. *Recommendation: keep it, first in the chain, since it fails soft.*
|
||||
- **D2** — Single-key heuristic: if the graph holds exactly one baseline key, use it as the fallback?
|
||||
Fixes the issue with zero git calls, but is implicit and surprising when several keys exist.
|
||||
*Recommendation: no.*
|
||||
- **D3** — Should detached HEAD write a baseline at all, or be read-only? Current behaviour mints a
|
||||
key. *Recommendation: read-only.*
|
||||
- **D4** — Should `pest()->tia()->defaultBranch()` validate that the branch exists, or accept any
|
||||
string? *Recommendation: accept any string; a nonexistent name degrades to a full run, which is
|
||||
safe.*
|
||||
|
||||
---
|
||||
|
||||
## Part 3 — HARD STOP
|
||||
|
||||
When the code is written:
|
||||
|
||||
1. Show Nuno the diff (`git -C /Users/nunomaduro/Work/projects/pestphp/pest diff`) and a one-paragraph
|
||||
summary of each file's change.
|
||||
2. Answer/raise the D1–D4 decisions.
|
||||
3. State explicitly that you have **not** committed and have **not** synced `vendor/`.
|
||||
4. Ask whether repo unit tests should be added (per `CLAUDE.md`), describing the tests you have in
|
||||
mind — do not write them yet.
|
||||
5. **Wait.** Nuno commits and applies the change to the playground.
|
||||
|
||||
**Never sync the playground's `vendor/` yourself.** `vendor/pestphp/pest` there is a dist copy, not a
|
||||
symlink (composer installed `dev-fix/tia-filtered as 5.2.0`), so pest-repo edits do not reach it. Say
|
||||
what is stale and wait. This applies to before/after contrasts too.
|
||||
|
||||
Once he confirms, verify the sync actually landed before measuring anything:
|
||||
|
||||
```bash
|
||||
V=/Users/nunomaduro/Work/projects/playground/laravel/vendor/pestphp/pest
|
||||
grep -c 'defaultBranch' "$V/src/Plugins/Tia/ChangedFiles.php" # must be ≥ 1
|
||||
for f in src/Plugins/Tia.php src/Plugins/Tia/Graph.php src/Plugins/Tia/ChangedFiles.php \
|
||||
src/Plugins/Tia/Configuration.php; do
|
||||
diff -q "/Users/nunomaduro/Work/projects/pestphp/pest/$f" "$V/$f" >/dev/null \
|
||||
&& echo "SAME $f" || echo "STALE $f"
|
||||
done
|
||||
```
|
||||
|
||||
If anything reports `STALE`, stop and tell him. State in your final report which pest commit produced
|
||||
the playground numbers.
|
||||
|
||||
---
|
||||
|
||||
## Part 4 — Measurement
|
||||
|
||||
### 4.1 Environment and traps
|
||||
|
||||
**Playground:** `/Users/nunomaduro/Work/projects/playground/laravel` (branch `master`).
|
||||
|
||||
1. **Every** pest invocation needs `PAO_DISABLE=1` — the app has `laravel/pao`, which emits JSON when
|
||||
it detects an agent.
|
||||
2. **Pin the interpreter to `php85`.** The playground requires PHP `>= 8.4.1` *and* pcov. Locally only
|
||||
`php85` (8.5.8) has both — `php84` (8.4.23) has no pcov, and the bare Herd `php` shim has been
|
||||
observed drifting to 8.3.32 mid-session, which kills every run in
|
||||
`vendor/composer/platform_check.php`. Also put a `php` → `php85` symlink first on `PATH`: the
|
||||
`--shard` list-tests probe spawns a subprocess via bare `php`, not `PHP_BINARY`.
|
||||
3. **Never run `git checkout .` or `git checkout -- composer.lock` in the playground.** It has four
|
||||
pre-existing user-modified files — `AGENTS.md`, `CLAUDE.md`, `composer.json`, `composer.lock`. Scope
|
||||
resets to `git checkout -- tests app` / `git clean -fd tests app`. For `phpunit.xml` and
|
||||
`composer.lock`, copy aside and copy back, verifying with `shasum`.
|
||||
4. **zsh does not word-split unquoted parameters.** A `$PEST` string containing a space becomes one
|
||||
command name. Route every invocation through `eval` (the `pest()` helper below does this).
|
||||
5. Comment-only edits to a test file are **not** changes (AST-level hashing). Use semantic edits.
|
||||
6. **The sentinel technique must not falsify `assertions` on zero-assertion tests.** A
|
||||
risky/skipped/incomplete status is *derived* from "performed no assertions", so patching those to
|
||||
`42` rewrites the status on replay and destroys the discriminator — it shows up as a phantom
|
||||
`status 5→0` defect. The `sentinel.php` below only patches `assertions` where it is already
|
||||
non-zero. With that, a full replay gives a clean `rewritten=0`.
|
||||
7. Restore branch state after every L row. Several rows rename or detach; leaving a stray `main` key
|
||||
in the graph is exactly what produced phase two's false H6/H7 passes.
|
||||
|
||||
### 4.2 Reference numbers
|
||||
|
||||
The playground already carries the phase-two fixtures at commit `fb2e77e` — **do not rebuild them.**
|
||||
A healthy sequential `--tia` graph is:
|
||||
|
||||
> **`files=27`, 10 edge keys (one per test file), self-edge on all 10, `n=25` results.**
|
||||
|
||||
`sha` will differ once Nuno commits the vendor sync — re-derive it once and use it throughout. The
|
||||
`files`/`edges`/`n` numbers hold as long as no test fixture changes. Suite shape: 10 test files, 25
|
||||
tests, including six deliberate status fixtures (skipped, todo, incomplete, risky, warning,
|
||||
deprecation), a 3-row dataset, a `smoke` group, an env-driven flaky test (green unless
|
||||
`FLAKY_FAIL=1`), and the annotation set (`covers`/`note`/`flaky`/`issue`/`pr`/`ticket`/`assignee`).
|
||||
|
||||
**Re-run H6 and H7 from a cold graph** (`rm -rf` the graph dir, record on `master` only, *then*
|
||||
branch/detach). Their phase-two results are void.
|
||||
|
||||
### 4.3 Harness
|
||||
|
||||
Write these to your scratchpad. `$SP` is your own scratchpad dir.
|
||||
|
||||
<details>
|
||||
<summary><code>lib.sh</code></summary>
|
||||
|
||||
```bash
|
||||
#!/bin/zsh
|
||||
export PAO_DISABLE=1
|
||||
PG=/Users/nunomaduro/Work/projects/playground/laravel
|
||||
SP="<your scratchpad>"
|
||||
PHPBIN="php85"
|
||||
PEST="$PHPBIN $PG/vendor/bin/pest"
|
||||
cd "$PG" || exit 1
|
||||
pest() { eval "$PEST $*"; } # zsh: no word-splitting, must eval
|
||||
GRAPHDIR="$(pest --baseline)"
|
||||
GRAPH="$GRAPHDIR/graph.json"
|
||||
mkdir -p "$SP/bin" && ln -sf "$(command -v php85)" "$SP/bin/php"
|
||||
export PATH="$SP/bin:$PATH" # --shard spawns bare `php`
|
||||
reset_tree() { git checkout -- tests app 2>/dev/null; git clean -qfd tests app 2>/dev/null; }
|
||||
seed() { rm -rf "$GRAPHDIR"; pest --tia >/dev/null 2>&1; $PHPBIN "$SP/sentinel.php" "$GRAPH" >/dev/null; cp "$GRAPH" "$SP/before.json"; }
|
||||
snap() { cp "$GRAPH" "$SP/before.json"; }
|
||||
delta() { $PHPBIN "$SP/cmp.php" "$SP/before.json" "$GRAPH"; }
|
||||
keys() { $PHPBIN -r '$g=json_decode(file_get_contents($argv[1]),true);echo "[".implode(",",array_keys($g["baselines"]??[]))."]";' "$GRAPH"; }
|
||||
tally() { sed -E $'s/\x1b\\[[0-9;]*[a-zA-Z]//g' <"$SP/out.txt" | grep -E 'Tests:' | sed -E 's/^ +//;s/Tests: +//'; }
|
||||
```
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><code>sentinel.php</code> — the write discriminator</summary>
|
||||
|
||||
```php
|
||||
<?php // sentinel.php <graph.json>
|
||||
$p = $argv[1];
|
||||
$g = json_decode((string) file_get_contents($p), true, 512, JSON_THROW_ON_ERROR);
|
||||
$n = 0;
|
||||
foreach ($g['baselines'] ?? [] as $br => $b) {
|
||||
foreach (array_keys($b['results'] ?? []) as $id) {
|
||||
$g['baselines'][$br]['results'][$id]['time'] = 9.999;
|
||||
// Only falsify a non-zero assertion count: risky/skipped/incomplete are
|
||||
// DERIVED from "performed no assertions", so patching those to 42 would
|
||||
// rewrite the status on replay and destroy the discriminator.
|
||||
if ((int) ($b['results'][$id]['assertions'] ?? 0) > 0) {
|
||||
$g['baselines'][$br]['results'][$id]['assertions'] = 42;
|
||||
}
|
||||
$n++;
|
||||
}
|
||||
}
|
||||
file_put_contents($p, json_encode($g, JSON_THROW_ON_ERROR));
|
||||
echo "sentinelled $n results\n";
|
||||
```
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><code>oneline.php</code> — one compact tier verdict per row</summary>
|
||||
|
||||
```php
|
||||
<?php // oneline.php <before.json> <after.json>
|
||||
function load(string $p): ?array {
|
||||
return is_file($p) ? json_decode((string) file_get_contents($p), true, 512, JSON_THROW_ON_ERROR) : null;
|
||||
}
|
||||
$a = load($argv[1]); $b = load($argv[2]);
|
||||
if ($a === null || $b === null) { echo 'GRAPH '.($b === null ? 'DELETED' : 'CREATED'); exit; }
|
||||
function edgeSets(array $g): array {
|
||||
$out = [];
|
||||
foreach ($g['edges'] ?? [] as $t => $ids) { $s = array_map(fn($i) => $g['files'][$i] ?? "?$i", (array) $ids); sort($s); $out[$t] = $s; }
|
||||
ksort($out); return $out;
|
||||
}
|
||||
$moved = [];
|
||||
if (edgeSets($a) !== edgeSets($b)) $moved[] = 'edges';
|
||||
if (($a['files'] ?? []) !== ($b['files'] ?? [])) $moved[] = 'files';
|
||||
if (($a['fingerprint'] ?? null) !== ($b['fingerprint'] ?? null)) $moved[] = 'fingerprint';
|
||||
$brA = array_keys($a['baselines'] ?? []); $brB = array_keys($b['baselines'] ?? []);
|
||||
if ($brA !== $brB) $moved[] = 'branchkeys('.implode('|', $brA).'->'.implode('|', $brB).')';
|
||||
$add = $rem = $wr = 0; $shaMoved = $treeMoved = false;
|
||||
foreach ($brB as $br) {
|
||||
$ra = $a['baselines'][$br]['results'] ?? []; $rb = $b['baselines'][$br]['results'] ?? [];
|
||||
if (($a['baselines'][$br]['sha'] ?? null) !== ($b['baselines'][$br]['sha'] ?? null)) $shaMoved = true;
|
||||
if (($a['baselines'][$br]['tree'] ?? null) !== ($b['baselines'][$br]['tree'] ?? null)) $treeMoved = true;
|
||||
$add += count(array_diff(array_keys($rb), array_keys($ra)));
|
||||
$rem += count(array_diff(array_keys($ra), array_keys($rb)));
|
||||
foreach ($ra as $id => $x) {
|
||||
if (! isset($rb[$id])) continue;
|
||||
foreach (['status','time','assertions','message'] as $f) {
|
||||
if (($x[$f] ?? null) !== ($rb[$id][$f] ?? null)) { $wr++; break; }
|
||||
}
|
||||
}
|
||||
}
|
||||
$n = 0; foreach ($brB as $br) $n = max($n, count($b['baselines'][$br]['results'] ?? []));
|
||||
printf('n=%d w=%-2d +%d -%d %s%s%s', $n, $wr, $add, $rem,
|
||||
$moved === [] ? 'struct:ok' : 'STRUCT:'.implode(',', $moved),
|
||||
$shaMoved ? ' sha:CHANGED' : '', $treeMoved ? ' tree:chg' : '');
|
||||
```
|
||||
</details>
|
||||
|
||||
`cmp.php` (verbose per-entry version of the same) and `summarise.php` / `edgediff.php` are in
|
||||
`PLAN_PHASE_TWO.md` §"Graph summariser" — reuse them for drill-downs. Reading the verdict:
|
||||
|
||||
- `w=` — entries actually **written**. Under sentinel patching this is the only reliable way to tell
|
||||
"wrote identical values" from "wrote nothing". A full replay must give `w=0`.
|
||||
- `struct:ok` + `+0 -0` — no prune, no edges/files/fingerprint movement. Required by RESULTS-ONLY.
|
||||
- `STRUCT:branchkeys(...)` — a new baseline key appeared. For section L this is the headline signal.
|
||||
|
||||
### 4.4 Section L — new rows for this fix
|
||||
|
||||
Tiers, unchanged from phase two: **COMPLETE** may change everything · **RESULTS-ONLY (RO)** may
|
||||
change only `baselines[<branch>].results` for tests that ran, and must never remove an entry, add a
|
||||
result for a test file absent from `edges`, or alter `sha`/`tree`/`edges`/`files`/`fingerprint` ·
|
||||
**HARD-SUPPRESSED** may change nothing.
|
||||
|
||||
Every L row starts from a **cold graph recorded on the named default branch only** — verify
|
||||
`keys=[<default>]` before branching. Restore branch state afterwards.
|
||||
|
||||
| # | Case | Target outcome |
|
||||
|---|---|---|
|
||||
| L1 | default `master`, record, `git switch -c feature-x`, `pest --tia`, zero changes | **all 25 replayed** (`w=0`), not `25 uncached`. The headline fix. |
|
||||
| L2 | as L1 but default `main` | still all replayed — regression guard, this already worked |
|
||||
| L3 | default `trunk`, then `develop` | replayed for both; the fix must not special-case two names |
|
||||
| L4 | L1, then a *second* new branch `feature-y` | replayed too — the toll must not return per branch |
|
||||
| L5 | L1 then edit `app/Services/Calculator.php` on `feature-x` | narrows to the 2 affected files (`CalculatorTest` + `AnnotationsTest`, which `covers` it); the other 17 replay |
|
||||
| L6 | L1 with `--tia --filtered` | filtered mode reads the fallback too (`testFilesToRerun`, `hasUnlocatedTestsToRerun`) → `No affected tests found`, zero delta |
|
||||
| L7 | L1 with `--tia --parallel` | fallback works in workers as well as the parent |
|
||||
| L8 | L1, then `pest --tia` twice on `feature-x` | idempotent; second run also `w=0` |
|
||||
| L9 | detached HEAD on a `master`-only graph | replays, and **no `main` key minted** — `keys` stays `[master]`. Bug (b). |
|
||||
| L10 | `pest()->tia()->defaultBranch('master')` in `tests/Pest.php`, repo default renamed away | config wins over autodetect |
|
||||
| L11 | config set to a nonexistent branch (`defaultBranch('nope')`) | degrades to a full run; no crash, no phantom key |
|
||||
| L12 | no remote at all (`git remote remove origin` if present) | still resolves (via `init.defaultBranch`) or degrades safely — must not throw |
|
||||
| L13 | branch name with a slash (`feature/x/y`) | replayed; no key-splitting bugs |
|
||||
| L14 | L1, then confirm writes | `feature-x` gets its **own** key; the `master` key is **not** written to (fallback is read-only) |
|
||||
| L15 | git worktree on a new branch (the issue's scenario) | replays from the default-branch baseline |
|
||||
| L16 | non-git dir, `pest --tia` | still `MissingDependency: The feature "Tia mode" requires "git".` — the soft resolver must not swallow it |
|
||||
| L17 | non-git dir, plain `pest` | runs normally; no baseline dir created |
|
||||
| L18 | count `git` subprocesses during one `--tia` run | default-branch resolution is cached, not one call per test. Probe by shimming `git` on `PATH` to a logging wrapper. |
|
||||
|
||||
L10–L11 need a `tests/Pest.php` edit — that file is tracked and **outside** the `tests app` reset
|
||||
scope in practice (it lives in `tests/`, so `git checkout -- tests` does restore it; verify with
|
||||
`git status` after).
|
||||
|
||||
For L16/L17, build a throwaway non-git project — and note the trap that burned phase two: a
|
||||
**symlinked** `vendor` makes Pest resolve the project root back to the playground (identical baseline
|
||||
hash), silently invalidating the test. Use a hardlinked copy:
|
||||
|
||||
```bash
|
||||
NG="$SP/nogit"; rm -rf "$NG"; mkdir -p "$NG"
|
||||
cp -R composer.json composer.lock phpunit.xml artisan tests app bootstrap config routes resources storage "$NG/"
|
||||
[ -f .env ] && cp .env "$NG/"
|
||||
cp -Rl vendor "$NG/vendor" || cp -R vendor "$NG/vendor"
|
||||
```
|
||||
|
||||
Confirm the baseline path differs (`nogit-<hash>`, not `laravel-4a455a95622ac0ec`), and delete both
|
||||
the temp project and its `~/.pest/tia/nogit-*` dir afterwards.
|
||||
|
||||
### 4.5 Re-run the phase-two matrix (A–K, 156 rows)
|
||||
|
||||
Re-run every row of `PLAN_PHASE_TWO.md` Part 2 against the fixed build. No row's status is trusted
|
||||
until re-measured — the fix touches `Graph`'s read path, which nearly every row exercises. Sections
|
||||
**A, B, H, I** are the load-bearing ones here (H is branch-key resolution; I is filtered mode; both
|
||||
consume the changed methods directly). **H6 and H7 must be re-derived from a cold graph** (Part 1.4).
|
||||
|
||||
Most rows batch cheaply — phase two ran C1–C20 in one call at roughly one line of output each. Use
|
||||
`oneline.php` for the sweep and `cmp.php` only to drill into anomalies.
|
||||
|
||||
### 4.6 Known pre-existing failures — do not report as regressions
|
||||
|
||||
| item | status |
|
||||
|---|---|
|
||||
| ~~**G4 / G4b** — parallel replay clobbers cached `time` on all non-executed tests.~~ | **Struck in phase four — does not reproduce.** `flushWorkerReplay()` applies `resultTime()` worker-side before writing the partial, so the parent's verbatim read of `$result['time']` is reading values that were already corrected. Pinned by `a parallel replay keeps the recorded time of tests that did not run` (`tests/Features/Tia/CompleteRunWriteTier.php`), which sentinels every cached `time` and asserts a parallel replay writes nothing. |
|
||||
| **C19** — `--tia --uses=…` cannot be fixtured. TIA hard-errors on PHPUnit classes (`EnsureTiaIsRunningPestTestsOnly`), and Pest has no chainable `->uses()`. | **Expected behaviour per Nuno.** Verify the tier (`w=0`, RO, notice) and move on. Not a defect. |
|
||||
| **J11** — `--repeat` is not a Pest option (`Unknown option "--repeat"`). | **Don't care per Nuno.** Mark SKIP. |
|
||||
| **J10** — `--random-order-seed` alone exits 1 with a WARN. Identical without `--tia`. | Pre-existing Pest behaviour, unrelated. Tier still holds. |
|
||||
|
||||
---
|
||||
|
||||
## Part 5 — Reporting
|
||||
|
||||
Per row: **tier respected (yes/no)**, the **graph delta** under sentinel patching, and for any failure
|
||||
the **pre-fix contrast** so a regression is told apart from a pre-existing defect. For a pre-fix
|
||||
contrast you need `db70017c` (or `bfd5b756` for "before phase three") files swapped into the
|
||||
playground's `vendor/` — that is a sync, so **ask Nuno first** and always restore afterwards.
|
||||
|
||||
State which pest commit produced the numbers. Close with:
|
||||
|
||||
1. Whether L1–L18 all pass (the fix works).
|
||||
2. Whether A–K regressed anywhere relative to phase two's 154/156.
|
||||
3. The D1–D4 decisions as implemented.
|
||||
4. Anything still open — including G4, which will still be failing.
|
||||
|
||||
Leave the playground on `master` with only the four user-modified files dirty, no stray branches, and
|
||||
no leftover `~/.pest/tia/*` dirs beyond `laravel-4a455a95622ac0ec`.
|
||||
@@ -1,466 +0,0 @@
|
||||
# TIA write-tier conformance — phase two
|
||||
|
||||
## Your task
|
||||
|
||||
**Re-run all 156 rows of the matrix in Part 2 from scratch, against the playground app** at
|
||||
`/Users/nunomaduro/Work/projects/playground/laravel`. Every row, including the ones already marked
|
||||
VERIFIED or PASS — the point of phase two is that no row's status is trusted until it has been
|
||||
re-measured against the current code. The `Phase 1` column is prior evidence and a hint at what to
|
||||
watch, never a reason to skip a row.
|
||||
|
||||
These are **not** the `pestphp/pest` repo's own tests (`composer test`) — do not run those. Each row is
|
||||
a `pest` invocation against the playground's suite, followed by a diff of the TIA graph it wrote.
|
||||
|
||||
Report, per row: tier respected (yes/no), the graph delta under sentinel patching, and — for any
|
||||
failure — the pre-fix contrast so a regression is told apart from a pre-existing defect. Do **not**
|
||||
stop at reading this file or summarising it; the deliverable is executed results.
|
||||
|
||||
Work in this order: **Part 1 (environment traps) → Part 1b (build the fixtures — the playground has
|
||||
none of them) → Part 2 (the matrix) → Part 3 (priorities)**. The matrix is too large for one context;
|
||||
take it one lettered section at a time and report as you finish each. Sections A, B, G, I and K are
|
||||
the load-bearing ones — do those first if you run short.
|
||||
|
||||
Phase one implemented `PLAN.md` Part 1 items 1–4 plus three §5 items. This file turns the matrix into
|
||||
a conformance check rather than a bug list.
|
||||
|
||||
Code under test: `pestphp/pest` at `/Users/nunomaduro/Work/projects/pestphp/pest`, branch
|
||||
`fix/tia-filtered`, commit **`bfd5b756`** or later. Verify with
|
||||
`grep -c 'recordsEdgesInWorkers\|recordsEdges' src/Plugins/Tia.php` → at least 3 hits. Pre-fix
|
||||
baseline for every contrast is **`db70017c`**.
|
||||
|
||||
---
|
||||
|
||||
## Part 0 — What phase one changed
|
||||
|
||||
| # | Change | Files |
|
||||
|---|---|---|
|
||||
| 1 | `hasUnlocatedTestsToRerun()` stats the file, so a deleted test file is "unlocated" | `src/Plugins/Tia/Graph.php` |
|
||||
| 2 | `enterReplayMode()` uses `activateLinkTracking()` under piggyback coverage | `src/Plugins/Tia.php` |
|
||||
| 3 | `enterReplayMode()` stamps `TIA_PIGGYBACK_COVERAGE` for workers | `src/Plugins/Tia.php` |
|
||||
| 4 | `replaceEdges(…, keepExisting:)` — piggyback edges seed empty sets, never overwrite populated ones | `Graph.php`, `Tia.php` |
|
||||
| 5 | `renderFreshGraph()` stops claiming "fresh graph" when the graph is kept; reason reworded to `recording a coverage baseline` | `Tia.php` |
|
||||
| 6 | `COVERAGE_REPORT_FLAGS` + `coverageReportActive()` union over `originalArguments`; new `pestCoverageActive()` keeps the coverage-cache marker/hijack on Pest's own `--coverage` | `Tia.php` |
|
||||
| 7 | `Tia::recordsEdgesInWorkers()` + `WrapperRunner::handleTia()` inject `-d pcov.directory=<root>` into worker argv | `Tia.php`, `src/Plugins/Parallel/Paratest/WrapperRunner.php` |
|
||||
| 8 | Sequential record runs announce structural drift via `renderFreshGraph()` | `Tia.php` |
|
||||
| 9 | `Graph::getTime()` + `cachedTimeByTestId` + `resultTime()` preserve replayed durations; edge-less write guard is now `$recordsEdges = $complete && ($markKnownTestFiles \|\| $this->recordingActive)` | `Graph.php`, `Tia.php` |
|
||||
|
||||
Deliberately **not** done: `PLAN.md` §5 SIGINT propagation, §5 warning/deprecation `status=0`
|
||||
mapping, and all of §6 (playground annotation fixtures). The vacuous C rows below stay vacuous.
|
||||
|
||||
### Target-outcome changes this forces
|
||||
|
||||
Two rows in the original matrix asserted the **old**, buggy behaviour. Their targets are updated
|
||||
below — do not report them as regressions:
|
||||
|
||||
- **B10** was "`time` differs; statuses stable". Change 9 means replayed entries now **keep** their
|
||||
recorded `time`. New target: `time` differs only for tests that actually executed.
|
||||
- **B12** gains an edge-preservation assertion it never had (see K1).
|
||||
|
||||
### One known-failing repo test
|
||||
|
||||
`tests/Unit/Plugins/Tia/Graph.php:69-76` asserts `hasUnlocatedTestsToRerun('main')` is `false` for
|
||||
`tests/Feature/FooTest.php` under `new Graph(sys_get_temp_dir())` — a path that does not exist. That
|
||||
assertion encodes the I9 bug and **will fail** under change 1. It needs re-pointing at a root/file
|
||||
that exists (e.g. `dirname(__DIR__, 4)` + `'tests/Unit/Plugins/Tia/Graph.php'`). Left untouched by
|
||||
request; it is a repo-test matter, not a playground one.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — Environment, and the traps in it
|
||||
|
||||
**Playground:** `/Users/nunomaduro/Work/projects/playground/laravel`
|
||||
|
||||
```bash
|
||||
cd /Users/nunomaduro/Work/projects/playground/laravel
|
||||
GRAPH="$(PAO_DISABLE=1 ./vendor/bin/pest --baseline)/graph.json" # ~/.pest/tia/laravel-4a455a95622ac0ec
|
||||
```
|
||||
|
||||
1. **Every** pest invocation needs `PAO_DISABLE=1` — the app has `laravel/pao`, which emits JSON
|
||||
when it detects an agent.
|
||||
2. **Never run `git checkout .` or `git checkout -- composer.lock` in the playground.** It has four
|
||||
pre-existing user-modified files — `AGENTS.md`, `CLAUDE.md`, `composer.json`, `composer.lock` —
|
||||
that a blanket reset would destroy. `PLAN.md`'s "reset ritual" is unsafe as written. Scope resets
|
||||
to what you touched: `git checkout -- tests app` / `git clean -fd tests app`, and for
|
||||
`composer.lock` (needed for the drift rows) **copy it aside and copy it back**, verifying with
|
||||
`shasum`.
|
||||
3. **`vendor/pestphp/pest` is a dist copy, not a symlink.** Composer installed
|
||||
`dev-fix/tia-filtered as 5.2.0`, so edits in the pest repo do **not** reach the playground.
|
||||
|
||||
**Tell Nuno whenever a sync is needed to move forward — do not sync silently.** Say what is stale
|
||||
and what the sync would be, then wait. This includes temporarily swapping in `db70017c` files for a
|
||||
before/after contrast. When you report any playground result, state which commit produced it.
|
||||
The sync itself, once he agrees:
|
||||
```bash
|
||||
PEST=/Users/nunomaduro/Work/projects/pestphp/pest
|
||||
V=/Users/nunomaduro/Work/projects/playground/laravel/vendor/pestphp/pest
|
||||
for f in src/Plugins/Tia.php src/Plugins/Tia/Graph.php \
|
||||
src/Plugins/Parallel/Paratest/WrapperRunner.php; do cp "$PEST/$f" "$V/$f"; done
|
||||
```
|
||||
Verify with `grep -c recordsEdgesInWorkers "$V/src/Plugins/Tia.php"` → `1`. **Always restore the
|
||||
current version before continuing** after a pre-fix contrast.
|
||||
4. **Coverage driver:** pcov only, no xdebug. `ini_get('pcov.directory')` is `''` by default — that
|
||||
emptiness is the entire mechanism behind G12.
|
||||
5. **Suite shape as found:** 7 tests in 5 files — `tests/Unit/{CalculatorTest,ExampleTest,GreeterTest,
|
||||
TrioTest}.php`, `tests/Feature/ExampleTest.php`. A healthy sequential graph is **`files=22`,
|
||||
5 edge keys, self-edge on every test** (`PLAN.md`'s `files=25` is stale). Pre-fix parallel gives
|
||||
`files=4` with zero self-edges. **These numbers shift the moment you add the Part 1b fixtures** —
|
||||
re-derive them once, after the fixtures land, and use the new numbers throughout. The invariants
|
||||
that do *not* shift: sequential and parallel must agree, and every test must have a self-edge.
|
||||
6. **Sentinel patching is the only reliable discriminator** between "wrote identical values" and
|
||||
"wrote nothing": rewrite every cached result to `time=9.999 assertions=42`, snapshot, run the
|
||||
case, diff. A canary test absent from `edges` is unreliable under `--filter` because it never
|
||||
matches the filter and so never runs.
|
||||
7. Seeding a cached failure needs `--fresh` (or an env-driven flaky fixture): `--tia` on a clean
|
||||
green tree replays rather than executes, so it can never cache a failure. Working recipe — break
|
||||
an assertion, `pest --tia --fresh`, then restore the source.
|
||||
8. Comment-only edits to a test file are **not** changes (AST-level hashing). Use semantic edits.
|
||||
9. The shell is zsh — build commands with arrays or `eval`; unquoted `$args` does not word-split.
|
||||
|
||||
### Graph summariser
|
||||
|
||||
Write this to a scratch path and use it for every diff.
|
||||
|
||||
```php
|
||||
<?php // summarise.php <graph.json> [label]
|
||||
$g = json_decode((string) file_get_contents($argv[1]), true, 512, JSON_THROW_ON_ERROR);
|
||||
$files = $g['files'] ?? []; $edges = $g['edges'] ?? [];
|
||||
echo ($argv[2] ?? $argv[1])."\n files=".count($files).' edges keys='.count($edges)."\n";
|
||||
$dirs = []; foreach ($files as $f) { $dirs[explode('/', (string) $f)[0]] = true; }
|
||||
ksort($dirs); echo ' file dirs: '.implode(', ', array_keys($dirs))."\n";
|
||||
ksort($edges);
|
||||
foreach ($edges as $test => $ids) {
|
||||
$self = 'no';
|
||||
foreach ((array) $ids as $id) { if (($files[$id] ?? null) === $test) { $self = 'YES'; break; } }
|
||||
echo sprintf(" %-42s n=%-3d self=%s\n", $test, count((array) $ids), $self);
|
||||
}
|
||||
foreach ($g['baselines'] ?? [] as $branch => $b) {
|
||||
$r = $b['results'] ?? [];
|
||||
echo " baseline[$branch]: n=".count($r).' sha='.substr((string) ($b['sha'] ?? '-'), 0, 7)."\n";
|
||||
foreach ($r as $id => $x) {
|
||||
echo sprintf(" %-58s status=%d time=%s asserts=%d file=%s\n", substr((string) $id, -58),
|
||||
$x['status'], $x['time'], $x['assertions'], $x['file'] ?? '-');
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Normalised edge-set equality (for G12 / I5 / K1):
|
||||
|
||||
```php
|
||||
<?php // edgediff.php <a.json> <b.json>
|
||||
function edges(string $p): array {
|
||||
$g = json_decode((string) file_get_contents($p), true, 512, JSON_THROW_ON_ERROR); $out = [];
|
||||
foreach ($g['edges'] as $t => $ids) { $s = array_map(fn ($i) => $g['files'][$i], $ids); sort($s); $out[$t] = $s; }
|
||||
ksort($out); return $out;
|
||||
}
|
||||
$a = edges($argv[1]); $b = edges($argv[2]);
|
||||
echo $a === $b ? "IDENTICAL edge sets\n" : "DIFFER\n";
|
||||
foreach ($a as $t => $s) {
|
||||
$m = array_diff($s, $b[$t] ?? []); $e = array_diff($b[$t] ?? [], $s);
|
||||
if ($m || $e) printf(" %s: -%d +%d\n", $t, count($m), count($e));
|
||||
}
|
||||
```
|
||||
|
||||
### Per-case loop
|
||||
|
||||
```bash
|
||||
git checkout -- tests app 2>/dev/null; git clean -qfd tests app
|
||||
rm -rf "$(PAO_DISABLE=1 ./vendor/bin/pest --baseline)"
|
||||
PAO_DISABLE=1 ./vendor/bin/pest --tia >/dev/null 2>&1 # seed a healthy graph
|
||||
# sentinel-patch every result to time=9.999 assertions=42, snapshot, run the case, diff
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 1b — Fixtures you must build first
|
||||
|
||||
**The playground has none of the fixtures the matrix depends on.** Verified inventory: the only test
|
||||
files are `tests/Unit/{CalculatorTest,ExampleTest,GreeterTest,TrioTest}.php` and
|
||||
`tests/Feature/ExampleTest.php` — 7 plain tests, zero occurrences of `->group()`, `->only()`,
|
||||
`->skip()`, `->todo()`, `->note()`, `->flaky()`, `->covers()`, `->uses()`, `->issue()`, `->pr()`,
|
||||
`->ticket()`, `->assignee()`, datasets, or any `FLAKY_OK`-style env hook. `phpunit.xml` defines only
|
||||
the `Unit` and `Feature` testsuites.
|
||||
|
||||
So roughly 30 rows below cannot run as written until you create the fixtures. Build them **all up
|
||||
front, in one commit**, then re-derive the baseline graph shape once and use those numbers for the
|
||||
whole sweep — every fixture you add changes `files`, the `edges` key count and `n`, so adding them
|
||||
piecemeal invalidates earlier rows.
|
||||
|
||||
| Fixture to create | Rows that need it |
|
||||
|---|---|
|
||||
| a test with `->group('smoke')` | C7, C8, C9, C10 |
|
||||
| an env-driven flaky test (passes iff `FLAKY_OK=1`) | E1, E2, E3, E4 |
|
||||
| a `->skip()`ed test | E9, D8 |
|
||||
| a `->todo()` test | E10, C23, C39 |
|
||||
| a risky test (no assertions; pair with `--disallow-test-output`) | E11, D7 |
|
||||
| a test that triggers a PHPUnit warning | D6 |
|
||||
| a test calling `markTestIncomplete()` | D9 |
|
||||
| a test that triggers a deprecation | D10 |
|
||||
| a dataset test with ≥3 rows | E13, E14 |
|
||||
| `->only()` — added and removed per case, not left in | C31, C32, G7 |
|
||||
| `->covers(App\Services\Calculator::class)` | C18 |
|
||||
| `->uses(...)` / `UsesClass` annotation | C19 |
|
||||
| `->note(...)` | C24 |
|
||||
| `->flaky()` **annotation** (distinct from the env-driven flaky test above) | C25 |
|
||||
| `->issue(123)`, `->pr(1)`, `->ticket('X')`, `->assignee('X')` | C26, C27, C28, C37, C38 |
|
||||
|
||||
This is `PLAN.md` §6's "test-harness gaps to close before re-running", now itemised: without these,
|
||||
the listed rows' graph-invariant assertions match zero tests and **prove nothing** — they pass
|
||||
vacuously. Any row still marked "(vacuous)" in Part 2 is vacuous *only because* its fixture is
|
||||
missing; once you add the fixture, treat the row as unverified and make it load-bearing.
|
||||
|
||||
Rows needing an *action* rather than a fixture — breaking `trio one` for the D rows, an uncommitted
|
||||
test edit for `--dirty`, branch renames and a non-git dir for H5–H10, `--mutate` against
|
||||
`app/Services` for the F rows — are fine as written; `pestphp/pest-plugin-mutate` is installed.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — Full case matrix
|
||||
|
||||
Tiers: **COMPLETE** may change everything · **RESULTS-ONLY** (RO) may change only
|
||||
`baselines[<branch>].results` for tests that ran, and must never remove an entry, add a result for a
|
||||
test file absent from `edges`, or alter `sha`/`tree`/`edges`/`files`/`fingerprint` ·
|
||||
**HARD-SUPPRESSED** may change nothing.
|
||||
|
||||
Phase-1 column: **VERIFIED** = re-run against the fixed code in the phase-one session, pre-fix
|
||||
contrast captured · **PASS (sweep)** = passed in the original `db70017` sweep and *not* re-checked
|
||||
since the changes — these are the bulk of phase two's work · **SKIP** = not runnable as written.
|
||||
|
||||
### A — Setup & sanity
|
||||
|
||||
| # | Case | Target outcome | Phase 1 |
|
||||
|---|---|---|---|
|
||||
| A1 | `composer show pestphp/pest` | version `dev-fix/tia-filtered` | PASS (sweep) |
|
||||
| A2 | `pest --baseline` | prints an existing dir; exit 0 | VERIFIED |
|
||||
| A3 | delete graph, `pest --tia` | graph.json created; one `edges` key per test file; one result per test | VERIFIED |
|
||||
| A4 | `git status` after reset ritual | clean *except the four user-modified files* (see trap 2) | VERIFIED |
|
||||
| A5 | `extension_loaded("pcov")` | `true` | VERIFIED |
|
||||
| A6 | delete graph, plain `pest` | no graph created | PASS (sweep) |
|
||||
| A7 | delete graph, `pest --filter=adds` | no graph created | PASS (sweep) |
|
||||
| A8 | two consecutive `pest --tia` | second replays everything; `sha` unchanged | VERIFIED |
|
||||
|
||||
### B — COMPLETE runs still write
|
||||
|
||||
| # | Case | Target outcome | Phase 1 |
|
||||
|---|---|---|---|
|
||||
| B1 | `pest --tia` (clean) | replays; graph written; `sha` = `git rev-parse HEAD` | VERIFIED |
|
||||
| B2 | `pest --no-tia` | results refreshed; prune applied; `sha`/`tree`/`edges` unchanged | PASS (sweep) — recheck under change 9 |
|
||||
| B3 | `pest` (plain) | same as B2 | PASS (sweep) — recheck under change 9 |
|
||||
| B4 | `pest --tia --fresh` | graph purged and rebuilt; `edges` rebuilt; `files` repopulated | VERIFIED |
|
||||
| B5 | `pest --bail`, all green | COMPLETE — graph written, prune applied | PASS (sweep) |
|
||||
| B6 | edit `Calculator.php`, `pest --tia` | `CalculatorTest` re-runs, others replay; `tree` updated | PASS (sweep) |
|
||||
| B7 | delete `trio three`, `pest --tia` | its result pruned; `trio one`/`two` remain | PASS (sweep) |
|
||||
| B8 | delete `GreeterTest.php`, `pest --tia` | result and edges survive (missing-file prune is `--fresh`-only) | PASS (sweep) |
|
||||
| B9 | B8 then `pest --tia --fresh` | entry gone; `files` drops `Greeter.php` too | PASS (sweep) |
|
||||
| B10 | `pest --tia` twice | **UPDATED TARGET:** `time` differs only for tests that executed; replayed entries keep their recorded `time`; statuses stable | VERIFIED |
|
||||
| B11 | add a test file, `pest --tia` | new `edges` key **and** new result appear in the **same** run | VERIFIED (regression guard for change 9) |
|
||||
| B12 | `pest --tia --coverage` | completes; graph written; coverage report prints | VERIFIED |
|
||||
|
||||
### C — Selection narrowing → RESULTS-ONLY
|
||||
|
||||
All rows: RO invariants hold. "Notice" = `TIA does not apply to partial runs — running the selected tests directly.`
|
||||
|
||||
| # | Case | Target outcome | Phase 1 |
|
||||
|---|---|---|---|
|
||||
| C1 | `pest --filter="adds numbers"` | RO; no notice | PASS (sweep) |
|
||||
| C2 | `pest --tia --filter="adds numbers"` | RO; notice | PASS (sweep) |
|
||||
| C3 | `pest --filter="trio one"` | RO; `trio two`/`three` byte-identical | PASS (sweep) |
|
||||
| C4 | `pest --filter="trio"` | RO; all three update; nothing else does | PASS (sweep) |
|
||||
| C5 | `pest --exclude-filter="Feature"` | RO; no notice; no prune | PASS (sweep) |
|
||||
| C6 | `pest --tia --exclude-filter="Feature"` | RO; notice | PASS (sweep) |
|
||||
| C7 | `pest --group=smoke` | RO; no notice | PASS (sweep) |
|
||||
| C8 | `pest --tia --group=smoke` | RO; notice | PASS (sweep) |
|
||||
| C9 | `pest --exclude-group=smoke` | RO; no notice; no prune | PASS (sweep) |
|
||||
| C10 | `pest --tia --exclude-group=smoke` | RO; notice | PASS (sweep) |
|
||||
| C11 | `pest tests/Unit` | RO; no notice; Feature entry not pruned | PASS (sweep) |
|
||||
| C12 | `pest --tia tests/Unit` | RO; notice | PASS (sweep) |
|
||||
| C13 | `pest tests/Unit/TrioTest.php` | RO; no notice | PASS (sweep) |
|
||||
| C14 | `pest --tia tests/Unit/TrioTest.php` | RO; notice | PASS (sweep) |
|
||||
| C15 | `pest --testsuite=Unit` | RO; no notice | PASS (sweep) |
|
||||
| C16 | `pest --tia --testsuite=Unit` | RO; notice | PASS (sweep) |
|
||||
| C17 | `pest --tia --exclude-testsuite=Feature` | RO; notice | PASS (sweep) |
|
||||
| C18 | `pest --tia --covers='App\Services\Calculator'` | RO; notice | PASS (sweep) |
|
||||
| C19 | `pest --tia --uses='App\Services\Calculator'` | RO; notice | PASS (vacuous — needs fixture) |
|
||||
| C20 | `pest --tia --test-suffix=Test.php` | RO even though all tests run; `edges`/`files`/`n` unchanged | PASS (sweep) |
|
||||
| C21 | `pest --dirty` with an uncommitted test edit | RO; no notice | PASS (sweep) |
|
||||
| C22 | `pest --tia --dirty` | RO; notice | PASS (sweep) |
|
||||
| C23 | `pest --tia --todos` | RO; notice | PASS (vacuous) |
|
||||
| C24 | `pest --tia --notes` | RO; notice | PASS (vacuous) |
|
||||
| C25 | `pest --tia --flaky` | RO; notice | PASS (vacuous) |
|
||||
| C26 | `pest --tia --issue=123` | RO; notice | PASS (vacuous) |
|
||||
| C27 | `pest --tia --pr=1` | RO; notice | PASS (vacuous) |
|
||||
| C28 | `pest --tia --pull-request=1` | RO; notice | PASS (vacuous) |
|
||||
| C29 | `pest --tia --shard=1/2` | RO; notice; no prune | PASS (sweep) |
|
||||
| C30 | `pest --tia --shard=2/2` | RO; notice; C29+C30 covers the suite; neither prunes | PASS (sweep) |
|
||||
| C31 | `->only()` on `trio one`, `pest --tia` | RO; no notice; siblings untouched | PASS (sweep) |
|
||||
| C32 | `->only()` on `trio one`, plain `pest` | RO; no notice | PASS (sweep) |
|
||||
| C33 | `pest --tia --filtered --filter=adds` | RO; notice; filtered yields to narrowing | PASS (sweep) |
|
||||
| C34 | `PEST_TIA=1 pest --filter=adds` | RO; notice | PASS (sweep) |
|
||||
| C35 | `PEST_TIA_FILTERED=1 pest --filter=adds` | RO; notice (regression guard) | PASS (sweep) |
|
||||
| C36 | `pest --filtered --filter=adds` | RO; notice | PASS (sweep) |
|
||||
| C37 | `pest --tia --ticket=X` | RO; notice. `--ticket` **is** a real option (`src/Plugins/Snapshot.php:83`) — it narrows legitimately | PASS (vacuous) |
|
||||
| C38 | `pest --tia --assignee=X` | RO; notice (`src/Plugins/Snapshot.php:81`) | PASS (vacuous) |
|
||||
| C39 | `pest --tia --todo` | RO; notice; matches nothing | PASS (vacuous) |
|
||||
|
||||
### D — Truncation → RESULTS-ONLY
|
||||
|
||||
D1–D5, D11–D13, D16 precondition: `trio one` broken. D6–D10 run against a green suite so the `--stop-on-*` trigger is the only narrowing.
|
||||
|
||||
| # | Case | Target outcome | Phase 1 |
|
||||
|---|---|---|---|
|
||||
| D1 | `pest --bail` | RO; `trio one` = `status=7`; siblings survive unchanged | PASS (sweep) |
|
||||
| D2 | `pest --retry` | RO; siblings survive | PASS (sweep) |
|
||||
| D3 | `pest --stop-on-failure` | RO | PASS (sweep) |
|
||||
| D4 | `pest --stop-on-defect` | RO | PASS (sweep) |
|
||||
| D5 | `pest --stop-on-error` (no error occurs) | COMPLETE — nothing stopped it | PASS (sweep) |
|
||||
| D6 | `pest --stop-on-warning` | RO when it fires. Warning stored as `status=0`, not `6` | PASS (sweep) |
|
||||
| D7 | `pest --stop-on-risky --disallow-test-output` | RO; `status=5` | PASS (sweep) |
|
||||
| D8 | `pest --stop-on-skipped` | RO; `status=1` | PASS (sweep) |
|
||||
| D9 | `pest --stop-on-incomplete` | RO; `status=2` | PASS (sweep) |
|
||||
| D10 | `pest --stop-on-deprecation` | RO. Deprecation stored as `status=0`, not `4` | PASS (sweep) |
|
||||
| D11 | `pest --tia --bail` | RO | PASS (sweep) |
|
||||
| D12 | `pest --bail --filter=trio` | RO (narrowed *and* truncated) | PASS (sweep) |
|
||||
| D13 | `stopOnFailure="true"` in `phpunit.xml`, plain `pest` | RO — caught via `stoppedEarly()`, not flag matching | PASS (sweep) |
|
||||
| D14 | D13 config, green suite | COMPLETE | PASS (sweep) |
|
||||
| D15 | `pest --bail`, last test in run order fails | RO — over-conservative but safe; nothing was skipped | PASS (sweep) |
|
||||
| D16 | `pest --tia --filtered --bail`, broken affected test | RO | PASS (sweep) |
|
||||
|
||||
### E — Result merge semantics
|
||||
|
||||
| # | Case | Target outcome | Phase 1 |
|
||||
|---|---|---|---|
|
||||
| E1 | cached `flaky`=7, `FLAKY_OK=1 pest --filter=flaky` | flips to `status=0`; all other results byte-identical | PASS (sweep) |
|
||||
| E2 | cached `flaky`=0, `pest --filter=flaky` (env unset) | flips to `status=7` | PASS (sweep) |
|
||||
| E3 | after E1, `pest --tia --filtered` clean | `No affected tests found`; zero graph delta | PASS (sweep) |
|
||||
| E4 | after E2, `pest --tia --filtered` | re-runs, `from 1 previously unsuccessful test` | PASS (sweep) |
|
||||
| E5 | any partial run | `assertions` and `time` update for the test that ran (it executed, so change 9 does not apply) | PASS (sweep) |
|
||||
| E6 | any partial run | `message` of untouched tests unchanged | PASS (sweep) |
|
||||
| E7 | two sequential partial runs on different tests | each updates only its own entry; both persist | PASS (sweep) |
|
||||
| E8 | `pest --filter="trio one"` | siblings keep exact `status`/`time`/`assertions`/`message` | PASS (sweep) |
|
||||
| E9 | partial run of a `->skip()`ed test | `status=1` | PASS (sweep) |
|
||||
| E10 | partial run of a `->todo()` test | `status=1`, `assertions=0`, `message="__TODO__"` | PASS (sweep) |
|
||||
| E11 | partial run of a risky test | `status=5` | PASS (sweep) |
|
||||
| E12 | any partial run | `fingerprint` byte-identical | PASS (sweep) |
|
||||
| E13 | dataset test, `--filter` matching one row | that row updates; other rows survive (the old prune bug) | PASS (sweep) |
|
||||
| E14 | dataset test, full `--tia` after deleting a row | the deleted row is pruned | PASS (sweep) |
|
||||
|
||||
### F — Guard rails
|
||||
|
||||
| # | Case | Target outcome | Phase 1 |
|
||||
|---|---|---|---|
|
||||
| F1 | test absent from `edges`, `pest --filter="brand new"` | no result recorded | VERIFIED |
|
||||
| F2 | complete `pest --tia` first, then the same filter | result is recorded | VERIFIED |
|
||||
| F3 | `pest --mutate --path=app/Services --covered-only` | statuses/`edges`/`sha`/`tree`/`files` identical; only `time` may change | PASS (sweep) |
|
||||
| F4 | `pest --mutate --parallel --path=… --covered-only` | same (observed: zero delta) | PASS (sweep) — recheck under change 7 |
|
||||
| F5 | weakened test so a mutant survives, `pest --mutate` | no status change in the baseline | PASS (sweep) |
|
||||
| F6 | `pest --mutate` with the graph deleted | no graph created | PASS (sweep) |
|
||||
| F7 | mutation subprocess env | carries `PEST_MUTATION_TESTING`; `--mutate` stripped from argv | PASS (sweep) |
|
||||
| F8 | grep baseline after `--mutate` | no mutation-flavoured messages | PASS (sweep) |
|
||||
|
||||
### G — Parallel
|
||||
|
||||
| # | Case | Target outcome | Phase 1 |
|
||||
|---|---|---|---|
|
||||
| G1 | `pest --tia --parallel` (clean) | COMPLETE — graph written; all 7 results present | VERIFIED |
|
||||
| G2 | `pest --tia --parallel --filter=adds` | RO | PASS (sweep) |
|
||||
| G3 | `pest --parallel --bail`, `trio one` broken | RO; siblings survive | PASS (sweep) |
|
||||
| G4 | `pest --tia --parallel --bail` | RO | PASS (sweep) |
|
||||
| G5 | `pest --tia --filtered --parallel` after a source edit | narrows to affected; writes | VERIFIED (see I5) |
|
||||
| G6 | `pest --tia --parallel` | worker results reach the parent baseline | VERIFIED |
|
||||
| G7 | `->only()` + `pest --tia --parallel` | RO | PASS (sweep) |
|
||||
| G8 | `pest --tia --parallel --shard=1/2` | RO | PASS (sweep) |
|
||||
| G9 | broken test in one worker, `pest --parallel --bail` | whole run is RO, not just that worker's slice | PASS (sweep) |
|
||||
| G10 | `pest --parallel --retry` | `InvalidOption`; graph untouched | PASS (sweep) |
|
||||
| G11 | `pest --tia --parallel --fresh` | graph rebuilt | VERIFIED |
|
||||
| G12 | `edges` after `pest --tia` vs `pest --tia --parallel --fresh` | **equivalent edge sets** — `files=22`, self-edge on all 5 tests, both | **VERIFIED (fixed)** — pre-fix: `files=4`, 0 self-edges |
|
||||
|
||||
### H — Baseline key / branch resolution
|
||||
|
||||
| # | Case | Target outcome | Phase 1 |
|
||||
|---|---|---|---|
|
||||
| H1 | `pest --tia` on `master` | only a `master` key | VERIFIED |
|
||||
| H2 | plain `pest` on `master` | writes `master`, not `main` | PASS (sweep) |
|
||||
| H3 | `pest --bail` truncated on `master` | no `main` key minted | PASS (sweep) |
|
||||
| H4 | `pest --filter=adds` on `master` | no `main` key minted | PASS (sweep) |
|
||||
| H5 | `git branch -m main`, full `pest --tia` | single `main` key | PASS (sweep) |
|
||||
| H6 | `git checkout -b feature/x`, `pest --tia` | `feature/x` key appears; reads fall back to the existing baseline | PASS (sweep) |
|
||||
| H7 | detached HEAD, `pest --tia` | falls back to the `main` key for both reads and writes; no `HEAD` key | PASS (sweep) |
|
||||
| H8 | after every C and D case | no baseline key other than the real branch | PASS (sweep) |
|
||||
| H9 | non-git dir, `pest --tia` | `MissingDependency` — `The feature "Tia mode" requires "git".` | PASS (sweep) |
|
||||
| H10 | non-git dir, plain `pest` | runs normally; no baseline dir created | PASS (sweep) |
|
||||
|
||||
### I — Filtered mode
|
||||
|
||||
| # | Case | Target outcome | Phase 1 |
|
||||
|---|---|---|---|
|
||||
| I1 | clean + green, `pest --tia --filtered` | `No affected tests found`; zero graph delta | PASS (sweep) |
|
||||
| I2 | edit `Calculator.php` | only `CalculatorTest` runs | VERIFIED |
|
||||
| I3 | cached failure, clean tree | re-runs it, `from 1 previously unsuccessful test` | PASS (sweep) |
|
||||
| I4 | I2 with sentinels | `tree` updated; exactly one entry rewritten; all others retained | PASS (sweep) |
|
||||
| I5 | `pest --tia --filtered --parallel` | same narrowing as I2 **and `edges` unchanged** | **VERIFIED (fixed)** — 1 affected test, edges identical |
|
||||
| I6 | `PEST_TIA_FILTERED=1 pest --tia` | identical delta to I4 | PASS (sweep) |
|
||||
| I7 | `pest --tia --filtered tests/Unit` | explicit path wins; filtered off; RO; notice | PASS (sweep) |
|
||||
| I8 | `pest --tia --filtered --coverage-text` | filtered mode disabled by an active coverage report → full suite runs | **VERIFIED (fixed)** — pre-fix: `No affected tests found` |
|
||||
| I9 | cached failure whose test file was deleted | WARN `Some cached tests due a re-run could not be located on disk` + `Running the full suite with replay instead of a filtered run` | **VERIFIED (fixed)** — pre-fix: `No tests found`, exit 0 green |
|
||||
| I10 | `pest --tia --filtered` with no baseline yet | records a baseline instead of filtering | PASS (sweep) |
|
||||
| I11 | edit a Blade view a Feature test renders | the Feature test is selected | PASS (sweep) |
|
||||
| I12 | edit `composer.lock` | fingerprint drift → full rebuild, **and the reason is printed sequentially**: `fresh graph (composer.lock changed)` | **VERIFIED (fixed)** — pre-fix: bare `Running in TIA mode.` |
|
||||
|
||||
### J — Interactions & regressions
|
||||
|
||||
| # | Case | Target outcome | Phase 1 |
|
||||
|---|---|---|---|
|
||||
| J1 | `pest --tia --fresh --filter=adds` | graph **not** purged; RO | PASS (sweep) |
|
||||
| J2 | `pest --tia --refetch --filter=adds` | refetch not performed; RO | PASS (sweep) |
|
||||
| J3 | `pest --no-tia --filter=adds` | RO — narrowing wins over `--no-tia`; no notice | PASS (sweep) |
|
||||
| J4 | `pest --tia --no-tia` | TIA disabled; COMPLETE | PASS (sweep) |
|
||||
| J5 | `<groups><exclude><group>integration</group></exclude></groups>`, `pest --tia` | COMPLETE, no notice — an always-in-force filter applies to baseline runs too | PASS (sweep) |
|
||||
| J6 | `pest --tia --compact` | COMPLETE | PASS (sweep) |
|
||||
| J7 | `pest --tia -v` | COMPLETE | PASS (sweep) |
|
||||
| J8 | `pest --tia --profile` | COMPLETE | PASS (sweep) |
|
||||
| J9 | `pest --tia --order-by=random` | COMPLETE | PASS (sweep) |
|
||||
| J10 | `pest --tia --random-order-seed=1234` | COMPLETE | PASS (sweep) |
|
||||
| J11 | `pest --tia --repeat=2` | COMPLETE — repetition is not narrowing | **SKIP** — `--repeat` is not a Pest option |
|
||||
| J12 | `pest --filter=adds` twice | only the executed test's `time` differs; rest byte-identical | PASS (sweep) |
|
||||
| J13 | `pest --tia --min=50` | COMPLETE (silent no-op without `--coverage`) | PASS (sweep) |
|
||||
| J14 | SIGINT mid-suite | RO — an interrupted run is truncated | PASS (sweep) — signal still does not reach the re-exec'd child (not fixed) |
|
||||
| J15 | delete the graph, `pest --tia --filtered` | records a baseline rather than erroring | PASS (sweep) |
|
||||
| J16 | corrupt `graph.json`, `pest --tia` | recovers by rebuilding; no crash | PASS (sweep) |
|
||||
|
||||
### K — New rows for the phase-one fixes
|
||||
|
||||
These assert behaviour no original row covered. All were verified in phase one; re-run them as
|
||||
regression guards.
|
||||
|
||||
| # | Case | Target outcome | Phase 1 |
|
||||
|---|---|---|---|
|
||||
| K1 | healthy `pest --tia` graph, then `pest --tia --coverage` | `edges` **byte-identical** — piggyback data may seed empty sets, never narrow populated ones; `files=22` stays `22` | **VERIFIED (fixed)** — pre-fix: `ExampleTest` 16→2 edges, self-edges lost |
|
||||
| K2 | same run's headline | `Experimental TIA mode enabled / recording a coverage baseline.` — no false `fresh graph` | **VERIFIED (fixed)** |
|
||||
| K3 | `pest --tia` ×3 on a clean tree | replayed entries keep their recorded `time` across all three | **VERIFIED (fixed)** — pre-fix: `0.053 → 0.001 → 0.001 → 0.001` |
|
||||
| K4 | add a test file, one `pest --tia` (graph exists, fingerprint matches → replay+refresh) | result **and** edges appear in that same run; `n` grows by 1 | **VERIFIED** — regression guard for change 9 |
|
||||
| K5 | add a test file, plain `pest --no-tia` | no result written for it; `n` unchanged; no edges | **VERIFIED (fixed)** — pre-fix: edge-less result inflated `n` |
|
||||
| K6 | `pest --tia --parallel` worker argv | carries `-d pcov.directory=<projectRoot>`; `PEST_TIA` unset; not re-exec'd | **VERIFIED** — probe `bin/worker.php` |
|
||||
| K7 | second `pest --tia --coverage` (cache primed → replay) | recorder uses link tracking only, does not clear PHPUnit's data mid-collection; coverage report intact | Not yet measured — **new work** |
|
||||
| K8 | `pest --tia --parallel --coverage` | workers read `TIA_PIGGYBACK_COVERAGE`; no widened pcov scope; report intact | Not yet measured — **new work** |
|
||||
| K9 | `pest --tia --filtered --coverage-html=<dir>` / `--coverage-clover=<file>` | filtered mode off, same as I8, for every flag in `COVERAGE_REPORT_FLAGS` | Not yet measured — **new work** |
|
||||
|
||||
---
|
||||
|
||||
## Part 3 — Priorities for phase two
|
||||
|
||||
0. **Part 1b fixtures** — nothing in C, D6–D10, or E9–E14 means anything until they exist.
|
||||
1. **K7, K8, K9** — the only rows never measured. K7/K8 exercise changes 2 and 3, which were
|
||||
reasoned about but not observed; K9 covers the seven `COVERAGE_REPORT_FLAGS` beyond
|
||||
`--coverage-text`.
|
||||
2. **B2, B3, F4** and all of **D** and **E** — change 9 touched the shared result-write path, and
|
||||
these are the rows that exercise it hardest. `$recordsEdges` is the thing to falsify: it must be
|
||||
false for every partial and every non-recording run.
|
||||
3. **F3–F8** — change 7 injects a `-d` into worker argv; `--mutate --parallel` is the one place that
|
||||
both spawns workers and must write nothing.
|
||||
4. **H1–H10** — untouched by these changes; cheapest bulk confirmation.
|
||||
5. Raw PHPUnit coverage flags print **no report at all** in Pest, with or without TIA
|
||||
(`--no-tia --coverage-text` is equally silent). Pre-existing, unrelated to change 6 — do not
|
||||
chase it as a regression, but it means I8/K9 can only assert the selection half.
|
||||
|
||||
Report per row: tier respected (yes/no), the graph delta under sentinel patching, and for any
|
||||
failure the pre-fix contrast (write `git show db70017c:<file>` into vendor, re-run, restore) so a
|
||||
regression is told apart from a pre-existing defect.
|
||||
@@ -94,6 +94,7 @@
|
||||
"test:inline": "php bin/pest --configuration=phpunit.inline.xml",
|
||||
"test:parallel": "php bin/pest --exclude-group=integration --parallel --processes=3",
|
||||
"test:integration": "php bin/pest --group=integration -v",
|
||||
"test:tia": "php bin/pest --group=tia -v",
|
||||
"update:snapshots": "REBUILD_SNAPSHOTS=true php bin/pest --update-snapshots",
|
||||
"test": [
|
||||
"@test:lint",
|
||||
|
||||
@@ -19,6 +19,6 @@ final class MissingDependency extends InvalidArgumentException implements Except
|
||||
*/
|
||||
public function __construct(string $feature, string $dependency)
|
||||
{
|
||||
parent::__construct(sprintf('The feature "%s" requires "%s".', $feature, $dependency));
|
||||
parent::__construct(sprintf('The feature [%s] requires [%s].', $feature, $dependency));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace Pest\Exceptions;
|
||||
|
||||
use NunoMaduro\Collision\Contracts\RenderlessEditor;
|
||||
use NunoMaduro\Collision\Contracts\RenderlessTrace;
|
||||
use Pest\Contracts\Panicable;
|
||||
use RuntimeException;
|
||||
use Symfony\Component\Console\Exception\ExceptionInterface;
|
||||
use Symfony\Component\Console\Output\OutputInterface;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
*/
|
||||
final class TiaRequiresCommit extends RuntimeException implements ExceptionInterface, Panicable, RenderlessEditor, RenderlessTrace
|
||||
{
|
||||
public function __construct()
|
||||
{
|
||||
parent::__construct(
|
||||
'Tia mode requires a repository with at least one commit, so the baseline it records can be anchored to a revision.',
|
||||
);
|
||||
}
|
||||
|
||||
public function render(OutputInterface $output): void
|
||||
{
|
||||
$output->writeln([
|
||||
'',
|
||||
' <fg=white;options=bold;bg=red> ERROR </> Tia mode requires at least one commit.',
|
||||
'',
|
||||
' A baseline is anchored to the revision it was recorded at, and this repository',
|
||||
' has none yet, so there is nothing to record against and nothing to compare a',
|
||||
' later run to.',
|
||||
'',
|
||||
' Commit once, then run again:',
|
||||
'',
|
||||
' <fg=yellow>git add . && git commit -m "Initial commit"</>',
|
||||
'',
|
||||
' Runs without <fg=yellow>--tia</> are unaffected.',
|
||||
'',
|
||||
]);
|
||||
}
|
||||
|
||||
public function exitCode(): int
|
||||
{
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
+42
-90
@@ -12,6 +12,7 @@ use Pest\Contracts\Plugins\Terminable;
|
||||
use Pest\Exceptions\InvalidOption;
|
||||
use Pest\Exceptions\MissingDependency;
|
||||
use Pest\Exceptions\NoAffectedTestsFound;
|
||||
use Pest\Exceptions\TiaRequiresCommit;
|
||||
use Pest\Exceptions\TiaRequiresDefaultBranch;
|
||||
use Pest\Exceptions\TiaRequiresRemote;
|
||||
use Pest\Exceptions\TiaRequiresRepositoryRoot;
|
||||
@@ -32,13 +33,13 @@ use Pest\Plugins\Tia\Storage;
|
||||
use Pest\Plugins\Tia\TableExtractor;
|
||||
use Pest\Plugins\Tia\WatchPatterns;
|
||||
use Pest\Support\Container;
|
||||
use Pest\Support\Git;
|
||||
use Pest\Support\View;
|
||||
use Pest\TestCaseFilters\TiaTestCaseFilter;
|
||||
use Pest\TestSuite;
|
||||
use PHPUnit\Framework\TestStatus\TestStatus;
|
||||
use PHPUnit\TestRunner\TestResult\Facade as TestResultFacade;
|
||||
use Symfony\Component\Console\Output\OutputInterface;
|
||||
use Symfony\Component\Process\Process;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
@@ -123,18 +124,14 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
'--compact', '--ci-build-id', '--min',
|
||||
];
|
||||
|
||||
/**
|
||||
* @var list<string>
|
||||
*/
|
||||
/** @var list<string> */
|
||||
private const array COVERAGE_REPORT_FLAGS = [
|
||||
'--coverage-clover', '--coverage-cobertura', '--coverage-crap4j',
|
||||
'--coverage-html', '--coverage-openclover', '--coverage-php',
|
||||
'--coverage-text', '--coverage-xml',
|
||||
];
|
||||
|
||||
/**
|
||||
* @var list<string>
|
||||
*/
|
||||
/** @var list<string> */
|
||||
private const array PARTIAL_SELECTION_FLAGS = [
|
||||
'--filter', '--exclude-filter', '--group', '--exclude-group',
|
||||
'--covers', '--uses', '--testsuite', '--exclude-testsuite', '--test-suffix',
|
||||
@@ -159,19 +156,10 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
/** @var array<string, int> */
|
||||
private array $cachedAssertionsByTestId = [];
|
||||
|
||||
/**
|
||||
* The status a replayed test was replayed *as*, so the write-back records
|
||||
* what was cached rather than what the replay looked like from the
|
||||
* outside. A cached deprecation replays as a pass — recording that pass
|
||||
* would erase the deprecation from the baseline on the very next run.
|
||||
*
|
||||
* @var array<string, array{status: int, message: string}>
|
||||
*/
|
||||
/** @var array<string, array{status: int, message: string}> */
|
||||
private array $cachedStatusByTestId = [];
|
||||
|
||||
/**
|
||||
* @var array<string, float>
|
||||
*/
|
||||
/** @var array<string, float> */
|
||||
private array $cachedTimeByTestId = [];
|
||||
|
||||
private ?Graph $replayGraph = null;
|
||||
@@ -210,6 +198,8 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
|
||||
private bool $detachedHead = false;
|
||||
|
||||
private bool $graphUnreachable = false;
|
||||
|
||||
/** @var array<int, string> */
|
||||
private array $originalArguments = [];
|
||||
|
||||
@@ -269,11 +259,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return $graph;
|
||||
}
|
||||
|
||||
/**
|
||||
* Drop a graph that will not decode, so the next run that can record starts
|
||||
* clean instead of tripping over the same file forever — rebuilding needs a
|
||||
* coverage driver, and without one the file would stay corrupt for good.
|
||||
*/
|
||||
private function discardUnreadableGraph(): void
|
||||
{
|
||||
if (Parallel::isWorker()) {
|
||||
@@ -294,16 +279,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$this->renderBadge('WARN', 'The dependency graph could not be read — it will be rebuilt.');
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a state file, unless this checkout may not write.
|
||||
*
|
||||
* A detached HEAD names no branch, so {@see self::saveGraph()} refuses to
|
||||
* write — which means anything deleted here could never be rebuilt from
|
||||
* this checkout. Read-only has to mean deletes too, or a drifted
|
||||
* `composer.lock` on a detached CI checkout wipes the whole team's baseline.
|
||||
*
|
||||
* @return bool Whether the delete happened.
|
||||
*/
|
||||
private function deleteState(string $key): bool
|
||||
{
|
||||
if ($this->detachedHead) {
|
||||
@@ -315,9 +290,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
|
||||
private function saveGraph(Graph $graph): bool
|
||||
{
|
||||
// A detached HEAD names no branch of its own, so `$this->branch` is the
|
||||
// fallback — writing here would land this checkout's results in the
|
||||
// default branch's baseline. Leave the graph exactly as it was.
|
||||
if ($this->detachedHead) {
|
||||
return true;
|
||||
}
|
||||
@@ -577,9 +549,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$this->flushWorkerReplay();
|
||||
}
|
||||
|
||||
// `terminate()` also runs from the shutdown handler, which is how a run
|
||||
// that `exit()`s inside a test gets here — with a test prepared and
|
||||
// never finished, and so with no right to a complete write.
|
||||
if ($this->writesSuppressed || $this->resultsOnlyWrites || $this->hasUnfinishedTest()) {
|
||||
$this->recorder->reset();
|
||||
$this->coverageCollector->reset();
|
||||
@@ -692,7 +661,7 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return $exitCode;
|
||||
}
|
||||
|
||||
if ($this->replayRan) {
|
||||
if ($this->replayRan || $this->graphUnreachable) {
|
||||
$this->bumpRecordedSha();
|
||||
}
|
||||
|
||||
@@ -836,7 +805,17 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
Panic::with(new TiaRequiresRepositoryRoot($subdirectoryPrefix));
|
||||
}
|
||||
|
||||
$this->resolveBranch($projectRoot);
|
||||
try {
|
||||
$this->resolveBranch($projectRoot);
|
||||
} catch (MissingDependency $missingGit) {
|
||||
$repository = new ChangedFiles($projectRoot);
|
||||
|
||||
if ($repository->isRepository() && ! $repository->hasCommits()) {
|
||||
Panic::with(new TiaRequiresCommit);
|
||||
}
|
||||
|
||||
throw $missingGit;
|
||||
}
|
||||
|
||||
if (! $this->fallbackBranchResolved) {
|
||||
Panic::with(new ChangedFiles($projectRoot)->hasRemote()
|
||||
@@ -865,6 +844,7 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
&& $changedFiles->since($branchSha) === null) {
|
||||
$this->renderBadge('WARN', 'Recorded commit is no longer reachable — graph will be rebuilt.');
|
||||
$graph = null;
|
||||
$this->graphUnreachable = true;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -886,8 +866,14 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$this->state->write(self::KEY_COVERAGE_MARKER, '');
|
||||
}
|
||||
|
||||
if (! $graph instanceof Graph && $this->piggybackCoverage) {
|
||||
$this->emitCoverageScopedRecordSkipped();
|
||||
|
||||
return $arguments;
|
||||
}
|
||||
|
||||
if ($coverageCacheOwned && ! $this->state->exists(self::KEY_COVERAGE_CACHE)) {
|
||||
if ($graph instanceof Graph && $this->driftLabel === null) {
|
||||
if ($this->driftLabel === null) {
|
||||
$this->freshGraphReason = 'recording a coverage baseline';
|
||||
}
|
||||
|
||||
@@ -1042,7 +1028,7 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return $arguments;
|
||||
}
|
||||
|
||||
$affectedFromChanges = $changed === [] ? [] : $graph->affected($changed);
|
||||
$affectedFromChanges = $changed === [] ? [] : $graph->testFilesOnDisk($graph->affected($changed));
|
||||
$rerunFromCache = [];
|
||||
|
||||
if ($this->filteredMode && $graph->hasUnlocatedTestsToRerun($this->branch)) {
|
||||
@@ -1295,6 +1281,14 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$this->renderChild('Running in TIA mode, however TIA is skipped as it needs ext-pcov or Xdebug.');
|
||||
}
|
||||
|
||||
private function emitCoverageScopedRecordSkipped(): void
|
||||
{
|
||||
$this->output->writeln('');
|
||||
|
||||
$this->renderChild('Running in TIA mode, however TIA is skipped as an active coverage report narrows the edges it could record.');
|
||||
$this->renderChild('Record the baseline with a plain --tia run first; coverage runs then reuse it.');
|
||||
}
|
||||
|
||||
/**
|
||||
* @param array<string, array<int, string>> $perTestFiles
|
||||
* @param array<string, array<int, string>> $perTestTables
|
||||
@@ -1342,15 +1336,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$this->renderChild('Install / enable pcov or xdebug (mode: coverage) in the worker PHP and rerun.');
|
||||
}
|
||||
|
||||
/**
|
||||
* A parallel run keeps its results in the workers, so the parent's collector
|
||||
* is empty and nothing would ever reach the graph. Ask the workers to flush
|
||||
* what they ran, so a parallel run refreshes — and prunes — exactly like the
|
||||
* sequential run of the same command.
|
||||
*
|
||||
* Gated on a graph already existing: a project that has never run TIA must
|
||||
* not gain a baseline from a plain `--parallel` run.
|
||||
*/
|
||||
private function requestWorkerResults(): void
|
||||
{
|
||||
if (Parallel::isWorker() || ! Parallel::isEnabled() || $this->writesSuppressed) {
|
||||
@@ -1722,16 +1707,8 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$collector->reset();
|
||||
}
|
||||
|
||||
/**
|
||||
* Give back what the graph no longer needs. Only ever called from a
|
||||
* complete write — the RESULTS-ONLY and HARD-SUPPRESSED tiers may not
|
||||
* remove an entry, and a narrowed run has not seen enough to judge.
|
||||
*/
|
||||
private function reclaim(Graph $graph): void
|
||||
{
|
||||
// The fallback branch never layers under itself, so marking it would
|
||||
// write the graph for no reader's benefit — and cost a clean green run
|
||||
// its "wrote nothing at all".
|
||||
if ($this->branch !== $this->fallbackBranch) {
|
||||
$graph->markBaselineComplete($this->branch);
|
||||
}
|
||||
@@ -1745,10 +1722,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return;
|
||||
}
|
||||
|
||||
// A shallow, single-branch CI checkout can see almost no refs, and
|
||||
// "git has never heard of it" would then mean "this clone is narrow",
|
||||
// not "that branch is gone". Only reclaim from a checkout that can at
|
||||
// least see the branch everything else falls back to.
|
||||
if (! in_array($this->fallbackBranch, $branches, true)) {
|
||||
return;
|
||||
}
|
||||
@@ -1916,10 +1889,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return TestResultFacade::shouldStop();
|
||||
}
|
||||
|
||||
/**
|
||||
* A test that was prepared and never finished means this process is being
|
||||
* torn down mid-file, so it has not seen enough of that file to prune it.
|
||||
*/
|
||||
private function hasUnfinishedTest(): bool
|
||||
{
|
||||
$collector = Container::getInstance()->get(ResultCollector::class);
|
||||
@@ -2035,13 +2004,7 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
}
|
||||
}
|
||||
|
||||
foreach ($testPaths as $testPath) {
|
||||
if (str_starts_with($testPath, $candidate.DIRECTORY_SEPARATOR)) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
return array_all($testPaths, fn (string $testPath): bool => ! str_starts_with($testPath, $candidate.DIRECTORY_SEPARATOR));
|
||||
}
|
||||
|
||||
private function resolveArgumentPath(string $arg, string $projectRoot): ?string
|
||||
@@ -2173,16 +2136,7 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
*/
|
||||
private function gitSubdirectoryPrefix(string $projectRoot): ?string
|
||||
{
|
||||
$process = new Process(['git', 'rev-parse', '--show-prefix'], $projectRoot);
|
||||
$process->run();
|
||||
|
||||
if (! $process->isSuccessful()) {
|
||||
return null;
|
||||
}
|
||||
|
||||
$prefix = trim($process->getOutput());
|
||||
|
||||
return $prefix === '' ? null : rtrim(str_replace(DIRECTORY_SEPARATOR, '/', $prefix), '/');
|
||||
return new Git($projectRoot)->subdirectoryPrefix();
|
||||
}
|
||||
|
||||
private function composerLockDelta(string $projectRoot, string $sha): string
|
||||
@@ -2192,15 +2146,13 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return '';
|
||||
}
|
||||
|
||||
$process = new Process(['git', 'show', $sha.':composer.lock'], $projectRoot);
|
||||
$process->setTimeout(5.0);
|
||||
$process->run();
|
||||
$baseline = new Git($projectRoot)->show($sha, 'composer.lock');
|
||||
|
||||
if (! $process->isSuccessful()) {
|
||||
if ($baseline === null) {
|
||||
return '';
|
||||
}
|
||||
|
||||
$oldVersions = $this->lockVersions($process->getOutput());
|
||||
$oldVersions = $this->lockVersions($baseline);
|
||||
$newVersions = $this->lockVersions($current);
|
||||
|
||||
if ($oldVersions === [] && $newVersions === []) {
|
||||
|
||||
@@ -5,14 +5,19 @@ declare(strict_types=1);
|
||||
namespace Pest\Plugins\Tia;
|
||||
|
||||
use Pest\Exceptions\MissingDependency;
|
||||
use Symfony\Component\Process\Process;
|
||||
use Pest\Support\Git;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
*/
|
||||
final readonly class ChangedFiles
|
||||
{
|
||||
public function __construct(private string $projectRoot) {}
|
||||
private Git $git;
|
||||
|
||||
public function __construct(private string $projectRoot)
|
||||
{
|
||||
$this->git = new Git($projectRoot);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param array<int, string> $files project-relative paths.
|
||||
@@ -155,15 +160,7 @@ final readonly class ChangedFiles
|
||||
|
||||
private function contentAtSha(string $sha, string $path): ?string
|
||||
{
|
||||
$process = new Process(['git', 'show', $sha.':'.$path], $this->projectRoot);
|
||||
$process->setTimeout(5.0);
|
||||
$process->run();
|
||||
|
||||
if (! $process->isSuccessful()) {
|
||||
return null;
|
||||
}
|
||||
|
||||
return $process->getOutput();
|
||||
return $this->git->show($sha, $path);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -176,21 +173,17 @@ final readonly class ChangedFiles
|
||||
return $candidates;
|
||||
}
|
||||
|
||||
$process = new Process(
|
||||
['git', 'check-ignore', '--no-index', '-z', '--stdin'],
|
||||
$this->projectRoot,
|
||||
$result = $this->git->result(
|
||||
['check-ignore', '--no-index', '-z', '--stdin'],
|
||||
implode("\x00", array_keys($candidates)),
|
||||
);
|
||||
$process->setTimeout(5.0);
|
||||
$process->setInput(implode("\x00", array_keys($candidates)));
|
||||
$process->run();
|
||||
|
||||
$exitCode = $process->getExitCode();
|
||||
|
||||
if ($exitCode !== 0 && $exitCode !== 1) {
|
||||
// `check-ignore` exits 1 when nothing matched — that is not a failure.
|
||||
if ($result['exitCode'] !== 0 && $result['exitCode'] !== 1) {
|
||||
throw new MissingDependency('Tia mode', 'git');
|
||||
}
|
||||
|
||||
$output = $process->getOutput();
|
||||
$output = $result['output'];
|
||||
|
||||
if ($output === '') {
|
||||
return $candidates;
|
||||
@@ -207,21 +200,20 @@ final readonly class ChangedFiles
|
||||
|
||||
public function currentBranch(): ?string
|
||||
{
|
||||
$process = new Process(['git', 'rev-parse', '--abbrev-ref', 'HEAD'], $this->projectRoot);
|
||||
$process->run();
|
||||
$output = $this->git->raw(['rev-parse', '--abbrev-ref', 'HEAD']);
|
||||
|
||||
if (! $process->isSuccessful()) {
|
||||
if ($output === null) {
|
||||
throw new MissingDependency('Tia mode', 'git');
|
||||
}
|
||||
|
||||
$branch = trim($process->getOutput());
|
||||
$branch = trim($output);
|
||||
|
||||
return $branch === '' || $branch === 'HEAD' ? null : $branch;
|
||||
}
|
||||
|
||||
public function defaultBranch(): ?string
|
||||
{
|
||||
$head = $this->gitOutput(['git', 'symbolic-ref', '--short', 'refs/remotes/origin/HEAD']);
|
||||
$head = $this->git->output(['symbolic-ref', '--short', 'refs/remotes/origin/HEAD']);
|
||||
|
||||
if ($head !== null) {
|
||||
$branch = preg_replace('#^origin/#', '', $head);
|
||||
@@ -231,41 +223,32 @@ final readonly class ChangedFiles
|
||||
}
|
||||
}
|
||||
|
||||
$configured = $this->gitOutput(['git', 'config', '--get', 'init.defaultBranch']);
|
||||
$configured = $this->git->output(['config', '--get', 'init.defaultBranch']);
|
||||
|
||||
if ($configured === null) {
|
||||
return null;
|
||||
}
|
||||
|
||||
$exists = $this->gitOutput(['git', 'rev-parse', '--verify', '--quiet', 'refs/heads/'.$configured]) !== null
|
||||
|| $this->gitOutput(['git', 'rev-parse', '--verify', '--quiet', 'refs/remotes/origin/'.$configured]) !== null;
|
||||
$exists = $this->git->hasRef('refs/heads/'.$configured)
|
||||
|| $this->git->hasRef('refs/remotes/origin/'.$configured);
|
||||
|
||||
return $exists ? $configured : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Every branch name this checkout knows, local and remote alike. Remotes
|
||||
* count: a branch that only lives on the origin is still a branch someone
|
||||
* will check out, and its baseline must survive.
|
||||
*
|
||||
* @return list<string>|null `null` when git cannot answer.
|
||||
* @return list<string>|null
|
||||
*/
|
||||
public function branchNames(): ?array
|
||||
{
|
||||
$process = new Process(
|
||||
['git', 'for-each-ref', '--format=%(refname)', 'refs/heads', 'refs/remotes'],
|
||||
$this->projectRoot,
|
||||
);
|
||||
$process->setTimeout(5.0);
|
||||
$process->run();
|
||||
$output = $this->git->raw(['for-each-ref', '--format=%(refname)', 'refs/heads', 'refs/remotes']);
|
||||
|
||||
if (! $process->isSuccessful()) {
|
||||
if ($output === null) {
|
||||
return null;
|
||||
}
|
||||
|
||||
$names = [];
|
||||
|
||||
foreach ($this->splitLines($process->getOutput()) as $ref) {
|
||||
foreach ($this->splitLines($output) as $ref) {
|
||||
if (str_starts_with($ref, 'refs/heads/')) {
|
||||
$names[substr($ref, strlen('refs/heads/'))] = true;
|
||||
|
||||
@@ -295,36 +278,31 @@ final readonly class ChangedFiles
|
||||
|
||||
public function hasRemote(): bool
|
||||
{
|
||||
return $this->gitOutput(['git', 'remote']) !== null;
|
||||
return $this->git->hasRemote();
|
||||
}
|
||||
|
||||
public function isRepository(): bool
|
||||
{
|
||||
return $this->git->isRepository();
|
||||
}
|
||||
|
||||
public function hasCommits(): bool
|
||||
{
|
||||
return $this->git->hasCommits();
|
||||
}
|
||||
|
||||
/**
|
||||
* @param array<int, string> $command
|
||||
* Working-tree scans get a longer leash than metadata queries — on a large
|
||||
* repository with a cold cache, `status` and `diff` are not instant.
|
||||
*/
|
||||
private function gitOutput(array $command): ?string
|
||||
private function scan(): Git
|
||||
{
|
||||
$process = new Process($command, $this->projectRoot);
|
||||
$process->setTimeout(5.0);
|
||||
$process->run();
|
||||
|
||||
if (! $process->isSuccessful()) {
|
||||
return null;
|
||||
}
|
||||
|
||||
$output = trim($process->getOutput());
|
||||
|
||||
return $output === '' ? null : $output;
|
||||
return $this->git->withTimeout(60.0);
|
||||
}
|
||||
|
||||
private function shaIsReachable(string $sha): bool
|
||||
{
|
||||
$process = new Process(
|
||||
['git', 'merge-base', '--is-ancestor', $sha, 'HEAD'],
|
||||
$this->projectRoot,
|
||||
);
|
||||
$process->run();
|
||||
|
||||
return $process->getExitCode() === 0;
|
||||
return $this->git->succeeds(['merge-base', '--is-ancestor', $sha, 'HEAD']);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -332,17 +310,13 @@ final readonly class ChangedFiles
|
||||
*/
|
||||
private function diffSinceSha(string $sha): array
|
||||
{
|
||||
$process = new Process(
|
||||
['git', 'diff', '--name-only', $sha.'..HEAD'],
|
||||
$this->projectRoot,
|
||||
);
|
||||
$process->run();
|
||||
$output = $this->scan()->raw(['diff', '--name-only', '--no-renames', $sha.'..HEAD']);
|
||||
|
||||
if (! $process->isSuccessful()) {
|
||||
if ($output === null) {
|
||||
throw new MissingDependency('Tia mode', 'git');
|
||||
}
|
||||
|
||||
return $this->splitLines($process->getOutput());
|
||||
return $this->splitLines($output);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -350,18 +324,12 @@ final readonly class ChangedFiles
|
||||
*/
|
||||
private function workingTreeChanges(): array
|
||||
{
|
||||
$process = new Process(
|
||||
['git', 'status', '--porcelain', '-z', '--untracked-files=all'],
|
||||
$this->projectRoot,
|
||||
);
|
||||
$process->run();
|
||||
$output = $this->scan()->raw(['status', '--porcelain', '-z', '--untracked-files=all']);
|
||||
|
||||
if (! $process->isSuccessful()) {
|
||||
if ($output === null) {
|
||||
throw new MissingDependency('Tia mode', 'git');
|
||||
}
|
||||
|
||||
$output = $process->getOutput();
|
||||
|
||||
if ($output === '') {
|
||||
return [];
|
||||
}
|
||||
@@ -399,14 +367,13 @@ final readonly class ChangedFiles
|
||||
|
||||
public function currentSha(): ?string
|
||||
{
|
||||
$process = new Process(['git', 'rev-parse', 'HEAD'], $this->projectRoot);
|
||||
$process->run();
|
||||
$output = $this->git->raw(['rev-parse', 'HEAD']);
|
||||
|
||||
if (! $process->isSuccessful()) {
|
||||
if ($output === null) {
|
||||
throw new MissingDependency('Tia mode', 'git');
|
||||
}
|
||||
|
||||
$sha = trim($process->getOutput());
|
||||
$sha = trim($output);
|
||||
|
||||
return $sha === '' ? null : $sha;
|
||||
}
|
||||
|
||||
@@ -4,56 +4,31 @@ declare(strict_types=1);
|
||||
|
||||
namespace Pest\Plugins\Tia;
|
||||
|
||||
use Pest\Plugins\Tia\Contracts\Ci;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
*/
|
||||
final class CiDefaultBranch
|
||||
{
|
||||
/**
|
||||
* @var array<int, class-string<Ci>>
|
||||
*/
|
||||
private const array CIS = [
|
||||
Cis\GitLab::class,
|
||||
Cis\GitHub::class,
|
||||
];
|
||||
|
||||
public static function detect(): ?string
|
||||
{
|
||||
return self::fromGitLab() ?? self::fromGitHubEvent();
|
||||
}
|
||||
foreach (self::CIS as $class) {
|
||||
$branch = (new $class)->defaultBranch();
|
||||
|
||||
private static function fromGitLab(): ?string
|
||||
{
|
||||
return self::environment('CI_DEFAULT_BRANCH');
|
||||
}
|
||||
|
||||
private static function fromGitHubEvent(): ?string
|
||||
{
|
||||
$path = self::environment('GITHUB_EVENT_PATH');
|
||||
|
||||
if ($path === null || ! is_file($path) || ! is_readable($path)) {
|
||||
return null;
|
||||
if ($branch !== null) {
|
||||
return $branch;
|
||||
}
|
||||
}
|
||||
|
||||
$contents = @file_get_contents($path);
|
||||
|
||||
if ($contents === false) {
|
||||
return null;
|
||||
}
|
||||
|
||||
$payload = json_decode($contents, true);
|
||||
|
||||
if (! is_array($payload) || ! is_array($payload['repository'] ?? null)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
$branch = $payload['repository']['default_branch'] ?? null;
|
||||
|
||||
return is_string($branch) && $branch !== '' ? $branch : null;
|
||||
}
|
||||
|
||||
private static function environment(string $name): ?string
|
||||
{
|
||||
$value = getenv($name);
|
||||
|
||||
if (! is_string($value)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
$value = trim($value);
|
||||
|
||||
return $value === '' ? null : $value;
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace Pest\Plugins\Tia\Cis\Concerns;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
*/
|
||||
trait ReadsEnvironment
|
||||
{
|
||||
private function environment(string $name): ?string
|
||||
{
|
||||
$value = getenv($name);
|
||||
|
||||
if (! is_string($value)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
$value = trim($value);
|
||||
|
||||
return $value === '' ? null : $value;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace Pest\Plugins\Tia\Cis;
|
||||
|
||||
use Pest\Plugins\Tia\Cis\Concerns\ReadsEnvironment;
|
||||
use Pest\Plugins\Tia\Contracts\Ci;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
*/
|
||||
final readonly class GitHub implements Ci
|
||||
{
|
||||
use ReadsEnvironment;
|
||||
|
||||
public function defaultBranch(): ?string
|
||||
{
|
||||
$path = $this->environment('GITHUB_EVENT_PATH');
|
||||
|
||||
if ($path === null || ! is_file($path) || ! is_readable($path)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
$contents = @file_get_contents($path);
|
||||
|
||||
if ($contents === false) {
|
||||
return null;
|
||||
}
|
||||
|
||||
$payload = json_decode($contents, true);
|
||||
|
||||
if (! is_array($payload) || ! is_array($payload['repository'] ?? null)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
$branch = $payload['repository']['default_branch'] ?? null;
|
||||
|
||||
return is_string($branch) && $branch !== '' ? $branch : null;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace Pest\Plugins\Tia\Cis;
|
||||
|
||||
use Pest\Plugins\Tia\Cis\Concerns\ReadsEnvironment;
|
||||
use Pest\Plugins\Tia\Contracts\Ci;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
*/
|
||||
final readonly class GitLab implements Ci
|
||||
{
|
||||
use ReadsEnvironment;
|
||||
|
||||
public function defaultBranch(): ?string
|
||||
{
|
||||
return $this->environment('CI_DEFAULT_BRANCH');
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace Pest\Plugins\Tia\Contracts;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
*/
|
||||
interface Ci
|
||||
{
|
||||
/**
|
||||
* The default branch advertised by this CI, or `null` when the run
|
||||
* does not happen on it — or when it exposes no such information.
|
||||
*/
|
||||
public function defaultBranch(): ?string;
|
||||
}
|
||||
@@ -29,10 +29,6 @@ enum ReplayType
|
||||
$status->isRisky() => self::Risky,
|
||||
$status->isSkipped() => self::Skipped,
|
||||
$status->isIncomplete() => self::Incomplete,
|
||||
// A recorded notice, deprecation or warning only reaches replay when
|
||||
// the configured failOn* / displayDetailsOn* policies say it is not
|
||||
// worth re-running — which means the test passed. Folding it into
|
||||
// Failure below would turn a green run red on cache alone.
|
||||
$status->isNotice(), $status->isDeprecation(), $status->isWarning() => self::Pass,
|
||||
$status->isFailure(), $status->isError() => self::Failure,
|
||||
default => self::None,
|
||||
|
||||
+27
-59
@@ -119,6 +119,24 @@ final class Graph
|
||||
return array_keys($affectedSet);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param array<int, string> $testFiles Project-relative paths.
|
||||
* @return list<string>
|
||||
*/
|
||||
public function testFilesOnDisk(array $testFiles): array
|
||||
{
|
||||
$root = rtrim($this->projectRoot, DIRECTORY_SEPARATOR).DIRECTORY_SEPARATOR;
|
||||
$onDisk = [];
|
||||
|
||||
foreach ($testFiles as $testFile) {
|
||||
if (is_file($root.$testFile)) {
|
||||
$onDisk[] = $testFile;
|
||||
}
|
||||
}
|
||||
|
||||
return $onDisk;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param array<int, string> $changedFiles
|
||||
* @return array{0: list<string>, 1: list<string>}
|
||||
@@ -651,9 +669,6 @@ final class Graph
|
||||
|
||||
$r = $baseline['results'][$testId];
|
||||
|
||||
// A status this build does not know — a graph written by a newer Pest,
|
||||
// or a corrupt one — is not a result. Returning null re-executes the
|
||||
// test rather than replaying an outcome nobody can interpret.
|
||||
return match ($r['status']) {
|
||||
0 => TestStatus::success(),
|
||||
1 => TestStatus::skipped($r['message']),
|
||||
@@ -691,8 +706,6 @@ final class Graph
|
||||
|
||||
$rel = $this->relative($file);
|
||||
|
||||
// A test file that is no longer on disk cannot be re-run by anyone,
|
||||
// so selecting it would only widen the run for nothing.
|
||||
if ($rel !== null && is_file($this->projectRoot.'/'.$rel)) {
|
||||
$files[$rel] = true;
|
||||
}
|
||||
@@ -701,15 +714,6 @@ final class Graph
|
||||
return array_keys($files);
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a cached result due a re-run names a test file this project
|
||||
* cannot address — an empty path, or one that resolves outside the project
|
||||
* root. Those are genuinely lost, so the caller widens to the full suite.
|
||||
*
|
||||
* A path that resolves fine but is simply absent is *deleted*, not lost:
|
||||
* widening would not run it either, and treating it as unlocated used to
|
||||
* strand `--filtered` on a full replay for good.
|
||||
*/
|
||||
public function hasUnlocatedTestsToRerun(string $branch, ?string $fallbackBranch = null): bool
|
||||
{
|
||||
$baseline = $this->baselineFor($branch, $fallbackBranch);
|
||||
@@ -826,24 +830,6 @@ final class Graph
|
||||
}
|
||||
|
||||
/**
|
||||
* The baseline a read sees for this branch: its own entries layered over the
|
||||
* default branch's, so a key minted by a narrowed run — which only holds the
|
||||
* handful of tests that ran — does not shadow the fallback for everything else.
|
||||
*
|
||||
* Once this branch has had a complete run, the layering becomes per *file*
|
||||
* rather than per test id: the branch's entries for a file it executed are
|
||||
* the whole truth, so the fallback's entries for that same file are dropped
|
||||
* rather than merged. Without that, a test the branch renamed or removed —
|
||||
* and {@see self::pruneStaleResults()} therefore unset — is resurrected by
|
||||
* the default branch on the very next read, and never stops coming back.
|
||||
*
|
||||
* A branch whose key was minted by a *narrowed* run holds only the handful
|
||||
* of tests that ran, and has no business speaking for the rest of their
|
||||
* file, so it keeps the per-test-id merge.
|
||||
*
|
||||
* Read-only: the layering never reaches `$this->baselines`, so writes stay on
|
||||
* the branch that ran.
|
||||
*
|
||||
* @return array{sha: ?string, tree: array<string, string>, complete?: bool, results: array<string, array{status: int, message: string, time: float, assertions?: int, file?: string}>}
|
||||
*/
|
||||
private function baselineFor(string $branch, ?string $fallbackBranch): array
|
||||
@@ -1486,12 +1472,6 @@ final class Graph
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Record that this branch has run the whole suite at least once, which is
|
||||
* what lets {@see self::baselineFor()} treat its entries as authoritative
|
||||
* for the files they cover. Never mints a key: a run that recorded nothing
|
||||
* has nothing to be authoritative about.
|
||||
*/
|
||||
public function markBaselineComplete(string $branch): void
|
||||
{
|
||||
if (isset($this->baselines[$branch])) {
|
||||
@@ -1499,12 +1479,6 @@ final class Graph
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Drop this branch's result entries whose test file is no longer on disk.
|
||||
*
|
||||
* Without this nothing but `--fresh` ever reclaims them, and a *failing*
|
||||
* one keeps `--filtered` widened to a full replay on every later run.
|
||||
*/
|
||||
public function pruneResultsForMissingFiles(string $branch): void
|
||||
{
|
||||
if (! isset($this->baselines[$branch]['results'])) {
|
||||
@@ -1515,14 +1489,18 @@ final class Graph
|
||||
|
||||
foreach ($this->baselines[$branch]['results'] as $testId => $result) {
|
||||
$file = $result['file'] ?? null;
|
||||
|
||||
if (! is_string($file) || $file === '') {
|
||||
if (! is_string($file)) {
|
||||
continue;
|
||||
}
|
||||
if ($file === '') {
|
||||
continue;
|
||||
}
|
||||
|
||||
$rel = $this->relative($file);
|
||||
|
||||
if ($rel === null || is_file($root.$rel)) {
|
||||
if ($rel === null) {
|
||||
continue;
|
||||
}
|
||||
if (is_file($root.$rel)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -1531,10 +1509,7 @@ final class Graph
|
||||
}
|
||||
|
||||
/**
|
||||
* Drop baselines for branches git no longer knows, so the graph does not
|
||||
* carry one full copy of the suite per branch ever created.
|
||||
*
|
||||
* @param array<int, string> $keep Branch names that must survive.
|
||||
* @param array<int, string> $keep
|
||||
*/
|
||||
public function pruneMissingBranches(array $keep): void
|
||||
{
|
||||
@@ -1692,11 +1667,6 @@ final class Graph
|
||||
}
|
||||
|
||||
/**
|
||||
* A graph is state on disk that any process may have written: a newer Pest,
|
||||
* a half-finished write, a hand edit. Every branch, every entry and every
|
||||
* field is checked here so that a malformed one is dropped rather than
|
||||
* reaching a read path and taking the run down with it.
|
||||
*
|
||||
* @return array<string, array{sha: ?string, tree: array<string, string>, complete?: bool, results: array<string, array{status: int, message: string, time: float, assertions?: int, file?: string}>}>
|
||||
*/
|
||||
private static function decodeBaselines(mixed $section): array
|
||||
@@ -1708,8 +1678,6 @@ final class Graph
|
||||
$baselines = [];
|
||||
|
||||
foreach ($section as $key => $baseline) {
|
||||
// A branch named `12345` decodes as an integer key, and must not be
|
||||
// mistaken for a malformed one.
|
||||
$branch = (string) $key;
|
||||
|
||||
if ($branch === '') {
|
||||
|
||||
@@ -360,7 +360,9 @@ final class Recorder
|
||||
}
|
||||
|
||||
$lineKeys = array_keys($lines);
|
||||
if ($lineKeys !== [] && count($covered) === 1 && $covered[0] === max($lineKeys)) {
|
||||
$reportsUnexecutedLines = count($covered) < count($lines);
|
||||
|
||||
if ($reportsUnexecutedLines && $lineKeys !== [] && count($covered) === 1 && $covered[0] === max($lineKeys)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
|
||||
@@ -38,11 +38,6 @@ final class ResultCollector
|
||||
return;
|
||||
}
|
||||
|
||||
// PHPUnit reports a test that triggered a notice, deprecation or
|
||||
// warning as passed, and emits Passed for it. Recording success here
|
||||
// would erase the issue from the baseline, and a later replay under
|
||||
// --fail-on-deprecation (and friends) would come back green where a
|
||||
// fresh run fails. Keep the issue; only refresh what it cannot know.
|
||||
if (isset($this->triggered[$this->currentTestId])) {
|
||||
$this->refreshTime();
|
||||
|
||||
@@ -120,12 +115,6 @@ final class ResultCollector
|
||||
return $this->results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a test was prepared but never finished — the process is being
|
||||
* torn down in the middle of it (an `exit()` inside a test, a killed
|
||||
* worker). What it collected is therefore a partial view of that test
|
||||
* file, and must not license pruning the siblings it never reached.
|
||||
*/
|
||||
public function hasUnfinishedTest(): bool
|
||||
{
|
||||
return $this->currentTestId !== null;
|
||||
@@ -164,11 +153,6 @@ final class ResultCollector
|
||||
$this->startTime = null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Record an issue raised while the test was running. The most important
|
||||
* one wins, exactly as PHPUnit ranks them, so a deprecation does not
|
||||
* shadow the warning that followed it — or the failure.
|
||||
*/
|
||||
private function recordIssue(TestStatus $status): void
|
||||
{
|
||||
if ($this->currentTestId === null) {
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace Pest\Support;
|
||||
|
||||
use Symfony\Component\Process\Process;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
*/
|
||||
final readonly class Git
|
||||
{
|
||||
private const float TIMEOUT = 5.0;
|
||||
|
||||
public function __construct(
|
||||
private ?string $directory = null,
|
||||
private float $timeout = self::TIMEOUT,
|
||||
) {}
|
||||
|
||||
public function withTimeout(float $timeout): self
|
||||
{
|
||||
return new self($this->directory, $timeout);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param array<int, string> $arguments
|
||||
*/
|
||||
public function raw(array $arguments): ?string
|
||||
{
|
||||
$result = $this->result($arguments);
|
||||
|
||||
return $result['exitCode'] === 0 ? $result['output'] : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param array<int, string> $arguments
|
||||
*/
|
||||
public function output(array $arguments): ?string
|
||||
{
|
||||
$output = $this->raw($arguments);
|
||||
|
||||
if ($output === null) {
|
||||
return null;
|
||||
}
|
||||
|
||||
$output = trim($output);
|
||||
|
||||
return $output === '' ? null : $output;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param array<int, string> $arguments
|
||||
*/
|
||||
public function succeeds(array $arguments): bool
|
||||
{
|
||||
return $this->result($arguments)['exitCode'] === 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param array<int, string> $arguments
|
||||
* @return array{exitCode: int, output: string}
|
||||
*/
|
||||
public function result(array $arguments, ?string $input = null): array
|
||||
{
|
||||
$process = new Process(['git', ...$arguments], $this->directory);
|
||||
$process->setTimeout($this->timeout);
|
||||
|
||||
if ($input !== null) {
|
||||
$process->setInput($input);
|
||||
}
|
||||
|
||||
$process->run();
|
||||
|
||||
return [
|
||||
'exitCode' => $process->getExitCode() ?? 1,
|
||||
'output' => $process->getOutput(),
|
||||
];
|
||||
}
|
||||
|
||||
public function isRepository(): bool
|
||||
{
|
||||
return $this->succeeds(['rev-parse', '--git-dir']);
|
||||
}
|
||||
|
||||
public function hasCommits(): bool
|
||||
{
|
||||
return $this->succeeds(['rev-parse', '--verify', '--quiet', 'HEAD']);
|
||||
}
|
||||
|
||||
public function hasRemote(): bool
|
||||
{
|
||||
return $this->output(['remote']) !== null;
|
||||
}
|
||||
|
||||
public function hasRef(string $ref): bool
|
||||
{
|
||||
return $this->output(['rev-parse', '--verify', '--quiet', $ref]) !== null;
|
||||
}
|
||||
|
||||
public function show(string $sha, string $path): ?string
|
||||
{
|
||||
return $this->raw(['show', $sha.':'.$path]);
|
||||
}
|
||||
|
||||
public function subdirectoryPrefix(): ?string
|
||||
{
|
||||
$prefix = $this->output(['rev-parse', '--show-prefix']);
|
||||
|
||||
if ($prefix === null) {
|
||||
return null;
|
||||
}
|
||||
|
||||
return rtrim(str_replace(DIRECTORY_SEPARATOR, '/', $prefix), '/');
|
||||
}
|
||||
}
|
||||
@@ -8,8 +8,8 @@ use Pest\Contracts\TestCaseFilter;
|
||||
use Pest\Exceptions\MissingDependency;
|
||||
use Pest\Exceptions\NoDirtyTestsFound;
|
||||
use Pest\Panic;
|
||||
use Pest\Support\Git;
|
||||
use Pest\TestSuite;
|
||||
use Symfony\Component\Process\Process;
|
||||
|
||||
final class GitDirtyTestCaseFilter implements TestCaseFilter
|
||||
{
|
||||
@@ -52,14 +52,13 @@ final class GitDirtyTestCaseFilter implements TestCaseFilter
|
||||
*/
|
||||
private function loadChangedFiles(): void
|
||||
{
|
||||
$process = new Process(['git', 'status', '--short', '--', '*.php']);
|
||||
$process->run();
|
||||
$status = new Git(timeout: 60.0)->raw(['status', '--short', '--', '*.php']);
|
||||
|
||||
if (! $process->isSuccessful()) {
|
||||
if ($status === null) {
|
||||
throw new MissingDependency('Filter by dirty files', 'git');
|
||||
}
|
||||
|
||||
$output = preg_split('/\R+/', $process->getOutput(), flags: PREG_SPLIT_NO_EMPTY);
|
||||
$output = preg_split('/\R+/', $status, flags: PREG_SPLIT_NO_EMPTY);
|
||||
assert(is_array($output));
|
||||
|
||||
$dirtyFiles = [];
|
||||
|
||||
@@ -1583,6 +1583,10 @@
|
||||
✓ a narrowed run does not reclaim anything
|
||||
✓ a detached HEAD does not reclaim anything either
|
||||
✓ the default branch baseline survives every branch that comes and goes
|
||||
✓ a project below the git repository root refuses to run and writes nothing with dataset "sequential"
|
||||
✓ a project below the git repository root refuses to run and writes nothing with dataset "parallel"
|
||||
✓ a repository with no commits says so, and leaves plain runs alone
|
||||
✓ a directory with no repository at all still asks for git
|
||||
|
||||
PASS Tests\Features\Tia\CompleteRunWriteTier
|
||||
✓ a complete run prunes a deleted test with dataset "sequential"
|
||||
@@ -1603,6 +1607,13 @@
|
||||
✓ a test edit narrows to the affected file and replays the rest
|
||||
✓ a parallel run merges worker results into the parent baseline
|
||||
|
||||
PASS Tests\Features\Tia\CoveragePiggyback
|
||||
✓ a coverage report does not found a dependency graph with dataset "pest coverage"
|
||||
✓ a coverage report does not found a dependency graph with dataset "phpunit coverage report"
|
||||
✓ a coverage report does not found a dependency graph with dataset "parallel"
|
||||
✓ a plain run after a coverage run records the whole project scope
|
||||
✓ a coverage report leaves the edges of an existing graph alone
|
||||
|
||||
PASS Tests\Features\Tia\DefaultBranchReplay
|
||||
✓ replays the default branch baseline on a new branch
|
||||
✓ replays whatever the default branch is called with ('main')
|
||||
@@ -1710,6 +1721,32 @@
|
||||
✓ a shard is a partial run
|
||||
✓ a parallel partial run records the test that ran, like a sequential one
|
||||
|
||||
PASS Tests\Features\Tia\RemoteBaseline
|
||||
✓ a published baseline is fetched instead of recorded locally
|
||||
✓ a fetched baseline that will not decode is discarded rather than trusted
|
||||
✓ a fetched baseline recorded against another tree is not used
|
||||
✓ an artifact without a graph in it fails loudly
|
||||
✓ a baseline that cannot be authenticated for fails loudly
|
||||
✓ a workflow or artifact that is not there fails loudly
|
||||
✓ a network failure warns and lets the suite run with dataset "querying the runs"
|
||||
✓ a network failure warns and lets the suite run with dataset "downloading the artifact"
|
||||
✓ no published baseline yet starts a cooldown, and a corrupt cooldown does not break the run
|
||||
|
||||
PASS Tests\Features\Tia\SelectionPaths
|
||||
✓ a committed rename selects the tests that depended on the old path with dataset "sequential"
|
||||
✓ a committed rename selects the tests that depended on the old path with dataset "parallel"
|
||||
✓ an affected test file that is gone does not strand a filtered run with dataset "sequential"
|
||||
✓ an affected test file that is gone does not strand a filtered run with dataset "parallel"
|
||||
✓ a plain run reclaims the edge of a test file that is gone
|
||||
✓ a changed view selects the test that rendered it
|
||||
✓ a changed partial selects the test that rendered its ancestor with dataset "direct @include"
|
||||
✓ a changed partial selects the test that rendered its ancestor with dataset "transitive @include"
|
||||
✓ a changed partial selects the test that rendered its ancestor with dataset "x- component"
|
||||
✓ a changed partial selects the test that rendered its ancestor with dataset "include cycle"
|
||||
✓ a changed Inertia page selects the test that rendered its component
|
||||
✓ a changed shared JS module selects the tests of the pages that import it
|
||||
✓ a changed frontend runtime file selects every Inertia test
|
||||
|
||||
PASS Tests\Features\Tia\StateReclamation
|
||||
✓ a detached HEAD does not purge the graph on structural drift with dataset "sequential"
|
||||
✓ a detached HEAD does not purge the graph on structural drift with dataset "parallel"
|
||||
@@ -1748,6 +1785,8 @@
|
||||
✓ --fresh on a partial run neither purges nor prunes with dataset "parallel"
|
||||
✓ a second green run on a feature branch writes nothing at all with dataset "sequential"
|
||||
✓ a second green run on a feature branch writes nothing at all with dataset "parallel"
|
||||
✓ a graph whose recorded commit is gone is re-anchored, not warned about forever with dataset "sequential"
|
||||
✓ a graph whose recorded commit is gone is re-anchored, not warned about forever with dataset "parallel"
|
||||
|
||||
PASS Tests\Features\Ticket
|
||||
✓ it may be associated with an ticket #1, #2
|
||||
@@ -2378,4 +2417,4 @@
|
||||
✓ pass with dataset with ('my-datas-set-value')
|
||||
✓ within describe → pass with dataset with ('my-datas-set-value')
|
||||
|
||||
Tests: 1 deprecated, 4 warnings, 5 incomplete, 2 notices, 40 todos, 35 skipped, 1710 passed (3845 assertions)
|
||||
Tests: 1 deprecated, 4 warnings, 5 incomplete, 2 notices, 40 todos, 35 skipped, 1743 passed (3953 assertions)
|
||||
@@ -8,11 +8,6 @@ afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
|
||||
/*
|
||||
* Invariant 5 — writes land on the branch that ran and only there — and
|
||||
* invariant 6 — nothing is unbounded — under every branch shape git allows.
|
||||
*/
|
||||
|
||||
test('a branch name git allows is a branch key TIA can hold', function (string $branch): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
@@ -159,3 +154,44 @@ test('the default branch baseline survives every branch that comes and goes', fu
|
||||
->and($delta->removed())->toBe(0, $delta->summary())
|
||||
->and($project->graph()['baselines']['master']['results'])->toHaveCount(Project::TOTAL_TESTS);
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a project below the git repository root refuses to run and writes nothing', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$nested = $project->nested();
|
||||
|
||||
$result = $project->pestIn($nested, '--tia', ...$arguments);
|
||||
|
||||
expect($result->exitCode)->toBe(1, $result->describe())
|
||||
->and($result->output)->toContain('Tia mode requires the git repository root')
|
||||
->and($project->path('.home/.pest'))->not->toBeDirectory()
|
||||
->and($nested.DIRECTORY_SEPARATOR.'.pest')->not->toBeDirectory();
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a repository with no commits says so, and leaves plain runs alone', function (): void {
|
||||
$project = Project::withoutGit();
|
||||
$project->git()->run(['init', '--quiet']);
|
||||
$project->git()->run(['checkout', '--quiet', '-b', 'master']);
|
||||
$project->git()->addOrigin();
|
||||
|
||||
$tia = $project->pest('--tia');
|
||||
|
||||
expect($tia->exitCode)->toBe(1, $tia->describe())
|
||||
->and($tia->output)->toContain('Tia mode requires at least one commit')
|
||||
->and($tia->output)->not->toContain('requires [git]')
|
||||
->and($project->graphExists())->toBeFalse();
|
||||
|
||||
$plain = $project->pest();
|
||||
|
||||
expect($plain->exitCode)->toBe(0, $plain->describe())
|
||||
->and($plain->tally())->toContain(Project::TOTAL_TESTS.' passed');
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a directory with no repository at all still asks for git', function (): void {
|
||||
$project = Project::withoutGit();
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->exitCode)->toBe(1, $result->describe())
|
||||
->and($result->output)->toContain('requires "git"')
|
||||
->and($project->graphExists())->toBeFalse();
|
||||
})->skipOnWindows();
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
|
||||
test('a coverage report does not found a dependency graph', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
|
||||
$project->pest('--tia', ...$arguments);
|
||||
|
||||
expect($project->graphExists())->toBeFalse();
|
||||
})->with([
|
||||
'pest coverage' => [['--coverage']],
|
||||
'phpunit coverage report' => [['--coverage-text']],
|
||||
'parallel' => [['--coverage', '--parallel', '--processes=2']],
|
||||
])->skipOnWindows();
|
||||
|
||||
test('a plain run after a coverage run records the whole project scope', function (): void {
|
||||
$project = Project::make('master');
|
||||
|
||||
$project->pest('--tia', '--coverage');
|
||||
$project->pest('--tia');
|
||||
|
||||
$graph = $project->graph();
|
||||
|
||||
if ($graph === null) {
|
||||
expect($project->graphExists())->toBeFalse();
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
expect(array_keys($graph['edges']))->toEqualCanonicalizing(array_keys(Project::EDGES))
|
||||
->and($graph['files'])->toContain('tests/Unit/CalculatorTest.php')
|
||||
->and($graph['files'])->toContain('app/Calculator.php');
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a coverage report leaves the edges of an existing graph alone', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->pest('--tia', '--coverage');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($delta->edgesMoved())->toBeFalse($delta->summary())
|
||||
->and($delta->filesMoved())->toBeFalse($delta->summary())
|
||||
->and($delta->removed())->toBe(0, $delta->summary())
|
||||
->and($delta->added())->toBe(0, $delta->summary());
|
||||
})->skipOnWindows();
|
||||
@@ -175,7 +175,7 @@ test('tia still requires git', function (): void {
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->output)->toContain('The feature "Tia mode" requires "git".')
|
||||
expect($result->output)->toContain('The feature [Tia mode[ requires [git].')
|
||||
->and($result->exitCode)->not->toBe(0);
|
||||
})->skipOnWindows();
|
||||
|
||||
|
||||
@@ -75,9 +75,6 @@ test('filtered mode falls back to a full replay when a cached failure cannot be
|
||||
|
||||
$project->seed('master', failing: ['adds two numbers']);
|
||||
|
||||
// A path this project cannot address at all — recorded on another machine.
|
||||
// A path that merely no longer exists is a *deleted* test, not a lost one,
|
||||
// and widening the run would not find it either; see StateReclamation.
|
||||
$project->mutateGraph(function (array $graph): array {
|
||||
$testId = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
|
||||
|
||||
@@ -8,11 +8,6 @@ afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
|
||||
/*
|
||||
* Invariant 7 — a hostile state dir cannot break a run. Whatever is in
|
||||
* graph.json, the suite still runs and exits on the tests' merit.
|
||||
*/
|
||||
|
||||
test('a graph mangled beyond use still lets the suite run', function (string $contents): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
@@ -8,13 +8,6 @@ afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
|
||||
/*
|
||||
* A test that triggers a notice, deprecation or warning is reported by PHPUnit
|
||||
* as passed, and emits Passed. Recording it as a plain success made the cache
|
||||
* hide the issue: a later run under --fail-on-* came back green where a fresh
|
||||
* run failed. Invariant 3 — replay is faithful — at its most dangerous.
|
||||
*/
|
||||
|
||||
function tiaTriggering(string $call): string
|
||||
{
|
||||
return <<<PHP
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
|
||||
/**
|
||||
* @param callable(array<string, mixed>): array<string, mixed>|null $mutator
|
||||
* @return array{0: Project, 1: array<string, string>}
|
||||
*/
|
||||
function tiaPublishedBaseline(string $mode = 'ok', ?callable $mutator = null): array
|
||||
{
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$payload = $project->detachGraph();
|
||||
|
||||
if ($mutator !== null) {
|
||||
/** @var array<string, mixed> $decoded */
|
||||
$decoded = json_decode($payload, true);
|
||||
$payload = (string) json_encode($mutator($decoded), JSON_UNESCAPED_SLASHES);
|
||||
}
|
||||
|
||||
return [$project, $project->gh($mode, $payload)];
|
||||
}
|
||||
|
||||
test('a published baseline is fetched instead of recorded locally', function (): void {
|
||||
[$project, $environment] = tiaPublishedBaseline();
|
||||
|
||||
$result = $project->pestWithEnvironment($project->path(), $environment, '--tia', '--baselined');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->output)->toContain('Downloading TIA baseline')
|
||||
->and($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($project->graphExists())->toBeTrue();
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a fetched baseline that will not decode is discarded rather than trusted', function (): void {
|
||||
[$project, $environment] = tiaPublishedBaseline('corrupt');
|
||||
|
||||
$result = $project->pestWithEnvironment($project->path(), $environment, '--tia', '--baselined');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->output)->toContain('The dependency graph could not be read')
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed')
|
||||
->and($result->replayed())->toBe(0, $result->describe());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a fetched baseline recorded against another tree is not used', function (): void {
|
||||
[$project, $environment] = tiaPublishedBaseline('ok', function (array $graph): array {
|
||||
$graph['fingerprint']['structural']['composer_lock'] = 'a-lockfile-this-project-never-had';
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$result = $project->pestWithEnvironment($project->path(), $environment, '--tia', '--baselined');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed')
|
||||
->and($result->replayed())->toBe(0, $result->describe());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('an artifact without a graph in it fails loudly', function (): void {
|
||||
[$project, $environment] = tiaPublishedBaseline('missing-asset');
|
||||
|
||||
$result = $project->pestWithEnvironment($project->path(), $environment, '--tia', '--baselined');
|
||||
|
||||
expect($result->exitCode)->toBe(1, $result->describe())
|
||||
->and($result->output)->toContain('the artifact is missing expected files')
|
||||
->and($project->graphExists())->toBeFalse();
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a baseline that cannot be authenticated for fails loudly', function (): void {
|
||||
[$project, $environment] = tiaPublishedBaseline('unauthenticated');
|
||||
|
||||
$result = $project->pestWithEnvironment($project->path(), $environment, '--tia', '--baselined');
|
||||
|
||||
expect($result->exitCode)->toBe(1, $result->describe())
|
||||
->and($result->output)->toContain('is not authenticated')
|
||||
->and($project->graphExists())->toBeFalse();
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a workflow or artifact that is not there fails loudly', function (): void {
|
||||
[$project, $environment] = tiaPublishedBaseline('list-404');
|
||||
|
||||
$result = $project->pestWithEnvironment($project->path(), $environment, '--tia', '--baselined');
|
||||
|
||||
expect($result->exitCode)->toBe(1, $result->describe())
|
||||
->and($result->output)->toContain('not found in repo')
|
||||
->and($project->graphExists())->toBeFalse();
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a network failure warns and lets the suite run', function (string $mode): void {
|
||||
[$project, $environment] = tiaPublishedBaseline($mode);
|
||||
|
||||
$result = $project->pestWithEnvironment($project->path(), $environment, '--tia', '--baselined');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->output)->toContain('network error')
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed');
|
||||
})->with([
|
||||
'querying the runs' => ['list-network'],
|
||||
'downloading the artifact' => ['download-network'],
|
||||
])->skipOnWindows();
|
||||
|
||||
test('no published baseline yet starts a cooldown, and a corrupt cooldown does not break the run', function (): void {
|
||||
[$project, $environment] = tiaPublishedBaseline('no-runs');
|
||||
|
||||
$discardGraph = function () use ($project): void {
|
||||
if ($project->graphExists()) {
|
||||
$project->detachGraph();
|
||||
}
|
||||
};
|
||||
|
||||
$first = $project->pestWithEnvironment($project->path(), $environment, '--tia', '--baselined');
|
||||
|
||||
expect($first->exitCode)->toBe(0, $first->describe())
|
||||
->and($first->output)->toContain('No baseline published yet')
|
||||
->and($project->graphDir().DIRECTORY_SEPARATOR.'fetch-cooldown.json')->toBeFile();
|
||||
|
||||
$discardGraph();
|
||||
|
||||
$second = $project->pestWithEnvironment($project->path(), $environment, '--tia', '--baselined');
|
||||
|
||||
expect($second->exitCode)->toBe(0, $second->describe())
|
||||
->and($second->output)->toContain('next auto-retry in');
|
||||
|
||||
file_put_contents($project->graphDir().DIRECTORY_SEPARATOR.'fetch-cooldown.json', 'not json{');
|
||||
|
||||
$discardGraph();
|
||||
|
||||
$third = $project->pestWithEnvironment($project->path(), $environment, '--tia', '--baselined');
|
||||
|
||||
expect($third->exitCode)->toBe(0, $third->describe())
|
||||
->and($third->output)->toContain('No baseline published yet')
|
||||
->and($third->tally())->toContain(Project::TOTAL_TESTS.' passed');
|
||||
})->skipOnWindows();
|
||||
@@ -0,0 +1,218 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
|
||||
function tiaSeedWithView(Project $project, string $view): void
|
||||
{
|
||||
$project->seed('master');
|
||||
|
||||
$project->mutateGraph(function (array $graph) use ($view): array {
|
||||
$id = count($graph['files']);
|
||||
$graph['files'][$id] = $view;
|
||||
$graph['edges']['tests/Unit/GreeterTest.php'][] = $id;
|
||||
|
||||
return $graph;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* @param array<int, string> $components
|
||||
* @param array<string, array<int, string>> $jsFileToComponents
|
||||
*/
|
||||
function tiaSeedWithInertia(Project $project, array $components, array $jsFileToComponents = []): void
|
||||
{
|
||||
$project->seed('master');
|
||||
|
||||
$project->mutateGraph(function (array $graph) use ($components, $jsFileToComponents): array {
|
||||
$graph['test_inertia_components'] = ['tests/Unit/GreeterTest.php' => $components];
|
||||
$graph['js_file_to_components'] = $jsFileToComponents;
|
||||
|
||||
return $graph;
|
||||
});
|
||||
}
|
||||
|
||||
test('a committed rename selects the tests that depended on the old path', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->write('resources/views/greeting.blade.php', "<p>Hello</p>\n");
|
||||
$project->git()->commit('add view');
|
||||
|
||||
tiaSeedWithView($project, 'resources/views/greeting.blade.php');
|
||||
|
||||
$project->git()->run(['mv', 'resources/views/greeting.blade.php', 'resources/views/hello.blade.php']);
|
||||
$project->git()->commit('move the view');
|
||||
$project->snapshot();
|
||||
|
||||
$result = $project->pest('--tia', ...$arguments);
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->affected())->toBe(2, $result->describe())
|
||||
->and($result->replayed())->toBe(4, $result->describe());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('an affected test file that is gone does not strand a filtered run', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->write('resources/views/page.blade.php', "<p>one</p>\n");
|
||||
$project->git()->commit('add view');
|
||||
$project->seed('master');
|
||||
|
||||
$project->mutateGraph(function (array $graph): array {
|
||||
$id = count($graph['files']);
|
||||
$graph['files'][$id] = 'resources/views/page.blade.php';
|
||||
$graph['edges']['tests/Unit/GhostTest.php'] = [$id];
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$project->write('resources/views/page.blade.php', "<p>two</p>\n");
|
||||
$project->snapshot();
|
||||
|
||||
$result = $project->pest('--tia', '--filtered', ...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->output)->toContain('No affected tests found')
|
||||
->and($result->output)->not->toContain('tests/Unit/GhostTest.php')
|
||||
->and($delta->isHardSuppressed())->toBeTrue($delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a plain run reclaims the edge of a test file that is gone', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->write('resources/views/page.blade.php', "<p>one</p>\n");
|
||||
$project->git()->commit('add view');
|
||||
$project->seed('master');
|
||||
|
||||
$project->mutateGraph(function (array $graph): array {
|
||||
$id = count($graph['files']);
|
||||
$graph['files'][$id] = 'resources/views/page.blade.php';
|
||||
$graph['edges']['tests/Unit/GhostTest.php'] = [$id];
|
||||
$graph['baselines']['master']['results']['P\\Tests\\Unit\\GhostTest::ghostly'] = [
|
||||
'status' => 0,
|
||||
'message' => '',
|
||||
'time' => 9.999,
|
||||
'assertions' => 42,
|
||||
'file' => 'tests/Unit/GhostTest.php',
|
||||
];
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$project->write('resources/views/page.blade.php', "<p>two</p>\n");
|
||||
$project->snapshot();
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($delta->removed())->toBe(1, $delta->summary())
|
||||
->and($project->graph()['edges'])->not->toHaveKey('tests/Unit/GhostTest.php');
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a changed view selects the test that rendered it', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->write('resources/views/page.blade.php', "<p>one</p>\n");
|
||||
$project->git()->commit('add view');
|
||||
|
||||
tiaSeedWithView($project, 'resources/views/page.blade.php');
|
||||
|
||||
$project->write('resources/views/page.blade.php', "<p>two</p>\n");
|
||||
$project->snapshot();
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->affected())->toBe(2, $result->describe())
|
||||
->and($result->replayed())->toBe(4, $result->describe());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a changed partial selects the test that rendered its ancestor', function (array $views, string $changed): void {
|
||||
$project = Project::make('master');
|
||||
|
||||
foreach ($views as $path => $contents) {
|
||||
$project->write($path, $contents);
|
||||
}
|
||||
|
||||
$project->git()->commit('add views');
|
||||
|
||||
tiaSeedWithView($project, 'resources/views/page.blade.php');
|
||||
|
||||
$project->write($changed, $views[$changed]."<span>edited</span>\n");
|
||||
$project->snapshot();
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->affected())->toBe(2, $result->describe())
|
||||
->and($result->replayed())->toBe(4, $result->describe());
|
||||
})->with([
|
||||
'direct @include' => [[
|
||||
'resources/views/page.blade.php' => "@include('partials.nav')\n",
|
||||
'resources/views/partials/nav.blade.php' => "<nav>one</nav>\n",
|
||||
], 'resources/views/partials/nav.blade.php'],
|
||||
'transitive @include' => [[
|
||||
'resources/views/page.blade.php' => "@include('partials.wrapper')\n",
|
||||
'resources/views/partials/wrapper.blade.php' => "@include('partials.nav')\n",
|
||||
'resources/views/partials/nav.blade.php' => "<nav>one</nav>\n",
|
||||
], 'resources/views/partials/nav.blade.php'],
|
||||
'x- component' => [[
|
||||
'resources/views/page.blade.php' => "<x-card>hi</x-card>\n",
|
||||
'resources/views/components/card.blade.php' => "<div>one</div>\n",
|
||||
], 'resources/views/components/card.blade.php'],
|
||||
'include cycle' => [[
|
||||
'resources/views/page.blade.php' => "@include('partials.a')\n",
|
||||
'resources/views/partials/a.blade.php' => "@include('partials.b')\n",
|
||||
'resources/views/partials/b.blade.php' => "@include('partials.a')\n",
|
||||
], 'resources/views/partials/b.blade.php'],
|
||||
])->skipOnWindows();
|
||||
|
||||
test('a changed Inertia page selects the test that rendered its component', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->write('resources/js/Pages/Foo.vue', "<template>one</template>\n");
|
||||
$project->git()->commit('add page');
|
||||
|
||||
tiaSeedWithInertia($project, ['Foo']);
|
||||
|
||||
$project->write('resources/js/Pages/Foo.vue', "<template>two</template>\n");
|
||||
$project->snapshot();
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->affected())->toBe(2, $result->describe())
|
||||
->and($result->replayed())->toBe(4, $result->describe());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a changed shared JS module selects the tests of the pages that import it', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->write('resources/js/Pages/Foo.vue', "<template>one</template>\n");
|
||||
$project->write('resources/js/Shared/Nav.vue', "<template>nav</template>\n");
|
||||
$project->git()->commit('add pages');
|
||||
|
||||
tiaSeedWithInertia($project, ['Foo'], ['resources/js/Shared/Nav.vue' => ['Foo']]);
|
||||
|
||||
$project->write('resources/js/Shared/Nav.vue', "<template>nav two</template>\n");
|
||||
$project->snapshot();
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->affected())->toBe(2, $result->describe())
|
||||
->and($result->replayed())->toBe(4, $result->describe());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a changed frontend runtime file selects every Inertia test', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->write('resources/js/app.js', "console.log(1)\n");
|
||||
$project->git()->commit('add runtime');
|
||||
|
||||
tiaSeedWithInertia($project, ['Foo']);
|
||||
|
||||
$project->write('resources/js/app.js', "console.log(2)\n");
|
||||
$project->snapshot();
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->affected())->toBe(2, $result->describe())
|
||||
->and($result->replayed())->toBe(4, $result->describe());
|
||||
})->skipOnWindows();
|
||||
@@ -129,8 +129,6 @@ test('the fallback still reaches a branch that has never run a test file', funct
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
|
||||
// A narrowed run mints the branch key holding only the Greeter entries, so
|
||||
// the layering must still serve master's cached failure for the Calculator.
|
||||
$project->pest('--filter=greets a person');
|
||||
|
||||
$result = $project->pest('--tia', '--filtered');
|
||||
@@ -212,12 +210,6 @@ test('a malformed baseline entry cannot break the run', function (array $argumen
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed');
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
/*
|
||||
* Invariant 1 — sequential and parallel must agree — under a process that is
|
||||
* torn down in the middle of a test file. A worker that flushed what it got to
|
||||
* before dying has not seen enough of that file to license pruning the
|
||||
* siblings it never reached.
|
||||
*/
|
||||
test('a run torn down mid-file does not prune the tests it never reached', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
@@ -281,10 +273,6 @@ test('a fatal error mid-file is a test error, not a truncation', function (array
|
||||
->and($delta->structureMoved())->toBeFalse($delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
/*
|
||||
* Invariant 2 — every command lands in exactly one tier and stays inside it —
|
||||
* for the combinations that were never exercised.
|
||||
*/
|
||||
test('a green complete run leaves the graph exactly as it found it', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
@@ -343,3 +331,30 @@ test('a second green run on a feature branch writes nothing at all', function (a
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($delta->isHardSuppressed())->toBeTrue($delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a graph whose recorded commit is gone is re-anchored, not warned about forever', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->git()->commit('second');
|
||||
$project->seed('master');
|
||||
|
||||
$recordedSha = $project->graph()['baselines']['master']['sha'];
|
||||
|
||||
$project->git()->run(['reset', '--quiet', '--hard', 'HEAD~1']);
|
||||
$project->snapshot();
|
||||
|
||||
$first = $project->pest('--tia', ...$arguments);
|
||||
|
||||
expect($first->exitCode)->toBe(0, $first->describe())
|
||||
->and($first->output)->toContain('no longer reachable')
|
||||
->and($first->tally())->toContain(Project::TOTAL_TESTS.' passed')
|
||||
->and($project->graph()['baselines']['master']['sha'])->not->toBe($recordedSha)
|
||||
->and($project->graph()['baselines']['master']['sha'])->toBe($project->git()->sha());
|
||||
|
||||
$project->snapshot();
|
||||
$second = $project->pest('--tia', ...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($second->output)->not->toContain('no longer reachable')
|
||||
->and($second->replayed())->toBe(Project::TOTAL_TESTS, $second->describe())
|
||||
->and($delta->writtenCount())->toBe(0, $delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
@@ -130,7 +130,7 @@ final readonly class GitRepo
|
||||
private function process(array $arguments, bool $mustSucceed): Process
|
||||
{
|
||||
$process = new Process(['git', ...$arguments], $this->path, self::ENV);
|
||||
$process->setTimeout(30.0);
|
||||
$process->setTimeout(120.0);
|
||||
$process->run();
|
||||
|
||||
if ($mustSucceed && ! $process->isSuccessful()) {
|
||||
|
||||
@@ -43,9 +43,6 @@ final class Project
|
||||
public const int TOTAL_TESTS = 6;
|
||||
|
||||
/**
|
||||
* A dataset for the rule that TIA must reach the same outcome sequentially
|
||||
* and in parallel: the same command, run both ways, must leave the same graph.
|
||||
*
|
||||
* @var array<string, array<int, array<int, string>>>
|
||||
*/
|
||||
public const array SEQUENTIAL_AND_PARALLEL = [
|
||||
@@ -124,6 +121,20 @@ final class Project
|
||||
}
|
||||
}
|
||||
|
||||
public function nested(string $directory = 'nested'): string
|
||||
{
|
||||
$path = $this->path($directory);
|
||||
|
||||
if (! is_dir($path) && ! @mkdir($path, 0755, true) && ! is_dir($path)) {
|
||||
throw new RuntimeException(sprintf('Unable to create [%s].', $path));
|
||||
}
|
||||
|
||||
self::copy(__DIR__.'/app', $path);
|
||||
$this->scaffoldVendor($path);
|
||||
|
||||
return $path;
|
||||
}
|
||||
|
||||
public function worktree(string $branch): string
|
||||
{
|
||||
$path = $this->path.'-worktree-'.preg_replace('/[^a-z0-9]+/i', '-', $branch);
|
||||
@@ -159,6 +170,7 @@ final class Project
|
||||
'COLLISION_IGNORE_DURATION' => 'true',
|
||||
'PARATEST' => '0',
|
||||
'PAO_DISABLE' => '1',
|
||||
'XDEBUG_MODE' => 'coverage',
|
||||
'HOME' => $this->home(),
|
||||
'GITHUB_EVENT_PATH' => '',
|
||||
'CI_DEFAULT_BRANCH' => '',
|
||||
@@ -239,6 +251,40 @@ final class Project
|
||||
$sentinel ? $this->sentinel() : $this->snapshot();
|
||||
}
|
||||
|
||||
public function detachGraph(): string
|
||||
{
|
||||
$json = $this->state()->read(Tia::KEY_GRAPH);
|
||||
|
||||
if ($json === null) {
|
||||
throw new RuntimeException('There is no graph to detach.');
|
||||
}
|
||||
|
||||
if (! $this->state()->delete(Tia::KEY_GRAPH)) {
|
||||
throw new RuntimeException('Unable to remove the detached graph.');
|
||||
}
|
||||
|
||||
$this->snapshot();
|
||||
|
||||
return $json;
|
||||
}
|
||||
|
||||
/**
|
||||
* @return array<string, string>
|
||||
*/
|
||||
public function gh(string $mode = 'ok', string $payload = '{}'): array
|
||||
{
|
||||
self::mirror(__DIR__.'/stubs/gh', $this->path('stub/gh'));
|
||||
chmod($this->path('stub/gh'), 0755);
|
||||
|
||||
$this->write('payload/graph.json', $payload);
|
||||
|
||||
return [
|
||||
'PATH' => $this->path('stub').PATH_SEPARATOR.getenv('PATH'),
|
||||
'GH_STUB_MODE' => $mode,
|
||||
'GH_STUB_PAYLOAD' => $this->path('payload/graph.json'),
|
||||
];
|
||||
}
|
||||
|
||||
public static function testId(string $testFile, string $description): string
|
||||
{
|
||||
$basename = basename($testFile, '.php');
|
||||
@@ -270,10 +316,6 @@ final class Project
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Adds a second, empty baseline key, so a lone recorded baseline can no
|
||||
* longer stand in for the default branch.
|
||||
*/
|
||||
public function addBaseline(string $branch): void
|
||||
{
|
||||
$this->mutateGraph(function (array $graph) use ($branch): array {
|
||||
@@ -322,7 +364,6 @@ final class Project
|
||||
{
|
||||
$baselines = $this->graph()['baselines'] ?? [];
|
||||
|
||||
// A branch named `12345` comes back from json_decode as an integer key.
|
||||
return is_array($baselines) ? array_map(strval(...), array_keys($baselines)) : [];
|
||||
}
|
||||
|
||||
|
||||
Executable
+47
@@ -0,0 +1,47 @@
|
||||
#!/bin/sh
|
||||
|
||||
if [ "$1" = "auth" ]; then
|
||||
[ "$GH_STUB_MODE" = "unauthenticated" ] && exit 1
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$1" = "run" ] && [ "$2" = "list" ]; then
|
||||
case "$GH_STUB_MODE" in
|
||||
no-runs) exit 0 ;;
|
||||
list-404) echo "HTTP 404: Not Found" >&2; exit 1 ;;
|
||||
list-network) echo "could not resolve host: api.github.com" >&2; exit 1 ;;
|
||||
esac
|
||||
echo 987654321
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$1" = "api" ]; then
|
||||
echo 2048
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$1" = "run" ] && [ "$2" = "download" ]; then
|
||||
case "$GH_STUB_MODE" in
|
||||
download-403) echo "HTTP 403: Forbidden" >&2; exit 1 ;;
|
||||
download-network) echo "connection refused" >&2; exit 1 ;;
|
||||
esac
|
||||
|
||||
dir=""
|
||||
previous=""
|
||||
for argument in "$@"; do
|
||||
[ "$previous" = "-D" ] && dir="$argument"
|
||||
previous="$argument"
|
||||
done
|
||||
|
||||
[ -z "$dir" ] && exit 1
|
||||
|
||||
case "$GH_STUB_MODE" in
|
||||
missing-asset) echo "{}" > "$dir/other.json" ;;
|
||||
corrupt) printf 'not json at all' > "$dir/graph.json" ;;
|
||||
*) cp "$GH_STUB_PAYLOAD" "$dir/graph.json" ;;
|
||||
esac
|
||||
|
||||
exit 0
|
||||
fi
|
||||
|
||||
exit 1
|
||||
+1
-1
@@ -19,7 +19,7 @@ pest()->in('PHPUnit/GlobPatternTests/SubFolder2/*AsPattern.php')->use(CustomTest
|
||||
|
||||
pest()->in('Visual')->group('integration');
|
||||
|
||||
pest()->in('Features/Tia')->group('integration');
|
||||
pest()->in('Features/Tia')->group('integration', 'tia');
|
||||
|
||||
// NOTE: global test value container to be mutated and checked across files, as needed
|
||||
$_SERVER['globalHook'] = (object) ['calls' => (object) ['beforeAll' => 0, 'afterAll' => 0]];
|
||||
|
||||
Reference in New Issue
Block a user