mirror of
https://github.com/pestphp/pest.git
synced 2026-09-05 06:13:35 +02:00
Compare commits
8 Commits
e90e4f70fc
...
7bfb2a6185
| Author | SHA1 | Date | |
|---|---|---|---|
| 7bfb2a6185 | |||
| ca56ca5216 | |||
| b49ba062d5 | |||
| 09699847a2 | |||
| 545d0c2784 | |||
| b795af3b1b | |||
| a40cc6bc0e | |||
| 4d3d0105b7 |
@@ -16,3 +16,19 @@ composer test:integration # visual and snapshot tests
|
||||
composer test # everything CI runs, in CI's order
|
||||
composer update:snapshots # only when a test was added or removed
|
||||
```
|
||||
|
||||
## TIA scenario tests
|
||||
|
||||
`tests/Features/Tia/*` scaffold a throwaway git project, run a real `pest` subprocess against it, and diff the TIA graph it wrote. They exist because TIA's contract is about what a run *writes* — replay, branch keys, and the COMPLETE / RESULTS-ONLY / HARD-SUPPRESSED tiers are invisible to ordinary assertions, and every case used to be measured by hand against a playground app.
|
||||
|
||||
Add one whenever a change touches branch resolution, replay, filtered mode, or the write tiers. How:
|
||||
|
||||
- `Project::make('master')` scaffolds; `seed('master')` writes a graph and sentinels every cached result (`time=9.999`, `assertions=42`) so any rewrite shows up.
|
||||
- `$project->pest('--tia', …)` runs it; `$project->delta()` compares against that snapshot. `writtenCount()` is the discriminator — `0` means "replayed", not "wrote the same values". `mutateGraph()` bends one entry; overlays in `tests/Fixtures/Tia/overlays/<name>/` supply a different `tests/Pest.php`.
|
||||
- Keep expectations driver-independent: a cold recording run needs pcov/xdebug and behaves differently without one. Seed a graph instead of recording one.
|
||||
|
||||
Run them by file (a directory argument finds nothing) or by `--filter`:
|
||||
|
||||
```bash
|
||||
php bin/pest tests/Features/Tia/PartialRunWriteTier.php
|
||||
```
|
||||
|
||||
@@ -90,9 +90,9 @@ these are one fix or two.
|
||||
| Complete non-TIA runs write edge-less results | `src/Plugins/Tia.php:1645`, `:638` | Guard is `! $complete && ! knowsTest()`; complete bypasses it and `markKnownTestFiles` stays false. Inert — replay guards on the same predicate at `:360` — but inflates `n` |
|
||||
| SIGINT never reaches the suite | `PcovRestarter` re-exec | Parent ignores it; the child holds PHPUnit's handler. CI `timeout`/Ctrl-C will not stop a run |
|
||||
| Structural drift never announced sequentially | `src/Plugins/Tia.php:1178-1181` | `enterRecordMode()` prints a bare `Running in TIA mode.`; `renderFreshGraph()` (`fresh graph (composer.lock changed)`) only runs under `--parallel` or coverage piggyback |
|
||||
| Replay clobbers cached `time` | write path | `0.058` → `0.001`; timing data degrades toward zero across runs |
|
||||
| `pest --parallel` without `--tia` writes nothing | G3, G6 | Sequential `pest --filter=…` does record. Parallel CI contributes nothing to the cached-failure replay path |
|
||||
| `--tia --parallel --filter`/`--shard` record nothing | G2, G8 | More conservative than the results-only contract requires |
|
||||
| ~~Replay clobbers cached `time`~~ | write path | **Struck in phase four — does not reproduce**, sequentially or in parallel. Both write paths route through `resultTime()`; a replay with every cached `time` sentinelled writes nothing |
|
||||
| ~~`pest --parallel` without `--tia` writes nothing~~ | G3, G6 | **Closed in phase four.** Workers flush their results through `requestWorkerResults()`, so a parallel run refreshes and prunes exactly like the sequential run of the same command |
|
||||
| ~~`--tia --parallel --filter`/`--shard` record nothing~~ | G2, G8 | **Closed in phase four** (partial parity landed first, complete parity with the row above) |
|
||||
| `--min=50` without `--coverage` is a silent no-op | — | — |
|
||||
| `pest --repeat=2` is not a Pest option | J11 | Case unrunnable as written; drop it or add the option |
|
||||
|
||||
|
||||
@@ -0,0 +1,312 @@
|
||||
# TIA deep audit — phase five
|
||||
|
||||
## Your task
|
||||
|
||||
Phases one through four fixed the defects that were *reported*. This phase is the opposite shape: go
|
||||
looking. Read TIA's read/write path adversarially, find what is wrong or fragile, and leave behind a
|
||||
scenario suite that covers the edges nobody has exercised yet.
|
||||
|
||||
Two deliverables, both required:
|
||||
|
||||
1. **A ranked findings list.** Every finding needs a *reproduction*, not a reading of the code. A
|
||||
finding you cannot reproduce is a hypothesis — say so and rank it separately.
|
||||
2. **New scenario tests** in `tests/Features/Tia/*`, covering the edges you probed. Rows that pass go
|
||||
in the repo (they are the regression net). Rows that fail stay in your scratchpad until the fix
|
||||
lands — **never commit a red test.**
|
||||
|
||||
Work in passes, and **report between passes** rather than at the very end:
|
||||
|
||||
- **Pass A** — reproduce the leads in Part 3 below. Report which are real.
|
||||
- **Pass B** — your own hunt: the invariants in Part 2, attacked with inputs nobody tried.
|
||||
- **Pass C** — fixes, smallest first, each with the row that pins it.
|
||||
|
||||
Fix what is clearly a defect with an obvious correct answer. **Stop and ask** when the fix is a
|
||||
behaviour *choice* (what should TIA do when two runs race for one graph?) — those are Nuno's calls,
|
||||
and a reproduction with a crisp yes/no question is worth more than a guessed fix.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — The harness
|
||||
|
||||
### 1.1 What TIA is
|
||||
|
||||
Test Impact Analysis. `pest --tia` records a dependency graph (test file → source files it touched)
|
||||
plus a per-branch baseline of results, then on later runs replays the tests whose dependencies did not
|
||||
change. State lives in a per-project dir under `~/.pest/tia/`; `graph.json` is the whole thing.
|
||||
|
||||
Source of truth: `src/Plugins/Tia.php` (the plugin, ~1900 lines) and `src/Plugins/Tia/Graph.php` (the
|
||||
graph model + read/write API). Supporting: `src/Plugins/Tia/ChangedFiles.php` (git), `Fingerprint.php`
|
||||
(environment/structure hashing), `State.php` (the state dir).
|
||||
|
||||
Read `PLAN.md`, `PLAN_PHASE_TWO.md`, `PLAN_PHASE_THREE.md`, `PLAN_PHASE_FOUR.md` first — in that
|
||||
order. They carry the history, the tier contract, and the decisions already made. Struck-through rows
|
||||
are fixed; do not re-report them.
|
||||
|
||||
### 1.2 The scenario harness
|
||||
|
||||
`tests/Features/Tia/*` scaffold a throwaway git project into a temp dir, run a **real `pest`
|
||||
subprocess** against it, and diff the graph it wrote. Everything lives in `tests/Fixtures/Tia/`:
|
||||
|
||||
| Class | What it gives you |
|
||||
|---|---|
|
||||
| `Project` | `make(branch, overlay:)`, `withoutGit()`, `seed(branch, sentinel:, failing:)`, `seedFor($root, …)`, `pest(...$args)`, `pestWithEnvironment($dir, $env, ...$args)`, `pestIn($dir, ...)`, `write($rel, $contents)`, `path($rel)`, `graph()`, `branchKeys()`, `graphDir()`, `graphExists()`, `snapshot()`, `delta()`, `mutateGraph(fn)`, `addBaseline($branch)`, `worktree($branch)`, `destroy()`, `destroyAll()`, `SEQUENTIAL_AND_PARALLEL` |
|
||||
| `GitRepo` (`$project->git()`) | `switchTo($b, new:)`, `rename($from, $to)`, `detach()`, `config($k, $v)`, `addOrigin()`, `removeOrigin()`, `setOriginHead($b)`, `unsetOriginHead()`, `worktree()`, `sha()`, `branchNames()` |
|
||||
| `GraphDelta` (`$project->delta()`) | `writtenCount()`, `added()`, `removed()`, `branchKeys()`, `baselineUntouched($b)`, `shaMoved()`, `treeMoved()`, `edgesMoved()`, `filesMoved()`, `fingerprintMoved()`, `structureMoved()`, `isResultsOnly()`, `isHardSuppressed()`, `summary()` |
|
||||
| `PestResult` (returned by `pest()`) | `replayed()`, `uncached()`, `affected()`, `tally()`, `output`, `exitCode`, `describe()` |
|
||||
|
||||
The fixture app is 3 test files / **6 tests** (`Project::TOTAL_TESTS`); `Project::EDGES` and
|
||||
`Project::TESTS` describe the graph `seed()` writes. `Project::testId($file, $description)` builds a
|
||||
result key. Overlays in `tests/Fixtures/Tia/overlays/<name>/` supply a different `tests/Pest.php` —
|
||||
that is how you configure `pest()->tia()->…` for a scenario.
|
||||
|
||||
**If the fixture cannot express your case, extend the fixture.** A new `GitRepo` verb or a new overlay
|
||||
is a legitimate part of the deliverable — several findings below need one. Do not water down a
|
||||
scenario to fit the current helpers.
|
||||
|
||||
### 1.3 The sentinel discriminator — read before writing any assertion
|
||||
|
||||
`seed()` writes a graph **and then rewrites every cached result to `time=9.999`, `assertions=42`**
|
||||
(only where assertions are non-zero — risky/skipped/incomplete statuses are *derived* from "performed
|
||||
no assertions", so falsifying those would rewrite the status on replay), then snapshots. Therefore:
|
||||
|
||||
- `$delta->writtenCount()` is the only reliable way to tell **"replayed"** from **"executed and wrote
|
||||
back the same values"**. `0` means nothing was written.
|
||||
- `isResultsOnly()` = no structure moved, no `sha`/`tree` movement, nothing added or removed.
|
||||
- `isHardSuppressed()` = the graph is byte-identical.
|
||||
- A replayed suite reports inflated assertion counts (6 tests × 42). That is the sentinel, not a bug.
|
||||
|
||||
The three write tiers, unchanged since phase two:
|
||||
|
||||
- **COMPLETE** — may change everything.
|
||||
- **RESULTS-ONLY** — may change only `baselines[<branch>].results` for tests that ran; never removes
|
||||
an entry, never adds a result for a test file absent from `edges`, never touches
|
||||
`sha`/`tree`/`edges`/`files`/`fingerprint`.
|
||||
- **HARD-SUPPRESSED** — may change nothing.
|
||||
|
||||
### 1.4 Running them
|
||||
|
||||
They are in the `integration` group (`tests/Pest.php`), so `composer test:unit` skips them. **A
|
||||
directory argument finds nothing** — pass files, space-separated, as separate argv entries:
|
||||
|
||||
```bash
|
||||
PAO_DISABLE=1 php84 bin/pest tests/Features/Tia/DefaultBranchReplay.php tests/Features/Tia/DefaultBranchResolution.php \
|
||||
tests/Features/Tia/DefaultBranchWriteTier.php tests/Features/Tia/PartialRunWriteTier.php \
|
||||
tests/Features/Tia/CompleteRunWriteTier.php tests/Features/Tia/FilteredMode.php
|
||||
|
||||
PAO_DISABLE=1 php bin/pest <same list>
|
||||
```
|
||||
|
||||
**Both interpreters must be green.** `php84` is 8.4.x with **no pcov** — that is what CI has
|
||||
(`.github/workflows/tests.yml` sets `coverage: none`). `php` is 8.5.x with pcov — that is the dev
|
||||
machine. Any assertion that depends on a coverage driver passes locally and fails in CI. Concretely:
|
||||
|
||||
- A **cold recording run writes no graph at all** without pcov/xdebug (it prints `Running in TIA mode,
|
||||
however TIA is skipped as it needs ext-pcov or Xdebug`). **Seed a graph, never record one**, unless
|
||||
the driver *is* the point of the row.
|
||||
- A **PHP source file edit** driverless triggers `Detected PHP source changes but no coverage driver
|
||||
is available` → full suite, `affected=0`. Edit *test* files, not `app/` files.
|
||||
|
||||
`PAO_DISABLE=1` is mandatory on every pest invocation and every probe: `laravel/pao` emits JSON under
|
||||
agents and corrupts the captured output.
|
||||
|
||||
Baseline before you touch anything: **72 passed** on both interpreters, at `HEAD` plus the phase-four
|
||||
working-tree changes. If that number does not reproduce, stop and say so.
|
||||
|
||||
### 1.5 Measure before you assert
|
||||
|
||||
Never guess an expectation from reading the code. Probe first, in your scratchpad:
|
||||
|
||||
```php
|
||||
<?php
|
||||
require '/Users/nunomaduro/Work/projects/pestphp/pest/vendor/autoload.php';
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
$p = Project::make('master');
|
||||
$p->seed('master');
|
||||
$p->git()->switchTo('feature-x', new: true);
|
||||
$r = $p->pest('--tia');
|
||||
printf("tally=[%s] keys=[%s] %s\n", $r->tally(), implode(',', $p->branchKeys()), $p->delta()->summary());
|
||||
$p->destroy();
|
||||
```
|
||||
|
||||
`PAO_DISABLE=1 php probe.php`. One project per scenario; always `destroy()`; `Project::destroyAll()`
|
||||
at the end. Run every probe on **both** interpreters before believing it.
|
||||
|
||||
### 1.6 Ground rules
|
||||
|
||||
- **Do not** run `composer test`. It takes minutes and you do not need it.
|
||||
- **Do not** run `composer update:snapshots`. `tests/.snapshots/success.txt` and the tally in
|
||||
`tests/Visual/Parallel.php` encode the whole suite's result, so every row you add breaks them.
|
||||
That is expected — **report that they need regenerating and let Nuno run it.**
|
||||
- **Do not commit.** Leave the tree dirty.
|
||||
- **Do not touch the playground's `vendor/`.** If a finding truly needs the playground (25 real tests,
|
||||
real timings), say so and ask — syncing it is a manual step Nuno owns.
|
||||
- Run `vendor/bin/phpstan analyse <the files you touched> --memory-limit=-1 --no-progress` and
|
||||
`vendor/bin/pint <the files you touched>` before reporting. **Scope both to files you changed** —
|
||||
Nuno edits `src/` live, and a repo-wide fixer will revert his work.
|
||||
- The fixture projects **hardlink `src/`**. If you edit `src/` while a scenario run is in flight,
|
||||
those rows go red for no reason. Finish the edit, then run.
|
||||
- Keep scratch probes out of the repo.
|
||||
- **Never weaken an existing assertion to make something pass.** If an existing row contradicts your
|
||||
fix, that is a finding: report the contradiction and ask.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — The invariants to attack
|
||||
|
||||
These are the properties TIA is supposed to have. Each one is a place to hunt: construct the input
|
||||
that breaks it.
|
||||
|
||||
1. **Parity.** `pest <args>` and `pest --parallel --processes=N <args>` must leave the **same graph**
|
||||
and reach the same tally. This is a hard rule from Nuno — sequential and parallel must *always*
|
||||
agree. `Project::SEQUENTIAL_AND_PARALLEL` is the dataset that encodes it; consider making every
|
||||
new write-path row use it. Vary `--processes` (1, 2, 8 — more processes than test files).
|
||||
2. **The tiers hold.** Every command lands in exactly one of COMPLETE / RESULTS-ONLY /
|
||||
HARD-SUPPRESSED, and stays inside it. Combinations are where this frays: `--fresh --parallel
|
||||
--filter`, `--bail --shard`, `--filtered` plus an explicit path, `--tia --no-tia`, `--retry`.
|
||||
3. **Replay is faithful.** A replayed test reports the same status, message, time and assertion count
|
||||
as the recorded run — and replay itself writes nothing. Statuses beyond pass/fail are the soft
|
||||
spot: skipped, incomplete, risky, notice, deprecation, warning, todo, and a test that failed with a
|
||||
multi-line message.
|
||||
4. **Reads never write.** No read path may mint a baseline key, move a `sha`, or create the state dir.
|
||||
A project that has never run TIA must gain nothing from a plain `pest` run.
|
||||
5. **A branch never corrupts another branch's baseline.** Writes land on the branch that ran, and
|
||||
only there. Reads may *layer* the default branch under the current one (phase four, B1) — but that
|
||||
layering must not leak into a write.
|
||||
6. **Nothing is unbounded.** Baseline keys, `files`, `edges`, worker partials, state files: something
|
||||
must eventually reclaim them, or the graph grows forever.
|
||||
7. **A hostile state dir cannot break a run.** Corrupt, truncated, empty, wrong-schema, read-only,
|
||||
absent, or *someone else's* `graph.json` — the suite still runs and exits on the tests' merit.
|
||||
8. **Git shapes are all handled.** Detached HEAD (read-only, per phase four B2), worktrees, no commits
|
||||
yet, no `origin`, no `origin/HEAD`, submodules, a repo whose root is above the pest project (that
|
||||
one panics deliberately — `TiaRequiresRepositoryRoot`), renamed branches, deleted branches.
|
||||
|
||||
---
|
||||
|
||||
## Part 3 — Leads to reproduce first (Pass A)
|
||||
|
||||
These came out of reading the phase-four diff. **Each is a hypothesis, not a finding** — several may
|
||||
turn out to be fine. Reproduce or refute each, in order, and report the measured delta either way.
|
||||
|
||||
### L1 — the per-entry fallback may resurrect a pruned or deleted test · **highest value**
|
||||
|
||||
Phase four made `Graph::baselineFor()` layer the branch's results **over** the default branch's. Two
|
||||
consequences worth probing:
|
||||
|
||||
- `pruneStaleResults()` unsets an entry from `baselines[branch].results`. The very next read layers the
|
||||
**fallback's** entry for that same test id back in. So a delete may not stick from a branch's point
|
||||
of view.
|
||||
- `hasUnlocatedTestsToRerun()` returns true when a *failing* cached result names a file that no longer
|
||||
exists on disk — and that forces a **full suite**. If a feature branch deletes a test file that
|
||||
fails on the default branch, the merged read still carries master's entry pointing at the now-absent
|
||||
file. Suspected symptom: **that branch runs the full suite forever.**
|
||||
|
||||
Probe: seed master with a failing test (`seed('master', failing: [...])`), branch off, delete the test
|
||||
file that holds it, run `--tia`, and compare `replayed`/`uncached`/`affected` against the same shape
|
||||
where the failure is on the branch instead. Then the mirror case with a green deleted test.
|
||||
|
||||
### L2 — environment drift clears one branch's results and the fallback serves them right back
|
||||
|
||||
`reconcileFingerprint()` on environmental drift calls `$graph->clearResults($this->branch)` and warns
|
||||
`results dropped, edges reused`. On a feature branch that clears only the *branch's* results — the
|
||||
layered read then re-serves the default branch's results, which were recorded under the **old**
|
||||
environment. Suspected symptom: the drop is a no-op on any branch that is not the default one.
|
||||
|
||||
Probe: `pestWithEnvironment()` to shift whatever `Fingerprint` reads as environmental (check
|
||||
`Fingerprint::environmentalDrift()` for the exact keys), on the default branch vs a feature branch,
|
||||
and compare what survives.
|
||||
|
||||
### L3 — `sha`/`tree` may be taken from a different commit than the results
|
||||
|
||||
`baselineFor()` takes `sha` from the branch when non-null and otherwise from the fallback; `tree`
|
||||
likewise when the branch's is empty. So a branch can end up computing "what changed since" against the
|
||||
**default branch's** recorded sha while reading its own results — or vice versa. Is there a shape where
|
||||
that under-reports changed files (a test replays that should have run)? That is the dangerous
|
||||
direction: a false replay is a lie about a passing test.
|
||||
|
||||
Probe: seed master, commit a test edit on the branch so the shas genuinely differ, and check whether
|
||||
the edited test is treated as affected.
|
||||
|
||||
### L4 — pruning from merged worker partials
|
||||
|
||||
Phase four made a complete `--parallel` run write and prune from merged worker results. The stated
|
||||
safety net is that `pruneStaleResults()` only prunes files it saw results for, and that a truncated
|
||||
worker sets results-only. Try to defeat it: a worker that reports results for a test file it did not
|
||||
finish. Candidate shapes — a fatal error mid-file (not an assertion failure), `exit()` inside a test,
|
||||
an uncaught error in an `afterEach`, a test that kills its own process, `--stop-on-failure` variants,
|
||||
`--processes` greater than the number of test files.
|
||||
|
||||
### L5 — two runs racing for one `graph.json`
|
||||
|
||||
`State::write()` has no locking. Two pest processes on one project (a watcher plus a manual run, two CI
|
||||
jobs sharing a cache dir, `--parallel` where the parent writes while a straggler worker flushes) can
|
||||
lose an update or interleave. Probe by launching two `pest --tia` subprocesses concurrently against one
|
||||
project and diffing. **This is likely a design decision, not a bug** — if you reproduce a lost update,
|
||||
report it as a question (accept last-writer-wins, or lock?), do not invent a locking scheme.
|
||||
|
||||
### L6 — a detached HEAD still purges on structural drift
|
||||
|
||||
Phase four made a detached HEAD read-only *for writes* (`saveGraph()` refuses). But
|
||||
`reconcileFingerprint()` deletes the whole graph on structural drift (`Tia.php`, the
|
||||
`state->delete(KEY_GRAPH)` in the structural branch) before any write happens. So `--tia` from a
|
||||
detached checkout with a changed `composer.lock` can still wipe the default branch's baseline. Confirm
|
||||
it, then ask: should the detached-HEAD guard cover the purge too?
|
||||
|
||||
### L7 — statuses that may not round-trip
|
||||
|
||||
`PLAN.md` §5 claims warnings and deprecations record as `status=0`, and that codes `6` and `4` look
|
||||
unreachable. `Graph::getResult()` maps 0–8 to `TestStatus`. Verify each status end to end: record it,
|
||||
replay it, and check the replayed run reports the same thing — including the message, the exit code,
|
||||
and whether `shouldRerunStatus()` decides to re-execute it. `failOnRisky` / `failOnSkipped` /
|
||||
`displayDetailsOn*` change that decision, so an overlay that flips those config flags is part of this.
|
||||
A status that replays as a pass would be the most serious class of bug in TIA.
|
||||
|
||||
### L8 — branch-key hygiene
|
||||
|
||||
Nothing appears to reclaim baseline keys. Probe: create and delete 5 branches, rename one
|
||||
(`GitRepo::rename()`), and check what `branchKeys()` holds afterwards. Also try names that stress the
|
||||
JSON keying: `feature/x/y` (already covered), a name differing from another only in case (macOS is
|
||||
case-insensitive — does the key match the ref?), a name with a space or a unicode character, a branch
|
||||
literally called `HEAD`, and a very long name. Then: is unbounded growth acceptable, or does this need
|
||||
a cap / GC? Ask rather than build.
|
||||
|
||||
### L9 — `soleRecordedBranch()` as a fallback source
|
||||
|
||||
When config, CI env and git all fail to name a default branch, resolution falls back to "the only
|
||||
branch in the graph". If that sole key was minted by a *narrowed* run on a feature branch (which phase
|
||||
four's B1 made a live possibility), the fallback now names a feature branch, and every other branch
|
||||
layers **its** results underneath. Probe: `withoutGit()` or `removeOrigin()` + no config, with a graph
|
||||
whose only key is `feature-x`.
|
||||
|
||||
### L10 — the state dir as an adversary
|
||||
|
||||
Beyond corrupt JSON (fixed in phase four by deleting it): a valid-JSON graph with `schema: 2`; a graph
|
||||
whose `files` and `edges` disagree; `results` entries with a `file` pointing outside the project root
|
||||
or at an absolute path from another machine; a `graph.json` that is a directory; a state dir with no
|
||||
write permission; `$HOME` unset. Each should degrade to "run the tests", never crash and never write
|
||||
garbage.
|
||||
|
||||
---
|
||||
|
||||
## Part 4 — Reporting
|
||||
|
||||
**Between passes**, not just at the end. Per finding:
|
||||
|
||||
- **Reproduced (yes / no / hypothesis only)**, with the exact command and the measured
|
||||
`tally` + `delta()->summary()` on **both** interpreters.
|
||||
- **Severity**, and say why in one line. The scale that matters here: *a test wrongly replayed as
|
||||
passing* (worst) > *cache silently useless, full suite forever* > *graph grows / stale data* >
|
||||
*cosmetic*.
|
||||
- **Fixed / deferred / needs-a-decision**, and the row that pins it.
|
||||
- For anything needing a decision: **one yes/no question**, no essay.
|
||||
|
||||
Close with:
|
||||
|
||||
1. The count of `tests/Features/Tia/*` green on **both** `php84` (no pcov) and `php85` (pcov), against
|
||||
the 72 baseline.
|
||||
2. Every row you added, and what invariant from Part 2 it defends.
|
||||
3. Which findings are still open, as yes/no questions.
|
||||
4. That `tests/.snapshots/success.txt` and the `tests/Visual/Parallel.php` tally need regenerating —
|
||||
**do not regenerate them.**
|
||||
5. **What you looked at and found solid.** A list of attacks that did not break anything is a real
|
||||
result: it tells the next phase where not to spend its time.
|
||||
@@ -0,0 +1,351 @@
|
||||
# TIA defect sweep — phase four
|
||||
|
||||
## Your task
|
||||
|
||||
Five defects in TIA's read/write path, found while building the repo's TIA scenario suite. One is
|
||||
confirmed and load-bearing (**B1**); four need a decision before a fix (**B2**–**B5**).
|
||||
|
||||
Work **one bug at a time, in order**, and for each:
|
||||
|
||||
1. **Reproduce it as a repo test first.** The reproduction is the deliverable even when the fix is
|
||||
deferred — a red test that pins the exact symptom is worth more than a prose report. Do not commit
|
||||
a red test to the suite; keep it in a scratch file until the fix lands (see Part 1.4).
|
||||
2. Confirm the measured numbers in this file still hold. They were taken at commit `4d3d0105` plus
|
||||
the two uncommitted changes described in Part 2. If a number has moved, say so and stop.
|
||||
3. Fix, then re-run **the whole `tests/Features/Tia/*` set on two interpreters** (Part 1.3).
|
||||
4. **HARD STOP after B1.** Report the diff and wait — B1's fix changes `Graph`'s read semantics for
|
||||
every caller, and Nuno wants to see it before B2–B5 pile on top.
|
||||
|
||||
Per `CLAUDE.md`: do not run `composer test`, and do not regenerate snapshots unless told. Do not
|
||||
commit. Do not touch the playground's `vendor/`.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — The harness
|
||||
|
||||
### 1.1 What exists
|
||||
|
||||
`tests/Features/Tia/*` scaffold a throwaway git project into a temp dir, run a **real `pest`
|
||||
subprocess** against it, and diff the TIA graph it wrote. Everything lives in
|
||||
`tests/Fixtures/Tia/`:
|
||||
|
||||
| Class | What it gives you |
|
||||
|---|---|
|
||||
| `Project` | `make(branch, overlay:)`, `withoutGit()`, `seed(branch, sentinel:, failing:)`, `pest(...$args)`, `pestWithEnvironment($dir, $env, ...$args)`, `pestIn($dir, ...)`, `write($rel, $contents)`, `graph()`, `branchKeys()`, `graphDir()`, `graphExists()`, `snapshot()`, `delta()`, `mutateGraph(fn)`, `addBaseline($branch)`, `worktree($branch)`, `destroyAll()` |
|
||||
| `GitRepo` (`$project->git()`) | `switchTo($b, new:)`, `rename($from, $to)`, `detach()`, `config($k, $v)`, `addOrigin()`, `removeOrigin()`, `setOriginHead($b)`, `unsetOriginHead()`, `worktree()`, `sha()`, `branchNames()` |
|
||||
| `GraphDelta` (`$project->delta()`) | `writtenCount()`, `added()`, `removed()`, `branchKeys()`, `baselineUntouched($b)`, `shaMoved()`, `treeMoved()`, `edgesMoved()`, `filesMoved()`, `fingerprintMoved()`, `structureMoved()`, `isResultsOnly()`, `isHardSuppressed()`, `summary()` |
|
||||
| `PestResult` (returned by `pest()`) | `replayed()`, `uncached()`, `affected()`, `tally()`, `output`, `exitCode`, `describe()` |
|
||||
|
||||
The fixture app is 3 test files / **6 tests** (`Project::TOTAL_TESTS`), with `Project::EDGES` and
|
||||
`Project::TESTS` describing the graph `seed()` writes. `Project::testId($file, $description)` builds
|
||||
a result key.
|
||||
|
||||
### 1.2 The sentinel discriminator — read this before writing an assertion
|
||||
|
||||
`seed()` writes a graph **and then rewrites every cached result to `time=9.999`,
|
||||
`assertions=42`** (non-zero assertion counts only — risky/skipped/incomplete statuses are *derived*
|
||||
from "performed no assertions", so falsifying those would rewrite the status on replay), and takes a
|
||||
snapshot. Therefore:
|
||||
|
||||
- `$delta->writtenCount()` is the only reliable way to tell **"replayed"** from **"executed and wrote
|
||||
back the same values"**. `0` means nothing was written.
|
||||
- `isResultsOnly()` = no structure moved, no `sha`/`tree` movement, nothing added or removed.
|
||||
- `isHardSuppressed()` = the graph is byte-identical.
|
||||
|
||||
Tiers, unchanged since phase two: **COMPLETE** may change everything · **RESULTS-ONLY** may change
|
||||
only `baselines[<branch>].results` for tests that ran, never removing an entry, never adding a result
|
||||
for a test file absent from `edges`, never touching `sha`/`tree`/`edges`/`files`/`fingerprint` ·
|
||||
**HARD-SUPPRESSED** may change nothing.
|
||||
|
||||
### 1.3 Running them
|
||||
|
||||
They are in the `integration` group (`tests/Pest.php:22`), so `composer test:unit` skips them. A
|
||||
**directory argument finds nothing** — pass files:
|
||||
|
||||
```bash
|
||||
F="tests/Features/Tia/DefaultBranchReplay.php tests/Features/Tia/DefaultBranchResolution.php \
|
||||
tests/Features/Tia/DefaultBranchWriteTier.php tests/Features/Tia/PartialRunWriteTier.php \
|
||||
tests/Features/Tia/CompleteRunWriteTier.php tests/Features/Tia/FilteredMode.php"
|
||||
|
||||
PAO_DISABLE=1 php84 bin/pest $F # 8.4.23, NO pcov — this is what CI has
|
||||
PAO_DISABLE=1 php bin/pest $F # 8.5.8, pcov — this is what your machine has
|
||||
```
|
||||
|
||||
**Both must be green.** `.github/workflows/tests.yml` sets `coverage: none`, so any assertion that
|
||||
depends on a coverage driver fails in CI while passing locally. Two tests were already caught by
|
||||
this. Concretely: a **cold recording run writes no graph at all** without pcov/xdebug (it prints
|
||||
`Running in TIA mode, however TIA is skipped as it needs ext-pcov or Xdebug`), and a **PHP source
|
||||
file edit** triggers `Detected PHP source changes but no coverage driver is available` → full suite,
|
||||
`affected=0`. Seed a graph instead of recording one, and edit *test* files rather than `app/` files,
|
||||
unless the point of the row is the driver itself.
|
||||
|
||||
### 1.4 Measure before you assert
|
||||
|
||||
Do not guess expectations from reading the code — every number in Part 3 came from a scratch probe.
|
||||
The pattern (put it in your scratchpad, not in the repo):
|
||||
|
||||
```php
|
||||
<?php
|
||||
require '/Users/nunomaduro/Work/projects/pestphp/pest/vendor/autoload.php';
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
$p = Project::make('master');
|
||||
$p->seed('master');
|
||||
$p->git()->switchTo('feature-x', new: true);
|
||||
$r = $p->pest('--tia');
|
||||
printf("tally=[%s] keys=[%s] %s\n", $r->tally(), implode(',', $p->branchKeys()), $p->delta()->summary());
|
||||
$p->destroy();
|
||||
```
|
||||
|
||||
`PAO_DISABLE=1 php probe.php`. One project per scenario; always `destroy()`.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — What the code looks like right now
|
||||
|
||||
Phase three landed default-branch resolution: `ChangedFiles::defaultBranch()`, the
|
||||
`pest()->tia()->defaultBranch()` config surface, `Graph::setFallbackBranch()` + `?string
|
||||
$fallbackBranch = null` on the seven read methods, and `Tia::resolveFallbackBranch()`
|
||||
(`Tia.php:~1776`) resolving **config → CI env (`CiDefaultBranch`) → git (`origin/HEAD`, then
|
||||
`init.defaultBranch` if the branch exists) → `soleRecordedBranch()`**, failing loudly when nothing
|
||||
can name it.
|
||||
|
||||
On top of that, **two uncommitted changes** you will see in `git diff`:
|
||||
|
||||
1. `TIA_RESULTS_ONLY` global — a *partial* parallel run with an existing graph now purges stale
|
||||
worker partials, sets the global, and workers flush their results through the existing
|
||||
`flushWorkerReplay()` / `mergeWorkerReplayPartials()` path; the parent writes them with
|
||||
`complete: false`. Gated on a graph already existing, so a TIA-less project still creates no
|
||||
baseline dir. This gave `--parallel --filter` parity with sequential — **and, per B1, handed it
|
||||
the shadowing bug too.**
|
||||
2. `loadGraph()` emits `WARN The dependency graph could not be read — it will be rebuilt.` once per
|
||||
parent process when `graph.json` exists but will not decode. Previously silent.
|
||||
|
||||
62 scenario tests cover this and pass on both interpreters.
|
||||
|
||||
---
|
||||
|
||||
## Part 3 — The defects
|
||||
|
||||
### B1 — a thin baseline key permanently shadows the default-branch fallback · **confirmed, priority 1**
|
||||
|
||||
**Symptom.** Any *narrowed* run on a new branch (`--filter`, `--group`, a path, `--bail`, `--shard`,
|
||||
and now `--parallel --filter`) writes a baseline key holding only the tests that ran. From then on
|
||||
`--tia` on that branch reads that thin key instead of falling back to the default branch, so
|
||||
everything else is uncached — **one full suite per branch, forever**, which is the exact cost
|
||||
issue [#1823](https://github.com/pestphp/pest/issues/1823) was about, re-entering through a side door.
|
||||
|
||||
**Measured** (fixture: 6 tests, graph seeded on `master`):
|
||||
|
||||
```
|
||||
switch -c feature-x; pest --filter="adds two numbers"
|
||||
→ keys=[master,feature-x] (feature-x holds 1 result)
|
||||
pest --tia
|
||||
→ 6 passed (6 assertions, 5 uncached, 1 replayed) ← want: 6 replayed, 0 uncached
|
||||
|
||||
same via: pest --parallel --processes=2 --filter="adds two numbers" → identical
|
||||
```
|
||||
|
||||
**Where.** `src/Plugins/Tia/Graph.php::baselineFor()` (~line 814):
|
||||
|
||||
```php
|
||||
if (isset($this->baselines[$branch])) return $this->baselines[$branch];
|
||||
if ($branch !== $fallbackBranch && isset($this->baselines[$fallbackBranch])) return $this->baselines[$fallbackBranch];
|
||||
```
|
||||
|
||||
The fallback is all-or-nothing: it fires only when the branch has **no** key at all. The key itself
|
||||
is minted by `Graph::setResult()` → `ensureBaseline($branch)` (~599 / ~829), reached from
|
||||
`Tia::snapshotTestResults()` on partial runs.
|
||||
|
||||
**Reproduction to add** (`tests/Features/Tia/DefaultBranchReplay.php`):
|
||||
|
||||
```php
|
||||
test('a narrowed run on a new branch does not cost the fallback', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
$project->pest('--filter=adds two numbers');
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($result->uncached())->toBe(0, $result->describe());
|
||||
})->skipOnWindows();
|
||||
```
|
||||
|
||||
Add the `--parallel --processes=2 --filter=…` variant as a second row (dataset), since the two write
|
||||
paths are different code.
|
||||
|
||||
**Fix direction.** Make the fallback **per entry** rather than per baseline: in `baselineFor()`,
|
||||
when the branch has its own baseline *and* a distinct fallback baseline exists, return
|
||||
`results` = the branch's results **layered over** the fallback's (branch wins per test id), and take
|
||||
`sha`/`tree` from the fallback when the branch's are `null`/empty. `baselineFor()` is the single
|
||||
funnel for `recordedAtSha()`, `lastRunTree()`, `getResult()`, `getTime()`, `getAssertions()`,
|
||||
`testFilesToRerun()` and `hasUnlocatedTestsToRerun()`, so one change covers every reader. An
|
||||
alternative — never mint a key from a partial run — is smaller but loses the executed result
|
||||
entirely, which regresses the parity just gained.
|
||||
|
||||
**Done when.** Both reproduction rows are green on both interpreters, and none of these move:
|
||||
|
||||
- `the branch that ran gets its own key and the default branch keeps its baseline` — writes stay on
|
||||
the real branch; the merge must be **read-only** and must not leak into `ensureBaseline()`/`setResult()`.
|
||||
- `a declared default branch that does not exist degrades to a full run` — with a fallback that names
|
||||
nothing, a branch's own thin results must still be all you get.
|
||||
- `a detached HEAD replays without minting a branch key`, `writes nothing on a second run on the same
|
||||
branch`, `filtered mode finds nothing to do…` (both rows) — a merged read must not make a clean
|
||||
replay start writing.
|
||||
- The whole `PartialRunWriteTier.php` / `CompleteRunWriteTier.php` set — tier semantics are unchanged
|
||||
by this fix.
|
||||
|
||||
---
|
||||
|
||||
### B2 — a partial run on detached HEAD writes into the default branch's baseline · **needs a decision**
|
||||
|
||||
**Symptom.** With `HEAD` detached, `Tia::resolveBranch()` (`Tia.php:~1756`) sets
|
||||
`$this->branch = $changedFiles->currentBranch() ?? $this->fallbackBranch` — and that branch is used
|
||||
for **writes**. A `--tia` run in this state happens to be harmless (a clean replay writes nothing),
|
||||
but any run that *executes* tests writes their results into the default branch's baseline.
|
||||
|
||||
**Measured.**
|
||||
|
||||
```
|
||||
seed on master; git checkout --detach; pest --filter="adds two numbers"
|
||||
→ keys=[master] w=1 struct:ok ← master's baseline rewritten from a detached checkout
|
||||
seed on master; git checkout --detach; pest --tia
|
||||
→ keys=[master] w=0 ← read-only, as intended
|
||||
```
|
||||
|
||||
**The decision.** `PLAN_PHASE_THREE.md` §2.3 **D3** recommended detached HEAD be *read-only*. If that
|
||||
still stands, suppress writes when `currentBranch()` is `null` (a dedicated flag — note
|
||||
`resultsOnlyWrites` is **not** enough, it still writes results). If Nuno prefers the current
|
||||
behaviour, add a test pinning it and close this out.
|
||||
|
||||
**Reproduction** (`tests/Features/Tia/DefaultBranchWriteTier.php`), written for the read-only answer:
|
||||
|
||||
```php
|
||||
test('a detached HEAD does not write into the default branch baseline', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->detach();
|
||||
$project->pest('--filter=adds two numbers');
|
||||
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($delta->baselineUntouched('master'))->toBeTrue($delta->summary())
|
||||
->and($project->branchKeys())->toBe(['master']);
|
||||
})->skipOnWindows();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### B3 — an unreadable graph is never repaired on a machine with no coverage driver · **needs a decision**
|
||||
|
||||
**Symptom.** A corrupt `graph.json` is now *reported* (Part 2, change 2) but only *rebuilt* when a
|
||||
coverage driver is present, because rebuilding means recording. Driverless, the file stays corrupt
|
||||
run after run and TIA is silently inert until someone deletes it by hand — while the WARN claims
|
||||
`it will be rebuilt`.
|
||||
|
||||
**Measured** (`php84`, no pcov):
|
||||
|
||||
```
|
||||
overwrite graph.json with '{not json'
|
||||
run 1: exit=0, 6 passed, file still '{not json'
|
||||
run 2: exit=0, 6 passed, file still '{not json'
|
||||
headline: "Running in TIA mode, however TIA is skipped as it needs ext-pcov or Xdebug"
|
||||
```
|
||||
|
||||
**Options.** (a) delete the file when it cannot be decoded, so the next drivered run starts clean and
|
||||
the state dir does not carry a permanent landmine; (b) keep the file but reword the WARN when no
|
||||
driver is available. (a) is the honest one and costs one `State::delete()`.
|
||||
|
||||
**Reproduction** (`tests/Features/Tia/FilteredMode.php`, extending the existing corrupt-graph row):
|
||||
|
||||
```php
|
||||
expect($result->output)->toContain('The dependency graph could not be read')
|
||||
->and(file_get_contents($project->graphDir().'/graph.json'))->not->toBe('{not json');
|
||||
```
|
||||
|
||||
Must pass on **both** interpreters — that is the whole point of the row.
|
||||
|
||||
---
|
||||
|
||||
### B4 — a complete `--parallel` run writes nothing and prunes nothing · **needs a decision**
|
||||
|
||||
**Symptom.** With a graph present and TIA not flagged, a sequential run refreshes results and applies
|
||||
the prune; the same run under `--parallel` does neither, because the parent's `ResultCollector` is
|
||||
empty (results live in the workers) and workers only flush when they were told to record, replay, or
|
||||
— since Part 2's change — results-only for a *partial* run. So parallel CI contributes nothing to the
|
||||
cache, and a deleted test's entry survives forever.
|
||||
|
||||
**Measured** (graph seeded on `master`, sentinelled):
|
||||
|
||||
```
|
||||
pest → w=6 +0 -0 struct:ok (sequential baseline)
|
||||
pest --parallel --processes=2 → w=0 +0 -0 struct:ok ← writes nothing
|
||||
delete a test, then:
|
||||
pest --parallel --processes=2 → w=0 +0 -0 struct:ok ← and does not prune (sequential gives -1)
|
||||
pest --tia --parallel --processes=2 → w=0 ← correct: everything replayed
|
||||
```
|
||||
|
||||
**The decision.** Extending the `TIA_RESULTS_ONLY` mechanism to complete parallel runs is
|
||||
mechanically easy, but a *complete* run also prunes, and pruning from merged worker partials is the
|
||||
risky half: a worker that dies, or a shard that never ran, would look like "these tests no longer
|
||||
exist". If it is done, the prune must key off "every worker reported" and fall back to
|
||||
results-only when it cannot prove that. `PLAN.md` §5 lists this as a known gap, not a regression.
|
||||
|
||||
**Reproduction** (`tests/Features/Tia/CompleteRunWriteTier.php`) — mirror the two sequential rows
|
||||
that already exist (`a complete run prunes a deleted test`, `--no-tia refreshes results…`) with
|
||||
`--parallel --processes=2` added, and assert the same deltas.
|
||||
|
||||
---
|
||||
|
||||
### B5 — G4 ("parallel replay clobbers cached `time`") no longer reproduces · **verify, then correct the record**
|
||||
|
||||
**Symptom.** `PLAN_PHASE_THREE.md` §4.6 lists as still-present: *"parallel replay clobbers cached
|
||||
`time` on all non-executed tests — `mergeWorkerReplayPartials()` takes `$result['time']` verbatim,
|
||||
never routing through `resultTime()`"*. The repo fixture disagrees: after a parallel replay the
|
||||
sentinelled `time=9.999` / `assertions=42` survive on every non-executed test.
|
||||
|
||||
**Measured.** `a parallel run merges worker results into the parent baseline` edits one test file,
|
||||
then runs `--tia --parallel --processes=2`: `2 affected, 4 replayed, w=2`. If replayed times were
|
||||
being clobbered, `w` would be `6`.
|
||||
|
||||
**Why the record may be stale.** `flushWorkerReplay()` (`Tia.php:~1286`) already applies
|
||||
`resultTime()` **worker-side** before writing the partial, so the parent's verbatim read is reading
|
||||
values that were already corrected.
|
||||
|
||||
**What to do.** Either find a shape where it still reproduces (the playground has 25 tests and real
|
||||
timings; the fixture has 6 and may be too small), or confirm it is fixed and strike it from §4.6.
|
||||
Add a direct row either way:
|
||||
|
||||
```php
|
||||
test('a parallel replay keeps the recorded time of tests that did not run', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->pest('--tia', '--parallel', '--processes=2');
|
||||
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($delta->writtenCount())->toBe(0, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 4 — Reporting
|
||||
|
||||
Per bug: **reproduced (yes/no)** with the measured delta, **fixed (yes/no/deferred)**, and the test
|
||||
that now pins it. Close with:
|
||||
|
||||
1. Whether all `tests/Features/Tia/*` are green on **both** `php84` (no pcov) and `php85` (pcov).
|
||||
2. Which of B2–B5 still need Nuno's decision, phrased as a yes/no question each.
|
||||
3. Whether `tests/.snapshots/success.txt` and the `tests/Visual/Parallel.php` tally now need
|
||||
regenerating (they will, if you added rows) — **do not run `composer update:snapshots` unless
|
||||
asked.**
|
||||
4. Anything you found that is not in this file.
|
||||
|
||||
Leave the tree uncommitted and the scratch probes out of the repo.
|
||||
@@ -0,0 +1,386 @@
|
||||
# TIA deep audit — phase six
|
||||
|
||||
## Your task
|
||||
|
||||
Phase five went looking rather than fixing what was reported, and found nine defects in the
|
||||
**read/write path and the state dir**. All nine are fixed and pinned by rows in the repo. It stopped
|
||||
there deliberately: roughly half of TIA was never opened. Phase six is that other half.
|
||||
|
||||
Two deliverables, both required:
|
||||
|
||||
1. **A ranked findings list.** Every finding needs a *reproduction*, not a reading of the code. A
|
||||
finding you cannot reproduce is a hypothesis — say so and rank it separately.
|
||||
2. **New scenario tests** in `tests/Features/Tia/*`. Rows that pass go in the repo (they are the
|
||||
regression net). Rows that fail stay in your scratchpad until the fix lands — **never commit a red
|
||||
test.**
|
||||
|
||||
Work in passes, and **report between passes** rather than at the very end:
|
||||
|
||||
- **Pass A** — reproduce the leads in Part 3. Report which are real.
|
||||
- **Pass B** — your own hunt: the invariants in Part 2, attacked with inputs nobody tried.
|
||||
- **Pass C** — fixes, smallest first, each with the row that pins it.
|
||||
|
||||
Fix what is clearly a defect with an obvious correct answer. **Stop and ask** when the fix is a
|
||||
behaviour *choice* — those are Nuno's calls, and a reproduction with a crisp yes/no question is worth
|
||||
more than a guessed fix.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — Where things stand
|
||||
|
||||
### 1.1 What TIA is
|
||||
|
||||
Test Impact Analysis. `pest --tia` records a dependency graph (test file → source files it touched)
|
||||
plus a per-branch baseline of results, then on later runs replays the tests whose dependencies did not
|
||||
change. State lives in a per-project dir under `~/.pest/tia/`; `graph.json` is the whole thing.
|
||||
|
||||
Source of truth: `src/Plugins/Tia.php` (the plugin) and `src/Plugins/Tia/Graph.php` (the graph model +
|
||||
read/write API). Supporting: `ChangedFiles.php` (git), `Fingerprint.php` (environment/structure
|
||||
hashing), `Storage.php` / `FileState.php` (the state dir), `ResultCollector.php` (what a run observed),
|
||||
`Recorder.php` + `CoverageCollector.php` (how edges are recorded).
|
||||
|
||||
Read `PLAN.md`, then `PLAN_PHASE_TWO.md` → `THREE` → `FOUR` → `FIVE` for the history and the tier
|
||||
contract. Struck-through rows are fixed; do not re-report them. **Section 1.2 below supersedes any
|
||||
phase-five row that contradicts it.**
|
||||
|
||||
### 1.2 What phase five settled — do not re-report these
|
||||
|
||||
Nine defects, all fixed, each with a row that fails if it comes back:
|
||||
|
||||
| # | Defect | Fix | Pinned by |
|
||||
|---|---|---|---|
|
||||
| 1 | `pruneStaleResults()` unset an entry; the next read layered the default branch's entry for the same test id back in, so a renamed/removed test stayed "previously unsuccessful" on a feature branch forever | `Graph::baselineFor()` layers per *file* once a branch has had a complete run (new `complete` flag on the baseline); a key minted by a narrowed run keeps the per-test-id merge | `StateReclamation` → *a pruned result does not come back from the fallback*, *the fallback still reaches a branch that has never run a test file* |
|
||||
| 2 | A cached failure whose test file was deleted was never reclaimed, so `--filtered` degraded to a full replay on every later run | `hasUnlocatedTestsToRerun()` widens only for a path it cannot *address*; `Graph::pruneResultsForMissingFiles()` + `pruneMissingTests()` run on every complete write | `StateReclamation` → *a cached failure whose test file was deleted stops widening later runs*, *a complete run reclaims the entry and the edge of a deleted test file* |
|
||||
| 3 | A detached HEAD is read-only for writes, but three paths still *deleted*: structural drift, `--fresh` (`Storage::purge`), and the corrupt-graph discard. The checkout that wiped the baseline could never rebuild it | `Tia::deleteState()` no-ops when detached; `Storage::purge` guarded | `StateReclamation` → the three *a detached HEAD does not purge…* rows |
|
||||
| 4 | A status int outside 0–8 became `TestStatus::unknown()`, which `ReplayType` folded into `Failure` — a green test went red, exit 1 | `Graph::getResult()` returns `null` for an unknown status (re-run, don't replay); `shouldRerunStatus()` treats unknown as re-run | `HostileState` → *a cached status this build cannot interpret is re-run, not replayed* |
|
||||
| 5 | Notice/deprecation/warning (3/4/6) decode fine but `ReplayType` had no case, so they also folded into `Failure` | Explicit `Pass` cases — those statuses only reach replay when the configured `failOn*` / `displayDetailsOn*` policies say they do not matter | `HostileState` → *a cached status with no replay of its own does not fail the run* |
|
||||
| 6 | `Graph::decode()` took `baselines`, `edges` and `files` verbatim; one malformed entry raised a `TypeError` inside a test | `decodeBaselines()` / `decodeResults()` / `decodeEdges()` / `decodeFiles()` validate every field. Numeric-looking keys are cast, not filtered — a branch named `12345` decodes as an `int` | `HostileState` → *a graph whose shape is wrong everywhere is repaired rather than trusted* |
|
||||
| 7 | Baseline keys grew forever — one full copy of the suite per branch ever created | `ChangedFiles::branchNames()` (local + remote refs) + `Graph::pruneMissingBranches()`, on complete writes only, and only when the fallback branch is visible in the refs | `BranchShapes` → *deleting many branches reclaims every one of their baselines* and the four rows around it |
|
||||
| 8 | A test that triggered a deprecation was recorded as `status=0`, because PHPUnit emits `Passed` for it and TIA had no issue subscribers. A fresh `--fail-on-deprecation` run exited 1; the replayed one exited 0 | Six subscribers (`Notice`/`PhpNotice`/`Deprecation`/`PhpDeprecation`/`Warning`/`PhpWarning`) feed `ResultCollector`; most-important-status-wins; a plain `Passed` does not downgrade a triggered issue; `@`-suppressed issues are ignored | `IssueStatuses` → *a triggered issue is recorded as itself, not as a pass*, *a cached deprecation still fails the run that asked to fail on one* |
|
||||
| 9 | A replay wrote back the status it *looked* like from outside, so a cached deprecation replaying as a pass was persisted as `0` — defect 8's fix eroded after one run | `Tia::replayedAsRecorded()` writes back the cached status and message for replayed tests, in the sequential path and in the worker flush | `IssueStatuses` → *replaying a cached issue does not downgrade it to a pass* |
|
||||
| 10 | A run torn down mid-file (an `exit()` inside a test) still flushed what it had, and the parent read that as licence to prune the siblings it never reached. Sequential and parallel disagreed | `ResultCollector::hasUnfinishedTest()` demotes such a run to results-only, in `terminate()` (the shutdown path) and in `addOutput()` | `StateReclamation` → *a run torn down mid-file does not prune the tests it never reached* |
|
||||
|
||||
**One existing assertion was changed.** `DefaultBranchWriteTier > filtered mode falls back to a full
|
||||
replay when a cached failure cannot be located` used `tests/Unit/DeletedTest.php` — a path that
|
||||
resolves but does not exist, which is precisely the shape behind defect 2. It now points at
|
||||
`/build/agent/…`, so it still pins the widening safety net for the case where widening can help. If
|
||||
you disagree with that reading, that is a finding, not a licence to change it back quietly.
|
||||
|
||||
### 1.3 Attacked in phase five and found solid — do not spend time here again
|
||||
|
||||
- **Parity.** Sequential vs `--processes=1/2/8` across replay-with-edit, `--filtered` with a cached
|
||||
failure, a first run on a feature branch, `--bail`, `--stop-on-failure`, `--compact`: identical
|
||||
tally *and* identical graph delta every time.
|
||||
- **Hostile state dir.** Empty / truncated / not-JSON / JSON scalar / JSON list / `null` / `{}` / NUL
|
||||
bytes; `graph.json` as a directory; a read-only state dir; dangling and negative edge ids; a result
|
||||
`file` pointing outside the project; `schema: 2`. All degrade to "run the tests", exit 0.
|
||||
- **Status int mapping.** `ResultCollector` (`asInt()`), `Graph::getResult()` and `TestStatus::from()`
|
||||
agree exactly on 0–8. No off-by-one.
|
||||
- **`FileState::write`** is tmp + rename, so concurrent runs cannot tear a file. Racing runs are
|
||||
last-writer-wins **by design** — treat as a decision, not a bug, unless you can show data loss
|
||||
worse than that.
|
||||
- **Branch names.** Slashes, dots, unicode, digits-only, 180 characters, case-only differences,
|
||||
remote-only branches, worktree branches: all keyed and reclaimed correctly.
|
||||
- **`sha`/`tree` layering.** No shape found that under-reports changed files. The fallback `tree` only
|
||||
drops a file whose current content hashes identically to what the fallback ran with, which is sound.
|
||||
|
||||
### 1.4 Known and deliberately unfixed
|
||||
|
||||
| Item | Why it is still open |
|
||||
|---|---|
|
||||
| Reclamation is skipped when a complete run executes **zero** tests | Stale edges linger until a run that executes at least one test. Writing the graph from a run that produced nothing costs more than it buys. Revisit if you find a real project stuck there. |
|
||||
| SIGINT never reaches the suite (`PcovRestarter` re-exec) | Real, but signal handling outside the TIA read/write path. Needs its own pass. |
|
||||
| Orphaned state dirs when the origin URL changes | `Storage::projectKey()` keys on the origin identity so clones share a graph. Changing the remote silently moves TIA to a fresh dir and nothing reclaims the old one. Unbounded growth in `~/.pest/tia`. |
|
||||
| `environmental` fingerprint holds only `php_minor` | Nuno declined. The `clearResults()` drift path is therefore latent — it becomes live the moment that bucket grows. **Do not re-flag the `PHP_MAJOR_VERSION` line itself.** |
|
||||
| PLAN.md §3 (raw coverage flags leave filtered mode on) | **No longer reproduces** — `coverageReportActive()` consults `COVERAGE_REPORT_FLAGS` now. Struck. |
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — The invariants to attack
|
||||
|
||||
Same list as phase five; 1, 2, 4, 5, 7 and 8 are now well covered, so weight your effort toward 3 and
|
||||
6 and toward the *recording* half of the system, which nothing below the plugin has ever probed.
|
||||
|
||||
1. **Parity.** `pest <args>` and `pest --parallel --processes=N <args>` must leave the same graph and
|
||||
reach the same tally. Hard rule. `Project::SEQUENTIAL_AND_PARALLEL` encodes it; use it on every new
|
||||
write-path row.
|
||||
2. **The tiers hold.** COMPLETE / RESULTS-ONLY / HARD-SUPPRESSED, exactly one each, no leaking.
|
||||
3. **Replay is faithful.** ← *weak spot.* A replayed test reports the same status, message, time and
|
||||
assertion count as the recorded run. Phase five fixed statuses; **edges** are unproven.
|
||||
4. **Reads never write.**
|
||||
5. **A branch never corrupts another branch's baseline.**
|
||||
6. **Nothing is unbounded.** ← *weak spot.* Baseline keys are reclaimed now; `files`, `edges`, worker
|
||||
partials, coverage caches and orphaned state dirs are not.
|
||||
7. **A hostile state dir cannot break a run.**
|
||||
8. **Git shapes are all handled.**
|
||||
|
||||
---
|
||||
|
||||
## Part 3 — Leads to reproduce first (Pass A)
|
||||
|
||||
Ranked by expected value. Everything here is **unexplored**, not merely unfixed — phase five never
|
||||
opened these files.
|
||||
|
||||
### M1 — the recording path with a real coverage driver · **highest value**
|
||||
|
||||
Everything phase five did was driver-independent by design (seed a graph, never record one). Nothing
|
||||
verified that a *recorded* graph is correct. This is the biggest blind spot in the audit.
|
||||
|
||||
- Do the recorded edges match what the test actually touched? Record with pcov, then hand-check
|
||||
`edges` against the source files each fixture test uses.
|
||||
- `PLAN.md` §4: **`pest --tia --coverage` narrows edges** — `Feature/ExampleTest` recorded with 2
|
||||
files instead of 16, dropping self-edges. Same observable shape as the parallel bug G12 that phase
|
||||
four closed, likely a different mechanism (the piggyback collector is scoped by `phpunit.xml
|
||||
<include>`, the pcov-restarted recorder is not). **Confirm whether these are one fix or two.**
|
||||
- `Recorder::activateLinkTracking()` (piggyback) vs `activate()` (pcov restart) must produce the same
|
||||
edge set for the same suite. Compare them directly.
|
||||
- `keepExisting: $this->piggybackCoverage` in `replaceEdges()` — what happens to a test whose edges
|
||||
genuinely shrank while piggybacking?
|
||||
|
||||
Rows for this **must** be `->skipOnPhpVersionsWithoutCoverage()`-style guarded, or seeded, or CI goes
|
||||
red on `php84`. That constraint is why phase five skipped it; solve it deliberately rather than by
|
||||
accident. Adding a coverage-driver guard helper to the fixture is a legitimate deliverable.
|
||||
|
||||
### M2 — `BaselineSync` (621 lines, never opened)
|
||||
|
||||
The remote-baseline fetch is the only path where a graph arrives from **another machine**, which is
|
||||
exactly where the hostile-state work matters most and where none of it has been exercised.
|
||||
|
||||
- A fetched baseline whose `fingerprint` matches but whose `files`/`edges` describe a different tree.
|
||||
- A fetched baseline recorded on a branch this checkout does not have.
|
||||
- `fetchIfAvailable()` under a broken network, a 404, a truncated download, a non-gzip body.
|
||||
- `KEY_FETCH_COOLDOWN` — does it bound retries, and does a corrupt cooldown file break a run?
|
||||
- Interaction with defect 7's branch GC: a fetched baseline carries branch keys this clone has never
|
||||
heard of. **They will be pruned on the next complete write.** Is that right, or must fetched keys be
|
||||
exempt? This is a real question, answer it.
|
||||
|
||||
### M3 — selection paths nobody has probed
|
||||
|
||||
`Graph::affected()` is ~600 lines and phase five only exercised the plain PHP-edge path.
|
||||
|
||||
- **Migrations** → `TableExtractor` → `testTables` intersection. What happens with an unparseable
|
||||
migration, a migration that drops a table, a squashed schema dump?
|
||||
- **Blade** — `bladeAncestorsFor()` walks `@include`/`@extends`/`<x-*>` transitively. Cycles?
|
||||
Depth? A component referenced only dynamically?
|
||||
- **Inertia** — `componentForInertiaPage()`, `jsFileToComponents`, `JsModuleGraph::buildStrict()`.
|
||||
What if `vite` is missing, or the resolver returns garbage?
|
||||
- **`usesSiblingHeuristicForUnknownPhp()`** — a hard-coded list of Laravel directories. A changed file
|
||||
in `app/Providers/` widens to every test whose deps share that directory. Measure how much that
|
||||
over-selects on a real tree.
|
||||
- **Arch tests** — `testSourceDeclaresArchGroup()` greps the source with three regexes. False
|
||||
positives (the string `arch(` in a comment) select the file on *every* PHP source change.
|
||||
|
||||
### M4 — git shapes phase five left alone
|
||||
|
||||
- A repo with **no commits yet** (`currentSha()` returns null / git fails).
|
||||
- **Submodules** — a changed file inside one; `git status --porcelain` reports the submodule path.
|
||||
- A repo whose root is **above** the pest project — `TiaRequiresRepositoryRoot` panics deliberately;
|
||||
confirm it still panics and writes nothing.
|
||||
- **Shallow / single-branch CI checkouts.** Defect 7's guard (`fallbackBranch` must be visible in the
|
||||
refs) was reasoned about, not measured. Build one and check nothing is over-pruned.
|
||||
- A branch **behind** the recorded sha, so `merge-base --is-ancestor` fails and the graph is declared
|
||||
unreachable. Does it recover, or thrash?
|
||||
|
||||
### M5 — a real concurrent race
|
||||
|
||||
Phase five established `FileState::write` is atomic and called it last-writer-wins by design. Nobody
|
||||
launched two runs. Launch them: a watcher plus a manual run, two `--tia` processes on one project,
|
||||
`--parallel` where the parent writes while a straggler worker flushes. Look for **loss worse than
|
||||
last-writer-wins** — a partially-merged baseline, a pruned entry from a run that never saw the file,
|
||||
worker partials from run A consumed by run B (`KEY_WORKER_*` are not namespaced per run).
|
||||
|
||||
That last one is the sharpest: `purgeWorkerPartials()` deletes *all* partials by prefix, so two
|
||||
concurrent parallel runs will eat each other's. Probe it.
|
||||
|
||||
### M6 — the unbounded remainder
|
||||
|
||||
Defect 7 reclaimed baseline keys. These still grow:
|
||||
|
||||
- `files` and `edges` — a deleted *source* file's entry is never removed (`pruneMissingTests()` only
|
||||
covers test files). Every rename leaves an orphan id forever.
|
||||
- Orphaned state dirs under `~/.pest/tia` (see 1.4).
|
||||
- `KEY_COVERAGE_CACHE` / `KEY_COVERAGE_MARKER` — who deletes them, and when?
|
||||
- Worker partials when a worker dies before `terminate()`.
|
||||
|
||||
Measure the growth on a realistic tree before proposing anything. Then **ask** — a cap or a GC is a
|
||||
behaviour choice.
|
||||
|
||||
---
|
||||
|
||||
## Part 4 — The harness
|
||||
|
||||
### 4.1 The scenario suite
|
||||
|
||||
`tests/Features/Tia/*` scaffold a throwaway git project into a temp dir, run a **real `pest`
|
||||
subprocess** against it, and diff the graph it wrote. Everything lives in `tests/Fixtures/Tia/`:
|
||||
|
||||
| Class | What it gives you |
|
||||
|---|---|
|
||||
| `Project` | `make(branch, overlay:)`, `withoutGit()`, `seed(branch, sentinel:, failing:)`, `seedFor($root, …)`, `pest(...$args)`, `pestWithEnvironment($dir, $env, ...$args)`, `pestIn($dir, ...)`, `write($rel, $contents)`, `path($rel)`, `graph()`, `branchKeys()`, `graphDir()`, `graphExists()`, `snapshot()`, `delta()`, `mutateGraph(fn)`, `addBaseline($branch)`, `worktree($branch)`, `destroy()`, `destroyAll()`, `SEQUENTIAL_AND_PARALLEL` |
|
||||
| `GitRepo` (`$project->git()`) | `switchTo($b, new:)`, `rename($from, $to)`, `detach()`, `commit($msg)`, `config($k, $v)`, `addOrigin()`, `removeOrigin()`, `setOriginHead($b)`, `unsetOriginHead()`, `worktree()`, `sha()`, `branchNames()`, `run([...])` for anything else |
|
||||
| `GraphDelta` (`$project->delta()`) | `writtenCount()`, `added()`, `removed()`, `branchKeys()`, `baselineUntouched($b)`, `shaMoved()`, `treeMoved()`, `edgesMoved()`, `filesMoved()`, `fingerprintMoved()`, `structureMoved()`, `isResultsOnly()`, `isHardSuppressed()`, `summary()` |
|
||||
| `PestResult` (returned by `pest()`) | `replayed()`, `uncached()`, `affected()`, `tally()`, `output`, `exitCode`, `describe()` |
|
||||
|
||||
The fixture app is 3 test files / **6 tests** (`Project::TOTAL_TESTS`); `Project::EDGES` and
|
||||
`Project::TESTS` describe the graph `seed()` writes. `Project::testId($file, $description)` builds a
|
||||
result key. Overlays in `tests/Fixtures/Tia/overlays/<name>/` supply a different `tests/Pest.php` —
|
||||
that is how you configure `pest()->tia()->…` for a scenario.
|
||||
|
||||
**If the fixture cannot express your case, extend the fixture.** A new `GitRepo` verb, a new overlay,
|
||||
or a coverage-driver guard is a legitimate part of the deliverable. Do not water down a scenario to
|
||||
fit the current helpers. Phase five added `GitRepo::commit()` usage, numeric-key handling in
|
||||
`Project::branchKeys()` and `GraphDelta`, and used `$project->write()` to author test files inline —
|
||||
follow that pattern.
|
||||
|
||||
### 4.2 The sentinel discriminator — read before writing any assertion
|
||||
|
||||
`seed()` writes a graph **and then rewrites every cached result to `time=9.999`, `assertions=42`**
|
||||
(only where assertions are non-zero — risky/skipped/incomplete statuses are *derived* from "performed
|
||||
no assertions", so falsifying those would rewrite the status on replay), then snapshots. Therefore:
|
||||
|
||||
- `$delta->writtenCount()` is the only reliable way to tell **"replayed"** from **"executed and wrote
|
||||
back the same values"**. `0` means nothing was written.
|
||||
- `isResultsOnly()` = no structure moved, no `sha`/`tree` movement, nothing added or removed.
|
||||
- `isHardSuppressed()` = the graph is byte-identical.
|
||||
- A replayed suite reports inflated assertion counts (6 tests × 42). That is the sentinel, not a bug.
|
||||
|
||||
The three write tiers, unchanged since phase two:
|
||||
|
||||
- **COMPLETE** — may change everything.
|
||||
- **RESULTS-ONLY** — may change only `baselines[<branch>].results` for tests that ran; never removes
|
||||
an entry, never adds a result for a test file absent from `edges`, never touches
|
||||
`sha`/`tree`/`edges`/`files`/`fingerprint`.
|
||||
- **HARD-SUPPRESSED** — may change nothing.
|
||||
|
||||
One addition from phase five: a complete run on a **non-default** branch also sets
|
||||
`baselines[<branch>].complete = true`. It is deliberately *not* set on the default branch, so a clean
|
||||
green run there stays byte-identical.
|
||||
|
||||
### 4.3 Running them
|
||||
|
||||
They are in the `integration` group (`tests/Pest.php`), so `composer test:unit` skips them. **A
|
||||
directory argument finds nothing** — pass files, space-separated, as separate argv entries:
|
||||
|
||||
```bash
|
||||
PAO_DISABLE=1 php84 bin/pest tests/Features/Tia/BranchShapes.php tests/Features/Tia/CompleteRunWriteTier.php \
|
||||
tests/Features/Tia/DefaultBranchReplay.php tests/Features/Tia/DefaultBranchResolution.php \
|
||||
tests/Features/Tia/DefaultBranchWriteTier.php tests/Features/Tia/FilteredMode.php \
|
||||
tests/Features/Tia/HostileState.php tests/Features/Tia/IssueStatuses.php \
|
||||
tests/Features/Tia/PartialRunWriteTier.php tests/Features/Tia/StateReclamation.php
|
||||
|
||||
PAO_DISABLE=1 php bin/pest <same list>
|
||||
```
|
||||
|
||||
**Both interpreters must be green.** `php84` is 8.4.x with **no pcov** — that is what CI has
|
||||
(`.github/workflows/tests.yml` sets `coverage: none`). `php` is 8.5.x with pcov — that is the dev
|
||||
machine. Any assertion that depends on a coverage driver passes locally and fails in CI. Concretely:
|
||||
|
||||
- A **cold recording run writes no graph at all** without pcov/xdebug (it prints `Running in TIA mode,
|
||||
however TIA is skipped as it needs ext-pcov or Xdebug`). **Seed a graph, never record one**, unless
|
||||
the driver *is* the point of the row — see M1, which has to solve this properly.
|
||||
- A **PHP source file edit** driverless triggers `Detected PHP source changes but no coverage driver
|
||||
is available` → full suite, `affected=0`. Edit *test* files, not `app/` files.
|
||||
- The `terminate()` path differs by driver: with pcov the plugin reaches the complete write through
|
||||
the shutdown handler, without it the run exits earlier. Defect 10 only reproduced on `php`. **Run
|
||||
every probe on both before believing it.**
|
||||
|
||||
`PAO_DISABLE=1` is mandatory on every pest invocation and every probe: `laravel/pao` emits JSON under
|
||||
agents and corrupts the captured output.
|
||||
|
||||
**Baseline: 161 scenario rows green on both interpreters**, at `b49ba062`. Per file:
|
||||
|
||||
| File | Rows |
|
||||
|---|---|
|
||||
| `StateReclamation.php` | 37 |
|
||||
| `HostileState.php` | 25 |
|
||||
| `CompleteRunWriteTier.php` | 17 |
|
||||
| `DefaultBranchResolution.php` | 14 |
|
||||
| `BranchShapes.php` | 14 |
|
||||
| `DefaultBranchReplay.php` | 13 |
|
||||
| `IssueStatuses.php` | 13 |
|
||||
| `PartialRunWriteTier.php` | 12 |
|
||||
| `DefaultBranchWriteTier.php` | 10 |
|
||||
| `FilteredMode.php` | 6 |
|
||||
|
||||
Plus 80 unit/arch rows (`tests/Unit/Plugins/Tia/*`, `tests/Arch.php`). If those numbers do not
|
||||
reproduce, stop and say so.
|
||||
|
||||
### 4.4 Measure before you assert
|
||||
|
||||
Never guess an expectation from reading the code. Probe first, in your scratchpad:
|
||||
|
||||
```php
|
||||
<?php
|
||||
require '/Users/nunomaduro/Work/projects/pestphp/pest/vendor/autoload.php';
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
$p = Project::make('master');
|
||||
$p->seed('master');
|
||||
$p->git()->switchTo('feature-x', new: true);
|
||||
$r = $p->pest('--tia');
|
||||
printf("tally=[%s] keys=[%s] %s\n", $r->tally(), implode(',', $p->branchKeys()), $p->delta()->summary());
|
||||
$p->destroy();
|
||||
```
|
||||
|
||||
`PAO_DISABLE=1 php probe.php`. One project per scenario; always `destroy()`; `Project::destroyAll()`
|
||||
at the end. Run every probe on **both** interpreters before believing it.
|
||||
|
||||
---
|
||||
|
||||
## Part 5 — Ground rules
|
||||
|
||||
- **Do not** run `composer test`. It takes minutes and you do not need it.
|
||||
- **`tests/.snapshots/success.txt` and the tally in `tests/Visual/Parallel.php` are stale right now** —
|
||||
phase five added 89 rows and did not regenerate them. **Report that they need regenerating and let
|
||||
Nuno run `composer update:snapshots`.** Do not run it yourself.
|
||||
- **Do not commit.** Leave the tree dirty.
|
||||
- **Do not touch the playground's `vendor/`.** If a finding truly needs the playground (25 real tests,
|
||||
real timings), say so and ask — syncing it is a manual step Nuno owns.
|
||||
- Run `vendor/bin/phpstan analyse <the files you touched> --memory-limit=-1 --no-progress` and
|
||||
`vendor/bin/pint <the files you touched>` before reporting. **Scope both to files you changed** —
|
||||
Nuno edits `src/` live, and a repo-wide fixer will revert his work.
|
||||
- The fixture projects **hardlink `src/`**. If you edit `src/` while a scenario run is in flight,
|
||||
those rows go red for no reason. Finish the edit, then run.
|
||||
- Keep scratch probes out of the repo.
|
||||
- **Never weaken an existing assertion to make something pass.** If an existing row contradicts your
|
||||
fix, that is a finding: report the contradiction and ask. Phase five hit this once (see 1.2) and
|
||||
rewrote the row to pin the *narrower* contract rather than deleting it — that is the bar.
|
||||
|
||||
---
|
||||
|
||||
## Part 6 — Reporting
|
||||
|
||||
**Between passes**, not just at the end. Per finding:
|
||||
|
||||
- **Reproduced (yes / no / hypothesis only)**, with the exact command and the measured
|
||||
`tally` + `delta()->summary()` on **both** interpreters.
|
||||
- **Severity**, and say why in one line. The scale that matters here: *a test wrongly replayed as
|
||||
passing* (worst) > *cache silently useless, full suite forever* > *graph grows / stale data* >
|
||||
*cosmetic*.
|
||||
- **Fixed / deferred / needs-a-decision**, and the row that pins it.
|
||||
- For anything needing a decision: **one yes/no question**, no essay.
|
||||
|
||||
Close with:
|
||||
|
||||
1. The count of `tests/Features/Tia/*` green on **both** `php84` (no pcov) and `php` (pcov), against
|
||||
the 161 baseline.
|
||||
2. Every row you added, and what invariant from Part 2 it defends.
|
||||
3. Which findings are still open, as yes/no questions.
|
||||
4. That the snapshots need regenerating — **do not regenerate them.**
|
||||
5. **What you looked at and found solid.** A list of attacks that did not break anything is a real
|
||||
result: it tells the next phase where not to spend its time.
|
||||
|
||||
---
|
||||
|
||||
## Part 7 — Open questions carried into this phase
|
||||
|
||||
Answer these before or during Pass C; they change what the fixes should be.
|
||||
|
||||
1. A fetched remote baseline carries branch keys this clone has never heard of, and the branch GC will
|
||||
prune them on the next complete write. **Should fetched keys be exempt?**
|
||||
2. Two concurrent parallel runs share the `worker-edges-*` / `worker-results-*` prefixes and
|
||||
`purgeWorkerPartials()` deletes by prefix. **Should partials be namespaced per run, or is "do not
|
||||
run two TIA suites at once" the contract?**
|
||||
3. `files` and `edges` never lose a deleted *source* file. **Cap, GC, or accept?**
|
||||
4. Orphaned state dirs accumulate under `~/.pest/tia` whenever a project's origin URL changes.
|
||||
**Reclaim them, or accept?**
|
||||
+1
-1
@@ -430,7 +430,7 @@ Most rows batch cheaply — phase two ran C1–C20 in one call at roughly one li
|
||||
|
||||
| item | status |
|
||||
|---|---|
|
||||
| **G4 / G4b** — parallel replay clobbers cached `time` on all non-executed tests. `mergeWorkerReplayPartials()` takes `$result['time']` verbatim at `Tia.php:1451`, never routing through `resultTime()` as the sequential sites (1681, 1762) do. Assertions survive because workers replay those themselves. | Pre-existing (pre-fix had no preservation at all), **out of scope for phase three**. Report it as still-present; do not fix it unless Nuno asks. |
|
||||
| ~~**G4 / G4b** — parallel replay clobbers cached `time` on all non-executed tests.~~ | **Struck in phase four — does not reproduce.** `flushWorkerReplay()` applies `resultTime()` worker-side before writing the partial, so the parent's verbatim read of `$result['time']` is reading values that were already corrected. Pinned by `a parallel replay keeps the recorded time of tests that did not run` (`tests/Features/Tia/CompleteRunWriteTier.php`), which sentinels every cached `time` and asserts a parallel replay writes nothing. |
|
||||
| **C19** — `--tia --uses=…` cannot be fixtured. TIA hard-errors on PHPUnit classes (`EnsureTiaIsRunningPestTestsOnly`), and Pest has no chainable `->uses()`. | **Expected behaviour per Nuno.** Verify the tier (`w=0`, RO, notice) and move on. Not a defect. |
|
||||
| **J11** — `--repeat` is not a Pest option (`Unknown option "--repeat"`). | **Don't care per Nuno.** Mark SKIP. |
|
||||
| **J10** — `--random-order-seed` alone exits 1 with a WARN. Identical without `--tia`. | Pre-existing Pest behaviour, unrelated. Tier still holds. |
|
||||
|
||||
@@ -28,8 +28,6 @@ return RectorConfig::configure()
|
||||
->withSkip([
|
||||
__DIR__.'/src/Plugins/Parallel/Paratest/WrapperRunner.php',
|
||||
__DIR__.'/tests/Fixtures/Arch',
|
||||
// Fixture suites are pinned by the TeamCity / JUnit snapshots, down to
|
||||
// the line numbers — rewriting their source would break them.
|
||||
__DIR__.'/tests/Fixtures/Suites',
|
||||
ReturnNeverTypeRector::class,
|
||||
ArrowFunctionDelegatingCallToFirstClassCallableRector::class,
|
||||
|
||||
@@ -35,6 +35,12 @@ final readonly class BootSubscribers implements Bootstrapper
|
||||
Subscribers\EnsureTiaResultIsRecordedOnSkipped::class,
|
||||
Subscribers\EnsureTiaResultIsRecordedOnIncomplete::class,
|
||||
Subscribers\EnsureTiaResultIsRecordedOnRisky::class,
|
||||
Subscribers\EnsureTiaResultIsRecordedOnNoticeTriggered::class,
|
||||
Subscribers\EnsureTiaResultIsRecordedOnPhpNoticeTriggered::class,
|
||||
Subscribers\EnsureTiaResultIsRecordedOnDeprecationTriggered::class,
|
||||
Subscribers\EnsureTiaResultIsRecordedOnPhpDeprecationTriggered::class,
|
||||
Subscribers\EnsureTiaResultIsRecordedOnWarningTriggered::class,
|
||||
Subscribers\EnsureTiaResultIsRecordedOnPhpWarningTriggered::class,
|
||||
Subscribers\EnsureTiaAssertionsAreRecordedOnFinished::class,
|
||||
];
|
||||
|
||||
|
||||
@@ -286,9 +286,6 @@ trait Testable
|
||||
if ($replay !== ReplayType::None) {
|
||||
assert($status !== null);
|
||||
|
||||
// Marks the replay before the branches below throw, so `tearDown`
|
||||
// short-circuits for every replayed result — the throwing branches
|
||||
// never reach `parent::setUp`, so no user hook may run after them.
|
||||
$this->__replay = $replay;
|
||||
|
||||
match ($replay) {
|
||||
|
||||
@@ -205,15 +205,6 @@ final class WrapperRunner implements RunnerInterface
|
||||
}
|
||||
|
||||
/**
|
||||
* Widens pcov's instrumentation scope to the whole project for workers that
|
||||
* record TIA edges.
|
||||
*
|
||||
* pcov's default scope is a single source directory it auto-detects, so
|
||||
* `config/`, `routes/`, `bootstrap/` and every test's own file never reach
|
||||
* the recorder — a worker-recorded graph selects a fraction of what a
|
||||
* sequential one does. `pcov.directory` is only settable at startup, hence
|
||||
* the command line rather than an `ini_set()` inside the worker.
|
||||
*
|
||||
* @param array<int, non-empty-string> $parameters
|
||||
* @return array<int, non-empty-string>
|
||||
*/
|
||||
|
||||
+229
-306
@@ -63,11 +63,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
|
||||
private const string BASELINE_PATH_OPTION = '--baseline';
|
||||
|
||||
/**
|
||||
* Set by the mutation plugin on the subprocess running a single mutant,
|
||||
* and nowhere else. Its own `--mutate` flag is popped before the argv is
|
||||
* handed to that subprocess, so the flag cannot be matched instead.
|
||||
*/
|
||||
private const string ENV_MUTATION_TESTING = 'PEST_MUTATION_TESTING';
|
||||
|
||||
private const string ENV_TIA = 'PEST_TIA';
|
||||
@@ -100,21 +95,12 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
|
||||
private const string FILTERED_GLOBAL = 'TIA_FILTERED';
|
||||
|
||||
private const string WORKER_RESULTS_GLOBAL = 'TIA_WORKER_RESULTS';
|
||||
|
||||
private const string PIGGYBACK_COVERAGE_GLOBAL = 'TIA_PIGGYBACK_COVERAGE';
|
||||
|
||||
/**
|
||||
* The parent's resolved fallback branch, handed to the workers.
|
||||
*
|
||||
* A worker cannot resolve it for itself: the restarters run before
|
||||
* `tests/Pest.php` is loaded, so a `defaultBranch()` declared there is
|
||||
* invisible to it — and autodetecting again would spend a git call per
|
||||
* worker to reach the answer the parent already has.
|
||||
*/
|
||||
private const string FALLBACK_BRANCH_GLOBAL = 'TIA_FALLBACK_BRANCH';
|
||||
|
||||
/**
|
||||
* The branch assumed when a repository cannot name its own default.
|
||||
*/
|
||||
private const string DEFAULT_BRANCH = 'main';
|
||||
|
||||
/**
|
||||
@@ -138,19 +124,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
];
|
||||
|
||||
/**
|
||||
* PHPUnit flags that make this run produce a coverage report.
|
||||
*
|
||||
* Pest's own `--coverage` is tracked by the Coverage plugin, but a raw
|
||||
* PHPUnit report flag never reaches it. A run that reports coverage must
|
||||
* not be narrowed to the affected tests — the report would then describe a
|
||||
* subset of the suite — and must let PHPUnit own the coverage driver rather
|
||||
* than have the TIA recorder clear it mid-collection.
|
||||
*
|
||||
* Flags that only shape collection or an existing report — `--coverage-filter`,
|
||||
* `--path-coverage`, `--warm-coverage-cache`, `--only-summary-for-coverage-text`,
|
||||
* `--show-uncovered-for-coverage-text`, `--disable-coverage-ignore` — produce no
|
||||
* report on their own, so they are deliberately absent.
|
||||
*
|
||||
* @var list<string>
|
||||
*/
|
||||
private const array COVERAGE_REPORT_FLAGS = [
|
||||
@@ -160,24 +133,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
];
|
||||
|
||||
/**
|
||||
* Flags that narrow this run to a subset of the suite.
|
||||
*
|
||||
* Only user-supplied, per-run narrowing belongs here. A filter that is
|
||||
* always in force — `<groups>` in phpunit.xml, or a plugin registering a
|
||||
* test case filter from `boot()` — applies equally to the runs that build
|
||||
* the baseline, so it does not make this run narrower than the baseline
|
||||
* and must not disable baseline writes.
|
||||
*
|
||||
* `--shard` is rewritten to `--filter` before this plugin sees the
|
||||
* arguments, so it is covered here too. The `bin/pest`-only flags are
|
||||
* stripped from the handled arguments, so they are matched against the
|
||||
* original argv instead.
|
||||
*
|
||||
* Flags that cut a run short instead of narrowing it — `--bail`, `--retry`,
|
||||
* `--stop-on-*` — do not belong here either. They only narrow the run when
|
||||
* something actually fails, and that is not known until it is over, so they
|
||||
* are handled by stoppedEarly() from addOutput().
|
||||
*
|
||||
* @var list<string>
|
||||
*/
|
||||
private const array PARTIAL_SELECTION_FLAGS = [
|
||||
@@ -187,15 +142,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
'--assignee', '--issue', '--ticket', '--pr', '--pull-request',
|
||||
];
|
||||
|
||||
/**
|
||||
* Options that cannot be combined with Tia mode.
|
||||
*
|
||||
* `--covers` and `--uses` select on coverage metadata Tia does not model,
|
||||
* so they resolve to no tests at all rather than to the ones the user meant.
|
||||
* `--random-order-seed` exits non-zero on its own, with or without Tia.
|
||||
* Either way the run cannot honour both things it was asked for, so it says
|
||||
* so instead of silently dropping Tia and running something else.
|
||||
*/
|
||||
private const array UNSUPPORTED_OPTIONS = [
|
||||
'--covers', '--uses', '--random-order-seed',
|
||||
];
|
||||
@@ -214,46 +160,26 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
private array $cachedAssertionsByTestId = [];
|
||||
|
||||
/**
|
||||
* Recorded durations of the tests this run replayed rather than executed.
|
||||
*
|
||||
* A replayed test never runs, so the duration PHPUnit reports for it is the
|
||||
* cost of replaying it — near zero. Writing that back would decay every
|
||||
* cached timing toward zero one run at a time.
|
||||
* The status a replayed test was replayed *as*, so the write-back records
|
||||
* what was cached rather than what the replay looked like from the
|
||||
* outside. A cached deprecation replays as a pass — recording that pass
|
||||
* would erase the deprecation from the baseline on the very next run.
|
||||
*
|
||||
* @var array<string, array{status: int, message: string}>
|
||||
*/
|
||||
private array $cachedStatusByTestId = [];
|
||||
|
||||
/**
|
||||
* @var array<string, float>
|
||||
*/
|
||||
private array $cachedTimeByTestId = [];
|
||||
|
||||
private ?Graph $replayGraph = null;
|
||||
|
||||
/**
|
||||
* The baseline this run reads from and writes to.
|
||||
*
|
||||
* The repository's default branch is only the fallback for a checkout whose
|
||||
* branch cannot be read — a detached HEAD. It is also the branch every
|
||||
* other baseline falls back to reading, so writing there by accident
|
||||
* corrupts the shared baseline. Resolved through resolveBranch() rather
|
||||
* than at every use site, because the git call it needs is not free.
|
||||
*/
|
||||
private string $branch = self::DEFAULT_BRANCH;
|
||||
|
||||
/**
|
||||
* The baseline branches with none of their own read from.
|
||||
*
|
||||
* Read-only, and the whole point of the exercise: without it the first run
|
||||
* on every new branch re-runs a suite whose results the default branch
|
||||
* already holds.
|
||||
*/
|
||||
private string $fallbackBranch = self::DEFAULT_BRANCH;
|
||||
|
||||
/**
|
||||
* Whether anything actually named the branch above.
|
||||
*
|
||||
* When nothing did, the value is a guess, and a guess is what the TIA path
|
||||
* refuses to run on: an unresolved fallback reads no baseline at all, which
|
||||
* looks exactly like a hit in the output. Runs that never asked for TIA
|
||||
* still have to write somewhere, so the guess stands for them.
|
||||
*/
|
||||
private bool $fallbackBranchResolved = false;
|
||||
|
||||
private bool $branchResolved = false;
|
||||
@@ -272,28 +198,18 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
|
||||
private bool $baselineFetchAttemptedForDrift = false;
|
||||
|
||||
private bool $freshRebuild = false;
|
||||
|
||||
private bool $filteredMode = false;
|
||||
|
||||
/**
|
||||
* Bars this run from touching the graph at all, results included.
|
||||
*
|
||||
* Reserved for runs whose results describe something other than the code
|
||||
* in the working tree, which is nothing the baseline can ever use.
|
||||
*/
|
||||
private bool $writesSuppressed = false;
|
||||
|
||||
/**
|
||||
* Narrows this run's writes to the results of the tests it actually ran.
|
||||
*
|
||||
* A run that covered only part of the suite still learns something true
|
||||
* about the tests it did reach. What it cannot do is speak for the rest:
|
||||
* pruning results, advancing the recorded sha and replacing the edge map
|
||||
* all claim the whole suite reported, so they stay behind a complete run.
|
||||
*/
|
||||
private bool $resultsOnlyWrites = false;
|
||||
|
||||
private bool $flushesWorkerResults = false;
|
||||
|
||||
private bool $unreadableGraphReported = false;
|
||||
|
||||
private bool $detachedHead = false;
|
||||
|
||||
/** @var array<int, string> */
|
||||
private array $originalArguments = [];
|
||||
|
||||
@@ -342,15 +258,70 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
|
||||
$graph = Graph::decode($json, $projectRoot);
|
||||
|
||||
// Every read of a baseline goes through a graph loaded here, so this is
|
||||
// the one place the resolved fallback has to reach.
|
||||
$graph?->setFallbackBranch($this->fallbackBranch);
|
||||
if (! $graph instanceof Graph) {
|
||||
$this->discardUnreadableGraph();
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
$graph->setFallbackBranch($this->fallbackBranch);
|
||||
|
||||
return $graph;
|
||||
}
|
||||
|
||||
/**
|
||||
* Drop a graph that will not decode, so the next run that can record starts
|
||||
* clean instead of tripping over the same file forever — rebuilding needs a
|
||||
* coverage driver, and without one the file would stay corrupt for good.
|
||||
*/
|
||||
private function discardUnreadableGraph(): void
|
||||
{
|
||||
if (Parallel::isWorker()) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (! $this->deleteState(self::KEY_GRAPH)) {
|
||||
return;
|
||||
}
|
||||
|
||||
if ($this->unreadableGraphReported) {
|
||||
return;
|
||||
}
|
||||
|
||||
$this->unreadableGraphReported = true;
|
||||
|
||||
$this->output->writeln('');
|
||||
$this->renderBadge('WARN', 'The dependency graph could not be read — it will be rebuilt.');
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a state file, unless this checkout may not write.
|
||||
*
|
||||
* A detached HEAD names no branch, so {@see self::saveGraph()} refuses to
|
||||
* write — which means anything deleted here could never be rebuilt from
|
||||
* this checkout. Read-only has to mean deletes too, or a drifted
|
||||
* `composer.lock` on a detached CI checkout wipes the whole team's baseline.
|
||||
*
|
||||
* @return bool Whether the delete happened.
|
||||
*/
|
||||
private function deleteState(string $key): bool
|
||||
{
|
||||
if ($this->detachedHead) {
|
||||
return false;
|
||||
}
|
||||
|
||||
return $this->state->delete($key);
|
||||
}
|
||||
|
||||
private function saveGraph(Graph $graph): bool
|
||||
{
|
||||
// A detached HEAD names no branch of its own, so `$this->branch` is the
|
||||
// fallback — writing here would land this checkout's results in the
|
||||
// default branch's baseline. Leave the graph exactly as it was.
|
||||
if ($this->detachedHead) {
|
||||
return true;
|
||||
}
|
||||
|
||||
$json = $graph->encode();
|
||||
|
||||
if ($json === null) {
|
||||
@@ -388,19 +359,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return ! self::argumentPresent('--ci', $arguments);
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the workers of this run record their own coverage edges.
|
||||
*
|
||||
* Stamped by the parent before paratest spawns anything, because a worker
|
||||
* cannot tell on its own: its argv carries no `--tia`, and the restarters
|
||||
* run before `tests/Pest.php` is loaded, so {@see self::isEnabledForRun()}
|
||||
* sees an empty {@see WatchPatterns} too. Left unanswered, pcov keeps its
|
||||
* default scope — a single auto-detected source directory — and every edge
|
||||
* outside it, test self-edges included, is silently dropped.
|
||||
*
|
||||
* Piggyback runs are excluded: their edges come from PHPUnit's own coverage
|
||||
* session, so widening pcov there costs time and buys nothing.
|
||||
*/
|
||||
public static function recordsEdgesInWorkers(): bool
|
||||
{
|
||||
return (string) Parallel::getGlobal(self::RECORDING_GLOBAL) === '1'
|
||||
@@ -483,6 +441,10 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
}
|
||||
|
||||
$this->replayedCount++;
|
||||
$this->cachedStatusByTestId[$testId] = [
|
||||
'status' => $result->asInt(),
|
||||
'message' => $result->message(),
|
||||
];
|
||||
$assertions = $this->replayGraph->getAssertions($this->branch, $testId);
|
||||
$this->cachedAssertionsByTestId[$testId] = $assertions ?? 0;
|
||||
|
||||
@@ -541,11 +503,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$partial = ! $isWorker && ($hasExplicitPath || $this->hasPartialSelection($arguments));
|
||||
$disabled = $disabled || $partial;
|
||||
|
||||
// A mutation subprocess runs the suite against source the mutation
|
||||
// plugin has deliberately broken. Its failures describe the mutant, not
|
||||
// the working tree, so unlike every other narrowed run there is nothing
|
||||
// in its results worth keeping. The parent `--mutate` run is untouched
|
||||
// by this: it runs the whole suite against real source.
|
||||
if (getenv(self::ENV_MUTATION_TESTING) !== false) {
|
||||
$this->writesSuppressed = true;
|
||||
}
|
||||
@@ -565,19 +522,11 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$arguments = $this->popArgument(self::BASELINED_OPTION, $arguments);
|
||||
|
||||
if ($disabled) {
|
||||
$this->requestWorkerResults();
|
||||
|
||||
if ($partial) {
|
||||
// TIA cannot choose what runs here — the user already did — but
|
||||
// the tests they picked still report honestly, so their results
|
||||
// are kept and everything that would speak for the excluded ones
|
||||
// is not. `--no-tia` needs none of this: it still runs the whole
|
||||
// suite, so it remains a complete run.
|
||||
$this->resultsOnlyWrites = true;
|
||||
|
||||
// `$this->filteredMode` counts as asking for it: reaching here
|
||||
// means the narrowing came from the command line while filtered
|
||||
// mode came from the environment or the config, and a run that
|
||||
// silently declines what the config asked for is the one most
|
||||
// in need of the explanation.
|
||||
if ($cliEnabled || $freshRequested || $this->forceRefetch || $this->filteredMode) {
|
||||
$this->output->writeln('');
|
||||
$this->renderChild('TIA does not apply to partial runs — running the selected tests directly.');
|
||||
@@ -586,15 +535,22 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
|
||||
$this->forceRefetch = false;
|
||||
$this->filteredMode = false;
|
||||
$this->freshRebuild = false;
|
||||
|
||||
return $arguments;
|
||||
}
|
||||
|
||||
if ($isWorker && (string) Parallel::getGlobal(self::WORKER_RESULTS_GLOBAL) === '1') {
|
||||
$this->flushesWorkerResults = true;
|
||||
$this->resultsOnlyWrites = true;
|
||||
|
||||
return $arguments;
|
||||
}
|
||||
|
||||
$forceRebuild = $freshRequested && ($enabled || $recordingGlobal || $replayingGlobal);
|
||||
$this->freshRebuild = $forceRebuild;
|
||||
|
||||
if (! $enabled && ! $this->forceRefetch && ! $recordingGlobal && ! $replayingGlobal) {
|
||||
$this->requestWorkerResults();
|
||||
|
||||
return $arguments;
|
||||
}
|
||||
|
||||
@@ -617,16 +573,14 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return;
|
||||
}
|
||||
|
||||
if (Parallel::isWorker() && ($this->replayGraph instanceof Graph || $this->recordingActive)) {
|
||||
if (Parallel::isWorker() && ($this->replayGraph instanceof Graph || $this->recordingActive || $this->flushesWorkerResults)) {
|
||||
$this->flushWorkerReplay();
|
||||
}
|
||||
|
||||
// Both only ever set for the parent — addOutput() returns early in
|
||||
// workers, whose partials are ephemeral and only reach the baseline if
|
||||
// the parent consumes them. Everything this method goes on to write is
|
||||
// whole-suite by nature — the edge map above all — so a narrowed run
|
||||
// stops here too, its results already persisted by addOutput().
|
||||
if ($this->writesSuppressed || $this->resultsOnlyWrites) {
|
||||
// `terminate()` also runs from the shutdown handler, which is how a run
|
||||
// that `exit()`s inside a test gets here — with a test prepared and
|
||||
// never finished, and so with no right to a complete write.
|
||||
if ($this->writesSuppressed || $this->resultsOnlyWrites || $this->hasUnfinishedTest()) {
|
||||
$this->recorder->reset();
|
||||
$this->coverageCollector->reset();
|
||||
|
||||
@@ -699,10 +653,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$graph->replaceTestInertiaComponents($perTestInertia);
|
||||
$graph->replaceJsFileToComponents(JsModuleGraph::build($projectRoot));
|
||||
|
||||
if ($this->freshRebuild) {
|
||||
$graph->pruneMissingTests();
|
||||
}
|
||||
|
||||
$this->seedResultsInto($graph);
|
||||
|
||||
if (! $this->saveGraph($graph)) {
|
||||
@@ -722,20 +672,12 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return $exitCode;
|
||||
}
|
||||
|
||||
// `->only()` narrows the executed set exactly like `--filter` does, but
|
||||
// is only knowable once the suite has been collected — too late to turn
|
||||
// TIA off up front. Sampled in addOutput() because Only's lock file is
|
||||
// already gone by the time terminate() runs (its plugin terminates
|
||||
// first). Whether the run was cut short is likewise only knowable now.
|
||||
if (Only::isEnabled() || $this->stoppedEarly()) {
|
||||
if (Only::isEnabled() || $this->stoppedEarly() || $this->hasUnfinishedTest()) {
|
||||
$this->resultsOnlyWrites = true;
|
||||
}
|
||||
|
||||
$this->reportMissingWorkerDrivers();
|
||||
|
||||
// Runs before the checks below: it is what fills the parent's result
|
||||
// collector in parallel, and a worker that stopped early narrows the
|
||||
// whole run.
|
||||
if (Parallel::isEnabled()) {
|
||||
$this->mergeWorkerReplayPartials();
|
||||
}
|
||||
@@ -815,10 +757,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$graph->replaceTestInertiaComponents($finalisedInertia);
|
||||
$graph->replaceJsFileToComponents(JsModuleGraph::build($projectRoot));
|
||||
|
||||
if ($this->freshRebuild) {
|
||||
$graph->pruneMissingTests();
|
||||
}
|
||||
|
||||
if (! $this->saveGraph($graph)) {
|
||||
$this->renderBadge('ERROR', 'Could not write the dependency graph.');
|
||||
|
||||
@@ -861,8 +799,8 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return $this->reconcileFingerprint($rebuilt, $current);
|
||||
}
|
||||
|
||||
$this->state->delete(self::KEY_GRAPH);
|
||||
$this->state->delete(self::KEY_COVERAGE_CACHE);
|
||||
$this->deleteState(self::KEY_GRAPH);
|
||||
$this->deleteState(self::KEY_COVERAGE_CACHE);
|
||||
|
||||
return null;
|
||||
}
|
||||
@@ -878,7 +816,7 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$graph->clearResults($this->branch);
|
||||
$graph->setFingerprint($current);
|
||||
$this->saveGraph($graph);
|
||||
$this->state->delete(self::KEY_COVERAGE_CACHE);
|
||||
$this->deleteState(self::KEY_COVERAGE_CACHE);
|
||||
}
|
||||
|
||||
return $graph;
|
||||
@@ -900,12 +838,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
|
||||
$this->resolveBranch($projectRoot);
|
||||
|
||||
// After resolveBranch(), so a directory that is no repository at all
|
||||
// still reports the missing git dependency rather than an unresolved
|
||||
// default branch. Nothing named the branch every other baseline reads
|
||||
// through, so every new branch would re-run the whole suite while the
|
||||
// output called it a hit. A repository with no remote is the likeliest
|
||||
// reason and gets said out loud.
|
||||
if (! $this->fallbackBranchResolved) {
|
||||
Panic::with(new ChangedFiles($projectRoot)->hasRemote()
|
||||
? new TiaRequiresDefaultBranch
|
||||
@@ -915,7 +847,7 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$fingerprint = Fingerprint::compute($projectRoot);
|
||||
$this->startFingerprint = $fingerprint;
|
||||
|
||||
if ($forceRebuild) {
|
||||
if ($forceRebuild && ! $this->detachedHead) {
|
||||
Storage::purge($projectRoot);
|
||||
}
|
||||
|
||||
@@ -948,11 +880,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
}
|
||||
}
|
||||
|
||||
// Both of these belong to the coverage cache, which only Pest's own
|
||||
// `--coverage` ever writes or merges. A raw PHPUnit report flag takes
|
||||
// the piggyback path — it must not drive the driver itself — but must
|
||||
// not leave a marker behind, nor force a recording run to prime a cache
|
||||
// that nothing on its path will fill.
|
||||
$coverageCacheOwned = $this->piggybackCoverage && $this->pestCoverageActive();
|
||||
|
||||
if ($coverageCacheOwned) {
|
||||
@@ -1157,9 +1084,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
|
||||
if (! Parallel::isEnabled()) {
|
||||
if ($canRefreshReplayEdges) {
|
||||
// Piggyback runs read PHPUnit's own coverage session. Driving
|
||||
// the driver alongside it would clear the data PHPUnit is about
|
||||
// to read, so only link tracking may run here.
|
||||
if ($this->piggybackCoverage) {
|
||||
$this->recorder->activateLinkTracking();
|
||||
} else {
|
||||
@@ -1329,9 +1253,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$recorder->activate();
|
||||
$this->recordingActive = true;
|
||||
|
||||
// Why this run is rebuilding is worth saying whenever there is a reason
|
||||
// for it — the parallel and piggyback branches above already do. Runs
|
||||
// that are simply recording for the first time have nothing to explain.
|
||||
if ($this->driftLabel !== null || $this->freshGraphReason !== null) {
|
||||
$this->output->writeln('');
|
||||
$this->renderFreshGraph();
|
||||
@@ -1347,8 +1268,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
private function renderFreshGraph(): void
|
||||
{
|
||||
if ($this->driftLabel === null && $this->freshGraphReason !== null) {
|
||||
// The reason is only ever set for a run that keeps its graph and
|
||||
// records alongside it, so "fresh graph" would be a lie here.
|
||||
$headline = sprintf('Experimental TIA mode enabled / %s.', $this->freshGraphReason);
|
||||
} else {
|
||||
$headline = 'Experimental TIA mode enabled / fresh graph';
|
||||
@@ -1423,6 +1342,30 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$this->renderChild('Install / enable pcov or xdebug (mode: coverage) in the worker PHP and rerun.');
|
||||
}
|
||||
|
||||
/**
|
||||
* A parallel run keeps its results in the workers, so the parent's collector
|
||||
* is empty and nothing would ever reach the graph. Ask the workers to flush
|
||||
* what they ran, so a parallel run refreshes — and prunes — exactly like the
|
||||
* sequential run of the same command.
|
||||
*
|
||||
* Gated on a graph already existing: a project that has never run TIA must
|
||||
* not gain a baseline from a plain `--parallel` run.
|
||||
*/
|
||||
private function requestWorkerResults(): void
|
||||
{
|
||||
if (Parallel::isWorker() || ! Parallel::isEnabled() || $this->writesSuppressed) {
|
||||
return;
|
||||
}
|
||||
|
||||
if ($this->state->read(self::KEY_GRAPH) === null) {
|
||||
return;
|
||||
}
|
||||
|
||||
$this->purgeWorkerPartials();
|
||||
|
||||
Parallel::setGlobal(self::WORKER_RESULTS_GLOBAL, '1');
|
||||
}
|
||||
|
||||
private function purgeWorkerPartials(): void
|
||||
{
|
||||
foreach ($this->collectWorkerEdgesPartials() as $key) {
|
||||
@@ -1444,14 +1387,8 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return;
|
||||
}
|
||||
|
||||
// A replayed result carries the duration PHPUnit measured for a test
|
||||
// that never ran — near zero. Unlike the assertion count, which the
|
||||
// replay injects into the result itself, the cached duration lives only
|
||||
// in this process: the parent replayed nothing of its own, so once the
|
||||
// partial is written the real value is unrecoverable. Launder it here
|
||||
// and the parent's verbatim read is correct by construction.
|
||||
foreach ($results as $testId => $result) {
|
||||
$results[$testId]['time'] = $this->resultTime($testId, $result['time']);
|
||||
$results[$testId] = $this->replayedAsRecorded($testId, $result);
|
||||
}
|
||||
|
||||
$json = json_encode([
|
||||
@@ -1459,9 +1396,7 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
'replayed' => $this->replayedCount,
|
||||
'affected' => $this->affectedCount,
|
||||
'executed' => $this->executedCount,
|
||||
// Only the worker knows it stopped early — the parent runs no tests
|
||||
// of its own, so its own check would always come back clean.
|
||||
'truncated' => $this->stoppedEarly(),
|
||||
'truncated' => $this->stoppedEarly() || $collector->hasUnfinishedTest(),
|
||||
], JSON_UNESCAPED_SLASHES);
|
||||
|
||||
if ($json === false) {
|
||||
@@ -1498,9 +1433,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
continue;
|
||||
}
|
||||
|
||||
// One worker stopping early leaves the whole suite incomplete: the
|
||||
// tests it never reached are missing from the merged result set just
|
||||
// as if they had been filtered out.
|
||||
if (($decoded['truncated'] ?? false) === true) {
|
||||
$this->resultsOnlyWrites = true;
|
||||
}
|
||||
@@ -1728,15 +1660,29 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return $coverage;
|
||||
}
|
||||
|
||||
/**
|
||||
* The duration to record for a test: its own, unless it was replayed rather
|
||||
* than executed, in which case the duration it was recorded with stands.
|
||||
*/
|
||||
private function resultTime(string $testId, float $time): float
|
||||
{
|
||||
return $this->cachedTimeByTestId[$testId] ?? $time;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param array{status: int, message: string, time: float, assertions: int, file?: string} $result
|
||||
* @return array{status: int, message: string, time: float, assertions: int, file?: string}
|
||||
*/
|
||||
private function replayedAsRecorded(string $testId, array $result): array
|
||||
{
|
||||
$result['time'] = $this->resultTime($testId, $result['time']);
|
||||
|
||||
$cached = $this->cachedStatusByTestId[$testId] ?? null;
|
||||
|
||||
if ($cached !== null) {
|
||||
$result['status'] = $cached['status'];
|
||||
$result['message'] = $cached['message'];
|
||||
}
|
||||
|
||||
return $result;
|
||||
}
|
||||
|
||||
private function seedResultsInto(Graph $graph): void
|
||||
{
|
||||
/** @var ResultCollector $collector */
|
||||
@@ -1756,12 +1702,14 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$touchedFiles[$file] = true;
|
||||
}
|
||||
|
||||
$result = $this->replayedAsRecorded($testId, $result);
|
||||
|
||||
$graph->setResult(
|
||||
$this->branch,
|
||||
$testId,
|
||||
$result['status'],
|
||||
$result['message'],
|
||||
$this->resultTime($testId, $result['time']),
|
||||
$result['time'],
|
||||
$result['assertions'],
|
||||
$file,
|
||||
);
|
||||
@@ -1769,17 +1717,45 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
|
||||
$graph->markKnownTestFiles(array_keys($touchedFiles));
|
||||
$graph->pruneStaleResults($this->branch, array_keys($touchedFiles), array_keys($results));
|
||||
$this->reclaim($graph);
|
||||
|
||||
$collector->reset();
|
||||
}
|
||||
|
||||
/**
|
||||
* Folds the run's results into the existing graph.
|
||||
*
|
||||
* An incomplete run passes `$complete: false`, which keeps the additive
|
||||
* half — the results of the tests it did run — and drops the half that
|
||||
* speaks for the suite as a whole.
|
||||
* Give back what the graph no longer needs. Only ever called from a
|
||||
* complete write — the RESULTS-ONLY and HARD-SUPPRESSED tiers may not
|
||||
* remove an entry, and a narrowed run has not seen enough to judge.
|
||||
*/
|
||||
private function reclaim(Graph $graph): void
|
||||
{
|
||||
// The fallback branch never layers under itself, so marking it would
|
||||
// write the graph for no reader's benefit — and cost a clean green run
|
||||
// its "wrote nothing at all".
|
||||
if ($this->branch !== $this->fallbackBranch) {
|
||||
$graph->markBaselineComplete($this->branch);
|
||||
}
|
||||
|
||||
$graph->pruneMissingTests();
|
||||
$graph->pruneResultsForMissingFiles($this->branch);
|
||||
|
||||
$branches = new ChangedFiles(TestSuite::getInstance()->rootPath)->branchNames();
|
||||
|
||||
if ($branches === null) {
|
||||
return;
|
||||
}
|
||||
|
||||
// A shallow, single-branch CI checkout can see almost no refs, and
|
||||
// "git has never heard of it" would then mean "this clone is narrow",
|
||||
// not "that branch is gone". Only reclaim from a checkout that can at
|
||||
// least see the branch everything else falls back to.
|
||||
if (! in_array($this->fallbackBranch, $branches, true)) {
|
||||
return;
|
||||
}
|
||||
|
||||
$graph->pruneMissingBranches([...$branches, $this->branch, $this->fallbackBranch]);
|
||||
}
|
||||
|
||||
private function snapshotTestResults(bool $markKnownTestFiles = false, bool $complete = true): void
|
||||
{
|
||||
/** @var ResultCollector $collector */
|
||||
@@ -1802,23 +1778,12 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
try {
|
||||
$this->resolveBranch($projectRoot);
|
||||
} catch (MissingDependency) {
|
||||
// This run never asked for TIA, so a missing git must not turn it
|
||||
// into a failure the way it does on the TIA path. Writing to the
|
||||
// fallback baseline is the lesser of the two evils.
|
||||
}
|
||||
|
||||
// The graph above was loaded before the branch was known — this path
|
||||
// only writes, but a graph carrying an unresolved fallback is the exact
|
||||
// bug this whole change is about.
|
||||
$graph->setFallbackBranch($this->fallbackBranch);
|
||||
|
||||
$touchedFiles = [];
|
||||
|
||||
// Whether this run is the one that records the edges its results will be
|
||||
// invalidated through. A recording run's edges are written after this
|
||||
// (terminate() runs last), and a parallel one's arrive with the worker
|
||||
// partials that ask for $markKnownTestFiles — either way the graph on
|
||||
// disk cannot be asked yet, so the run is taken at its word.
|
||||
$recordsEdges = $complete && ($markKnownTestFiles || $this->recordingActive);
|
||||
|
||||
foreach ($results as $testId => $result) {
|
||||
@@ -1832,22 +1797,18 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$touchedFiles[$file] = true;
|
||||
}
|
||||
|
||||
// A result is only ever invalidated through the edges of the test
|
||||
// that produced it, so one recorded for a test the graph has no
|
||||
// edges for could never be invalidated again — it would be replayed
|
||||
// as settled however far the code around it moved. A run that
|
||||
// records no edges leaves such a test exactly as unknown as it
|
||||
// found it, whether or not it ran the whole suite.
|
||||
if (! $recordsEdges && (! is_string($file) || ! $graph->knowsTest($file))) {
|
||||
continue;
|
||||
}
|
||||
|
||||
$result = $this->replayedAsRecorded($testId, $result);
|
||||
|
||||
$graph->setResult(
|
||||
$this->branch,
|
||||
$testId,
|
||||
$result['status'],
|
||||
$result['message'],
|
||||
$this->resultTime($testId, $result['time']),
|
||||
$result['time'],
|
||||
$result['assertions'],
|
||||
$file,
|
||||
);
|
||||
@@ -1857,10 +1818,9 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
$graph->markKnownTestFiles(array_keys($touchedFiles));
|
||||
}
|
||||
|
||||
// Pruning reads the absence of a test from this run as the test being
|
||||
// gone. That only holds if every test was invited to report.
|
||||
if ($complete) {
|
||||
$graph->pruneStaleResults($this->branch, array_keys($touchedFiles), array_keys($results));
|
||||
$this->reclaim($graph);
|
||||
}
|
||||
|
||||
$this->saveGraph($graph);
|
||||
@@ -1898,15 +1858,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether this run produces a coverage report, however it was asked for.
|
||||
*
|
||||
* The original argv, not the handled arguments: Pest's own Coverage plugin
|
||||
* appends `--coverage-php <path>` to those and runs before this one, and a
|
||||
* paratest worker's arguments always carry it too. `bin/worker.php` never
|
||||
* hands over the original argv, so a worker sees `[]` here and keeps taking
|
||||
* this from {@see self::PIGGYBACK_COVERAGE_GLOBAL} instead.
|
||||
*/
|
||||
private function coverageReportActive(): bool
|
||||
{
|
||||
if ($this->pestCoverageActive()) {
|
||||
@@ -1916,10 +1867,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return array_any(self::COVERAGE_REPORT_FLAGS, fn (string $flag): bool => $this->hasArgument($flag, $this->originalArguments));
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether Pest's own `--coverage` was given — the only entry point that
|
||||
* writes the coverage cache these two flags read and clean up.
|
||||
*/
|
||||
private function pestCoverageActive(): bool
|
||||
{
|
||||
$coverage = Container::getInstance()->get(Coverage::class);
|
||||
@@ -1929,11 +1876,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
}
|
||||
|
||||
/**
|
||||
* Panics when the run asks for Tia alongside an option Tia cannot honour.
|
||||
*
|
||||
* Checked against the original argv as well, because `bin/pest` consumes
|
||||
* some of these itself before PHPUnit ever sees them.
|
||||
*
|
||||
* @param array<int, string> $arguments
|
||||
*/
|
||||
private function guardUnsupportedOptions(array $arguments): void
|
||||
@@ -1952,11 +1894,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a selection-narrowing flag was given, either among the arguments
|
||||
* PHPUnit receives or — for the flags `bin/pest` consumes itself — among
|
||||
* the original argv. Explicit path arguments and `->only()` are detected
|
||||
* separately.
|
||||
*
|
||||
* @param array<int, string> $arguments
|
||||
*/
|
||||
private function hasPartialSelection(array $arguments): bool
|
||||
@@ -1974,36 +1911,23 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the run stopped before reaching every test it had queued.
|
||||
*
|
||||
* Covers `--bail`, `--retry` and every `--stop-on-*` flag, the equivalent
|
||||
* `phpunit.xml` attributes, and an interrupted run — none of which narrow
|
||||
* the selection up front, so hasPartialSelection() cannot see them. The
|
||||
* tests queued behind the defect that halted the run never reported, and
|
||||
* folding what did report into the baseline prunes the cached results of
|
||||
* their siblings in every file the run had already entered.
|
||||
*
|
||||
* Deliberately unguarded. Both callers run only once PHPUnit's
|
||||
* configuration is registered — the kernel reads it unguarded itself just
|
||||
* before dispatching addOutput(), and flushWorkerReplay() bails out unless
|
||||
* the worker actually executed something. Swallowing a failure here would
|
||||
* report every truncated run as complete, which is the corruption this
|
||||
* guards against in the first place.
|
||||
*/
|
||||
private function stoppedEarly(): bool
|
||||
{
|
||||
return TestResultFacade::shouldStop();
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the baselines this run reads from and writes to, once.
|
||||
*
|
||||
* Results are written on runs where TIA itself took no part, and those
|
||||
* never reach handleParent(). Without this the default would stand and
|
||||
* every such run would write its results to `main`, whatever branch it
|
||||
* actually ran on.
|
||||
* A test that was prepared and never finished means this process is being
|
||||
* torn down mid-file, so it has not seen enough of that file to prune it.
|
||||
*/
|
||||
private function hasUnfinishedTest(): bool
|
||||
{
|
||||
$collector = Container::getInstance()->get(ResultCollector::class);
|
||||
assert($collector instanceof ResultCollector);
|
||||
|
||||
return $collector->hasUnfinishedTest();
|
||||
}
|
||||
|
||||
private function resolveBranch(string $projectRoot): void
|
||||
{
|
||||
if ($this->branchResolved) {
|
||||
@@ -2014,9 +1938,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
|
||||
$changedFiles = new ChangedFiles($projectRoot);
|
||||
|
||||
// Resolved before the current branch, which throws where git is
|
||||
// missing: the fallback is advisory, so a run that cannot name its
|
||||
// branch at all should still carry the best answer available.
|
||||
$resolved = $this->resolveFallbackBranch($changedFiles);
|
||||
|
||||
$this->fallbackBranchResolved = $resolved !== null;
|
||||
@@ -2024,24 +1945,12 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
|
||||
Parallel::setGlobal(self::FALLBACK_BRANCH_GLOBAL, $this->fallbackBranch);
|
||||
|
||||
// A detached HEAD has no branch of its own to write to. The default
|
||||
// branch is the honest key there — it is the commit the checkout most
|
||||
// likely sits on, and it keeps a phantom baseline from being minted
|
||||
// under a branch name the repository never had.
|
||||
$this->branch = $changedFiles->currentBranch() ?? $this->fallbackBranch;
|
||||
$currentBranch = $changedFiles->currentBranch();
|
||||
|
||||
$this->detachedHead = $currentBranch === null;
|
||||
$this->branch = $currentBranch ?? $this->fallbackBranch;
|
||||
}
|
||||
|
||||
/**
|
||||
* The branch every other baseline falls back to reading, or null when
|
||||
* nothing in the checkout can name it.
|
||||
*
|
||||
* Ordered by how much the source actually knows. Configuration first: it is
|
||||
* the escape hatch for a repository whose git-side answers disagree with its
|
||||
* branches. Then the CI provider, which states the answer outright where git
|
||||
* is at its least informed. Then git itself. Then the recorded graph, whose
|
||||
* single baseline can only have come from the branch this repository
|
||||
* integrates on.
|
||||
*/
|
||||
private function resolveFallbackBranch(ChangedFiles $changedFiles): ?string
|
||||
{
|
||||
$inherited = Parallel::getGlobal(self::FALLBACK_BRANCH_GLOBAL);
|
||||
@@ -2056,14 +1965,6 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
?? $this->soleRecordedBranch();
|
||||
}
|
||||
|
||||
/**
|
||||
* The one branch a recorded graph holds a baseline for.
|
||||
*
|
||||
* Last in the chain and deliberately narrow: with a single baseline on disk
|
||||
* there is only one branch whose results can be read at all, so naming it is
|
||||
* strictly better than resolving to a branch that holds nothing. Two or more
|
||||
* baselines carry no such implication and are left alone.
|
||||
*/
|
||||
private function soleRecordedBranch(): ?string
|
||||
{
|
||||
$json = $this->state->read(self::KEY_GRAPH);
|
||||
@@ -2098,11 +1999,15 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
if (str_starts_with($arg, '-')) {
|
||||
continue;
|
||||
}
|
||||
if ($index > 0) {
|
||||
$previous = $arguments[$index - 1] ?? '';
|
||||
if (in_array($previous, self::VALUE_TAKING_FLAGS, true)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if ($index === 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
$previous = $arguments[$index - 1] ?? '';
|
||||
|
||||
if (in_array($previous, self::VALUE_TAKING_FLAGS, true)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
$candidate = $this->resolveArgumentPath($arg, $projectRoot);
|
||||
@@ -2111,16 +2016,34 @@ final class Tia implements AddsOutput, HandlesArguments, HandlesOriginalArgument
|
||||
continue;
|
||||
}
|
||||
|
||||
foreach ($testPaths as $testPath) {
|
||||
if ($candidate === $testPath || str_starts_with($candidate, $testPath.DIRECTORY_SEPARATOR)) {
|
||||
return true;
|
||||
}
|
||||
if ($this->narrowsSuite($candidate, $testPaths)) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param array<int, string> $testPaths
|
||||
*/
|
||||
private function narrowsSuite(string $candidate, array $testPaths): bool
|
||||
{
|
||||
foreach ($testPaths as $testPath) {
|
||||
if ($candidate === $testPath || str_starts_with($candidate, $testPath.DIRECTORY_SEPARATOR)) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
foreach ($testPaths as $testPath) {
|
||||
if (str_starts_with($testPath, $candidate.DIRECTORY_SEPARATOR)) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
private function resolveArgumentPath(string $arg, string $projectRoot): ?string
|
||||
{
|
||||
$candidates = [$arg, rtrim($projectRoot, DIRECTORY_SEPARATOR).DIRECTORY_SEPARATOR.ltrim($arg, DIRECTORY_SEPARATOR)];
|
||||
|
||||
@@ -219,16 +219,6 @@ final readonly class ChangedFiles
|
||||
return $branch === '' || $branch === 'HEAD' ? null : $branch;
|
||||
}
|
||||
|
||||
/**
|
||||
* The repository's default branch — the one every other branch's baseline
|
||||
* falls back to reading.
|
||||
*
|
||||
* Advisory, unlike {@see self::currentBranch()}: a repository that cannot
|
||||
* answer the question is not a broken repository. A remote-less checkout
|
||||
* has no `origin/HEAD`, and plenty of CI checkouts never run
|
||||
* `git remote set-head`, so every step here fails soft and the caller is
|
||||
* left to pick its own default.
|
||||
*/
|
||||
public function defaultBranch(): ?string
|
||||
{
|
||||
$head = $this->gitOutput(['git', 'symbolic-ref', '--short', 'refs/remotes/origin/HEAD']);
|
||||
@@ -241,10 +231,6 @@ final readonly class ChangedFiles
|
||||
}
|
||||
}
|
||||
|
||||
// `init.defaultBranch` is a setting of the machine, not of the
|
||||
// repository — it names what `git init` would have called the first
|
||||
// branch here, which is worth nothing once the repository disagrees.
|
||||
// Taken only when a branch by that name actually exists.
|
||||
$configured = $this->gitOutput(['git', 'config', '--get', 'init.defaultBranch']);
|
||||
|
||||
if ($configured === null) {
|
||||
@@ -258,11 +244,55 @@ final readonly class ChangedFiles
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the repository has any remote configured.
|
||||
* Every branch name this checkout knows, local and remote alike. Remotes
|
||||
* count: a branch that only lives on the origin is still a branch someone
|
||||
* will check out, and its baseline must survive.
|
||||
*
|
||||
* Advisory like {@see self::defaultBranch()} — a `git` that cannot answer
|
||||
* is reported as "no remote", and the caller decides what that means.
|
||||
* @return list<string>|null `null` when git cannot answer.
|
||||
*/
|
||||
public function branchNames(): ?array
|
||||
{
|
||||
$process = new Process(
|
||||
['git', 'for-each-ref', '--format=%(refname)', 'refs/heads', 'refs/remotes'],
|
||||
$this->projectRoot,
|
||||
);
|
||||
$process->setTimeout(5.0);
|
||||
$process->run();
|
||||
|
||||
if (! $process->isSuccessful()) {
|
||||
return null;
|
||||
}
|
||||
|
||||
$names = [];
|
||||
|
||||
foreach ($this->splitLines($process->getOutput()) as $ref) {
|
||||
if (str_starts_with($ref, 'refs/heads/')) {
|
||||
$names[substr($ref, strlen('refs/heads/'))] = true;
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
if (! str_starts_with($ref, 'refs/remotes/')) {
|
||||
continue;
|
||||
}
|
||||
|
||||
$tail = substr($ref, strlen('refs/remotes/'));
|
||||
$slash = strpos($tail, '/');
|
||||
|
||||
if ($slash === false) {
|
||||
continue;
|
||||
}
|
||||
|
||||
$branch = substr($tail, $slash + 1);
|
||||
|
||||
if ($branch !== '' && $branch !== 'HEAD') {
|
||||
$names[$branch] = true;
|
||||
}
|
||||
}
|
||||
|
||||
return array_keys($names);
|
||||
}
|
||||
|
||||
public function hasRemote(): bool
|
||||
{
|
||||
return $this->gitOutput(['git', 'remote']) !== null;
|
||||
|
||||
@@ -5,23 +5,10 @@ declare(strict_types=1);
|
||||
namespace Pest\Plugins\Tia;
|
||||
|
||||
/**
|
||||
* The default branch as the CI provider itself reports it.
|
||||
*
|
||||
* Worth asking before git: a CI checkout is the one place where git knows the
|
||||
* least. `actions/checkout` builds the working copy with `git init` plus a
|
||||
* single-ref `fetch` rather than a `clone`, so `origin/HEAD` is never set and
|
||||
* `init.defaultBranch` — a setting of the runner image, not of the repository —
|
||||
* is all git has left to offer. The provider, meanwhile, states the answer
|
||||
* outright in the environment it handed us.
|
||||
*
|
||||
* @internal
|
||||
*/
|
||||
final class CiDefaultBranch
|
||||
{
|
||||
/**
|
||||
* Advisory, like every other source in the chain: anything unreadable,
|
||||
* unparsable, or simply absent means "no answer", never a failure.
|
||||
*/
|
||||
public static function detect(): ?string
|
||||
{
|
||||
return self::fromGitLab() ?? self::fromGitHubEvent();
|
||||
@@ -32,11 +19,6 @@ final class CiDefaultBranch
|
||||
return self::environment('CI_DEFAULT_BRANCH');
|
||||
}
|
||||
|
||||
/**
|
||||
* GitHub publishes no default-branch variable, but every repository-scoped
|
||||
* event payload carries `repository.default_branch`, and the path to that
|
||||
* payload is in the environment.
|
||||
*/
|
||||
private static function fromGitHubEvent(): ?string
|
||||
{
|
||||
$path = self::environment('GITHUB_EVENT_PATH');
|
||||
|
||||
@@ -61,12 +61,6 @@ final class Configuration
|
||||
}
|
||||
|
||||
/**
|
||||
* The branch whose baseline every other branch falls back to reading.
|
||||
*
|
||||
* Autodetected from the repository when left unset; declare it here when
|
||||
* the repository cannot answer for itself — no `origin/HEAD`, or an
|
||||
* `init.defaultBranch` that disagrees with reality.
|
||||
*
|
||||
* @return $this
|
||||
*/
|
||||
public function defaultBranch(string $branch): self
|
||||
|
||||
@@ -29,7 +29,13 @@ enum ReplayType
|
||||
$status->isRisky() => self::Risky,
|
||||
$status->isSkipped() => self::Skipped,
|
||||
$status->isIncomplete() => self::Incomplete,
|
||||
default => self::Failure,
|
||||
// A recorded notice, deprecation or warning only reaches replay when
|
||||
// the configured failOn* / displayDetailsOn* policies say it is not
|
||||
// worth re-running — which means the test passed. Folding it into
|
||||
// Failure below would turn a green run red on cache alone.
|
||||
$status->isNotice(), $status->isDeprecation(), $status->isWarning() => self::Pass,
|
||||
$status->isFailure(), $status->isError() => self::Failure,
|
||||
default => self::None,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
+305
-48
@@ -43,20 +43,12 @@ final class Graph
|
||||
* @var array<string, array{
|
||||
* sha: ?string,
|
||||
* tree: array<string, string>,
|
||||
* complete?: bool,
|
||||
* results: array<string, array{status: int, message: string, time: float, assertions?: int, file?: string}>
|
||||
* }>
|
||||
*/
|
||||
private array $baselines = [];
|
||||
|
||||
/**
|
||||
* The baseline a branch with none of its own reads from.
|
||||
*
|
||||
* Only ever read from: a branch writes to its own key, so a fallback that
|
||||
* leaked into the write path would corrupt the baseline every other branch
|
||||
* depends on. Resolved once per run by the plugin — see
|
||||
* {@see self::setFallbackBranch()} — because the git calls it takes are not
|
||||
* free and the read path runs per test.
|
||||
*/
|
||||
private string $fallbackBranch = 'main';
|
||||
|
||||
private readonly string $projectRoot;
|
||||
@@ -659,6 +651,9 @@ final class Graph
|
||||
|
||||
$r = $baseline['results'][$testId];
|
||||
|
||||
// A status this build does not know — a graph written by a newer Pest,
|
||||
// or a corrupt one — is not a result. Returning null re-executes the
|
||||
// test rather than replaying an outcome nobody can interpret.
|
||||
return match ($r['status']) {
|
||||
0 => TestStatus::success(),
|
||||
1 => TestStatus::skipped($r['message']),
|
||||
@@ -669,7 +664,7 @@ final class Graph
|
||||
6 => TestStatus::warning($r['message']),
|
||||
7 => TestStatus::failure($r['message']),
|
||||
8 => TestStatus::error($r['message']),
|
||||
default => TestStatus::unknown(),
|
||||
default => null,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -696,7 +691,9 @@ final class Graph
|
||||
|
||||
$rel = $this->relative($file);
|
||||
|
||||
if ($rel !== null) {
|
||||
// A test file that is no longer on disk cannot be re-run by anyone,
|
||||
// so selecting it would only widen the run for nothing.
|
||||
if ($rel !== null && is_file($this->projectRoot.'/'.$rel)) {
|
||||
$files[$rel] = true;
|
||||
}
|
||||
}
|
||||
@@ -705,13 +702,13 @@ final class Graph
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether any cached result due a re-run points at a test file that is not
|
||||
* on disk — deleted, or never locatable in the first place (`eval()`'d code,
|
||||
* a path outside the project).
|
||||
* Whether a cached result due a re-run names a test file this project
|
||||
* cannot address — an empty path, or one that resolves outside the project
|
||||
* root. Those are genuinely lost, so the caller widens to the full suite.
|
||||
*
|
||||
* A filtered run cannot honour such an entry: it would select a file that
|
||||
* collects no tests, so the run reports green without ever re-running the
|
||||
* failure — and does so again on every subsequent invocation.
|
||||
* A path that resolves fine but is simply absent is *deleted*, not lost:
|
||||
* widening would not run it either, and treating it as unlocated used to
|
||||
* strand `--filtered` on a full replay for good.
|
||||
*/
|
||||
public function hasUnlocatedTestsToRerun(string $branch, ?string $fallbackBranch = null): bool
|
||||
{
|
||||
@@ -728,12 +725,7 @@ final class Graph
|
||||
return true;
|
||||
}
|
||||
|
||||
$rel = $this->relative($file);
|
||||
|
||||
// Results are stored relative, so `relative()` answers "is this
|
||||
// inside the project" without ever touching the filesystem. The
|
||||
// stat is what tells a deleted test file apart from a live one.
|
||||
if ($rel === null || ! is_file($this->projectRoot.'/'.$rel)) {
|
||||
if ($this->relative($file) === null) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
@@ -757,6 +749,10 @@ final class Graph
|
||||
return true;
|
||||
}
|
||||
|
||||
if ($testStatus->isUnknown()) {
|
||||
return true;
|
||||
}
|
||||
|
||||
$configuration = Registry::get();
|
||||
|
||||
if ($testStatus->isRisky()) {
|
||||
@@ -830,21 +826,82 @@ final class Graph
|
||||
}
|
||||
|
||||
/**
|
||||
* @return array{sha: ?string, tree: array<string, string>, results: array<string, array{status: int, message: string, time: float, assertions?: int, file?: string}>}
|
||||
* The baseline a read sees for this branch: its own entries layered over the
|
||||
* default branch's, so a key minted by a narrowed run — which only holds the
|
||||
* handful of tests that ran — does not shadow the fallback for everything else.
|
||||
*
|
||||
* Once this branch has had a complete run, the layering becomes per *file*
|
||||
* rather than per test id: the branch's entries for a file it executed are
|
||||
* the whole truth, so the fallback's entries for that same file are dropped
|
||||
* rather than merged. Without that, a test the branch renamed or removed —
|
||||
* and {@see self::pruneStaleResults()} therefore unset — is resurrected by
|
||||
* the default branch on the very next read, and never stops coming back.
|
||||
*
|
||||
* A branch whose key was minted by a *narrowed* run holds only the handful
|
||||
* of tests that ran, and has no business speaking for the rest of their
|
||||
* file, so it keeps the per-test-id merge.
|
||||
*
|
||||
* Read-only: the layering never reaches `$this->baselines`, so writes stay on
|
||||
* the branch that ran.
|
||||
*
|
||||
* @return array{sha: ?string, tree: array<string, string>, complete?: bool, results: array<string, array{status: int, message: string, time: float, assertions?: int, file?: string}>}
|
||||
*/
|
||||
private function baselineFor(string $branch, ?string $fallbackBranch): array
|
||||
{
|
||||
$fallbackBranch ??= $this->fallbackBranch;
|
||||
|
||||
if (isset($this->baselines[$branch])) {
|
||||
return $this->baselines[$branch];
|
||||
$fallback = $branch !== $fallbackBranch ? ($this->baselines[$fallbackBranch] ?? null) : null;
|
||||
$own = $this->baselines[$branch] ?? null;
|
||||
|
||||
if ($own === null) {
|
||||
return $fallback ?? ['sha' => null, 'tree' => [], 'results' => []];
|
||||
}
|
||||
|
||||
if ($branch !== $fallbackBranch && isset($this->baselines[$fallbackBranch])) {
|
||||
return $this->baselines[$fallbackBranch];
|
||||
if ($fallback === null) {
|
||||
return $own;
|
||||
}
|
||||
|
||||
return ['sha' => null, 'tree' => [], 'results' => []];
|
||||
$under = ($own['complete'] ?? false) === true
|
||||
? $this->withoutFilesCoveredBy($fallback['results'], $own['results'])
|
||||
: $fallback['results'];
|
||||
|
||||
return [
|
||||
'sha' => $own['sha'] ?? $fallback['sha'],
|
||||
'tree' => $own['tree'] !== [] ? $own['tree'] : $fallback['tree'],
|
||||
'results' => array_replace($under, $own['results']),
|
||||
];
|
||||
}
|
||||
|
||||
/**
|
||||
* @param array<string, array{status: int, message: string, time: float, assertions?: int, file?: string}> $results
|
||||
* @param array<string, array{status: int, message: string, time: float, assertions?: int, file?: string}> $authoritative
|
||||
* @return array<string, array{status: int, message: string, time: float, assertions?: int, file?: string}>
|
||||
*/
|
||||
private function withoutFilesCoveredBy(array $results, array $authoritative): array
|
||||
{
|
||||
$covered = [];
|
||||
|
||||
foreach ($authoritative as $entry) {
|
||||
$file = $entry['file'] ?? null;
|
||||
|
||||
if (is_string($file) && $file !== '') {
|
||||
$covered[$file] = true;
|
||||
}
|
||||
}
|
||||
|
||||
if ($covered === []) {
|
||||
return $results;
|
||||
}
|
||||
|
||||
foreach ($results as $testId => $entry) {
|
||||
$file = $entry['file'] ?? null;
|
||||
|
||||
if (is_string($file) && isset($covered[$file])) {
|
||||
unset($results[$testId]);
|
||||
}
|
||||
}
|
||||
|
||||
return $results;
|
||||
}
|
||||
|
||||
private function ensureBaseline(string $branch): void
|
||||
@@ -856,13 +913,7 @@ final class Graph
|
||||
|
||||
/**
|
||||
* @param array<string, array<int, string>> $testToFiles
|
||||
* @param bool $keepExisting Leave already-recorded edge sets alone. For runs
|
||||
* whose edges are piggybacked off a PHPUnit coverage
|
||||
* session: that data is scoped by `<source>`, so it
|
||||
* can only ever be narrower than what the TIA
|
||||
* recorder sees — it never contains the test's own
|
||||
* file, for one — and a narrower edge set silently
|
||||
* stops selecting the tests it used to select.
|
||||
* @param bool $keepExisting Leave already-recorded edge sets alone.
|
||||
*/
|
||||
public function replaceEdges(array $testToFiles, bool $keepExisting = false): void
|
||||
{
|
||||
@@ -873,8 +924,6 @@ final class Graph
|
||||
continue;
|
||||
}
|
||||
|
||||
// An empty set means "known, covers nothing", so piggyback data is
|
||||
// still an improvement there — only a populated set is protected.
|
||||
if ($keepExisting && ($this->edges[$testRel] ?? []) !== []) {
|
||||
continue;
|
||||
}
|
||||
@@ -1437,6 +1486,67 @@ final class Graph
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Record that this branch has run the whole suite at least once, which is
|
||||
* what lets {@see self::baselineFor()} treat its entries as authoritative
|
||||
* for the files they cover. Never mints a key: a run that recorded nothing
|
||||
* has nothing to be authoritative about.
|
||||
*/
|
||||
public function markBaselineComplete(string $branch): void
|
||||
{
|
||||
if (isset($this->baselines[$branch])) {
|
||||
$this->baselines[$branch]['complete'] = true;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Drop this branch's result entries whose test file is no longer on disk.
|
||||
*
|
||||
* Without this nothing but `--fresh` ever reclaims them, and a *failing*
|
||||
* one keeps `--filtered` widened to a full replay on every later run.
|
||||
*/
|
||||
public function pruneResultsForMissingFiles(string $branch): void
|
||||
{
|
||||
if (! isset($this->baselines[$branch]['results'])) {
|
||||
return;
|
||||
}
|
||||
|
||||
$root = rtrim($this->projectRoot, DIRECTORY_SEPARATOR).DIRECTORY_SEPARATOR;
|
||||
|
||||
foreach ($this->baselines[$branch]['results'] as $testId => $result) {
|
||||
$file = $result['file'] ?? null;
|
||||
|
||||
if (! is_string($file) || $file === '') {
|
||||
continue;
|
||||
}
|
||||
|
||||
$rel = $this->relative($file);
|
||||
|
||||
if ($rel === null || is_file($root.$rel)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
unset($this->baselines[$branch]['results'][$testId]);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Drop baselines for branches git no longer knows, so the graph does not
|
||||
* carry one full copy of the suite per branch ever created.
|
||||
*
|
||||
* @param array<int, string> $keep Branch names that must survive.
|
||||
*/
|
||||
public function pruneMissingBranches(array $keep): void
|
||||
{
|
||||
$survivors = array_fill_keys($keep, true);
|
||||
|
||||
foreach (array_keys($this->baselines) as $branch) {
|
||||
if (! isset($survivors[$branch])) {
|
||||
unset($this->baselines[$branch]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Prune baseline result entries whose test files were just executed but whose
|
||||
* test IDs are no longer present (e.g. the test method was removed or renamed).
|
||||
@@ -1483,13 +1593,6 @@ final class Graph
|
||||
}
|
||||
|
||||
/**
|
||||
* The branches a recorded graph holds baselines for, read straight from the
|
||||
* encoded form.
|
||||
*
|
||||
* Answerable before the graph is hydrated because the default branch has to
|
||||
* be resolved first: the fallback is what every hydrated graph reads its
|
||||
* baselines through.
|
||||
*
|
||||
* @return list<string>
|
||||
*/
|
||||
public static function branchesIn(string $json): array
|
||||
@@ -1521,10 +1624,10 @@ final class Graph
|
||||
|
||||
$graph = new self($projectRoot);
|
||||
$graph->fingerprint = is_array($data['fingerprint'] ?? null) ? $data['fingerprint'] : [];
|
||||
$graph->files = is_array($data['files'] ?? null) ? array_values($data['files']) : [];
|
||||
$graph->files = self::decodeFiles($data['files'] ?? null);
|
||||
$graph->fileIds = array_flip($graph->files);
|
||||
$graph->edges = is_array($data['edges'] ?? null) ? $data['edges'] : [];
|
||||
$graph->baselines = is_array($data['baselines'] ?? null) ? $data['baselines'] : [];
|
||||
$graph->edges = self::decodeEdges($data['edges'] ?? null);
|
||||
$graph->baselines = self::decodeBaselines($data['baselines'] ?? null);
|
||||
|
||||
$graph->testTables = self::decodeStringMap($data['test_tables'] ?? null);
|
||||
$graph->testInertiaComponents = self::decodeStringMap($data['test_inertia_components'] ?? null);
|
||||
@@ -1533,6 +1636,160 @@ final class Graph
|
||||
return $graph;
|
||||
}
|
||||
|
||||
/**
|
||||
* @return array<int, string>
|
||||
*/
|
||||
private static function decodeFiles(mixed $section): array
|
||||
{
|
||||
if (! is_array($section)) {
|
||||
return [];
|
||||
}
|
||||
|
||||
$files = [];
|
||||
|
||||
foreach ($section as $path) {
|
||||
if (is_string($path) && $path !== '') {
|
||||
$files[] = $path;
|
||||
}
|
||||
}
|
||||
|
||||
return $files;
|
||||
}
|
||||
|
||||
/**
|
||||
* @return array<string, array<int, int>>
|
||||
*/
|
||||
private static function decodeEdges(mixed $section): array
|
||||
{
|
||||
if (! is_array($section)) {
|
||||
return [];
|
||||
}
|
||||
|
||||
$edges = [];
|
||||
|
||||
foreach ($section as $key => $ids) {
|
||||
$testFile = (string) $key;
|
||||
|
||||
if ($testFile === '') {
|
||||
continue;
|
||||
}
|
||||
if (! is_array($ids)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
$clean = [];
|
||||
|
||||
foreach ($ids as $id) {
|
||||
if (is_int($id)) {
|
||||
$clean[] = $id;
|
||||
}
|
||||
}
|
||||
|
||||
$edges[$testFile] = $clean;
|
||||
}
|
||||
|
||||
return $edges;
|
||||
}
|
||||
|
||||
/**
|
||||
* A graph is state on disk that any process may have written: a newer Pest,
|
||||
* a half-finished write, a hand edit. Every branch, every entry and every
|
||||
* field is checked here so that a malformed one is dropped rather than
|
||||
* reaching a read path and taking the run down with it.
|
||||
*
|
||||
* @return array<string, array{sha: ?string, tree: array<string, string>, complete?: bool, results: array<string, array{status: int, message: string, time: float, assertions?: int, file?: string}>}>
|
||||
*/
|
||||
private static function decodeBaselines(mixed $section): array
|
||||
{
|
||||
if (! is_array($section)) {
|
||||
return [];
|
||||
}
|
||||
|
||||
$baselines = [];
|
||||
|
||||
foreach ($section as $key => $baseline) {
|
||||
// A branch named `12345` decodes as an integer key, and must not be
|
||||
// mistaken for a malformed one.
|
||||
$branch = (string) $key;
|
||||
|
||||
if ($branch === '') {
|
||||
continue;
|
||||
}
|
||||
if (! is_array($baseline)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
$sha = $baseline['sha'] ?? null;
|
||||
$tree = [];
|
||||
|
||||
if (is_array($baseline['tree'] ?? null)) {
|
||||
foreach ($baseline['tree'] as $path => $hash) {
|
||||
if (is_string($path) && is_string($hash)) {
|
||||
$tree[$path] = $hash;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$baselines[$branch] = [
|
||||
'sha' => is_string($sha) ? $sha : null,
|
||||
'tree' => $tree,
|
||||
'results' => self::decodeResults($baseline['results'] ?? null),
|
||||
];
|
||||
|
||||
if (($baseline['complete'] ?? null) === true) {
|
||||
$baselines[$branch]['complete'] = true;
|
||||
}
|
||||
}
|
||||
|
||||
return $baselines;
|
||||
}
|
||||
|
||||
/**
|
||||
* @return array<string, array{status: int, message: string, time: float, assertions?: int, file?: string}>
|
||||
*/
|
||||
private static function decodeResults(mixed $section): array
|
||||
{
|
||||
if (! is_array($section)) {
|
||||
return [];
|
||||
}
|
||||
|
||||
$results = [];
|
||||
|
||||
foreach ($section as $key => $entry) {
|
||||
$testId = (string) $key;
|
||||
|
||||
if ($testId === '') {
|
||||
continue;
|
||||
}
|
||||
if (! is_array($entry)) {
|
||||
continue;
|
||||
}
|
||||
if (! is_int($entry['status'] ?? null)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
$time = $entry['time'] ?? null;
|
||||
|
||||
$result = [
|
||||
'status' => $entry['status'],
|
||||
'message' => is_string($entry['message'] ?? null) ? $entry['message'] : '',
|
||||
'time' => is_int($time) || is_float($time) ? (float) $time : 0.0,
|
||||
];
|
||||
|
||||
if (is_int($entry['assertions'] ?? null)) {
|
||||
$result['assertions'] = $entry['assertions'];
|
||||
}
|
||||
|
||||
if (is_string($entry['file'] ?? null) && $entry['file'] !== '') {
|
||||
$result['file'] = $entry['file'];
|
||||
}
|
||||
|
||||
$results[$testId] = $result;
|
||||
}
|
||||
|
||||
return $results;
|
||||
}
|
||||
|
||||
/**
|
||||
* @return array<string, list<string>>
|
||||
*/
|
||||
|
||||
@@ -16,6 +16,9 @@ final class ResultCollector
|
||||
*/
|
||||
private array $results = [];
|
||||
|
||||
/** @var array<string, true> */
|
||||
private array $triggered = [];
|
||||
|
||||
private ?string $currentTestId = null;
|
||||
|
||||
private ?string $currentTestFile = null;
|
||||
@@ -35,9 +38,35 @@ final class ResultCollector
|
||||
return;
|
||||
}
|
||||
|
||||
// PHPUnit reports a test that triggered a notice, deprecation or
|
||||
// warning as passed, and emits Passed for it. Recording success here
|
||||
// would erase the issue from the baseline, and a later replay under
|
||||
// --fail-on-deprecation (and friends) would come back green where a
|
||||
// fresh run fails. Keep the issue; only refresh what it cannot know.
|
||||
if (isset($this->triggered[$this->currentTestId])) {
|
||||
$this->refreshTime();
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
$this->record(TestStatus::success());
|
||||
}
|
||||
|
||||
public function testTriggeredNotice(string $message): void
|
||||
{
|
||||
$this->recordIssue(TestStatus::notice($message));
|
||||
}
|
||||
|
||||
public function testTriggeredDeprecation(string $message): void
|
||||
{
|
||||
$this->recordIssue(TestStatus::deprecation($message));
|
||||
}
|
||||
|
||||
public function testTriggeredWarning(string $message): void
|
||||
{
|
||||
$this->recordIssue(TestStatus::warning($message));
|
||||
}
|
||||
|
||||
public function testFailed(string $message): void
|
||||
{
|
||||
if ($this->currentTestId === null) {
|
||||
@@ -91,6 +120,17 @@ final class ResultCollector
|
||||
return $this->results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a test was prepared but never finished — the process is being
|
||||
* torn down in the middle of it (an `exit()` inside a test, a killed
|
||||
* worker). What it collected is therefore a partial view of that test
|
||||
* file, and must not license pruning the siblings it never reached.
|
||||
*/
|
||||
public function hasUnfinishedTest(): bool
|
||||
{
|
||||
return $this->currentTestId !== null;
|
||||
}
|
||||
|
||||
public function recordAssertions(string $testId, int $assertions): void
|
||||
{
|
||||
if (isset($this->results[$testId])) {
|
||||
@@ -111,6 +151,7 @@ final class ResultCollector
|
||||
public function reset(): void
|
||||
{
|
||||
$this->results = [];
|
||||
$this->triggered = [];
|
||||
$this->currentTestId = null;
|
||||
$this->currentTestFile = null;
|
||||
$this->startTime = null;
|
||||
@@ -123,6 +164,43 @@ final class ResultCollector
|
||||
$this->startTime = null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Record an issue raised while the test was running. The most important
|
||||
* one wins, exactly as PHPUnit ranks them, so a deprecation does not
|
||||
* shadow the warning that followed it — or the failure.
|
||||
*/
|
||||
private function recordIssue(TestStatus $status): void
|
||||
{
|
||||
if ($this->currentTestId === null) {
|
||||
return;
|
||||
}
|
||||
|
||||
$existing = $this->results[$this->currentTestId]['status'] ?? null;
|
||||
|
||||
if (is_int($existing) && $existing >= $status->asInt()) {
|
||||
return;
|
||||
}
|
||||
|
||||
$this->triggered[$this->currentTestId] = true;
|
||||
|
||||
$this->record($status);
|
||||
}
|
||||
|
||||
private function refreshTime(): void
|
||||
{
|
||||
if ($this->currentTestId === null) {
|
||||
return;
|
||||
}
|
||||
if (! isset($this->results[$this->currentTestId])) {
|
||||
return;
|
||||
}
|
||||
if ($this->startTime === null) {
|
||||
return;
|
||||
}
|
||||
|
||||
$this->results[$this->currentTestId]['time'] = round(microtime(true) - $this->startTime, 3);
|
||||
}
|
||||
|
||||
private function record(TestStatus $status): void
|
||||
{
|
||||
if ($this->currentTestId === null) {
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace Pest\Subscribers;
|
||||
|
||||
use Pest\Plugins\Tia\ResultCollector;
|
||||
use PHPUnit\Event\Test\DeprecationTriggered;
|
||||
use PHPUnit\Event\Test\DeprecationTriggeredSubscriber;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
*/
|
||||
final readonly class EnsureTiaResultIsRecordedOnDeprecationTriggered implements DeprecationTriggeredSubscriber
|
||||
{
|
||||
public function __construct(private ResultCollector $collector) {}
|
||||
|
||||
public function notify(DeprecationTriggered $event): void
|
||||
{
|
||||
if ($event->wasSuppressed()) {
|
||||
return;
|
||||
}
|
||||
|
||||
$this->collector->testTriggeredDeprecation($event->message());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace Pest\Subscribers;
|
||||
|
||||
use Pest\Plugins\Tia\ResultCollector;
|
||||
use PHPUnit\Event\Test\NoticeTriggered;
|
||||
use PHPUnit\Event\Test\NoticeTriggeredSubscriber;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
*/
|
||||
final readonly class EnsureTiaResultIsRecordedOnNoticeTriggered implements NoticeTriggeredSubscriber
|
||||
{
|
||||
public function __construct(private ResultCollector $collector) {}
|
||||
|
||||
public function notify(NoticeTriggered $event): void
|
||||
{
|
||||
if ($event->wasSuppressed()) {
|
||||
return;
|
||||
}
|
||||
|
||||
$this->collector->testTriggeredNotice($event->message());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace Pest\Subscribers;
|
||||
|
||||
use Pest\Plugins\Tia\ResultCollector;
|
||||
use PHPUnit\Event\Test\PhpDeprecationTriggered;
|
||||
use PHPUnit\Event\Test\PhpDeprecationTriggeredSubscriber;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
*/
|
||||
final readonly class EnsureTiaResultIsRecordedOnPhpDeprecationTriggered implements PhpDeprecationTriggeredSubscriber
|
||||
{
|
||||
public function __construct(private ResultCollector $collector) {}
|
||||
|
||||
public function notify(PhpDeprecationTriggered $event): void
|
||||
{
|
||||
if ($event->wasSuppressed()) {
|
||||
return;
|
||||
}
|
||||
|
||||
$this->collector->testTriggeredDeprecation($event->message());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace Pest\Subscribers;
|
||||
|
||||
use Pest\Plugins\Tia\ResultCollector;
|
||||
use PHPUnit\Event\Test\PhpNoticeTriggered;
|
||||
use PHPUnit\Event\Test\PhpNoticeTriggeredSubscriber;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
*/
|
||||
final readonly class EnsureTiaResultIsRecordedOnPhpNoticeTriggered implements PhpNoticeTriggeredSubscriber
|
||||
{
|
||||
public function __construct(private ResultCollector $collector) {}
|
||||
|
||||
public function notify(PhpNoticeTriggered $event): void
|
||||
{
|
||||
if ($event->wasSuppressed()) {
|
||||
return;
|
||||
}
|
||||
|
||||
$this->collector->testTriggeredNotice($event->message());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace Pest\Subscribers;
|
||||
|
||||
use Pest\Plugins\Tia\ResultCollector;
|
||||
use PHPUnit\Event\Test\PhpWarningTriggered;
|
||||
use PHPUnit\Event\Test\PhpWarningTriggeredSubscriber;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
*/
|
||||
final readonly class EnsureTiaResultIsRecordedOnPhpWarningTriggered implements PhpWarningTriggeredSubscriber
|
||||
{
|
||||
public function __construct(private ResultCollector $collector) {}
|
||||
|
||||
public function notify(PhpWarningTriggered $event): void
|
||||
{
|
||||
if ($event->wasSuppressed()) {
|
||||
return;
|
||||
}
|
||||
|
||||
$this->collector->testTriggeredWarning($event->message());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace Pest\Subscribers;
|
||||
|
||||
use Pest\Plugins\Tia\ResultCollector;
|
||||
use PHPUnit\Event\Test\WarningTriggered;
|
||||
use PHPUnit\Event\Test\WarningTriggeredSubscriber;
|
||||
|
||||
/**
|
||||
* @internal
|
||||
*/
|
||||
final readonly class EnsureTiaResultIsRecordedOnWarningTriggered implements WarningTriggeredSubscriber
|
||||
{
|
||||
public function __construct(private ResultCollector $collector) {}
|
||||
|
||||
public function notify(WarningTriggered $event): void
|
||||
{
|
||||
if ($event->wasSuppressed()) {
|
||||
return;
|
||||
}
|
||||
|
||||
$this->collector->testTriggeredWarning($event->message());
|
||||
}
|
||||
}
|
||||
@@ -1568,6 +1568,41 @@
|
||||
PASS Tests\Features\Tia
|
||||
✓ it does not run user hooks when replaying cached skipped and incomplete results
|
||||
|
||||
PASS Tests\Features\Tia\BranchShapes
|
||||
✓ a branch name git allows is a branch key TIA can hold with dataset "slashes"
|
||||
✓ a branch name git allows is a branch key TIA can hold with dataset "dots"
|
||||
✓ a branch name git allows is a branch key TIA can hold with dataset "unicode"
|
||||
✓ a branch name git allows is a branch key TIA can hold with dataset "digits"
|
||||
✓ a branch name git allows is a branch key TIA can hold with dataset "underscores"
|
||||
✓ a branch name git allows is a branch key TIA can hold with dataset "very long"
|
||||
✓ a branch differing from the default only in case gets its own key
|
||||
✓ a branch that only lives on the remote keeps its baseline
|
||||
✓ a branch checked out in a worktree keeps its baseline
|
||||
✓ deleting many branches reclaims every one of their baselines with dataset "sequential"
|
||||
✓ deleting many branches reclaims every one of their baselines with dataset "parallel"
|
||||
✓ a narrowed run does not reclaim anything
|
||||
✓ a detached HEAD does not reclaim anything either
|
||||
✓ the default branch baseline survives every branch that comes and goes
|
||||
|
||||
PASS Tests\Features\Tia\CompleteRunWriteTier
|
||||
✓ a complete run prunes a deleted test with dataset "sequential"
|
||||
✓ a complete run prunes a deleted test with dataset "parallel"
|
||||
✓ a complete run records nothing for a test file the graph does not know
|
||||
✓ a partial run records nothing for a test file the graph does not know
|
||||
✓ a truncated run does not prune with dataset "sequential"
|
||||
✓ a truncated run does not prune with dataset "parallel"
|
||||
✓ a green bail run is complete
|
||||
✓ --no-tia refreshes results without enabling tia with dataset "sequential"
|
||||
✓ --no-tia refreshes results without enabling tia with dataset "parallel"
|
||||
✓ a plain run refreshes the results it executed with dataset "sequential"
|
||||
✓ a plain run refreshes the results it executed with dataset "parallel"
|
||||
✓ a parallel replay keeps the recorded time of tests that did not run
|
||||
✓ a run that never enables tia creates no graph with dataset "plain"
|
||||
✓ a run that never enables tia creates no graph with dataset "filtered"
|
||||
✓ a run that never enables tia creates no graph with dataset "parallel filtered"
|
||||
✓ a test edit narrows to the affected file and replays the rest
|
||||
✓ a parallel run merges worker results into the parent baseline
|
||||
|
||||
PASS Tests\Features\Tia\DefaultBranchReplay
|
||||
✓ replays the default branch baseline on a new branch
|
||||
✓ replays whatever the default branch is called with ('main')
|
||||
@@ -1577,12 +1612,23 @@
|
||||
✓ replays on a second new branch too
|
||||
✓ writes nothing on a second run on the same branch
|
||||
✓ replays on a branch whose name contains slashes
|
||||
✓ replays again once back on the default branch
|
||||
✓ replays on a new branch when tia is enabled by configuration
|
||||
✓ a narrowed run on a new branch does not cost the fallback with dataset "sequential"
|
||||
✓ a narrowed run on a new branch does not cost the fallback with dataset "parallel"
|
||||
✓ replays inside a worktree on a new branch
|
||||
|
||||
PASS Tests\Features\Tia\DefaultBranchResolution
|
||||
✓ a declared default branch beats autodetection
|
||||
✓ a declared default branch that does not exist degrades to a full run
|
||||
✓ a renamed default branch replays and writes under its new name
|
||||
✓ the CI provider names the default branch where the checkout cannot
|
||||
✓ GitLab names the default branch through its own variable
|
||||
✓ a lone recorded baseline names the default branch
|
||||
✓ a default branch nothing can name is refused rather than guessed
|
||||
✓ an init.defaultBranch naming a branch that exists is still trusted
|
||||
✓ a repository with no remote is refused rather than silently re-run
|
||||
✓ a remote-less repository holding one baseline is not refused
|
||||
✓ a declared default branch stands in for a missing remote
|
||||
✓ tia still requires git
|
||||
✓ a plain run outside a repository creates no baseline
|
||||
@@ -1592,10 +1638,117 @@
|
||||
✓ narrows to the affected tests on a new branch
|
||||
✓ filtered mode reads the fallback too
|
||||
✓ filtered mode finds nothing to do on a clean green feature branch
|
||||
✓ filtered mode falls back to a full replay when a cached failure cannot be located
|
||||
✓ filtered mode finds nothing to do on the default branch itself
|
||||
✓ a detached HEAD replays without minting a branch key
|
||||
✓ a detached HEAD does not write into the default branch baseline with dataset "sequential"
|
||||
✓ a detached HEAD does not write into the default branch baseline with dataset "parallel"
|
||||
✓ the branch that ran gets its own key and the default branch keeps its baseline
|
||||
✓ the fallback reaches parallel workers
|
||||
|
||||
PASS Tests\Features\Tia\FilteredMode
|
||||
✓ re-runs a cached failure on a clean tree
|
||||
✓ an explicit path turns filtered mode off
|
||||
✓ filtered mode runs the whole suite when there is no baseline
|
||||
✓ filtered mode finds nothing to do in parallel either
|
||||
✓ a corrupt graph is reported and does not crash the run
|
||||
✓ --parallel --retry is refused and leaves the graph alone
|
||||
|
||||
PASS Tests\Features\Tia\HostileState
|
||||
✓ a graph mangled beyond use still lets the suite run with dataset "empty"
|
||||
✓ a graph mangled beyond use still lets the suite run with dataset "truncated"
|
||||
✓ a graph mangled beyond use still lets the suite run with dataset "not json"
|
||||
✓ a graph mangled beyond use still lets the suite run with dataset "json scalar"
|
||||
✓ a graph mangled beyond use still lets the suite run with dataset "json list"
|
||||
✓ a graph mangled beyond use still lets the suite run with dataset "json null"
|
||||
✓ a graph mangled beyond use still lets the suite run with dataset "empty object"
|
||||
✓ a graph mangled beyond use still lets the suite run with dataset "nul bytes"
|
||||
✓ a graph whose shape is wrong everywhere is repaired rather than trusted with dataset "sequential"
|
||||
✓ a graph whose shape is wrong everywhere is repaired rather than trusted with dataset "parallel"
|
||||
✓ a cached status this build cannot interpret is re-run, not replayed with dataset "below the range"
|
||||
✓ a cached status this build cannot interpret is re-run, not replayed with dataset "one past the range"
|
||||
✓ a cached status this build cannot interpret is re-run, not replayed with dataset "far past the range"
|
||||
✓ a cached status this build cannot interpret is re-run, not replayed with dataset "huge"
|
||||
✓ a cached status with no replay of its own does not fail the run with dataset "notice"
|
||||
✓ a cached status with no replay of its own does not fail the run with dataset "deprecation"
|
||||
✓ a cached status with no replay of its own does not fail the run with dataset "warning"
|
||||
✓ a cached skip or todo replays with its message intact with dataset "skipped"
|
||||
✓ a cached skip or todo replays with its message intact with dataset "incomplete"
|
||||
✓ a cached failure with a multi-line message re-runs rather than replaying the text
|
||||
✓ a result pointing outside the project is not addressable and widens the run
|
||||
✓ an edge pointing at a file id that does not exist is ignored
|
||||
✓ a graph from a schema this build does not know is rebuilt, not read
|
||||
✓ graph.json being a directory does not stop the run
|
||||
✓ a state dir it cannot write to still replays
|
||||
|
||||
PASS Tests\Features\Tia\IssueStatuses
|
||||
✓ a triggered issue is recorded as itself, not as a pass with dataset "sequential" / dataset "deprecation"
|
||||
✓ a triggered issue is recorded as itself, not as a pass with dataset "sequential" / dataset "notice"
|
||||
✓ a triggered issue is recorded as itself, not as a pass with dataset "sequential" / dataset "warning"
|
||||
✓ a triggered issue is recorded as itself, not as a pass with dataset "parallel" / dataset "deprecation"
|
||||
✓ a triggered issue is recorded as itself, not as a pass with dataset "parallel" / dataset "notice"
|
||||
✓ a triggered issue is recorded as itself, not as a pass with dataset "parallel" / dataset "warning"
|
||||
✓ a cached deprecation still fails the run that asked to fail on one with dataset "sequential"
|
||||
✓ a cached deprecation still fails the run that asked to fail on one with dataset "parallel"
|
||||
✓ replaying a cached issue does not downgrade it to a pass with dataset "sequential"
|
||||
✓ replaying a cached issue does not downgrade it to a pass with dataset "parallel"
|
||||
✓ a failure outranks an issue triggered on the way to it
|
||||
✓ a skip outranks an issue triggered on the way to it
|
||||
✓ a suppressed issue is not recorded
|
||||
|
||||
PASS Tests\Features\Tia\PartialRunWriteTier
|
||||
✓ a filtered run rewrites only the test that ran
|
||||
✓ a filtered run under --tia announces that tia does not apply
|
||||
✓ a test suffix narrows the tier even though every test runs
|
||||
✓ a dirty run narrows to the uncommitted test edit
|
||||
✓ filtered mode yields to an explicit filter
|
||||
✓ an env flag narrows exactly like the option it mirrors with ('PEST_TIA')
|
||||
✓ an env flag narrows exactly like the option it mirrors with ('PEST_TIA_FILTERED')
|
||||
✓ a partial run does not purge the graph even with --fresh
|
||||
✓ --no-tia does not stop a partial run from refreshing its own entry
|
||||
✓ two partial runs each keep the other entry
|
||||
✓ a shard is a partial run
|
||||
✓ a parallel partial run records the test that ran, like a sequential one
|
||||
|
||||
PASS Tests\Features\Tia\StateReclamation
|
||||
✓ a detached HEAD does not purge the graph on structural drift with dataset "sequential"
|
||||
✓ a detached HEAD does not purge the graph on structural drift with dataset "parallel"
|
||||
✓ a detached HEAD does not purge the graph with --fresh either with dataset "sequential"
|
||||
✓ a detached HEAD does not purge the graph with --fresh either with dataset "parallel"
|
||||
✓ a detached HEAD leaves an unreadable graph for a checkout that can rebuild it
|
||||
✓ a cached failure whose test file was deleted stops widening later runs with dataset "sequential"
|
||||
✓ a cached failure whose test file was deleted stops widening later runs with dataset "parallel"
|
||||
✓ a complete run reclaims the entry and the edge of a deleted test file with dataset "sequential"
|
||||
✓ a complete run reclaims the entry and the edge of a deleted test file with dataset "parallel"
|
||||
✓ a pruned result does not come back from the fallback with dataset "sequential"
|
||||
✓ a pruned result does not come back from the fallback with dataset "parallel"
|
||||
✓ the fallback still reaches a branch that has never run a test file
|
||||
✓ a branch that git no longer knows loses its baseline
|
||||
✓ an unknown cached status is re-run rather than replayed as a failure with dataset "unknown"
|
||||
✓ an unknown cached status is re-run rather than replayed as a failure with dataset "future"
|
||||
✓ an unknown cached status is re-run rather than replayed as a failure with dataset "garbage"
|
||||
✓ a cached notice, deprecation or warning does not replay as a failure with dataset "notice"
|
||||
✓ a cached notice, deprecation or warning does not replay as a failure with dataset "deprecation"
|
||||
✓ a cached notice, deprecation or warning does not replay as a failure with dataset "warning"
|
||||
✓ a malformed baseline entry cannot break the run with dataset "sequential"
|
||||
✓ a malformed baseline entry cannot break the run with dataset "parallel"
|
||||
✓ a run torn down mid-file does not prune the tests it never reached with dataset "sequential"
|
||||
✓ a run torn down mid-file does not prune the tests it never reached with dataset "parallel"
|
||||
✓ a fatal error mid-file is a test error, not a truncation with dataset "sequential"
|
||||
✓ a fatal error mid-file is a test error, not a truncation with dataset "parallel"
|
||||
✓ a green complete run leaves the graph exactly as it found it with dataset "bail"
|
||||
✓ a green complete run leaves the graph exactly as it found it with dataset "stop-on-failure"
|
||||
✓ a green complete run leaves the graph exactly as it found it with dataset "compact"
|
||||
✓ a green complete run leaves the graph exactly as it found it with dataset "parallel bail"
|
||||
✓ a green complete run leaves the graph exactly as it found it with dataset "parallel one process"
|
||||
✓ a green complete run leaves the graph exactly as it found it with dataset "parallel more processes than files"
|
||||
✓ --tia --no-tia is a plain run that still refreshes what it executed with dataset "sequential"
|
||||
✓ --tia --no-tia is a plain run that still refreshes what it executed with dataset "parallel"
|
||||
✓ --fresh on a partial run neither purges nor prunes with dataset "sequential"
|
||||
✓ --fresh on a partial run neither purges nor prunes with dataset "parallel"
|
||||
✓ a second green run on a feature branch writes nothing at all with dataset "sequential"
|
||||
✓ a second green run on a feature branch writes nothing at all with dataset "parallel"
|
||||
|
||||
PASS Tests\Features\Ticket
|
||||
✓ it may be associated with an ticket #1, #2
|
||||
✓ nested → it may be associated with an ticket #1, #4, #5, #6, #3
|
||||
@@ -2225,4 +2378,4 @@
|
||||
✓ pass with dataset with ('my-datas-set-value')
|
||||
✓ within describe → pass with dataset with ('my-datas-set-value')
|
||||
|
||||
Tests: 1 deprecated, 4 warnings, 5 incomplete, 2 notices, 40 todos, 35 skipped, 1571 passed (3446 assertions)
|
||||
Tests: 1 deprecated, 4 warnings, 5 incomplete, 2 notices, 40 todos, 35 skipped, 1710 passed (3845 assertions)
|
||||
@@ -49,7 +49,7 @@ it('does not run user hooks when replaying cached skipped and incomplete results
|
||||
expect($storage->write(Tia::KEY_GRAPH, (string) $json))->toBeTrue();
|
||||
|
||||
$process = new Process(
|
||||
['php', 'bin/pest', $fixture, '--tia'],
|
||||
['php', 'bin/pest', '--configuration', 'tests/Fixtures/Suites/TiaReplayHooks.xml', '--tia'],
|
||||
$projectRoot,
|
||||
[
|
||||
'COLLISION_PRINTER' => 'DefaultPrinter',
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
|
||||
/*
|
||||
* Invariant 5 — writes land on the branch that ran and only there — and
|
||||
* invariant 6 — nothing is unbounded — under every branch shape git allows.
|
||||
*/
|
||||
|
||||
test('a branch name git allows is a branch key TIA can hold', function (string $branch): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->switchTo($branch, new: true);
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($result->uncached())->toBe(0, $result->describe())
|
||||
->and($project->branchKeys())->toBe(['master', $branch])
|
||||
->and($delta->baselineUntouched('master'))->toBeTrue($delta->summary());
|
||||
})->with([
|
||||
'slashes' => 'feature/deep/nesting',
|
||||
'dots' => 'release.1.2.x',
|
||||
'unicode' => 'feature-café-日本',
|
||||
'digits' => '12345',
|
||||
'underscores' => 'feature_x_y',
|
||||
'very long' => 'feature-'.str_repeat('x', 180),
|
||||
])->skipOnWindows();
|
||||
|
||||
test('a branch differing from the default only in case gets its own key', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->switchTo('MASTER-2', new: true);
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($project->branchKeys())->toBe(['master', 'MASTER-2']);
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a branch that only lives on the remote keeps its baseline', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->switchTo('remote-only', new: true);
|
||||
$project->pest('--tia');
|
||||
|
||||
$project->git()->switchTo('master');
|
||||
$project->git()->run(['update-ref', 'refs/remotes/origin/remote-only', 'HEAD']);
|
||||
$project->git()->run(['branch', '-D', 'remote-only']);
|
||||
|
||||
$project->pest('--tia');
|
||||
|
||||
expect($project->branchKeys())->toBe(['master', 'remote-only']);
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a branch checked out in a worktree keeps its baseline', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->switchTo('wt-branch', new: true);
|
||||
$project->pest('--tia');
|
||||
|
||||
$project->git()->switchTo('master');
|
||||
$project->git()->run(['worktree', 'add', '--quiet', $project->path().'-wt', 'wt-branch']);
|
||||
|
||||
$project->pest('--tia');
|
||||
|
||||
$project->git()->run(['worktree', 'remove', '--force', $project->path().'-wt']);
|
||||
|
||||
expect($project->branchKeys())->toBe(['master', 'wt-branch']);
|
||||
})->skipOnWindows();
|
||||
|
||||
test('deleting many branches reclaims every one of their baselines', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
foreach (range(1, 4) as $index) {
|
||||
$project->git()->switchTo('feature-'.$index, new: true);
|
||||
$project->pest('--tia', ...$arguments);
|
||||
}
|
||||
|
||||
expect($project->branchKeys())->toHaveCount(5);
|
||||
|
||||
$project->git()->switchTo('master');
|
||||
|
||||
foreach (range(1, 4) as $index) {
|
||||
$project->git()->run(['branch', '-D', 'feature-'.$index]);
|
||||
}
|
||||
|
||||
$project->pest('--tia', ...$arguments);
|
||||
|
||||
expect($project->branchKeys())->toBe(['master']);
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a narrowed run does not reclaim anything', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
$project->pest('--tia');
|
||||
|
||||
$project->git()->switchTo('master');
|
||||
$project->git()->run(['branch', '-D', 'feature-x']);
|
||||
|
||||
$project->snapshot();
|
||||
$project->pest('--tia', '--filter=adds two numbers');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($project->branchKeys())->toBe(['master', 'feature-x'])
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a detached HEAD does not reclaim anything either', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
$project->pest('--tia');
|
||||
|
||||
$project->git()->switchTo('master');
|
||||
$project->git()->run(['branch', '-D', 'feature-x']);
|
||||
$project->git()->detach();
|
||||
|
||||
$project->snapshot();
|
||||
$project->pest('--tia');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($project->branchKeys())->toBe(['master', 'feature-x'])
|
||||
->and($delta->isHardSuppressed())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('the default branch baseline survives every branch that comes and goes', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
$project->pest('--tia');
|
||||
|
||||
$project->git()->switchTo('master');
|
||||
$project->git()->run(['branch', '-D', 'feature-x']);
|
||||
|
||||
$project->snapshot();
|
||||
$result = $project->pest('--tia');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($delta->writtenCount())->toBe(0, $delta->summary())
|
||||
->and($delta->added())->toBe(0, $delta->summary())
|
||||
->and($delta->removed())->toBe(0, $delta->summary())
|
||||
->and($project->graph()['baselines']['master']['results'])->toHaveCount(Project::TOTAL_TESTS);
|
||||
})->skipOnWindows();
|
||||
@@ -0,0 +1,234 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
|
||||
test('a complete run prunes a deleted test', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/GreeterTest.php', <<<'PHP'
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Fixture\App\Greeter;
|
||||
|
||||
test('greets a person', function (): void {
|
||||
expect((new Greeter)->greet('Nuno'))->toBe('Hello, Nuno!');
|
||||
});
|
||||
PHP);
|
||||
|
||||
$result = $project->pest(...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->tally())->toContain('5 passed')
|
||||
->and($delta->removed())->toBe(1, $delta->summary())
|
||||
->and($delta->added())->toBe(0, $delta->summary())
|
||||
->and($delta->structureMoved())->toBeFalse($delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a complete run records nothing for a test file the graph does not know', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/BrandNewTest.php', <<<'PHP'
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
test('brand new thing', function (): void {
|
||||
expect(true)->toBeTrue();
|
||||
});
|
||||
PHP);
|
||||
|
||||
$result = $project->pest();
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->tally())->toContain('7 passed')
|
||||
->and($delta->added())->toBe(0, $delta->summary())
|
||||
->and($delta->removed())->toBe(0, $delta->summary())
|
||||
->and($delta->writtenCount())->toBe(Project::TOTAL_TESTS, $delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a partial run records nothing for a test file the graph does not know', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/BrandNewTest.php', <<<'PHP'
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
test('brand new thing', function (): void {
|
||||
expect(true)->toBeTrue();
|
||||
});
|
||||
PHP);
|
||||
|
||||
$result = $project->pest('--filter=brand new thing');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->tally())->toContain('1 passed')
|
||||
->and($delta->added())->toBe(0, $delta->summary())
|
||||
->and($delta->writtenCount())->toBe(0, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a truncated run does not prune', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/GreeterTest.php', <<<'PHP'
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Fixture\App\Greeter;
|
||||
|
||||
test('greets a person', function (): void {
|
||||
expect((new Greeter)->greet('Nuno'))->toBe('Goodbye, Nuno!');
|
||||
});
|
||||
|
||||
test('greets the world', function (): void {
|
||||
expect((new Greeter)->greet('world'))->toBe('Hello, world!');
|
||||
});
|
||||
PHP);
|
||||
|
||||
$result = $project->pest('--bail', ...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->exitCode)->toBe(1, $result->describe())
|
||||
->and($result->tally())->toContain('1 failed')
|
||||
->and($delta->removed())->toBe(0, $delta->summary())
|
||||
->and($delta->structureMoved())->toBeFalse($delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a green bail run is complete', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--bail');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($delta->writtenCount())->toBe(Project::TOTAL_TESTS, $delta->summary())
|
||||
->and($delta->removed())->toBe(0, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('--no-tia refreshes results without enabling tia', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--tia', '--no-tia', ...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->output)->not->toContain('Experimental TIA mode enabled')
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed')
|
||||
->and($delta->writtenCount())->toBe(Project::TOTAL_TESTS, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a plain run refreshes the results it executed', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest(...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->tally())->toContain(Project::TOTAL_TESTS.' passed')
|
||||
->and($delta->writtenCount())->toBe(Project::TOTAL_TESTS, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a parallel replay keeps the recorded time of tests that did not run', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->pest('--tia', '--parallel', '--processes=2');
|
||||
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($delta->writtenCount())->toBe(0, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a run that never enables tia creates no graph', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
|
||||
$result = $project->pest(...$arguments);
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($project->graphExists())->toBeFalse();
|
||||
})->with([
|
||||
'plain' => [[]],
|
||||
'filtered' => [['--filter=adds two numbers']],
|
||||
'parallel filtered' => [['--parallel', '--processes=2', '--filter=adds two numbers']],
|
||||
])->skipOnWindows();
|
||||
|
||||
test('a test edit narrows to the affected file and replays the rest', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/GreeterTest.php', <<<'PHP'
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Fixture\App\Greeter;
|
||||
|
||||
test('greets a person', function (): void {
|
||||
expect((new Greeter)->greet('Nuno'))->toBe('Hello, Nuno!');
|
||||
expect((new Greeter)->greet('Nuno'))->toBeString();
|
||||
});
|
||||
|
||||
test('greets the world', function (): void {
|
||||
expect((new Greeter)->greet('world'))->toBe('Hello, world!');
|
||||
});
|
||||
PHP);
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->affected())->toBe(2, $result->describe())
|
||||
->and($result->replayed())->toBe(4, $result->describe())
|
||||
->and($delta->writtenCount())->toBe(2, $delta->summary())
|
||||
->and($delta->edgesMoved())->toBeFalse($delta->summary())
|
||||
->and($delta->removed())->toBe(0, $delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a parallel run merges worker results into the parent baseline', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/GreeterTest.php', <<<'PHP'
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Fixture\App\Greeter;
|
||||
|
||||
test('greets a person', function (): void {
|
||||
expect((new Greeter)->greet('Nuno'))->toBe('Hello, Nuno!');
|
||||
expect((new Greeter)->greet('Nuno'))->toBeString();
|
||||
});
|
||||
|
||||
test('greets the world', function (): void {
|
||||
expect((new Greeter)->greet('world'))->toBe('Hello, world!');
|
||||
});
|
||||
PHP);
|
||||
|
||||
$result = $project->pest('--tia', '--parallel', '--processes=2');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->affected())->toBe(2, $result->describe())
|
||||
->and($result->replayed())->toBe(4, $result->describe())
|
||||
->and($delta->writtenCount())->toBe(2, $delta->summary())
|
||||
->and($delta->edgesMoved())->toBeFalse($delta->summary())
|
||||
->and($delta->removed())->toBe(0, $delta->summary());
|
||||
})->skipOnWindows();
|
||||
@@ -4,15 +4,6 @@ declare(strict_types=1);
|
||||
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
/**
|
||||
* Reading a baseline recorded on another branch.
|
||||
*
|
||||
* Without the default-branch fallback, the first `--tia` run on every new branch
|
||||
* re-runs a whole suite whose results the default branch already holds — once
|
||||
* per branch, forever, on any repository not named `main`.
|
||||
*
|
||||
* @see https://github.com/pestphp/pest/issues/1823
|
||||
*/
|
||||
afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
@@ -54,8 +45,6 @@ test('replays on a second new branch too', function (): void {
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
// The toll is one full run per new branch. It must not come back for the
|
||||
// second branch either.
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($result->uncached())->toBe(0, $result->describe());
|
||||
})->skipOnWindows();
|
||||
@@ -89,16 +78,58 @@ test('replays on a branch whose name contains slashes', function (): void {
|
||||
->and($project->branchKeys())->toContain('feature/x/y');
|
||||
})->skipOnWindows();
|
||||
|
||||
test('replays again once back on the default branch', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
$project->pest('--tia');
|
||||
|
||||
$project->git()->switchTo('master');
|
||||
|
||||
$project->snapshot();
|
||||
$result = $project->pest('--tia');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($result->uncached())->toBe(0, $result->describe())
|
||||
->and($delta->writtenCount())->toBe(0, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('replays on a new branch when tia is enabled by configuration', function (): void {
|
||||
$project = Project::make('master', overlay: 'always-enabled');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
|
||||
$result = $project->pest();
|
||||
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($result->uncached())->toBe(0, $result->describe())
|
||||
->and($project->branchKeys())->toBe(['master', 'feature-x']);
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a narrowed run on a new branch does not cost the fallback', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
$project->pest(...$arguments);
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($result->uncached())->toBe(0, $result->describe());
|
||||
})->with([
|
||||
'sequential' => [['--filter=adds two numbers']],
|
||||
'parallel' => [['--parallel', '--processes=2', '--filter=adds two numbers']],
|
||||
])->skipOnWindows();
|
||||
|
||||
test('replays inside a worktree on a new branch', function (): void {
|
||||
$project = Project::make('master');
|
||||
$worktree = $project->worktree('feature-worktree');
|
||||
|
||||
// Seeded against the worktree rather than the main checkout: a worktree's
|
||||
// `.git` is a file, so `Storage::originIdentity()` cannot read the remote
|
||||
// from it and the worktree resolves a storage key of its own. That gap is
|
||||
// separate from the branch fallback, and it is the fallback this row is
|
||||
// about — the worktree is checked out on a branch the baseline does not
|
||||
// name, which is the scenario from the issue.
|
||||
$project->seedFor($worktree, 'master');
|
||||
|
||||
$result = $project->pestIn($worktree, '--tia');
|
||||
|
||||
@@ -5,17 +5,11 @@ declare(strict_types=1);
|
||||
use Symfony\Component\Process\ExecutableFinder;
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
/**
|
||||
* How the default branch gets named: declared in `tests/Pest.php`, autodetected
|
||||
* from the repository, or not answerable at all.
|
||||
*/
|
||||
afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
|
||||
test('a declared default branch beats autodetection', function (): void {
|
||||
// The repository autodetects `develop`, which holds no baseline. Only the
|
||||
// declaration in `tests/Pest.php` can reach the `master` one.
|
||||
$project = Project::make('develop', overlay: 'configured-default-branch');
|
||||
$project->seed('master');
|
||||
|
||||
@@ -35,20 +29,32 @@ test('a declared default branch that does not exist degrades to a full run', fun
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
// Nothing to fall back to, so everything runs — and no baseline is minted
|
||||
// under the name that resolved to nothing.
|
||||
expect($result->uncached())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($project->branchKeys())->toBe(['master', 'feature-x']);
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a renamed default branch replays and writes under its new name', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->rename('master', 'main');
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($result->uncached())->toBe(0, $result->describe())
|
||||
->and($project->branchKeys())->toBe(['master', 'main'])
|
||||
->and($delta->baselineUntouched('master'))->toBeTrue($delta->summary())
|
||||
->and($delta->writtenCount())->toBe(0, $delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('the CI provider names the default branch where the checkout cannot', function (): void {
|
||||
// A CI checkout: `actions/checkout` fetches a single ref instead of cloning,
|
||||
// so there is no `origin/HEAD` for git to read the default branch from. The
|
||||
// event payload GitHub hands the job says it outright.
|
||||
$project = Project::make('master');
|
||||
$project->git()->unsetOriginHead();
|
||||
$project->seed('master');
|
||||
$project->addBaseline('legacy');
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
|
||||
@@ -68,6 +74,7 @@ test('GitLab names the default branch through its own variable', function (): vo
|
||||
$project = Project::make('master');
|
||||
$project->git()->unsetOriginHead();
|
||||
$project->seed('master');
|
||||
$project->addBaseline('legacy');
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
|
||||
@@ -80,10 +87,6 @@ test('GitLab names the default branch through its own variable', function (): vo
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a lone recorded baseline names the default branch', function (): void {
|
||||
// Nothing left to ask: no `origin/HEAD`, no CI provider, and an
|
||||
// `init.defaultBranch` that names a branch this repository does not have.
|
||||
// The graph holds exactly one baseline, and it is the only one any branch
|
||||
// could read — so it is the answer.
|
||||
$project = Project::make('master');
|
||||
$project->git()->unsetOriginHead();
|
||||
$project->git()->config('init.defaultBranch', 'main');
|
||||
@@ -98,9 +101,6 @@ test('a lone recorded baseline names the default branch', function (): void {
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a default branch nothing can name is refused rather than guessed', function (): void {
|
||||
// Same checkout as above, without the graph that answered it. Guessing here
|
||||
// is what made this bug expensive: the guess reads no baseline at all, and
|
||||
// the output calls that a hit.
|
||||
$project = Project::make('master');
|
||||
$project->git()->unsetOriginHead();
|
||||
$project->git()->config('init.defaultBranch', 'main');
|
||||
@@ -116,9 +116,6 @@ test('a default branch nothing can name is refused rather than guessed', functio
|
||||
})->skipOnWindows();
|
||||
|
||||
test('an init.defaultBranch naming a branch that exists is still trusted', function (): void {
|
||||
// The setting is the machine's, not the repository's — worth taking only
|
||||
// where the repository has a branch by that name. It does here, and with no
|
||||
// graph on disk it is the only source left, so the run must not be refused.
|
||||
$project = Project::make('master');
|
||||
$project->git()->unsetOriginHead();
|
||||
$project->git()->config('init.defaultBranch', 'master');
|
||||
@@ -129,7 +126,7 @@ test('an init.defaultBranch naming a branch that exists is still trusted', funct
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->output)->not->toContain('could not determine the default branch')
|
||||
->and($project->branchKeys())->toBe(['feature-x']);
|
||||
->and($project->branchKeys())->not->toContain('master');
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a repository with no remote is refused rather than silently re-run', function (): void {
|
||||
@@ -141,16 +138,11 @@ test('a repository with no remote is refused rather than silently re-run', funct
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
// Nothing can name the default branch, so every new branch would re-run the
|
||||
// whole suite with no explanation. Saying so beats doing that quietly. The
|
||||
// missing remote is the likeliest reason and gets named as such.
|
||||
expect($result->output)->toContain('Tia mode requires a repository with a remote.')
|
||||
->and($result->exitCode)->toBe(1, $result->describe());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a remote-less repository holding one baseline is not refused', function (): void {
|
||||
// The refusal above exists to stop a guess, not to demand a remote for its
|
||||
// own sake. With a baseline on disk there is nothing left to guess at.
|
||||
$project = Project::make('master');
|
||||
|
||||
$project->git()->removeOrigin();
|
||||
@@ -165,8 +157,6 @@ test('a remote-less repository holding one baseline is not refused', function ()
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a declared default branch stands in for a missing remote', function (): void {
|
||||
// The escape hatch the refusal above points at: with the branch named by
|
||||
// hand there is nothing left for a remote to answer.
|
||||
$project = Project::make('master', overlay: 'configured-default-branch');
|
||||
|
||||
$project->git()->removeOrigin();
|
||||
@@ -185,8 +175,6 @@ test('tia still requires git', function (): void {
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
// The soft default-branch resolver runs before the branch is named, and it
|
||||
// must not swallow this.
|
||||
expect($result->output)->toContain('The feature "Tia mode" requires "git".')
|
||||
->and($result->exitCode)->not->toBe(0);
|
||||
})->skipOnWindows();
|
||||
@@ -212,8 +200,6 @@ test('the default branch is resolved once per run, not once per test', function
|
||||
|
||||
expect($git)->not->toBeNull();
|
||||
|
||||
// A `git` first on `PATH` that records what it was asked before handing over
|
||||
// to the real one.
|
||||
$log = $project->path('git-calls.log');
|
||||
$project->write('shim/git', implode("\n", [
|
||||
'#!/bin/sh',
|
||||
|
||||
@@ -4,12 +4,6 @@ declare(strict_types=1);
|
||||
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
/**
|
||||
* What the fallback is allowed to touch.
|
||||
*
|
||||
* Reading another branch's baseline must stay a read: writes belong to the
|
||||
* branch that ran, and a branch that has no name of its own must not mint one.
|
||||
*/
|
||||
afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
@@ -20,58 +14,43 @@ test('narrows to the affected tests on a new branch', function (): void {
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
|
||||
// Semantic, not cosmetic: PHP is hashed at the AST level, so a comment
|
||||
// would not register as a change at all.
|
||||
$project->write('app/Calculator.php', <<<'PHP'
|
||||
$project->write('tests/Unit/GreeterTest.php', <<<'PHP'
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace Fixture\App;
|
||||
use Fixture\App\Greeter;
|
||||
|
||||
final class Calculator
|
||||
{
|
||||
public function add(int $a, int $b): int
|
||||
{
|
||||
return $a + $b;
|
||||
}
|
||||
test('greets a person', function (): void {
|
||||
expect((new Greeter)->greet('Nuno'))->toBe('Hello, Nuno!');
|
||||
expect((new Greeter)->greet('Nuno'))->toBeString();
|
||||
});
|
||||
|
||||
public function subtract(int $a, int $b): int
|
||||
{
|
||||
return $a - $b;
|
||||
}
|
||||
|
||||
public function multiply(int $a, int $b): int
|
||||
{
|
||||
return $a * $b;
|
||||
}
|
||||
}
|
||||
test('greets the world', function (): void {
|
||||
expect((new Greeter)->greet('world'))->toBe('Hello, world!');
|
||||
});
|
||||
PHP);
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
$delta = $project->delta();
|
||||
|
||||
// Two of the three test files cover `Calculator`; the third replays from the
|
||||
// default branch's baseline.
|
||||
expect($result->affected())->toBe(4, $result->describe())
|
||||
->and($result->replayed())->toBe(2, $result->describe())
|
||||
->and($result->exitCode)->toBe(0, $result->describe());
|
||||
expect($result->affected())->toBe(2, $result->describe())
|
||||
->and($result->replayed())->toBe(4, $result->describe())
|
||||
->and($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($project->branchKeys())->toBe(['master', 'feature-x'])
|
||||
->and($delta->baselineUntouched('master'))->toBeTrue($delta->summary())
|
||||
->and($delta->edgesMoved())->toBeFalse($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('filtered mode reads the fallback too', function (): void {
|
||||
$project = Project::make('master');
|
||||
|
||||
// A failure cached on the default branch. Filtered mode asks the graph which
|
||||
// test files are due a re-run, and that read has to reach the fallback as
|
||||
// well: without it a new branch sees nothing to do, reports green, and never
|
||||
// re-runs the failure — on every subsequent invocation.
|
||||
$project->seed('master', failing: ['adds two numbers']);
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
|
||||
$result = $project->pest('--tia', '--filtered');
|
||||
|
||||
// Selection is by file, so the failure's two siblings in `CalculatorTest`
|
||||
// come along; the other two test files stay out of the run entirely.
|
||||
expect($result->output)->toContain('from 1 previously unsuccessful test')
|
||||
->and($result->output)->not->toContain('No affected tests found')
|
||||
->and($result->tally())->toContain('2 passed');
|
||||
@@ -91,6 +70,43 @@ test('filtered mode finds nothing to do on a clean green feature branch', functi
|
||||
->and($delta->isHardSuppressed())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('filtered mode falls back to a full replay when a cached failure cannot be located', function (): void {
|
||||
$project = Project::make('master');
|
||||
|
||||
$project->seed('master', failing: ['adds two numbers']);
|
||||
|
||||
// A path this project cannot address at all — recorded on another machine.
|
||||
// A path that merely no longer exists is a *deleted* test, not a lost one,
|
||||
// and widening the run would not find it either; see StateReclamation.
|
||||
$project->mutateGraph(function (array $graph): array {
|
||||
$testId = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
|
||||
$graph['baselines']['master']['results'][$testId]['file'] = '/build/agent/tests/Unit/DeletedTest.php';
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
|
||||
$result = $project->pest('--tia', '--filtered');
|
||||
|
||||
expect($result->output)->toContain('Some cached tests due a re-run could not be located on disk.')
|
||||
->and($result->output)->toContain('Running the full suite with replay instead of a filtered run.')
|
||||
->and($result->output)->not->toContain('No affected tests found');
|
||||
})->skipOnWindows();
|
||||
|
||||
test('filtered mode finds nothing to do on the default branch itself', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--tia', '--filtered');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->output)->toContain('No affected tests found')
|
||||
->and($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($delta->isHardSuppressed())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a detached HEAD replays without minting a branch key', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
@@ -99,13 +115,27 @@ test('a detached HEAD replays without minting a branch key', function (): void {
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
// A detached HEAD has no branch of its own. The default branch is the
|
||||
// honest key, and no phantom one appears beside it.
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($result->uncached())->toBe(0, $result->describe())
|
||||
->and($project->branchKeys())->toBe(['master']);
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a detached HEAD does not write into the default branch baseline', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->detach();
|
||||
$project->pest(...$arguments);
|
||||
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($delta->baselineUntouched('master'))->toBeTrue($delta->summary())
|
||||
->and($project->branchKeys())->toBe(['master']);
|
||||
})->with([
|
||||
'sequential' => [['--filter=adds two numbers']],
|
||||
'parallel' => [['--parallel', '--processes=2', '--filter=adds two numbers']],
|
||||
])->skipOnWindows();
|
||||
|
||||
test('the branch that ran gets its own key and the default branch keeps its baseline', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
|
||||
test('re-runs a cached failure on a clean tree', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master', failing: ['adds two numbers']);
|
||||
|
||||
$result = $project->pest('--tia', '--filtered');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->output)->toContain('from 1 previously unsuccessful test')
|
||||
->and($result->affected())->toBe(2, $result->describe())
|
||||
->and($result->tally())->toContain('2 passed')
|
||||
->and($delta->writtenCount())->toBe(2, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('an explicit path turns filtered mode off', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--tia', '--filtered', 'tests/Unit');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->output)->toContain('TIA does not apply to partial runs')
|
||||
->and($result->output)->not->toContain('No affected tests found')
|
||||
->and($result->tally())->toContain('4 passed')
|
||||
->and($delta->writtenCount())->toBe(4, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('filtered mode runs the whole suite when there is no baseline', function (): void {
|
||||
$project = Project::make('master');
|
||||
|
||||
$result = $project->pest('--tia', '--filtered');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed')
|
||||
->and($result->output)->not->toContain('No affected tests found');
|
||||
})->skipOnWindows();
|
||||
|
||||
test('filtered mode finds nothing to do in parallel either', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--tia', '--filtered', '--parallel', '--processes=2');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->output)->toContain('No affected tests found')
|
||||
->and($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($delta->isHardSuppressed())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a corrupt graph is reported and does not crash the run', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$graph = $project->graphDir().'/graph.json';
|
||||
|
||||
file_put_contents($graph, '{not json');
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed')
|
||||
->and($result->output)->toContain('The dependency graph could not be read')
|
||||
->and(is_file($graph) ? file_get_contents($graph) : null)->not->toBe('{not json');
|
||||
})->skipOnWindows();
|
||||
|
||||
test('--parallel --retry is refused and leaves the graph alone', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--parallel', '--retry');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->exitCode)->not->toBe(0)
|
||||
->and($result->output)->toContain('--retry')
|
||||
->and($delta->isHardSuppressed())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
@@ -0,0 +1,226 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
|
||||
/*
|
||||
* Invariant 7 — a hostile state dir cannot break a run. Whatever is in
|
||||
* graph.json, the suite still runs and exits on the tests' merit.
|
||||
*/
|
||||
|
||||
test('a graph mangled beyond use still lets the suite run', function (string $contents): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
file_put_contents($project->graphDir().'/graph.json', $contents);
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed');
|
||||
})->with([
|
||||
'empty' => '',
|
||||
'truncated' => '{"schema":1,"files":["app/Calculator.php"],"edg',
|
||||
'not json' => '{not json',
|
||||
'json scalar' => '"just a string"',
|
||||
'json list' => '[1,2,3]',
|
||||
'json null' => 'null',
|
||||
'empty object' => '{}',
|
||||
'nul bytes' => "\0\0\0\0",
|
||||
])->skipOnWindows();
|
||||
|
||||
test('a graph whose shape is wrong everywhere is repaired rather than trusted', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->mutateGraph(function (array $graph): array {
|
||||
$id = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
|
||||
$graph['baselines']['master']['results'][$id] = 'nope';
|
||||
$graph['baselines']['master']['results'][7] = ['status' => 0, 'message' => '', 'time' => 0.1];
|
||||
$graph['baselines']['master']['tree'] = 'nope';
|
||||
$graph['baselines']['master']['sha'] = 42;
|
||||
$graph['baselines'][''] = ['sha' => null, 'tree' => [], 'results' => []];
|
||||
$graph['baselines']['broken'] = 'nope';
|
||||
$graph['edges']['tests/Unit/GreeterTest.php'] = 'nope';
|
||||
$graph['edges'][''] = [0];
|
||||
$graph['files'][] = ['nested'];
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$result = $project->pest('--tia', ...$arguments);
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->output)->not->toContain('TypeError')
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed')
|
||||
->and($project->branchKeys())->toBe(['master']);
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a cached status this build cannot interpret is re-run, not replayed', function (int $status): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->mutateGraph(function (array $graph) use ($status): array {
|
||||
$id = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
$graph['baselines']['master']['results'][$id]['status'] = $status;
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed')
|
||||
->and($result->uncached())->toBe(1, $result->describe());
|
||||
})->with([
|
||||
'below the range' => -1,
|
||||
'one past the range' => 9,
|
||||
'far past the range' => 99,
|
||||
'huge' => PHP_INT_MAX,
|
||||
])->skipOnWindows();
|
||||
|
||||
test('a cached status with no replay of its own does not fail the run', function (int $status): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->mutateGraph(function (array $graph) use ($status): array {
|
||||
$id = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
$graph['baselines']['master']['results'][$id]['status'] = $status;
|
||||
$graph['baselines']['master']['results'][$id]['message'] = 'cached detail';
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed');
|
||||
})->with(['notice' => 3, 'deprecation' => 4, 'warning' => 6])->skipOnWindows();
|
||||
|
||||
test('a cached skip or todo replays with its message intact', function (int $status, string $tally): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->mutateGraph(function (array $graph) use ($status): array {
|
||||
$id = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
$graph['baselines']['master']['results'][$id]['status'] = $status;
|
||||
$graph['baselines']['master']['results'][$id]['message'] = 'a recorded reason';
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->tally())->toContain($tally)
|
||||
->and($result->output)->toContain('a recorded reason')
|
||||
->and($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe());
|
||||
})->with([
|
||||
'skipped' => [1, '1 skipped'],
|
||||
'incomplete' => [2, '1 incomplete'],
|
||||
])->skipOnWindows();
|
||||
|
||||
test('a cached failure with a multi-line message re-runs rather than replaying the text', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->mutateGraph(function (array $graph): array {
|
||||
$id = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
$graph['baselines']['master']['results'][$id]['status'] = 7;
|
||||
$graph['baselines']['master']['results'][$id]['message'] = "line one\nline two\nline three";
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->uncached())->toBe(1, $result->describe())
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed');
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a result pointing outside the project is not addressable and widens the run', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master', failing: ['adds two numbers']);
|
||||
|
||||
$project->mutateGraph(function (array $graph): array {
|
||||
$id = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
$graph['baselines']['master']['results'][$id]['file'] = '/build/agent/tests/Unit/CalculatorTest.php';
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$result = $project->pest('--tia', '--filtered');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->output)->toContain('could not be located on disk')
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed');
|
||||
})->skipOnWindows();
|
||||
|
||||
test('an edge pointing at a file id that does not exist is ignored', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->mutateGraph(function (array $graph): array {
|
||||
$graph['edges']['tests/Unit/CalculatorTest.php'] = [0, 999, -5];
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a graph from a schema this build does not know is rebuilt, not read', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->mutateGraph(function (array $graph): array {
|
||||
$graph['schema'] = 2;
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed')
|
||||
->and($result->replayed())->toBe(0, $result->describe());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('graph.json being a directory does not stop the run', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
unlink($project->graphDir().'/graph.json');
|
||||
mkdir($project->graphDir().'/graph.json');
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed');
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a state dir it cannot write to still replays', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
chmod($project->graphDir(), 0500);
|
||||
|
||||
try {
|
||||
$result = $project->pest('--tia');
|
||||
} finally {
|
||||
chmod($project->graphDir(), 0700);
|
||||
}
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe());
|
||||
})->skipOnWindows();
|
||||
@@ -0,0 +1,143 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
|
||||
/*
|
||||
* A test that triggers a notice, deprecation or warning is reported by PHPUnit
|
||||
* as passed, and emits Passed. Recording it as a plain success made the cache
|
||||
* hide the issue: a later run under --fail-on-* came back green where a fresh
|
||||
* run failed. Invariant 3 — replay is faithful — at its most dangerous.
|
||||
*/
|
||||
|
||||
function tiaTriggering(string $call): string
|
||||
{
|
||||
return <<<PHP
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Fixture\App\Calculator;
|
||||
|
||||
test('adds two numbers', function (): void {
|
||||
{$call}
|
||||
expect((new Calculator)->add(1, 2))->toBe(3);
|
||||
});
|
||||
|
||||
test('subtracts two numbers', function (): void {
|
||||
expect((new Calculator)->subtract(3, 1))->toBe(2);
|
||||
});
|
||||
PHP;
|
||||
}
|
||||
|
||||
test('a triggered issue is recorded as itself, not as a pass', function (array $arguments, string $call, int $status): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/CalculatorTest.php', tiaTriggering($call));
|
||||
$project->git()->commit('trigger an issue');
|
||||
|
||||
$project->pest('--tia', ...$arguments);
|
||||
|
||||
$id = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
|
||||
expect($project->graph()['baselines']['master']['results'][$id]['status'])->toBe($status);
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->with([
|
||||
'deprecation' => ["trigger_error('legacy adder', E_USER_DEPRECATED);", 4],
|
||||
'notice' => ["trigger_error('a notice', E_USER_NOTICE);", 3],
|
||||
'warning' => ["trigger_error('a warning', E_USER_WARNING);", 6],
|
||||
])->skipOnWindows();
|
||||
|
||||
test('a cached deprecation still fails the run that asked to fail on one', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/CalculatorTest.php', tiaTriggering("trigger_error('legacy adder', E_USER_DEPRECATED);"));
|
||||
$project->git()->commit('trigger a deprecation');
|
||||
|
||||
$project->pest('--tia', ...$arguments);
|
||||
|
||||
$result = $project->pest('--tia', '--fail-on-deprecation', ...$arguments);
|
||||
|
||||
expect($result->exitCode)->toBe(1, $result->describe())
|
||||
->and($result->tally())->toContain('1 deprecated')
|
||||
->and($result->uncached())->toBe(1, $result->describe());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('replaying a cached issue does not downgrade it to a pass', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/CalculatorTest.php', tiaTriggering("trigger_error('legacy adder', E_USER_DEPRECATED);"));
|
||||
$project->git()->commit('trigger a deprecation');
|
||||
|
||||
$project->pest('--tia', ...$arguments);
|
||||
|
||||
$id = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
|
||||
foreach (range(1, 3) as $ignored) {
|
||||
$project->pest('--tia', ...$arguments);
|
||||
|
||||
expect($project->graph()['baselines']['master']['results'][$id]['status'])->toBe(4);
|
||||
}
|
||||
|
||||
$result = $project->pest('--tia', '--fail-on-deprecation', ...$arguments);
|
||||
|
||||
expect($result->exitCode)->toBe(1, $result->describe());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a failure outranks an issue triggered on the way to it', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/CalculatorTest.php', str_replace(
|
||||
'toBe(3)',
|
||||
'toBe(999)',
|
||||
tiaTriggering("trigger_error('legacy adder', E_USER_DEPRECATED);"),
|
||||
));
|
||||
$project->git()->commit('an issue then a failure');
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
$id = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
|
||||
expect($result->exitCode)->not->toBe(0)
|
||||
->and($project->graph()['baselines']['master']['results'][$id]['status'])->toBe(7);
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a skip outranks an issue triggered on the way to it', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/CalculatorTest.php', str_replace(
|
||||
'expect((new Calculator)->add(1, 2))->toBe(3);',
|
||||
"\$this->markTestSkipped('not today');",
|
||||
tiaTriggering("trigger_error('legacy adder', E_USER_DEPRECATED);"),
|
||||
));
|
||||
$project->git()->commit('an issue then a skip');
|
||||
|
||||
$project->pest('--tia');
|
||||
|
||||
$id = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
|
||||
expect($project->graph()['baselines']['master']['results'][$id]['status'])->toBe(1);
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a suppressed issue is not recorded', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/CalculatorTest.php', tiaTriggering("@trigger_error('quiet', E_USER_DEPRECATED);"));
|
||||
$project->git()->commit('a suppressed deprecation');
|
||||
|
||||
$project->pest('--tia');
|
||||
|
||||
$id = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
|
||||
expect($project->graph()['baselines']['master']['results'][$id]['status'])->toBe(0);
|
||||
})->skipOnWindows();
|
||||
@@ -0,0 +1,170 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
|
||||
test('a filtered run rewrites only the test that ran', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--filter=adds two numbers');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->tally())->toContain('1 passed')
|
||||
->and($result->output)->not->toContain('TIA does not apply to partial runs')
|
||||
->and($delta->writtenCount())->toBe(1, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a filtered run under --tia announces that tia does not apply', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--tia', '--filter=adds two numbers');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->output)->toContain('TIA does not apply to partial runs')
|
||||
->and($delta->writtenCount())->toBe(1, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a test suffix narrows the tier even though every test runs', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--tia', '--test-suffix=Test.php');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->output)->toContain('TIA does not apply to partial runs')
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed')
|
||||
->and($delta->writtenCount())->toBe(Project::TOTAL_TESTS, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a dirty run narrows to the uncommitted test edit', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/GreeterTest.php', <<<'PHP'
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Fixture\App\Greeter;
|
||||
|
||||
test('greets a person', function (): void {
|
||||
expect((new Greeter)->greet('Nuno'))->toBe('Hello, Nuno!');
|
||||
});
|
||||
|
||||
test('greets the world', function (): void {
|
||||
expect((new Greeter)->greet('world'))->toBe('Hello, world!');
|
||||
});
|
||||
|
||||
test('greets again', function (): void {
|
||||
expect((new Greeter)->greet('again'))->toBe('Hello, again!');
|
||||
});
|
||||
PHP);
|
||||
|
||||
$result = $project->pest('--tia', '--dirty');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->output)->toContain('TIA does not apply to partial runs')
|
||||
->and($result->tally())->toContain('3 passed')
|
||||
->and($delta->removed())->toBe(0, $delta->summary())
|
||||
->and($delta->structureMoved())->toBeFalse($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('filtered mode yields to an explicit filter', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--tia', '--filtered', '--filter=adds two numbers');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->output)->toContain('TIA does not apply to partial runs')
|
||||
->and($result->tally())->toContain('1 passed')
|
||||
->and($delta->writtenCount())->toBe(1, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('an env flag narrows exactly like the option it mirrors', function (string $variable): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pestWithEnvironment($project->path(), [
|
||||
$variable => '1',
|
||||
], '--filter=adds two numbers');
|
||||
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->output)->toContain('TIA does not apply to partial runs')
|
||||
->and($delta->writtenCount())->toBe(1, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->with(['PEST_TIA', 'PEST_TIA_FILTERED'])->skipOnWindows();
|
||||
|
||||
test('a partial run does not purge the graph even with --fresh', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--tia', '--fresh', '--filter=adds two numbers');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->output)->toContain('TIA does not apply to partial runs')
|
||||
->and($delta->writtenCount())->toBe(1, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('--no-tia does not stop a partial run from refreshing its own entry', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--no-tia', '--filter=adds two numbers');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->output)->not->toContain('TIA does not apply to partial runs')
|
||||
->and($delta->writtenCount())->toBe(1, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('two partial runs each keep the other entry', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->pest('--filter=adds two numbers');
|
||||
$project->pest('--filter=greets a person');
|
||||
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($delta->writtenCount())->toBe(2, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a shard is a partial run', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--tia', '--shard=1/2');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->output)->toContain('TIA does not apply to partial runs')
|
||||
->and($delta->removed())->toBe(0, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a parallel partial run records the test that ran, like a sequential one', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--tia', '--parallel', '--processes=2', '--filter=adds two numbers');
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->output)->toContain('TIA does not apply to partial runs')
|
||||
->and($result->tally())->toContain('1 passed')
|
||||
->and($delta->writtenCount())->toBe(1, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->skipOnWindows();
|
||||
@@ -0,0 +1,345 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Tests\Fixtures\Tia\Project;
|
||||
|
||||
afterEach(function (): void {
|
||||
Project::destroyAll();
|
||||
});
|
||||
|
||||
test('a detached HEAD does not purge the graph on structural drift', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->detach();
|
||||
|
||||
$project->write('composer.lock', (string) json_encode([
|
||||
'content-hash' => 'drifted',
|
||||
'packages' => [],
|
||||
'packages-dev' => [],
|
||||
]));
|
||||
|
||||
$result = $project->pest('--tia', ...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($project->graphExists())->toBeTrue('the detached run deleted graph.json')
|
||||
->and($delta->isHardSuppressed())->toBeTrue($delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a detached HEAD does not purge the graph with --fresh either', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->detach();
|
||||
|
||||
$result = $project->pest('--tia', '--fresh', ...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($project->graphExists())->toBeTrue('the detached --fresh run deleted graph.json')
|
||||
->and($delta->isHardSuppressed())->toBeTrue($delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a detached HEAD leaves an unreadable graph for a checkout that can rebuild it', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->detach();
|
||||
|
||||
file_put_contents($project->graphDir().'/graph.json', '{not json');
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed')
|
||||
->and(file_get_contents($project->graphDir().'/graph.json'))->toBe('{not json');
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a cached failure whose test file was deleted stops widening later runs', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master', failing: ['adds two numbers']);
|
||||
|
||||
unlink($project->path('tests/Unit/CalculatorTest.php'));
|
||||
$project->git()->commit('drop CalculatorTest');
|
||||
|
||||
$project->pest('--tia', '--filtered', ...$arguments);
|
||||
|
||||
$project->snapshot();
|
||||
$second = $project->pest('--tia', '--filtered', ...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($second->output)->not->toContain('could not be located on disk')
|
||||
->and($second->output)->toContain('No affected tests found')
|
||||
->and($delta->isHardSuppressed())->toBeTrue($delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a complete run reclaims the entry and the edge of a deleted test file', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master', failing: ['adds two numbers']);
|
||||
|
||||
unlink($project->path('tests/Unit/CalculatorTest.php'));
|
||||
$project->git()->commit('drop CalculatorTest');
|
||||
|
||||
$project->pest('--tia', ...$arguments);
|
||||
|
||||
$graph = $project->graph();
|
||||
|
||||
expect($graph['edges'] ?? [])->not->toHaveKey('tests/Unit/CalculatorTest.php')
|
||||
->and(array_column($graph['baselines']['master']['results'] ?? [], 'file'))
|
||||
->not->toContain('tests/Unit/CalculatorTest.php');
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a pruned result does not come back from the fallback', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master', failing: ['adds two numbers']);
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
|
||||
$project->write('tests/Unit/CalculatorTest.php', <<<'PHP'
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Fixture\App\Calculator;
|
||||
|
||||
test('adds two numbers, renamed', function (): void {
|
||||
expect((new Calculator)->add(1, 2))->toBe(3);
|
||||
});
|
||||
|
||||
test('subtracts two numbers', function (): void {
|
||||
expect((new Calculator)->subtract(3, 1))->toBe(2);
|
||||
});
|
||||
PHP);
|
||||
|
||||
$project->git()->commit('rename the test on the branch');
|
||||
|
||||
$project->pest('--tia', '--filtered', ...$arguments);
|
||||
|
||||
$second = $project->pest('--tia', '--filtered', ...$arguments);
|
||||
|
||||
expect($second->output)->not->toContain('previously unsuccessful')
|
||||
->and($second->output)->toContain('No affected tests found');
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('the fallback still reaches a branch that has never run a test file', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master', failing: ['adds two numbers']);
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
|
||||
// A narrowed run mints the branch key holding only the Greeter entries, so
|
||||
// the layering must still serve master's cached failure for the Calculator.
|
||||
$project->pest('--filter=greets a person');
|
||||
|
||||
$result = $project->pest('--tia', '--filtered');
|
||||
|
||||
expect($result->output)->toContain('previously unsuccessful')
|
||||
->and($result->affected())->toBe(2, $result->describe());
|
||||
})->skipOnWindows();
|
||||
|
||||
test('a branch that git no longer knows loses its baseline', function (): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
$project->pest('--tia');
|
||||
|
||||
expect($project->branchKeys())->toBe(['master', 'feature-x']);
|
||||
|
||||
$project->git()->switchTo('master');
|
||||
$project->git()->run(['branch', '-D', 'feature-x']);
|
||||
|
||||
$project->pest('--tia');
|
||||
|
||||
expect($project->branchKeys())->toBe(['master']);
|
||||
})->skipOnWindows();
|
||||
|
||||
test('an unknown cached status is re-run rather than replayed as a failure', function (int $status): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->mutateGraph(function (array $graph) use ($status): array {
|
||||
$id = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
$graph['baselines']['master']['results'][$id]['status'] = $status;
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed')
|
||||
->and($result->uncached())->toBe(1, $result->describe());
|
||||
})->with(['unknown' => -1, 'future' => 9, 'garbage' => 99])->skipOnWindows();
|
||||
|
||||
test('a cached notice, deprecation or warning does not replay as a failure', function (int $status): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->mutateGraph(function (array $graph) use ($status): array {
|
||||
$id = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
$graph['baselines']['master']['results'][$id]['status'] = $status;
|
||||
$graph['baselines']['master']['results'][$id]['message'] = 'cached detail';
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$result = $project->pest('--tia');
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed');
|
||||
})->with(['notice' => 3, 'deprecation' => 4, 'warning' => 6])->skipOnWindows();
|
||||
|
||||
test('a malformed baseline entry cannot break the run', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->mutateGraph(function (array $graph): array {
|
||||
$id = Project::testId('tests/Unit/CalculatorTest.php', 'adds two numbers');
|
||||
$graph['baselines']['master']['results'][$id] = 'nope';
|
||||
$graph['baselines']['master']['tree'] = 'nope';
|
||||
$graph['edges']['tests/Unit/GreeterTest.php'] = 'nope';
|
||||
|
||||
return $graph;
|
||||
});
|
||||
|
||||
$result = $project->pest('--tia', ...$arguments);
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($result->output)->not->toContain('TypeError')
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed');
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
/*
|
||||
* Invariant 1 — sequential and parallel must agree — under a process that is
|
||||
* torn down in the middle of a test file. A worker that flushed what it got to
|
||||
* before dying has not seen enough of that file to license pruning the
|
||||
* siblings it never reached.
|
||||
*/
|
||||
test('a run torn down mid-file does not prune the tests it never reached', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/CalculatorTest.php', <<<'PHP'
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Fixture\App\Calculator;
|
||||
|
||||
test('adds two numbers', function (): void {
|
||||
expect((new Calculator)->add(1, 2))->toBe(3);
|
||||
});
|
||||
|
||||
test('subtracts two numbers', function (): void {
|
||||
exit(0);
|
||||
});
|
||||
PHP);
|
||||
|
||||
$project->git()->commit('a test that kills its own process');
|
||||
|
||||
$project->pest('--tia', ...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($delta->removed())->toBe(0, $delta->summary())
|
||||
->and($delta->shaMoved())->toBeFalse($delta->summary())
|
||||
->and($delta->structureMoved())->toBeFalse($delta->summary())
|
||||
->and(array_keys($project->graph()['baselines']['master']['results']))
|
||||
->toContain(Project::testId('tests/Unit/CalculatorTest.php', 'subtracts two numbers'));
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a fatal error mid-file is a test error, not a truncation', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->write('tests/Unit/CalculatorTest.php', <<<'PHP'
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
use Fixture\App\Calculator;
|
||||
|
||||
test('adds two numbers', function (): void {
|
||||
expect((new Calculator)->add(1, 2))->toBe(3);
|
||||
});
|
||||
|
||||
test('subtracts two numbers', function (): void {
|
||||
undefined_function_here();
|
||||
});
|
||||
PHP);
|
||||
|
||||
$project->git()->commit('a test that fatals');
|
||||
|
||||
$result = $project->pest('--tia', ...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->exitCode)->not->toBe(0)
|
||||
->and($result->tally())->toContain('1 failed')
|
||||
->and($delta->removed())->toBe(0, $delta->summary())
|
||||
->and($delta->structureMoved())->toBeFalse($delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
/*
|
||||
* Invariant 2 — every command lands in exactly one tier and stays inside it —
|
||||
* for the combinations that were never exercised.
|
||||
*/
|
||||
test('a green complete run leaves the graph exactly as it found it', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--tia', ...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($delta->isHardSuppressed())->toBeTrue($delta->summary());
|
||||
})->with([
|
||||
'bail' => [['--bail']],
|
||||
'stop-on-failure' => [['--stop-on-failure']],
|
||||
'compact' => [['--compact']],
|
||||
'parallel bail' => [['--parallel', '--processes=2', '--bail']],
|
||||
'parallel one process' => [['--parallel', '--processes=1']],
|
||||
'parallel more processes than files' => [['--parallel', '--processes=8']],
|
||||
])->skipOnWindows();
|
||||
|
||||
test('--tia --no-tia is a plain run that still refreshes what it executed', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--tia', '--no-tia', ...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->replayed())->toBe(0, $result->describe())
|
||||
->and($result->tally())->toContain(Project::TOTAL_TESTS.' passed')
|
||||
->and($delta->writtenCount())->toBe(Project::TOTAL_TESTS, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('--fresh on a partial run neither purges nor prunes', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$result = $project->pest('--tia', '--fresh', '--filter=adds two numbers', ...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->exitCode)->toBe(0, $result->describe())
|
||||
->and($project->graphExists())->toBeTrue()
|
||||
->and($delta->writtenCount())->toBe(1, $delta->summary())
|
||||
->and($delta->isResultsOnly())->toBeTrue($delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
|
||||
test('a second green run on a feature branch writes nothing at all', function (array $arguments): void {
|
||||
$project = Project::make('master');
|
||||
$project->seed('master');
|
||||
|
||||
$project->git()->switchTo('feature-x', new: true);
|
||||
$project->pest('--tia', ...$arguments);
|
||||
|
||||
$project->snapshot();
|
||||
$result = $project->pest('--tia', ...$arguments);
|
||||
$delta = $project->delta();
|
||||
|
||||
expect($result->replayed())->toBe(Project::TOTAL_TESTS, $result->describe())
|
||||
->and($delta->isHardSuppressed())->toBeTrue($delta->summary());
|
||||
})->with(Project::SEQUENTIAL_AND_PARALLEL)->skipOnWindows();
|
||||
@@ -0,0 +1,17 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!--
|
||||
Runs TiaReplayHooks.php as a suite of its own, so `tests/Features/Tia.php` can
|
||||
replay it without narrowing the run to a path. TIA declines to replay a
|
||||
narrowed run, and a single-file invocation is exactly that.
|
||||
-->
|
||||
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/10.0/phpunit.xsd"
|
||||
bootstrap="../../../vendor/autoload.php"
|
||||
colors="true"
|
||||
>
|
||||
<testsuites>
|
||||
<testsuite name="default">
|
||||
<file>TiaReplayHooks.php</file>
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
</phpunit>
|
||||
@@ -4,19 +4,10 @@ declare(strict_types=1);
|
||||
|
||||
namespace Tests\Fixtures\Tia;
|
||||
|
||||
use Pest\Plugins\Tia\Storage;
|
||||
use RuntimeException;
|
||||
use Symfony\Component\Process\Process;
|
||||
|
||||
/**
|
||||
* A git repository the scenario tests drive.
|
||||
*
|
||||
* Hermetic by construction: every invocation neutralises the machine's global
|
||||
* and system config and carries its own committer identity. Without that, an
|
||||
* ambient `init.defaultBranch = main` would answer questions the scenario means
|
||||
* to leave unanswered, and rows like "resolves to nothing, degrades safely"
|
||||
* would pass by accident.
|
||||
*
|
||||
* @internal
|
||||
*/
|
||||
final readonly class GitRepo
|
||||
@@ -27,9 +18,6 @@ final readonly class GitRepo
|
||||
public const array ENV = [
|
||||
'GIT_CONFIG_GLOBAL' => '/dev/null',
|
||||
'GIT_CONFIG_SYSTEM' => '/dev/null',
|
||||
// `GIT_CONFIG_SYSTEM` does not cover every system-level file git reads:
|
||||
// Apple's git also loads one from inside Xcode, and it sets
|
||||
// `init.defaultBranch`. Only this suppresses all of them.
|
||||
'GIT_CONFIG_NOSYSTEM' => '1',
|
||||
'GIT_AUTHOR_NAME' => 'Pest Fixture',
|
||||
'GIT_AUTHOR_EMAIL' => 'fixture@pestphp.io',
|
||||
@@ -39,9 +27,6 @@ final readonly class GitRepo
|
||||
|
||||
public function __construct(public string $path) {}
|
||||
|
||||
/**
|
||||
* Initialises the repository on `$branch` and commits everything in it.
|
||||
*/
|
||||
public function init(string $branch = 'master'): void
|
||||
{
|
||||
$this->run(['init', '--quiet']);
|
||||
@@ -70,12 +55,6 @@ final readonly class GitRepo
|
||||
$this->run(['checkout', '--quiet', '--detach']);
|
||||
}
|
||||
|
||||
/**
|
||||
* Registers an `origin`, which also decides the graph's storage key:
|
||||
* {@see Storage::projectKey()} prefers the remote's
|
||||
* identity over the path, so two checkouts of one repository — a worktree,
|
||||
* say — share a single graph.
|
||||
*/
|
||||
public function addOrigin(string $url = 'git@github.com:pestphp/tia-fixture.git'): void
|
||||
{
|
||||
$this->run(['remote', 'add', 'origin', $url]);
|
||||
@@ -86,22 +65,12 @@ final readonly class GitRepo
|
||||
$this->run(['remote', 'remove', 'origin']);
|
||||
}
|
||||
|
||||
/**
|
||||
* Points `refs/remotes/origin/HEAD` at a local branch — what
|
||||
* `git remote set-head` would write, without a remote to talk to.
|
||||
*/
|
||||
public function setOriginHead(string $branch): void
|
||||
{
|
||||
$this->run(['update-ref', 'refs/remotes/origin/'.$branch, 'HEAD']);
|
||||
$this->run(['symbolic-ref', 'refs/remotes/origin/HEAD', 'refs/remotes/origin/'.$branch]);
|
||||
}
|
||||
|
||||
/**
|
||||
* Drops `refs/remotes/origin/HEAD` while keeping the remote-tracking branch
|
||||
* — what a CI checkout looks like. `actions/checkout` builds its working
|
||||
* copy with `git init` plus a single-ref `fetch` rather than a `clone`, and
|
||||
* only a `clone` writes that symbolic ref.
|
||||
*/
|
||||
public function unsetOriginHead(): void
|
||||
{
|
||||
$this->run(['symbolic-ref', '--delete', 'refs/remotes/origin/HEAD']);
|
||||
@@ -112,9 +81,6 @@ final readonly class GitRepo
|
||||
$this->run(['config', '--local', $key, $value]);
|
||||
}
|
||||
|
||||
/**
|
||||
* Adds a worktree for a new branch and returns its path.
|
||||
*/
|
||||
public function worktree(string $path, string $branch): string
|
||||
{
|
||||
$this->run(['worktree', 'add', '--quiet', '-b', $branch, $path]);
|
||||
|
||||
@@ -5,20 +5,6 @@ declare(strict_types=1);
|
||||
namespace Tests\Fixtures\Tia;
|
||||
|
||||
/**
|
||||
* What one run did to the graph.
|
||||
*
|
||||
* The three tiers a run may respect, from the conformance matrix:
|
||||
*
|
||||
* - COMPLETE — may change everything.
|
||||
* - RESULTS-ONLY — may change only `baselines[<branch>].results` for tests that
|
||||
* actually ran. It may never remove an entry, nor move `sha`, `tree`,
|
||||
* `edges`, `files` or `fingerprint`.
|
||||
* - HARD-SUPPRESSED — may change nothing at all.
|
||||
*
|
||||
* {@see self::writtenCount()} is the load-bearing measurement, and the reason
|
||||
* {@see Project::sentinel()} exists: without falsified cached values there is no
|
||||
* way to tell "wrote the same values back" from "wrote nothing".
|
||||
*
|
||||
* @internal
|
||||
*/
|
||||
final readonly class GraphDelta
|
||||
@@ -42,9 +28,6 @@ final readonly class GraphDelta
|
||||
return $this->before !== null && $this->after === null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Result entries whose stored values actually moved.
|
||||
*/
|
||||
public function writtenCount(): int
|
||||
{
|
||||
$written = 0;
|
||||
@@ -71,9 +54,6 @@ final readonly class GraphDelta
|
||||
return $written;
|
||||
}
|
||||
|
||||
/**
|
||||
* Result entries that appeared.
|
||||
*/
|
||||
public function added(): int
|
||||
{
|
||||
$added = 0;
|
||||
@@ -88,9 +68,6 @@ final readonly class GraphDelta
|
||||
return $added;
|
||||
}
|
||||
|
||||
/**
|
||||
* Result entries that were pruned.
|
||||
*/
|
||||
public function removed(): int
|
||||
{
|
||||
$removed = 0;
|
||||
@@ -106,14 +83,11 @@ final readonly class GraphDelta
|
||||
}
|
||||
|
||||
/**
|
||||
* The baseline keys after the run — the headline signal for the
|
||||
* default-branch rows, where a phantom key is the defect.
|
||||
*
|
||||
* @return array<int, string>
|
||||
*/
|
||||
public function branchKeys(): array
|
||||
{
|
||||
return array_keys($this->baselines($this->after));
|
||||
return array_map(strval(...), array_keys($this->baselines($this->after)));
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -121,7 +95,7 @@ final readonly class GraphDelta
|
||||
*/
|
||||
public function branchKeysBefore(): array
|
||||
{
|
||||
return array_keys($this->baselines($this->before));
|
||||
return array_map(strval(...), array_keys($this->baselines($this->before)));
|
||||
}
|
||||
|
||||
public function branchKeysMoved(): bool
|
||||
@@ -129,10 +103,6 @@ final readonly class GraphDelta
|
||||
return $this->branchKeysBefore() !== $this->branchKeys();
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the named branch's baseline is byte-identical — how a row proves
|
||||
* the fallback is read-only.
|
||||
*/
|
||||
public function baselineUntouched(string $branch): bool
|
||||
{
|
||||
return ($this->baselines($this->before)[$branch] ?? null)
|
||||
@@ -149,11 +119,6 @@ final readonly class GraphDelta
|
||||
return array_any($this->branchKeys(), fn (string $branch) => $this->baselineField($branch, 'tree', $this->before) !== $this->baselineField($branch, 'tree', $this->after));
|
||||
}
|
||||
|
||||
/**
|
||||
* Compares edges by the file paths they resolve to, not by file id: ids are
|
||||
* an implementation detail that shifts whenever `files` is rebuilt in a
|
||||
* different order.
|
||||
*/
|
||||
public function edgesMoved(): bool
|
||||
{
|
||||
return $this->edgeSets($this->before) !== $this->edgeSets($this->after);
|
||||
@@ -184,17 +149,11 @@ final readonly class GraphDelta
|
||||
return $this->branchKeysMoved();
|
||||
}
|
||||
|
||||
/**
|
||||
* Nothing moved at all.
|
||||
*/
|
||||
public function isHardSuppressed(): bool
|
||||
{
|
||||
return $this->before === $this->after;
|
||||
}
|
||||
|
||||
/**
|
||||
* Results may have moved for tests that ran; nothing structural did.
|
||||
*/
|
||||
public function isResultsOnly(): bool
|
||||
{
|
||||
return ! $this->graphWasCreated()
|
||||
@@ -206,9 +165,6 @@ final readonly class GraphDelta
|
||||
&& $this->added() === 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* A one-line verdict, for failure messages.
|
||||
*/
|
||||
public function summary(): string
|
||||
{
|
||||
if ($this->graphWasCreated()) {
|
||||
@@ -298,9 +254,7 @@ final readonly class GraphDelta
|
||||
$baselines = [];
|
||||
|
||||
foreach ($this->section($graph, 'baselines') as $branch => $baseline) {
|
||||
if (is_string($branch)) {
|
||||
$baselines[$branch] = $baseline;
|
||||
}
|
||||
$baselines[(string) $branch] = $baseline;
|
||||
}
|
||||
|
||||
return $baselines;
|
||||
|
||||
@@ -5,15 +5,10 @@ declare(strict_types=1);
|
||||
namespace Tests\Fixtures\Tia;
|
||||
|
||||
/**
|
||||
* The outcome of one `pest` invocation against a fixture project.
|
||||
*
|
||||
* @internal
|
||||
*/
|
||||
final readonly class PestResult
|
||||
{
|
||||
/**
|
||||
* The run's output, with the terminal's escape sequences taken back out.
|
||||
*/
|
||||
public string $output;
|
||||
|
||||
/**
|
||||
@@ -25,39 +20,26 @@ final readonly class PestResult
|
||||
public int $exitCode,
|
||||
) {
|
||||
$this->output = (string) preg_replace([
|
||||
'#\x1b[[][^A-Za-z]*[A-Za-z]#', // colours, cursor moves
|
||||
'#\x1b\]8;[^\x1b\x07]*(?:\x1b\\\\|\x07)#', // hyperlinks
|
||||
'#\x1b[[][^A-Za-z]*[A-Za-z]#',
|
||||
'#\x1b\]8;[^\x1b\x07]*(?:\x1b\\\\|\x07)#',
|
||||
], '', $output);
|
||||
}
|
||||
|
||||
/**
|
||||
* Tests whose cached result was replayed instead of executed.
|
||||
*/
|
||||
public function replayed(): int
|
||||
{
|
||||
return $this->recapFragment('replayed');
|
||||
}
|
||||
|
||||
/**
|
||||
* Tests that ran because the graph held nothing for them — the count that
|
||||
* betrays a fallback which never resolved.
|
||||
*/
|
||||
public function uncached(): int
|
||||
{
|
||||
return $this->recapFragment('uncached');
|
||||
}
|
||||
|
||||
/**
|
||||
* Tests that ran because a file they depend on changed.
|
||||
*/
|
||||
public function affected(): int
|
||||
{
|
||||
return $this->recapFragment('affected');
|
||||
}
|
||||
|
||||
/**
|
||||
* The `Tests:` summary line, without its label or leading whitespace.
|
||||
*/
|
||||
public function tally(): string
|
||||
{
|
||||
if (preg_match('/^\s*Tests:\s+(.+)$/m', $this->output, $matches) !== 1) {
|
||||
@@ -72,10 +54,6 @@ final readonly class PestResult
|
||||
return str_contains($this->output, $needle);
|
||||
}
|
||||
|
||||
/**
|
||||
* A description of the run, for failure messages that would otherwise say
|
||||
* only that 0 !== 6.
|
||||
*/
|
||||
public function describe(): string
|
||||
{
|
||||
return sprintf(
|
||||
@@ -86,11 +64,6 @@ final readonly class PestResult
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read off the `Tests:` line rather than the whole output: the TIA headline
|
||||
* counts affected *files*, and matching that instead would be a quietly
|
||||
* wrong number.
|
||||
*/
|
||||
private function recapFragment(string $label): int
|
||||
{
|
||||
if (preg_match('/(\d+) '.preg_quote($label, '/').'/', $this->tally(), $matches) !== 1) {
|
||||
|
||||
+47
-167
@@ -5,7 +5,6 @@ declare(strict_types=1);
|
||||
namespace Tests\Fixtures\Tia;
|
||||
|
||||
use FilesystemIterator;
|
||||
use Pest\Factories\TestCaseFactory;
|
||||
use Pest\Plugins\Tia;
|
||||
use Pest\Plugins\Tia\ChangedFiles;
|
||||
use Pest\Plugins\Tia\FileState;
|
||||
@@ -19,28 +18,11 @@ use RuntimeException;
|
||||
use Symfony\Component\Process\Process;
|
||||
|
||||
/**
|
||||
* A throwaway Pest project the TIA scenario tests drive.
|
||||
*
|
||||
* Two facts about Pest shape everything here:
|
||||
*
|
||||
* 1. `bin/pest` derives the project root from **the autoloader it finds**, not
|
||||
* from the working directory — `dirname($autoloadPath, 2)`. So the project
|
||||
* owns a real `vendor/autoload.php` and a real copy of `bin/pest` at the path
|
||||
* a composer install would have put them. A symlinked `vendor` would resolve
|
||||
* `__DIR__` straight back to the Pest repository, and every scenario would
|
||||
* silently measure the wrong project.
|
||||
* 2. TIA cannot *record* without pcov or Xdebug, and CI has neither. So a
|
||||
* scenario never records: {@see self::seed()} writes the graph a recording
|
||||
* run would have written, and the run under test exercises the read path.
|
||||
*
|
||||
* @internal
|
||||
*/
|
||||
final class Project
|
||||
{
|
||||
/**
|
||||
* Test file → the source files a recording run would have linked it to. The
|
||||
* self-edge every test file gets is added on top of these.
|
||||
*
|
||||
* @var array<string, array<int, string>>
|
||||
*/
|
||||
public const array EDGES = [
|
||||
@@ -50,8 +32,6 @@ final class Project
|
||||
];
|
||||
|
||||
/**
|
||||
* Test file → the descriptions it declares, in declaration order.
|
||||
*
|
||||
* @var array<string, array<int, string>>
|
||||
*/
|
||||
public const array TESTS = [
|
||||
@@ -60,15 +40,20 @@ final class Project
|
||||
'tests/Feature/CoversCalculatorTest.php' => ['adds within a feature test', 'subtracts within a feature test'],
|
||||
];
|
||||
|
||||
/**
|
||||
* Every test in the fixture suite.
|
||||
*/
|
||||
public const int TOTAL_TESTS = 6;
|
||||
|
||||
/**
|
||||
* Every project scaffolded so far, so a row cannot leak one by failing
|
||||
* before its own cleanup.
|
||||
* A dataset for the rule that TIA must reach the same outcome sequentially
|
||||
* and in parallel: the same command, run both ways, must leave the same graph.
|
||||
*
|
||||
* @var array<string, array<int, array<int, string>>>
|
||||
*/
|
||||
public const array SEQUENTIAL_AND_PARALLEL = [
|
||||
'sequential' => [[]],
|
||||
'parallel' => [['--parallel', '--processes=2']],
|
||||
];
|
||||
|
||||
/**
|
||||
* @var array<int, self>
|
||||
*/
|
||||
private static array $created = [];
|
||||
@@ -85,13 +70,6 @@ final class Project
|
||||
*/
|
||||
private array $extraPaths = [];
|
||||
|
||||
/**
|
||||
* The root whose graph this project reads and writes.
|
||||
*
|
||||
* Its own, except where a row seeds a worktree: a worktree's `.git` is a
|
||||
* file rather than a directory, so {@see Storage} cannot read the remote
|
||||
* from it and resolves a storage key of its own.
|
||||
*/
|
||||
private string $graphRoot;
|
||||
|
||||
private function __construct(public readonly string $path)
|
||||
@@ -99,14 +77,6 @@ final class Project
|
||||
$this->graphRoot = $path;
|
||||
}
|
||||
|
||||
/**
|
||||
* Scaffolds a project whose default branch is `$branch`, and hands it back
|
||||
* checked out there.
|
||||
*
|
||||
* The repository is realistic on purpose: it has an `origin`, and an
|
||||
* `origin/HEAD` naming `$branch`, which is what a checkout of a real project
|
||||
* looks like and what the default branch is autodetected from.
|
||||
*/
|
||||
public static function make(string $branch = 'master', ?string $overlay = null): self
|
||||
{
|
||||
$project = self::scaffold($overlay);
|
||||
@@ -118,9 +88,6 @@ final class Project
|
||||
return $project;
|
||||
}
|
||||
|
||||
/**
|
||||
* Destroys every project scaffolded so far. Belongs in an `afterEach`.
|
||||
*/
|
||||
public static function destroyAll(): void
|
||||
{
|
||||
while (self::$created !== []) {
|
||||
@@ -128,14 +95,6 @@ final class Project
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A project that is not a git repository at all — for the rows that assert
|
||||
* TIA still demands git, and that a plain run does not care.
|
||||
*
|
||||
* Lives in the system temp directory, so it is outside any enclosing
|
||||
* repository, and owns a real `vendor` rather than a symlinked one, so its
|
||||
* baseline key cannot collide with another fixture's.
|
||||
*/
|
||||
public static function withoutGit(?string $overlay = null): self
|
||||
{
|
||||
return self::scaffold($overlay);
|
||||
@@ -151,12 +110,6 @@ final class Project
|
||||
return $relative === '' ? $this->path : $this->path.DIRECTORY_SEPARATOR.$relative;
|
||||
}
|
||||
|
||||
/**
|
||||
* Overwrites a file in the project.
|
||||
*
|
||||
* Edits must be semantic: TIA hashes PHP at the AST level, so a
|
||||
* comment-only change is not a change at all.
|
||||
*/
|
||||
public function write(string $relative, string $contents): void
|
||||
{
|
||||
$path = $this->path($relative);
|
||||
@@ -171,14 +124,6 @@ final class Project
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Adds a worktree for a new branch, scaffolded so `pest` can run in it, and
|
||||
* returns its path.
|
||||
*
|
||||
* The graph is shared with the main checkout, which is the whole point: both
|
||||
* resolve the same storage key, because {@see Storage} prefers the `origin`
|
||||
* identity over the path.
|
||||
*/
|
||||
public function worktree(string $branch): string
|
||||
{
|
||||
$path = $this->path.'-worktree-'.preg_replace('/[^a-z0-9]+/i', '-', $branch);
|
||||
@@ -190,18 +135,11 @@ final class Project
|
||||
return $path;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs `pest` in the project and returns what happened.
|
||||
*/
|
||||
public function pest(string ...$arguments): PestResult
|
||||
{
|
||||
return $this->pestIn($this->path, ...$arguments);
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs `pest` in `$directory` — a worktree, say — against this project's
|
||||
* graph.
|
||||
*/
|
||||
public function pestIn(string $directory, string ...$arguments): PestResult
|
||||
{
|
||||
return $this->pestWithEnvironment($directory, [], ...$arguments);
|
||||
@@ -222,11 +160,6 @@ final class Project
|
||||
'PARATEST' => '0',
|
||||
'PAO_DISABLE' => '1',
|
||||
'HOME' => $this->home(),
|
||||
// Blanked for the same reason `GitRepo::ENV` blanks git's own
|
||||
// config: the default branch a CI provider reports is the one
|
||||
// *its* build is for. Pest's suite runs on GitHub Actions, so
|
||||
// without this every scenario would autodetect Pest's default
|
||||
// branch instead of the fixture's.
|
||||
'GITHUB_EVENT_PATH' => '',
|
||||
'CI_DEFAULT_BRANCH' => '',
|
||||
...$environment,
|
||||
@@ -244,17 +177,6 @@ final class Project
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes the graph a clean, green recording run on `$branch` would have
|
||||
* written, and remembers it as the snapshot the next {@see self::delta()}
|
||||
* compares against.
|
||||
*
|
||||
* Sentinelled by default: a row almost always wants to know which entries
|
||||
* were written, and only a real recording run's values can answer that.
|
||||
*
|
||||
* `$failing` names descriptions to record as failures — a clean green run
|
||||
* can never cache one, so a row that needs a cached failure to re-run has to
|
||||
* be handed it.
|
||||
*
|
||||
* @param array<int, string> $failing
|
||||
*/
|
||||
public function seed(string $branch, bool $sentinel = true, array $failing = []): void
|
||||
@@ -263,9 +185,6 @@ final class Project
|
||||
}
|
||||
|
||||
/**
|
||||
* Seeds the graph belonging to `$root` — a worktree, say, which resolves a
|
||||
* storage key of its own.
|
||||
*
|
||||
* @param array<int, string> $failing
|
||||
*/
|
||||
public function seedFor(string $root, string $branch, bool $sentinel = true, array $failing = []): void
|
||||
@@ -279,8 +198,6 @@ final class Project
|
||||
$graph->setFingerprint(Fingerprint::compute($root));
|
||||
$graph->setRecordedAtSha($branch, $sha);
|
||||
|
||||
// Hashes the tree as it stands, so the run under test sees nothing as
|
||||
// changed — the same call the recording path makes.
|
||||
$graph->setLastRunTree($branch, $changedFiles->snapshotTree($changedFiles->since($sha) ?? []));
|
||||
|
||||
$graph->markKnownTestFiles(array_keys(self::EDGES));
|
||||
@@ -322,13 +239,6 @@ final class Project
|
||||
$sentinel ? $this->sentinel() : $this->snapshot();
|
||||
}
|
||||
|
||||
/**
|
||||
* The id PHPUnit reports for a test in the fixture suite.
|
||||
*
|
||||
* Mirrors how Pest names generated test classes
|
||||
* ({@see TestCaseFactory}): a wrong id here shows up as
|
||||
* `0 replayed`, which every scenario asserts against.
|
||||
*/
|
||||
public static function testId(string $testFile, string $description): string
|
||||
{
|
||||
$basename = basename($testFile, '.php');
|
||||
@@ -343,41 +253,53 @@ final class Project
|
||||
return 'P\\'.str_replace(DIRECTORY_SEPARATOR, '\\', $relative).'::'.Str::evaluable($description);
|
||||
}
|
||||
|
||||
/**
|
||||
* Falsifies every cached value, so the next {@see self::delta()} can tell
|
||||
* "wrote the same values back" from "wrote nothing".
|
||||
*
|
||||
* `assertions` is only ever falsified where it is already non-zero: risky,
|
||||
* skipped and incomplete statuses are *derived* from "performed no
|
||||
* assertions", so patching those would rewrite the status on replay and
|
||||
* destroy the very discriminator this exists to provide.
|
||||
*/
|
||||
public function sentinel(): void
|
||||
{
|
||||
$this->mutateGraph(function (array $graph): array {
|
||||
foreach ($graph['baselines'] ?? [] as $branch => $baseline) {
|
||||
foreach (array_keys($baseline['results'] ?? []) as $testId) {
|
||||
$graph['baselines'][$branch]['results'][$testId]['time'] = 9.999;
|
||||
|
||||
if ((int) ($baseline['results'][$testId]['assertions'] ?? 0) > 0) {
|
||||
$graph['baselines'][$branch]['results'][$testId]['assertions'] = 42;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return $graph;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Adds a second, empty baseline key, so a lone recorded baseline can no
|
||||
* longer stand in for the default branch.
|
||||
*/
|
||||
public function addBaseline(string $branch): void
|
||||
{
|
||||
$this->mutateGraph(function (array $graph) use ($branch): array {
|
||||
$graph['baselines'][$branch] = ['sha' => null, 'tree' => [], 'results' => []];
|
||||
|
||||
return $graph;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* @param callable(array<string, mixed>): array<string, mixed> $callback
|
||||
*/
|
||||
public function mutateGraph(callable $callback): void
|
||||
{
|
||||
$graph = $this->graph();
|
||||
|
||||
if ($graph === null) {
|
||||
throw new RuntimeException('There is no graph to sentinel.');
|
||||
throw new RuntimeException('There is no graph to mutate.');
|
||||
}
|
||||
|
||||
foreach ($graph['baselines'] ?? [] as $branch => $baseline) {
|
||||
foreach (array_keys($baseline['results'] ?? []) as $testId) {
|
||||
$graph['baselines'][$branch]['results'][$testId]['time'] = 9.999;
|
||||
|
||||
if ((int) ($baseline['results'][$testId]['assertions'] ?? 0) > 0) {
|
||||
$graph['baselines'][$branch]['results'][$testId]['assertions'] = 42;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$this->state()->write(Tia::KEY_GRAPH, (string) json_encode($graph, JSON_UNESCAPED_SLASHES));
|
||||
$this->state()->write(Tia::KEY_GRAPH, (string) json_encode($callback($graph), JSON_UNESCAPED_SLASHES));
|
||||
|
||||
$this->snapshot();
|
||||
}
|
||||
|
||||
/**
|
||||
* The decoded graph, or `null` when there is none.
|
||||
*
|
||||
* @return array<string, mixed>|null
|
||||
*/
|
||||
public function graph(): ?array
|
||||
@@ -400,7 +322,8 @@ final class Project
|
||||
{
|
||||
$baselines = $this->graph()['baselines'] ?? [];
|
||||
|
||||
return is_array($baselines) ? array_keys($baselines) : [];
|
||||
// A branch named `12345` comes back from json_decode as an integer key.
|
||||
return is_array($baselines) ? array_map(strval(...), array_keys($baselines)) : [];
|
||||
}
|
||||
|
||||
public function graphDir(): string
|
||||
@@ -413,43 +336,21 @@ final class Project
|
||||
return is_file($this->graphDir().DIRECTORY_SEPARATOR.Tia::KEY_GRAPH);
|
||||
}
|
||||
|
||||
/**
|
||||
* Remembers the graph as it stands now.
|
||||
*/
|
||||
public function snapshot(): void
|
||||
{
|
||||
$this->snapshot = $this->graph();
|
||||
}
|
||||
|
||||
/**
|
||||
* What has happened to the graph since the last snapshot.
|
||||
*/
|
||||
public function delta(): GraphDelta
|
||||
{
|
||||
return new GraphDelta($this->snapshot, $this->graph());
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes the `vendor` a composer install would have produced.
|
||||
*
|
||||
* Pest is mirrored in at `vendor/pestphp/pest` rather than pointed at,
|
||||
* because Pest locates things from where its own files sit:
|
||||
* `bin/pest` finds the project root by walking up from the autoloader it
|
||||
* loads, and a parallel run picks its worker binary — and with it the
|
||||
* worker's project root — from the directory the runner class was loaded
|
||||
* from. Deferring to the repository's copy would resolve both back to the
|
||||
* Pest repository, and every scenario would quietly measure that instead.
|
||||
*
|
||||
* Hardlinked where the filesystem allows it, so the mirror costs almost
|
||||
* nothing and can never drift from the working tree.
|
||||
*/
|
||||
public function scaffoldVendor(string $directory): void
|
||||
{
|
||||
$pestRoot = dirname(__DIR__, 3);
|
||||
$pest = $directory.'/vendor/pestphp/pest';
|
||||
|
||||
// `overrides`, `resources` and `stubs` come along because Pest loads
|
||||
// them relative to `src` — the same list `BootExcludeList` walks.
|
||||
foreach (['src', 'overrides', 'resources', 'stubs'] as $tree) {
|
||||
self::mirror($pestRoot.'/'.$tree, $pest.'/'.$tree);
|
||||
}
|
||||
@@ -460,21 +361,11 @@ final class Project
|
||||
|
||||
self::mirror($pestRoot.'/composer.json', $pest.'/composer.json');
|
||||
|
||||
// Pest's own autoloader, with the mirrored copy taking precedence: the
|
||||
// repository's `vendor` supplies PHPUnit, Symfony and the plugin
|
||||
// packages, none of which care where they are loaded from.
|
||||
file_put_contents($directory.'/vendor/autoload.php', sprintf(
|
||||
"<?php\n\n\$loader = require %s;\n\$loader->addPsr4('Pest\\\\', __DIR__.'/pestphp/pest/src', true);\n\nreturn \$loader;\n",
|
||||
var_export($pestRoot.'/vendor/autoload.php', true),
|
||||
));
|
||||
|
||||
// Invoking the binary directly skips composer's bin proxy, which is
|
||||
// what would otherwise define `$GLOBALS['_composer_bin_dir']`. Without
|
||||
// it `Pest\Plugin\Loader` looks for `vendor/bin/../pest-plugins.json`
|
||||
// relative to the working directory — so that is where the plugin list
|
||||
// goes, and mirroring the repository's keeps it in step with
|
||||
// composer.json. `vendor/bin` has to exist for the `..` in that path to
|
||||
// resolve, empty though it is.
|
||||
if (! is_dir($directory.'/vendor/bin') && ! @mkdir($directory.'/vendor/bin', 0755, true)) {
|
||||
throw new RuntimeException(sprintf('Unable to create [%s].', $directory.'/vendor/bin'));
|
||||
}
|
||||
@@ -500,9 +391,6 @@ final class Project
|
||||
}
|
||||
|
||||
/**
|
||||
* `Storage` reads `HOME` from the environment, so the graph lands inside the
|
||||
* throwaway project instead of the developer's real `~/.pest`.
|
||||
*
|
||||
* @template TReturn
|
||||
*
|
||||
* @param callable(): TReturn $callback
|
||||
@@ -529,8 +417,6 @@ final class Project
|
||||
throw new RuntimeException(sprintf('Unable to create [%s].', $path));
|
||||
}
|
||||
|
||||
// Realpathed because Pest realpaths its own project root, and macOS
|
||||
// hands out a symlinked temp directory.
|
||||
$real = realpath($path);
|
||||
|
||||
$project = new self($real === false ? $path : $real);
|
||||
@@ -543,8 +429,6 @@ final class Project
|
||||
self::copy(__DIR__.'/overlays/'.$overlay, $project->path);
|
||||
}
|
||||
|
||||
// `ChangedFiles` asks git what changed, so anything the scaffold writes
|
||||
// but the project does not own has to be invisible to it.
|
||||
$project->write('.gitignore', implode("\n", ['/vendor/', '/.home/', '/.phpunit.cache/', '']));
|
||||
|
||||
$project->scaffoldVendor($project->path);
|
||||
@@ -553,10 +437,6 @@ final class Project
|
||||
return $project;
|
||||
}
|
||||
|
||||
/**
|
||||
* Mirrors a file or directory, hardlinking where the filesystem allows it
|
||||
* and copying where it does not.
|
||||
*/
|
||||
private static function mirror(string $from, string $to): void
|
||||
{
|
||||
if (is_dir($from)) {
|
||||
|
||||
@@ -4,8 +4,6 @@ declare(strict_types=1);
|
||||
|
||||
use Fixture\App\Calculator;
|
||||
|
||||
// A second test file over the same source file, so a `Calculator` edit narrows
|
||||
// to two of the three test files rather than to one.
|
||||
test('adds within a feature test', function (): void {
|
||||
expect((new Calculator)->add(10, 5))->toBe(15);
|
||||
});
|
||||
|
||||
@@ -2,8 +2,5 @@
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
// The fixture project has no composer autoloader of its own — its `vendor/`
|
||||
// holds nothing but a shim pointing back at Pest's. Requiring the two classes
|
||||
// here is enough for every test file in the suite.
|
||||
require_once __DIR__.'/../app/Calculator.php';
|
||||
require_once __DIR__.'/../app/Greeter.php';
|
||||
|
||||
@@ -4,9 +4,6 @@ declare(strict_types=1);
|
||||
|
||||
use Fixture\App\Calculator;
|
||||
|
||||
// `test()` rather than `it()`: the harness seeds results by test id, and `it()`
|
||||
// would prefix every description with `it `, leaving Project::TESTS a step away
|
||||
// from what is written here.
|
||||
test('adds two numbers', function (): void {
|
||||
expect((new Calculator)->add(1, 2))->toBe(3);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
require_once __DIR__.'/../app/Calculator.php';
|
||||
require_once __DIR__.'/../app/Greeter.php';
|
||||
|
||||
pest()->tia()->always();
|
||||
@@ -5,7 +5,4 @@ declare(strict_types=1);
|
||||
require_once __DIR__.'/../app/Calculator.php';
|
||||
require_once __DIR__.'/../app/Greeter.php';
|
||||
|
||||
// Declared default branch. Wins over whatever the repository autodetects — the
|
||||
// escape hatch for a checkout with no `origin/HEAD` and a misleading
|
||||
// `init.defaultBranch`.
|
||||
pest()->tia()->defaultBranch('master');
|
||||
|
||||
@@ -5,6 +5,4 @@ declare(strict_types=1);
|
||||
require_once __DIR__.'/../app/Calculator.php';
|
||||
require_once __DIR__.'/../app/Greeter.php';
|
||||
|
||||
// A branch the repository does not have. Accepted as configured — a name that
|
||||
// resolves to no baseline degrades to a full run, which is safe.
|
||||
pest()->tia()->defaultBranch('nope');
|
||||
|
||||
@@ -19,8 +19,6 @@ pest()->in('PHPUnit/GlobPatternTests/SubFolder2/*AsPattern.php')->use(CustomTest
|
||||
|
||||
pest()->in('Visual')->group('integration');
|
||||
|
||||
// Every row scaffolds a throwaway project and runs `pest` in it as a
|
||||
// subprocess, which is far too slow for the unit suite.
|
||||
pest()->in('Features/Tia')->group('integration');
|
||||
|
||||
// NOTE: global test value container to be mutated and checked across files, as needed
|
||||
|
||||
@@ -67,8 +67,6 @@ describe('applyMigrationChanges()', function (): void {
|
||||
|
||||
describe('rerun tracking', function (): void {
|
||||
beforeEach(function (): void {
|
||||
// `hasUnlocatedTestsToRerun()` stats each recorded file to tell a
|
||||
// deleted test apart from a live one, so the files have to exist.
|
||||
$this->projectRoot = sys_get_temp_dir().'/pest-tia-rerun-'.bin2hex(random_bytes(4));
|
||||
mkdir($this->projectRoot.'/tests/Feature', 0755, true);
|
||||
|
||||
|
||||
@@ -9,6 +9,8 @@ $run = function (): ?string {
|
||||
['COLLISION_PRINTER' => 'DefaultPrinter', 'COLLISION_IGNORE_DURATION' => 'true', 'PAO_DISABLE' => '1'],
|
||||
);
|
||||
|
||||
$process->setTimeout(300.0);
|
||||
|
||||
$process->run();
|
||||
|
||||
return removeAnsiEscapeSequences($process->getOutput());
|
||||
|
||||
@@ -17,6 +17,8 @@ test('visual snapshot of test suite on success', function (): void {
|
||||
['EXCLUDE' => 'integration', '--exclude-group' => 'integration', 'REBUILD_SNAPSHOTS' => false, 'PARATEST' => 0, 'COLLISION_PRINTER' => 'DefaultPrinter', 'COLLISION_IGNORE_DURATION' => 'true', 'PAO_DISABLE' => '1'],
|
||||
));
|
||||
|
||||
$process->setTimeout(300.0);
|
||||
|
||||
$process->run();
|
||||
|
||||
return preg_replace([
|
||||
|
||||
Reference in New Issue
Block a user