Making a test suite safe to run in parallel worker processes

What a .NET integration suite has to satisfy before Bobcat's supervisor can split it across worker processes, and the failure modes to expect on the way. Written from four real suites -- Wolverine's PersistenceTests and Redis.Tests, Polecat's Polecat.Tests, and Bobcat's own -- so the list is what actually broke, not what might.

Measured so far

Suite	Tests	Sequential	4 workers	Result
Polecat.Tests	1587	954s	366s (2.6x)	1587/1587 after two fixes
Wolverine PersistenceTests	78	164s	73s (2.2x)	78/78, no source changes
Wolverine Redis.Tests	144	452s	206s (2.2x)	144/144, no source changes

All three plateau quickly -- 8 workers bought 4% over 4 on Polecat and 3% on Redis -- but for two different reasons, and the difference decides whether the plateau is worth attacking.

On Polecat it is contention: beyond a handful of lanes the database container is the bottleneck rather than the worker count. More hardware would move it.

On Redis it is arithmetic. Partitioning is by class, so a class cannot be split, so the largest class is a floor no fleet size goes below:

ceiling = sum(all test durations) / largest class's total duration

Redis is 599s of test time with a 188s compliance class in it, so the ceiling is 3.19x and the floor is ~188s. Measured: 199s at 8 workers. The prediction cost one run; the curve that confirmed it cost four. Worth computing before provisioning a fleet -- if the ceiling is 3x, asking for 8 workers buys nothing but containers.

When the floor is what binds, the fix is not more workers. It is splitting that class, or making the tests in it faster: two compliance classes were 61% of the Redis suite's total test time.

Two ways to run under the supervisor

From the command line, bobcat run --workers 4 runs a spec project on four worker processes, partitioned by test class; --retries, --stall and --stall-action turn on the retry and stall handling below (see Supervised runs). Every lane gets the same environment, so that is enough for a suite whose classes already isolate themselves -- a container or a database per process, say.

In C#, a small host program builds a Supervisor over an MtpWorkerFactory and sets what the command line doesn't reach: a connection string per lane (Step 2), preflight checks, recyclable resources, failure policies, observers, the heartbeat and memory sampling. It drives any MTP test host, Bobcat or not:

var supervisor = new Supervisor(new MtpWorkerFactory(hostPath))
{
    MaxParallelWorkers = 4,
    RetryBudget = new RetryBudget { MaxAttemptsPerTest = 3 }
};

var results = await supervisor.Run();
var exitCode = results.ExitCode;

hostPath is the suite's test host executable, and exitCode is what the program's Main returns.

Across every project of a repository, a GateDeclaration runs one supervisor per (project, framework) unit under resource tracks and one worker budget, builds once, and exits with the worst unit's code -- the local commit-and-push gate. See Local Commit-and-Push Gates.

A TRX file for CI that reads one

A worker in MTP server mode doesn't write its own TRX, so --report-trx on the host gives you nothing under the supervisor. If your CI reads TRX (dorny/test-reporter, a scoreboard), write one from the results:

var startedAt = DateTimeOffset.Now;
var results = await supervisor.Run();

RunReport.WriteTrx(results, path, new TrxOptions
{
    StartedAt = startedAt,
    Storage = "path/to/MySpecs.dll"
});

RunReport.ToTrx(results) returns the same XML as a string. The file has one UnitTestResult per test, so the reporter's counts match the run's. A test that took more than one attempt lists each attempt under InnerResults, with its outcome, duration, message and stack trace. TRX has no outcome for every case the supervisor reports, so some are written out in text:

• A pass on retry is Passed. Its output starts with PASSED ON RETRY, and the run summary has a warning RunInfo naming the test.
• An indeterminate test (its worker died) is Failed, never skipped. Its message starts with INDETERMINATE, and each worker fault gets its own RunInfo.
• An aborted run has ResultSummary outcome="Aborted" and a RunInfo with the abort reason.
• A duration nobody measured is written as zero, because readers require the attribute. RunReport.ToJson reports it as null.

Running the supervisor from Nuke

The Critter Stack repositories host the supervisor in their Nuke build (build/), one target per suite, and CI runs ./build.sh <Target>. Three things to know before you do the same.

A Nuke bootstrap is not free, so a tiny suite is not worth supervising on its own. Weasel measured 7–11 s per job just to restore, compile and start the build project. Its Core, SQLite and MySQL suites run in 3–10 s, so supervising them made each job slower (10–37 s), and Polecat's small AspNetCore and EF Core suites each lost about 0.3 minutes per job. The supervisor pays off on a suite whose run time is large next to that fixed cost -- Weasel's SQL Server legs went from about 100 s to 47–66 s. As a rule of thumb, a suite that already finishes in under about 10 s gets nothing from lanes or retries that is worth the bootstrap. For those suites, either:

• Leave them on plain dotnet test (or run the test executable directly) in their own job, with no Nuke involved, or

• Run several of them from one build invocation, so the bootstrap is paid once. Nuke runs every target named on the command line in one process, in dependency order:

  ./build.sh TestCore TestSqlite TestMySql

  Or give them one aggregate target that depends on each, and have CI call that:

  Target TestSmallSuites => _ => _
      .DependsOn(TestCore, TestSqlite, TestMySql);

  Keep the large, database-backed suites in jobs of their own -- they are where the lanes pay, and they need their own containers anyway.

The build project inherits the repository's Directory.Build.props. MSBuild imports the nearest Directory.Build.props above a project, and build/build.csproj is a project like any other. If the root file sets the plural TargetFrameworks (as most library repositories do), the plural property wins over the build project's own singular TargetFramework, the build project becomes multi-targeted, and ./build.sh fails with:

Your project targets multiple frameworks. Specify which framework to run using '--framework'.

Polecat hit this, and Wolverine's runner hit the same rule before it. It can pass locally and fail only on CI, so check it before the first push. Either fix works:

• Clear the inherited property in build/build.csproj, next to its own TargetFramework:

  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <TargetFrameworks></TargetFrameworks>
  </PropertyGroup>

• Or stop the import altogether with a build/Directory.Build.props (and, if the root has one, a build/Directory.Build.targets) that imports nothing. Wolverine and Weasel do this, which also keeps the root's analyzers, packaging settings and warnings-as-errors out of the build project:

  <Project>
    <!-- Stops MSBuild importing the repository's Directory.Build.props into the build project. -->
  </Project>

The build project no longer needs a JasperFx pin. On 1.0.0-rc.1 and earlier, Bobcat.Supervisor pulled in Bobcat's JasperFx minimum, and a build project pinning JasperFx lower failed restore with NU1605 or NU1109. Since rc.2 neither package references JasperFx (bobcat#744, #745), so a build project referencing the supervisor needs no VersionOverride. If you are still on an earlier release, Compatibility has the recipe.

Durations from the last run: the ledger as a CI artifact

Lanes balance by test count until the supervisor knows what each test costs. TestLedger carries those durations from one run to the next, and the usual way to carry it is as a CI artifact: each run downloads the previous run's ledger, balances on it, folds its own results in, and uploads it again. Nothing is committed, so no commit or merge conflict comes out of a test run.

// A missing file is an empty ledger: the first run balances by count.
var ledger = TestLedger.Load(ledgerPath);

var supervisor = new Supervisor(factory)
{
    MaxParallelWorkers = 4,
    KnownTestDurations = ledger.KnownDurations()
};

var results = await supervisor.Run();

// The run id and stamp are yours to supply, so the fold never reads a clock.
var runId = Environment.GetEnvironmentVariable("GITHUB_RUN_ID") ?? Guid.NewGuid().ToString("N");
ledger.Record(SupervisorLedger.From(results, runId, DateTimeOffset.UtcNow)).Save(ledgerPath);

factory is your MtpWorkerFactory, and results.ExitCode is what the program returns.

In GitHub Actions, fetch the ledger from the newest successful run on main with the gh CLI, and upload it whatever the outcome (a red run's durations are still durations):

permissions:
  actions: read
  contents: read

steps:
  - uses: actions/checkout@v4

  - name: Download the previous run's ledger
    env:
      GH_TOKEN: ${{ github.token }}
    run: |
      run_id=$(gh run list --workflow "${{ github.workflow }}" --branch main --status success \
        --limit 1 --json databaseId --jq '.[0].databaseId')
      if [ -n "$run_id" ]; then
        gh run download "$run_id" --name bobcat-ledger --dir artifacts/ledger || echo "no ledger yet"
      fi

  - run: dotnet run --project build/SupervisedSpecs   # the host above

  - uses: actions/upload-artifact@v4
    if: always()
    with:
      name: bobcat-ledger
      path: artifacts/ledger/ledger.json
      retention-days: 30

What the ledger needs to know:

• Tests are keyed by display name, which stays the same across machines, configurations and target frameworks. The uid does not: xUnit v3's uid hashes the assembly path, so a ledger keyed by uid matched 0 of 2525 tests on another machine, and 2521 once re-keyed by display name. Where two tests share a display name, both are keyed by their uid instead, so two tests' durations are never merged into one. ledger.SharedDisplayNames lists those names; their durations only match on the machine that recorded them.
• The run says how many durations matched: Known durations matched 2521/2525 test(s), or Known durations matched 0/2525 test(s): lanes balanced by test count. These are the first lines of the report, and knownDurations in the JSON. A durations dictionary you build yourself can be keyed by uid or by the ledger's key; both are looked up.
• It keeps the newest 5 runs per test by default (TestLedger.DefaultMaxRunsPerTest). That is enough for a median and a trend, and keeps the artifact small. Choose another number with TestLedger.Empty(n), or TestLedger.Empty(n).Merge(ledger) to change an existing file.
• A ledger written before Bobcat keyed by display name (a version 1 or 2 file) still loads. It is re-keyed by display name, folding one test's uids from different machines into one test. Tests that shared a display name in one of its runs stay keyed by uid. The ledger is written back as version 3.
• Committing the ledger is opt-in. Commit it if you want a laptop's first run to balance by duration too. When two branches change it and conflict, load both sides, Merge them, and save. The merge always gives the same file, so the conflict can't be resolved wrongly.

Step 0 -- be an MTP host

The supervisor drives the test executable over Microsoft.Testing.Platform's server mode. It never loads the test assembly, so a test project needs no reference to Bobcat. It does need to be an MTP host rather than an xUnit native runner:

<OutputType>Exe</OutputType>
<UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>

Check with ./YourTests --list-tests. If that errors with unknown option, you have the xUnit native runner and the supervisor cannot drive it.

Recovery hints ride along in the spec manifest

A suite can say which failures recover and how, with Bobcat.Resilience's [ClearsOnRetry], [ClearsOnRecycle], [ClearsInFreshProcess] and [NeverRecovers] -- on the assembly, a test class or a single test method. (Before 1.0.0-rc.2 they lived in JasperFx's JasperFx.Testing; see Migrating to rc.2.) The sample corpus declares all three scopes, an assembly-wide default and a class and a method overriding it:

[assembly: ClearsInFreshProcess(typeof(Bobcat.Xunit.Samples.Specs.HintProbeException),
    Because = "the assembly-wide default for the recovery hint probes")]

[BobcatFeature("Recovery hints, by class and method"), BobcatScenario]
[ClearsOnRetry(typeof(HintProbeException), Because = "the class-wide override for the recovery hint probes")]
public class ClassHintProbes
{
    [Fact]
    public void recovers_on_a_retry_by_the_class_override()
        => HintProbe.FailsOnFirstAttempt(nameof(recovers_on_a_retry_by_the_class_override));

    [Fact]
    [NeverRecovers(typeof(HintProbeException), Because = "the method-level override for the recovery hint probes")]
    public void never_recovers_by_the_method_override()
        => HintProbe.AlwaysFails(nameof(never_recovers_by_the_method_override));
}

The supervisor still never loads your assembly, so it can't read them itself. The suite reads them, in its own process, and writes them into its spec manifest -- the file a Bobcat-projected suite writes when BOBCAT_LIST_SPECS asks for one. The supervisor asks during discovery and adds what comes back to Supervisor.RecoveryHints before anything runs. This is what the three declarations above write into the sample corpus's manifest:

"recoveryHints": [
  {
    "failureType": "Bobcat.Xunit.Samples.Specs.HintProbeException",
    "kind": "RetryInFreshProcess",
    "scope": "Global",
    "source": "Bobcat.Xunit.Samples",
    "because": "the assembly-wide default for the recovery hint probes"
  },
  {
    "failureType": "Bobcat.Xunit.Samples.Specs.HintProbeException",
    "kind": "RetryInProcess",
    "scope": "Group",
    "source": "ClassHintProbes",
    "because": "the class-wide override for the recovery hint probes",
    "testClass": "Bobcat.Xunit.Samples.Specs.ClassHintProbes"
  },
  {
    "failureType": "Bobcat.Xunit.Samples.Specs.HintProbeException",
    "kind": "FailAndContinue",
    "scope": "Test",
    "source": "ClassHintProbes.never_recovers_by_the_method_override",
    "because": "the method-level override for the recovery hint probes",
    "testClass": "Bobcat.Xunit.Samples.Specs.ClassHintProbes",
    "testMethod": "never_recovers_by_the_method_override"
  }
]

The scope rule survives the trip: assembly < class < method, and the narrowest declaration wins even when it names a less specific type, so a class can override a run-wide default without knowing what that default names. Hints you register on Supervisor.RecoveryHints by hand still work and merge with the suite's; at equal scope and an equally close match, yours win. Turn the reading off with ReadWorkerRecoveryHints = false.

Two things to know:

• This is where "no reference to Bobcat" stops being true. The manifest is written by code Bobcat generates into a projected suite, so a suite with no Bobcat reference -- or one that references Bobcat but has no projected specifications yet -- writes no manifest, and its hint attributes are read by nobody. It still runs, and traits still drive retries. Hand the hints to Supervisor.RecoveryHints instead.
• Over the wire a hint has to name the exact exception type. A failure reported over MTP carries one type name, not its inheritance chain, so a hint naming a base class abstains there -- and a class hint naming SystemException won't override an assembly hint naming TimeoutException the way it would in process. That degrades to a retry that didn't happen, never to one that shouldn't have. tUnit reports no exception type at all, so on tUnit hints never match.

A SpecCatastrophicException stops a supervised run

Throw SpecCatastrophicException when the environment is gone and nothing after this spec can pass -- the application under test will not start, the database is unreachable, a Playwright suite's site never came up. The supervisor reads the type name the worker reports (xUnit writes it into the failure message as Bobcat.Engine.SpecCatastrophicException : …), treats the failure as catastrophic, and aborts the run: no further tests are scheduled, nothing is retried -- not even under a Retry trait with budget to spare -- the report carries AbortReason (catastrophic failure: <your message>), and the exit code is 2.

• It is matched by name, exactly -- Bobcat.Engine.SpecCatastrophicException (or the bare simple name), plus Bobcat.Engine.SpecEnvironmentLostException, which is what the worker's later scenarios fail with once its environment is lost (the in-process halt). Any other subclass, or an exception of your own with the same simple name in another namespace, is an ordinary error.
• tUnit erases exception types on the wire, so under the supervisor a tUnit worker's SpecCatastrophicException is an ordinary error: the run carries on, and a Retry trait retries it. If that matters, register an IFailurePolicy that aborts on the message instead.
• It is not what stops ordinary exceptions from being retried. A plain exception (a timeout, a refused connection) is a critical failure, and critical failures are retried under a Retry trait -- that is where flakiness in an integration suite mostly comes from. Opt one type out with [NeverRecovers(typeof(X))].

UseMicrosoftTestingPlatformRunner changes only the executable's entry point -- the VSTest adapter path is untouched, so dotnet test, --filter, TRX and coverlet keep working. Verified on Wolverine, which runs xUnit v3 in VSTest compatibility mode: dotnet test --filter "Category!=Flaky" still reported 160 passed with the property set.

Hazard: same output path, different entry point. A build without the property silently yields a non-supervisable executable. Put it in Directory.Build.props, not a per-job -p: override.

Hazard (found on CritterWatch): a test project with <GenerateTargetFrameworkAttribute>false</GenerateTargetFrameworkAttribute> breaks the xUnit v3 VSTest adapter -- the assembly reports UnknownTargetFramework, the adapter's process launcher cannot decide how to start the (now-executable) project, and every dotnet test run dies with Catastrophic failure: … Could not launch test process. The exe itself still answers --list-tests and -automated, which makes the symptom look like an adapter bug rather than a project-property one. Strip the property from test projects when migrating to v3; suites carry it for assembly-version-reporting reasons that never apply to a test host.

Hazard (found on CritterWatch): pathological test output breaks the MTP wire. MTP server mode is JSON-RPC, and Microsoft.Testing.Platform's serializer throws ArgumentException: The JSON value of length N is too large on any single string past ~½GB -- at which point the worker faults and every unreported test goes Indeterminate. A CritterWatch test was dumping a 698MB tracked-session log into ITestOutputHelper; dotnet test had been silently absorbing it into an 868MB TRX nobody read, so the supervisor was the first consumer to notice. The fix is fixing the test (the giant output traced to a real message loop), not widening the wire -- but recognize the signature: worker fault carrying Json.SerializeAsync + ValueTooLarge, with a pass/indeterminate split at the choke point. Sister symptom for triage: on Bobcat before 0.7.0 those indeterminates printed as uid hashes, not display names -- fixed by issue #82, which carries the discovery display name onto synthesized never-reported outcomes.

Step 1 -- the partitioning contract

The supervisor splits work by test class, never by individual test. That is a correctness rule, not a tuning choice: every framework's isolation contract is per class or collection, so a class's fixtures and static state assume one process.

Splitting Wolverine's suite per test failed 1–4 tests non-deterministically; per class it was 78/78 at the same wall clock. The mechanism was a class whose setup read

var schemaName = "sqlserver" + ++count;   // static int

Split across four processes, each restarts at zero and all four collide. Nothing about that is visible in the test list, which is why the rule has to be conservative.

You inherit this for free. The thing you must supply is what follows.

Step 2 -- per-worker resource isolation

Class-level partitioning keeps a class together. It cannot stop two different classes that share a database landing in different workers. So each worker needs its own database:

ConnectionStrings = new WorkerConnectionStrings()
    .Add("postgres", worker => $"Host=localhost;Database=specs_w{worker.Lane}")
    .Add("rabbitmq", "amqp://guest:guest@localhost:5672")

That is the ConnectionStrings property of the MtpWorkerFactory, set in its initializer.

Each worker is launched with ConnectionStrings__postgres and ConnectionStrings__rabbitmq. That is the .NET Aspire convention, so the suite reads them the way an application does:

var connectionString = configuration.GetConnectionString("postgres")
                       ?? "Host=localhost;Port=5433;Database=postgres;Username=postgres;Password=postgres";

or, without IConfiguration, Environment.GetEnvironmentVariable("ConnectionStrings__postgres"). Bobcat.Supervisor.Tests proves it against a real worker process that reads its strings exactly that way. The same suite then runs unchanged in three places: under the supervisor; under an Aspire AppHost that passes the resource with WithReference(...); and by hand, as ConnectionStrings__postgres=... dotnet run. The fallback keeps a plain dotnet test working.

Rules the convention depends on (issue #414):

• Always a whole connection string, never a fragment. Hand a worker something it can use as-is, not a database name, schema or port to stitch onto a base string. Step 3 shows how stitching goes wrong.
• Brokers too. RabbitMQ, Kafka, Azure Service Bus and NATS each get their complete connection string or URI under a name, never a host or vhost fragment. Use the string overload of Add for a resource every worker shares.
• Provisioning is yours. Create one database (or vhost, or namespace) per lane before the run; WorkerConnectionStrings only says which one each worker gets. Lane is bounded by MaxParallelWorkers, so you provision as many as workers you asked for. Discovery, isolated and recycled launches report lane 0, and the provider receives the whole WorkerLaunchContext if one of them should get something else.

LaneDatabases provisions a database per lane on PostgreSQL or SQL Server (bobcat#700). It references no driver: pass your own ADO.NET provider, and the administrative connection string is the template each lane's string is built from, with the provider's connection string builder setting only the database:

var lanes = LaneDatabases.Postgres(provider,
    "Host=localhost;Port=5433;Database=postgres;Username=postgres;Password=postgres", prefix: "specs_w");
// or LaneDatabases.SqlServer(provider, "Server=localhost;Database=master;User Id=sa", prefix: "specs_w")

await lanes.Recreate(workers);        // drop and create specs_w0 .. specs_w{workers-1}

var connectionStrings = new WorkerConnectionStrings().Add("postgres", lanes.ForWorker);

provider is NpgsqlFactory.Instance or SqlClientFactory.Instance, and connectionStrings goes on the MtpWorkerFactory as above.

• Recreate drops and recreates every lane database, so each run starts empty: a database carrying the previous run's tables lets a schema change pass locally and fail on CI's fresh one. On PostgreSQL it uses DROP DATABASE … WITH (FORCE) (PostgreSQL 13 or later); on SQL Server it sets the catalog SINGLE_USER WITH ROLLBACK IMMEDIATE first. Both end the previous run's connections.
• EnsureCreated creates only the lane databases that are missing, for a suite whose tests reset their own schemas.
• The prefix must be lower-case letters, digits and _: it goes into DDL, where a name cannot be a parameter.
• ConnectionStringFor(lane) is there for an EnvironmentFor that names its own variable (["POLECAT_TESTING_DATABASE"] = lanes.ConnectionStringFor(context.Lane)).

EnvironmentFor is still there for anything that is not a connection string. It is layered above ConnectionStrings, so it wins when both set the same variable.

A suite that starts its own container per process already has this, for free. Wolverine's Redis tests spin up a Testcontainers Redis from a [ModuleInitializer], so every worker gets its own broker with its own port and there is nothing to wire -- which is why that suite parallelised with no source changes at all. Check for this before building per-worker plumbing.

Two things it costs, both worth knowing rather than discovering:

• Every process pays the container start, before Main, on the critical path -- ~0.85s for redis:7-alpine including discovery, and far more for heavier images. Irrelevant at four workers; it dominates any mode that starts one process per test.
• Nothing reaps those containers if the reaper is off. A [ModuleInitializer] that never disposes leaks one container per process; with Testcontainers' Ryuk disabled this reached 49 orphans on one machine. Not Bobcat's bug, but Bobcat's process multiplication is what makes it visible.

Step 3 -- the bugs this exposes

All were found in real suites, all silent, and all are the reason a first parallel run goes red.

Text-rewriting a connection string

// WRONG — only rewrites while the catalog happens to be "master"
ConnectionSource.ConnectionString.Replace("Initial Catalog=master", $"Database={name}")

Point the environment variable at anything else and the literal is absent, the replace matches nothing, and the "other" database quietly resolves to the current one. The test then asserts against itself and passes or fails for reasons unrelated to what it tests. Use a DbConnectionStringBuilder.

The same shape appears as assertions that bake the environment into an expected value:

var expected = new Uri("wolverinedb://postgresql/localhost/postgres/wolverine");

That one is honest about what it is -- it just cannot survive a renamed database.

Server-scoped names

A database a test creates is a sibling of the worker's database, not a child of it. Per-worker catalogs therefore do not isolate them, and a hardcoded name is one object shared by every worker:

// WRONG — every worker races to create and drop the same database
private const string DbA = "polecat_tenant_a";

// RIGHT — carries the worker's scope
private static readonly string DbA = ConnectionSource.Scoped("tenant_a");

Under four workers this surfaced as "User does not have permission to alter database 'polecat_tenant_a'" -- non-deterministically, which is the signature of contention rather than a deterministic bug.

Applies to anything at server scope: databases, logins, linked servers, Agent jobs, and any fixed TCP port a test binds.

Schema names do not need this. They live inside the catalog, so per-worker databases already separate them. Polecat has SchemaName = "doc_usage" in nine files and it is fine.

The same shape, read from the other side: server-wide catalogs. A test does not have to create anything at server scope to cross lanes. It only has to look at a view that shows the whole server: pg_locks, pg_stat_activity, or SQL Server's sys.dm_* views. A database per lane does not narrow those views. Each lane sees every other lane's sessions in them.

Found on Weasel's Postgres suite. A database per lane isolates advisory locks, because a lock is scoped to its database. But NextLockId() restarts at the same number in every process, and the test helper looked up the holders of a lock id in pg_locks and then terminated them. So one lane found the other lane's backends holding the same id and killed their connections. That failed whichever test in the other lane held the lock. Two tests failed at 2 and 4 workers, and never in process.

-- WRONG: sees every lane's backends
select pid from pg_locks where locktype = 'advisory' and objid = @id;

-- RIGHT: only this lane's database
select l.pid from pg_locks l
join pg_database d on d.oid = l.database
where l.locktype = 'advisory' and l.objid = @id and d.datname = current_database();

With the filter on current_database() there were no failures in 24 local and 8 CI runs. A test that observes a server-wide catalog gets wrong counts. A test that acts on what it sees (pg_terminate_backend, KILL) breaks other lanes. Filter every such query to the lane's own database: current_database() on Postgres, database_id = DB_ID() on SQL Server. Shipped code that runs the same query needs the filter too.

Environment-dependent test identities

A theory whose data is computed from the environment gives the same test a different identity in every worker:

// WRONG — the argument (and therefore the test's uid) depends on which worker computes it
public static IEnumerable<object[]> AgentUris =>
    [[$"wolverinedb://postgresql/localhost/{Servers.PostgresDatabaseName}/wolverine"], ...];

// RIGHT — a stable sentinel, resolved inside the test body
[InlineData("default")]
public async Task build_each_agent_smoke_test(string uriString)
{
    var uri = uriString == "default"
        ? new Uri($"wolverinedb://.../{Servers.PostgresDatabaseName}/wolverine") : uriString.ToUri();

The supervisor discovers in one process and executes in another with a different per-worker environment, so the uid the plan asks for does not exist in the worker that receives it. Found on Wolverine's MartenTests: the symptom is "the worker finished without reporting a result for this test" -- reported as indeterminate, and a retry appears to "fix" it whenever the retry process happens to compute the same environment discovery did, which makes it masquerade as an ordinary flake. Test identity must be a function of the code alone, never of the environment.

Theory data the framework cannot serialize

A theory whose rows xUnit can't serialize (a MemberData row holding a delegate, a type with no serializer, an object built in the data method) is discovered as one test, and every row then reports its verdict under that one test id. The supervisor folds those verdicts worst-first, so a failing row still fails the run, and lists the theory and its rows under Collapsed theories in the report (SupervisorResults.Collapsed). But the rows can't be selected, retried or timed one by one, and a direct run sees more tests than the supervised one.

// WRONG: the row is a lambda, so the theory collapses into one test
public static IEnumerable<object[]> Cases => [[(Func<int>)(() => 1)], [(Func<int>)(() => 2)]];

// RIGHT: pass a row key and look the value up inside the test
[Theory, InlineData("one"), InlineData("two")]
public void each_case(string key) => run(Cases[key]);

A collapse whose every row passes leaves the exit code at 0, so it is easy to carry forever. Set FailOnCollapsed (off by default) and such a run fails with exit code 1 and a RUN FAILED line naming each collapsed theory and the fix, with every row listed below it:

var supervisor = new Supervisor(factory) { FailOnCollapsed = true };

The JSON report carries it as failOnCollapsed and collapsedFailure, and GitHubAnnotations writes it as an ::error.

Step 4 -- containers

DockerComposeResource starts the containers and knows when they are usable. Readiness comes from the compose file's own healthcheck, not from a Bobcat package per image -- the healthcheck lives where the credentials and ports already are.

Four descending tiers, and the one used is reported in ReadinessSource so a weak check is never silent: a supplied Probe, then the declared healthcheck, then a TCP connect, then "the container is running" (which only means the entrypoint has not exited).

Recycle() uses --force-recreate rather than restart, because recycling means throwing the thing away.

Step 5 -- choosing how many workers

Three measurements decide the worker count. Take them in this order, because each one can make the next one unnecessary. They come from running the supervisor on Marten, Polecat, Weasel and Fisher CI (lessons of record).

1. Measure parallel efficiency first

Run the suite at one worker and read parallel efficiency in RunReport's Timing section. The figure is split by where the overlap came from: lanes running side by side, or the test framework's own parallelism inside one worker. A single-lane run above 1x means the framework is already running tests in parallel, and more workers compete for the same cores.

Fisher measured this. xUnit already runs Fisher.Tests' collections in parallel in one process, so its parallel efficiency read 11–41x locally, and 6.6–7.3x at 2 workers on CI. That predicted, before any tuning, that more workers would buy little. Going from 1 to 2 workers gave about 10–15%, and 4 workers was within noise. Fisher commits 2 workers.

A suite that runs serially in process (a shared database, DisableTestParallelization, one big collection) reads close to 1x, and that is the suite where workers pay. Marten's EventSourcingTests went from 4m36s at 1 worker to 1m27s at 4 locally.

2. Measure memory, then decide on a clamp

A common convention in CI build scripts is to clamp the requested workers to Environment.ProcessorCount / 2. Wolverine adopted it after four Marten hosts beside Postgres killed a 4-vCPU/16 GB GitHub-hosted runner (The runner has received a shutdown signal, which is the runner dying, not a test). Marten, Weasel and Fisher use the same clamp unchanged. On a 4-vCPU runner it gives 2 lanes. Marten measured what that costs: 2 workers give about 1.4–1.5x on the runner, where the same split gives 1.8x on a developer machine.

The clamp is a default, not a rule. Polecat's old runner had bypassed its own clamp and ran 4 SQL Server-backed workers on the same 4-vCPU runners for 100 of 100 green runs, at about 300 MB RSS per worker. Clamping it would have halved its parallelism for no measured reason, so polecat.yml passes 4 workers explicitly, with the evidence in a comment.

So measure before choosing. Set ResourceSampleInterval and read peak worker RSS in the report (RunResources), then pick the count the runner can hold. Keep the evidence next to the setting. One gap: the sampler sees worker processes only. On a database-backed suite the database container is often the largest process, and the report cannot see it.

3. Duration balancing helps only when a few classes are heavy

Supervisor.KnownTestDurations (from the committed ledger) balances lanes by measured duration instead of test count. That helps only when the suite has a few classes much heavier than the rest. Spreading those classes across lanes is the whole gain:

• Weasel's SQL Server suite, where it paid: 54–56 s by count to 25–36 s from the ledger at 4 local workers, and 62–67 s to 40–56 s on CI.
• Marten's EventSourcingTests, where it did not: 1m27s by count and 1m29s from the ledger at 4 workers. Many classes of similar size balance just as well by count.
• Polecat, unresolved: two ledger rounds measured more summed test time than the count round, with the same plan. That may be runner variance. It may also be that packing the heavy database classes onto one shared SQL Server raised contention. A per-test duration cannot express contention, and three runs could not tell the two apart.

The Timing section now separates the two (bobcat#705). With known durations supplied it reports a runner speed index: this run's first-attempt time for every test the ledger knows, against the ledger's median for the same tests. 1.30x means the runner was 30% slower than the history, whatever the plan did. With more than one lane it reports the lane plan, predicted against realised: what WorkPlan expected each lane to cost, what the lane's tests took, and the lane's own wall clock. The figure to compare across a count round and a ledger round is the slowest lane over the mean, planned and realised. It is a ratio, so a slow runner does not move it. A plan that expected 1.00x and realised 1.40x balanced badly; a speed index of 1.30x with an even realised spread means the runner was slow, not the plan. A count-balanced plan has no predicted time (the planner charges a nominal second per test) but still reports both spreads. The JSON carries the same figures as timing.runnerSpeed and timing.lanePlan. Report, don't act.

Balancing cannot go below the largest class (see the ceiling under "Measured so far"). The Timing section names that class, and that is the class to split or speed up.

Environment checks: run under the supervisor

The supervisor checks the environment once, before it launches any worker -- not even the discovery worker starts until every check has run. The contract is the same as JasperFx's IStatefulResource.Check: a check that throws has failed.

var supervisor = new Supervisor(factory);

// A check that throws has failed. This one asks whether anything answers on the database's port.
supervisor.Preflight.Add("Postgres is reachable", async token =>
{
    using var probe = new System.Net.Sockets.TcpClient();
    await probe.ConnectAsync("127.0.0.1", port, token);
});

For a Critter Stack application, WolverineFx.Bobcat adds Preflight.AddContainerChecks(...), which runs every check a host's container knows about through JasperFx's EnvironmentChecker; see Testing with Bobcat in the Wolverine docs.

A resource registered with AddRecyclableResource is checked too, through its own Check, without being added to Preflight. Every check runs, even after one fails, so one preflight reports everything that is wrong.

A failed check aborts the run. No worker is launched, no test runs, and the result is one red outcome with the reason in it rather than a test failure:

• SupervisorResults.AbortReason lists each failed check and its message:

  Environment preflight failed (1 of 2 checks):
    ✗ Postgres is reachable: Connection refused

• RunReport.ToText starts with RUN ABORTED: and the JSON report carries abortReason.

• The exit code is 2, the code for an aborted run, not 1, the code for test failures.

An in-process dotnet test run has no preflight. Nothing runs before the test framework starts executing tests, so a missing broker or database shows up as every test that touches it failing with its own connection error -- hundreds of red tests and one cause.

So run under the supervisor in CI and in fleets, where a red result should say what broke rather than how many tests it broke.

When a parallel run hangs, bloats, or is cut short

Everything here is off by default and reporting-first -- the same guardrail as the timing report.

• HeartbeatInterval emits one log line however many lanes are running, leading with the longest-running test -- a stuck run shows as that figure climbing before any threshold fires.
• StallThreshold (and per-test StallThresholdFor) fires a TestStalled observer callback and a STALLED: log line once per attempt, collected on SupervisorResults.StalledTests. RunReport.ToText puts a Stalled section at the top, ahead of the retries, whenever there is one -- green runs included -- naming each test, its lane, how long it had been in flight and whether it finished (finished Passed after 5m 12s, or never finished). Whether the supervisor then acts is the separate, opt-in StallAction: Report (default), KillAndRetry (kill the wedged worker, give the stalled test one fresh-process retry, resume its innocent batch-mates in their lane), or AbortRun. Stall-managed outcomes never spend the RetryBudget and never count as flaky -- a wedge is not a flake.
• MtpWorkerFactory.OnBeforeKill hands you the live process (with its pid -- the supervisor surfaces it, never guesses) before a forced kill, which is where a dotnet-dump collect belongs.
• ResourceSampleInterval brackets every attempt with worker RSS samples; per-test retained deltas are attributed only when the attempt had the process to itself, and unmeasured is never zero-filled.
• AddMemorySampleTarget samples something besides the workers at the same cadence -- on a hosted runner the database container is often the larger consumer, and what kills the runner is the total. MemorySampleTargets.Container("myapp-postgres-1", "postgres") reads docker stats --no-stream (the container's whole cgroup, so every Postgres backend counts, and it works under Docker Desktop's VM); MemorySampleTargets.Process("sqlserver", () => pid) reads one host process's resident set by a pid fetched on every sample. Each target is reported beside the worker peaks -- • postgres: 300 MB → 2000 MB peak (+900 MB) in the text report, a targets array in the JSON memory block. A target that cannot be measured (no docker on the PATH, the container gone) reads not measured with null figures, never zero. Only sampled while ResourceSampleInterval is set.
• Supervisor.Snapshot() is what a cancelled run leaves behind: wire your CI job's SIGTERM to it and write the flakiness ledger from a result stamped IsPartial, instead of losing the whole run to a capped job.

Triage order for a red first run

1. Deterministic and database-name-shaped -- text-rewritten connection strings, or assertions naming the database. Fix the suite; these are latent bugs regardless of parallelism.
2. Non-deterministic, moves between runs -- server-scoped name collisions. Scope the names. Connections dropped mid-test, or counts of locks or sessions that are too high, point at a query over a server-wide catalog. Filter it to the lane's database.
3. Non-deterministic, same class every time -- a class split across workers. Report it: the partitioner should have prevented this.

A fourth shape is worth ruling out before the first parallel run rather than triaging after it: a test that only ever passed because something else ran first. That sweep is built -- IsolationSweep (in Bobcat.Supervisor) runs every partition alone, diffs against a baseline full-suite run, and classifies what moved: OrderDependent (passed with the suite, failed alone -- the bug this exists to find), InterferenceVictim (the same defect seen from the other end), FailedInBoth (ordinary red), and EnvironmentSensitive (failed in the concurrent sweep but passed a serial confirmation run -- the sweep's own contention, deliberately reported rather than counted). Granularity defaults to per-class, matching what the supervisor actually guarantees; per-test is strictly finer and catches ordering within a class at the cost of one process per test. It is the one class of failure that parallelism exposes without having caused.

What this does not buy you

Parallelism speeds up a suite; it does not make a flaky one reliable. If stability is the problem, that is the retry/quarantine half of the supervisor (RetryBudget, the Retry, Isolated and RecycleOnRetry traits -- [Trait("Isolated", "true")] on xUnit -- and RecoveryHint), and it is a separate decision -- Polecat needed none of it.

Nor does it fix a suite dominated by one slow test. Wall clock is the slowest lane, so the largest partition sets the floor: Wolverine's run bottomed out at ~67s because one test slept for 61 of them. The evidence for attacking that is in the run itself: RunReport's Timing section ranks the slowest tests by share of wall clock, reports parallel efficiency, and prices retries and isolation -- report, don't act, so whether a slow test is a bug or honestly slow stays a judgement. Parallel efficiency is split by where the overlap came from: the lanes running side by side, or the test framework's own parallelism inside one worker. A single-lane run above 1x is the second kind, which more workers would not add to. Across runs, the ledger (TestLedger, see ledger-design.md) carries duration trends, and ledger.KnownDurations() feeds the lane balancer so a second run balances on measured durations instead of test counts. It is keyed by display name, so a ledger from another machine, configuration or target framework still matches, and the run says how many durations matched (see the ledger as a CI artifact).

And it does not shorten a CI run whose wall clock is set by a different job. Wolverine's matrix runs ~30 jobs concurrently, so the workflow takes as long as its slowest; parallelising the Redis job from 9m to 5m moved that number by zero, because two 18m jobs set it. Measure which suite is on the critical path before optimising the one in front of you.