# Mutalisk

Modern Elixir mutation testing. Mutalisk makes small changes ("mutants") to
your source — flip a `<` to `<=`, swap `Enum.filter` for `Enum.reject`, drop a
clause — then runs your test suite against each one. If a test fails, the
mutant is **killed** (your tests caught the bug). If every test still passes,
the mutant **survived** — a real gap in your suite's coverage that line
coverage can't see.

## Install

Add to your `mix.exs`:

    defp deps do
      [
        {:mutalisk, "~> 0.1", only: [:test], runtime: false}
      ]
    end

Run `MIX_ENV=test mix deps.get`.

Requires Elixir 1.19 or newer. Mutalisk is validated on Linux and macOS;
child-process cleanup uses Unix tools (`ps`, `pgrep`, and `kill`). Windows
is not currently tested.

## Usage

    MIX_ENV=test mix mut

That builds the mutants, runs your tests against each, prints a summary, and
writes `stryker.report.json` (viewable in the
[Stryker mutation testing elements](https://stryker-mutator.io/) HTML viewer).

Because the dependency is `only: [:test]`, the task only exists under the test
environment — always run it with `MIX_ENV=test`. `runtime: false` keeps Mutalisk out of your app's normal test start
graph; mutation workers enable the runtime dependency in their own overlay.
Plain `mix mut` in the default env fails with "task could not be found".

`mix mut` exits 1 when the mutation score is below **80%**, so CI fails by
default. Change the threshold with `--fail-at` or `fail_at:` in config;
`--fail-at 0` turns the gate off.

Useful flags (see `MIX_ENV=test mix help mut` or `MIX_ENV=test mix mut --help` for the full
list):

    MIX_ENV=test mix mut --files "lib/my_app/core/**/*.ex"   # only mutate some files
    MIX_ENV=test mix mut --concurrency 8                      # parallel workers
    MIX_ENV=test mix mut --fail-at 70                         # exit non-zero below 70%
    MIX_ENV=test mix mut --reporters terminal,html           # pick reporters

Mutalisk runs your suite as-is (it does **not** pass `--warnings-as-errors`), so
compiler warnings in the target do not affect the mutation score. One exception:
fallback-engine recompiles load your project and honour `elixirc_options:
[warnings_as_errors: true]` from `mix.exs`, exactly as `mix compile` would — a
mutation that introduces a warning on such a project is reported `CompileError`
(invalid), not scored. Schema-engine mutants are compiled once with the real
`mix compile` and are unaffected. Keep your own `mix compile
--warnings-as-errors` gate in CI if you rely on it.

## Interpreting the score

The terminal summary reports a **mutation score** and a per-status breakdown:

- **Killed** — a test failed on the mutant. Good: your suite caught it.
- **Survived** — all selected tests passed on the mutant. A gap: no test
  distinguishes the mutated behaviour from the original.
- **Timeout** — the mutant made a test exceed its time budget (usually an
  infinite loop the mutation introduced). Counted as **detected** (like
  killed).
- **CompileError** / **RuntimeError** — the mutant didn't compile, or crashed
  the run for reasons unrelated to a test assertion. Excluded from the score.
- **NoCoverage** — no selected test executed for the mutant. Counted as
  undetected, like a survivor, because the report cannot prove a test guards it.
- **Ignored** — skipped (unsupported target, `@mutalisk_ignore`, or an
  `exclude` match). Excluded from the score.

The score is `detected / (detected + survived + no_coverage)` where
`detected = killed + timeout`. CompileError, RuntimeError, and Ignored mutants
are excluded from both — they don't measure test quality. This matches the
score the Stryker HTML viewer derives from the same report.

## Handling surviving mutants

Each surviving mutant is a specific, located suggestion. For each one, pick:

1. **Write or strengthen a test** (the usual answer). The survivor tells you
   exactly which behaviour is unguarded — e.g. "replace `<` with `<=` on
   `lib/foo.ex:12` survived" means no test pins the boundary.
2. **Decide it's equivalent** — the mutation produces behaviour that is
   genuinely indistinguishable (e.g. a defensive branch the spec doesn't
   constrain). Mutation testing can't decide equivalence for you; some
   survivors are expected.
3. **Exclude it from mutation** when the code is intentionally untested
   (generated/DSL modules, glue): add `@mutalisk_ignore true` to the module
   (see below), or an `exclude` path pattern in config.

Use `--reporters html` for a clickable report of every survivor with its
source line and mutation, or `--reporters github-actions` to get inline PR
annotations in CI.

## Source-level ignores

Add `@mutalisk_ignore true` anywhere in a module's body to exclude every
mutant in that module:

    defmodule MyApp.Generated do
      @mutalisk_ignore true
      # ... no mutants are produced here ...
    end

## Configuration

Settings layer, lowest to highest precedence:

    .mutalisk.exs project file  <  legacy config :mut  <  config :mutalisk  <  CLI flags

(`config :mut` is a deprecated namespace kept for compatibility; prefer
`config :mutalisk`.)

`.mutalisk.exs` (project root, optional) is a plain keyword list:

    # .mutalisk.exs
    [
      selection: :coverage_with_static_fallback,
      fail_at: 75.0,
      concurrency: 8,
      exclude: [~r"lib/my_app_web/router.ex"]
    ]

Or `config :mutalisk, ...` in `config/test.exs`. CLI flags override both. Run
`MIX_ENV=test mix help mut` or `MIX_ENV=test mix mut --help` for every key and flag.

## Mutators

13 low-noise mutators run by default; 17 more are opt-in or explicit-only. See
[docs/MUTATORS.md](docs/MUTATORS.md) for the full catalogue and enablement
rules.

## Reporters

- `terminal` (default) — the summary printed to stdout.
- `stryker-json` (default) — `stryker.report.json` for the Stryker HTML viewer.
- `html` (opt-in) — a self-contained `stryker.report.html` listing survivors.
- `github-actions` (opt-in) — `::warning` annotations for inline PR comments.

Select with `--reporters a,b` or `config :mutalisk, reporters: [...]`. Config may
use atoms such as `:stryker_json`; CLI examples use the hyphenated names.

## Per-test timeout policy

Mutalisk runs ExUnit with a **10 000 ms per-test timeout** (not ExUnit's
60 000 ms default), passed as `mix test --timeout 10000`. Override with
`--test-timeout-ms` or `config :mutalisk, test_timeout_ms: N`.

Mutation-introduced bugs are usually infinite loops or unbounded recursion —
1–10 s is plenty of evidence to classify them, and a 60 s timeout per such
mutant dominates wall-clock. If a test legitimately needs longer under no
mutation, tag it (`@tag timeout: 60_000`); ExUnit per-test tags override the
global default.

### Whole-suite (host) deadline

The per-test timeout bounds one test; the host also bounds the **whole selected
suite** for a mutant and kills the test port when that budget elapses. Because
several individually valid slow tests can exceed a budget sized for one test —
and the resulting host timeout would be scored as a detection, inflating the
score — the host deadline is derived from the measured baseline run:

    max(test_timeout_ms, min(baseline_wall_ms * 2, 600 000)) + 10 000 ms

Set `--suite-timeout-ms N` (or `config :mutalisk, suite_timeout_ms: N`, range
1 000..3 600 000) to pin it explicitly. The host uses
`max(test_timeout_ms, suite_timeout_ms) + 10 000 ms`, keeping the per-test
timeout as a floor and adding time for ExUnit to report its own timeout first.
Each run prints the deadline it derived and why.

### Sandbox `priv/` reset

Every mutant runs in a sandbox whose `priv/` tree is restored between mutants,
so a database, generated asset, directory or symlink a test writes there never
leaks into the next mutant. Baseline `priv/` files are compared by size + mtime
(`priv_fingerprint: :stat`, the default) — cheap, but blind to a rewrite that
keeps the byte size **and** lands in the same mtime second. Set
`--priv-fingerprint hash` (or `config :mutalisk, priv_fingerprint: :hash`) for a
content-exact comparison; it costs one content read per `priv/` file per reset,
per worker.

## Test selection

Test selection decides which test files run for each mutant. It defaults to
`coverage_with_static_fallback`; `--selection static` is the fully-portable
escape hatch. `--selection coverage` prevents a whole-run downgrade to static
selection when coverage collection becomes too slow; the per-mutant fallback
ladder below still applies. Selection only ever affects *which* tests run — it
never enters a mutant's stable id — and every rule below errs towards running more tests, because
under-selecting a killing test would report a false survivor.

In the coverage modes each mutant takes the first non-empty scope from:

1. **exact line** — tests whose coverage includes the mutated line.
2. **enclosing function** — tests covering any line of the mutated function.
3. **static fallback** — tests statically referencing the mutant's module.
4. **all tests**.

Two rules deliberately widen that ladder:

- **Clause-head mutations skip exact-line coverage.** A mutation in a pattern
  or a guard can make its clause accept inputs that originally routed to a
  *different* clause, so the killing test need never have executed the mutated
  line. Those mutants start at enclosing-function scope (every clause of the
  function), then try enclosing-file coverage before static selection.
- **Degraded coverage is unknown coverage.** Any test file whose coverage
  collection timed out or failed is added to *every* mutant's selection. An
  absent measurement is not evidence that the test is irrelevant.

Static selection is **conservative and may select more tests than strictly
necessary**. It indexes the modules each test file names, and then follows a
module reference graph built over project source files, including custom source
roots, umbrella layouts, and test support (build and dependency trees are
pruned): a test is selected for a target module when it references that
module *or any module that transitively references it*. This is what makes a
test that reaches the target only through a facade module still run. When the
closure spans the whole project, selecting every test is the correct answer.
The graph is built from statically visible aliases, so runtime-computed module
names are still handled by the separate dynamic-dispatch escape hatch (a test
file using `apply/3` on a variable module, or `Module.concat/1`, is selected for
every mutant).

## Limitations

- Mutalisk does not mutate DSL-emitted code, macro bodies, or generated code.
- Static test selection over-approximates; see [Test selection](#test-selection).
- Mutants run via a fresh `mix test` worker each (one mutant per VM), in an
  isolated sandbox subprocess.

## Public API

The supported interfaces are `MIX_ENV=test mix mut`, its documented configuration
and reports, and the runtime helpers in `Mutalisk`: `active_key/0`,
`set_active/1`, `get_active/0`, and `clear/0`. The active mutant is VM-global;
use `try`/`after` to clear it when using these helpers directly. Modules under
`Mut.*` and the compiler integration are implementation details.

## Maintaining Mutalisk

The source repository includes the [release guide](https://github.com/lukaszsamson/mutalisk/blob/main/docs/RELEASE.md),
[architecture overview](https://github.com/lukaszsamson/mutalisk/blob/main/docs/ARCHITECTURE.md),
and [benchmark tools](https://github.com/lukaszsamson/mutalisk/blob/main/bench/README.md).