Development and quality checks

This document describes contributor workflow and repository quality gates. These practices verify the implementation but are not implementation architecture decisions.

Default checks

Use Ruby 3.4 or newer, install the complete development toolchain once, and run the unit, integration, documentation, and style suite:

bundle install
bundle exec rake

Additional repository checks are:

bundle exec rake frontend:check
bundle exec rake grammar:test
bundle exec rake quality:error_ux
bundle exec rake quality:error_ux_round2
bundle exec rake quality:workloads
bundle exec rake quality:direct_ielr_decision
bundle exec rake quality:maturity
bundle exec rake quality:runtime_abi
npm ci
npm run test:site
actionlint
zizmor .

CI runs the supported Ruby matrix separately from optional experimental Ruby implementations. Workflow and dependency-update configuration are authoritative for the current matrix, schedules, permissions, and pinned action versions. The matrix uses the intentionally unlocked gemfiles/compat.Gemfile so each supported Ruby resolves compatible test dependencies. Contributors use the root Gemfile and its committed lockfile for every development tool.

Feature development may proceed while a release decision is HOLD. Before merging the first user-visible change for a feature, check the active-track limits in stability.md and update that inventory in the same pull request.

For the active declarative parser-construction track, change the canonical frontend grammar and shadow grammar together, run bundle exec rake frontend:generate, and review the generated parser diff. Then run frontend:check, the parser-configuration frontend/LSP/IR/CLI tests, test:no_exec, and test:zero_cost. A source parser declaration must promote normalization to the current Grammar IR without changing parser-table format v6; declaration-free inputs must retain their existing current-format golden bytes.

Stage A safety net

The pre-v1 safety suite keeps independent property tests and adds reviewable cross-feature orchestration:

bundle exec rake test:matrix
bundle exec rake test:matrix:full
bundle exec rake test:zero_cost
bundle exec rake test:reproducible
bundle exec rake test:compat
bundle exec rake test:ir_schema
bundle exec rake test:no_exec
bundle exec rake test:adversarial
bundle exec rake gallery:build
bundle exec rake gallery:conflicts
bundle exec rake fuzz:short
bundle exec rake fuzz:direct
bundle exec rake fuzz:injection
bundle exec rake verify
bundle exec rake verify:injection
bundle exec rake verify:strict
bundle exec rake equiv:regression
bundle exec rake analysis:test
bundle exec rake fix:test
bundle exec rake i18n:coverage
bundle exec rake deps:zero
bundle exec rake network:zero

test/matrix.yml declares the algorithm, table, CST, semantic-location, and entry axes. Pull requests run twelve representative combinations; the scheduled workflow runs all 96. test/golden/generated/ contains reviewable feature-off generated bytes and test/golden/digests.yml detects an unreviewed change. Use bundle exec rake golden:update only when the generated source change is intentional and review the normal version-control diff.

test:no_exec is a representative static-path regression for sentence generation and internal differential fuzzing. Its fixture contains parser actions, generated lexer actions, and all user-code sections and fails if those commands execute any of them. Command-level tests retain responsibility for the other static entries in the execution trust matrix. test/runtime/cst_incremental_test.rb covers the generated syntax-only side with independent sentinels: lexer actions execute while parser production actions do not. These guarantees do not make a generated parser a sandbox; test generated files only when their embedded Ruby is trusted.

The committed gallery/ grammars are self-authored and separately licensed. Their valid/invalid corpora run through all algorithms and table formats, and their state/conflict metrics are fixed. The short fuzzer derives 100 fixed-seed sentences per grammar and applies one-token insert/delete/replace mutations; scheduled CI raises this to 100,000. All searches have token, depth, expansion, action, stack, or subprocess-trial bounds. Timing and memory remain observations, not pass/fail thresholds.

quality:workloads validates the public workload registry, including exact source and license-evidence identities, measured versus not-measured states, benchmark eligibility, and cross-manifest consistency. It recomputes repository-owned grammar counts without downloading external source. Synthetic gallery evidence remains distinct from public and production workloads.

quality:error_ux_round2 regenerates the fixed H003 multi-shape diagnostic and repair capture in memory, validates its closed schema and required case coverage, and compares it byte-for-byte with the committed evidence. It also keeps the completed repository capture separate from the still-HOLD external subjective gate, validates the digest-bound independent-review registry and reviewer/disagreement inventory, and binds the unchanged R001 normative snapshot.

quality:maturity validates the exact 18 Preview and two Experimental records in the maturity audit, including activation, evidence gaps, external-use classification, issue-audit freshness, decisions, source digests, feature budgets, release blockers, and the synchronized public summary. This is an audit-integrity gate, not a promise that every Preview implementation or historical evidence capture remains runtime-compatible before v1.0.

quality:runtime_abi is contract-only: it cross-checks the published runtime ABI evolution policy and test-interaction policy against implementation constants, schemas, the 96-case matrix, scheduled gates, and golden inputs, and ignores ambient GitHub event variables. The minimal runtime-abi-assessment CI job runs only checkout, Ruby setup, and the quality:runtime_abi_pr entry, which explicitly receives the pull-request event and requires the template's structured assessment when a declared runtime-facing path changes. Free-form prose is not parsed as an assessment; reviewers still decide whether the selected boundary and evidence are correct.

When ibex fuzz finds a differential failure it automatically performs trial-bounded reduction and atomically saves a versioned fixture under test/fuzz/regressions/. Reproduce the fixed-seed fixture, add a focused test, and only then commit it; incomplete reduction is explicitly recorded and must not be called minimal. External comparisons require an explicit --against-runtime description so their runtime configuration is not omitted from the report or saved fixture. Per-sentence time and output limits prevent a stuck external target from hanging the fuzz run, and child process groups are cleaned up so a checker cannot leave descendants behind. --against=COMMAND is nevertheless an explicit unsafe opt-in: it executes that program with the developer's host permissions and is not a sandbox.

rake verify covers every gallery grammar, construction algorithm, and plain or compact table representation with the default independent checks. verify:injection requires all twenty structurally valid Automaton IR mutations to be detected. Scheduled and manually dispatched CI additionally runs verify:strict, whose completeness checks have higher construction cost. equiv:regression fixes ten equivalent and ten non-equivalent grammar pairs, concrete shortest witnesses, tree mappings, and exit status 2 for exhausted product-state budgets. analysis:test validates the closed JSON contracts and added/removed/changed classification for diff, plus deterministic recursion and table counts for metrics. fix:test proves that emitted repairs eliminate their target without adding another conflict, pass the independent verifier, and find no language or mapped-tree difference within their reported bounds. It also replays the committed twenty-conflict measurement. The fuzz and reducer CLIs default to their versioned JSON reports and also exercise --format=text so automation and terminal use share the same result and exit-status contracts. Reducer subprocesses additionally have explicit time, output, input, and trial budgets; budget exhaustion is not accepted as a reproduced failure. reduce --command=COMMAND executes arbitrary checker code; those budgets do not sandbox its filesystem, network, process, or other side effects. i18n:coverage requires exact message-ID and interpolation parity across every built-in language catalog, and exercises option, environment, fallback, text, and JSON paths.

Release basis

Before selecting a release revision, run:

bundle exec rake release:reproducible

The gate normalizes the RBS declarations named in tool/quality/stable-api-v1.yml and requires zero differences from the v0.2.0 baseline. Comments and blank lines do not affect the fingerprint. The lock covers the Stable runtime, batch CST, current IR, and table surfaces; Preview- and Experimental-only signature files are not promoted by appearing elsewhere in the generated RBS tree.

The same task builds ibex-runtime and ibex twice in separate temporary directories with different locale, timezone, and frozen-string settings. It requires byte-identical .gem files, rejects unsafe or development-only package paths, and prints the SHA-256 digest for each artifact. It never publishes a package and leaves no release artifact in the repository.

Artifact signing is deliberately not simulated with a repository-local key. The release remains blocked until the outcome gates in release-readiness.md pass. When that happens, the release environment must provide a protected external signing identity and bind its signature or provenance attestation to the printed artifact digest.

Browser site and API documentation

Install the locked browser dependencies and build the same self-hosted site bundle exercised in CI:

npm ci
npm run test:site
bundle exec yard doc

tool/build_site.rb, the YARD configuration, and .github/workflows/pages.yml are authoritative for documentation inputs, build versions, artifact retention, deployment permissions, and the protected Pages environment. Publication runs from main; pull requests validate the site without receiving deployment credentials.

Changing those operational settings does not require an implementation ADR. A change to the shipped browser analyzer's worker isolation, execution limits, or Ruby/JavaScript data boundary does.

Self-hosted frontend

lib/ibex/frontend/grammar.y is the production grammar. Regenerate its committed parser after changing the frontend language:

bundle exec rake frontend:generate
bundle exec rake frontend:check
bundle exec ruby -Itest test/frontend/self_host_test.rb

lib/ibex/frontend/shadow_grammar.y independently describes the same language using parameterized and inline rules. Tests build it through the bootstrap frontend and compare its location-preserving AST with the production parser. The shadow source must change with the canonical grammar, but it never replaces the committed production parser.

RBS and Steep

The development bundle includes the type toolchain. Regenerate the committed RBS tree, validate it, type-check the library, and refresh the documented statistics with:

ruby -e '
  sources = Dir.glob("lib/**/*.rb").sort
  exec("bundle", "exec", "rbs-inline", "--opt-out", "--base=lib", "--output=sig", *sources)
'
bundle exec rbs -r digest -r fileutils -r json -r optparse -r tempfile -r timeout -r tmpdir -r uri -I sig validate
bundle exec steep check
bundle exec ruby tool/type_stats.rb --write

CI generates signatures into a clean temporary directory and compares the complete tree, so both missing and stale signature files fail.

Bounded mutation testing

Mutation analysis is part of the development bundle:

bundle exec rake quality:mutation

The job uses MRI 4.0, two workers, a ten-minute job budget, and the focused Ibex::Tables::Compact#initialize matcher. TablesTest owns the mutation coverage declaration. The mutation environment declares open-source usage and applies a one-second timeout to each deterministic mutation. Expanding the subject requires an explicit test owner, no surviving non-equivalent mutations, and a measured duration within the same bounded job.

The supported runtime matrix uses gemfiles/compat.Gemfile, so the mutation tool's Ruby floor does not change the library's Ruby floor.

Benchmarks and evidence

Performance observations are not ordinary CI timing thresholds. Follow the benchmark guide for workload identities, formal comparison commands, environment matching, artifact validation, and append-only result history. Use stable workload and problem IDs from the workload registry when proposing or prioritizing improvements.

The development bundle includes the exact-version external-grammar profiler:

bundle exec ruby benchmark/public_profile.rb --help

Its output is diagnostic-only and must remain outside benchmark/results/. Use it to locate costs, then repeat the uninstrumented public comparison to produce reviewable performance evidence.

The versioned error-experience snapshot and its review procedure live in docs/error-ux.md.