Cucumber for Elixir runs 43 of 49 samples from the Cucumber Compatibility Kit
(CCK) v31.0.0 at commit 0a3cc60bf3a783bfcbdeb8b754ce46860e0388e5. All vendored feature
files, reference streams, and the attachment PDF match that revision byte for
byte. These approvals establish compatibility for the covered behavior subject
to the allowances below; they do not establish complete Cucumber, Gherkin,
Cucumber Expressions, or formatter compatibility.
Messages contract
The emitter declares Messages 34.2.1. Its unmodified authoritative JSON
Schema bundle is vendored at test/fixtures/messages/messages.schema.json, from
cucumber/messages commit e5769e8
(tag v34.2.1). Every raw envelope in all supported samples is validated
against 34.2.1, including metadata and fields later removed for comparison.
Cucumber.CckSchema is a test-only, dependency-free evaluator of the vocabulary
used by this pinned bundle: references (including document-local definitions),
objects, arrays, required/unknown properties, scalar types, enums, numeric bounds,
and minimum array lengths. Unknown validation keywords fail closed. It is not a
general JSON Schema engine. Schema validation proves the emitted shape; optional
fields and unimplemented message types are not promises of implemented features.
Cucumber.CckStream additionally checks one message per envelope, metadata first
and exactly once, globally unique IDs, references to earlier envelopes of the
right message type, one completed run, paired run hooks, and complete ordered
steps within each active attempt. Attachments must reference that attempt's active
step, or — for attachments from before_all/after_all hooks — a run hook that
has started and not yet finished. Attempt state is independent so concurrent
scenarios may interleave. Both actual and reference streams pass these checks
before approval normalization. The Messages emission behavior tests also use
these checks, including abrupt process termination and hook failures. Mutation
tests exercise rejection paths and acceptance of interleaved attempts.
This validator targets complete execution streams for the supported subset. It does not validate arbitrary partial streams, every semantic relationship in the Messages model, or compatibility with every consumer.
Sample matrix
| Coverage | Samples |
|---|---|
| Approved: basic execution and arguments | minimal, cdata, empty, data-tables, data-tables-with-expression, backgrounds, doc-strings, doc-strings-with-expression, examples-tables, unused-steps, stack-traces, parameter-types, regular-expression |
| Approved: outcomes | undefined, ambiguous, pending, skipped, all-statuses, failedish-combinations, undefined-multiple, examples-tables-undefined, examples-tables-undefined-multiple |
| Approved: hooks | hooks, hooks-conditional, hooks-named, hooks-skipped, skipped-failing-hook, global-hooks, global-hooks-afterall-error, global-hooks-attachments, hooks-undefined |
| Approved: retries | retry, retry-pending, retry-undefined |
| Approved: structure and attachments | rules, rules-backgrounds, attachments, hooks-attachment, examples-tables-attachment, multiple-features |
| Approved with additional behavior/representation allowances | global-hooks-beforeall-error, retry-ambiguous, markdown (details below) |
| Excluded: a step with both a data table and a doc string (Messages 34) is not supported | data-tables-doc-strings |
| Excluded: reference CLI reverse-order option absent; ExUnit controls order | multiple-features-reversed |
| Excluded: pending/skipped use return values here, not exceptions; return-value samples are approved | pending-exception, skipped-exception |
| Excluded: reference CLI runner-crash injection absent | test-run-exception |
Excluded: discovery raises Cucumber.UndefinedParameterTypeError before a run, rather than emitting undefinedParameterType | unknown-parameter-type |
An inventory test requires every sample in the pinned upstream tree to be either approved or explicitly excluded. Broader Gherkin parser/dialect and Cucumber Expressions fixture adoption remains follow-up work, with separate fixture pins and explicit support matrices needed for each.
Comparison allowances
All approvals use these rules, implemented in Cucumber.CckApproval:
| Category | Allowance and boundary |
|---|---|
| Nondeterministic | Timestamps and durations are removed after raw type/range validation. IDs are consistently renumbered after raw uniqueness/reference checks. |
| Host/language specific | meta is omitted from equality, but validated and required to declare 34.2.1. sourceReference is removed because Elixir definitions differ from TypeScript. Error message and exception fields are removed because wording and stack traces differ. |
| Path representation | uri becomes its basename, ignoring fixture root paths. This does not prove disambiguation for duplicate basenames. |
| Parser representation | Every column is removed (our parser tracks lines). Descriptions have whitespace trimmed per line and at their boundaries. Exact columns and indentation are not approved. |
| Unsupported optional output | stepMatchArgumentsLists is removed: matching produces converted values, not source offsets. suggestion envelopes are removed: undefined-step suggestions are reported through errors rather than these envelopes. Parameter-type and hook source lines are not recorded; raw optional source references still undergo schema validation when present. |
| Valid emission order | Step definitions, hooks, and parameter types are sorted by content. Test cases are hoisted after testRunStarted for comparison; each raw case must already precede its attempt. |
Three sample-specific allowances remain:
| Sample | Additional allowance |
|---|---|
global-hooks-beforeall-error | The reference aborts before creating cases. ExUnit still executes and fails scenarios with Cucumber.BeforeAllError. Case and step lifecycle envelopes are removed only from equality; our entire emitted lifecycle must validate first. This is a behavior difference. |
retry-ambiguous | The reference registers identical patterns twice; discovery here rejects exact duplicates. Two distinct overlapping patterns reproduce ambiguity, and step-definition patterns are excluded from equality. |
markdown | Only the feature description is excluded. The reference has tokenizer-recovery text | boz | boo |; our Markdown parser emits no Markdown descriptions. |
Reproduce and update
Offline checks (no GitHub access or extra validator dependency):
mix test test/cucumber/cck_approval_test.exs test/cucumber/cck_validation_test.exs test/cucumber/cck_fixtures_test.exs test/cucumber/behavior/messages_emission_test.exs
Verify the immutable fixture provenance online with GitHub CLI authentication:
MIX_ENV=test mix run scripts/check_cck.exs
Detect changed reference files or added/removed upstream samples:
MIX_ENV=test mix run scripts/check_cck.exs main
test/fixtures/cck/upstream.json records the upstream revision, complete sample
inventory, and upstream Git blob hashes for every vendored file. Normal tests
verify file contents and inventory against the manifest; the online check verifies
those hashes and the complete sample inventory against GitHub's tree API. It fails
on truncated trees or changed/new samples. Upstream files not vendored here
(including TypeScript definitions) are not byte-checked; review those when updating.
To update deliberately:
- Resolve the desired CCK revision to a full commit SHA with
gh api repos/cucumber/compatibility-kit/commits/REV --jq .shaand run the check against it. - Fetch its recursive Git tree using
gh api 'repos/cucumber/compatibility-kit/git/trees/SHA?recursive=1'. Review sample additions/removals, reference step definitions, and protocol changes. - Replace each tracked fixture from
devkit/samples/PATHat that SHA usinggh api 'repos/cucumber/compatibility-kit/contents/devkit/samples/PATH?ref=SHA' -H 'Accept: application/vnd.github.raw+json'. Add fixtures and Elixir definitions for newly supported samples, or explicitly classify exclusions. - Update
upstream.jsonwith that SHA, the sorted complete upstream sample list, and the Git tree's blob hashes for the selected paths. Do not bless locally modified reference output by hashing it into the manifest. - If adopting a new Messages version, vendor its schema and license unchanged, record its immutable revision and SHA-256, and review the validator vocabulary and emitted shapes before changing metadata. Run raw validation and approvals, resolve differences, and update this matrix. Finish with the online pinned check.