Skip to content

Add a beginner guide to Beaker acceptance testing#368

Open
miharp wants to merge 2 commits into
OpenVoxProject:masterfrom
miharp:docs/beaker-acceptance-guide
Open

Add a beginner guide to Beaker acceptance testing#368
miharp wants to merge 2 commits into
OpenVoxProject:masterfrom
miharp:docs/beaker-acceptance-guide

Conversation

@miharp

@miharp miharp commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

What

Expands the DevKit "Acceptance Testing with Beaker" page from an 8-line stub (which only linked to voxpupuli-acceptance) into a full beginner-friendly guide.

Closes #304.

Contents

  • Prerequisites — a hypervisor (Docker by default), a current Ruby (via a version manager such as rbenv/rvm/mise), and the system_tests Bundler group, plus a macOS/Apple Silicon local-run note. The common newcomer traps are called out explicitly.
  • Setting up the framework — the three pieces that wire voxpupuli-acceptance into a module (the system_tests Gemfile group, spec_helper_acceptance.rb, and the Rakefile).
  • Anatomy of a test — the apply_manifest two-run idempotency idiom (with idempotency defined plainly), a worked chrony example that mirrors the real puppet-chrony acceptance test, and the shared-example shortcuts.
  • Choosing a platform (setfiles), running the suite, a BEAKER_* environment-variable reference, debugging a failure, running in CI (including a pointer to the shared gha-puppet reusable workflow that most Vox Pupuli modules use, with a note on the modulesync-managed ci.yml), and customizing nodes.
  • Beginner-facing terms (catalog, gem, hypervisor, rake, fixtures, apply_manifest, setfile, the package resource type) are linked or glossed on first use.

Validation

Every command in the guide was validated end to end:

  1. In CI — wired voxpupuli-acceptance into a real module and ran rake beaker in GitHub Actions (Docker, almalinux9-64, openvox8), green.
  2. Locally — ran the full flow against the upstream voxpupuli/puppet-chrony module on macOS + Docker: bundle install, puppet-metadata setfiles, and bundle exec rake beaker.

The worked example was run as written against real OpenVox containers on both almalinux9-64 and debian12-64 → 4 examples, 0 failures each. Local testing also pinned down two adjustments the worked example needs: chrony won't apply cleanly inside a Docker container without options => '-F 0 -x' (which disable clock-stepping and the seccomp filter), and the service is named chronyd on RHEL-family vs chrony on Debian-family, so the example selects it from os.family.

That validation is also where the guide's details were pinned down: the openvox8 collection name (vs. upstream puppet8), the puppet-metadata setfiles command (the older setfiles is now a deprecated alias), and the macOS/Apple Silicon and Ruby-version prerequisites.

Notes

The page lives in _ecosystem_8x and is surfaced via the existing _latest symlink. The nav entry is renamed from "Acceptance Testing" to "Acceptance Testing with Beaker" to match the page title.

@miharp miharp force-pushed the docs/beaker-acceptance-guide branch from f360527 to 81f3977 Compare June 25, 2026 17:13
@miharp miharp changed the title Expand the Beaker acceptance testing guide Add a beginner guide to Beaker acceptance testing Jun 25, 2026
@miharp miharp marked this pull request as ready for review June 25, 2026 17:21
@miharp miharp requested a review from a team as a code owner June 25, 2026 17:21
@miharp miharp force-pushed the docs/beaker-acceptance-guide branch 5 times, most recently from e2d53f1 to ed253c5 Compare June 26, 2026 13:10
@miharp miharp marked this pull request as draft June 26, 2026 13:21
@miharp miharp marked this pull request as ready for review June 26, 2026 13:32
@miharp miharp force-pushed the docs/beaker-acceptance-guide branch 4 times, most recently from 5322444 to c76cb92 Compare June 29, 2026 17:02
miharp and others added 2 commits June 30, 2026 06:03
Expand the DevKit "Acceptance Testing with Beaker" page from an 8-line
stub into a full beginner-friendly guide. It covers:

- prerequisites: a hypervisor, a current Ruby (via rbenv/rvm/mise), and the
  system_tests Bundler group, plus a macOS/Apple Silicon local-run note
- wiring voxpupuli-acceptance into a module (Gemfile, spec_helper, Rakefile)
- anatomy of a test: the apply_manifest two-run idempotency idiom with a
  worked chrony example that mirrors the real puppet-chrony test
- choosing a platform (setfiles), running the suite, the common BEAKER_*
  variables, debugging a failure, running in CI, and customizing nodes

Beginner-facing terms (catalog, gem, hypervisor, rake, fixtures,
apply_manifest, setfile, the package resource type) are linked or glossed
on first use.

The commands and the openvox8 collection name were validated end to end by
running the suite against real OpenVox containers (almalinux9-64 and
debian12-64, 4 examples / 0 failures) and in GitHub Actions CI.

Closes OpenVoxProject#304

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Michael Harp <mike@mikeharp.com>
The acceptance testing guide was reachable from the nav but not
mentioned on the DevKit components overview. Add a Beaker bullet
alongside the other testing tools, linking to the guide.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Michael Harp <mike@mikeharp.com>

@binford2k binford2k left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine the way it is, but I do have a few suggestions to make the whole set feel more cohesive.

If Docker reports an image platform mismatch, export `DOCKER_DEFAULT_PLATFORM=linux/amd64`, pull the base image once with that platform selected, and retry.

Second, you need a current Ruby.
The Ruby that ships with macOS and some Linux distributions is too old, and using it makes `bundle install` fail with a confusing dependency error rather than a clear "your Ruby is too old" message.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe worth calling out that macOS has deprecated system ruby, even though they've not actually removed it yet.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually, this comment makes me realize that the setup page should talk more about ruby versions. Filed #382

Beaker defaults to Docker, which is the easiest way to get started, so install it and make sure your user can run containers.
[Podman](https://podman.io/), [Vagrant](https://www.vagrantup.com/), and [libvirt](https://libvirt.org/) are also supported if you prefer them.

If you're running locally on macOS, two extra wrinkles come up. Neither affects Linux CI runners, which already provide the standard socket and architecture.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we maybe put these into a <details> disclosure widget or something? It's not super clear on first read which parts are the "two extra wrinkles".

Either that, or move it to the end of the prereqs section.

Install a 3.x Ruby, then run `ruby --version` in the module directory and confirm it reports 3.x before you go further; the [DevKit setup guide](setup.html) covers this in more detail.

Finally, you need the acceptance gems installed.
A module's `Gemfile` sorts its gems into named groups so that, for example, continuous integration can install only what each job needs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the with/without discussion should be moved to setup.md and this section should only say something like

You'll need the system_tests bundler group, but you can omit development and release if you want to keep things lean.

filed #383

That second run is the heart of acceptance testing, because it proves _idempotency_.
A manifest is idempotent when applying it more than once is safe: the first run brings the system to the desired state, and every run after that makes no changes because the system already matches.
Idempotency is a core property of well-written Puppet code, and a manifest that keeps making changes on every run is a bug.
Catching that here is one of the most valuable things an acceptance test does.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth mentioning the modules which require two runs to reach idempotency? (the ones that install something used to generate a fact that is then used on the next run to specify state)

Or is that a "by the time you have to do this, you'll know enough to figure it out" thing?

@binford2k binford2k requested a review from bastelfreak June 30, 2026 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature request]: Documentation on beaker and acceptance testing

2 participants