feat: add automated available modules table generation in README.md and related tests

.github/instructions/pre-commit-hooks.instructions.md

@@ -20,6 +20,7 @@ pre-commit install --hook-type commit-msg
 | Terraform format mismatch | `terraform fmt -recursive infrastructure/modules/` |
 | Documentation out of sync | `pre-commit run terraform_docs --all-files` |
 | Dependabot config out of sync | Commit the regenerated `.github/dependabot.yaml` (auto-generated) |
+| Available modules table out of sync | Commit the regenerated `README.md` Available modules section (auto-generated). Includes all modules: regular modules alphabetically, then legacy modules (older format, in `_legacy/`) with `[LEGACY]` markers at the end. |
 | Shell script errors | Review output; fix syntax errors; re-run `pre-commit run shellcheck` |
 | English/spelling mistakes | Check `.vale.ini` rules; update text if needed |
 | Trailing whitespace/EOL | `pre-commit run --all-files` (auto-fixed) |
@@ -46,6 +47,7 @@ git commit -m "type(scope): description"
 - `scan-secrets-whole-history` — scans entire git history for secrets (runs on `pre-commit run --all-files`)
 - `terraform_validate` — ensures modules are syntactically valid
 - `regenerate-dependabot-config` — ensures Dependabot watches all modules
+- `check-available-modules` — ensures README module table is up-to-date
 - `no-commit-to-branch` — enforces PR workflow
 
 ## Tool Invocation in Scripts

.github/workflows/stage-1-pre-commit.yml

@@ -42,6 +42,9 @@ jobs:
             hooks: >-
               generate-terraform-providers terraform_providers_lock
               terraform_validate terraform_docs
+          - name: "configuration-generation"
+            hooks: >
+              regenerate-dependabot-config check-available-modules
     steps:
       - name: "Checkout code"
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

.pre-commit-config.yaml

@@ -56,6 +56,27 @@ repos:
         files: infrastructure/modules/.*/versions\.tf$
         pass_filenames: false
 
+  # --------------------------------------------------------------------------
+  # Local: regenerate Available modules section in README.md
+  # --------------------------------------------------------------------------
+  # Automatically regenerates the Available modules table in README.md whenever
+  # modules are added/removed or module READMEs are updated. This ensures the
+  # documentation stays in sync with the actual modules available.
+  # Fails if the generated section differs from the committed version,
+  # forcing users to review and commit the updated documentation.
+  #
+  # Generates:
+  #   - README.md (Available modules table, between markers)
+  - repo: local
+    hooks:
+      - id: check-available-modules
+        name: check-available-modules
+        require_serial: true
+        entry: ./scripts/githooks/check-available-modules.sh
+        language: script
+        files: (infrastructure/modules/.*/README\.md|README\.md)$
+        pass_filenames: false
+
   # --------------------------------------------------------------------------
   # Terraform hooks (format, lint, validate, docs, lock)
   # --------------------------------------------------------------------------

README.md

@@ -197,7 +197,10 @@ screening-terraform-modules-aws/
 │       ├── iam/           # Exemplar: iam policies & roles
 │       ├── secrets-manager/
 │       ├── kms/
-│       └── ...            # Additional modules
+│       ├── ...            # Additional modules
+│       └── _legacy/       # Older-format modules (pre-restructure, screening-specific variants)
+│           ├── old-module-1/
+│           └── old-module-2/
 ├── scripts/               # Helper scripts (linting, hooks, Docker)
 ├── docs/                  # ADRs, developer guides, diagrams
 ├── .pre-commit-config.yaml # Pre-commit hook definitions
@@ -295,6 +298,7 @@ Rules:
 
 ## Available modules
 
+<!-- BEGIN_AVAILABLE_MODULES -->
 | Module | Wraps | Description |
 | --- | --- | --- |
 | `acm` | terraform-aws-modules/acm/aws | AWS Certificate Manager (ACM) certificate management |
@@ -306,7 +310,6 @@ Rules:
 | `cw-firehose-splunk` | — | CloudWatch logs to Splunk via Firehose |
 | `ecr` | — | ECR repository with security controls |
 | `ecs-cluster` | — | ECS Fargate cluster |
-| `ecs-service` | — | ECS service and task definition |
 | `elasticache` | — | ElastiCache cluster (Redis/Memcached) |
 | `github-config` | — | GitHub OIDC provider and runner configuration |
 | `guardduty` | — | GuardDuty threat detection |
@@ -316,11 +319,9 @@ Rules:
 | `lambda` | terraform-aws-modules/lambda/aws | Lambda function with runtime and layers |
 | `lambda-layer` | — | Lambda layer for function libraries |
 | `license-manager` | — | License Manager configuration |
-| `network-firewall` | — | Network Firewall rules and policies |
 | `parameter_store` | — | SSM Parameter Store configuration |
-| `r53` | — | Route 53 DNS records (legacy) |
+| `r53` | terraform-aws-modules/route53/aws | Route 53 DNS Zones, Records, Resolver and Resolver Firewall |
 | `r53-healthcheck` | — | Route 53 health checks |
-| `rds` | — | RDS database instance (legacy) |
 | `rds-database` | — | RDS database (logical) |
 | `rds-gateway-ecs-task` | — | RDS gateway ECS task definition |
 | `rds-instance` | — | RDS instance |
@@ -332,11 +333,13 @@ Rules:
 | `sns` | terraform-aws-modules/sns/aws | SNS topic with encryption and policies |
 | `sqs` | — | SQS queue with encryption |
 | `tags` | — | Foundation: naming and tagging context module |
-| `vpc` | — | VPC with subnets, routing, and gateways |
+| `vpc` | terraform-aws-modules/vpc/aws | VPC with subnets, routing, and gateways |
 | `vpce` | — | VPC endpoint (single service) |
 | `vpces` | — | VPC endpoints (multiple services) |
 | `waf` | — | WAF web ACL with rules |
 
+<!-- END_AVAILABLE_MODULES -->
+
 ## Pre-commit hooks
 
 This repository uses [pre-commit](https://pre-commit.com/) to run quality checks before code is committed locally, and in CI via the `stage-1-pre-commit.yml` GitHub Actions workflow.

docs/user-guides/Pre_commit_hooks_reference.md

@@ -334,6 +334,182 @@ yq eval '.' .github/dependabot.yaml
 
 ---
 
+#### `check-available-modules` — Update Available Modules Table in README
+
+**What it does:** Automatically regenerates the "Available modules" section in `README.md` by discovering all Terraform modules and reading their metadata from `scripts/config/generate-available-modules.yaml`. This ensures the module table stays in sync with actual modules in the codebase and their descriptions.
+
+**When it fails:**
+
+```text
+✗ Pre-commit check FAILED - README was regenerated
+
+Please review the updated Available modules section and commit it:
+  git add README.md
+  git commit -m 'docs: update Available modules section'
+```
+
+**What triggers it:**
+
+- Adding or removing a module (identified by presence of `main.tf` or `versions.tf`)
+- Changing module descriptions in `scripts/config/generate-available-modules.yaml`
+- Any changes to files that cause README to be regenerated
+
+**How it discovers modules:**
+
+The script scans `infrastructure/modules/` (including `infrastructure/modules/_legacy/`) for actual Terraform module directories by looking for:
+
+- `main.tf` files, OR
+- `versions.tf` files
+
+It automatically **excludes `.terraform/` directories** (cached provider plugins), so only real modules are included.
+
+**Legacy module handling:**
+
+Modules in `infrastructure/modules/_legacy/` are older-format modules (from before the major restructuring and compliance enforcement) that may include screening programme-specific variants:
+
+- **Included** in the generated table (not skipped) to prevent breaking changes
+- **Marked with [LEGACY]** annotation in both the module name and description
+- **Placed at the end** of the table, after all regular modules, visually signaling deprecated status
+- **Alphabetically sorted** within the legacy section
+- **Not used for new infrastructure** — they lack security baseline and compliance controls
+
+This keeps legacy modules discoverable while clearly separating them from active modules:
+
+```text
+acm
+api-gateway
+...
+vpc
+[LEGACY]-old-module-1  (if any legacy modules exist)
+[LEGACY]-old-module-2
+```
+
+**Migrating away from legacy modules:**
+
+1. Identify or create the modern equivalent in `infrastructure/modules/`
+2. Update all stacks that use the legacy module to use the modern equivalent
+3. Once all consumers have migrated, move the legacy module from `infrastructure/modules/{name}/` to `infrastructure/modules/_legacy/{name}/` to mark it deprecated
+
+**Module metadata:**
+
+Modules are listed alphabetically (regular modules first, then legacy modules). For each module found:
+
+- **If metadata exists** in `scripts/config/generate-available-modules.yaml`:
+  - Use the curated description and wrapped module reference
+  - Example: `s3-bucket` → "S3 bucket with full security baseline" | "terraform-aws-modules/s3-bucket/aws"
+  - Example legacy: `[LEGACY]-old-module` → "Old description [LEGACY]" | "—"
+
+- **If no metadata entry exists**:
+  - Module is still included (prevents gaps when new modules are added)
+  - Shows `—` (dash) for both Wraps and Description columns
+  - Example: `new-module` → "—" | "—"
+  - Example legacy: `[LEGACY]-new-legacy-module` → "[LEGACY]" | "—"
+
+**Fix:**
+
+The hook regenerates the table automatically. Simply review and commit it:
+
+```bash
+# Review the changes
+git diff README.md
+
+# Commit the regenerated table
+git add README.md
+git commit -m "docs: update Available modules section"
+```
+
+**How it works:**
+
+1. **Scans `infrastructure/modules/` and `infrastructure/modules/_legacy/`** for directories containing `main.tf` or `versions.tf` (excluding `.terraform/`)
+
+2. **Assigns sort keys** to modules based on location:
+   - Regular modules: sort key `0`
+   - Legacy modules (in `_legacy/`): sort key `1`
+
+3. **Looks up metadata** in `scripts/config/generate-available-modules.yaml` for each module found
+   - If found: uses curated description and wrapped module reference
+   - If missing: uses dashes (`—`) to indicate "add metadata if you want"
+
+4. **Generates markdown table** sorted by:
+   - Primary: sort key (0 = regular modules first, 1 = legacy modules at end)
+   - Secondary: module name alphabetically
+   - Between markers: `<!-- BEGIN_AVAILABLE_MODULES -->` / `<!-- END_AVAILABLE_MODULES -->`
+
+5. **Annotates legacy modules** with `[LEGACY]` in both the module name and description
+
+6. **Replaces only the table**, preserving all other README content
+
+**Example table generated:**
+
+```markdown
+| Module | Wraps | Description |
+| --- | --- | --- |
+| `new-module` | — | — |
+| `s3-bucket` | terraform-aws-modules/s3-bucket/aws | S3 bucket with full security baseline |
+| `tags` | — | Foundation: naming and tagging context module |
+| `vpc` | — | VPC with subnets, routing, and gateways |
+```
+
+**Maintenance:**
+
+**Adding or updating module metadata:**
+
+1. Edit `scripts/config/generate-available-modules.yaml`
+2. Add or update the module entry with description and wraps info
+3. Run the hook (it will regenerate README.md automatically on next commit)
+
+If you've just added a new module and don't want to document it yet, no action is needed — it will appear in the table with dashes until you add metadata.
+
+**Archiving a module as legacy:**
+
+1. Move the module from `infrastructure/modules/{module-name}/` to `infrastructure/modules/_legacy/{module-name}/`
+2. Update its metadata in `scripts/config/generate-available-modules.yaml` if desired
+3. Commit both changes — the hook will automatically regenerate README.md with `[LEGACY]` annotations and correct placement at end of table
+4. The legacy module remains discoverable and documented but visually separated from active modules
+
+Example metadata entry:
+
+```yaml
+s3-bucket:
+  description: "S3 bucket with full security baseline"
+  wraps: "terraform-aws-modules/s3-bucket/aws"
+
+new-module:
+  description: "Description here"
+  wraps: "—"
+```
+
+**Troubleshooting:**
+
+If markers are missing from README.md, add them:
+
+```markdown
+## Available modules
+
+<!-- BEGIN_AVAILABLE_MODULES -->
+(table will be inserted here)
+<!-- END_AVAILABLE_MODULES -->
+```
+
+To regenerate manually:
+
+```bash
+bash scripts/generate-available-modules.sh
+```
+
+To add metadata for a module (converting dashes to real descriptions):
+
+```bash
+# Edit the metadata file
+vi scripts/config/generate-available-modules.yaml
+
+# Then commit - the hook will regenerate README.md
+git add scripts/config/generate-available-modules.yaml
+git commit -m "docs: add metadata for new-module"
+```
+
+---
+
 ### File Hygiene
 
 These hooks catch common Git mistakes and enforce best practices.

infrastructure/AGENTS.md

@@ -25,9 +25,22 @@ infrastructure/
     ├── rds-database/       # RDS database module
     ├── sqs/                # SQS queue module
     ├── waf/                # WAF module
-    └── ...                 # Additional modules
+    ├── ...                 # Additional modules
+    └── _legacy/            # Archive of deprecated/superseded modules
+        ├── old-module-1/   # Legacy module (kept for backwards compatibility)
+        └── old-module-2/   # Legacy module (kept for backwards compatibility)
 ```
 
+**Note on `_legacy/` modules:**
+
+- Legacy modules are older-format modules (from before the major restructuring and compliance enforcement) that may include older screening programme-specific variants
+- They are kept in the `_legacy/` directory for backwards compatibility with existing infrastructure
+- They remain discoverable and documented in README.md (marked with `[LEGACY]` annotation) to prevent accidental breaking changes
+- They appear at the end of the Available modules table, after all active modules, visually signaling their deprecated status
+- **Do not use legacy modules for new infrastructure** — they lack the security baseline and compliance controls enforced in modern modules
+- To migrate away from a legacy module: identify or create the modern equivalent in `infrastructure/modules/`, update your stack, then deprecate the legacy module
+- To mark a module as legacy (after all consumers have migrated): move it from `infrastructure/modules/{name}/` to `infrastructure/modules/_legacy/{name}/`
+
 ## The Wrapper Module Pattern
 
 This repository's modules are **thin, opinionated wrappers** around well-known community Terraform modules from the registry (e.g., `terraform-aws-modules/*`). The wrapper's job is to: