From 827da1d16996a29216169ddaa7517cbd749bd3b0 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 19 May 2026 22:32:57 +0000 Subject: [PATCH 1/8] docs: add air-gapped environment walkthroughs for GitHub Actions and Azure DevOps Add step-by-step guides under docs/walkthrough/ covering: - Tarball preparation with --cli-package flag - Lock file strategy for offline npm ci - Self-hosted runner/agent configuration - Workflow/pipeline modifications for offline operation - Upgrade procedures and troubleshooting Update docs/README.md navigation with new walkthrough section. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: EMaher <9244742+EMaher@users.noreply.github.com> --- .squad/agents/docwriter/history.md | 16 + docs/README.md | 4 + docs/walkthrough/air-gapped-azure-devops.md | 335 ++++++++++++++++++ docs/walkthrough/air-gapped-github-actions.md | 259 ++++++++++++++ 4 files changed, 614 insertions(+) create mode 100644 docs/walkthrough/air-gapped-azure-devops.md create mode 100644 docs/walkthrough/air-gapped-github-actions.md diff --git a/.squad/agents/docwriter/history.md b/.squad/agents/docwriter/history.md index b78bf5d..6712071 100644 --- a/.squad/agents/docwriter/history.md +++ b/.squad/agents/docwriter/history.md @@ -16,6 +16,22 @@ Documentation authoring (2026-04-30 to 2026-05-17): 3-phase plan with 28 user-fa +### 2026-06-11: Air-Gapped Walkthroughs — GitHub Actions and Azure DevOps + +**Context:** Authored two walkthrough documents for using apiops-cli in air-gapped (network-restricted) environments. + +**Files Created:** +1. `docs/walkthrough/air-gapped-github-actions.md` — 8-step guide covering tarball preparation, `apiops init --cli-package`, lock file generation, npm cache transfer, self-hosted runner setup, workflow modifications for `npm ci --offline`, and upgrade procedures. +2. `docs/walkthrough/air-gapped-azure-devops.md` — 9-step guide with same core pattern plus Azure Artifacts feed as alternative to manual cache transfer, `npmAuthenticate@0` task usage, agent pool configuration, and AzureCLI@2 service connection authentication. + +**Files Updated:** +- `docs/README.md` — Added walkthrough section to Quick Links table and directory tree. + +**Learnings:** +- Air-gapped setups rely on `--cli-package` flag for local tarball mode, which generates `package.json` with `"file:.apiops/{tarball}"` dependency. The lock file is critical — without it, `npm ci` fails. +- Azure DevOps has a natural advantage for air-gapped setups via Azure Artifacts feeds with upstream sources (controlled sync windows). GitHub Actions environments must rely on pre-populated npm cache or vendored node_modules. +- Authentication differs: GitHub Actions in air-gapped may lose OIDC (can't reach token.actions.githubusercontent.com), requiring fallback to service principal secrets. Azure DevOps service connections work regardless since the agent-to-DevOps connectivity handles token exchange. + ### 2026-05-17: Phase 2 Docs — Azure DevOps, Filtering, Artifact Format, Config Reference, Glossary **Context:** Authored five Phase 2 documentation files completing CI/CD coverage, resource filtering guide, artifact format reference, configuration reference, and APIM glossary. diff --git a/docs/README.md b/docs/README.md index cfc6d9d..6fe88ef 100644 --- a/docs/README.md +++ b/docs/README.md @@ -9,6 +9,7 @@ | [Getting Started](getting-started.md) | Install and run your first extract → publish cycle in 10 minutes | | [Command Reference](commands/) | Detailed docs for [extract](commands/extract.md), [publish](commands/publish.md), [init](commands/init.md) | | [CI/CD Integration](ci-cd/) | Set up [GitHub Actions](ci-cd/github-actions.md) or [Azure DevOps](ci-cd/azure-devops.md) pipelines | +| [Walkthroughs](walkthrough/) | Step-by-step guides: [Air-gapped GitHub Actions](walkthrough/air-gapped-github-actions.md), [Air-gapped Azure DevOps](walkthrough/air-gapped-azure-devops.md) | ## How It Works @@ -78,6 +79,9 @@ docs/ ├── architecture/ │ ├── overview.md — System design overview │ └── design-principles.md — Architecture principles +├── walkthrough/ +│ ├── air-gapped-github-actions.md — Air-gapped setup with GitHub Actions +│ └── air-gapped-azure-devops.md — Air-gapped setup with Azure DevOps └── troubleshooting/ ├── common-errors.md — Error messages and solutions ├── debugging-guide.md — Debugging with --log-level diff --git a/docs/walkthrough/air-gapped-azure-devops.md b/docs/walkthrough/air-gapped-azure-devops.md new file mode 100644 index 0000000..21670f2 --- /dev/null +++ b/docs/walkthrough/air-gapped-azure-devops.md @@ -0,0 +1,335 @@ +# Air-Gapped Setup: Azure DevOps + +Deploy APIM configuration using apiops-cli on Azure DevOps agents with **no internet access** at runtime. This walkthrough covers preparing the npm tarball, lock file, and self-hosted agent configuration so that `npm ci` succeeds entirely offline. + +--- + +## When to Use This Guide + +- Self-hosted agents in a private network with no outbound internet +- Environments requiring artifact pre-staging for security compliance +- Corporate networks that block access to the npm registry + +--- + +## Architecture Overview + +```mermaid +flowchart LR + subgraph Connected Workstation + A[npm pack] --> B[apiops tarball .tgz] + C[npm install] --> D[package-lock.json] + end + subgraph Transfer + B -->|Approved media / Azure Artifacts feed| E[Self-hosted Agent] + D -->|Same transfer| E + end + subgraph Air-Gapped Agent + E --> F[npm ci --offline] + F --> G[apiops extract / publish] + end +``` + +--- + +## Prerequisites + +| Requirement | Details | +|-------------|---------| +| **Connected workstation** | A machine with internet access to download packages | +| **Node.js 22.x** | Installed on both the workstation and the agent | +| **npm 10+** | Comes with Node.js 22 | +| **Self-hosted Azure Pipelines agent** | Registered in your agent pool, running in the air-gapped network | +| **Azure connectivity from agent** | The agent must reach your APIM instance's ARM endpoint (network-level, not npm) | +| **Transfer mechanism** | USB drive, Azure Artifacts upstream feed, or approved file share for moving packages into the restricted zone | + +--- + +## Step 1 — Prepare the Tarball (Connected Workstation) + +On a machine with internet access: + +```bash +# Install the CLI globally to get access to the apiops command +npm install -g @peterhauge/apiops-cli + +# Pack the installed package into a tarball +npm pack @peterhauge/apiops-cli +# Produces: peterhauge-apiops-cli-.tgz +``` + +Keep the `.tgz` file — you'll transfer it to the air-gapped environment. + +--- + +## Step 2 — Scaffold the Repository + +Run `apiops init` with the `--cli-package` flag pointing to the tarball: + +```bash +apiops init \ + --ci azure-devops \ + --cli-package ./peterhauge-apiops-cli-0.1.5-alpha.1.tgz \ + --environments dev,prod \ + --non-interactive +``` + +This generates: + +| File | Purpose | +|------|---------| +| `package.json` | References the tarball as `"file:.apiops/peterhauge-apiops-cli-0.1.5-alpha.1.tgz"` | +| `.apiops/peterhauge-apiops-cli-0.1.5-alpha.1.tgz` | Local copy of the CLI package | +| `pipelines/run-extractor.yaml` | Extract pipeline | +| `pipelines/run-publisher.yaml` | Publish pipeline | +| `configuration.*.yaml` | Override templates | + +--- + +## Step 3 — Generate the Lock File + +The lock file pins every transitive dependency so `npm ci` can resolve them offline: + +```bash +# In the scaffolded repo directory +npm install +``` + +This creates `package-lock.json`. Commit it — the lock file is **required** for `npm ci` to work. + +--- + +## Step 4 — Cache Dependencies for Offline Install + +`npm ci --offline` requires a populated npm cache. On your connected workstation, populate the cache: + +```bash +# Clean the npm cache to start fresh (optional but recommended) +npm cache clean --force + +# Install to populate the cache with all resolved packages +npm ci + +# Locate the cache directory +npm config get cache +# Default: ~/.npm +``` + +Copy the entire npm cache directory (or the `_cacache` subfolder) to your transfer media. + +> **Alternative — Azure Artifacts upstream feed:** If your air-gapped network has an Azure Artifacts feed with an upstream source configured (accessible during a controlled sync window), you can use it as a local npm registry instead of transferring the cache manually. Configure `.npmrc` to point to the feed. + +> **Alternative — vendored `node_modules`:** If your transfer policy allows it, you can commit or transfer the entire `node_modules` directory. In that case, skip `npm ci` in the pipeline and run `npx apiops` directly. + +--- + +## Step 5 — Transfer Artifacts to the Air-Gapped Zone + +Move these items through your approved transfer channel: + +| Artifact | Destination on Agent | +|----------|-----------------------| +| Repository clone (with `.apiops/` tarball, `package.json`, `package-lock.json`) | Agent's working directory (handled by `checkout`) | +| npm cache (`_cacache/`) | `~/.npm/_cacache/` on the agent (or `%AppData%\npm-cache\_cacache\` on Windows) | + +If using Azure Artifacts, sync the feed during an approved window instead. + +--- + +## Step 6 — Configure the Self-Hosted Agent + +Install and register the agent in the air-gapped network: + +```bash +# On the agent machine (Linux example) +mkdir myagent && cd myagent +# Transfer the agent package via approved media +tar xzf vsts-agent-linux-x64-*.tar.gz +./config.sh --url https://dev.azure.com/ --auth pat --token +./svc.sh install && ./svc.sh start +``` + +Ensure: + +1. **Node.js 22.x** is installed and on `PATH` +2. **npm cache is pre-populated** at `~/.npm` (from Step 4) +3. **Network access to Azure ARM** — the agent must reach `management.azure.com` (or sovereign equivalent) +4. **Network access to Azure DevOps** — the agent must reach your Azure DevOps org for job dispatch (or use GHES-style on-prem server) +5. **Git** is installed (required by the `checkout` step) + +> **Agent pool:** Add your air-gapped agents to a dedicated agent pool (e.g., `air-gapped-pool`) so pipelines target them explicitly. + +--- + +## Step 7 — Modify Pipelines for Offline Operation + +Edit the generated pipelines to use offline npm and target your self-hosted agent pool. + +### `pipelines/run-extractor.yaml` + +```yaml +pool: + name: 'air-gapped-pool' # ← your self-hosted pool + +steps: + - checkout: self + + # Skip NodeTool@0 — Node.js is pre-installed on the agent + + - script: npm ci --offline + displayName: 'Install dependencies (offline)' + + - task: AzureCLI@2 + displayName: 'Run APIM Extract' + inputs: + azureSubscription: '$(AZURE_SERVICE_CONNECTION)' + scriptType: 'bash' + scriptLocation: 'inlineScript' + inlineScript: | + npx apiops extract \ + --resource-group $(APIM_RESOURCE_GROUP) \ + --service-name $(APIM_SERVICE_NAME) \ + --subscription-id $(AZURE_SUBSCRIPTION_ID) \ + --output ./apim-artifacts +``` + +### `pipelines/run-publisher.yaml` + +```yaml +pool: + name: 'air-gapped-pool' + +stages: + - stage: Publish_dev + variables: + - group: apim-dev + jobs: + - deployment: Deploy + environment: dev + strategy: + runOnce: + deploy: + steps: + - checkout: self + fetchDepth: 2 + + - script: npm ci --offline + displayName: 'Install dependencies (offline)' + + - task: AzureCLI@2 + displayName: 'Publish to dev (incremental)' + inputs: + azureSubscription: '$(AZURE_SERVICE_CONNECTION_DEV)' + scriptType: 'bash' + scriptLocation: 'inlineScript' + inlineScript: | + npx apiops publish \ + --resource-group $(APIM_RESOURCE_GROUP_DEV) \ + --service-name $(APIM_SERVICE_NAME_DEV) \ + --subscription-id $(AZURE_SUBSCRIPTION_ID) \ + --source ./apim-artifacts \ + --override configuration.dev.yaml \ + --commit-id $(Build.SourceVersion) +``` + +> **Authentication:** The `AzureCLI@2` task handles Azure authentication via the service connection. No additional credential configuration is needed in the pipeline — the service connection injects tokens into the shell environment for `DefaultAzureCredential`. + +--- + +## Step 8 — Configure Variable Groups and Service Connections + +Follow the standard [Azure DevOps integration guide](../ci-cd/azure-devops.md#variable-groups-configuration) to set up: + +1. **Variable group `apim-common`** — for the extract pipeline +2. **Variable groups `apim-dev`, `apim-prod`** — for the publish pipeline +3. **Service connections** — Azure Resource Manager connections scoped to your APIM instances + +These are configured in Azure DevOps (not in the air-gapped agent) and are injected at pipeline runtime. + +--- + +## Step 9 — Commit and Validate + +```bash +git add . +git commit -m "feat: air-gapped apiops setup with local tarball" +git push +``` + +Trigger the extract pipeline manually from **Pipelines → Run pipeline** and verify: + +1. `npm ci --offline` completes without network calls to npmjs.org +2. `apiops extract` authenticates via the service connection and runs successfully + +--- + +## Using Azure Artifacts as a Local Registry + +If your organization has an Azure Artifacts feed accessible from the air-gapped network (even via a controlled sync window), you can avoid manual cache transfers: + +### Setup + +1. Create an Azure Artifacts npm feed in your Azure DevOps project +2. Configure an upstream source to npmjs.org (used during sync windows only) +3. During a sync window, install all packages to populate the feed +4. After the sync window closes, the feed serves packages locally + +### `.npmrc` Configuration + +Create a `.npmrc` file in the repository root: + +```ini +registry=https://pkgs.dev.azure.com///_packaging//npm/registry/ +always-auth=true +``` + +### Pipeline Authentication + +Add the `npmAuthenticate@0` task before `npm ci`: + +```yaml +- task: npmAuthenticate@0 + inputs: + workingFile: .npmrc + +- script: npm ci + displayName: 'Install from Azure Artifacts feed' +``` + +This approach eliminates the need for `--offline` and manual cache management. + +--- + +## Upgrading the CLI Version + +When a new version is released: + +1. On the connected workstation: `npm pack @peterhauge/apiops-cli` (new version) +2. Replace `.apiops/*.tgz` in the repository +3. Update `package.json` to reference the new tarball filename +4. Run `npm install` to regenerate `package-lock.json` +5. Rebuild the npm cache (`npm ci`) and transfer the updated cache (or sync your Azure Artifacts feed) +6. Commit and push + +--- + +## Troubleshooting + +| Problem | Cause | Fix | +|---------|-------|-----| +| `npm ci` fails with `ENOTCACHED` | npm cache doesn't contain required packages | Re-populate cache on connected workstation and transfer | +| `npm ci` fails with "lockfile mismatch" | `package-lock.json` out of sync with `package.json` | Re-run `npm install` on connected workstation, commit updated lock file | +| `npx apiops` not found | `npm ci` didn't complete or `.bin` not in PATH | Verify `node_modules/.bin/apiops` exists after install | +| Azure auth fails | Agent can't reach Entra ID or ARM endpoint | Verify network allows traffic to `login.microsoftonline.com` and `management.azure.com` | +| `AzureCLI@2` service connection error | Service connection not linked or misconfigured | Verify variable group is linked to pipeline and connection name matches | +| Agent not picking up jobs | Pool name mismatch or agent offline | Confirm pool name in YAML matches the registered agent pool | +| `npmAuthenticate@0` fails | Feed permissions or `.npmrc` path wrong | Ensure the build service identity has Reader access to the feed | + +--- + +## Related + +- [apiops init reference](../commands/init.md) — all `--cli-package` details +- [Azure DevOps integration](../ci-cd/azure-devops.md) — standard (connected) setup +- [Authentication guide](../guides/authentication.md) — service principal and managed identity options +- [Air-gapped setup: GitHub Actions](air-gapped-github-actions.md) diff --git a/docs/walkthrough/air-gapped-github-actions.md b/docs/walkthrough/air-gapped-github-actions.md new file mode 100644 index 0000000..f9bb385 --- /dev/null +++ b/docs/walkthrough/air-gapped-github-actions.md @@ -0,0 +1,259 @@ +# Air-Gapped Setup: GitHub Actions + +Deploy APIM configuration using apiops-cli on GitHub Actions runners with **no internet access** at runtime. This walkthrough covers preparing the npm tarball, lock file, and self-hosted runner configuration so that `npm ci` succeeds entirely offline. + +--- + +## When to Use This Guide + +- Self-hosted runners in a private network with no outbound internet +- Environments requiring artifact pre-staging for security compliance +- Corporate networks that block access to the npm registry + +--- + +## Architecture Overview + +```mermaid +flowchart LR + subgraph Connected Workstation + A[npm pack] --> B[apiops tarball .tgz] + C[npm install] --> D[package-lock.json] + end + subgraph Transfer + B -->|Approved media / artifact feed| E[Self-hosted Runner] + D -->|Same transfer| E + end + subgraph Air-Gapped Runner + E --> F[npm ci --offline] + F --> G[apiops extract / publish] + end +``` + +--- + +## Prerequisites + +| Requirement | Details | +|-------------|---------| +| **Connected workstation** | A machine with internet access to download packages | +| **Node.js 22.x** | Installed on both the workstation and the runner | +| **npm 10+** | Comes with Node.js 22 | +| **Self-hosted GitHub Actions runner** | Registered in your repository, running in the air-gapped network | +| **Azure connectivity from runner** | The runner must reach your APIM instance's ARM endpoint (network-level, not npm) | +| **Transfer mechanism** | USB drive, internal artifact feed, or approved file share for moving packages into the restricted zone | + +--- + +## Step 1 — Prepare the Tarball (Connected Workstation) + +On a machine with internet access: + +```bash +# Install the CLI globally to get access to the apiops command +npm install -g @peterhauge/apiops-cli + +# Pack the installed package into a tarball +npm pack @peterhauge/apiops-cli +# Produces: peterhauge-apiops-cli-.tgz +``` + +Keep the `.tgz` file — you'll transfer it to the air-gapped environment. + +--- + +## Step 2 — Scaffold the Repository + +Run `apiops init` with the `--cli-package` flag pointing to the tarball: + +```bash +apiops init \ + --ci github-actions \ + --cli-package ./peterhauge-apiops-cli-0.1.5-alpha.1.tgz \ + --environments dev,prod \ + --non-interactive +``` + +This generates: + +| File | Purpose | +|------|---------| +| `package.json` | References the tarball as `"file:.apiops/peterhauge-apiops-cli-0.1.5-alpha.1.tgz"` | +| `.apiops/peterhauge-apiops-cli-0.1.5-alpha.1.tgz` | Local copy of the CLI package | +| `.github/workflows/run-extractor.yaml` | Extract workflow | +| `.github/workflows/run-publisher.yaml` | Publish workflow | +| `configuration.*.yaml` | Override templates | + +--- + +## Step 3 — Generate the Lock File + +The lock file pins every transitive dependency so `npm ci` can resolve them offline: + +```bash +# In the scaffolded repo directory +npm install +``` + +This creates `package-lock.json`. Commit it — the lock file is **required** for `npm ci` to work. + +--- + +## Step 4 — Cache Dependencies for Offline Install + +`npm ci --offline` requires a populated npm cache. On your connected workstation, populate the cache: + +```bash +# Clean the npm cache to start fresh (optional but recommended) +npm cache clean --force + +# Install to populate the cache with all resolved packages +npm ci + +# Export the cache directory +npm config get cache +# Default: ~/.npm +``` + +Copy the entire npm cache directory (or the `_cacache` subfolder) to your transfer media. + +> **Alternative — vendored `node_modules`:** If your transfer policy allows it, you can commit or transfer the entire `node_modules` directory. In that case, skip `npm ci` in the workflow and run `npx apiops` directly. + +--- + +## Step 5 — Transfer Artifacts to the Air-Gapped Zone + +Move these items through your approved transfer channel: + +| Artifact | Destination on Runner | +|----------|-----------------------| +| Repository clone (with `.apiops/` tarball, `package.json`, `package-lock.json`) | Runner's `$GITHUB_WORKSPACE` (handled by `actions/checkout`) | +| npm cache (`_cacache/`) | `~/.npm/_cacache/` on the runner | + +If using a vendored `node_modules`, transfer it as part of the repository. + +--- + +## Step 6 — Configure the Self-Hosted Runner + +Install and register the runner in the air-gapped network: + +```bash +# On the runner machine +mkdir actions-runner && cd actions-runner +# Transfer the runner package via approved media +tar xzf actions-runner-linux-x64-*.tar.gz +./config.sh --url https://github.com// --token +./svc.sh install && ./svc.sh start +``` + +Ensure: + +1. **Node.js 22.x** is installed and on `PATH` +2. **npm cache is pre-populated** at `~/.npm` (from Step 4) +3. **Network access to Azure ARM** — the runner must reach `management.azure.com` (or sovereign equivalent) for APIM API calls +4. **Git** is installed (required by `actions/checkout`) + +--- + +## Step 7 — Modify Workflows for Offline Operation + +Edit the generated workflows to use offline npm and target your self-hosted runner. + +### `.github/workflows/run-extractor.yaml` + +```yaml +jobs: + extract: + runs-on: [self-hosted, air-gapped] # ← target your runner labels + steps: + - uses: actions/checkout@v4 + + # Skip actions/setup-node — Node.js is pre-installed on the runner + + - name: Install dependencies (offline) + run: npm ci --offline + + - name: Run extract + run: | + npx apiops extract \ + --resource-group ${{ secrets.APIM_RESOURCE_GROUP }} \ + --service-name ${{ secrets.APIM_SERVICE_NAME }} \ + --subscription-id ${{ secrets.AZURE_SUBSCRIPTION_ID }} \ + --output ./apim-artifacts +``` + +### `.github/workflows/run-publisher.yaml` + +```yaml +jobs: + publish-dev: + runs-on: [self-hosted, air-gapped] + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 2 # needed for incremental publish + + - name: Install dependencies (offline) + run: npm ci --offline + + - name: Publish (incremental) + run: | + npx apiops publish \ + --resource-group ${{ secrets.APIM_RESOURCE_GROUP_DEV }} \ + --service-name ${{ secrets.APIM_SERVICE_NAME_DEV }} \ + --subscription-id ${{ secrets.AZURE_SUBSCRIPTION_ID }} \ + --source ./apim-artifacts \ + --commit-id ${{ github.sha }} +``` + +> **Authentication:** In air-gapped environments, OIDC federation may not be available if the runner can't reach `token.actions.githubusercontent.com`. Use a service principal with `--client-id`, `--client-secret`, and `--tenant-id` flags (or equivalent environment variables) instead. Credentials can be stored in GitHub repository secrets — the runner only needs to reach GitHub's API for secret injection during job startup. + +--- + +## Step 8 — Commit and Validate + +```bash +git add . +git commit -m "feat: air-gapped apiops setup with local tarball" +git push +``` + +Trigger the workflow manually from **Actions → Run workflow** and verify: + +1. `npm ci --offline` completes without network calls +2. `apiops extract` or `apiops publish` authenticates and runs successfully + +--- + +## Upgrading the CLI Version + +When a new version is released: + +1. On the connected workstation: `npm pack @peterhauge/apiops-cli` (new version) +2. Replace `.apiops/*.tgz` in the repository +3. Update `package.json` to reference the new tarball filename +4. Run `npm install` to regenerate `package-lock.json` +5. Rebuild the npm cache (`npm ci`) and transfer the updated cache +6. Commit and push + +--- + +## Troubleshooting + +| Problem | Cause | Fix | +|---------|-------|-----| +| `npm ci` fails with `ENOTCACHED` | npm cache doesn't contain required packages | Re-populate cache on connected workstation and transfer | +| `npm ci` fails with "lockfile mismatch" | `package-lock.json` out of sync with `package.json` | Re-run `npm install` on connected workstation, commit updated lock file | +| `npx apiops` not found | `npm ci` didn't complete or `.bin` not in PATH | Verify `node_modules/.bin/apiops` exists after install | +| Azure auth fails | Runner can't reach Entra ID or ARM endpoint | Verify network allows traffic to `login.microsoftonline.com` and `management.azure.com` | +| `actions/checkout` fails | Runner can't reach GitHub API | Ensure runner has network path to `github.com` (or use GHES with local clone) | + +--- + +## Related + +- [apiops init reference](../commands/init.md) — all `--cli-package` details +- [GitHub Actions integration](../ci-cd/github-actions.md) — standard (connected) setup +- [Authentication guide](../guides/authentication.md) — service principal and managed identity options +- [Air-gapped setup: Azure DevOps](air-gapped-azure-devops.md) From 4a22a7836c5cb642b46fbb7782736f14a197db1b Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 19 May 2026 22:35:05 +0000 Subject: [PATCH 2/8] docs: merge decision on air-gapped walkthrough documentation pattern - Merged inbox/docwriter-airgapped-walkthroughs.md into decisions.md - Established walkthrough pattern for complex multi-step deployment scenarios - Covers GitHub Actions and Azure DevOps air-gapped deployments - Pattern: docs/walkthrough/ (scenario-based), docs/ci-cd/ (reference), docs/guides/ (feature-oriented) - Deleted merged inbox file Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: EMaher <9244742+EMaher@users.noreply.github.com> --- .squad/decisions.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/.squad/decisions.md b/.squad/decisions.md index 023dfe4..3aabea4 100644 --- a/.squad/decisions.md +++ b/.squad/decisions.md @@ -2,6 +2,24 @@ ## Active Decisions +### 2026-06-11T22:34:04Z: Air-Gapped Walkthrough Documentation Pattern +**By:** DocWriter +**Status:** Proposed +**What:** Created a new `docs/walkthrough/` directory for scenario-based step-by-step guides that are more detailed than the standard CI/CD integration docs. First entries cover air-gapped deployments for both GitHub Actions and Azure DevOps. + +**Decision:** Establish walkthrough pattern for complex multi-step deployment scenarios that don't fit cleanly into standard integration guides. + +**Pattern:** +- **`docs/walkthrough/`** = scenario-based, multi-step guides for specific deployment patterns +- **`docs/ci-cd/`** = platform integration reference (standard setup) +- **`docs/guides/`** = feature-oriented how-to guides + +**Scope:** Future walkthrough candidates include multi-region APIM, sovereign cloud deployments, monorepo with multiple APIM instances. Any scenario requiring 5+ steps beyond the standard setup qualifies. + +**Next Steps:** Team review; if approved, publish guides and update docs/README index. + +--- + ### 2026-05-14T05:20:00Z: APIM v1 → v2 SKU Migration via apiops-cli **By:** ApimExpert + ApiOpsLead (joint research and decision) **Status:** Proposed for team governance review From 30b5808183c470cc538761ce522756a1100ce87b Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 19 May 2026 22:35:18 +0000 Subject: [PATCH 3/8] chore: record squad orchestration logs for air-gapped walkthrough task Agent-Logs-Url: https://github.com/Azure/apiops-cli/sessions/55ca6c66-0c45-4eec-9647-95bcedd33b90 Co-authored-by: EMaher <9244742+EMaher@users.noreply.github.com> --- .commit-msg.txt | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 .commit-msg.txt diff --git a/.commit-msg.txt b/.commit-msg.txt new file mode 100644 index 0000000..566469b --- /dev/null +++ b/.commit-msg.txt @@ -0,0 +1,9 @@ +docs: merge decision on air-gapped walkthrough documentation pattern + +- Merged inbox/docwriter-airgapped-walkthroughs.md into decisions.md +- Established walkthrough pattern for complex multi-step deployment scenarios +- Covers GitHub Actions and Azure DevOps air-gapped deployments +- Pattern: docs/walkthrough/ (scenario-based), docs/ci-cd/ (reference), docs/guides/ (feature-oriented) +- Deleted merged inbox file + +Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> From af26aafa24d419064c8cc2f0d87a28fab1f6a283 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 19 May 2026 22:35:32 +0000 Subject: [PATCH 4/8] chore: remove accidental temp commit message file Agent-Logs-Url: https://github.com/Azure/apiops-cli/sessions/55ca6c66-0c45-4eec-9647-95bcedd33b90 Co-authored-by: EMaher <9244742+EMaher@users.noreply.github.com> --- .commit-msg.txt | 9 --------- 1 file changed, 9 deletions(-) delete mode 100644 .commit-msg.txt diff --git a/.commit-msg.txt b/.commit-msg.txt deleted file mode 100644 index 566469b..0000000 --- a/.commit-msg.txt +++ /dev/null @@ -1,9 +0,0 @@ -docs: merge decision on air-gapped walkthrough documentation pattern - -- Merged inbox/docwriter-airgapped-walkthroughs.md into decisions.md -- Established walkthrough pattern for complex multi-step deployment scenarios -- Covers GitHub Actions and Azure DevOps air-gapped deployments -- Pattern: docs/walkthrough/ (scenario-based), docs/ci-cd/ (reference), docs/guides/ (feature-oriented) -- Deleted merged inbox file - -Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> From a4f5edbfc07812a85fbb179d6b66ae0f6273b623 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 19 May 2026 23:13:24 +0000 Subject: [PATCH 5/8] feat: revise air-gapped Azure DevOps doc per PR feedback - Make local npm registry (Azure Artifacts) the primary approach - Demote tarball flow to fallback section - Remove redundant npm and transfer mechanism prerequisites - Add official Microsoft docs links (agents, feeds, on-prem, sovereign clouds) - Replace brittle embedded YAML with edit guidance referencing generated files - Add sovereign clouds and on-premises section Closes #44 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: EMaher <9244742+EMaher@users.noreply.github.com> --- .squad/agents/docwriter/history.md | 18 ++ docs/walkthrough/air-gapped-azure-devops.md | 288 +++++++------------- 2 files changed, 115 insertions(+), 191 deletions(-) diff --git a/.squad/agents/docwriter/history.md b/.squad/agents/docwriter/history.md index 6712071..d8c7be8 100644 --- a/.squad/agents/docwriter/history.md +++ b/.squad/agents/docwriter/history.md @@ -131,3 +131,21 @@ Documentation authoring (2026-04-30 to 2026-05-17): 3-phase plan with 28 user-fa **Patterns:** Examples-first, Mermaid workflows, relative links, search-optimized errors, progressive disclosure **Gotchas:** Auth flags set env vars (credential precedence). Overrides: names consistent, properties differ. `--commit-id`/`--delete-unmatched` exclusive. + +### 2026-06-11: Air-Gapped Azure DevOps — PR Feedback Revision + +**Context:** Revised `docs/walkthrough/air-gapped-azure-devops.md` based on PR review comments. + +**Changes:** +- Made local npm registry (Azure Artifacts) the primary/default approach; tarball flow demoted to fallback section. +- Removed redundant "npm 10+" prerequisite (implied by Node.js 22.x). +- Removed "transfer mechanism" prerequisite row (not needed when local registry is primary). +- Added official Microsoft docs links: self-hosted agents, Azure Artifacts npm feeds, Azure DevOps Server on-prem, sovereign cloud identity endpoints. +- Replaced full embedded pipeline YAML with concise edit table referencing generated files (`pipelines/run-extractor.yaml`, `pipelines/run-publisher.yaml`). +- Added "Sovereign Clouds and On-Premises" section with doc links. +- Simplified architecture diagram to reflect local registry flow. + +**Learnings:** +- PR reviewers strongly prefer docs that match the air-gapped reality: local registries over manual transfers. Azure Artifacts is a natural fit since it's bundled with Azure DevOps Server. +- Large embedded YAML examples are fragile — they drift from generated templates. Better to describe minimal edits to the generated files. +- Always link to official Microsoft docs for infrastructure setup (agents, feeds, sovereign clouds) rather than inlining instructions that will go stale. diff --git a/docs/walkthrough/air-gapped-azure-devops.md b/docs/walkthrough/air-gapped-azure-devops.md index 21670f2..9e527c0 100644 --- a/docs/walkthrough/air-gapped-azure-devops.md +++ b/docs/walkthrough/air-gapped-azure-devops.md @@ -1,6 +1,6 @@ # Air-Gapped Setup: Azure DevOps -Deploy APIM configuration using apiops-cli on Azure DevOps agents with **no internet access** at runtime. This walkthrough covers preparing the npm tarball, lock file, and self-hosted agent configuration so that `npm ci` succeeds entirely offline. +Deploy APIM configuration using apiops-cli on [self-hosted Azure Pipelines agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents) with **no internet access** at runtime. This walkthrough uses a local npm registry (Azure Artifacts) as the primary package source, with a tarball-based fallback for environments without any registry infrastructure. --- @@ -8,7 +8,7 @@ Deploy APIM configuration using apiops-cli on Azure DevOps agents with **no inte - Self-hosted agents in a private network with no outbound internet - Environments requiring artifact pre-staging for security compliance -- Corporate networks that block access to the npm registry +- Corporate networks that block access to the public npm registry --- @@ -16,17 +16,14 @@ Deploy APIM configuration using apiops-cli on Azure DevOps agents with **no inte ```mermaid flowchart LR - subgraph Connected Workstation - A[npm pack] --> B[apiops tarball .tgz] - C[npm install] --> D[package-lock.json] + subgraph Connected Zone + A[npm publish to local feed] --> B[Azure Artifacts feed] end - subgraph Transfer - B -->|Approved media / Azure Artifacts feed| E[Self-hosted Agent] - D -->|Same transfer| E - end - subgraph Air-Gapped Agent - E --> F[npm ci --offline] - F --> G[apiops extract / publish] + subgraph Air-Gapped Network + B -->|Controlled sync window| C[Local feed replica] + C --> D[Self-hosted Agent] + D --> E[npm ci] + E --> F[apiops extract / publish] end ``` @@ -37,39 +34,57 @@ flowchart LR | Requirement | Details | |-------------|---------| | **Connected workstation** | A machine with internet access to download packages | -| **Node.js 22.x** | Installed on both the workstation and the agent | -| **npm 10+** | Comes with Node.js 22 | -| **Self-hosted Azure Pipelines agent** | Registered in your agent pool, running in the air-gapped network | +| **Node.js 22.x** | Installed on both the workstation and the agent (includes npm) | +| **[Self-hosted Azure Pipelines agent](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents)** | Registered in your agent pool, running in the air-gapped network | | **Azure connectivity from agent** | The agent must reach your APIM instance's ARM endpoint (network-level, not npm) | -| **Transfer mechanism** | USB drive, Azure Artifacts upstream feed, or approved file share for moving packages into the restricted zone | +| **Local npm registry** | An [Azure Artifacts npm feed](https://learn.microsoft.com/en-us/azure/devops/artifacts/npm/npmrc?view=azure-devops) accessible from the air-gapped network | + +> **On-premises Azure DevOps:** If you run [Azure DevOps Server](https://learn.microsoft.com/en-us/azure/devops/server/install/get-started?view=azure-devops-2022) (formerly TFS), the same approach applies — Azure Artifacts is included in the server installation. --- -## Step 1 — Prepare the Tarball (Connected Workstation) +## Step 1 — Configure the Local npm Registry + +Set up an [Azure Artifacts npm feed](https://learn.microsoft.com/en-us/azure/devops/artifacts/npm/npmrc?view=azure-devops) that serves packages to your air-gapped agents without requiring internet access at install time. + +### Create the Feed -On a machine with internet access: +1. In Azure DevOps, go to **Artifacts → Create Feed** +2. Name it (e.g., `npm-internal`) and set visibility to your project or organization +3. Add an **upstream source** pointing to `https://registry.npmjs.org` (used only during controlled sync windows) + +### Populate During a Sync Window + +On the connected workstation (or during a controlled connectivity window): ```bash -# Install the CLI globally to get access to the apiops command -npm install -g @peterhauge/apiops-cli +# Point npm at your Azure Artifacts feed +npm config set registry https://pkgs.dev.azure.com///_packaging//npm/registry/ -# Pack the installed package into a tarball -npm pack @peterhauge/apiops-cli -# Produces: peterhauge-apiops-cli-.tgz +# Install the CLI — this caches the package and all dependencies in the feed +npm install @peterhauge/apiops-cli ``` -Keep the `.tgz` file — you'll transfer it to the air-gapped environment. +Once the feed has cached the package, close the upstream connection. The feed now serves packages locally. ---- +### Repository `.npmrc` + +Create a `.npmrc` file in the repository root so all `npm` commands resolve against the local feed: -## Step 2 — Scaffold the Repository +```ini +registry=https://pkgs.dev.azure.com///_packaging//npm/registry/ +always-auth=true +``` + +> See [Azure Artifacts npm feed documentation](https://learn.microsoft.com/en-us/azure/devops/artifacts/npm/npmrc?view=azure-devops) for detailed setup instructions including scoped registries and authentication. + +--- -Run `apiops init` with the `--cli-package` flag pointing to the tarball: +## Step 2 — Initialize the Repository ```bash apiops init \ --ci azure-devops \ - --cli-package ./peterhauge-apiops-cli-0.1.5-alpha.1.tgz \ --environments dev,prod \ --non-interactive ``` @@ -78,8 +93,7 @@ This generates: | File | Purpose | |------|---------| -| `package.json` | References the tarball as `"file:.apiops/peterhauge-apiops-cli-0.1.5-alpha.1.tgz"` | -| `.apiops/peterhauge-apiops-cli-0.1.5-alpha.1.tgz` | Local copy of the CLI package | +| `package.json` | Declares the CLI as a dependency | | `pipelines/run-extractor.yaml` | Extract pipeline | | `pipelines/run-publisher.yaml` | Publish pipeline | | `configuration.*.yaml` | Override templates | @@ -88,10 +102,7 @@ This generates: ## Step 3 — Generate the Lock File -The lock file pins every transitive dependency so `npm ci` can resolve them offline: - ```bash -# In the scaffolded repo directory npm install ``` @@ -99,144 +110,49 @@ This creates `package-lock.json`. Commit it — the lock file is **required** fo --- -## Step 4 — Cache Dependencies for Offline Install - -`npm ci --offline` requires a populated npm cache. On your connected workstation, populate the cache: - -```bash -# Clean the npm cache to start fresh (optional but recommended) -npm cache clean --force - -# Install to populate the cache with all resolved packages -npm ci - -# Locate the cache directory -npm config get cache -# Default: ~/.npm -``` - -Copy the entire npm cache directory (or the `_cacache` subfolder) to your transfer media. - -> **Alternative — Azure Artifacts upstream feed:** If your air-gapped network has an Azure Artifacts feed with an upstream source configured (accessible during a controlled sync window), you can use it as a local npm registry instead of transferring the cache manually. Configure `.npmrc` to point to the feed. - -> **Alternative — vendored `node_modules`:** If your transfer policy allows it, you can commit or transfer the entire `node_modules` directory. In that case, skip `npm ci` in the pipeline and run `npx apiops` directly. - ---- - -## Step 5 — Transfer Artifacts to the Air-Gapped Zone - -Move these items through your approved transfer channel: - -| Artifact | Destination on Agent | -|----------|-----------------------| -| Repository clone (with `.apiops/` tarball, `package.json`, `package-lock.json`) | Agent's working directory (handled by `checkout`) | -| npm cache (`_cacache/`) | `~/.npm/_cacache/` on the agent (or `%AppData%\npm-cache\_cacache\` on Windows) | +## Step 4 — Configure the Self-Hosted Agent -If using Azure Artifacts, sync the feed during an approved window instead. - ---- - -## Step 6 — Configure the Self-Hosted Agent - -Install and register the agent in the air-gapped network: - -```bash -# On the agent machine (Linux example) -mkdir myagent && cd myagent -# Transfer the agent package via approved media -tar xzf vsts-agent-linux-x64-*.tar.gz -./config.sh --url https://dev.azure.com/ --auth pat --token -./svc.sh install && ./svc.sh start -``` +Install and register the agent in the air-gapped network per the [self-hosted agent documentation](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/linux-agent?view=azure-devops): Ensure: 1. **Node.js 22.x** is installed and on `PATH` -2. **npm cache is pre-populated** at `~/.npm` (from Step 4) -3. **Network access to Azure ARM** — the agent must reach `management.azure.com` (or sovereign equivalent) -4. **Network access to Azure DevOps** — the agent must reach your Azure DevOps org for job dispatch (or use GHES-style on-prem server) +2. **Network access to the Azure Artifacts feed** — the agent can resolve packages from the local feed +3. **Network access to Azure ARM** — the agent must reach `management.azure.com` (or [sovereign cloud equivalent](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud)) +4. **Network access to Azure DevOps** — the agent must reach your Azure DevOps org for job dispatch 5. **Git** is installed (required by the `checkout` step) -> **Agent pool:** Add your air-gapped agents to a dedicated agent pool (e.g., `air-gapped-pool`) so pipelines target them explicitly. +> **Agent pool:** Add your air-gapped agents to a dedicated pool (e.g., `air-gapped-pool`) so pipelines target them explicitly. --- -## Step 7 — Modify Pipelines for Offline Operation +## Step 5 — Modify Pipelines for Air-Gapped Operation -Edit the generated pipelines to use offline npm and target your self-hosted agent pool. +The generated pipelines (`pipelines/run-extractor.yaml` and `pipelines/run-publisher.yaml`) need minimal edits for air-gapped operation: -### `pipelines/run-extractor.yaml` +| Edit | What to Change | +|------|----------------| +| **Agent pool** | Replace `pool: vmImage: ubuntu-latest` with `pool: name: air-gapped-pool` | +| **Remove NodeTool task** | Delete the `NodeTool@0` step (Node.js is pre-installed on the agent) | +| **Add feed auth** | Insert `npmAuthenticate@0` before the `npm ci` step | -```yaml -pool: - name: 'air-gapped-pool' # ← your self-hosted pool - -steps: - - checkout: self - - # Skip NodeTool@0 — Node.js is pre-installed on the agent - - - script: npm ci --offline - displayName: 'Install dependencies (offline)' - - - task: AzureCLI@2 - displayName: 'Run APIM Extract' - inputs: - azureSubscription: '$(AZURE_SERVICE_CONNECTION)' - scriptType: 'bash' - scriptLocation: 'inlineScript' - inlineScript: | - npx apiops extract \ - --resource-group $(APIM_RESOURCE_GROUP) \ - --service-name $(APIM_SERVICE_NAME) \ - --subscription-id $(AZURE_SUBSCRIPTION_ID) \ - --output ./apim-artifacts -``` +### Feed Authentication -### `pipelines/run-publisher.yaml` +Add this task before any `npm ci` step in both pipelines: ```yaml -pool: - name: 'air-gapped-pool' - -stages: - - stage: Publish_dev - variables: - - group: apim-dev - jobs: - - deployment: Deploy - environment: dev - strategy: - runOnce: - deploy: - steps: - - checkout: self - fetchDepth: 2 - - - script: npm ci --offline - displayName: 'Install dependencies (offline)' - - - task: AzureCLI@2 - displayName: 'Publish to dev (incremental)' - inputs: - azureSubscription: '$(AZURE_SERVICE_CONNECTION_DEV)' - scriptType: 'bash' - scriptLocation: 'inlineScript' - inlineScript: | - npx apiops publish \ - --resource-group $(APIM_RESOURCE_GROUP_DEV) \ - --service-name $(APIM_SERVICE_NAME_DEV) \ - --subscription-id $(AZURE_SUBSCRIPTION_ID) \ - --source ./apim-artifacts \ - --override configuration.dev.yaml \ - --commit-id $(Build.SourceVersion) +- task: npmAuthenticate@0 + inputs: + workingFile: .npmrc ``` -> **Authentication:** The `AzureCLI@2` task handles Azure authentication via the service connection. No additional credential configuration is needed in the pipeline — the service connection injects tokens into the shell environment for `DefaultAzureCredential`. +The `npm ci` step in the generated pipelines works as-is — it resolves packages from the local feed configured in `.npmrc`. + +> **Authentication:** The `AzureCLI@2` task handles Azure authentication via the service connection. The service connection injects tokens for `DefaultAzureCredential`. --- -## Step 8 — Configure Variable Groups and Service Connections +## Step 6 — Configure Variable Groups and Service Connections Follow the standard [Azure DevOps integration guide](../ci-cd/azure-devops.md#variable-groups-configuration) to set up: @@ -244,72 +160,57 @@ Follow the standard [Azure DevOps integration guide](../ci-cd/azure-devops.md#va 2. **Variable groups `apim-dev`, `apim-prod`** — for the publish pipeline 3. **Service connections** — Azure Resource Manager connections scoped to your APIM instances -These are configured in Azure DevOps (not in the air-gapped agent) and are injected at pipeline runtime. +These are configured in Azure DevOps and are injected at pipeline runtime. --- -## Step 9 — Commit and Validate +## Step 7 — Commit and Validate ```bash git add . -git commit -m "feat: air-gapped apiops setup with local tarball" +git commit -m "feat: air-gapped apiops setup with local registry" git push ``` Trigger the extract pipeline manually from **Pipelines → Run pipeline** and verify: -1. `npm ci --offline` completes without network calls to npmjs.org +1. `npm ci` resolves all packages from the local Azure Artifacts feed (no calls to npmjs.org) 2. `apiops extract` authenticates via the service connection and runs successfully --- -## Using Azure Artifacts as a Local Registry - -If your organization has an Azure Artifacts feed accessible from the air-gapped network (even via a controlled sync window), you can avoid manual cache transfers: - -### Setup +## Fallback: Tarball with Offline Cache -1. Create an Azure Artifacts npm feed in your Azure DevOps project -2. Configure an upstream source to npmjs.org (used during sync windows only) -3. During a sync window, install all packages to populate the feed -4. After the sync window closes, the feed serves packages locally +If your environment has no local registry infrastructure, you can use a tarball-based approach instead. This replaces Steps 1–3 above: -### `.npmrc` Configuration +1. **Pack the CLI** on a connected workstation: `npm pack @peterhauge/apiops-cli` +2. **Initialize with `--cli-package`** (instead of the standard init shown in Step 2): + `apiops init --ci azure-devops --cli-package ./peterhauge-apiops-cli-.tgz` +3. **Generate the lock file**: `npm install` +4. **Populate the npm cache**: Run `npm ci` on the connected workstation, then copy `~/.npm/_cacache/` to the agent +5. **Use `npm ci --offline`** in the pipeline instead of `npm ci` (no `npmAuthenticate@0` needed) -Create a `.npmrc` file in the repository root: +This approach requires manual cache transfers whenever dependencies change. Use the local registry method above when possible. -```ini -registry=https://pkgs.dev.azure.com///_packaging//npm/registry/ -always-auth=true -``` - -### Pipeline Authentication - -Add the `npmAuthenticate@0` task before `npm ci`: +--- -```yaml -- task: npmAuthenticate@0 - inputs: - workingFile: .npmrc +## Upgrading the CLI Version -- script: npm ci - displayName: 'Install from Azure Artifacts feed' -``` +**With local registry:** Sync the feed during a connectivity window to pull the new version, then update `package.json` and regenerate `package-lock.json`. -This approach eliminates the need for `--offline` and manual cache management. +**With tarball fallback:** Pack the new version, replace `.apiops/*.tgz`, update `package.json`, regenerate the lock file, rebuild the cache, and transfer. --- -## Upgrading the CLI Version +## Sovereign Clouds and On-Premises -When a new version is released: +| Scenario | Documentation | +|----------|---------------| +| Azure Government, Azure China, other sovereign clouds | [National cloud identity endpoints](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud) | +| Azure DevOps Server (on-premises) | [Install Azure DevOps Server](https://learn.microsoft.com/en-us/azure/devops/server/install/get-started?view=azure-devops-2022) | +| Self-hosted agents on-prem | [Self-hosted agent setup](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents) | -1. On the connected workstation: `npm pack @peterhauge/apiops-cli` (new version) -2. Replace `.apiops/*.tgz` in the repository -3. Update `package.json` to reference the new tarball filename -4. Run `npm install` to regenerate `package-lock.json` -5. Rebuild the npm cache (`npm ci`) and transfer the updated cache (or sync your Azure Artifacts feed) -6. Commit and push +For sovereign clouds, ensure your service connection targets the correct ARM endpoint and that your agent can reach the corresponding [Entra ID (Azure AD) endpoint](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud#azure-ad-authentication-endpoints) for token acquisition. --- @@ -317,10 +218,11 @@ When a new version is released: | Problem | Cause | Fix | |---------|-------|-----| -| `npm ci` fails with `ENOTCACHED` | npm cache doesn't contain required packages | Re-populate cache on connected workstation and transfer | +| `npm ci` fails with `ENOTCACHED` | npm cache missing packages (tarball flow) | Re-populate cache on connected workstation and transfer | +| `npm ci` fails with `E404` | Package not in local feed | Sync the feed during a connectivity window | | `npm ci` fails with "lockfile mismatch" | `package-lock.json` out of sync with `package.json` | Re-run `npm install` on connected workstation, commit updated lock file | | `npx apiops` not found | `npm ci` didn't complete or `.bin` not in PATH | Verify `node_modules/.bin/apiops` exists after install | -| Azure auth fails | Agent can't reach Entra ID or ARM endpoint | Verify network allows traffic to `login.microsoftonline.com` and `management.azure.com` | +| Azure auth fails | Agent can't reach Entra ID or ARM endpoint | Verify network allows traffic to `login.microsoftonline.com` and `management.azure.com` (or sovereign equivalents) | | `AzureCLI@2` service connection error | Service connection not linked or misconfigured | Verify variable group is linked to pipeline and connection name matches | | Agent not picking up jobs | Pool name mismatch or agent offline | Confirm pool name in YAML matches the registered agent pool | | `npmAuthenticate@0` fails | Feed permissions or `.npmrc` path wrong | Ensure the build service identity has Reader access to the feed | @@ -333,3 +235,7 @@ When a new version is released: - [Azure DevOps integration](../ci-cd/azure-devops.md) — standard (connected) setup - [Authentication guide](../guides/authentication.md) — service principal and managed identity options - [Air-gapped setup: GitHub Actions](air-gapped-github-actions.md) +- [Azure Artifacts npm feeds](https://learn.microsoft.com/en-us/azure/devops/artifacts/npm/npmrc?view=azure-devops) — official feed setup docs +- [Self-hosted agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents) — agent installation and configuration +- [Azure DevOps Server](https://learn.microsoft.com/en-us/azure/devops/server/install/get-started?view=azure-devops-2022) — on-premises installation +- [National cloud endpoints](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud) — sovereign cloud identity configuration From 0829109b372ff99196a7f07b5afba8e937a0a82e Mon Sep 17 00:00:00 2001 From: Elizabeth Maher Date: Thu, 21 May 2026 12:08:18 -0700 Subject: [PATCH 6/8] updating articles for air-gapped environments --- docs/README.md | 10 +- .../air-gapped-azure-devops-local-registry.md | 192 +++++++++++++ ...air-gapped-azure-devops-offline-tarball.md | 216 +++++++++++++++ docs/walkthrough/air-gapped-azure-devops.md | 238 +--------------- ...ir-gapped-github-actions-local-registry.md | 227 ++++++++++++++++ ...r-gapped-github-actions-offline-tarball.md | 251 +++++++++++++++++ docs/walkthrough/air-gapped-github-actions.md | 256 +----------------- 7 files changed, 912 insertions(+), 478 deletions(-) create mode 100644 docs/walkthrough/air-gapped-azure-devops-local-registry.md create mode 100644 docs/walkthrough/air-gapped-azure-devops-offline-tarball.md create mode 100644 docs/walkthrough/air-gapped-github-actions-local-registry.md create mode 100644 docs/walkthrough/air-gapped-github-actions-offline-tarball.md diff --git a/docs/README.md b/docs/README.md index 6fe88ef..7f41ecb 100644 --- a/docs/README.md +++ b/docs/README.md @@ -9,7 +9,7 @@ | [Getting Started](getting-started.md) | Install and run your first extract → publish cycle in 10 minutes | | [Command Reference](commands/) | Detailed docs for [extract](commands/extract.md), [publish](commands/publish.md), [init](commands/init.md) | | [CI/CD Integration](ci-cd/) | Set up [GitHub Actions](ci-cd/github-actions.md) or [Azure DevOps](ci-cd/azure-devops.md) pipelines | -| [Walkthroughs](walkthrough/) | Step-by-step guides: [Air-gapped GitHub Actions](walkthrough/air-gapped-github-actions.md), [Air-gapped Azure DevOps](walkthrough/air-gapped-azure-devops.md) | +| [Walkthroughs](walkthrough/) | Step-by-step guides: [Air-gapped GitHub Actions](walkthrough/air-gapped-github-actions.md) (local registry or offline tarball), [Air-gapped Azure DevOps](walkthrough/air-gapped-azure-devops.md) (local registry or offline tarball) | ## How It Works @@ -80,8 +80,12 @@ docs/ │ ├── overview.md — System design overview │ └── design-principles.md — Architecture principles ├── walkthrough/ -│ ├── air-gapped-github-actions.md — Air-gapped setup with GitHub Actions -│ └── air-gapped-azure-devops.md — Air-gapped setup with Azure DevOps +│ ├── air-gapped-github-actions.md — Air-gapped GitHub Actions (chooser) +│ ├── air-gapped-github-actions-local-registry.md — GitHub Actions via internal npm registry (GHES Packages) +│ ├── air-gapped-github-actions-offline-tarball.md — GitHub Actions via offline tarball + npm cache +│ ├── air-gapped-azure-devops.md — Air-gapped Azure DevOps (chooser) +│ ├── air-gapped-azure-devops-local-registry.md — Azure DevOps via Azure Artifacts feed +│ └── air-gapped-azure-devops-offline-tarball.md — Azure DevOps via offline tarball + npm cache └── troubleshooting/ ├── common-errors.md — Error messages and solutions ├── debugging-guide.md — Debugging with --log-level diff --git a/docs/walkthrough/air-gapped-azure-devops-local-registry.md b/docs/walkthrough/air-gapped-azure-devops-local-registry.md new file mode 100644 index 0000000..a21e39d --- /dev/null +++ b/docs/walkthrough/air-gapped-azure-devops-local-registry.md @@ -0,0 +1,192 @@ +# Air-Gapped Setup: Azure DevOps — Local npm Registry + +Deploy APIM configuration using apiops-cli on [self-hosted Azure Pipelines agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents) with **no internet access** at runtime. This walkthrough uses an Azure Artifacts npm feed as the package source so agents never reach the public npm registry. + +> Looking for the alternative that doesn't require a registry? See [Offline Tarball walkthrough](air-gapped-azure-devops-offline-tarball.md). + +--- + +## When to Use This Guide + +- Self-hosted agents in a private network with no outbound internet +- You can host (or already have) an Azure Artifacts npm feed reachable from the agent +- Corporate networks that block access to the public npm registry + +--- + +## Architecture Overview + +```mermaid +flowchart LR + subgraph Connected Zone + A[npm install from public registry] --> B[Azure Artifacts feed] + end + subgraph Air-Gapped Network + B -->|Controlled sync window| C[Local feed replica] + C --> D[Self-hosted Agent] + D --> E[npm ci] + E --> F[apiops extract / publish] + end +``` + +--- + +## Prerequisites + +| Requirement | Details | +|-------------|---------| +| **Connected workstation** | A machine with internet access to seed the feed | +| **Node.js 22.x** | Installed on both the workstation and the agent (includes npm) | +| **[Self-hosted Azure Pipelines agent](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents)** | Registered in your agent pool, running in the air-gapped network | +| **Azure connectivity from agent** | The agent must reach your APIM instance's ARM endpoint (network-level, not npm) | +| **Local npm registry** | An [Azure Artifacts npm feed](https://learn.microsoft.com/en-us/azure/devops/artifacts/npm/npmrc?view=azure-devops) accessible from the air-gapped network | + +> **On-premises Azure DevOps:** If you run [Azure DevOps Server](https://learn.microsoft.com/en-us/azure/devops/server/install/get-started?view=azure-devops-2022) (formerly TFS), the same approach applies — Azure Artifacts is included in the server installation. + +--- + +## Step 1 — Configure the Azure Artifacts npm Feed + +Set up an Azure Artifacts npm feed that serves packages to your air-gapped agents without requiring internet access at install time. + +1. **[Create a new feed](https://learn.microsoft.com/en-us/azure/devops/artifacts/get-started-npm?view=azure-devops#create-a-feed)** in your Azure DevOps organization. +2. **[Configure an upstream source](https://learn.microsoft.com/en-us/azure/devops/artifacts/how-to/set-up-upstream-sources?view=azure-devops)** pointing to `https://registry.npmjs.org`. The upstream is only used during controlled sync windows; once `@peterhauge/apiops-cli` and its dependencies are cached, the feed serves them locally. +3. **[Populate the feed](https://learn.microsoft.com/en-us/azure/devops/artifacts/npm/npmrc?view=azure-devops)** from a connected workstation by running `npm install @peterhauge/apiops-cli` against the feed registry URL. This pulls the package and its transitive dependencies into the feed cache. +4. **[Add a project `.npmrc`](https://learn.microsoft.com/en-us/azure/devops/artifacts/npm/npmrc?view=azure-devops)** that points `registry=` at your feed URL and sets `always-auth=true`. Commit this file so pipelines and developers resolve against the local feed. + +> **Tip:** Use the **Connect to feed** button in the Azure Artifacts UI to get the exact registry URL and a ready-to-copy `.npmrc` snippet for your feed. + +--- + +## Step 2 — Initialize the Repository + +```bash +apiops init \ + --ci azure-devops \ + --environments dev,prod \ + --non-interactive +``` + +This generates: + +| File | Purpose | +|------|---------| +| `package.json` | Declares the CLI as a dependency | +| `pipelines/run-extractor.yaml` | Extract pipeline | +| `pipelines/run-publisher.yaml` | Publish pipeline | +| `configuration.*.yaml` | Override templates | + +--- + +## Step 3 — Generate the Lock File + +```bash +npm install +``` + +This creates `package-lock.json`. Commit it — the lock file is **required** for `npm ci` to work. + +--- + +## Step 4 — Configure the Self-Hosted Agent + +Install and register the agent in the air-gapped network per the [self-hosted agent documentation](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/linux-agent?view=azure-devops): + +Ensure: + +1. **Node.js 22.x** is installed and on `PATH` +2. **Network access to the Azure Artifacts feed** — the agent can resolve packages from the local feed +3. **Network access to Azure ARM** — the agent must reach `management.azure.com` (or [sovereign cloud equivalent](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud)) +4. **Network access to Azure DevOps** — the agent must reach your Azure DevOps org for job dispatch +5. **Git** is installed (required by the `checkout` step) + +> **Agent pool:** Add your air-gapped agents to a [dedicated agent pool](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/pools-queues?view=azure-devops) (e.g., `air-gapped-pool`) so pipelines target them explicitly. + +--- + +## Step 5 — Modify Pipelines for Air-Gapped Operation + +The generated pipelines (`pipelines/run-extractor.yaml` and `pipelines/run-publisher.yaml`) need minimal edits: + +| Edit | What to Change | +|------|----------------| +| **Agent pool** | Replace `pool: vmImage: ubuntu-latest` with `pool: name: air-gapped-pool` ([pool YAML schema](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/pool?view=azure-pipelines)) | +| **Remove NodeTool task** | Delete the `NodeTool@0` step (Node.js is pre-installed on the agent) | +| **Add feed auth** | Insert [`npmAuthenticate@0`](https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/npm-authenticate-v0?view=azure-pipelines) before the `npm ci` step (see below) | + +### Feed Authentication + +Add this task before any `npm ci` step in both pipelines: + +```yaml +- task: npmAuthenticate@0 + inputs: + workingFile: .npmrc +``` + +`npmAuthenticate@0` reads the committed `.npmrc` and injects credentials for the Azure Artifacts feed using the pipeline's built-in build service identity — no service connection needed for feeds in the same organization. The `npm ci` step then works as-is. + +> **Azure authentication:** The `AzureCLI@2` task handles Azure authentication separately, via the service connection. The service connection injects tokens for `DefaultAzureCredential`. + +--- + +## Step 6 — Configure Variable Groups and Service Connections + +Follow the standard [Azure DevOps integration guide](../ci-cd/azure-devops.md#variable-groups-configuration) to set up: + +1. **Variable group `apim-common`** — variables shared across all environments (e.g., subscription ID, tenant ID, common tags) +2. **Variable groups `apim-dev`, `apim-prod`** — environment-specific variables (e.g., APIM instance name, resource group) that override values from `apim-common` +3. **Service connections** — Azure Resource Manager connections scoped to your APIM instances; they can be referenced from either variable group + +These are configured in Azure DevOps and are injected at pipeline runtime. + +--- + +## Step 7 — Commit and Validate + +```bash +git add . +git commit -m "feat: air-gapped apiops setup with local registry" +git push +``` + +Trigger the extract pipeline manually from **Pipelines → Run pipeline** and verify: + +1. `npm ci` resolves all packages from the local Azure Artifacts feed (no calls to npmjs.org) +2. `apiops extract` authenticates via the service connection and runs successfully + +> **✅ Setup complete.** Your air-gapped apiops pipelines are now operational. The remaining sections cover ongoing maintenance and reference material — read them as needed. + +--- + +## Upgrading the CLI Version + +Sync the feed during a connectivity window to pull the new version, then update `package.json` and regenerate `package-lock.json`. Commit both. + +--- + +## Troubleshooting + +| Problem | Cause | Fix | +|---------|-------|-----| +| `npm ci` fails with `E404` | Package not in local feed | Sync the feed during a connectivity window | +| `npm ci` fails with "lockfile mismatch" | `package-lock.json` out of sync with `package.json` | Re-run `npm install` on connected workstation, commit updated lock file | +| `npx apiops` not found | `npm ci` didn't complete or `.bin` not in PATH | Verify `node_modules/.bin/apiops` exists after install | +| Azure auth fails | Agent can't reach Entra ID or ARM endpoint | Verify network allows traffic to `login.microsoftonline.com` and `management.azure.com` (or sovereign equivalents) | +| `AzureCLI@2` service connection error | Service connection not linked or misconfigured | Verify variable group is linked to pipeline and connection name matches | +| Agent not picking up jobs | Pool name mismatch or agent offline | Confirm pool name in YAML matches the registered agent pool | +| `npmAuthenticate@0` fails | Feed permissions or `.npmrc` path wrong | Ensure the build service identity has Reader access to the feed | + +--- + +## Further Reading + +- [Offline Tarball walkthrough](air-gapped-azure-devops-offline-tarball.md) — alternative for environments without a registry +- [apiops init reference](../commands/init.md) +- [Azure DevOps integration](../ci-cd/azure-devops.md) — standard (connected) setup +- [Authentication guide](../guides/authentication.md) — service principal and managed identity options +- [Azure Artifacts npm feeds](https://learn.microsoft.com/en-us/azure/devops/artifacts/npm/npmrc?view=azure-devops) — official feed setup docs +- [Self-hosted agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents) — agent installation and configuration +- [Azure DevOps Server](https://learn.microsoft.com/en-us/azure/devops/server/install/get-started?view=azure-devops-2022) — on-premises installation +- [National cloud endpoints](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud) — sovereign cloud identity configuration +- [Entra ID authentication endpoints](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud#azure-ad-authentication-endpoints) — per-cloud token acquisition endpoints diff --git a/docs/walkthrough/air-gapped-azure-devops-offline-tarball.md b/docs/walkthrough/air-gapped-azure-devops-offline-tarball.md new file mode 100644 index 0000000..4f16bcf --- /dev/null +++ b/docs/walkthrough/air-gapped-azure-devops-offline-tarball.md @@ -0,0 +1,216 @@ +# Air-Gapped Setup: Azure DevOps — Offline Tarball + +Deploy APIM configuration using apiops-cli on [self-hosted Azure Pipelines agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents) with **no internet access** and **no internal npm registry**. The CLI is packaged as a `.tgz` and the npm cache is pre-staged on the agent so `npm ci --offline` resolves every dependency from disk. + +> If you can host an internal npm feed instead, prefer the [Local npm Registry walkthrough](air-gapped-azure-devops-local-registry.md) — it requires far less manual artifact transfer. + +--- + +## When to Use This Guide + +- Self-hosted agents in a private network with no outbound internet +- No Azure Artifacts feed (or any other internal npm registry) reachable from the agent +- You accept the operational cost of re-transferring the tarball and npm cache whenever dependencies change + +--- + +## Architecture Overview + +```mermaid +flowchart LR + subgraph Connected Workstation + A[npm pack apiops-cli] --> B[apiops-cli.tgz] + B --> C[npm ci] + C --> D[~/.npm/_cacache/] + end + subgraph Air-Gapped Network + B -->|Commit to repo| E[.apiops/*.tgz] + D -->|File copy| F[Agent ~/.npm/_cacache/] + E --> G[Self-hosted Agent] + F --> G + G --> H[npm ci --offline] + H --> I[apiops extract / publish] + end +``` + +--- + +## Prerequisites + +| Requirement | Details | +|-------------|---------| +| **Connected workstation** | A machine with internet access to download the CLI and its dependencies | +| **Node.js 22.x** | Installed on both the workstation and the agent (includes npm) | +| **[Self-hosted Azure Pipelines agent](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents)** | Registered in your agent pool, running in the air-gapped network | +| **Azure connectivity from agent** | The agent must reach your APIM instance's ARM endpoint (network-level only) | +| **File transfer mechanism** | A way to copy the npm cache directory into the air-gapped network | + +> **On-premises Azure DevOps:** If you run [Azure DevOps Server](https://learn.microsoft.com/en-us/azure/devops/server/install/get-started?view=azure-devops-2022), the same approach applies. + +--- + +## Step 1 — Pack the CLI + +On the connected workstation: + +```bash +npm pack @peterhauge/apiops-cli +``` + +This produces `peterhauge-apiops-cli-.tgz` in the current directory. + +Commit the tarball into your repository (e.g., under `.apiops/`) so the pipeline can reference it by path: + +```bash +mkdir -p .apiops +mv peterhauge-apiops-cli-*.tgz .apiops/ +git add .apiops/peterhauge-apiops-cli-*.tgz +``` + +--- + +## Step 2 — Initialize the Repository + +Pass `--cli-package` so the generated `package.json` references the local tarball instead of the public registry: + +```bash +apiops init \ + --ci azure-devops \ + --environments dev,prod \ + --cli-package ./.apiops/peterhauge-apiops-cli-.tgz \ + --non-interactive +``` + +This command generates: + +| File | Purpose | +|------|---------| +| `package.json` | Declares the CLI as a `file:` dependency pointing at the tarball | +| `pipelines/run-extractor.yaml` | Extract pipeline | +| `pipelines/run-publisher.yaml` | Publish pipeline | +| `configuration.*.yaml` | Override templates | + +--- + +## Step 3 — Generate the Lock File and Pre-Stage the npm Cache + +On the **connected workstation**, run: + +```bash +npm install # creates package-lock.json +npm ci # populates ~/.npm/_cacache/ with every package the lock file references +``` + +Commit `package-lock.json` to the repository. The command `npm ci --offline` requires it. + +Then transfer the npm cache to the agent: + +```bash +# On the workstation +tar -czf npm-cacache.tar.gz -C ~/.npm _cacache + +# Transfer npm-cacache.tar.gz into the air-gapped network, then on the agent: +mkdir -p ~/.npm +tar -xzf npm-cacache.tar.gz -C ~/.npm +``` + +> Repeat this cache transfer every time dependencies change (CLI upgrade, new package, etc.). + +--- + +## Step 4 — Configure the Self-Hosted Agent + +Install and register the agent in the air-gapped network per the [self-hosted agent documentation](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/linux-agent?view=azure-devops). + +Verify the following: + +1. **Node.js 22.x** is installed and on `PATH` +2. **npm cache is pre-staged** at the agent service account's `~/.npm/_cacache/` (see Step 3) +3. **Network access to Azure ARM** — the agent must reach `management.azure.com` (or [sovereign cloud equivalent](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud)) +4. **Network access to Azure DevOps** — the agent must reach your Azure DevOps org for job dispatch +5. **Git** is installed (required by the `checkout` step) + +> **Agent pool:** Add your air-gapped agents to a [dedicated agent pool](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/pools-queues?view=azure-devops) (e.g., `air-gapped-pool`) so pipelines target them explicitly. + +--- + +## Step 5 — Modify Pipelines for Air-Gapped Operation + +The generated pipelines (`pipelines/run-extractor.yaml` and `pipelines/run-publisher.yaml`) need three edits: + +| Edit | What to Change | +|------|----------------| +| **Agent pool** | Replace `pool: vmImage: ubuntu-latest` with `pool: name: air-gapped-pool` ([pool YAML schema](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/pool?view=azure-pipelines)) | +| **Remove NodeTool task** | Delete the `NodeTool@0` step (Node.js is pre-installed on the agent) | +| **Use offline `npm ci`** | Change `npm ci` to `npm ci --offline` so npm resolves entirely from the pre-staged cache | + +No `npmAuthenticate@0` task is needed — there is no registry to authenticate against. The `--offline` flag makes npm fail fast if any package is missing from the cache instead of attempting a network call. + +> **Azure authentication:** The `AzureCLI@2` task handles Azure authentication via the service connection. The service connection injects tokens for `DefaultAzureCredential`. + +--- + +## Step 6 — Configure Variable Groups and Service Connections + +Follow the standard [Azure DevOps integration guide](../ci-cd/azure-devops.md#variable-groups-configuration) to set up: + +1. **Variable group `apim-common`** — variables shared across all environments (e.g., subscription ID, tenant ID, common tags) +2. **Variable groups `apim-dev`, `apim-prod`** — environment-specific variables (e.g., APIM instance name, resource group) that override values from `apim-common` +3. **Service connections** — Azure Resource Manager connections scoped to your APIM instances; they can be referenced from either variable group + +These are configured in Azure DevOps and are injected at pipeline runtime. + +--- + +## Step 7 — Commit and Validate + +```bash +git add . +git commit -m "feat: air-gapped apiops setup with offline tarball" +git push +``` + +Trigger the extract pipeline manually from **Pipelines → Run pipeline** and verify: + +1. `npm ci --offline` completes with no network calls +2. `apiops extract` authenticates via the service connection and runs successfully + +> **✅ Setup complete.** Your air-gapped apiops pipelines are now operational. The remaining sections cover ongoing maintenance and reference material — read them as needed. + +--- + +## Upgrading the CLI Version + +1. On a connected workstation, run `npm pack @peterhauge/apiops-cli` for the new version +2. Replace `.apiops/peterhauge-apiops-cli-*.tgz` with the new tarball and update the `file:` path in `package.json` +3. Regenerate `package-lock.json` (`npm install`) +4. Re-populate and re-transfer the npm cache (`npm ci` on the workstation, then copy `~/.npm/_cacache/`) +5. Commit the tarball and updated lock file + +--- + +## Troubleshooting + +| Problem | Cause | Fix | +|---------|-------|-----| +| `npm ci` fails with `ENOTCACHED` | npm cache missing one or more packages | Re-run `npm ci` on the connected workstation and re-transfer `~/.npm/_cacache/` | +| `npm ci` fails with "lockfile mismatch" | `package-lock.json` out of sync with `package.json` | Re-run `npm install` on connected workstation, commit updated lock file, refresh cache | +| `npm install` complains about missing tarball | Path in `package.json` doesn't match the file on disk | Verify the `file:` reference matches the committed `.tgz` filename exactly | +| `npx apiops` not found | `npm ci --offline` didn't complete or `.bin` not in PATH | Verify `node_modules/.bin/apiops` exists after install | +| Azure auth fails | Agent can't reach Entra ID or ARM endpoint | Verify network allows traffic to `login.microsoftonline.com` and `management.azure.com` (or sovereign equivalents) | +| `AzureCLI@2` service connection error | Service connection not linked or misconfigured | Verify variable group is linked to pipeline and connection name matches | +| Agent not picking up jobs | Pool name mismatch or agent offline | Confirm pool name in YAML matches the registered agent pool | + +--- + +## Further Reading + +- [Local npm Registry walkthrough](air-gapped-azure-devops-local-registry.md) — recommended when an internal feed is available +- [apiops init reference](../commands/init.md) — full `--cli-package` documentation +- [Azure DevOps integration](../ci-cd/azure-devops.md) — standard (connected) setup +- [Authentication guide](../guides/authentication.md) — service principal and managed identity options +- [Air-gapped setup: GitHub Actions](air-gapped-github-actions.md) +- [Self-hosted agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents) — agent installation and configuration +- [Azure DevOps Server](https://learn.microsoft.com/en-us/azure/devops/server/install/get-started?view=azure-devops-2022) — on-premises installation +- [National cloud endpoints](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud) — sovereign cloud identity configuration +- [Entra ID authentication endpoints](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud#azure-ad-authentication-endpoints) — per-cloud token acquisition endpoints diff --git a/docs/walkthrough/air-gapped-azure-devops.md b/docs/walkthrough/air-gapped-azure-devops.md index 9e527c0..0b1cf57 100644 --- a/docs/walkthrough/air-gapped-azure-devops.md +++ b/docs/walkthrough/air-gapped-azure-devops.md @@ -1,241 +1,21 @@ # Air-Gapped Setup: Azure DevOps -Deploy APIM configuration using apiops-cli on [self-hosted Azure Pipelines agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents) with **no internet access** at runtime. This walkthrough uses a local npm registry (Azure Artifacts) as the primary package source, with a tarball-based fallback for environments without any registry infrastructure. +Deploy APIM configuration using apiops-cli on [self-hosted Azure Pipelines agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents) with **no internet access** at runtime. There are two supported approaches — pick the one that matches your environment and follow that walkthrough end-to-end. --- -## When to Use This Guide +## Choose Your Approach -- Self-hosted agents in a private network with no outbound internet -- Environments requiring artifact pre-staging for security compliance -- Corporate networks that block access to the public npm registry - ---- - -## Architecture Overview - -```mermaid -flowchart LR - subgraph Connected Zone - A[npm publish to local feed] --> B[Azure Artifacts feed] - end - subgraph Air-Gapped Network - B -->|Controlled sync window| C[Local feed replica] - C --> D[Self-hosted Agent] - D --> E[npm ci] - E --> F[apiops extract / publish] - end -``` - ---- - -## Prerequisites - -| Requirement | Details | -|-------------|---------| -| **Connected workstation** | A machine with internet access to download packages | -| **Node.js 22.x** | Installed on both the workstation and the agent (includes npm) | -| **[Self-hosted Azure Pipelines agent](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents)** | Registered in your agent pool, running in the air-gapped network | -| **Azure connectivity from agent** | The agent must reach your APIM instance's ARM endpoint (network-level, not npm) | -| **Local npm registry** | An [Azure Artifacts npm feed](https://learn.microsoft.com/en-us/azure/devops/artifacts/npm/npmrc?view=azure-devops) accessible from the air-gapped network | - -> **On-premises Azure DevOps:** If you run [Azure DevOps Server](https://learn.microsoft.com/en-us/azure/devops/server/install/get-started?view=azure-devops-2022) (formerly TFS), the same approach applies — Azure Artifacts is included in the server installation. - ---- - -## Step 1 — Configure the Local npm Registry - -Set up an [Azure Artifacts npm feed](https://learn.microsoft.com/en-us/azure/devops/artifacts/npm/npmrc?view=azure-devops) that serves packages to your air-gapped agents without requiring internet access at install time. - -### Create the Feed - -1. In Azure DevOps, go to **Artifacts → Create Feed** -2. Name it (e.g., `npm-internal`) and set visibility to your project or organization -3. Add an **upstream source** pointing to `https://registry.npmjs.org` (used only during controlled sync windows) - -### Populate During a Sync Window - -On the connected workstation (or during a controlled connectivity window): - -```bash -# Point npm at your Azure Artifacts feed -npm config set registry https://pkgs.dev.azure.com///_packaging//npm/registry/ - -# Install the CLI — this caches the package and all dependencies in the feed -npm install @peterhauge/apiops-cli -``` - -Once the feed has cached the package, close the upstream connection. The feed now serves packages locally. - -### Repository `.npmrc` - -Create a `.npmrc` file in the repository root so all `npm` commands resolve against the local feed: - -```ini -registry=https://pkgs.dev.azure.com///_packaging//npm/registry/ -always-auth=true -``` - -> See [Azure Artifacts npm feed documentation](https://learn.microsoft.com/en-us/azure/devops/artifacts/npm/npmrc?view=azure-devops) for detailed setup instructions including scoped registries and authentication. - ---- - -## Step 2 — Initialize the Repository - -```bash -apiops init \ - --ci azure-devops \ - --environments dev,prod \ - --non-interactive -``` - -This generates: - -| File | Purpose | -|------|---------| -| `package.json` | Declares the CLI as a dependency | -| `pipelines/run-extractor.yaml` | Extract pipeline | -| `pipelines/run-publisher.yaml` | Publish pipeline | -| `configuration.*.yaml` | Override templates | - ---- - -## Step 3 — Generate the Lock File - -```bash -npm install -``` - -This creates `package-lock.json`. Commit it — the lock file is **required** for `npm ci` to work. - ---- - -## Step 4 — Configure the Self-Hosted Agent - -Install and register the agent in the air-gapped network per the [self-hosted agent documentation](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/linux-agent?view=azure-devops): - -Ensure: - -1. **Node.js 22.x** is installed and on `PATH` -2. **Network access to the Azure Artifacts feed** — the agent can resolve packages from the local feed -3. **Network access to Azure ARM** — the agent must reach `management.azure.com` (or [sovereign cloud equivalent](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud)) -4. **Network access to Azure DevOps** — the agent must reach your Azure DevOps org for job dispatch -5. **Git** is installed (required by the `checkout` step) - -> **Agent pool:** Add your air-gapped agents to a dedicated pool (e.g., `air-gapped-pool`) so pipelines target them explicitly. - ---- - -## Step 5 — Modify Pipelines for Air-Gapped Operation - -The generated pipelines (`pipelines/run-extractor.yaml` and `pipelines/run-publisher.yaml`) need minimal edits for air-gapped operation: - -| Edit | What to Change | -|------|----------------| -| **Agent pool** | Replace `pool: vmImage: ubuntu-latest` with `pool: name: air-gapped-pool` | -| **Remove NodeTool task** | Delete the `NodeTool@0` step (Node.js is pre-installed on the agent) | -| **Add feed auth** | Insert `npmAuthenticate@0` before the `npm ci` step | - -### Feed Authentication - -Add this task before any `npm ci` step in both pipelines: - -```yaml -- task: npmAuthenticate@0 - inputs: - workingFile: .npmrc -``` - -The `npm ci` step in the generated pipelines works as-is — it resolves packages from the local feed configured in `.npmrc`. - -> **Authentication:** The `AzureCLI@2` task handles Azure authentication via the service connection. The service connection injects tokens for `DefaultAzureCredential`. - ---- - -## Step 6 — Configure Variable Groups and Service Connections - -Follow the standard [Azure DevOps integration guide](../ci-cd/azure-devops.md#variable-groups-configuration) to set up: - -1. **Variable group `apim-common`** — for the extract pipeline -2. **Variable groups `apim-dev`, `apim-prod`** — for the publish pipeline -3. **Service connections** — Azure Resource Manager connections scoped to your APIM instances - -These are configured in Azure DevOps and are injected at pipeline runtime. - ---- - -## Step 7 — Commit and Validate - -```bash -git add . -git commit -m "feat: air-gapped apiops setup with local registry" -git push -``` - -Trigger the extract pipeline manually from **Pipelines → Run pipeline** and verify: - -1. `npm ci` resolves all packages from the local Azure Artifacts feed (no calls to npmjs.org) -2. `apiops extract` authenticates via the service connection and runs successfully - ---- - -## Fallback: Tarball with Offline Cache - -If your environment has no local registry infrastructure, you can use a tarball-based approach instead. This replaces Steps 1–3 above: - -1. **Pack the CLI** on a connected workstation: `npm pack @peterhauge/apiops-cli` -2. **Initialize with `--cli-package`** (instead of the standard init shown in Step 2): - `apiops init --ci azure-devops --cli-package ./peterhauge-apiops-cli-.tgz` -3. **Generate the lock file**: `npm install` -4. **Populate the npm cache**: Run `npm ci` on the connected workstation, then copy `~/.npm/_cacache/` to the agent -5. **Use `npm ci --offline`** in the pipeline instead of `npm ci` (no `npmAuthenticate@0` needed) - -This approach requires manual cache transfers whenever dependencies change. Use the local registry method above when possible. - ---- - -## Upgrading the CLI Version - -**With local registry:** Sync the feed during a connectivity window to pull the new version, then update `package.json` and regenerate `package-lock.json`. - -**With tarball fallback:** Pack the new version, replace `.apiops/*.tgz`, update `package.json`, regenerate the lock file, rebuild the cache, and transfer. - ---- - -## Sovereign Clouds and On-Premises - -| Scenario | Documentation | -|----------|---------------| -| Azure Government, Azure China, other sovereign clouds | [National cloud identity endpoints](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud) | -| Azure DevOps Server (on-premises) | [Install Azure DevOps Server](https://learn.microsoft.com/en-us/azure/devops/server/install/get-started?view=azure-devops-2022) | -| Self-hosted agents on-prem | [Self-hosted agent setup](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents) | - -For sovereign clouds, ensure your service connection targets the correct ARM endpoint and that your agent can reach the corresponding [Entra ID (Azure AD) endpoint](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud#azure-ad-authentication-endpoints) for token acquisition. - ---- - -## Troubleshooting - -| Problem | Cause | Fix | -|---------|-------|-----| -| `npm ci` fails with `ENOTCACHED` | npm cache missing packages (tarball flow) | Re-populate cache on connected workstation and transfer | -| `npm ci` fails with `E404` | Package not in local feed | Sync the feed during a connectivity window | -| `npm ci` fails with "lockfile mismatch" | `package-lock.json` out of sync with `package.json` | Re-run `npm install` on connected workstation, commit updated lock file | -| `npx apiops` not found | `npm ci` didn't complete or `.bin` not in PATH | Verify `node_modules/.bin/apiops` exists after install | -| Azure auth fails | Agent can't reach Entra ID or ARM endpoint | Verify network allows traffic to `login.microsoftonline.com` and `management.azure.com` (or sovereign equivalents) | -| `AzureCLI@2` service connection error | Service connection not linked or misconfigured | Verify variable group is linked to pipeline and connection name matches | -| Agent not picking up jobs | Pool name mismatch or agent offline | Confirm pool name in YAML matches the registered agent pool | -| `npmAuthenticate@0` fails | Feed permissions or `.npmrc` path wrong | Ensure the build service identity has Reader access to the feed | +| Approach | Best For | Walkthrough | +|----------|----------|-------------| +| **Local npm Registry** (recommended) | You have (or can host) an Azure Artifacts npm feed reachable from the agent. Packages flow through the feed; only the feed needs controlled connectivity to refresh. | [Local npm Registry walkthrough](air-gapped-azure-devops-local-registry.md) | +| **Offline Tarball** | No internal npm registry is available. The CLI ships as a `.tgz` in your repo and the npm cache is transferred manually to the agent. Requires re-staging on every dependency change. | [Offline Tarball walkthrough](air-gapped-azure-devops-offline-tarball.md) | --- ## Related -- [apiops init reference](../commands/init.md) — all `--cli-package` details -- [Azure DevOps integration](../ci-cd/azure-devops.md) — standard (connected) setup -- [Authentication guide](../guides/authentication.md) — service principal and managed identity options - [Air-gapped setup: GitHub Actions](air-gapped-github-actions.md) -- [Azure Artifacts npm feeds](https://learn.microsoft.com/en-us/azure/devops/artifacts/npm/npmrc?view=azure-devops) — official feed setup docs -- [Self-hosted agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops#self-hosted-agents) — agent installation and configuration -- [Azure DevOps Server](https://learn.microsoft.com/en-us/azure/devops/server/install/get-started?view=azure-devops-2022) — on-premises installation -- [National cloud endpoints](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud) — sovereign cloud identity configuration +- [Azure DevOps integration](../ci-cd/azure-devops.md) — standard (connected) setup +- [apiops init reference](../commands/init.md) +- [Authentication guide](../guides/authentication.md) diff --git a/docs/walkthrough/air-gapped-github-actions-local-registry.md b/docs/walkthrough/air-gapped-github-actions-local-registry.md new file mode 100644 index 0000000..68ccc6c --- /dev/null +++ b/docs/walkthrough/air-gapped-github-actions-local-registry.md @@ -0,0 +1,227 @@ +# Air-Gapped Setup: GitHub Actions — Local npm Registry + +Deploy APIM configuration using apiops-cli on [self-hosted GitHub Actions runners](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners) with **no internet access** at runtime. This walkthrough uses [GitHub Packages on GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@latest/packages/working-with-a-github-packages-registry/working-with-the-npm-registry) as the npm registry so runners never reach the public npm registry. + +> Looking for the alternative that doesn't require a registry? See [Offline Tarball walkthrough](air-gapped-github-actions-offline-tarball.md). + +> **GitHub.com vs. GitHub Enterprise Server:** GitHub.com's hosted GitHub Packages registry (`npm.pkg.github.com`) is an internet service and cannot serve as an air-gapped registry. This walkthrough assumes [GitHub Enterprise Server (GHES)](https://docs.github.com/en/enterprise-server@latest/admin/overview/about-github-enterprise-server) running inside your network, which can host [GitHub Packages](https://docs.github.com/en/enterprise-server@latest/admin/packages/getting-started-with-github-packages-for-your-enterprise) including the npm registry. + +--- + +## When to Use This Guide + +- Self-hosted runners in a private network with no outbound internet +- You run GHES with GitHub Packages enabled +- Corporate network that blocks access to the public npm registry + +--- + +## Architecture Overview + +```mermaid +flowchart LR + subgraph Connected Zone + A[npm install from public registry] --> B[GHES Packages npm registry] + end + subgraph Air-Gapped Network + B -->|Controlled sync window| C[Local registry replica] + C --> D[Self-hosted Runner] + D --> E[npm ci] + E --> F[apiops extract / publish] + end +``` + +--- + +## Prerequisites + +| Requirement | Details | +|-------------|---------| +| **Connected workstation** | A machine with internet access to seed the registry | +| **Node.js 22.x** | Installed on both the workstation and the runner (includes npm) | +| **[Self-hosted GitHub Actions runner](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners)** | Registered in your repository or organization, running in the air-gapped network | +| **Azure connectivity from runner** | The runner must reach your APIM instance's ARM endpoint (network-level, not npm) | +| **Internal npm registry** | A [GitHub Packages npm registry](https://docs.github.com/en/enterprise-server@latest/packages/working-with-a-github-packages-registry/working-with-the-npm-registry) on your GHES instance, reachable from the air-gapped network. | + +> **GHES on-premises:** If you run [GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@latest/admin/overview/about-github-enterprise-server), the runner, the npm registry, and job dispatch can all live entirely inside your network with no internet egress. GHES is a self-contained appliance "governed by access and security controls that you define, such as firewalls, network policies, IAM, monitoring, and VPNs" ([About GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@latest/admin/overview/about-github-enterprise-server)), and outbound connectivity to GitHub.com is only required if you opt in to [GitHub Connect](https://docs.github.com/en/enterprise-server@latest/admin/configuring-settings/configuring-github-connect/enabling-github-connect-for-githubcom). + +--- + +## Step 1 — Configure the GHES Packages npm Registry + +Set up the [GitHub Packages](https://docs.github.com/en/enterprise-server@latest/admin/packages/getting-started-with-github-packages-for-your-enterprise) npm registry on your GHES instance so it serves packages to your air-gapped runners without requiring internet access at install time. + +1. **[Enable GitHub Packages on GHES](https://docs.github.com/en/enterprise-server@latest/admin/packages/getting-started-with-github-packages-for-your-enterprise)** — turn on the Packages service for your enterprise and configure the storage backend. The npm registry endpoint is `https://npm./`. +2. **Populate the registry** from a connected workstation by running `npm install @peterhauge/apiops-cli` against the GHES npm registry URL. This pulls the package and its transitive dependencies into the registry cache. +3. **Add a project `.npmrc`** that points `registry=` at your GHES npm endpoint and sets `//npm./:_authToken=${NODE_AUTH_TOKEN}` so authentication is read from an environment variable injected at workflow runtime. Commit `.npmrc` so workflows and developers resolve against the local registry. + +> **Tip:** Follow [Authenticating to GitHub Packages](https://docs.github.com/en/enterprise-server@latest/packages/working-with-a-github-packages-registry/working-with-the-npm-registry#authenticating-to-github-packages) for the exact `.npmrc` format your GHES version expects. + +--- + +## Step 2 — Initialize the Repository + +```bash +apiops init \ + --ci github-actions \ + --environments dev,prod \ + --non-interactive +``` + +This generates: + +| File | Purpose | +|------|---------| +| `package.json` | Declares the CLI as a dependency | +| `.github/workflows/run-extractor.yaml` | Extract workflow | +| `.github/workflows/run-publisher.yaml` | Publish workflow | +| `configuration.*.yaml` | Override templates | + +--- + +## Step 3 — Generate the Lock File + +```bash +npm install +``` + +This creates `package-lock.json`. Commit it — the lock file is **required** for `npm ci` to work and pins every transitive dependency to a registry-resolved tarball URL. + +--- + +## Step 4 — Configure the Self-Hosted Runner + +Install and register the runner in the air-gapped network per the [self-hosted runner documentation](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners). + +Verify the following: + +1. **Node.js 22.x** is installed and on `PATH` +2. **Network access to the GHES Packages npm registry** — the runner can resolve packages from `https://npm./` +3. **Network access to Azure ARM** — the runner must reach `management.azure.com` (or [sovereign cloud equivalent](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud)) +4. **Network access to GHES** — the runner must reach your GHES instance for job dispatch, `actions/checkout`, and secret injection +5. **Git** is installed (required by `actions/checkout`) + +> **Runner labels:** Add a custom label (e.g., `air-gapped`) when registering the runner so workflows can target it via `runs-on: [self-hosted, air-gapped]`. + +--- + +## Step 5 — Modify Workflows for Air-Gapped Operation + +The generated workflows need a few edits per job: + +| Edit | What to Change | +|------|----------------| +| **Runner target** | Replace `runs-on: ubuntu-latest` with `runs-on: [self-hosted, air-gapped]` | +| **`setup-node` registry** | Configure `actions/setup-node@v4` with `registry-url:` pointing at your GHES npm endpoint, and pass `NODE_AUTH_TOKEN` as an env var on the `npm ci` step (see below) | +| **Keep `npm ci`** | `.npmrc` already points at the GHES registry; no `--offline` flag is needed | + +### Registry Authentication + +```yaml +jobs: + extract: + runs-on: [self-hosted, air-gapped] + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: '22' + registry-url: 'https://npm./' # your GHES Packages npm endpoint + + - name: Install dependencies + env: + NODE_AUTH_TOKEN: ${{ secrets.GHES_PACKAGES_TOKEN }} + run: npm ci + + - name: Run extract + env: + AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }} + AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }} + AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }} + run: | + npx apiops extract \ + --resource-group ${{ secrets.APIM_RESOURCE_GROUP }} \ + --service-name ${{ secrets.APIM_SERVICE_NAME }} \ + --subscription-id ${{ secrets.AZURE_SUBSCRIPTION_ID }} \ + --output ./apim-artifacts +``` + +`actions/setup-node@v4` writes the `registry-url` and an auth-token placeholder into a runner-local `.npmrc`; `NODE_AUTH_TOKEN` is substituted in at install time. + +> **Authentication to Azure:** +> +> 1. **Managed identity (preferred)** — if your self-hosted runner is an Azure VM, attach a [user-assigned or system-assigned managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview), grant it APIM roles, and call `azure/login@v3` with `auth-type: IDENTITY`. No secrets to store, no internet egress required, and the token request stays inside the Azure backbone. See [Login With User-assigned Managed Identity](https://github.com/Azure/login#login-with-user-assigned-managed-identity) in the `azure/login` README. +> 2. **OIDC federation** — requires the runner to reach the GitHub Actions OIDC issuer at `token.actions.githubusercontent.com` to request the JWT that Entra ID validates ([GitHub OIDC overview](https://docs.github.com/en/actions/concepts/security/openid-connect#understanding-the-oidc-token), [Configuring OIDC in Azure](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-azure)). Use this when the runner is not on Azure compute but can still reach that endpoint. +> 3. **Service principal secret (fallback)** — when neither managed identity nor OIDC is viable, supply `AZURE_CLIENT_ID`/`AZURE_CLIENT_SECRET`/`AZURE_TENANT_ID` from repository secrets. +> +> See the [authentication guide](../guides/authentication.md) for full details. + +--- + +## Step 6 — Configure Repository Secrets + +Configure the secrets your workflows reference under **Settings → Secrets and variables → Actions**: + +| Secret | Purpose | +|--------|---------| +| `GHES_PACKAGES_TOKEN` | Personal access token / service token with `read:packages` scope on your GHES Packages registry | +| `AZURE_SUBSCRIPTION_ID` | Target subscription | +| `AZURE_TENANT_ID` | Entra ID tenant | +| `AZURE_CLIENT_ID` | Service principal app ID (if not using OIDC) | +| `AZURE_CLIENT_SECRET` | Service principal secret (if not using OIDC) | +| `APIM_RESOURCE_GROUP`, `APIM_SERVICE_NAME` | Per-environment APIM identifiers | + +Use [environment-scoped secrets](https://docs.github.com/en/actions/deployment/targeting-different-environments/managing-environments-for-deployment) for per-environment values (`dev`, `prod`). + +--- + +## Step 7 — Commit and Validate + +```bash +git add . +git commit -m "feat: air-gapped apiops setup with local registry" +git push +``` + +Trigger the extract workflow manually from **Actions → Run workflow** and verify: + +1. `npm ci` resolves all packages from the GHES Packages registry (no calls to `registry.npmjs.org`) +2. `apiops extract` authenticates and runs successfully + +> **✅ Setup complete.** Your air-gapped apiops workflows are now operational. The remaining sections cover ongoing maintenance and reference material — read them as needed. + +--- + +## Upgrading the CLI Version + +Sync the GHES Packages registry during a connectivity window to pull the new version, then update `package.json` and regenerate `package-lock.json`. Commit both. + +--- + +## Troubleshooting + +| Problem | Cause | Fix | +|---------|-------|-----| +| `npm ci` fails with `E404` | Package not in the GHES Packages registry | Sync the registry during a connectivity window | +| `npm ci` fails with `E401` / `E403` | `NODE_AUTH_TOKEN` missing, expired, or lacks `read:packages` | Re-create the PAT / service token and update the repository secret | +| `npm ci` fails with "lockfile mismatch" | `package-lock.json` out of sync with `package.json` | Re-run `npm install` on connected workstation, commit updated lock file | +| `npx apiops` not found | `npm ci` didn't complete or `.bin` not in PATH | Verify `node_modules/.bin/apiops` exists after install | +| Azure auth fails | Runner can't reach Entra ID or ARM endpoint | Verify network allows traffic to `login.microsoftonline.com` and `management.azure.com` (or sovereign equivalents) | +| OIDC token request fails | Runner blocked from `token.actions.githubusercontent.com` | Switch to service principal credentials in repository secrets | +| `actions/checkout` fails | Runner can't reach GHES API | Ensure runner has network path to your GHES instance | +| Runner not picking up jobs | Label mismatch or runner offline | Confirm `runs-on` labels match the registered runner | + +--- + +## Further Reading + +- [Offline Tarball walkthrough](air-gapped-github-actions-offline-tarball.md) — alternative for environments without a registry +- [apiops init reference](../commands/init.md) +- [GitHub Actions integration](../ci-cd/github-actions.md) — standard (connected) setup +- [Authentication guide](../guides/authentication.md) — service principal and managed identity options +- [GitHub Packages npm registry](https://docs.github.com/en/enterprise-server@latest/packages/working-with-a-github-packages-registry/working-with-the-npm-registry) — official GHES npm registry docs +- [Self-hosted runners](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners) — runner installation and configuration +- [GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@latest/admin/overview/about-github-enterprise-server) — on-premises GitHub +- [National cloud endpoints](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud) — sovereign cloud identity configuration +- [Entra ID authentication endpoints](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud#azure-ad-authentication-endpoints) — per-cloud token acquisition endpoints diff --git a/docs/walkthrough/air-gapped-github-actions-offline-tarball.md b/docs/walkthrough/air-gapped-github-actions-offline-tarball.md new file mode 100644 index 0000000..a70fbca --- /dev/null +++ b/docs/walkthrough/air-gapped-github-actions-offline-tarball.md @@ -0,0 +1,251 @@ +# Air-Gapped Setup: GitHub Actions — Offline Tarball + +Deploy APIM configuration using apiops-cli on [self-hosted GitHub Actions runners](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners) with **no internet access** and **no internal npm registry**. The CLI is packaged as a `.tgz` and the npm cache is pre-staged on the runner so `npm ci --offline` resolves every dependency from disk. + +> If you can host an internal npm feed instead (e.g., [GitHub Packages on GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@latest/packages/working-with-a-github-packages-registry/working-with-the-npm-registry)), prefer the [Local npm Registry walkthrough](air-gapped-github-actions-local-registry.md) — it requires far less manual artifact transfer. + +--- + +## When to Use This Guide + +- Self-hosted runners in a private network with no outbound internet +- No GitHub Packages / GitHub Enterprise Server / other internal npm registry reachable from the runner +- You accept the operational cost of re-transferring the tarball and npm cache whenever dependencies change + +--- + +## Architecture Overview + +```mermaid +flowchart LR + subgraph Connected Workstation + A[npm pack apiops-cli] --> B[apiops-cli.tgz] + B --> C[npm ci] + C --> D[~/.npm/_cacache/] + end + subgraph Air-Gapped Network + B -->|Commit to repo| E[.apiops/*.tgz] + D -->|File copy| F[Runner ~/.npm/_cacache/] + E --> G[Self-hosted Runner] + F --> G + G --> H[npm ci --offline] + H --> I[apiops extract / publish] + end +``` + +--- + +## Prerequisites + +| Requirement | Details | +|-------------|---------| +| **Connected workstation** | A machine with internet access to download the CLI and its dependencies | +| **Node.js 22.x** | Installed on both the workstation and the runner (includes npm) | +| **[Self-hosted GitHub Actions runner](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners)** | Registered in your repository or organization, running in the air-gapped network | +| **Azure connectivity from runner** | The runner must reach your APIM instance's ARM endpoint (network-level only) | +| **GitHub connectivity from runner** | The runner must reach `github.com` (or your [GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@latest/admin/overview/about-github-enterprise-server)) for job dispatch, checkout, and secret injection | +| **File transfer mechanism** | A way to copy the npm cache directory into the air-gapped network | + +--- + +## Step 1 — Pack the CLI + +On the connected workstation: + +```bash +npm pack @peterhauge/apiops-cli +``` + +This produces `peterhauge-apiops-cli-.tgz` in the current directory. + +Commit the tarball into your repository (e.g., under `.apiops/`) so the workflow can reference it by path: + +```bash +mkdir -p .apiops +mv peterhauge-apiops-cli-*.tgz .apiops/ +git add .apiops/peterhauge-apiops-cli-*.tgz +``` + +--- + +## Step 2 — Initialize the Repository + +Pass `--cli-package` so the generated `package.json` references the local tarball instead of the public registry: + +```bash +apiops init \ + --ci github-actions \ + --environments dev,prod \ + --cli-package ./.apiops/peterhauge-apiops-cli-.tgz \ + --non-interactive +``` + +This command generates: + +| File | Purpose | +|------|---------| +| `package.json` | Declares the CLI as a `file:` dependency pointing at the tarball | +| `.github/workflows/run-extractor.yaml` | Extract workflow | +| `.github/workflows/run-publisher.yaml` | Publish workflow | +| `configuration.*.yaml` | Override templates | + +--- + +## Step 3 — Generate the Lock File and Pre-Stage the npm Cache + +On the **connected workstation**, run: + +```bash +npm install # creates package-lock.json +npm ci # populates ~/.npm/_cacache/ with every package the lock file references +``` + +Commit `package-lock.json` to the repository. The command `npm ci --offline` requires it. + +Then transfer the npm cache to the runner: + +```bash +# On the workstation +tar -czf npm-cacache.tar.gz -C ~/.npm _cacache + +# Transfer npm-cacache.tar.gz into the air-gapped network, then on the runner: +mkdir -p ~/.npm +tar -xzf npm-cacache.tar.gz -C ~/.npm +``` + +> Repeat this cache transfer every time dependencies change (CLI upgrade, new package, etc.). + +--- + +## Step 4 — Configure the Self-Hosted Runner + +Install and register the runner in the air-gapped network per the [self-hosted runner documentation](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners). + +Verify the following: + +1. **Node.js 22.x** is installed and on `PATH` +2. **npm cache is pre-staged** at the runner service account's `~/.npm/_cacache/` (see Step 3) +3. **Network access to Azure ARM** — the runner must reach `management.azure.com` (or [sovereign cloud equivalent](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud)) +4. **Network access to GitHub** — the runner must reach `github.com` (or your GHES instance) for job dispatch, `actions/checkout`, and secret injection +5. **Git** is installed (required by `actions/checkout`) + +> **Runner labels:** Add a custom label (e.g., `air-gapped`) when registering the runner so workflows can target it via `runs-on: [self-hosted, air-gapped]`. + +--- + +## Step 5 — Modify Workflows for Air-Gapped Operation + +The generated workflows need three edits per job: + +| Edit | What to Change | +|------|----------------| +| **Runner target** | Replace `runs-on: ubuntu-latest` with `runs-on: [self-hosted, air-gapped]` | +| **Remove `setup-node`** | Delete the `actions/setup-node@v4` step (Node.js is pre-installed on the runner) | +| **Use offline `npm ci`** | Change `npm ci` to `npm ci --offline` so npm resolves entirely from the pre-staged cache | + +Example `.github/workflows/run-extractor.yaml`: + +```yaml +jobs: + extract: + runs-on: [self-hosted, air-gapped] + steps: + - uses: actions/checkout@v4 + + - name: Install dependencies (offline) + run: npm ci --offline + + - name: Run extract + env: + AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }} + AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }} + AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }} + run: | + npx apiops extract \ + --resource-group ${{ secrets.APIM_RESOURCE_GROUP }} \ + --service-name ${{ secrets.APIM_SERVICE_NAME }} \ + --subscription-id ${{ secrets.AZURE_SUBSCRIPTION_ID }} \ + --output ./apim-artifacts +``` + +The `--offline` flag makes npm fail fast if any package is missing from the cache instead of attempting a network call. + +> **Authentication to Azure (recommended order):** +> +> 1. **Managed identity (preferred)** — if your self-hosted runner is an Azure VM, attach a [user-assigned or system-assigned managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview), grant it APIM roles, and call `azure/login@v3` with `auth-type: IDENTITY`. No secrets to store, no internet egress required. See [Login With User-assigned Managed Identity](https://github.com/Azure/login#login-with-user-assigned-managed-identity) in the `azure/login` README. +> 2. **OIDC federation** — requires the runner to reach `token.actions.githubusercontent.com`. Use this when the runner is not on Azure compute but can still reach that endpoint. +> 3. **Service principal secret (fallback)** — when neither managed identity nor OIDC is viable, supply `AZURE_CLIENT_ID`/`AZURE_CLIENT_SECRET`/`AZURE_TENANT_ID` from repository secrets. +> +> See the [authentication guide](../guides/authentication.md) for full details. + +--- + +## Step 6 — Configure Repository Secrets + +Configure the secrets your workflows reference under **Settings → Secrets and variables → Actions**: + +| Secret | Purpose | +|--------|---------| +| `AZURE_SUBSCRIPTION_ID` | Target subscription | +| `AZURE_TENANT_ID` | Entra ID tenant | +| `AZURE_CLIENT_ID` | Service principal app ID (if not using OIDC) | +| `AZURE_CLIENT_SECRET` | Service principal secret (if not using OIDC) | +| `APIM_RESOURCE_GROUP`, `APIM_SERVICE_NAME` | Per-environment APIM identifiers | + +Use [environment-scoped secrets](https://docs.github.com/en/actions/deployment/targeting-different-environments/managing-environments-for-deployment) for per-environment values (`dev`, `prod`). + +--- + +## Step 7 — Commit and Validate + +```bash +git add . +git commit -m "feat: air-gapped apiops setup with offline tarball" +git push +``` + +Trigger the extract workflow manually from **Actions → Run workflow** and verify: + +1. `npm ci --offline` completes with no network calls +2. `apiops extract` authenticates and runs successfully + +> **✅ Setup complete.** Your air-gapped apiops workflows are now operational. The remaining sections cover ongoing maintenance and reference material — read them as needed. + +--- + +## Upgrading the CLI Version + +1. On a connected workstation, run `npm pack @peterhauge/apiops-cli` for the new version +2. Replace `.apiops/peterhauge-apiops-cli-*.tgz` with the new tarball and update the `file:` path in `package.json` +3. Regenerate `package-lock.json` (`npm install`) +4. Re-populate and re-transfer the npm cache (`npm ci` on the workstation, then copy `~/.npm/_cacache/`) +5. Commit the tarball and updated lock file + +--- + +## Troubleshooting + +| Problem | Cause | Fix | +|---------|-------|-----| +| `npm ci` fails with `ENOTCACHED` | npm cache missing one or more packages | Re-run `npm ci` on the connected workstation and re-transfer `~/.npm/_cacache/` | +| `npm ci` fails with "lockfile mismatch" | `package-lock.json` out of sync with `package.json` | Re-run `npm install` on connected workstation, commit updated lock file, refresh cache | +| `npm install` complains about missing tarball | Path in `package.json` doesn't match the file on disk | Verify the `file:` reference matches the committed `.tgz` filename exactly | +| `npx apiops` not found | `npm ci --offline` didn't complete or `.bin` not in PATH | Verify `node_modules/.bin/apiops` exists after install | +| Azure auth fails | Runner can't reach Entra ID or ARM endpoint | Verify network allows traffic to `login.microsoftonline.com` and `management.azure.com` (or sovereign equivalents) | +| `actions/checkout` fails | Runner can't reach GitHub API | Ensure runner has network path to `github.com` (or your GHES instance) | +| OIDC token request fails | Runner blocked from `token.actions.githubusercontent.com` | Switch to service principal credentials in repository secrets | +| Runner not picking up jobs | Label mismatch or runner offline | Confirm `runs-on` labels match the registered runner | + +--- + +## Further Reading + +- [Local npm Registry walkthrough](air-gapped-github-actions-local-registry.md) — recommended when GitHub Packages on GHES (or another internal feed) is available +- [apiops init reference](../commands/init.md) — full `--cli-package` documentation +- [GitHub Actions integration](../ci-cd/github-actions.md) — standard (connected) setup +- [Authentication guide](../guides/authentication.md) — service principal and managed identity options +- [Air-gapped setup: Azure DevOps](air-gapped-azure-devops.md) +- [Self-hosted runners](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners) — runner installation and configuration +- [GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@latest/admin/overview/about-github-enterprise-server) — on-premises GitHub +- [National cloud endpoints](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud) — sovereign cloud identity configuration +- [Entra ID authentication endpoints](https://learn.microsoft.com/en-us/azure/developer/identity/national-cloud#azure-ad-authentication-endpoints) — per-cloud token acquisition endpoints diff --git a/docs/walkthrough/air-gapped-github-actions.md b/docs/walkthrough/air-gapped-github-actions.md index f9bb385..e04a0a6 100644 --- a/docs/walkthrough/air-gapped-github-actions.md +++ b/docs/walkthrough/air-gapped-github-actions.md @@ -1,259 +1,23 @@ # Air-Gapped Setup: GitHub Actions -Deploy APIM configuration using apiops-cli on GitHub Actions runners with **no internet access** at runtime. This walkthrough covers preparing the npm tarball, lock file, and self-hosted runner configuration so that `npm ci` succeeds entirely offline. +Deploy APIM configuration using apiops-cli on [self-hosted GitHub Actions runners](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners) with **no internet access** at runtime. There are two supported approaches — pick the one that matches your environment and follow that walkthrough end-to-end. --- -## When to Use This Guide +## Choose Your Approach -- Self-hosted runners in a private network with no outbound internet -- Environments requiring artifact pre-staging for security compliance -- Corporate networks that block access to the npm registry +| Approach | Best For | Walkthrough | +|----------|----------|-------------| +| **Local npm Registry** (recommended) | You run [GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@latest/admin/overview/about-github-enterprise-server) with [GitHub Packages](https://docs.github.com/en/enterprise-server@latest/packages/working-with-a-github-packages-registry/working-with-the-npm-registry), or any other internal npm registry reachable from the runner (Verdaccio, Artifactory, Nexus). Packages flow through the registry; only the registry needs controlled connectivity to refresh. | [Local npm Registry walkthrough](air-gapped-github-actions-local-registry.md) | +| **Offline Tarball** | No internal npm registry is available. The CLI ships as a `.tgz` in your repo and the npm cache is transferred manually to the runner. Requires re-staging on every dependency change. | [Offline Tarball walkthrough](air-gapped-github-actions-offline-tarball.md) | ---- - -## Architecture Overview - -```mermaid -flowchart LR - subgraph Connected Workstation - A[npm pack] --> B[apiops tarball .tgz] - C[npm install] --> D[package-lock.json] - end - subgraph Transfer - B -->|Approved media / artifact feed| E[Self-hosted Runner] - D -->|Same transfer| E - end - subgraph Air-Gapped Runner - E --> F[npm ci --offline] - F --> G[apiops extract / publish] - end -``` - ---- - -## Prerequisites - -| Requirement | Details | -|-------------|---------| -| **Connected workstation** | A machine with internet access to download packages | -| **Node.js 22.x** | Installed on both the workstation and the runner | -| **npm 10+** | Comes with Node.js 22 | -| **Self-hosted GitHub Actions runner** | Registered in your repository, running in the air-gapped network | -| **Azure connectivity from runner** | The runner must reach your APIM instance's ARM endpoint (network-level, not npm) | -| **Transfer mechanism** | USB drive, internal artifact feed, or approved file share for moving packages into the restricted zone | - ---- - -## Step 1 — Prepare the Tarball (Connected Workstation) - -On a machine with internet access: - -```bash -# Install the CLI globally to get access to the apiops command -npm install -g @peterhauge/apiops-cli - -# Pack the installed package into a tarball -npm pack @peterhauge/apiops-cli -# Produces: peterhauge-apiops-cli-.tgz -``` - -Keep the `.tgz` file — you'll transfer it to the air-gapped environment. - ---- - -## Step 2 — Scaffold the Repository - -Run `apiops init` with the `--cli-package` flag pointing to the tarball: - -```bash -apiops init \ - --ci github-actions \ - --cli-package ./peterhauge-apiops-cli-0.1.5-alpha.1.tgz \ - --environments dev,prod \ - --non-interactive -``` - -This generates: - -| File | Purpose | -|------|---------| -| `package.json` | References the tarball as `"file:.apiops/peterhauge-apiops-cli-0.1.5-alpha.1.tgz"` | -| `.apiops/peterhauge-apiops-cli-0.1.5-alpha.1.tgz` | Local copy of the CLI package | -| `.github/workflows/run-extractor.yaml` | Extract workflow | -| `.github/workflows/run-publisher.yaml` | Publish workflow | -| `configuration.*.yaml` | Override templates | - ---- - -## Step 3 — Generate the Lock File - -The lock file pins every transitive dependency so `npm ci` can resolve them offline: - -```bash -# In the scaffolded repo directory -npm install -``` - -This creates `package-lock.json`. Commit it — the lock file is **required** for `npm ci` to work. - ---- - -## Step 4 — Cache Dependencies for Offline Install - -`npm ci --offline` requires a populated npm cache. On your connected workstation, populate the cache: - -```bash -# Clean the npm cache to start fresh (optional but recommended) -npm cache clean --force - -# Install to populate the cache with all resolved packages -npm ci - -# Export the cache directory -npm config get cache -# Default: ~/.npm -``` - -Copy the entire npm cache directory (or the `_cacache` subfolder) to your transfer media. - -> **Alternative — vendored `node_modules`:** If your transfer policy allows it, you can commit or transfer the entire `node_modules` directory. In that case, skip `npm ci` in the workflow and run `npx apiops` directly. - ---- - -## Step 5 — Transfer Artifacts to the Air-Gapped Zone - -Move these items through your approved transfer channel: - -| Artifact | Destination on Runner | -|----------|-----------------------| -| Repository clone (with `.apiops/` tarball, `package.json`, `package-lock.json`) | Runner's `$GITHUB_WORKSPACE` (handled by `actions/checkout`) | -| npm cache (`_cacache/`) | `~/.npm/_cacache/` on the runner | - -If using a vendored `node_modules`, transfer it as part of the repository. - ---- - -## Step 6 — Configure the Self-Hosted Runner - -Install and register the runner in the air-gapped network: - -```bash -# On the runner machine -mkdir actions-runner && cd actions-runner -# Transfer the runner package via approved media -tar xzf actions-runner-linux-x64-*.tar.gz -./config.sh --url https://github.com// --token -./svc.sh install && ./svc.sh start -``` - -Ensure: - -1. **Node.js 22.x** is installed and on `PATH` -2. **npm cache is pre-populated** at `~/.npm` (from Step 4) -3. **Network access to Azure ARM** — the runner must reach `management.azure.com` (or sovereign equivalent) for APIM API calls -4. **Git** is installed (required by `actions/checkout`) - ---- - -## Step 7 — Modify Workflows for Offline Operation - -Edit the generated workflows to use offline npm and target your self-hosted runner. - -### `.github/workflows/run-extractor.yaml` - -```yaml -jobs: - extract: - runs-on: [self-hosted, air-gapped] # ← target your runner labels - steps: - - uses: actions/checkout@v4 - - # Skip actions/setup-node — Node.js is pre-installed on the runner - - - name: Install dependencies (offline) - run: npm ci --offline - - - name: Run extract - run: | - npx apiops extract \ - --resource-group ${{ secrets.APIM_RESOURCE_GROUP }} \ - --service-name ${{ secrets.APIM_SERVICE_NAME }} \ - --subscription-id ${{ secrets.AZURE_SUBSCRIPTION_ID }} \ - --output ./apim-artifacts -``` - -### `.github/workflows/run-publisher.yaml` - -```yaml -jobs: - publish-dev: - runs-on: [self-hosted, air-gapped] - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 2 # needed for incremental publish - - - name: Install dependencies (offline) - run: npm ci --offline - - - name: Publish (incremental) - run: | - npx apiops publish \ - --resource-group ${{ secrets.APIM_RESOURCE_GROUP_DEV }} \ - --service-name ${{ secrets.APIM_SERVICE_NAME_DEV }} \ - --subscription-id ${{ secrets.AZURE_SUBSCRIPTION_ID }} \ - --source ./apim-artifacts \ - --commit-id ${{ github.sha }} -``` - -> **Authentication:** In air-gapped environments, OIDC federation may not be available if the runner can't reach `token.actions.githubusercontent.com`. Use a service principal with `--client-id`, `--client-secret`, and `--tenant-id` flags (or equivalent environment variables) instead. Credentials can be stored in GitHub repository secrets — the runner only needs to reach GitHub's API for secret injection during job startup. - ---- - -## Step 8 — Commit and Validate - -```bash -git add . -git commit -m "feat: air-gapped apiops setup with local tarball" -git push -``` - -Trigger the workflow manually from **Actions → Run workflow** and verify: - -1. `npm ci --offline` completes without network calls -2. `apiops extract` or `apiops publish` authenticates and runs successfully - ---- - -## Upgrading the CLI Version - -When a new version is released: - -1. On the connected workstation: `npm pack @peterhauge/apiops-cli` (new version) -2. Replace `.apiops/*.tgz` in the repository -3. Update `package.json` to reference the new tarball filename -4. Run `npm install` to regenerate `package-lock.json` -5. Rebuild the npm cache (`npm ci`) and transfer the updated cache -6. Commit and push - ---- - -## Troubleshooting - -| Problem | Cause | Fix | -|---------|-------|-----| -| `npm ci` fails with `ENOTCACHED` | npm cache doesn't contain required packages | Re-populate cache on connected workstation and transfer | -| `npm ci` fails with "lockfile mismatch" | `package-lock.json` out of sync with `package.json` | Re-run `npm install` on connected workstation, commit updated lock file | -| `npx apiops` not found | `npm ci` didn't complete or `.bin` not in PATH | Verify `node_modules/.bin/apiops` exists after install | -| Azure auth fails | Runner can't reach Entra ID or ARM endpoint | Verify network allows traffic to `login.microsoftonline.com` and `management.azure.com` | -| `actions/checkout` fails | Runner can't reach GitHub API | Ensure runner has network path to `github.com` (or use GHES with local clone) | +> **GitHub.com SaaS is not an air-gapped registry.** `npm.pkg.github.com` is an internet service. The Local npm Registry approach assumes [GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@latest/admin/overview/about-github-enterprise-server) running inside your network (or another self-hosted npm registry). --- ## Related -- [apiops init reference](../commands/init.md) — all `--cli-package` details -- [GitHub Actions integration](../ci-cd/github-actions.md) — standard (connected) setup -- [Authentication guide](../guides/authentication.md) — service principal and managed identity options - [Air-gapped setup: Azure DevOps](air-gapped-azure-devops.md) +- [GitHub Actions integration](../ci-cd/github-actions.md) — standard (connected) setup +- [apiops init reference](../commands/init.md) +- [Authentication guide](../guides/authentication.md) From 46935bd2387f47070d41fb6457da10eea211c35b Mon Sep 17 00:00:00 2001 From: Elizabeth Maher Date: Thu, 21 May 2026 12:22:24 -0700 Subject: [PATCH 7/8] reverting changes to squad --- .squad/agents/docwriter/history.md | 34 ------------------------------ .squad/decisions.md | 18 ---------------- 2 files changed, 52 deletions(-) diff --git a/.squad/agents/docwriter/history.md b/.squad/agents/docwriter/history.md index d8c7be8..b78bf5d 100644 --- a/.squad/agents/docwriter/history.md +++ b/.squad/agents/docwriter/history.md @@ -16,22 +16,6 @@ Documentation authoring (2026-04-30 to 2026-05-17): 3-phase plan with 28 user-fa -### 2026-06-11: Air-Gapped Walkthroughs — GitHub Actions and Azure DevOps - -**Context:** Authored two walkthrough documents for using apiops-cli in air-gapped (network-restricted) environments. - -**Files Created:** -1. `docs/walkthrough/air-gapped-github-actions.md` — 8-step guide covering tarball preparation, `apiops init --cli-package`, lock file generation, npm cache transfer, self-hosted runner setup, workflow modifications for `npm ci --offline`, and upgrade procedures. -2. `docs/walkthrough/air-gapped-azure-devops.md` — 9-step guide with same core pattern plus Azure Artifacts feed as alternative to manual cache transfer, `npmAuthenticate@0` task usage, agent pool configuration, and AzureCLI@2 service connection authentication. - -**Files Updated:** -- `docs/README.md` — Added walkthrough section to Quick Links table and directory tree. - -**Learnings:** -- Air-gapped setups rely on `--cli-package` flag for local tarball mode, which generates `package.json` with `"file:.apiops/{tarball}"` dependency. The lock file is critical — without it, `npm ci` fails. -- Azure DevOps has a natural advantage for air-gapped setups via Azure Artifacts feeds with upstream sources (controlled sync windows). GitHub Actions environments must rely on pre-populated npm cache or vendored node_modules. -- Authentication differs: GitHub Actions in air-gapped may lose OIDC (can't reach token.actions.githubusercontent.com), requiring fallback to service principal secrets. Azure DevOps service connections work regardless since the agent-to-DevOps connectivity handles token exchange. - ### 2026-05-17: Phase 2 Docs — Azure DevOps, Filtering, Artifact Format, Config Reference, Glossary **Context:** Authored five Phase 2 documentation files completing CI/CD coverage, resource filtering guide, artifact format reference, configuration reference, and APIM glossary. @@ -131,21 +115,3 @@ Documentation authoring (2026-04-30 to 2026-05-17): 3-phase plan with 28 user-fa **Patterns:** Examples-first, Mermaid workflows, relative links, search-optimized errors, progressive disclosure **Gotchas:** Auth flags set env vars (credential precedence). Overrides: names consistent, properties differ. `--commit-id`/`--delete-unmatched` exclusive. - -### 2026-06-11: Air-Gapped Azure DevOps — PR Feedback Revision - -**Context:** Revised `docs/walkthrough/air-gapped-azure-devops.md` based on PR review comments. - -**Changes:** -- Made local npm registry (Azure Artifacts) the primary/default approach; tarball flow demoted to fallback section. -- Removed redundant "npm 10+" prerequisite (implied by Node.js 22.x). -- Removed "transfer mechanism" prerequisite row (not needed when local registry is primary). -- Added official Microsoft docs links: self-hosted agents, Azure Artifacts npm feeds, Azure DevOps Server on-prem, sovereign cloud identity endpoints. -- Replaced full embedded pipeline YAML with concise edit table referencing generated files (`pipelines/run-extractor.yaml`, `pipelines/run-publisher.yaml`). -- Added "Sovereign Clouds and On-Premises" section with doc links. -- Simplified architecture diagram to reflect local registry flow. - -**Learnings:** -- PR reviewers strongly prefer docs that match the air-gapped reality: local registries over manual transfers. Azure Artifacts is a natural fit since it's bundled with Azure DevOps Server. -- Large embedded YAML examples are fragile — they drift from generated templates. Better to describe minimal edits to the generated files. -- Always link to official Microsoft docs for infrastructure setup (agents, feeds, sovereign clouds) rather than inlining instructions that will go stale. diff --git a/.squad/decisions.md b/.squad/decisions.md index 3aabea4..023dfe4 100644 --- a/.squad/decisions.md +++ b/.squad/decisions.md @@ -2,24 +2,6 @@ ## Active Decisions -### 2026-06-11T22:34:04Z: Air-Gapped Walkthrough Documentation Pattern -**By:** DocWriter -**Status:** Proposed -**What:** Created a new `docs/walkthrough/` directory for scenario-based step-by-step guides that are more detailed than the standard CI/CD integration docs. First entries cover air-gapped deployments for both GitHub Actions and Azure DevOps. - -**Decision:** Establish walkthrough pattern for complex multi-step deployment scenarios that don't fit cleanly into standard integration guides. - -**Pattern:** -- **`docs/walkthrough/`** = scenario-based, multi-step guides for specific deployment patterns -- **`docs/ci-cd/`** = platform integration reference (standard setup) -- **`docs/guides/`** = feature-oriented how-to guides - -**Scope:** Future walkthrough candidates include multi-region APIM, sovereign cloud deployments, monorepo with multiple APIM instances. Any scenario requiring 5+ steps beyond the standard setup qualifies. - -**Next Steps:** Team review; if approved, publish guides and update docs/README index. - ---- - ### 2026-05-14T05:20:00Z: APIM v1 → v2 SKU Migration via apiops-cli **By:** ApimExpert + ApiOpsLead (joint research and decision) **Status:** Proposed for team governance review From c8937ccf174212c655d2edcacc0423f6142cdedb Mon Sep 17 00:00:00 2001 From: Elizabeth Maher Date: Thu, 21 May 2026 12:25:29 -0700 Subject: [PATCH 8/8] modifying doc tree --- docs/README.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/docs/README.md b/docs/README.md index 7f41ecb..9bbfaf2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -80,12 +80,8 @@ docs/ │ ├── overview.md — System design overview │ └── design-principles.md — Architecture principles ├── walkthrough/ -│ ├── air-gapped-github-actions.md — Air-gapped GitHub Actions (chooser) -│ ├── air-gapped-github-actions-local-registry.md — GitHub Actions via internal npm registry (GHES Packages) -│ ├── air-gapped-github-actions-offline-tarball.md — GitHub Actions via offline tarball + npm cache -│ ├── air-gapped-azure-devops.md — Air-gapped Azure DevOps (chooser) -│ ├── air-gapped-azure-devops-local-registry.md — Azure DevOps via Azure Artifacts feed -│ └── air-gapped-azure-devops-offline-tarball.md — Azure DevOps via offline tarball + npm cache +│ ├── air-gapped-github-actions.md — Using `apiops` in air-gapped CI/CD (GitHub Actions) +│ ├── air-gapped-azure-devops.md — Using `apiops` in air-gapped CI/CD (Azure DevOps) └── troubleshooting/ ├── common-errors.md — Error messages and solutions ├── debugging-guide.md — Debugging with --log-level