Skip to content

fix(release): quay promotion in Go#5255

Open
deepsm007 wants to merge 1 commit into
openshift:mainfrom
deepsm007:fix/promotion-quay-go
Open

fix(release): quay promotion in Go#5255
deepsm007 wants to merge 1 commit into
openshift:mainfrom
deepsm007:fix/promotion-quay-go

Conversation

@deepsm007

@deepsm007 deepsm007 commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

/hold
/cc @danilo-gemoli

Convert [promotion-quay] from bash promotion pod to Go while keeping the same promotion logic and routing ([promotion] vs [promotion-quay]

Go Implementation of the promotion-quay Promotion Step

This pull request replaces the bash-based promotion-quay promotion pod with a Go-native implementation, while preserving the intentional separation between the general promotion flow and the Quay-specific promotion-quay routing.

What changes for CI operators

  • The promotion-quay step is now executed by Go code (runQuayPromotion) instead of being driven by the generic promotion pod’s embedded shell logic.
  • When consolidated Quay promotion is enabled, the default registry promotion step is skipped so Quay-specific promotion is the only promotion path taken.
  • When build_if_affected is enabled, promotion planning now skips optional tool-image tag promotion unless those images are explicitly required, reducing unnecessary mirroring/tagging work.

Key behavior changes

  • Routing switch (pkg/steps/release/promote.go)
    • PromotionStep now detects the promotion-quay step name and routes execution to runQuayPromotion(...) rather than applying Quay-specific behavior inside the generic promotion pod.
  • Promotion planning skips tool-image tags (build_if_affected)
    • The promotion planning API is extended with a skippedImages set.
    • Tag selection omits tags whose tool images are in skippedImages (unless the tag’s image is also in requiredImages).
  • Generic promotion pod simplification
    • The generic promotion pod no longer contains Quay-specific resolve/tag retry logic; it derives shell behavior solely from computed images/tags/prune images.
  • Quay promotion implemented in Go (pkg/steps/release/quay_promotion.go)
    • Introduces a dedicated Go orchestration for Quay promotion that runs oc commands inside a promotion-quay pod.
    • Implements a multi-phase workflow:
      • float promotions (including prune backup and restore-on-failure behavior),
      • “official imagestream” digest-pinned tagging via Quay proxy resolution,
      • template tag mirroring/tagging with retry semantics.
    • Includes pod lifecycle handling, node selection (amd64 vs arm64), and targeted retry/backoff around oc operations.

Tests and fixtures

  • Adds a focused Quay test suite (pkg/steps/release/quay_promote_test.go) covering promotion JSON generation, float phases, digest-pinned tagging, retry behavior, resolveAndTag error cases, and pod spec generation (amd64 and arm64).
  • Updates promotion planning tests (pkg/steps/release/promote_test.go) to validate skippedImages behavior under build_if_affected, and removes older quay-shell-specific promotion pod fixture coverage.
  • Removes legacy bash-based Kubernetes YAML fixtures for the previous promotion-quay pod and replaces them with a Go-based promotion-quay pod fixture for spec validation.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci openshift-ci Bot requested a review from danilo-gemoli June 16, 2026 15:20
@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 16, 2026
@coderabbitai

coderabbitai Bot commented Jun 16, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 82e22b5f-e8a9-48f3-a71d-da4db53e687e

📥 Commits

Reviewing files that changed from the base of the PR and between 193b133 and 72c356f.

📒 Files selected for processing (10)
  • pkg/defaults/defaults.go
  • pkg/steps/release/promote.go
  • pkg/steps/release/promote_test.go
  • pkg/steps/release/quay_promote_test.go
  • pkg/steps/release/quay_promotion.go
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay.yaml
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_4.12.yaml
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_multiple_tags.yaml
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_non_release_namespace.yaml
  • pkg/steps/release/testdata/zz_fixture_TestQuayPromotionPod.yaml
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • openshift/release (manual)
  • openshift/ci-docs (manual)
  • openshift/release-controller (manual)
  • openshift/ci-chat-bot (manual)
💤 Files with no reviewable changes (4)
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_non_release_namespace.yaml
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_4.12.yaml
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay.yaml
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_multiple_tags.yaml
✅ Files skipped from review due to trivial changes (1)
  • pkg/steps/release/testdata/zz_fixture_TestQuayPromotionPod.yaml
🚧 Files skipped from review as they are similar to previous changes (5)
  • pkg/defaults/defaults.go
  • pkg/steps/release/quay_promote_test.go
  • pkg/steps/release/promote.go
  • pkg/steps/release/promote_test.go
  • pkg/steps/release/quay_promotion.go

📝 Walkthrough

Walkthrough

Implements Go-native Quay promotion orchestration via dedicated quay_promotion.go module replacing shell-script generation in generic pod; threads skippedImages through promotion APIs to exclude build_if_affected tool images; routes PromotionQuayStepName to runQuayPromotion and gates default registry step on ConsolidatedQuayPromotion config; simplifies getPromotionPod by removing Quay-specific logic; adds comprehensive test coverage including mock client and new pod fixture.

Changes

Quay Promotion Refactor and skippedImages Threading

Layer / File(s) Summary
skippedImages API and threading through promotion
pkg/steps/release/promote.go
Adds skippedImages field to promotionStep struct and PromotedTagsOptions; exports WithSkippedImages(...) option; updates PromotionStep function signature to accept skippedImages parameter; threads exclusion through toPromote and PromotedTagsWithRequiredImages, where requiredImages override skipped tags.
Consolidation gate and Quay routing
pkg/defaults/defaults.go, pkg/steps/release/promote.go
defaults.go conditionally appends default PromotionStep only when !ConsolidatedQuayPromotion; promotionStep.run branches to runQuayPromotion(...) for PromotionQuayStepName, bypassing generic promotion pod for Quay-specific handling.
getPromotionPod refactor and generic command generation
pkg/steps/release/promote.go
Removes getResolveAndTagRetryShell helper; simplifies getPromotionPod to derive commands from images, tags, and pruneImages only; refactors pod spec wiring to use shared helpers for node selector, kubeconfig path, and volume mounts/volumes.
quay_promotion.go: data models and entry point
pkg/steps/release/quay_promotion.go
Defines constants, retry/backoff config, quayPromotion JSON payload models, and internal quayPromotionOCClient interface; implements runQuayPromotion (ConfigMap marshaling, pod creation/restart, container readiness polling, executor wiring) and buildQuayPromotion (deterministic float/template/ISTag instruction generation).
quay_promotion.go: float and tag promotion workflows
pkg/steps/release/quay_promotion.go
Implements promote orchestration, promoteFloats multi-phase mirroring (incoming/latch/post flows with existence detection), tagAfterMirror, resolveAndTag with digest retry, restoreFloats, mirrorChunks (chunked best-effort and bounded-retry mirroring), tagWithRetry (3-attempt backoff), and pair-formatting helpers.
quay_promotion.go: OC execution layer and pod lifecycle
pkg/steps/release/quay_promotion.go
Implements pod-exec oc client methods (Mirror, ImageExists with "not found" classification, digest extraction, Tag); quayPromotionPod pod spec construction with node selector and volume mounting; waitForQuayPromotionPod and deleteQuayPromotionPod lifecycle helpers; utilities for node selection, volume/mount setup, registry config path, and digest field parsing.
promote_test.go: updates for skippedImages and Quay removal
pkg/steps/release/promote_test.go
TestToPromote adds skippedImages field and cases for default exclusion with requiredImages override; TestPromotedTags and TestPromotedTagsWithRequiredImages update fixtures from "roger"/"fred" to "ci"/"tools" and expand build_if_affected scenarios; TestGetPromotionPod removes four Quay shell-script entries; TestGetResolveAndTagRetryShell replaced with TestQuayProxyTagFromISKey.
quay_promote_test.go: comprehensive Quay promotion coverage
pkg/steps/release/quay_promote_test.go
Adds recordingQuayPromotionOCClient mock, retry-sleep bypass helper, and full test suites (TestBuildQuayPromotion, TestQuayPromotionRunnerPromoteFloats, TestQuayPromotionRunnerPromote, TestQuayPromotionRunnerResolveAndTag, TestQuayPromotionPod, TestQuayPromotionPodArm64Only).
Test fixtures: new Quay pod and shell-script removal
pkg/steps/release/testdata/zz_fixture_TestQuayPromotionPod.yaml (added); four zz_fixture_TestGetPromotionPod_promotion_quay*.yaml (removed)
Adds new pod manifest fixture with sleep container, volume/env setup, and node selector; removes obsolete YAML fixtures that defined shell-script-based Quay promotion logic.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~65 minutes

Possibly related PRs

  • openshift/ci-tools#5249: Directly modifies the same getPromotionPod shell/retry generation code in pkg/steps/release/promote.go that this PR refactors and removes Quay-specific shell logic from.

Suggested reviewers

  • droslean

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 2 warnings)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error Line 455 in quay_promotion.go logs stderr from oc commands without sanitization: fmt.Errorf("%w: %s", err, msg) where msg comes from oc commands using registry credentials, risking exposure of au... Sanitize stderr/stdout before logging: filter out auth tokens, registry credentials, and internal hostnames, or replace with generic error messages without command output details.
Docstring Coverage ⚠️ Warning Docstring coverage is 24.32% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Go Error Handling ⚠️ Warning The quay_promotion.go file returns errors without context wrapping in 6 places (lines 191, 201, 263, 355, 390, 430) and ignores errors with _ = in 2 places (lines 182, 310) without clear justificat... Wrap bare error returns with fmt.Errorf and %w to add context. Add inline comments justifying ignored best-effort mirror errors (lines 182, 310) per check requirements.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the primary change: converting promotion-quay from bash to Go implementation. It's concise and specific enough for scanning history.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Test Coverage For New Features ✅ Passed New Quay promotion implementation (quay_promotion.go, 591 lines) has 6 comprehensive test functions in quay_promote_test.go; new WithSkippedImages option is tested in promote_test.go with multiple...
Stable And Deterministic Test Names ✅ Passed No Ginkgo tests found in PR; only Go standard library tests with stable, descriptive table-driven test names. Custom check not applicable.
Test Structure And Quality ✅ Passed Tests use standard Go testing package, not Ginkgo. All 6 new test functions follow Go best practices: table-driven structure with single responsibility, proper cleanup via t.Cleanup(), meaningful a...
Microshift Test Compatibility ✅ Passed This PR adds standard Go unit tests (not Ginkgo e2e tests) to the ci-tools repository. The custom check applies only to new Ginkgo e2e tests, which this PR does not add.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No Ginkgo e2e tests are added in this PR. All new tests are standard Go unit tests (using testing.T) in pkg/steps/release/, which is a library package. The check does not apply.
Topology-Aware Scheduling Compatibility ✅ Passed PR adds topology-agnostic single pod with only kubernetes.io/arch nodeSelector; no control-plane targeting, affinity rules, or topology spread constraints that would break SNO/TNF/TNA/HyperShift.
Ote Binary Stdout Contract ✅ Passed This PR modifies ci-operator tool code, not an OTE binary. The OTE Binary Stdout Contract check applies to OpenShift Tests Extension binaries; ci-operator is a separate CI orchestration tool unrela...
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR adds only standard Go unit tests (testing.T) in pkg/steps/release/, not Ginkgo e2e tests. Custom check applies only to new Ginkgo e2e tests.
No-Weak-Crypto ✅ Passed No weak crypto algorithms (MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB), custom crypto implementations, or insecure secret comparisons found in PR changes.
Container-Privileges ✅ Passed No privileged container settings found: no privileged flag, hostPID/Network/IPC, SYS_ADMIN capability, or allowPrivilegeEscalation in new or modified pod/container specs.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci

openshift-ci Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deepsm007

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 16, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
pkg/steps/release/quay_promotion.go (2)

574-586: 💤 Low value

String-based JSON parsing is fragile.

parseImageInfoDigest uses substring matching rather than json.Unmarshal. This works for simple cases but could break if the JSON contains escaped quotes within the digest value (unlikely but possible) or if the output format changes.

Consider using proper JSON unmarshaling for robustness:

Alternative implementation
func parseImageInfoDigest(output string) (string, bool) {
	var info struct {
		Digest string `json:"digest"`
	}
	if err := json.Unmarshal([]byte(output), &info); err != nil || info.Digest == "" {
		return "", false
	}
	return info.Digest, true
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/steps/release/quay_promotion.go` around lines 574 - 586, The
parseImageInfoDigest function uses fragile string-based JSON parsing with
substring matching (looking for the prefix "digest":" and finding the closing
quote) which can break if the JSON contains escaped quotes within the digest
value or if the output format changes. Replace this with proper JSON
unmarshaling by defining a struct with a Digest field tagged with json:"digest",
then use json.Unmarshal to parse the output string into this struct, returning
the Digest field value and a bool indicating success (checking both for
unmarshal errors and empty digest values).

485-494: ⚖️ Poor tradeoff

Missing resource limits and security context on container.

Per coding guidelines, Kubernetes containers should have resource limits (cpu, memory) and security hardening (runAsNonRoot, readOnlyRootFilesystem: false since KUBECACHEDIR writes to /tmp, allowPrivilegeEscalation: false). This pod runs in CI namespaces but should still follow best practices.

Suggested addition
Containers: []coreapi.Container{{
    Name:    quayPromotionContainer,
    Image:   cliImage,
    Command: []string{"sleep", fmt.Sprintf("%d", quayPromotionPodSleepSec)},
    Env: []coreapi.EnvVar{
        {Name: "KUBECONFIG", Value: promotionPodKubeconfigPath},
        {Name: "KUBECACHEDIR", Value: "/tmp/.kube/cache"},
    },
    Resources: coreapi.ResourceRequirements{
        Requests: coreapi.ResourceList{
            coreapi.ResourceCPU:    resource.MustParse("100m"),
            coreapi.ResourceMemory: resource.MustParse("256Mi"),
        },
        Limits: coreapi.ResourceList{
            coreapi.ResourceCPU:    resource.MustParse("500m"),
            coreapi.ResourceMemory: resource.MustParse("512Mi"),
        },
    },
    SecurityContext: &coreapi.SecurityContext{
        AllowPrivilegeEscalation: ptr.To(false),
        RunAsNonRoot:             ptr.To(true),
        Capabilities:             &coreapi.Capabilities{Drop: []coreapi.Capability{"ALL"}},
    },
    VolumeMounts: mounts,
}},
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/steps/release/quay_promotion.go` around lines 485 - 494, Add resource
limits and security hardening to the quayPromotionContainer in the Containers
array. Add a Resources field with cpu and memory requests (100m cpu, 256Mi
memory) and limits (500m cpu, 512Mi memory), and add a SecurityContext field
configured with AllowPrivilegeEscalation set to false, RunAsNonRoot set to true,
and Capabilities with all capabilities dropped. These additions follow
Kubernetes best practices for pod security and resource management.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/steps/release/quay_promotion.go`:
- Around line 273-276: The resolveAndTag method in quayPromotionRunner has a
potential panic because strings.LastIndex returns -1 when the colon substring is
not found in quayProxyTag, and using -1 as a slice bound causes a bounds panic.
Add a defensive check after the strings.LastIndex call to verify the returned
index is not -1; if it is, return an error indicating that quayProxyTag is
malformed (missing the expected colon separator). This ensures invalid input
cannot cause the function to panic.

---

Nitpick comments:
In `@pkg/steps/release/quay_promotion.go`:
- Around line 574-586: The parseImageInfoDigest function uses fragile
string-based JSON parsing with substring matching (looking for the prefix
"digest":" and finding the closing quote) which can break if the JSON contains
escaped quotes within the digest value or if the output format changes. Replace
this with proper JSON unmarshaling by defining a struct with a Digest field
tagged with json:"digest", then use json.Unmarshal to parse the output string
into this struct, returning the Digest field value and a bool indicating success
(checking both for unmarshal errors and empty digest values).
- Around line 485-494: Add resource limits and security hardening to the
quayPromotionContainer in the Containers array. Add a Resources field with cpu
and memory requests (100m cpu, 256Mi memory) and limits (500m cpu, 512Mi
memory), and add a SecurityContext field configured with
AllowPrivilegeEscalation set to false, RunAsNonRoot set to true, and
Capabilities with all capabilities dropped. These additions follow Kubernetes
best practices for pod security and resource management.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: e6aaa0e5-336f-4f00-bab0-1449d7acd2b8

📥 Commits

Reviewing files that changed from the base of the PR and between 4220cce and 6b84945.

📒 Files selected for processing (10)
  • pkg/defaults/defaults.go
  • pkg/steps/release/promote.go
  • pkg/steps/release/promote_test.go
  • pkg/steps/release/quay_promote_test.go
  • pkg/steps/release/quay_promotion.go
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay.yaml
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_4.12.yaml
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_multiple_tags.yaml
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_non_release_namespace.yaml
  • pkg/steps/release/testdata/zz_fixture_TestQuayPromotionPod.yaml
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • openshift/release (manual)
  • openshift/ci-docs (manual)
  • openshift/release-controller (manual)
  • openshift/ci-chat-bot (manual)
💤 Files with no reviewable changes (4)
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_4.12.yaml
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_non_release_namespace.yaml
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay.yaml
  • pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_multiple_tags.yaml

Comment thread pkg/steps/release/quay_promotion.go
@deepsm007 deepsm007 force-pushed the fix/promotion-quay-go branch 2 times, most recently from 71c6b73 to 193b133 Compare June 16, 2026 15:43
@deepsm007 deepsm007 force-pushed the fix/promotion-quay-go branch from 193b133 to 72c356f Compare June 16, 2026 16:41
@deepsm007

Copy link
Copy Markdown
Contributor Author

/test e2e

@deepsm007

Copy link
Copy Markdown
Contributor Author

/test all

@deepsm007

Copy link
Copy Markdown
Contributor Author

/override ci/prow/e2e

@openshift-ci

openshift-ci Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

@deepsm007: Overrode contexts on behalf of deepsm007: ci/prow/e2e

Details

In response to this:

/override ci/prow/e2e

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@deepsm007

Copy link
Copy Markdown
Contributor Author

/test integration

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

@openshift-ci

openshift-ci Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

@deepsm007: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant