test: add e2e test for OpenCensus to OpenTelemetry metrics migration by khrm · Pull Request #2811 · tektoncd/pipelines-as-code

khrm · 2026-06-29T12:51:00Z

Adds TestOTelMetrics, a consolidated e2e test for the OC→OTel metrics migration in Pipelines-as-Code (PR #2567). The test scrapes two pods:

Controller (app.kubernetes.io/name=controller):

Asserts http_client_* metrics from knative k8s client OTel instrumentation
Asserts go_* runtime metrics
Checks PAC application metrics (pipelines_as_code_*) are absent/present
Asserts old OC metric names are absent

Watcher (app.kubernetes.io/name=watcher):

Asserts kn_workqueue_* metrics (watcher uses knative reconciler)
Asserts go_* runtime metrics

Verified locally with PAC controller and watcher deployed to kind via ko.

Relates to #2567

📝 Description of the Change

🔗 Linked GitHub Issue

Fixes #

🧪 Testing Strategy

🤖 AI Assistance

AI assistance can be used for various tasks, such as code generation,
documentation, or testing.

Please indicate whether you have used AI assistance
for this PR and provide details if applicable.

I have not used any AI assistance for this PR.
I have used AI assistance for this PR.

Important

Slop will be simply rejected, if you are using AI assistance you need to make sure you
understand the code generated and that it meets the project's standards. you
need at least know how to run the code and deploy it (if needed). See
startpaac to make it easy
to deploy and test your code changes.

If the majority of the code in this PR was generated by an AI, please add a Co-authored-by trailer to your commit message.
For example:

Co-authored-by: Claude noreply@anthropic.com

✅ Submitter Checklist

gemini-code-assist

Code Review

This pull request introduces a new end-to-end test (test/metrics_otel_test.go) to verify the OpenCensus-to-OpenTelemetry metrics migration in Pipelines-as-Code by scraping the controller and watcher pods' metrics endpoints. The review feedback focuses on improving test robustness and efficiency: it recommends refactoring the metrics scraping helper to return errors instead of calling t.Fatalf (which would prematurely abort the test on transient errors during polling), reusing a single pre-initialized Kubernetes client across helper functions, and adding the necessary fmt import.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

codecov · 2026-06-29T12:57:09Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.62%. Comparing base (2a84139) to head (4ce4736).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2811   +/-   ##
=======================================
  Coverage   61.62%   61.62%           
=======================================
  Files         213      213           
  Lines       17512    17512           
=======================================
  Hits        10792    10792           
  Misses       5901     5901           
  Partials      819      819

Flag	Coverage Δ
unit-tests	`61.62% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

khrm · 2026-06-29T13:00:31Z

//gemini help

gemini-code-assist · 2026-06-29T13:01:08Z

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

khrm · 2026-06-29T13:01:54Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a new E2E test file test/metrics_otel_test.go to verify the OpenCensus-to-OpenTelemetry metrics migration in Pipelines-as-Code by scraping and asserting metrics from the controller and watcher pods. The feedback recommends parameterizing the waitForPACMetric helper function with a labelSelector parameter. This allows reusing the helper to poll and wait for the watcher pod's metrics, resolving potential test flakiness caused by scraping the watcher metrics only once without retries.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

chmouel · 2026-06-29T13:35:11Z

+
+	// ========== Infrastructure metric assertions (OTel renames) ==========
+
+	t.Run("Renames/k8s_client_uses_http_client_prefix", func(t *testing.T) {


plesae use testcase as per convention on this repo insteda of t.Run...

theakshaypant · 2026-06-30T14:13:09Z

+// and asserts that pipelines_as_code_pipelinerun_count_total increments by
+// exactly 1. The test skips automatically if Gitea is not configured
+// (TEST_GITEA_API_URL and related env vars are unset).
+func TestOthersOTelMetricsAfterPACRun(t *testing.T) {


This e2e test is failing in the CI
=== RUN TestOthersOTelMetricsAfterPACRun {"level":"info","ts":1782788949.960166,"caller":"gitea/gitea.go:290","msg":"gitea: initialized API client with provided credentials user=pac providerURL=http://localhost:3000"} {"level":"info","ts":1782788949.964509,"caller":"cctx/ctx.go:19","msg":"Found pipelines-as-code installation in namespace pipelines-as-code"} {"level":"info","ts":1782788949.9646285,"caller":"cctx/ctx.go:22","msg":"Pipelines as Code Controller: &{Name:default Configmap:pipelines-as-code Secret:pipelines-as-code-secret GlobalRepository:pipelines-as-code}"} {"level":"info","ts":1782788949.9674673,"caller":"repository/create.go:17","msg":"Namespace pac-e2e-test-2w5pv created"} {"level":"info","ts":1782788949.9676106,"caller":"gitea/scm.go:151","msg":"Creating gitea repository pac-e2e-test-2w5pv for user pac"} {"level":"info","ts":1782788950.389976,"caller":"gitea/scm.go:163","msg":"Creating webhook to smee url on gitea repository pac-e2e-test-2w5pv"} {"level":"info","ts":1782788950.909148,"caller":"repository/create.go:26","msg":"PipelinesAsCode Repository pac-e2e-test-2w5pv has been created in namespace pac-e2e-test-2w5pv"} {"level":"info","ts":1782788951.527173,"caller":"scm/scm.go:48","msg":"Pushed files to repo https://gitea.paac-127-0-0-1.nip.io/pac/pac-e2e-test-2w5pv branch pac-e2e-test-2w5pv"} {"level":"info","ts":1782788956.5320754,"caller":"gitea/test.go:254","msg":"Creating PullRequest"} {"level":"info","ts":1782788957.0262525,"caller":"gitea/test.go:269","msg":"PullRequest https://gitea.paac-127-0-0-1.nip.io/pac/pac-e2e-test-2w5pv/pulls/1 has been created"} {"level":"info","ts":1782788957.17538,"caller":"gitea/test.go:477","msg":"Number of gitea status on PR: 0/1"} {"level":"info","ts":1782788962.3255165,"caller":"gitea/test.go:474","msg":"Status on SHA: pac-e2e-test-2w5pv is success from Pipelines as Code CI / always-good-pipelinerun"} {"level":"info","ts":1782788962.325611,"caller":"gitea/test.go:477","msg":"Number of gitea status on PR: 1/1"} {"level":"info","ts":1782788962.3256638,"caller":"gitea/test.go:542","msg":"Looking for regexp \".*Pipelines as Code CI.*has.*successfully.*validated your commit.*\" in PR comments"} {"level":"info","ts":1782788962.4651678,"caller":"gitea/test.go:548","msg":"Found regexp in comment: Pipelines as Code CI/always-good-pipelinerun-z9rnx has successfully validated your commit.\n
\n
Namespace: pac-e2e-test-2w5pv
\n
PipelineRun: always-good-pipelinerun-z9rnx
\n
\n
\n
Task Statuses:
\n\n\n \n\n\n\n
Status Duration Name
Succeeded 3 seconds \n\n[The Task name is Task](http://dashboard.paac-127-0-0-1.nip.io/#/namespaces/pac-e2e-test-2w5pv/pipelineruns/always-good-pipelinerun-z9rnx?pipelineTask=task)\n\n
"} metrics_otel_test.go:314: Timed out waiting for metric "pipelines_as_code_pipelinerun_count_total" (selector=app.kubernetes.io/name=controller,app.kubernetes.io/part-of=pipelines-as-code, waited 2m0s): context deadline exceeded setup.go:87: Deleted gitea repo pac/pac-e2e-test-2w5pv --- FAIL: TestOthersOTelMetricsAfterPACRun (132.88s) FAIL FAIL github.com/openshift-pipelines/pipelines-as-code/test 1007.891s

zakisk · 2026-06-30T15:10:02Z

+	if err != nil {
+		t.Skipf("PAC controller metrics not reachable, skipping: %v", err)
+	}
+	baseCount := counterValue(baseline, "pipelines_as_code_pipelinerun_count_total")


@khrm actually it's pipelines_as_code_pipelinerun_count not pipelines_as_code_pipelinerun_count_total that's why the test is failing

The OTEL exporter should add total suffix. Need to check why it's not happening?

Actually, I was scapping from controller instead of watcher.

theakshaypant

@khrm You should try running the e2e on a local setup for validation.
Refs 1, 2

Moving this PR to draft while you resolve these issues.

Adds TestOTelMetrics, a consolidated e2e test for the OC→OTel metrics migration in Pipelines-as-Code (PR tektoncd#2567). The test scrapes two pods: Controller (app.kubernetes.io/name=controller): - Asserts http_client_* metrics from knative k8s client OTel instrumentation - Asserts go_* runtime metrics - Checks PAC application metrics (pipelines_as_code_*) are absent/present - Asserts old OC metric names are absent Watcher (app.kubernetes.io/name=watcher): - Asserts kn_workqueue_* metrics (watcher uses knative reconciler) - Asserts go_* runtime metrics Verified locally with PAC controller and watcher deployed to kind via ko. Relates to tektoncd#2567 Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

theakshaypant

Looks good, overall.
Green CI is a very positive sign. Got a couple minor comments.

theakshaypant · 2026-07-10T07:07:15Z

+// exactly 1 on the watcher pod. The test skips automatically if Gitea is not
+// configured (TEST_GITEA_API_URL and related env vars are unset).


The test skips automatically if Gitea is not // configured

Is that correct? I believe the other tests fail if the env vars are not set.

theakshaypant · 2026-07-10T07:10:41Z

+	// Old OC metric names must be absent.
+	// TODO: Remove in a future release once no OC-based release is supported.
+	for name := range families {
+		for _, prefix := range []string{"pipelines_as_code/", "tekton_pipelines_as_code_"} {
+			if strings.HasPrefix(name, prefix) {
+				t.Errorf("Old OC metric %q still present; expected removal after OTel migration", name)
+			}
+		}
+	}


Should the same check be present for watcher metrics test as well?

chmouel

I do not know much about metrics, but this LGTM to me, please see comments of other reviewers tho

gemini-code-assist Bot reviewed Jun 29, 2026

View reviewed changes

Comment thread test/metrics_otel_test.go

Comment thread test/metrics_otel_test.go Outdated

Comment thread test/metrics_otel_test.go Outdated

Comment thread test/metrics_otel_test.go Outdated

Comment thread test/metrics_otel_test.go Outdated

khrm force-pushed the e2e-otel-metrics-test branch from d4ec0f5 to 7c391cc Compare June 29, 2026 12:57

gemini-code-assist Bot reviewed Jun 29, 2026

View reviewed changes

Comment thread test/metrics_otel_test.go Outdated

Comment thread test/metrics_otel_test.go Outdated

Comment thread test/metrics_otel_test.go Outdated

khrm force-pushed the e2e-otel-metrics-test branch from 7c391cc to 16c8b75 Compare June 29, 2026 13:12

chmouel reviewed Jun 29, 2026

View reviewed changes

khrm force-pushed the e2e-otel-metrics-test branch 8 times, most recently from 317d87a to da6459d Compare June 29, 2026 14:19

zakisk force-pushed the e2e-otel-metrics-test branch from da6459d to 3337cc1 Compare June 29, 2026 15:18

zakisk added the ok-to-test label Jun 29, 2026

tekton-pac-bot removed the ok-to-test label Jun 29, 2026

zakisk reviewed Jun 29, 2026

View reviewed changes

Comment thread test/metrics_otel_test.go Outdated

khrm force-pushed the e2e-otel-metrics-test branch 4 times, most recently from 44bfe89 to b6ca870 Compare June 29, 2026 18:37

zakisk added the ok-to-test label Jun 30, 2026

tekton-pac-bot removed the ok-to-test label Jun 30, 2026

theakshaypant reviewed Jun 30, 2026

View reviewed changes

zakisk reviewed Jun 30, 2026

View reviewed changes

khrm force-pushed the e2e-otel-metrics-test branch from b6ca870 to 258e7e9 Compare June 30, 2026 15:21

khrm force-pushed the e2e-otel-metrics-test branch from 258e7e9 to 731dc35 Compare June 30, 2026 15:38

zakisk added the ok-to-test label Jun 30, 2026

khrm force-pushed the e2e-otel-metrics-test branch from 731dc35 to 91da0b3 Compare June 30, 2026 16:01

zakisk added ok-to-test and removed ok-to-test labels Jun 30, 2026

tekton-pac-bot removed the ok-to-test label Jun 30, 2026

theakshaypant reviewed Jul 3, 2026

View reviewed changes

theakshaypant marked this pull request as draft July 3, 2026 11:50

khrm force-pushed the e2e-otel-metrics-test branch from 2d5d489 to b34fae7 Compare July 8, 2026 14:51

Merge branch 'main' into e2e-otel-metrics-test

4ce4736

khrm marked this pull request as ready for review July 8, 2026 14:56

theakshaypant added the ok-to-test label Jul 8, 2026

tekton-pac-bot removed the ok-to-test label Jul 8, 2026

theakshaypant reviewed Jul 10, 2026

View reviewed changes

chmouel approved these changes Jul 10, 2026

View reviewed changes


		// ========== Infrastructure metric assertions (OTel renames) ==========

		t.Run("Renames/k8s_client_uses_http_client_prefix", func(t *testing.T) {

		// exactly 1 on the watcher pod. The test skips automatically if Gitea is not
		// configured (TEST_GITEA_API_URL and related env vars are unset).

Uh oh!

Conversation

khrm commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Description of the Change

🔗 Linked GitHub Issue

🧪 Testing Strategy

🤖 AI Assistance

✅ Submitter Checklist

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

khrm commented Jun 29, 2026

Uh oh!

gemini-code-assist Bot commented Jun 29, 2026

Uh oh!

khrm commented Jun 29, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chmouel Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

theakshaypant Jun 30, 2026

Choose a reason for hiding this comment

Task Statuses:

Uh oh!

zakisk Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

khrm Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

khrm Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

theakshaypant left a comment

Choose a reason for hiding this comment

Uh oh!

theakshaypant left a comment

Choose a reason for hiding this comment

Uh oh!

theakshaypant Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

theakshaypant Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

chmouel left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

khrm commented Jun 29, 2026 •

edited

Loading

codecov Bot commented Jun 29, 2026 •

edited

Loading