[skills] Update CI/test skills for public pipeline + NUnit dotnet test#11670
Draft
simonrozsival wants to merge 3 commits into
Draft
[skills] Update CI/test skills for public pipeline + NUnit dotnet test#11670simonrozsival wants to merge 3 commits into
dotnet test#11670simonrozsival wants to merge 3 commits into
Conversation
Adapt the local agent skills and workflow docs to two recent CI changes: - dotnet#11578 made dnceng-public `dotnet-android` the single PR pipeline (full matrix for every PR) and set `pr: none` on the DevDiv `Xamarin.Android-PR` pipeline. Rework the `ci-status` skill around one public build: build-id extraction, AZDO test queries via `az rest` (ResultsByBuild), queue-driven ETA variance, a gating-vs-flaky verdict (build result + checks are authoritative; `continueOnError` device-test failures are non-gating), and device-test `logcat-*.txt` artifacts. - dotnet#11224 migrated Mono.Android device tests from NUnitLite/`-t:RunTestApp` to stock NUnit via `dotnet test` + Microsoft Testing Platform. Update the `tests` skill, `copilot-instructions.md`, and `UnitTests.md` to the new `-t:Install` + `dotnet test --no-build --report-trx` flow (TRX output), and drop the removed NUnitLite row from the `update-tpn` skill. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apply the skill-creator "concise is key" principle to the ci-status skill: cut content that is obvious to the agent and keep only what is unique to dotnet/android CI. - SKILL.md 306 -> 78 lines (~70% smaller always-loaded body): drop the prerequisites table, PowerShell duplicates, step-by-step arithmetic (elapsed/median/job-counter explanations) and the verbose output template; state the repo-specific facts once (single public pipeline, Xamarin.Android-PR pr:none, test-area 404 -> az rest, gating-vs-flaky continueOnError lanes, queue-time ETA variance, logcat artifact). - Move deep-dive commands (ETA query, per-test error/stack, test-runs list, log fetch) to references/azdo-queries.md, loaded only when needed. Validated with skill-creator quick_validate and by re-running the lean workflow against live PRs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Per review feedback: the explicit "Present summary" template produced predictable, familiar output. Bring back an explicit (but lean) report template — PR header + fork badge, the dotnet-android build block (result / elapsed / ETA / job counters), a Stage > Job status table, Failures, gating-vs-flaky Failed tests, Verdict, and What next — in place of the prose-only guidance from the previous commit. SKILL.md is still ~107 lines (vs 306 originally). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adapts the local agent skills (
.github/skills/) and workflow docs to two recent CI changes. Docs/skills only — no product code.1. Public PR pipeline (#11578)
PR validation now runs on a single public pipeline —
dotnet-androidondev.azure.com/dnceng-public(projectpublic, def333) — running the full matrix for every PR (direct + fork). The DevDivXamarin.Android-PRpipeline gotpr: noneand no longer runs on PRs.Reworked the
ci-statusskill around one public build:dotnet-androidbuild per PR;BUILD_IDextraction from the check links; org/project =dnceng-public/public./azp runapproval), not which pipeline runs.test-areaaz devops invokeis broken on dnceng-public → switched failed-test queries toaz rest+ResultsByBuild.result+ GitHub checks are authoritative —continueOnErrordevice-test failures (e.g. flakySslTest.*) are published even on green builds and must not be reported as red.2. NUnitLite → stock NUnit via
dotnet test/MTP (#11224)Mono.Android device tests now run stock NUnit through
dotnet test+ Microsoft Testing Platform (-t:Installthendotnet test --no-build --report-trx, TRX output,logcat-*.txtartifacts) instead of NUnitLite/-t:RunTestApp/TestResult-*.xml.Updated:
testsskill (+test-catalog.md),copilot-instructions.md,Documentation/workflow/UnitTests.md, and removed the now-deletedNUnitLite/row from theupdate-tpnskill. Theci-statusskill +binlog-analysis.mdgained the device-test logcat/TRX notes.Validation
Exercised the updated
ci-statusskill end-to-end against several live PRs and iterated:Package Testsstage + failed tests.ResultsByBuildlisted 4SslTestfailures on an all-green build; the gating-vs-flaky logic reports green + a non-blocking note.All
az/gh/az restcommands in the skill were run againstdnceng-publicand confirmed working.🤖 Drafted with GitHub Copilot CLI.