fix: precompute project group merge suggestion counts (CM-1230)

[skwowet]

Summary

CM-1230 addressed membersToMerge / organizationsToMerge timeouts by scoping merge suggestions against rollup rows in memberSegmentsAgg / organizationSegmentsAgg — matching the selected project-group segment directly instead of expanding to leaf subprojects and hitting the agg tables with huge segmentId IN (...) lists.

That fixed the worst leaf-expansion cases, but unfiltered COUNT requests were still slow for large project groups: even with a single rollup segment ID, Postgres still scanned all merge rows and ran expensive EXISTS checks against very large agg buckets, which could exceed API timeouts.

This PR adds precomputed per–project-group merge suggestion counts (same idea as other agg-backed reads: expensive work offline, cheap read online). Unfiltered badge/list counts are served from segmentMergeSuggestionCounts; filtered counts still use the live query — that is intentional and fine: the production incident was unfiltered countOnly on huge project groups (e.g. LF OSS Index), not filtered data-quality views. Filtered requests (similarity, displayName, memberId/organizationId) keep the existing live COUNT path; they were not timing out before this PR and do not need precomputation.

Also restores project / subproject filtering on the merge suggestions list page (Projects dropdown). Segment scoping had regressed to project-group-only; list and live COUNT again use filter.projectIds / filter.subprojectIds when set, with precomputed counts only for unfiltered project-group scope.

Counts are refreshed on a schedule and adjusted immediately when users merge or dismiss suggestions.

How it works now

1. Cron (every 4h) computes member and org merge suggestion counts per active project group and upserts segmentMergeSuggestionCounts (skips write when values are unchanged).
2. API COUNT (unfiltered) — project group segment (parent/grandparent null), no similarity / display name / entity filters → read precomputed row (~ms). CDP UI always sends the selected project group as the sole segment.
3. API COUNT (filtered) — any filter present → existing live COUNT against memberToMerge / organizationToMerge + agg EXISTS. Same query shape as before this PR; acceptable at query time because filtered views are not the high-volume unfiltered badge path and were not the timeout source.
4. LIST — still paginated live queries; respects project/subproject filters when provided. Only the count side uses the table when unfiltered at project-group scope.
5. User actions — on member/org merge or no-merge, decrement the precomputed count for project groups where both entities share a rollup segment, so list badges update without waiting for cron.

Changes

• Migration: segmentMergeSuggestionCounts table (segmentId, member/org counts, updatedAt).
• DAL (segments/index.ts): fetch project groups; calculate/upsert/get counts; common project-group segment lookup; decrement helpers.
• Cron: refresh-segment-merge-suggestion-counts job (batched parallelism, fail-fast on segment errors).
• Repositories: hybrid getTotalCount() — precomputed vs live; aligned member/org list + COUNT SQL (NOT EXISTS for mergeActions, consistent EXISTS); project-group guard for cache reads; simplified org list query (removed CTE / hash dedup).
• Merge suggestions filtering: restore project/subproject segment filters on list + live COUNT (projectIds / subprojectIds); precomputed table remains project-group only when no project filter and no count filters.
• Services: decrement precomputed counts after member merge, member no-merge, org merge, and org no-merge.

[Copilot]

Pull request overview

This PR introduces a precomputed, per–project-group cache of merge-suggestion counts to avoid expensive COUNT(*) queries timing out on large project groups, while keeping filtered counts and list pagination as live queries. It adds a cron refresher plus immediate count adjustments when users merge or dismiss suggestions.

Changes:

• Adds segmentMergeSuggestionCounts table + DAL helpers to compute, upsert, fetch, and decrement counts per project-group segment.
• Adds a cron job to refresh cached counts every 4 hours (only writing when counts change).
• Updates member/org merge-suggestion repositories and merge/no-merge flows to use cached totals for unfiltered single-project-group counts and decrement counts on user actions.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
services/libs/data-access-layer/src/segments/index.ts	Adds DAL functions for project-group segment enumeration, count calculation, cached count upsert/get, common-segment lookup, and decrement helpers.
services/libs/common_services/src/services/common.member.service.ts	Decrements cached member merge-suggestion counts after a successful member merge.
services/apps/cron_service/src/jobs/refreshSegmentMergeSuggestionCounts.job.ts	New cron job that refreshes cached counts on a 4h schedule.
backend/src/services/organizationService.ts	Decrements cached org merge-suggestion counts after org merge and org no-merge actions.
backend/src/services/memberService.ts	Decrements cached member merge-suggestion counts after member no-merge actions.
backend/src/database/repositories/organizationRepository.ts	Uses cached totals for unfiltered single-project-group counts; adjusts mergeActions filtering in list/count SQL.
backend/src/database/repositories/memberRepository.ts	Uses cached totals for unfiltered single-project-group counts; adjusts mergeActions filtering in list/count SQL.
backend/src/database/migrations/V1782128525__segment-merge-suggestion-counts.sql	Adds the segmentMergeSuggestionCounts table schema.

Comments suppressed due to low confidence (1)

backend/src/database/repositories/organizationRepository.ts:820

• Using a mergeActions LEFT JOIN + (ma.id IS NULL OR ma.state = ERROR) can multiply rows and can incorrectly include a suggestion if there are mergeActions recorded in both directions with different states (the ERROR row still passes). A NOT EXISTS predicate avoids join-multiplication and correctly excludes suggestions when any non-ERROR mergeAction exists.

      LEFT JOIN "mergeActions" ma
        ON ma.type = :mergeActionType
        AND (
          (ma."primaryId" = otm."organizationId" AND ma."secondaryId" = otm."toMergeId")
          OR (ma."primaryId" = otm."toMergeId" AND ma."secondaryId" = otm."organizationId")

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.