Skip to content

[FIP-37] Add bitmap scalar functions and register via FlussCatalog#3492

Open
Prajwal-banakar wants to merge 1 commit into
apache:mainfrom
Prajwal-banakar:fip-37-scalar-functions
Open

[FIP-37] Add bitmap scalar functions and register via FlussCatalog#3492
Prajwal-banakar wants to merge 1 commit into
apache:mainfrom
Prajwal-banakar:fip-37-scalar-functions

Conversation

@Prajwal-banakar

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: Part of #3289

This PR completes the Phase 1 implementation of FIP-37 by adding the six RoaringBitmap scalar functions. It builds on the infrastructure from PR #3319 and the aggregate functions from PR #3398

These functions are registered as built-in functions in FlussCatalog, allowing users to call them directly in Flink SQL without manual CREATE FUNCTION statements.

Brief change log

New files in fluss-flink/fluss-flink-common:

  • functions/bitmap/RbCardinalityFunction.java: rb_cardinality(BYTES) -> BIGINT — Returns the number of distinct integers in the bitmap.
  • functions/bitmap/RbBuildFunction.java: rb_build(ARRAY<INT>) -> BYTES — Constructs a bitmap from an array of integers.
  • functions/bitmap/RbContainsFunction.java: rb_contains(BYTES, INT) -> BOOLEAN — Checks if a bitmap contains a specific integer.
  • functions/bitmap/RbToArrayFunction.java: rb_to_array(BYTES) -> ARRAY<INT> — Converts a bitmap into an array of its integers.
  • functions/bitmap/RbOrFunction.java: rb_or(BYTES, BYTES) -> BYTES — Computes the union of two bitmaps.
  • functions/bitmap/RbAndFunction.java: rb_and(BYTES, BYTES) -> BYTES — Computes the intersection of two bitmaps.
  • test/java/.../functions/bitmap/RbScalarFunctionsTest.java: Unit tests for all six scalar functions, covering correctness, null handling, and edge cases.

Modified files:

  • flink/catalog/FlinkCatalog.java: Registers the six new scalar functions in the BUILTIN_BITMAP_FUNCTIONS map.
  • test/.../catalog/FlinkCatalogTest.java: Adds a unit test to verify the scalar functions are correctly registered.
  • test/.../functions/bitmap/RbFunctionsCatalogITCase.java: Adds a new integration test (testScalarFunctions) that exercises all scalar functions through a TableEnvironment, validating the end-to-end catalog registration and execution path.

Tests

  • Unit tests: RbScalarFunctionsTest contains comprehensive tests for each new scalar function.
  • Catalog integration: FlinkCatalogTest verifies that the functions are discoverable through the catalog API.
  • End-to-end SQL ITCase: RbFunctionsCatalogITCase now includes a test that runs SQL queries with rb_cardinality, rb_build, rb_contains, rb_to_array, rb_or, and rb_and against a live Flink cluster.

Verified locally:

  • ./mvnw spotless:apply — 0 violations
  • ./mvnw test -pl fluss-flink/fluss-flink-common -Dtest="RbScalarFunctionsTest,FlinkCatalogTest" — BUILD SUCCESS
  • ./mvnw verify -pl fluss-flink/fluss-flink-common -Dit.test=RbFunctionsCatalogITCase — BUILD SUCCESS

API and Format

This change is purely additive and does not affect any storage formats or wire protocols. The functions operate on BYTES columns containing standard RoaringBitmap serialized data.

Documentation

User-facing documentation will be created in a follow-up PR now that the full set of Phase 1 functions is complete.

@Prajwal-banakar

Copy link
Copy Markdown
Contributor Author

Hi @polyzos , could you please help review here!?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant