[FIP-37] Add bitmap scalar functions and register via FlussCatalog#3492
Open
Prajwal-banakar wants to merge 1 commit into
Open
[FIP-37] Add bitmap scalar functions and register via FlussCatalog#3492Prajwal-banakar wants to merge 1 commit into
Prajwal-banakar wants to merge 1 commit into
Conversation
Contributor
Author
|
Hi @polyzos , could you please help review here!? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: Part of #3289
This PR completes the Phase 1 implementation of FIP-37 by adding the six RoaringBitmap scalar functions. It builds on the infrastructure from PR #3319 and the aggregate functions from PR #3398
These functions are registered as built-in functions in
FlussCatalog, allowing users to call them directly in Flink SQL without manualCREATE FUNCTIONstatements.Brief change log
New files in
fluss-flink/fluss-flink-common:functions/bitmap/RbCardinalityFunction.java:rb_cardinality(BYTES) -> BIGINT— Returns the number of distinct integers in the bitmap.functions/bitmap/RbBuildFunction.java:rb_build(ARRAY<INT>) -> BYTES— Constructs a bitmap from an array of integers.functions/bitmap/RbContainsFunction.java:rb_contains(BYTES, INT) -> BOOLEAN— Checks if a bitmap contains a specific integer.functions/bitmap/RbToArrayFunction.java:rb_to_array(BYTES) -> ARRAY<INT>— Converts a bitmap into an array of its integers.functions/bitmap/RbOrFunction.java:rb_or(BYTES, BYTES) -> BYTES— Computes the union of two bitmaps.functions/bitmap/RbAndFunction.java:rb_and(BYTES, BYTES) -> BYTES— Computes the intersection of two bitmaps.test/java/.../functions/bitmap/RbScalarFunctionsTest.java: Unit tests for all six scalar functions, covering correctness, null handling, and edge cases.Modified files:
flink/catalog/FlinkCatalog.java: Registers the six new scalar functions in theBUILTIN_BITMAP_FUNCTIONSmap.test/.../catalog/FlinkCatalogTest.java: Adds a unit test to verify the scalar functions are correctly registered.test/.../functions/bitmap/RbFunctionsCatalogITCase.java: Adds a new integration test (testScalarFunctions) that exercises all scalar functions through aTableEnvironment, validating the end-to-end catalog registration and execution path.Tests
RbScalarFunctionsTestcontains comprehensive tests for each new scalar function.FlinkCatalogTestverifies that the functions are discoverable through the catalog API.RbFunctionsCatalogITCasenow includes a test that runs SQL queries withrb_cardinality,rb_build,rb_contains,rb_to_array,rb_or, andrb_andagainst a live Flink cluster.Verified locally:
./mvnw spotless:apply— 0 violations./mvnw test -pl fluss-flink/fluss-flink-common -Dtest="RbScalarFunctionsTest,FlinkCatalogTest"— BUILD SUCCESS./mvnw verify -pl fluss-flink/fluss-flink-common -Dit.test=RbFunctionsCatalogITCase— BUILD SUCCESSAPI and Format
This change is purely additive and does not affect any storage formats or wire protocols. The functions operate on
BYTEScolumns containing standard RoaringBitmap serialized data.Documentation
User-facing documentation will be created in a follow-up PR now that the full set of Phase 1 functions is complete.