Antalya 26.3: S3 tables iceberg support

[subkanthi]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added initial support for s3 tables(Iceberg REST catalog)

Documentation entry for user-facing changes

closes: ClickHouse#95340
Support for s3 tables(Iceberg serverless REST Catalog) using sigv4 authentication.
Can be enabled with allow_experimental_database_s3_tables

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [1e8c14e]

[subkanthi]

TESTING
Using Athena to create iceberg tables in s3Tables

CREATE TABLE `nyc`.daily_sales (
sale_date date, 
product_category string, 
sales_amount double)
PARTITIONED BY (month(sale_date))
TBLPROPERTIES ('table_type' = 'iceberg')

INSERT INTO daily_sales
VALUES
(DATE '2024-01-15', 'Laptop', 900.00),
(DATE '2024-01-15', 'Monitor', 250.00),
(DATE '2024-01-16', 'Laptop', 1350.00),
(DATE '2024-02-01', 'Monitor', 300.00),
(DATE '2024-02-01', 'Keyboard', 60.00),
(DATE '2024-02-02', 'Mouse', 25.00),
(DATE '2024-02-02', 'Laptop', 1050.00),
(DATE '2024-02-03', 'Laptop', 1200.00),
(DATE '2024-02-03', 'Monitor', 375.00);

SET allow_experimental_database_s3_tables = 1;

CREATE DATABASE my_s3_tables_sales
ENGINE = DataLakeCatalog('https://s3tables.us-east-2.amazonaws.com/iceberg')
SETTINGS
    catalog_type = 's3tables',
    warehouse = 'arn:aws:s3tables:us-east-2:<account_id>:bucket/nyc-taxis',
    aws_access_key_id = '',
    aws_secret_access_key = '',
    region = 'us-east-2';

use my_s3_tables_sales;

USE my_s3_tables_sales

Query id: 6e72f6fd-568f-4dd2-95fa-b1ff6bddd7df

Ok.

0 rows in set. Elapsed: 0.002 sec. 

Ubuntu-2404-noble-amd64-base :) select * from my_s3_tables_sales.`nyc.daily_sales`;

SELECT *
FROM my_s3_tables_sales.`nyc.daily_sales`

Query id: 1cec644c-8927-4b9c-a2c3-b770375f7734

   ┌──sale_date─┬─product_category─┬─sales_amount─┐
1. │ 2024-01-15 │ Laptop           │          900 │
2. │ 2024-01-15 │ Monitor          │          250 │
3. │ 2024-01-16 │ Laptop           │         1350 │
4. │ 2024-02-01 │ Monitor          │          300 │
5. │ 2024-02-01 │ Keyboard         │           60 │
6. │ 2024-02-02 │ Mouse            │           25 │
7. │ 2024-02-02 │ Laptop           │         1050 │
8. │ 2024-02-03 │ Laptop           │         1200 │
9. │ 2024-02-03 │ Monitor          │          375 │
   └────────────┴──────────────────┴──────────────┘

[alsugiliazova]

Audit: PR #1808 — Antalya 26.3: S3 tables iceberg support

AI audit note: This review comment was generated by AI (Cursor agent, audit-review skill).

Confirmed defects

Medium: Empty IAM credentials injected when vended credentials are missing

• Impact: If the catalog response has no vended S3 credentials and credentials_provider->GetAWSCredentials() returns empty keys (misconfigured env/IAM role), ClickHouse still attaches the table; reads fail later with opaque S3 auth errors instead of a clear catalog-time failure.
• Anchor: S3TablesCatalog::tryGetTableMetadata (need_credentials branch)
• Trigger: CREATE DATABASE with valid SigV4 catalog access but no usable AWS credentials on the server when loading table metadata without vended creds
• Why defect: Code injects S3Credentials from GetAWSCredentials() without checking isEmpty() or throwing
• Fix direction (short): After GetAWSCredentials(), throw BAD_ARGUMENTS if keys are empty when injection is required
• Regression test direction (short): Mock empty provider; assert tryGetTableMetadata fails with explicit message, not silent empty creds

[subkanthi]

Audit: branch antalya_26_3_s3_tables -- S3 Tables Iceberg support (re-run)

Scope: commits 00cba16, 284666c, plus working-tree fix (duplicate settings entry removal and empty-creds guard). 18 changed files covering new S3TablesCatalog, AWSV4Signer, RestCatalog signature changes, settings/enum registration, resolveS3Endpoint utility.

Confirmed defects
No confirmed defects in reviewed scope.

Coverage summary
Scope reviewed: Full call graph from CREATE DATABASE registration through S3TablesCatalog construction, getTables, tryGetTableMetadata, dropTable, getAuthHeaders/signRequestWithAWSV4; all settings/enum/history changes; resolveS3Endpoint utility and its gtest.
Categories passed: Thread safety (concurrent signing via thread pool), memory lifetime, exception safety, credential injection (empty-creds guard present), URL construction (config.prefix per Iceberg REST spec), SigV4 header signing consistency, settings registration (single entry in 26.3, enum/setting/experimental gate/enableAll all consistent), error handling (404-idempotent delete, HTTP error wrapping), no sensitive data leakage in logs.
Categories failed: None.
Assumptions/limits: AWS SDK AWSAuthV4Signer::SignRequest thread-safety per AWS documentation; std::filesystem::path URL construction consistent with pre-existing RestCatalog pattern; no runtime/integration test executed.

Audit: PR #1808 — Antalya 26.3: S3 tables iceberg support

AI audit note: This review comment was generated by AI (Cursor agent, audit-review skill).

Confirmed defects

Medium: Empty IAM credentials injected when vended credentials are missing

• Impact: If the catalog response has no vended S3 credentials and credentials_provider->GetAWSCredentials() returns empty keys (misconfigured env/IAM role), ClickHouse still attaches the table; reads fail later with opaque S3 auth errors instead of a clear catalog-time failure.
• Anchor: S3TablesCatalog::tryGetTableMetadata (need_credentials branch)
• Trigger: CREATE DATABASE with valid SigV4 catalog access but no usable AWS credentials on the server when loading table metadata without vended creds
• Why defect: Code injects S3Credentials from GetAWSCredentials() without checking isEmpty() or throwing
• Fix direction (short): After GetAWSCredentials(), throw BAD_ARGUMENTS if keys are empty when injection is required
• Regression test direction (short): Mock empty provider; assert tryGetTableMetadata fails with explicit message, not silent empty creds

[arthurpassos]

lgtm

[alsugiliazova]

Audit: PR #1808 — Antalya 26.3: S3 Tables Iceberg REST support (SigV4)

• PR: Antalya 26.3: S3 tables iceberg support #1808
• Base: antalya-26.3
• Head: antalya_26_3_s3_tables
• Scope: 17 files, +608 / -16
  • src/Core/Settings.cpp, SettingsChangesHistory.cpp, SettingsEnums.{h,cpp} — new allow_experimental_database_s3_tables setting and S3_TABLES enum value.
  • src/Databases/DataLake/AWSV4Signer.{h,cpp} (new) — Poco → AWS SigV4 bridge for REST requests.
  • src/Databases/DataLake/S3TablesCatalog.{h,cpp} (new) — s3tables REST catalog over SigV4.
  • src/Databases/DataLake/RestCatalog.{h,cpp} — extended getAuthHeaders signature and routed write endpoints through config.prefix.
  • src/Databases/DataLake/DatabaseDataLake.cpp — dispatch / validation for S3_TABLES.
  • src/Databases/DataLake/ICatalog.{h,cpp} — TableMetadata::hasDataLakeSpecificProperties helper.
  • src/Databases/DataLake/StorageCredentials.h — S3Credentials::isEmpty.
  • src/Databases/enableAllExperimentalSettings.cpp — opt-in for stateless test envs.
  • src/IO/S3/URI.{h,cpp} + gtest_s3_uri.cpp — DB::S3::resolveS3Endpoint(region) and tests.

Note: This review was produced by AI (composer-1) using static reasoning only;
no live S3 Tables endpoint was exercised. Severity is bounded accordingly.

Confirmed defects

Medium — Vended-credentials refresh callback returns nullptr for catalog-IAM-cred S3 Tables

• Impact: Long-running queries against S3 Tables that rely on catalog IAM credentials (the path that injects credentials_provider->GetAWSCredentials() in S3TablesCatalog::tryGetTableMetadata) get a one-shot snapshot of access key / secret / session token. When the underlying chain (e.g. assume-role) rotates the session token, the storage layer asks the inherited RestCatalog::getCredentialsConfigurationCallback, which re-fetches LoadTableResponse and calls getCredentialsAndEndpoint. For S3 Tables tables without vended creds the parsed credentials are nullptr, so the callback returns nullptr and the storage keeps using the now-expired snapshot.
• Anchor: src/Databases/DataLake/S3TablesCatalog.cpp (tryGetTableMetadata) + src/Databases/DataLake/RestCatalog.cpp (getCredentialsConfigurationCallback).
• Trigger: Create S3 Tables database without explicit aws_access_key_id / aws_secret_access_key; run a query whose lifetime exceeds the role session token TTL (default 1h for sts:AssumeRole).
• Why defect: S3TablesCatalog short-circuits credential vending only inside the initial metadata fetch; the refresh callback is unaware of the IAM fallback and yields no credentials, so the S3 reader cannot rotate them.
• Fix direction: Override getCredentialsConfigurationCallback in S3TablesCatalog to re-call credentials_provider->GetAWSCredentials() when the inner callback returns nullptr or empty creds.
• Regression test direction: Unit test that mocks an AWSCredentialsProvider returning two distinct credential sets across calls and asserts that the callback returned by S3TablesCatalog reflects the rotated values.

Retracted — empty() recurses into ?parent=<ns> on a flat S3 Tables catalog

Originally flagged as Medium; after a deeper call-graph trace, this is no longer in scope for PR #1808.

• Why retracted:
  • EXISTS DATABASE (validated live: EXISTS DATABASE my_s3_tables_sales -> 1) goes through DatabaseCatalog::isDatabaseExist, which only consults the in-memory registry. It does not call IDatabase::empty.
  • DROP DATABASE / DETACH DATABASE reaches db->empty only when shouldBeEmptyOnDetach() returns true. DatabaseDataLake::shouldBeEmptyOnDetach() returns false, so the check is skipped.
  • The other in-tree caller (loadMetadata.cpp:451) only runs on ordinary/atomic database recovery, not on DatabaseDataLake.
• Result: DatabaseDataLake::empty -> RestCatalog::empty is effectively dead code in the current call graph; the S3 Tables recursion path it would hit isn't reachable in PR scope.
• Note (not introduced here): RestCatalog::empty returns found_table instead of !found_table — inverted compared to every peer (DatabaseS3, DatabaseFilesystem, PaimonRestCatalog, GlueCatalog, HiveCatalog, DatabaseWithOwnTablesBase). This is pre-existing in master, untouched by PR Antalya 26.3: S3 tables iceberg support #1808, and currently masked by the unreachability above. Worth tracking as a separate bug, but not a finding for this PR.

Low — S3TablesCatalog::tryGetTableMetadata rewrites iceberg_metadata_file_location before getMetadataLocation runs

• Impact: For S3 Tables responses where metadata-location is s3://<bucket>/<path>/<file>.metadata.json, the catalog strips s3://<bucket>/ and writes back the relative path/.../file.metadata.json. TableMetadata::getMetadataLocation already handles the storage_type_str (s3://) prefix and the optional endpoint prefix; after pre-stripping, neither code path matches and the value flows through unchanged. The downstream IcebergMetadata code in IcebergMetadata.cpp then prepends persistent_components.table_path, which can produce a different absolute path than the catalog returned when the data and metadata reside under different prefixes (e.g. cross-bucket vended layouts).
• Anchor: src/Databases/DataLake/S3TablesCatalog.cpp lines 195–207; src/Databases/DataLake/ICatalog.cpp TableMetadata::getMetadataLocation; src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp around line 1639.
• Trigger: S3 Tables table whose metadata-location does not share its bucket/path prefix with the catalog-reported location (rare but legal under vended credentials).
• Why defect: Two layers ("strip scheme+bucket in the catalog" and "strip storage_type_str / data_location in getMetadataLocation") duplicate responsibility and can disagree.
• Fix direction: Either drop the pre-stripping in S3TablesCatalog and rely on getMetadataLocation, or move the S3 Tables-specific normalization into getMetadataLocation / setLocation so both sides see the same representation.
• Regression test direction: Add a getMetadataLocation unit test fed with the S3 Tables-style absolute s3://bucket/path/.../v1.metadata.json value and assert the resulting relative path matches the location returned by the catalog response in tests.

Low — resolveS3Endpoint non-SDK fallback ignores non-default partitions

• Impact: When the AWS SDK endpoint resolver fails (initialization order, unknown region, sandbox), the catch-all return "https://s3." + region + ".amazonaws.com" produces an incorrect endpoint for China (amazonaws.com.cn) and GovCloud. The same resolveS3Endpoint is invoked from S3TablesCatalog::tryGetTableMetadata to inject an endpoint into TableMetadata when the catalog did not vend one, so a failed resolution silently downgrades to an unusable endpoint.
• Anchor: src/IO/S3/URI.cpp resolveS3Endpoint lines 307–318.
• Trigger: Run against a China / GovCloud region while the SDK's endpoint rule engine is unavailable (e.g. SDK initialization not run yet, or an SDK upgrade regressing the rules).
• Why defect: The fallback contradicts the comment in URI.h ("handles all partitions: standard, China, GovCloud, etc.") because only the success branch handles them.
• Fix direction: Either throw on resolver failure or partition-aware the fallback (cn-* → amazonaws.com.cn, us-gov-* → amazonaws.com).
• Regression test direction: Extend IOTestS3URI.ResolveS3Endpoint with a forced-failure path (mock provider) and assert the expected partition-correct host.

Low — Dead / unused declarations in the S3 Tables module

• Impact: Confuses maintainers and creates lurking link-time risks.
  • TableMetadata::hasDataLakeSpecificProperties() has no callers.
  • Three settings (s3_max_connections, s3_connect_timeout_ms, s3_request_timeout_ms) are declared extern in S3TablesCatalog.cpp but never read.
  • signing_service member duplicates the signer's constructor argument.
  • #include <Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.h> is unused.
• Anchor: src/Databases/DataLake/ICatalog.h line 52 / ICatalog.cpp lines 275–278; src/Databases/DataLake/S3TablesCatalog.cpp lines 20, 38–47, 68, 245.
• Trigger: N/A (static).
• Why defect: Either the settings were intended to flow into the poco / Aws config (then they're missing), or they should be removed.
• Fix direction: Drop the unused externs and unused include; either remove hasDataLakeSpecificProperties or call it from the existing data_lake_specific_metadata.has_value() sites; remove the redundant signing_service member.
• Regression test direction: None (cleanup only).

Coverage summary

• Scope reviewed: all 17 changed files (Settings / Enums plumbing, DatabaseDataLake dispatch, RestCatalog write-path prefix changes + getAuthHeaders signature, new S3TablesCatalog, new AWSV4Signer, IO/S3/URI::resolveS3Endpoint + test, StorageCredentials::isEmpty, enableAllExperimentalSettings).
• Categories failed:
  • Vended-credentials refresh contract.
  • Layered path-normalization between catalog and getMetadataLocation.
  • Endpoint resolver fallback completeness across AWS partitions.
  • Dead-code hygiene.
• Categories retracted:
  • Recursive namespace traversal for flat catalogs — DatabaseDataLake::empty is not reached from EXISTS/DROP/DETACH DATABASE (validated live), so the recursion is dead code in PR scope.
• Categories passed:
  • Virtual dispatch ordering in the constructor — signer is built before loadConfig, and S3TablesCatalog uses the 7-arg protected RestCatalog ctor that does not call loadConfig from the base.
  • Thread safety of AWSAuthV4Signer reuse across the getTables thread pool (no shared mutable state during SignRequest).
  • Body / hash consistency between signing and the out_stream_callback (same body_str is passed to both).
  • Content-Type non-duplication after filtering signer headers to authorization / x-amz-*.
  • Behavior of std::move(headers.begin(), …) on const HTTPHeaderEntries & — degrades to copy, retry path safe.
  • Error propagation when region is empty (validated in both validateSettings and the catalog ctor).
• Assumptions / limits: Static analysis only. Signature / path canonicalization assumptions about Aws::Http::URI (%2F in the warehouse ARN, urlEscapePath=false) were reasoned about but not exercised against a live S3 Tables endpoint.

[subkanthi]

Low: resolveS3Endpoint partition-incorrect fallback

Impact: When the AWS SDK endpoint resolver fails, S3 Tables tables without a catalog-vended endpoint get https://s3..amazonaws.com, which is wrong for China (.amazonaws.com.cn) and GovCloud partitions; metadata/data reads then hit the wrong host.
Anchor: src/IO/S3/URI.cpp / DB::S3::resolveS3Endpoint
Trigger: catalog_type = 's3tables', table metadata has no vended endpoint, SDK ResolveEndpoint fails (init order, unknown region, SDK regression).
Why defect: Success path uses Smithy rules (tested for cn-north-1, us-gov-west-1); fallback ignores partition semantics despite the header comment claiming otherwise.
Fix direction (short): Throw on resolver failure, or make the fallback partition-aware (cn-, us-gov-*).
Regression test direction (short): Mock a failing S3EndpointProvider and assert China/GovCloud hosts, or that an exception is thrown.

Low: Credential refresh silently keeps stale creds when IAM provider returns empty

Impact: A long-running query whose session token expires gets nullptr from getCatalogIAMCredentials on refresh; getClient keeps the old key/token and the query fails with an opaque S3 auth error instead of a clear catalog/credentials message.
Anchor: src/Databases/DataLake/S3TablesCredentialRefresh.cpp / getCatalogIAMCredentials; src/Disks/DiskObjectStorage/ObjectStorages/S3/diskSettings.cpp / getClient refresh branch
Trigger: Catalog-IAM path (no vended creds), query lifetime crosses STS session expiry, then GetAWSCredentials() returns empty keys mid-refresh.
Why defect: Initial attach fails fast on empty IAM creds, but the refresh path returns nullptr and the S3 client path silently reuses stale credentials.
Fix direction (short): Propagate an exception from the refresh callback when fallback IAM creds are empty.
Regression test direction (short): Mock provider returning empty on second call; assert refresh fails with an explicit error, not silent reuse.

Coverage summary:

Scope reviewed: All 21 changed files — settings/enum plumbing, DatabaseDataLake dispatch and validation, RestCatalog getAuthHeaders signature and write-path config.prefix routing, new S3TablesCatalog / AWSV4Signer / S3TablesCredentialRefresh, resolveS3Endpoint + gtests, StorageCredentials::isEmpty, experimental gate.
Categories failed: Endpoint resolver fallback completeness; refresh error contract when IAM creds disappear mid-query.
Categories passed: SigV4 signing (AWS SDK stores lowercase header keys — authorization filter is correct); constructor ordering (signer before loadConfig); empty IAM guard at metadata time; credential refresh override + unit tests (gtest_s3tables_credential_refresh); metadata-location normalization (pre-stripping removed, gtests added); dead-code cleanup from prior audit; thread safety of concurrent SignRequest on shared signer (const method, per-request HttpRequest); RestCatalog prefix routing fix for write endpoints; no credential leakage in new log lines.
Assumptions/limits: Static analysis only; no live S3 Tables endpoint exercised. Prior Medium findings (empty creds at attach, refresh callback, metadata stripping, dead code) are addressed in commits 55635ea and 202c052.