{"id":5807,"date":"2026-09-30T17:26:17","date_gmt":"2026-09-30T18:26:17","guid":{"rendered":"https:\/\/cloudobjectivity.co.uk\/?p=5807"},"modified":"2026-10-07T13:27:05","modified_gmt":"2026-10-07T14:27:05","slug":"amazon-s3-tables-now-support-all-apache-iceberg-v3-data-types","status":"publish","type":"post","link":"http:\/\/cloudobjectivity.co.uk\/index.php\/2026\/09\/30\/amazon-s3-tables-now-support-all-apache-iceberg-v3-data-types\/","title":{"rendered":"Amazon S3 Tables now support all Apache Iceberg V3 data types"},"content":{"rendered":"\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

Published: September 30, 2026<\/p>\n\n\n\n

Executive Overview<\/h5>\n\n\n\n

The modernization of enterprise data architecture has reached a decisive milestone around open table formats. Over the past decade, the rapid accumulation of structured, semi-structured, and telemetry datasets transformed object storage into the de facto enterprise data lakehouse tier. Among open formats, Apache Iceberg emerged as the enterprise standard, providing ACID transactional guarantees, schema evolution, hidden partitioning, and engine-agnostic interoperability across distributed compute frameworks. However, enterprise data platform teams operating at petabyte and exabyte scale have encountered severe operational friction under the Apache Iceberg V2 specification. High-frequency row modifications, granular regulatory compliance purges (such as GDPR and CCPA “right-to-be-forgotten” requests), and streaming change data capture (CDC) pipelines created massive numbers of positional delete files. These positional delete files caused severe read amplification, degraded interactive query latencies, and required continuous, resource-intensive maintenance compaction jobs to maintain table health.<\/p>\n\n\n\n

Furthermore, modern analytical workloads require native support for diverse data structures\u2014including deeply nested, polymorphic semi-structured payloads, high-precision geospatial coordinates, and sub-millisecond temporal metrics\u2014without forcing data engineers to encode data into rigid string or integer representations. To resolve these architectural pain points and provide a fully managed, engine-agnostic lakehouse tier, Amazon Web Services announced that Amazon S3 Tables now support all data types and core capabilities defined in the Apache Iceberg V3 specification. This release introduces native support for deletion vectors, the polymorphic variant data type, nanosecond-precision timestamps, native geometry and geography geospatial primitives, column default values without data backfills, and native row lineage tracking. Supported by S3 Tables’ automatic, continuous background compaction and maintenance, this release equips enterprise platform architects and data engineering leaders with an optimized foundation to modernize analytical lakehouses, accelerate change data capture pipelines, and lower total cost of ownership across distributed analytics engines.<\/p>\n\n\n\n

Features<\/h5>\n\n\n\n

The technical capabilities introduced in Amazon S3 Tables for the Apache Iceberg V3 specification represent a comprehensive expansion of storage primitives, binary delete structures, complex data types, schema management, and automated table maintenance.<\/p>\n\n\n\n

    \n
  • Deletion Vectors Replacing Positional Delete Files: Apache Iceberg V3 replaces V2’s file-level positional delete structures with compact binary deletion vectors. Under V2, deleting or updating rows generated separate positional delete files that query engines had to merge into memory during query execution (merge-on-read), introducing substantial read amplification and memory overhead. Deletion vectors utilize compact Roaring Bitmaps stored directly within the data file boundary or adjacent metadata, indicating exactly which rows are logically invalidated. This allows analytical engines to skip deleted records with near-zero deserialization latency, drastically reducing query scan overhead and shrinking write-amplification during row modifications.<\/li>\n\n\n\n
  • Complete Native Apache Iceberg V3 Data Type Ecosystem: S3 Tables introduces native storage and query support for all V3 data types, eliminating previous requirements to serialize complex or specialized attributes into generic strings or binary blobs:\n
      \n
    • Variant Data Type: Provides native storage for semi-structured, polymorphic JSON payloads without requiring predefined schemas. The variant type enables high-performance schema-on-read querying using the variant_get function, allowing disparate event structures (such as clickstream telemetry with variable attributes) to reside within a single table column.<\/li>\n\n\n\n
    • Nanosecond Timestamp (timestamp_ns and timestamptz_ns): Supports nanosecond-level temporal precision with and without time zone context. This capability is purpose-built for high-frequency financial trading algorithms, high-throughput IoT sensor telemetry, and distributed system event logs where microsecond timestamps are insufficient to preserve causal event order.<\/li>\n\n\n\n
    • Native Geospatial Types (Geometry and Geography): Enables direct storage of spatial vectors, polygon shapes, and geographic coordinates adhering to open geospatial consortium standards, streamlining spatial joins and GIS mapping analytics directly inside the lakehouse.<\/li>\n\n\n\n
    • Unknown Data Type: Allows downstream ingest engines to record columns with initially undefined or un-inferable data types, preserving raw source ingestion pipelines without failing schema validation jobs.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
    • Column Default Values Without Historical Data Rewrites: Under Iceberg V3, adding a new column with a default value no longer requires scanning and rewriting every historical Parquet data file in the table. The default value is recorded exclusively in the table’s metadata layer. When query engines read historical rows authored prior to the column addition, the query engine automatically evaluates the metadata-defined default value, enabling instantaneous, zero-cost schema evolution across multi-petabyte datasets.<\/li>\n\n\n\n
    • Native Row Lineage and Audit Tracking (_row_id and _last_updated_sequence_number): S3 Tables automatically populates and tracks row lineage fields across table lifecycles. Every row receives an immutable _row_id and a monotonically increasing _last_updated_sequence_number. This capability enables data engineers to construct efficient, stateless incremental data pipelines that query changes committed since a specific sequence boundary without maintaining external state tracking databases.<\/li>\n\n\n\n
    • Fully Managed Continuous Background Compaction: S3 Tables eliminates the manual administrative burden of managing Iceberg table health. The managed maintenance engine automatically monitors table fragmentation, tracks deletion vector density, and schedules automated background compaction runs. Compaction cycles merge deletion vectors back into base Parquet files, rewrite small files into optimal block sizes, and safely clean up unreferenced snapshots without manual human intervention or external orchestrator jobs.<\/li>\n\n\n\n
    • In-Place Atomic Upgrades from V2 to V3: Existing Iceberg V2 tables hosted in S3 Tables can be upgraded to V3 atomically in place using the standard SQL command ALTER TABLE my_table SET TBLPROPERTIES (‘format-version’ = ‘3’);. Upgrading does not rewrite underlying Parquet data files; existing V2 readers continue to read legacy files, while newly inserted and modified rows leverage V3 primitives. On subsequent automated compaction cycles, S3 Tables merges and retires legacy V2 delete files automatically.<\/li>\n\n\n\n
    • Broad Integration Across AWS Analytics Engines and Iceberg REST Catalog: S3 Tables exposes standard Iceberg REST Catalog (IRC) APIs, providing seamless multi-engine interoperability across Amazon EMR Spark, AWS Glue, Amazon Athena, and Amazon Redshift, alongside third-party compute frameworks such as Snowflake, Databricks, and Apache Flink.<\/li>\n\n\n\n
    • Global Regional Availability and Transparent Pricing: Native Iceberg V3 support is generally available across all commercial AWS Regions where Amazon S3 Tables is supported. Adopting V3 introduces no incremental feature licensing fees; organizations pay standard Amazon S3 Tables storage, maintenance, and request charges.<\/li>\n<\/ul>\n\n\n\n
      Benefits<\/h5>\n\n\n\n

      Deploying the Apache Iceberg V3 specification on Amazon S3 Tables yields tangible strategic, operational, financial, and architectural advantages for enterprise data organizations.<\/p>\n\n\n\n

        \n
      • Significant Query Latency Reduction and Read Amplification Elimination: The transition from positional delete files to binary deletion vectors delivers immediate performance dividends. Analytical queries executing against tables with frequent updates or deletions experience up to 40% to 60% faster scan times because compute engines evaluate compact Roaring Bitmaps directly during byte scanning rather than executing expensive distributed hash joins across separate delete files. This improvement directly stabilizes interactive query performance for business intelligence dashboards and operational analytics.<\/li>\n\n\n\n
      • Elimination of Manual Lakehouse Table Maintenance Overhead: In self-managed Iceberg architectures, platform teams spend substantial engineering effort building and monitoring Apache Spark maintenance jobs to execute periodic rewrite_data_files and expire_snapshots procedures. S3 Tables completely abstracts this operational complexity. The service’s managed maintenance engine continuously evaluates table health metrics, executing compaction and garbage collection in the background, eliminating the risk of un-compacted tables degrading query performance across the enterprise.<\/li>\n\n\n\n
      • Modernized FinOps Storage and Compute Cost Optimization: Iceberg V3 combined with managed S3 Tables generates substantial cloud cost savings across both storage and compute dimensions. Deletion vectors consume a fraction of the raw storage footprint required by legacy positional delete files, reducing monthly S3 storage expenditures. Concurrently, because queries read significantly fewer bytes and avoid memory-intensive delete merges, downstream compute engines (such as Amazon EMR clusters or Amazon Athena queries) consume fewer vCPU hours and scan fewer gigabytes, driving down the overall total cost of ownership for big data analytics.<\/li>\n\n\n\n
      • Agile Schema Evolution Without Data Pipeline Latency: Adding new business metrics or operational flags to multi-terabyte historical tables has historically been a high-risk, time-consuming operation requiring exhaustive data backfill jobs. The ability to declare column default values in metadata allows data engineering squads to deploy schema modifications instantaneously. Downstream applications gain immediate access to updated table definitions without taking tables offline or expending compute budget rewriting millions of historical data files.<\/li>\n\n\n\n
      • Native Streamlined Change Data Capture and Incremental Processing: Native row lineage tracking via _row_id and _last_updated_sequence_number transforms how enterprise engineering teams build downstream data pipelines. Rather than designing complex, brittle watermarking tables or maintaining intermediate change data capture (CDC) message queues, ETL workflows can query S3 Tables directly for records updated past a specific sequence number. This simplifies data pipeline architectures, reduces intermediate infrastructure dependencies, and accelerates data freshness across downstream data marts.<\/li>\n<\/ul>\n\n\n\n
        Use cases<\/h5>\n\n\n\n

        The combination of deletion vectors, flexible semi-structured data types, nanosecond precision, and automated compaction enables Amazon S3 Tables to resolve critical data processing bottlenecks across multiple enterprise industries.<\/p>\n\n\n\n

          \n
        • High-Volume Financial Trading Telemetry and Algorithmic Order Audit: A global quantitative trading firm processes millions of multi-asset equity transactions, market tick changes, and algorithmic order submissions every second. Historically, the firm struggled to preserve nanosecond-precision timestamps without serializing them into complex integer pairs or strings, which introduced compute overhead during quantitative analytics. By utilizing S3 Tables with native nanosecond timestamp support, the firm ingests raw market execution telemetry directly into Apache Iceberg V3 tables. Algorithmic researchers and regulatory compliance systems execute high-speed distributed SQL queries across billions of trades, analyzing microscopic market microstructure anomalies while maintaining strict compliance with financial trade reconstruction mandates.<\/li>\n\n\n\n
        • Polymorphic Clickstream and Digital Advertising Event Ingestion: A multinational digital advertising network ingests billions of tracking events daily from mobile devices, web applications, and smart connected televisions. Each event type possesses distinct nested attributes: ad impressions contain bidding metadata, clicks contain referral coordinates, and video views contain playback milestones. Rather than maintaining dozens of isolated tables or maintaining rigid schema normalization pipelines, the platform ingests all telemetry into a single S3 Table utilizing the variant data type. Analytics engineers execute ad-hoc SQL queries using variant_get expressions to evaluate campaign performance metrics dynamically, eliminating ETL schema drift errors while maintaining rapid write throughput.<\/li>\n\n\n\n
        • GDPR and CCPA Regulatory Compliance Deletions at Petabyte Scale: A global consumer e-commerce conglomerate stores five petabytes of historical customer browsing, order, and behavioral data across Iceberg tables. When customers submit “right-to-be-forgotten” regulatory data deletion requests, the compliance pipeline must purge specific customer records across historical datasets. Under Iceberg V2, frequent individual row deletions generated thousands of small positional delete files, severely degrading analytical query performance across the data lake. By upgrading tables to Iceberg V3, deletion requests write compact deletion vectors. S3 Tables’ automatic maintenance engine subsequently merges these vectors during regular maintenance windows, ensuring regulatory compliance without impacting production analytics or requiring custom compaction engineering.<\/li>\n\n\n\n
        • Real-Time Vehicle Telematics and Spatial Fleet Routing: An automotive manufacturer operates an autonomous vehicle fleet management platform tracking hundreds of thousands of commercial vehicles worldwide. Telematics gateways continuously stream vehicle GPS coordinates, sensor health metrics, and route polygons into S3 Tables using native geometry and geography data types. Route optimization engines execute spatial analytical queries directly across the lakehouse, identifying localized weather hazards, tracking geofencing boundary violations, and computing fleet routing efficiencies without relying on specialized external geographic information system (GIS) databases.<\/li>\n<\/ul>\n\n\n\n
          Alternatives<\/h5>\n\n\n\n

          Enterprise data architects, principal cloud engineers, and IT leadership evaluating lakehouse table formats, storage engines, and managed big data platforms must weigh S3 Tables with Iceberg V3 against alternative technological options.<\/p>\n\n\n\n