Iceberg Golang

iceberg is a Golang implementation of the Iceberg table spec.

Build From Source

Prerequisites

• Go 1.25 or later

Build

$ git clone https://github.com/apache/iceberg-go.git
$ cd iceberg-go/cmd/iceberg && go build .

Running Tests

Use the Makefile so commands stay in sync with CI (e.g. golangci-lint version).

Unit tests

make test

Linting

make lint

Install the linter first

make lint-install
or: go install github.com/golangci/golangci-lint/v2/cmd/[email protected]

Integration tests

Prerequisites: Docker, Docker Compose

1. Start the Docker containers using docker compose:

make integration-setup

1. Export the required environment variables:

export AWS_S3_ENDPOINT=http://$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' minio):9000
   export AWS_REGION=us-east-1
   export SPARK_CONTAINER_ID=$(docker ps -qf 'name=spark-iceberg')
   export DOCKER_API_VERSION=$(docker version -f '{{.Server.APIVersion}}')

1. Run the integration tests:

make integration-test

Or run a single suite: make integration-scanner, make integration-io, make integration-rest, make integration-rest-scan-planning, make integration-spark, make integration-hive, make integration-hadoop.

Running against Spark 4.0

To validate against Spark 4.0 (used by v3 feature tests), substitute the setup step:

make integration-setup-spark4

The exported env vars and make integration-test invocation are identical to the Spark 3.5 flow above.

Tests requiring Spark 4 (currently the variant and unknown-type tests) automatically skip on the Spark 3.5 setup.

Feature Support / Roadmap

FileSystem Support

| Filesystem Type | Supported | | :------------------: | :-------: | | S3 | X | | Google Cloud Storage | X | | Azure Blob Storage | X | | Local Filesystem | X |

Metadata

| Operation | Supported | | :----------------------- | :-------: | | Get Schema | X | | Get Snapshots | X | | Get Sort Orders | X | | Get Partition Specs | X | | Get Manifests | X | | Create New Manifests | X | | Plan Scan | x | | Plan Scan for Snapshot | x |

Catalog Support

| Operation | REST | Hive | Glue | SQL | Hadoop | |:----------------------------|:----:|:----:|:------:|:----:|:------:| | Load Table | X | X | X | X | X | | List Tables | X | X | X | X | X | | Create Table | X | X | X | X | X | | Register Table | X | X | X | | | | Update Current Snapshot | X | X | X | X | X | | Create New Snapshot | X | X | X | X | X | | Rename Table | X | X | X | X | | | Drop Table | X | X | X | X | X | | Alter Table | X | X | X | X | X | | Check Table Exists | X | X | X | X | X | | Set Table Properties | X | X | X | X | X | | List Namespaces | X | X | X | X | X | | Create Namespace | X | X | X | X | X | | Check Namespace Exists | X | X | X | X | X | | Drop Namespace | X | X | X | X | X | | Update Namespace Properties | X | X | X | X | | | Create View | X | X | | X | | | Load View | X | X | | X | | | List View | X | X | | X | | | Drop View | X | X | | X | | | Rename View | X | | | | | | Check View Exists | X | X | | X | |

Read/Write Data Support

• Data can currently be read as an Arrow Table or as a stream of Arrow record batches.

Supported Write Operations

As long as the FileSystem is supported and the Catalog supports altering the table, the following tracks the current write support:

| Operation | Supported | |:---------------------|:---------:| | Append Stream | X | | Append Data Files | X | | Rewrite Files | X | | Rewrite manifests | X | | Overwrite Files | X | | Copy-On-Write Delete | X | | Write Pos Delete | X | | Write Eq Delete | X | | Row Delta | X |

CLI Usage

The iceberg CLI usage is very similar to pyiceberg CLI \ You can pass the catalog URI with --uri argument.

Example: You can start the Iceberg REST API docker image which runs on default in port 8181

docker pull apache/iceberg-rest-fixture:latest
docker run -p 8181:8181 apache/iceberg-rest-fixture:latest

./iceberg --uri http://0.0.0.0:8181 list
┌─────┐
| IDs |
| --- |
└─────┘

./iceberg --uri http://0.0.0.0:8181 create namespace taxitrips

List Namespace

./iceberg --uri http://0.0.0.0:8181 list
┌───────────┐
| IDs       |
| --------- |
| taxitrips |
└───────────┘

Get in Touch

• Iceberg community