[![[parquet-tools]](https://github.com/hangxie/parquet-tools/actions/workflows/release.yml/badge.svg)](https://github.com/hangxie/parquet-tools/actions/workflows/release.yml)
parquet-tools
A utility to inspect Parquet files.Quick Start
Pre-built binaries or packages can be found on the release page, on Mac you can install with brew:
$ brew install go-parquet-toolsOnce it is installed:
$ parquet-tools
Usage: parquet-tools <command>
A utility to inspect Parquet files, for full usage see https://github.com/hangxie/parquet-tools/blob/main/README.md
Flags:
-h, --help Show context-sensitive help.
Commands:
cat Prints the content of a Parquet file, data only.
import Create Parquet file from other source data.
inspect Inspect Parquet file structure in detail.
merge Merge multiple parquet files into one.
meta Prints the metadata.
retype Change column type.
row-count Prints the count of rows.
schema Prints the schema.
shell-completions Install/uninstall shell completions
size Prints the size.
split Split into multiple parquet files.
transcode Transcode Parquet file with different compression.
version Show build version.
Run "parquet-tools <command> --help" for more information on a command.
parquet-tools: error: expected one of "cat", "import", "inspect", "merge", "meta", "row-count", ...
Table of Contents
- Quick Start - Table of Contents - Installation - Install from Source - Download Pre-built Binaries - Brew Install - Container Image - Prebuilt RPM and deb Packages - Usage - Canceling a Command - Obtain Help - Parquet File Location - File System - S3 Bucket - GCS Bucket - Azure Storage Container - HDFS File - HTTP Endpoint - Reading Encrypted Parquet Files - Writing Encrypted Parquet Files - File Format Options - Compression Codecs - Compression Levels - Data Page Version - Writer Tuning Options - Encoding - Geo Data Type Support - GEOGRAPHY/GEOMETRY vs GeoParquet - Geospatial Format - Variant Data Type Support - Physical Storage - Logical View - Advanced: Fine-Grained Compression - Shredded VARIANT - cat Command - Full Data Set - Skip Rows - CSV/TSV Format - Limit Number of Rows - Sampling - Compound Rule - Output Format - Non-finite Floating Point Values - UNKNOWN Logical Type - import Command - Import from CSV - Import from JSON - Import from JSONL - inspect Command - Inspect File Level - Inspect Row Group Level - Inspect Column Chunk Level - Inspect Page Level - Inspect Encryption - merge Command - meta Command - Show Meta Data - Bloom Filter Sizes - Meta Encryption - Discovering KMS Key IDs - retype Command - Convert INT96 to Timestamp - Convert BSON to String - Convert VARIANT to String - Convert UUID to String - Convert Repeated Primitive to LIST - Remove Geospatial Logical Type - Remove JSON Logical Type - Convert FLOAT16 to FLOAT32 - row-count Command - Show Number of Rows - schema Command - JSON Format - Raw Format - Go Struct Format - CSV Format - shell-completions Command (Experimental) - Install Shell Completions - Uninstall Shell Completions - Use Shell Completions - size Command - Show Raw Size - Show Footer Size in JSON Format - Show All Sizes in JSON Format - split Command - Name format - Exact number of output files - Maximum records in a file - transcode Command - Change Compression Codec - Change Data Page Version - Field-Specific Encoding - Control Statistics - Field-Specific Compression - Combine Multiple Options - INT96 Field Detection - version Command - Print Version - Print All Information - Print Version and Build Time in JSON Format - Print Version in JSON Format - Credit - LicenseInstallation
You can choose one of the installation methods from below, the functionality will be mostly the same.
Install from Source
Good for people who are familiar with Go, you need 1.26 or newer version.
$ go install github.com/hangxie/parquet-tools@latestThe above command installs the latest released version of parquet-tools to $GOPATH/bin, parquet-tools installed from source will not report proper version and build time, so if you run parquet-tools version, it will just give you an empty line, all other functions are not affected.
[!TIP]
If you do not setGOPATHenvironment variable explicitly, then its default value can be obtained by runninggo env GOPATH, usually it isgo/directory under your home directory.
Download Pre-built Binaries
Good for people who do not want to build and all other installation approaches do not work.
Go to release page, pick the release and platform you want to run, download the corresponding gz/zip file, extract it to your local disk, make sure the execution bit is set if you are running on Linux, Mac, or FreeBSD, then run the program.
For Windows 10 on ARM (like Surface Pro X), use windows-arm64, if you are using Windows 11 on ARM, both windows-arm64 and windows-amd64 build should work.
Brew Install
Mac users can use Homebrew to install:
$ brew install go-parquet-toolsTo upgrade to the latest version:
$ brew upgrade go-parquet-toolsContainer Image
Container image supports amd64, arm64, and arm/v7, it is hosted in two registries:
You can pull the image from either location:$ docker run --rm hangxie/parquet-tools version
v1.47.7
$ podman run --rm ghcr.io/hangxie/parquet-tools version
v1.47.7Prebuilt RPM and deb Packages
RPM and deb package can be found on release page, only amd64/x86_64 and arm64/aarch64 arch are available at this moment, download the proper package and run corresponding installation command:
- On Debian/Ubuntu:
$ sudo dpkg -i parquet-tools_1.47.7_amd64.deb
Preparing to unpack parquet-tools_1.47.7_amd64.deb ...
Unpacking parquet-tools (1.47.7) ...
Setting up parquet-tools (1.47.7) ...- On CentOS/Fedora:
$ sudo rpm -Uhv parquet-tools-1.47.7-1.x86_64.rpm
Verifying... ################################# [100%]
Preparing... ################################# [100%]
Updating / installing...
1:parquet-tools-1.47.7-1 ################################# [100%]Usage
Canceling a Command
Press Ctrl+C to cancel an active command. parquet-tools also handles SIGTERM, allowing long-running reads, writes, and inspection operations to stop cleanly.
Obtain Help
parquet-tools provides help information through -h flag, whenever you are not sure about a parameter for a command, just add -h to the end of the line then it will give you all available options, for example:
$ parquet-tools inspect -h
Usage: parquet-tools inspect <uri> [flags]
Inspect Parquet file structure in detail.
Arguments:
<uri> URI of Parquet file.
Flags:
-h, --help Show context-sensitive help.
--row-group=INDEX Row group index to inspect.
--column-chunk=INDEX Column chunk index to inspect.
--page=INDEX Page index to inspect.
--anonymous (S3, GCS, and Azure only) object is publicly accessible.
--http-extra-headers= (HTTP URI only) extra HTTP headers.
--http-ignore-tls-error (HTTP and S3 URI) ignore TLS error.
--http-multiple-connection (HTTP URI only) use multiple HTTP connection.
--object-version="" (S3, GCS, and Azure only) object version.
Most commands can output JSON format result which can be processed by utilities like jq or JSON parser online.
Parquet File Location
parquet-tools can read and write parquet files from these locations:
- file system
- AWS Simple Storage Service (S3) bucket
- Google Cloud Storage (GCS) bucket
- Azure Storage Container
- HDFS file
parquet-tools can read parquet files from these locations:
- HTTP/HTTPS URL
[!IMPORTANT]
You need to have proper permission on the file you are going to process.
File System
For files from the file system, you can specify file:// scheme or just ignore it:
$ parquet-tools row-count testdata/good.parquet
3
$ parquet-tools row-count file://testdata/good.parquet
3
$ parquet-tools row-count file://./testdata/good.parquet
3An existing local path containing a colon, such as foo:bar.parquet, is also
recognized as a file when the prefix is not a supported location scheme. To
make the location explicit, prefix a relative path with file://./:
$ cp testdata/good.parquet foo:bar.parquet
$ parquet-tools row-count foo:bar.parquet
3
$ parquet-tools row-count file://./foo:bar.parquet
3
$ rm foo:bar.parquetWhen writing a new file whose path contains a colon, use the explicit
file://./ form because the output path does not exist yet.
Supported location schemes always take precedence over identically named local
paths. This also applies to malformed URLs: for example,
s3://somewhere/%zz reports an invalid URL escape instead of reading a local
file at that path. To address that path explicitly as a local file, use the
file://./ form and URL-escape special characters, such as
file://./s3://somewhere/%25zz.
S3 Bucket
Use full S3 URL to indicate S3 object location, it starts with s3://. You need to make sure you have permission to read or write the S3 object, the easiest way to verify that is using AWS cli:
$ aws sts get-caller-identity
{
"UserId": "REDACTED",
"Account": "123456789012",
"Arn": "arn:aws:iam::123456789012:user/redacted"
}
aws s3 ls s3://daylight-openstreetmap/parquet/osm_features/release=v1.58/type=way/20241112_191814_00139_grr7u_0041fe64-a5ba-4375-88bf-ef790dfedfff
2024-11-12 19:26:50 1070476803 20241112_191814_00139_grr7u_0041fe64-a5ba-4375-88bf-ef790dfedfff
$ parquet-tools row-count s3://daylight-openstreetmap/parquet/osm_features/release=v1.58/type=way/20241112_191814_00139_grr7u_0041fe64-a5ba-4375-88bf-ef790dfedfff
7124019If an S3 object is publicly accessible and you do not have AWS credential, you can use --anonymous flag to bypass AWS authentication:
$ aws sts get-caller-identity
Unable to locate credentials. You can configure credentials by running "aws configure".
$ aws s3 --no-sign-request ls s3://daylight-openstreetmap/parquet/osm_features/release=v1.58/type=way/20241112_191814_00139_grr7u_0041fe64-a5ba-4375-88bf-ef790dfedfff
2024-11-12 19:26:50 1070476803 20241112_191814_00139_grr7u_0041fe64-a5ba-4375-88bf-ef790dfedfff
$ parquet-tools row-count --anonymous s3://daylight-openstreetmap/parquet/osm_features/release=v1.58/type=way/20241112_191814_00139_grr7u_0041fe64-a5ba-4375-88bf-ef790dfedfff
7124019
S3-compatible object stores such as RustFS, MinIO, Ceph, Garage, SeaweedFS,
Cloudflare R2, and LocalStack can be configured with the standard AWS SDK
endpoint environment variables.
AWS_ENDPOINT_URL_S3 affects only S3 and takes precedence over the global
AWS_ENDPOINT_URL. When either setting selects a custom endpoint,
parquet-tools uses path-style bucket addressing and does not query Amazon S3
for the bucket region.
For example, to access a local MinIO server:
$ export AWS_ENDPOINT_URL_S3=http://localhost:9000
$ export AWS_REGION=us-east-1
$ export AWS_ACCESS_KEY_ID=minioadmin
$ export AWS_SECRET_ACCESS_KEY=minioadmin
$ parquet-tools row-count s3://example-bucket/example.parquetThe endpoint and region can also come from the standard AWS shared config
file. Use --anonymous for an S3-compatible endpoint that does not require
signed requests.
Optionally, you can specify object version by using --object-version when you perform read operation (like cat, row-count, schema, etc.) for S3, parquet-tools will access current version if this parameter is omitted.
If version for the S3 object does not exist or bucket does not have version enabled, parquet-tools will report error:
$ parquet-tools row-count s3://daylight-openstreetmap/parquet/osm_features/release=v1.58/type=way/20241112_191814_00139_grr7u_0041fe64-a5ba-4375-88bf-ef790dfedfff --object-version non-existent-version
parquet-tools: error: unable to open file [s3://daylight-openstreetmap/parquet/osm_features/release=v1.58/type=way/20241112_191814_00139_grr7u_0041fe64-a5ba-4375-88bf-ef790dfedfff]: new s3 reader with client: open s3 object: head object: operation error S3: HeadObject, https response error StatusCode: 400, RequestID: TTBSQ5AW6JDA84CR, HostID: I5fmnGiUtkrxDRblY7Nkww10Hu6wc9IyNnk/nA825PEU6Yu7qcZpmXUFoQ7E3nESctHIACjjuEE=, api error BadRequest: Bad Request[!TIP]
According to HeadObject and GetObject, status code for non-existent object or version will be 403 instead of 404 if the caller does not have permission to ListBucket, or return 400 if bucket does not have version enabled.
[!NOTE]
S3 bucket names containing dots (e.g.,my.bucket.name) cause TLS certificate validation failures when accessed via virtual-hosted style URLs because AWS's wildcard certificate*.s3.amazonaws.comdoes not cover multi-level subdomains. Use the--http-ignore-tls-errorflag to access such buckets. AWS has postponed deprecation of path-style access multiple times, but virtual-hosted style remains the default for newer buckets.
parquet-tools loads only necessary data from S3 bucket, for most cases it is footer only, so it is much faster than downloading the file from S3 bucket and running parquet-tools on a local file. Size of the S3 object used in above sample is more than 1GB, but the row-count command takes just several seconds to finish.
GCS Bucket
Use full gsutil URI to point to GCS object location, it starts with gs://. You need to make sure you have permission to read or write to the GCS object, either use application default or GOOGLE_APPLICATION_CREDENTIALS, you can refer to Google Cloud document for more details.
$ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service/account/key.json
$ parquet-tools import -s testdata/csv.source -m testdata/csv.schema gs://REDACTED/csv.parquet
$ parquet-tools row-count gs://REDACTED/csv.parquet
7Similar to S3, parquet-tools downloads only necessary data from GCS bucket.
If the GCS object is publicly accessible, you can use --anonymous option to indicate that anonymous access is expected:
$ parquet-tools row-count gs://cloud-samples-data/bigquery/us-states/us-states.parquet
parquet-tools: error: failed to create GCS client: dialing: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
$ parquet-tools row-count --anonymous gs://cloud-samples-data/bigquery/us-states/us-states.parquet
50Optionally, you can specify object generation by using --object-version when you perform read operation (like cat, row-count, schema, etc.), parquet-tools will access latest generation if this parameter is omitted.
$ parquet-tools row-count --anonymous gs://cloud-samples-data/bigquery/us-states/us-states.parquet
50
$ parquet-tools row-count --anonymous --object-version=-1 gs://cloud-samples-data/bigquery/us-states/us-states.parquet
50parquet-tools reports error on invalid or non-existent generations:
$ parquet-tools row-count --anonymous --object-version=123 gs://cloud-samples-data/bigquery/us-states/us-states.parquet
parquet-tools: error: unable to open file [gs://cloud-samples-data/bigquery/us-states/us-states.parquet]: create new reader: storage: object doesn't exist: googleapi: Error 404: No such object: cloud-samples-data/bigquery/us-states/us-states.parquet, notFound
$ parquet-tools row-count --anonymous --object-version=foo-bar gs://cloud-samples-data/bigquery/us-states/us-states.parquet
parquet-tools: error: unable to open file [gs://cloud-samples-data/bigquery/us-states/us-states.parquet]: invalid GCS generation [foo-bar]: strconv.ParseInt: parsing "foo-bar": invalid syntaxAzure Storage Container
parquet-tools accepts all common spellings of an Azure blob URI, following the HDFS URL format:
- one of these schemes:
wasbs://, the blob-style scheme used by HDFS, with wasb:// as its alias
* abfss://, the filesystem-style scheme used by ADLS Gen2, Databricks, Synapse, and Fabric, with abfs:// as its alias
* az://, the scheme used by adlfs
- container as user name, followed by
- storage account as host, followed by
- blob name as path
[!NOTE]
Althoughwasb://andabfs://mean plain HTTP in Hadoop,parquet-toolstreats them as naming aliases only: all Azure schemes use HTTPS.
The host can carry either the blob endpoint suffix .blob.core.windows.net or the DFS endpoint suffix .dfs.core.windows.net: ADLS Gen2 serves the same data on both, and parquet-tools translates a DFS host to the blob host that the Azure SDK talks to. These URIs all refer to the same blob:
wasbs://laborstatisticscontainer@azureopendatastorage.blob.core.windows.net/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet
> abfss://laborstatisticscontainer@azureopendatastorage.dfs.core.windows.net/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet
> az://laborstatisticscontainer@azureopendatastorage.dfs.core.windows.net/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet
means the parquet file is at:
- storage account
azureopendatastorage - container
laborstatisticscontainer - blob
lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet
az:// additionally accepts a shorthand form that omits the storage account, which then has to come from the AZURE_STORAGE_ACCOUNT_NAME environment variable:
$ AZURE_STORAGE_ACCOUNT_NAME=azureopendatastorage parquet-tools row-count --anonymous az://laborstatisticscontainer/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet
6582726
$ parquet-tools row-count --anonymous az://laborstatisticscontainer/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet
parquet-tools: error: unable to open file [az://laborstatisticscontainer/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet]: az:// URI without a storage account requires environment variable AZURE_STORAGE_ACCOUNT_NAME to name itparquet-tools uses AZURE_STORAGE_ACCESS_KEY environment variable to identify access:
$ AZURE_STORAGE_ACCESS_KEY=REDACTED parquet-tools import -s testdata/csv.source -m testdata/csv.schema wasbs://[email protected]/test/csv.parquet
$ AZURE_STORAGE_ACCESS_KEY=REDACTED parquet-tools row-count wasbs://[email protected]/test/csv.parquet
7If the blob is publicly accessible, either unset AZURE_STORAGE_ACCESS_KEY or use --anonymous option to indicate that anonymous access is expected:
$ AZURE_STORAGE_ACCESS_KEY= parquet-tools row-count wasbs://laborstatisticscontainer@azureopendatastorage.blob.core.windows.net/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet
6582726
$ parquet-tools row-count --anonymous wasbs://laborstatisticscontainer@azureopendatastorage.blob.core.windows.net/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet
6582726Optionally, you can specify object version by using --object-version when you perform read operation (like cat, row-count, schema, etc.) for Azure blob, parquet-tools will access current version if this parameter is omitted.
[!NOTE]
Azure blob returns different errors for non-existent version and invalid version id:
$ parquet-tools row-count --anonymous wasbs://laborstatisticscontainer@azureopendatastorage.blob.core.windows.net/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet --object-version foo-bar
parquet-tools: error: unable to open file [wasbs://laborstatisticscontainer@azureopendatastorage.blob.core.windows.net/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet]: HEAD https://azureopendatastorage.blob.core.windows.net/laborstatisticscontainer/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet
--------------------------------------------------------------------------------
RESPONSE 400: 400 Value for one of the query parameters specified in the request URI is invalid.
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
$ parquet-tools row-count --anonymous wasbs://laborstatisticscontainer@azureopendatastorage.blob.core.windows.net/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet --object-version 2025-05-20T01:27:08.0552942Z
parquet-tools: error: unable to open file [wasbs://laborstatisticscontainer@azureopendatastorage.blob.core.windows.net/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet]: HEAD https://azureopendatastorage.blob.core.windows.net/laborstatisticscontainer/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet
--------------------------------------------------------------------------------
RESPONSE 404: 404 The specified blob does not exist.
ERROR CODE: BlobNotFound
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------Similar to S3 and GCS, parquet-tools downloads only necessary data from blob.
HDFS File
parquet-tools can read and write files under HDFS with schema hdfs://username@hostname:port/path/to/file, if username is not provided then current OS user will be used.
$ parquet-tools import -f jsonl -m testdata/jsonl.schema -s testdata/jsonl.source hdfs://localhost:9000/temp/good.parquet
parquet-tools: error: failed to create JSON writer: failed to open HDFS source [hdfs://localhost:9000/temp/good.parquet]: create /temp/good.parquet: permission denied
$ parquet-tools import -f jsonl -m testdata/jsonl.schema -s testdata/jsonl.source hdfs://root@localhost:9000/temp/good.parquet
$ parquet-tools row-count hdfs://localhost:9000/temp/good.parquet
7Similar to cloud storage, parquet-tools downloads only necessary data from HDFS.
HTTP Endpoint
parquet-tools supports URI with http or https scheme, the remote server needs to support Range header, particularly with unit of bytes.
HTTP endpoint does not support write operation so it cannot be used as destination of import, merge, or split command.
These options can be used along with HTTP endpoints:
--http-multiple-connectionwill enable dedicated transport for concurrent requests,parquet-toolswill establish multiple TCP connections to remote server. This may or may not have performance impact depending on how remote server handles concurrent connections, it is recommended to leave it to defaultfalsevalue for all commands exceptcat, and test performance carefully withcatcommand.--http-extra-headersin the format ofkey1=value1,key2=value2,..., they will be used as extra HTTP headers, a use case is to provideAuthorizationheader or JWT token that is required by remote server.--http-ignore-tls-errorwill ignore TLS errors, this is generally a bad idea.
$ parquet-tools row-count https://azureopendatastorage.blob.core.windows.net/laborstatisticscontainer/lfs/part-00000-tid-6312913918496818658-3a88e4f5-ebeb-4691-bfb6-e7bd5d4f2dd0-63558-c000.snappy.parquet
6582726
$ parquet-tools size https://dpla-provider-export.s3.amazonaws.com/2021/04/all.parquet/part-00000-471427c6-8097-428d-9703-a751a6572cca-c000.snappy.parquet
4632041101Similar to S3 and other remote endpoints, parquet-tools downloads only necessary data from remote server through Range header.
[!TIP]
parquet-toolswill use HTTP/2 if remote server supports this, however you can disable this if things are not working well by setting environment variableGODEBUGtohttp2client=0:
$ parquet-tools row-count https://...
2022/09/05 09:54:52 protocol error: received DATA after END_STREAM
2022/09/05 09:54:52 protocol error: received DATA after END_STREAM
2022/09/05 09:54:53 protocol error: received DATA after END_STREAM
2022/09/05 09:54:53 protocol error: received DATA after END_STREAM
2022/09/05 09:54:53 protocol error: received DATA after END_STREAM
2022/09/05 09:54:53 protocol error: received DATA after END_STREAM
2022/09/05 09:54:53 protocol error: received DATA after END_STREAM
2022/09/05 09:54:53 protocol error: received DATA after END_STREAM
2022/09/05 09:54:53 protocol error: received DATA after END_STREAM
18141856
$ GODEBUG=http2client=0 parquet-tools row-count https://...
18141856
Reading Encrypted Parquet Files
Read commands support AES-GCM encrypted Parquet files with explicit base64-encoded keys.
Note on KMS: KMS (Key Management Service) use cases are not directly supported. If your files are protected by a KMS, you must manually retrieve the key value from the KMS provider and provide it through the CLI parameters below.
All supplied key and AAD values must be standard base64 with padding (RFC 4648 §4). URL-safe and unpadded variants are rejected.
--footer-key: AES-128, AES-192, or AES-256 footer key.--column-key: column key incolumn.path=base64keyform; repeat the flag for multiple columns.--aad-prefix: base64-encoded AAD prefix for files that require the reader to supply it.
. as their delimiter by default. Set --field-delimiter to use another non-empty string that does not contain = (the assignment separator). This is useful when a field name contains a literal dot: for example, --field-delimiter=/ --column-key 'user.info/ssn=KEY' addresses the nested field ssn under a field literally named user.info. The delimiter applies to --column-key, --writer-column-key, --field-encoding, --field-compression, --field-bloom-filter, and column paths in key files.
For example:
$ parquet-tools cat --footer-key MDEyMzQ1Njc4OTAxMjM0NQ== encrypted.parquet
$ parquet-tools cat \
--footer-key MDEyMzQ1Njc4OTAxMjM0NQ== \
--column-key double_field=MTIzNDU2Nzg5MDEyMzQ1MA== \
--column-key float_field=MTIzNDU2Nzg5MDEyMzQ1MQ== \
encrypted.parquet
$ parquet-tools cat \
--footer-key MDEyMzQ1Njc4OTAxMjM0NQ== \
--aad-prefix dGVzdGVy \
encrypted.parquet
To keep keys out of shell history and ps output, use --key-file instead:
$ parquet-tools cat --key-file ~/.parquet/keys.json encrypted.parquetThe file is JSON with all fields optional:
{
"footer_key": "base64-encoded AES-128/192/256 key",
"aad_prefix": "base64-encoded AAD prefix",
"column_keys": {
"user.email": "base64-encoded AES key",
"user.ssn": "base64-encoded AES key"
}
}CLI flags take precedence over file values. --column-key flags are merged with column_keys from the file; CLI wins per column path. Unknown fields and trailing data in the JSON file are rejected. Set the file's permissions to 600 to prevent other users from reading it:
chmod 600 ~/.parquet/keys.jsonThese flags — including --key-file — are available on cat, schema, meta, size, row-count, inspect, split, retype, merge, and transcode. For merge, all encrypted source files must use the same supplied key set. To merge files with different keys, first strip encryption by transcoding each file to a plain Parquet file, then merge the plain outputs.
[!NOTE]
Partial keys and plaintext-signed footers. Files with a plaintext-signed footer (PAR1 with per-column encryption) expose structural metadata — row counts, sizes, schema, encodings, andkey_metadatahints — without any keys. Commands that only read footer/column metadata (schema,row-count,size,meta,inspectat file or row-group level) succeed on such files even when some or all column keys are missing. Missing-key errors are deferred to the moment an encrypted column's data or page-level details are actually requested (e.g.cat, orinspect --column-chunk Non an encrypted column). Files with an encrypted footer (PARE) still require the footer key to read anything.
Writing Encrypted Parquet Files
Writer-encryption flags are available on import, transcode, split, merge, and retype. They accept explicit base64-encoded AES keys; KMS and key_metadata resolution are not supported when writing, so provide the raw key values through CLI flags. For split, the same encryption settings apply to every output file.
Command-line arguments can be visible in shell history and process listings. Avoid passing production key material directly on shared systems; use short-lived shells, clear shell history, and prefer local operational controls that limit process inspection.
All writer keys must be standard base64 with padding (RFC 4648 §4) and must decode to 16 bytes (AES-128), 24 bytes (AES-192), or 32 bytes (AES-256). URL-safe and unpadded variants are rejected.
--writer-footer-key is required for every encryption mode. It encrypts the footer; with --plaintext-footer it signs the footer with AES-GCM instead. It is also the key used for any column whose --writer-column-key directive selects the footer key, and for unlisted columns when --encrypt-all-columns is set.
--writer-column-key column.path=VALUE selects per-column encryption for the listed column. column.path is the file-schema path of a leaf column without the schema root (e.g. Parent.Child, not parquet_go_root.Parent.Child) and uses --field-delimiter between components. VALUE is either a base64-encoded AES key (the column gets its own dedicated key) or the literal @footer-key (the column is encrypted with --writer-footer-key). The leading @ is outside the base64 alphabet, so the sentinel cannot collide with a key value.
By default, columns not listed in --writer-column-key are written as plaintext. Set --encrypt-all-columns to encrypt every leaf column not otherwise listed with --writer-footer-key.
To keep writer keys out of shell history and ps output, use --writer-key-file instead:
$ parquet-tools import -f csv \
-s testdata/csv.source \
-m testdata/csv.schema \
--writer-key-file ~/.parquet/writer-keys.json \
output.parquet--writer-key-file uses the same JSON schema as --key-file. aad_prefix is accepted but ignored by the writer:
{
"footer_key": "base64-encoded AES-128/192/256 key",
"column_keys": {
"col.path": "base64-encoded AES key",
"other.col": "@footer-key"
}
}CLI flags override file values. --writer-column-key flags merge with column_keys; CLI wins per column path. Unknown fields and trailing data in the JSON file are rejected. Recommend chmod 600 on the file.
Supported write scenarios:
| Scenario | Flags | Result |
| -------- | ----- | ------ |
| Encrypted footer, all columns plaintext | --writer-footer-key | Encrypted PARE footer; every column ships plaintext (the footer key is needed to read footer metadata but not column data) |
| Encrypted footer, some columns with dedicated keys | --writer-footer-key, one or more --writer-column-key column.path=base64key | Encrypted PARE footer; listed columns use their dedicated keys; unlisted columns ship plaintext |
| Encrypted footer, some columns use the footer key | --writer-footer-key, one or more --writer-column-key column.path=@footer-key | Encrypted PARE footer; listed columns are encrypted with the footer key; unlisted columns ship plaintext |
| Encrypted footer, every column encrypted | --writer-footer-key, --encrypt-all-columns | Encrypted PARE footer; every column is encrypted with the footer key |
| Encrypted footer, mixed dedicated keys + footer-key fill-in | --writer-footer-key, --writer-column-key column.path=base64key (one or more), --encrypt-all-columns | Encrypted PARE footer; listed columns use their dedicated keys; unlisted columns use the footer key |
| Plaintext footer, some columns encrypted | --writer-footer-key, one or more --writer-column-key column.path=base64key (or =@footer-key), --plaintext-footer | Plaintext PAR1 footer signed by the footer key; listed columns are encrypted; unlisted columns ship plaintext |
| Plaintext footer, every column encrypted | --writer-footer-key, --encrypt-all-columns, --plaintext-footer | Plaintext PAR1 footer signed by the footer key; every column is encrypted with the footer key |
| Plaintext footer, no columns encrypted (sign-only) | --writer-footer-key, --plaintext-footer | Plaintext PAR1 footer signed by the footer key; every column ships plaintext. Provides footer integrity (tamper detection of schema, statistics, offsets) without confidentiality |
[!NOTE]
Default treatment of unlisted columns is plaintext. This mirrors the underlying parquet-go writer default. If you want every column encrypted with the footer key (the previous all-or-nothing behavior), add--encrypt-all-columns.--plaintext-footerno longer implies any column encryption on its own; without column flags it produces a sign-only file where the footer signature still proves the schema and statistics have not been altered.
[!NOTE]
Encrypted footer with all columns plaintext (--writer-footer-keyalone) is permitted by the Parquet spec, but does not really make sense (personal opinion). The footer key is required to read the footer metadata, yet the column data itself is exposed in the clear — anyone with file access can scan column values without ever obtaining the footer key. If your goal is to protect data, pair--writer-footer-keywith--writer-column-keyand/or--encrypt-all-columns.
Examples:
# Encrypted footer; every column ships plaintext.
$ parquet-tools import -f csv \
-s testdata/csv.source \
-m testdata/csv.schema \
--writer-footer-key MDEyMzQ1Njc4OTAxMjM0NQ== \
/tmp/encrypted-footer.parquet
Encrypted footer; Bool gets its own key, every other column ships plaintext.
$ parquet-tools import -f csv \
-s testdata/csv.source \
-m testdata/csv.schema \
--writer-footer-key MDEyMzQ1Njc4OTAxMjM0NQ== \
--writer-column-key Bool=MTIzNDU2Nzg5MDEyMzQ1MA== \
/tmp/mixed-columns.parquet
Encrypted footer; every column encrypted with the footer key.
$ parquet-tools import -f csv \
-s testdata/csv.source \
-m testdata/csv.schema \
--writer-footer-key MDEyMzQ1Njc4OTAxMjM0NQ== \
--encrypt-all-columns \
/tmp/encrypted-all.parquet
Plaintext signed footer; shoe_name is encrypted with the footer key; other
columns ship plaintext.
$ parquet-tools transcode \
-s testdata/good.parquet \
--writer-footer-key MDEyMzQ1Njc4OTAxMjM0NQ== \
--writer-column-key shoe_name=@footer-key \
--plaintext-footer \
/tmp/plaintext-footer.parquet--encryption-algorithm controls the Parquet modular encryption algorithm used when --writer-footer-key is set. It does not enable encryption on its own: with no other encryption flag the file is written unencrypted, and the other encryption flags (--plaintext-footer, --encrypt-all-columns, --writer-column-key) all require --writer-footer-key.
AES-GCM-V1(default): authenticates every encrypted module.AES-GCM-CTR-V1: uses AES-CTR for page bodies. This has lower overhead, but page body tampering is not detected by the cipher.
$ parquet-tools transcode \
-s encrypted-source.parquet \
--key-file ~/.parquet/reader-keys.json \
--writer-key-file ~/.parquet/writer-keys.json \
re-encrypted-output.parquetFile Format Options
This section describes format options for commands that write Parquet files, including compression, data page version, and encoding settings.
Compression Codecs
The --compression / -z parameter sets the file-level default compression codec for commands that expose physical write options: import and transcode. Different compression codecs offer different trade-offs between compression ratio, speed, and compatibility.
The default codec is used only for columns that do not already specify a codec through a more specific setting:
import: column compression in the input schema takes precedence over--compression.transcode:--field-compression field.path=CODECtakes precedence over--compression; columns without a field-specific codec use the file-level default.
BROTLI- Excellent compression ratio with moderate speed (Note: not supported by most Java tools)GZIP- Better compression ratio, slower than SNAPPYLZ4_RAW- LZ4 raw format, fast compression/decompressionSNAPPY- Fast compression/decompression (default forimportandtranscode)UNCOMPRESSED- No compressionZSTD- Excellent compression ratio with good speed
LZ4- Deprecated in Parquet format. UseLZ4_RAWinstead for LZ4 compression. Written with the Hadoop block framing that parquet-mr and Arrow read; files written by parquet-tools v1.54.3 and earlier carry the LZ4 frame format instead, and both are still read
LZO- Not supported. See parquet-go#150 for details.
[!TIP]
Choose compression codecs based on your use case:
- SNAPPY (default): Best for balanced performance and moderate compression
- ZSTD or BROTLI: Best for storage optimization when compression ratio is priority
- GZIP: Good compression ratio with wide compatibility
- LZ4_RAW: Best for write-heavy workloads requiring maximum speed
- UNCOMPRESSED: Best for debugging or when data is already compressed
Compression Levels
The --compression-level option is available only for import and transcode. It accepts repeatable or comma-separated CODEC=LEVEL values:
$ parquet-tools transcode -s input.parquet --compression-level GZIP=9,ZSTD=3 output.parquetCompression levels are file-level writer settings, not per-column settings. There is currently no per-column compression-level control; see parquet-go#270.
The codec name in CODEC=LEVEL selects which compression codec the level applies to anywhere in the output file. It does not need to match --compression; it is used for any output column that is eventually written with that codec:
import: a column may use the file-level--compressioncodec, or a codec specified in the schema.transcode: a column may use the file-level--compressioncodec, or a codec specified with--field-compression.
3 for any column that uses ZSTD through the schema or a field-specific compression override:
$ parquet-tools transcode -s input.parquet -z GZIP --compression-level GZIP=9,ZSTD=3 output.parquetField-specific compression can therefore be paired with matching file-level compression levels:
$ parquet-tools transcode -s input.parquet \
--field-compression shoe_brand=GZIP \
--field-compression shoe_name=ZSTD \
--compression-level GZIP=9,ZSTD=3 \
output.parquetIn this example, GZIP-compressed columns use level 9 and ZSTD-compressed columns use level 3. Columns using codecs without a matching --compression-level value use the default level for that codec. A level for a codec that no output column uses is accepted and has no effect. Repeatable flags are processed left to right, and later values override earlier ones for the same codec.
Level validation:
| Codec | Accepted levels | Notes |
| -------- | --------------- | ------------------------------------ |
| GZIP | 1-9 | Parsed and range-checked |
| ZSTD | 1-22 | Parsed and range-checked |
| BROTLI | 0-11 | Parsed and range-checked |
| LZ4 | 1-9 | Deprecated Parquet codec |
| LZ4_RAW | 0-9 | Parsed and range-checked |
SNAPPY and UNCOMPRESSED do not accept compression levels. Invalid formats, unsupported codecs, and out-of-range values fail before writing output.
Data Page Version
Use --data-page-version to specify the data page format version. This option is available for import and transcode commands.
Data page version 2 is preferred as it offers better compression efficiency and read performance by separating repetition/definition levels from the data. However, version 1 has wider support among older Parquet readers.
Supported versions:
1- DATA_PAGE format (legacy, compatible with older readers)2- DATA_PAGE_V2 format (default, more efficient, requires Parquet format v2.0 readers)
Writer Tuning Options
These options allow fine-grained control over how Parquet files are written.
Binary Min/Max Bound Length (--binary-min-max-truncate-length):
Sets a target byte limit for min/max statistics and column-index bounds on unannotated binary columns and STRING/UTF8 columns. The default value 0 leaves bounds untruncated. STRING/UTF8 bounds may exceed the target when a shorter bound would not be valid UTF-8; other annotated binary types are not truncated.
Truncated bounds remain safe for filtering and are marked as inexact in Parquet metadata. The option is available on import, merge, retype, split, and transcode.
$ parquet-tools transcode -s input.parquet --binary-min-max-truncate-length 64 output.parquetMaximum Dictionary Size (--max-dictionary-size):
Limits encoded dictionary value bytes per column and row group. The default value 0 uses the writer default of 1 MiB. When a dictionary reaches the limit, dictionary-encoded columns fall back to plain encoding for subsequent values in that row group.
The option is available on import, merge, retype, split, and transcode.
$ parquet-tools transcode -s input.parquet --max-dictionary-size 4194304 output.parquetPage Size (--page-size):
Controls the target size of data pages in bytes. Smaller pages allow more granular access but increase metadata overhead. Default is 1048576 (1 MB).
$ parquet-tools import -f csv -s data.csv -m schema.json --page-size 524288 output.parquetRow Group Size (--row-group-size):
Controls the target size of row groups in bytes. Larger row groups improve compression and reduce metadata overhead, but require more memory during read/write operations. Default is 134217728 (128 MB).
$ parquet-tools transcode -s input.parquet --row-group-size 268435456 output.parquetPage Checksums (--write-crc):
Writes a CRC32 checksum into every data and dictionary page header, letting readers detect silent page corruption. Disabled by default. The option is available on import, merge, retype, split, and transcode.
$ parquet-tools transcode -s input.parquet --write-crc output.parquetInput Validation (--enforce-utf8):
Rejects invalid UTF-8 sequences (such as unpaired surrogate \uXXXX escapes or raw non-UTF-8 bytes) in STRING, UTF8, JSON, and ENUM values while importing JSON or CSV input. By default invalid sequences are replaced with the Unicode replacement character; with this flag set, the write fails instead. The option only affects text parsing in import; it has no effect on commands that read typed Parquet input.
$ parquet-tools import -f json -s data.json -m schema.json --enforce-utf8 output.parquet[!TIP]
- Use smaller --page-size for better random access performance at the cost of higher metadata overhead
- Use larger --row-group-size for better compression ratios, but ensure sufficient memory is available
Encoding
Parquet supports various encodings for different data types. Encoding can be specified in the schema file (f
... (README truncated for length)