eakmanrq/sqlframe: Turning PySpark Into a Universal DataFrame API

Your PySpark code. Your database.

Run the PySpark DataFrame API directly on your database.
No Spark cluster required.

Quickstart · Documentation · Report an issue

Why SQLFrame?

• Familiar API. Write transformations with PySpark DataFrame operations, joins, and window functions.
• Direct execution. Run your transformations on your database engine.
• Visible SQL. Inspect, debug, and share the queries behind your DataFrame code.

Try it locally

Install SQLFrame with DuckDB. You need Python 3.10 or later; no cloud account or Spark installation is required.

pip install "sqlframe[duckdb]"

from sqlframe.duckdb import DuckDBSession
session = DuckDBSession()
df = session.createDataFrame(
    [("US", 120), ("UK", 80), ("US", 30)],
    ["country", "revenue"],
)
df.groupBy("country").sum("revenue").orderBy("country").show()

+---------+--------------+
| country | sum(revenue) |
+---------+--------------+
|    UK   |      80      |
|    US   |     150      |
+---------+--------------+

To inspect a transformation's SQL, call df.sql(optimize=True). Follow the quickstart for a complete walkthrough.

Choose your engine

| Engine | Install | | :----- | :------ | | DuckDB | pip install "sqlframe[duckdb]" | | BigQuery | pip install "sqlframe[bigquery]" | | Snowflake | pip install "sqlframe[snowflake]" | | Postgres | pip install "sqlframe[postgres]" | | ClickHouse | pip install "sqlframe[clickhouse]" | | Databricks | pip install "sqlframe[databricks]" | | Spark | pip install "sqlframe[spark]" | | Standalone | pip install sqlframe |

Standalone sessions generate SQL without a database connection. For local Parquet support, install sqlframe[standalone].

Community integrations: GizmoSQL · Redshift (pip install "sqlframe[redshift]").

SQLFrame is also available on conda-forge: conda install -c conda-forge sqlframe.

Already using PySpark?

Activate SQLFrame before importing PySpark to run supported transformations on another engine.

from sqlframe import activate
activate("duckdb")
from pyspark.sql import SparkSession
session = SparkSession.builder.getOrCreate()

You can also import SQLFrame directly, as shown in the quickstart. See the activation guide and your engine's API compatibility reference for details.

Keep exploring

• Examples: longer pipelines with aggregations and window functions.
• Generated SQL: formatting, optimization, dialect overrides, and OpenAI enrichment.
• Input and output dialects: configure SQLFrame for your engine's conventions.
• Standalone SQL generation: generate queries without connecting to an engine.