BigQuery DataFrames (BigFrames) ===============================
|GA| |pypi| |versions|
BigQuery DataFrames (also known as BigFrames) provides a Pythonic DataFrame and machine learning (ML) API powered by the BigQuery engine. It provides modules for many use cases, including:
bigframes.pandas_
bigframes.bigquery.ai_
bigframes.ml_
BigQuery DataFrames is an open-source package _.
.. |GA| image:: https://img.shields.io/badge/support-GA-gold.svg :target: https://github.com/googleapis/google-cloud-python/blob/main/README.rst#general-availability .. |pypi| image:: https://img.shields.io/pypi/v/bigframes.svg :target: https://pypi.org/project/bigframes/ .. |versions| image:: https://img.shields.io/pypi/pyversions/bigframes.svg :target: https://pypi.org/project/bigframes/
Why Use BigQuery DataFrames? ----------------------------
BigQuery DataFrames eliminates the "data movement bottleneck." Instead of downloading large datasets to a local environment, BigFrames translates your Python code into optimized SQL executed across BigQuery's distributed compute engine.
- Petabyte-Scale Scalability: Effortlessly process datasets exceeding local memory constraints without provisioning or managing Spark or Ray clusters.
- Familiar Python Ecosystem: Use the pandas operations you already know (`
read_gbq,groupby,merge,pivot_table) directly on BigQuery tables. - Generative AI & Multimodality: Seamlessly leverage Gemini models, AI functions, multimodal processing (audio, video, PDF), vector embeddings, and vector search with
bigframes.bigquery.ai. - Optimized Execution: Take advantage of BigFrames partial ordering mode (
bpd.options.bigquery.ordering_mode = "partial") for considerable performance gains on analytical workloads. - Enterprise Security: Keep your data securely governed within the BigQuery security perimeter.
- Hybrid Flexibility: Seamlessly transition between distributed BigQuery processing and local pandas analysis with
to_pandas()andread_pandas().
Notebooks ~~~~~~~~~
The easiest way to get started is to open a sample notebook in
Colab _
or
BigQuery Studio _.
More details can be found in the BigFrames quickstart _
and in the introduction to notebooks _.
Local environment ~~~~~~~~~~~~~~~~~
To use BigFrames in your local development environment:
- Run
pip install --upgrade bigframesto install the latest version.
- Set up Application Default Credentials
_
- Create a Google Cloud project with the BigQuery API enabled
_.
- Use the
bigframespackage to query data.
import bigframes.pandas as bpd
bpd.options.bigquery.project = your_gcp_project_id # Optional in BQ Studio. bpd.options.bigquery.ordering_mode = "partial" # Recommended for performance. df = bpd.read_gbq("bigquery-public-data.usa_names.usa_1910_2013") print( df.groupby("name") .agg({"number": "sum"}) .sort_values("number", ascending=False) .head(10) .to_pandas() )
Documentation -------------
To learn more about BigQuery DataFrames, visit these pages
- Introduction to BigQuery DataFrames (BigFrames)
_ - Sample notebooks
_ - API reference
_ - Source code (GitHub)
_
BigQuery DataFrames is distributed with the Apache-2.0 license
_.
It also contains code derived from the following third-party packages:
- Ibis
_ - pandas
_ - Python
_ - scikit-learn
_ - XGBoost
_ - SQLGlot
_
_
directory.
Contact Us
----------
For further help and provide feedback, you can email us at
[email protected]