Open BigQuery Functions, SQL Superpowers
โฏ BigFunctions website โฎ
BigFunctions is both:
- Open-Source BigQuery functions (๐ช hello SQL-superpowers) callable without install from any GCP project
- A framework to define, test, deploy, document and monitor your own BigQuery functions. Features:
- ๐ A standard to define BigQuery routines as yaml files be it sql/js/remote UDF or procedures
- ๐
bigfun
CLI (Command Line Interface) to test, deploy, document and monitor these BigQuery routines in your project
BigQuery, with native and remote routines, is really powerful. BigFunctions take the most of that by offering ready-to-use functions with useful features for data-teams.
We believe no-one should be reinventing the wheel and open-source is the best way to fight against that. Hey Data-People! Let's share our work, help each other and inspire from each other. ๐ Play and contribute to BigFunctions!
All BigFunctions represented by a 'yaml' file in bigfunctions folder are automatically deployed in public datasets so that you can call them directly without install from your BigQuery project.
To explore available bigfunctions and to get started, visit BigFunctions website.
You can also deploy any bigfunction in your project! To deploy my_bigfunction defined in bigfunctions/my_bigfunction.yaml file, simply call:
bigfun deploy my_bigfunction
Details about bigfun
command line are given below.
bigfun
CLI (command-line-interface) facilitates BigFunctions development, test, deployment, documentation and monitoring.
Clone the repo and from the repo directory run:
virtualenv venv
. venv/bin/activate
pip install --editable .
$ bigfun --help
Usage: bigfun [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
deploy Deploy BIGFUNCTION
doc Generate, serve and publish documentation
test Test BIGFUNCTION
- Make sure the
gcloud
command is installed on your computer- Activate the application-default account with
gcloud auth application-default login
. A browser window should open, and you should be prompted to log into your Google account. Once you've done that,bigfun
will use your oauth'd credentials to connect to BigQuery through BigQuery python client!- Get permissions to edit a dataset
DATASET
in a projectPROJECT
and to run BigQuery queries in that project.
You should then be able to deploy the function is_email_valid
by simply running
bigfun deploy PROJECT.DATASET.is_email_valid
Test it with ๐ select PROJECT.DATASET.is_email_valid('paul.marcombes@unytics.io')
!
To deploy a remote function (e.g. python function), there are additional requirements in addition to the ones above.
- A Cloud Run service will be deployed to host the code (as seen here). So you must have permissions to deploy a Cloud Run service in your project
PROJECT
.gcloud
CLI will be used directly to deploy the service (usinggcloud run deploy
). Then, make sure you are logged in withgcloud
by calling:gcloud auth login
. A browser window should also open, and you should be prompted to log into your Google account. WARNING: you read correctly: you have to authenticate twice. Once for bigquery python client (to deploy any function including remote as seen above.) and once now to usegcloud
(to deploy a Cloud Run service).- A BigQuery Remote Connection will be created to link BigQuery with the Cloud Run service. You then should have permissions to create a remote connection. BigQuery Connection Admin or BigQuery Admin roles have these permissions.
- A service account will be automatically created by Google along with the BigQuery Remote Connection. BigQuery will use this service account of the remote connection to invoke the Cloud Run service. You then must have the permission to authorize this service account to invoke the Cloud Run service. This permission is provided in the role roles/run.admin
You now should be able to deploy sentiment_score
function by running
bigfun deploy PROJECT.DATASET.sentiment_score
To make this specific function work, you first must enable Google Cloud NLP API in your project PROJECT
. You can then test it with ๐ select PROJECT.DATASET.sentiment_score('This is awesome!')
!
You are more than welcome to contribute by:
- adding a โญ on this repo ๐
- reporting an issue here.
- suggesting new functions here
- adding/improving BigFunctions
- improving the framework and
bigfun
CLI
See contributing instructions for more details.
Contributors