PyAthena is a Python DB API 2.0 (PEP 249) compliant client for Amazon Athena.
- Python
- CPython 2,7, 3.5, 3.6, 3.7
Extra packages:
| Package | Install command | Version |
|---|---|---|
| Pandas | pip install PyAthena[Pandas] |
>=0.24.0 |
| SQLAlchemy | pip install PyAthena[SQLAlchemy] |
>=1.0.0, <2.0.0 |
from pyathena import connect
cursor = connect(aws_access_key_id='YOUR_ACCESS_KEY_ID',
aws_secret_access_key='YOUR_SECRET_ACCESS_KEY',
s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',
region_name='us-west-2').cursor()
cursor.execute("SELECT * FROM one_row")
print(cursor.description)
print(cursor.fetchall())Supported DB API paramstyle is only PyFormat. PyFormat only supports named placeholders with old % operator style and parameters specify dictionary format.
from pyathena import connect
cursor = connect(aws_access_key_id='YOUR_ACCESS_KEY_ID',
aws_secret_access_key='YOUR_SECRET_ACCESS_KEY',
s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',
region_name='us-west-2').cursor()
cursor.execute("""
SELECT col_string FROM one_row_complex
WHERE col_string = %(param)s
""", {'param': 'a string'})
print(cursor.fetchall())if % character is contained in your query, it must be escaped with %% like the following:
Install SQLAlchemy with pip install "SQLAlchemy>=1.0.0, <2.0.0" or pip install PyAthena[SQLAlchemy]. Supported SQLAlchemy is 1.0.0 or higher and less than 2.0.0.
from urllib.parse import quote_plus # PY2: from urllib import quote_plus
from sqlalchemy.engine import create_engine
from sqlalchemy.sql.expression import select
from sqlalchemy.sql.functions import func
from sqlalchemy.sql.schema import Table, MetaData
conn_str = 'awsathena+rest://{aws_access_key_id}:{aws_secret_access_key}@athena.{region_name}.amazonaws.com:443/'\
'{schema_name}?s3_staging_dir={s3_staging_dir}'
engine = create_engine(conn_str.format(
aws_access_key_id=quote_plus('YOUR_ACCESS_KEY_ID'),
aws_secret_access_key=quote_plus('YOUR_SECRET_ACCESS_KEY'),
region_name='us-west-2',
schema_name='default',
s3_staging_dir=quote_plus('s3://YOUR_S3_BUCKET/path/to/')))
many_rows = Table('many_rows', MetaData(bind=engine), autoload=True)
print(select([func.count('*')], from_obj=many_rows).scalar())The connection string has the following format:
If you do not specify aws_access_key_id and aws_secret_access_key using instance profile or boto3 configuration file:
NOTE: s3_staging_dir requires quote. If aws_access_key_id, aws_secret_access_key and other parameter contain special characters, quote is also required.
Minimal example for Pandas DataFrame:
As Pandas DataFrame:
from pyathena import connect
from pyathena.util import as_pandas
cursor = connect(aws_access_key_id='YOUR_ACCESS_KEY_ID',
aws_secret_access_key='YOUR_SECRET_ACCESS_KEY',
s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',
region_name='us-west-2').cursor()
cursor.execute("SELECT * FROM many_rows")
df = as_pandas(cursor)
print(df.describe())If you want to use Pandas DataFrame object directly, you can use PandasCursor.
AsynchronousCursor is a simple implementation using the concurrent.futures package. Python 2.7 uses backport of the concurrent.futures package. This cursor is not DB API 2.0 (PEP 249) compliant.
You can use the AsynchronousCursor by specifying the cursor_class with the connect method or connection object.
It can also be used by specifying the cursor class when calling the connection object's cursor method.
The default number of workers is 5 or cpu number * 5. If you want to change the number of workers you can specify like the following.
The execute method of the AsynchronousCursor returns the tuple of the query ID and the future object.
The return value of the future object is an AthenaResultSet object. This object has an interface that can fetch and iterate query results similar to synchronous cursors. It also has information on the result of query execution.
from pyathena import connect
from pyathena.async_cursor import AsyncCursor
cursor = connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',
region_name='us-west-2',
cursor_class=AsyncCursor).cursor()
query_id, future = cursor.execute("SELECT * FROM many_rows")
result_set = future.result()
print(result_set.state)
print(result_set.state_change_reason)
print(result_set.completion_date_time)
print(result_set.submission_date_time)
print(result_set.data_scanned_in_bytes)
print(result_set.execution_time_in_millis)
print(result_set.output_location)
print(result_set.description)
for row in result_set:
print(row)from pyathena import connect
from pyathena.async_cursor import AsyncCursor
cursor = connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',
region_name='us-west-2',
cursor_class=AsyncCursor).cursor()
query_id, future = cursor.execute("SELECT * FROM many_rows")
result_set = future.result()
print(result_set.fetchall())A query ID is required to cancel a query with the AsynchronousCursor.
NOTE: The cancel method of the future object does not cancel the query.
PandasCursor directly handles the CSV file of the query execution result output to S3. This cursor is to download the CSV file after executing the query, and then loaded into DataFrame object. Performance is better than fetching data with a cursor.
You can use the PandasCursor by specifying the cursor_class with the connect method or connection object.
It can also be used by specifying the cursor class when calling the connection object's cursor method.
The as_pandas method returns a DataFrame object.
Support fetch and iterate query results.
from pyathena import connect
from pyathena.pandas_cursor import PandasCursor
cursor = connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',
region_name='us-west-2',
cursor_class=PandasCursor).cursor()
cursor.execute("SELECT * FROM many_rows")
print(cursor.fetchone())
print(cursor.fetchmany())
print(cursor.fetchall())The DATE and TIMESTAMP of Athena's data type are returned as pandas.Timestamp type.
from pyathena import connect
from pyathena.pandas_cursor import PandasCursor
cursor = connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',
region_name='us-west-2',
cursor_class=PandasCursor).cursor()
cursor.execute("SELECT col_timestamp FROM one_row_complex")
print(type(cursor.fetchone()[0])) # <class 'pandas._libs.tslibs.timestamps.Timestamp'>Execution information of the query can also be retrieved.
from pyathena import connect
from pyathena.pandas_cursor import PandasCursor
cursor = connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',
region_name='us-west-2',
cursor_class=PandasCursor).cursor()
cursor.execute("SELECT * FROM many_rows")
print(cursor.state)
print(cursor.state_change_reason)
print(cursor.completion_date_time)
print(cursor.submission_date_time)
print(cursor.data_scanned_in_bytes)
print(cursor.execution_time_in_millis)
print(cursor.output_location)NOTE: PandasCursor handles the CSV file on memory. Pay attention to the memory capacity.
AsyncPandasCursor is an AsyncCursor that can handle Pandas DataFrame. This cursor directly handles the CSV of query results output to S3 in the same way as PandasCursor.
You can use the AsyncPandasCursor by specifying the cursor_class with the connect method or connection object.
It can also be used by specifying the cursor class when calling the connection object's cursor method.
The default number of workers is 5 or cpu number * 5. If you want to change the number of workers you can specify like the following.
The execute method of the AsynchronousPandasCursor returns the tuple of the query ID and the future object.
The return value of the future object is an AthenaPandasResultSet object. This object has an interface similar to AthenaResultSetObject.
from pyathena import connect
from pyathena.async_pandas_cursor import AsyncPandasCursor
cursor = connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',
region_name='us-west-2',
cursor_class=AsyncPandasCursor).cursor()
query_id, future = cursor.execute("SELECT * FROM many_rows")
result_set = future.result()
print(result_set.state)
print(result_set.state_change_reason)
print(result_set.completion_date_time)
print(result_set.submission_date_time)
print(result_set.data_scanned_in_bytes)
print(result_set.execution_time_in_millis)
print(result_set.output_location)
print(result_set.description)
for row in result_set:
print(row)from pyathena import connect
from pyathena.async_pandas_cursor import AsyncPandasCursor
cursor = connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',
region_name='us-west-2',
cursor_class=AsyncPandasCursor).cursor()
query_id, future = cursor.execute("SELECT * FROM many_rows")
result_set = future.result()
print(result_set.fetchall())This object also has an as_pandas method that returns a DataFrame object similar to the PandasCursor.
from pyathena import connect
from pyathena.async_pandas_cursor import AsyncPandasCursor
cursor = connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',
region_name='us-west-2',
cursor_class=AsyncPandasCursor).cursor()
query_id, future = cursor.execute("SELECT * FROM many_rows")
result_set = future.result()
df = result_set.as_pandas()
print(df.describe())
print(df.head())The DATE and TIMESTAMP of Athena's data type are returned as pandas.Timestamp type.
from pyathena import connect
from pyathena.async_pandas_cursor import AsyncPandasCursor
cursor = connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',
region_name='us-west-2',
cursor_class=AsyncPandasCursor).cursor()
query_id, future = cursor.execute("SELECT col_timestamp FROM one_row_complex")
result_set = future.result()
print(type(result_set.fetchone()[0])) # <class 'pandas._libs.tslibs.timestamps.Timestamp'>As with AsynchronousCursor, you need a query ID to cancel a query.
from pyathena import connect
from pyathena.async_pandas_cursor import AsyncPandasCursor
cursor = connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',
region_name='us-west-2',
cursor_class=AsyncPandasCursor).cursor()
query_id, future = cursor.execute("SELECT * FROM many_rows")
cursor.cancel(query_id)You can attempt to re-use the results from a previously run query to help save time and money in the cases where your underlying data isn't changing. Set the cache_size parameter of cursor.execute() to a number larger than 0 to enable cacheing.
from pyathena import connect
cursor = connect(aws_access_key_id='YOUR_ACCESS_KEY_ID',
aws_secret_access_key='YOUR_SECRET_ACCESS_KEY',
s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',
region_name='us-west-2').cursor()
cursor.execute("SELECT * FROM one_row") # run once
print(cursor.query_id)
cursor.execute("SELECT * FROM one_row", cache_size=10) # re-use earlier results
print(cursor.query_id) # You should expect to see the same Query IDResults will only be re-used if the query strings match exactly, and the query was a DML statement (the assumption being that you always want to re-run queries like CREATE TABLE and DROP TABLE).
The S3 staging directory is not checked, so it's possible that the location of the results is not in your provided s3_staging_dir.
Support Boto3 credentials.
Additional environment variable:
Depends on the following environment variables: