Loading data

Loading a dataset's actual rows (.to_pandas(), .to_polars(), .to_duckdb(), .sql(), .to_csv()) requires a read-scoped API key — unless the dataset is currently keyless (ds.keyless), in which case no key is needed. Create a key from your Aesops account's API keys settings (see also the API's Authentication page).

client = Client(api_key="Aes_...")
ds = client.load_dataset("kenya-housing-prices")

Your own unpublished datasets

load_dataset() also works for a dataset you've uploaded but not published, as long as the client has an API key from the account that owns it. Anyone else gets a not-found error.

pandas

df = ds.to_pandas()

Requires pandas (pip install 'aesops[pandas]').

polars

lf = ds.to_polars()

Requires polars (pip install 'aesops[polars]').

DuckDB

to_duckdb() returns an in-memory DuckDB connection with a data view bound to the dataset. DuckDB's httpfs extension does predicate + projection pushdown, so only the row-groups your query actually touches are fetched:

con = ds.to_duckdb()
con.sql("SELECT county, avg(price) FROM data GROUP BY county").df()

Several datasets in one connection

To query datasets together (joins, etc.), create one connection and pass it in with a view_name. The view is created directly in your connection and is still read lazily from Parquet:

import duckdb

con = duckdb.connect()
housing = client.load_dataset("kenya-housing-prices")
census = client.load_dataset("knbs-census-2019")

housing.to_duckdb(con=con, view_name="housing")
census.to_duckdb(con=con, view_name="census")

con.sql("SELECT * FROM housing JOIN census USING (county)").df()

view_name defaults to the dataset's slug (quote it in SQL if it contains hyphens, e.g. "kenya-housing-prices"). Calling it again for the same name refreshes the view. ds.sql(query, con=con, view_name=...) works the same way. Keep the Client open for as long as you use the connection — see Context manager.

Long sessions

Each view reads from a signed URL that expires after about an hour. ds.sql() and to_pandas() refresh it automatically, but queries you run yourself on a connection (con.sql(...)) can't. If one starts failing with an HTTP 403 error, call ds.to_duckdb(con=con, view_name=...) again — it replaces the view with a fresh URL.

New to DuckDB or DuckDBPyRelation? See the DuckDB primer.

Running SQL directly

ds.sql() skips the intermediate con — same pushdown behavior, reference the dataset as data. Returns a DuckDBPyRelation; call .df() for pandas or .pl() for polars:

ds.sql("SELECT county, avg(price) FROM data GROUP BY 1").df()

# anything more specific than a simple limit — offset, filters, ordering —
# also goes through ds.sql() directly
ds.sql("SELECT * FROM data ORDER BY price DESC LIMIT 20").df()

CSV export

Client-side conversion — no extra server compute, DuckDB reads the Parquet directly and writes the CSV locally:

ds.to_csv("/tmp/housing.csv")

Limiting rows

to_pandas(), to_polars(), and to_csv() all take a limit kwarg. It's pushed down through read_parquet, so it also cuts what crosses the network — not just what lands in the DataFrame or file:

sample = ds.to_pandas(limit=100)
ds.to_polars(limit=100)
ds.to_csv("/tmp/sample.csv", limit=100)