Pandas

Pandas is a fast, powerful, flexible, and easy-to-use open-source data analysis and manipulation tool built on top of Python. Since QuestDB 10.0 the QuestDB Python client covers both directions natively over QWP: db.dataframe() ingests a whole frame column by column, and query results stream back as Arrow batches that materialize into a DataFrame with to_pandas().

Both paths are columnar end to end, so there is no SQLAlchemy, ODBC, or ConnectorX layer to install and no row-by-row conversion in the middle.

Prerequisites​

  • QuestDB 10.0 or later, running and accessible. See the quick start.
  • Python 3.10 or later.
  • The client and pandas:
python3 -m pip install -U questdb pandas

pyarrow is optional. to_pandas() and iter_pandas() work without it, and it is only needed if you also want to_arrow() or pyarrow-backed dtypes.

Query into a DataFrame​

db.query() returns a result that streams Arrow record batches from the server. Call to_pandas() to materialize the whole result:

import questdb

with questdb.connect("ws::addr=localhost:9000;") as db:
    with db.query(
        "SELECT timestamp, symbol, price, amount FROM trades "
        "WHERE timestamp IN '$now-1h..$now'"
    ) as result:
        df = result.to_pandas()

print(df.head())

Bind values with $1..$N placeholders instead of interpolating them into the SQL string:

df = db.query(
    "SELECT * FROM trades WHERE symbol = $1 AND price > $2",
    ["ETH-USDT", 2615.0],
).to_pandas()

Stream large results​

to_pandas() holds the complete result in memory. For results that do not fit comfortably, iterate batch by batch with iter_pandas() and reduce as you go:

import questdb

with questdb.connect("ws::addr=localhost:9000;") as db:
    with db.query("SELECT price, amount FROM trades") as result:
        notional = sum(
            (chunk["price"] * chunk["amount"]).sum()
            for chunk in result.iter_pandas()
        )

print(notional)

A result is single-use and must stay on the thread that created it. Use a with block, or call close(), so the connection returns to the pool.

Types and nulls​

SYMBOL columns arrive as a Categorical sharing one dictionary across batches, which keeps them compact. INT and LONG become plain int32 / int64 when the column has no nulls and nullable Int32 / Int64 with pd.NA when it does. QuestDB's sentinel values, such as NaN doubles and INT64_MIN longs, are decoded as nulls rather than leaking as magic numbers.

to_pandas(dtype_backend="pyarrow") selects pyarrow-backed dtypes instead, matching the pd.read_sql convention. See result types and nulls for the full mapping.

Ingest a DataFrame​

db.dataframe() publishes the frame in batches and blocks until the server acknowledges the last one:

import pandas as pd
import questdb

df = pd.DataFrame({
    "symbol": pd.Categorical(["ETH-USDT", "BTC-USDT"]),
    "price": [2615.54, 65432.10],
    "amount": [0.00044, 0.00120],
    "timestamp": pd.to_datetime([
        "2025-01-01T00:00:00Z",
        "2025-01-01T00:00:01Z",
    ]),
})

with questdb.connect("ws::addr=localhost:9000;") as db:
    db.dataframe(df, table_name="trades", symbols=["symbol"], at="timestamp")

symbols defaults to "auto", which maps categorical columns to SYMBOL; pass a list of column names to be explicit. at names the designated timestamp column, or takes a fixed timestamp shared by every row, or questdb.ServerTimestamp to let the server assign one.

note

Naive timestamps, both DataFrame columns and a scalar at, are interpreted as UTC, matching the numpy datetime64 convention. Prefer timezone-aware values throughout.

Columns of float64 numpy arrays become DOUBLE[], and None, NaN, and pd.NA are stored as SQL nulls. A frame the columnar path cannot express raises UnsupportedDataFrameShapeError listing the offending columns. See DataFrame ingestion for batching, retries, and the full parameter set.

Legacy: ILP DataFrame ingestion​

warning

Since QuestDB 10.0 the recommended way to move data between pandas and QuestDB is the native client shown above. The API below is documented for legacy reasons: it encodes the frame as InfluxDB Line Protocol text and only covers ingestion. Use it for existing code, or when talking to a server older than 10.0.

The 4.x-style standalone Sender ships in the same questdb package and implements dataframe() on top of ILP:

import sys

import pandas as pd
from questdb import Sender, QuestDBError


def example(conf: str = "http::addr=localhost:9000;"):
    df = pd.DataFrame({
        "symbol": ["ETH-USDT", "BTC-USDT"],
        "price": [2615.54, 65432.10],
        "amount": [0.00044, 0.00120],
        "timestamp": [
            pd.Timestamp("2025-01-01 00:00:00", tz="UTC"),
            pd.Timestamp("2025-01-01 00:00:01", tz="UTC"),
        ],
    })
    try:
        with Sender.from_conf(conf) as sender:
            sender.dataframe(
                df,
                table_name="trades",   # Table name to insert into.
                symbols=["symbol"],    # Columns to insert as SYMBOL.
                at="timestamp")        # Designated timestamp column.

    except QuestDBError as e:
        sys.stderr.write(f"Got error: {e}\n")


if __name__ == "__main__":
    example()

The questdb.ingress import path still works as a deprecated alias module, where IngressError remains an alias of QuestDBError. Importing from it raises a DeprecationWarning, so prefer questdb directly as shown above. The legacy API is documented on ReadTheDocs, and the 5.0 migration guide maps each 4.x call to its pooled equivalent.

See also​