Ducklake

Overview

Ducklake is Orvanta’s lakehouse option for persistent storage: data lives as Parquet files in your S3 (or compatible) bucket rather than in a managed database, while a catalog keeps track of schemas and table versions so you can still query it with ordinary SQL. Reach for it over data tables when the dataset is large enough that you’d rather pay for object storage than for database disk and compute.

Getting started

Prerequisites include:

• Configured workspace storage
• A Postgres or MySQL connection (unless you’re a superuser with access to a custom instance database)

Configuration occurs in workspace settings under “Object storage (S3)” where you can set up a Ducklake instance.

Usage in scripts

Ducklakes are referenced by name, with main as the default that can be omitted.

TypeScript example:

import * as orvanta from 'orvanta-client';

export async function main(user_id: string) {
  let sql = orvanta.ducklake();
  let friend = await sql`SELECT * FROM friend WHERE id = ${user_id}`.fetchOne();
  return friend;
}

Python example:

import orvanta

def main(user_id: str):
  dl = orvanta.ducklake()
  friend = dl.query('SELECT * FROM friend WHERE id = $id', id=user_id).fetch_one()
  return friend

DuckDB example:

ATTACH 'ducklake' AS dl;
USE dl;
SELECT * FROM friend WHERE id = $user_id;

Database manager

The “Explore” button in Ducklake settings opens a database manager for CRUD operations via UI or SQL REPL.

Backend architecture

Ducklake stores metadata in catalog tables and maintains data as columnar Parquet files in S3, enabling efficient analytical processing without loading everything into RAM.

Related

• Data tables: Relational data with minimal setup.
• Large data files: Object storage for unstructured data.