Persistent storage & databases
Persistent storage is any method of storing data that remains intact and accessible even after a system is powered off, restarted, or experiences a crash.
Primary storage options
Section titled “Primary storage options”The recommended approach is to use data tables to store relational data, and ducklakes to store massive datasets, with external databases available as resources.
Data tables
Section titled “Data tables”Built-in relational database functionality requiring minimal setup for storing structured data directly within Orvanta.
Ducklake
Section titled “Ducklake”A data lakehouse solution enabling massive amounts of data stored in parquet files on S3 while querying it with natural SQL syntax.
External databases
Section titled “External databases”Connect existing databases through integrations or create custom resource types if your provider isn’t pre-integrated.
Additional storage methods
Section titled “Additional storage methods”Large unstructured data: S3, Cloudflare R2, MinIO, Azure Blob Storage, and Google Cloud Storage are highly scalable and durable object storage services with native Orvanta integration.
Structured SQL data: Postgres-based solutions like Supabase and Neon.tech are recommended for schema-based storage with defined entity relationships.
File persistence: Volumes provide persistent file storage that can be attached to scripts, with automatic syncing to object storage between runs.
NoSQL solutions: MongoDB Atlas, Redis, and Upstash support flexible, non-structured data management.
Internal storage
Section titled “Internal storage”Orvanta offers internal persistence between job executions, but this approach is explicitly marked as not recommended.