Architecture and data exchange

In Orvanta, a workflow is a JSON serializable value in the OpenFlow format that consists of an input spec (similar to Scripts), and a linear sequence of steps, also referred to as modules. Each step consists of either:

1. Reference to a Script from the Orvanta Hub.
2. Reference to a Script in your workspace.
3. Inlined Script in TypeScript (Deno), Python, Go, Bash, SQL or non-supported languages.
4. Trigger scripts which are a kind of Scripts that are meant to be the first step of a scheduled Flow, that watch for external events and early exit the Flow if there are no new events.
5. For loop that iterates over elements and triggers the execution of an embedded flow for each element. The list is calculated dynamically as an input transform.
6. Branch to the first subflow that has a truthy predicate (evaluated in-order).
7. Branches to all subflows and collect the results of each branch into an array.
8. Approval/Suspend steps which suspend the flow at no cost until it is resumed by getting an approval/resume signal.
9. Inner flows.

How flows are executed

Each step of a flow is executed as an individual job picked up by a worker. Sequential steps run one after another, while parallel branches are queued simultaneously and can be processed by different workers at the same time. A join step (like a final result aggregation) waits until every branch has completed before it is queued. With enough workers, parallel branches overlap and the total flow duration equals the longest branch — not the sum of all branches.

Input transform

With the mechanism of input transforms, the input of any step can be the output of any previous step, hence every Flow is actually a Directed Acyclic Graph (DAG) rather than a simple sequence. You can refer to the result of any step using its ID.

Every step has an input transform that maps from:

• the Flow input
• any step’s result, not only the previous step’s result
• Resource/Variable.

to the different parameters of this specific step.

It does that using a JavaScript expression that operates in a more restricted setting. That JavaScript uses a restricted subset of the standard library and a few more functions which are the following:

• flow_input: the dict/object containing the different parameters of the Flow itself.
• results.{id}: the result of the step with given ID.
• resource(path): the Resource at path.
• variable(path): the Variable at path.

Using JavaScript in this manner, for every parameter, is extremely flexible and allows Orvanta to be extremely generic in the kind of modules it runs.

JavaScript evaluation engine

Flow expressions (input transforms, branch predicates, for-loop iterators) are evaluated using a JavaScript engine. Two engines are available:

• QuickJS: A lightweight JavaScript engine with 8-16x faster startup times for simple expressions. It uses lazy evaluation — resource() and variable() calls are resolved on demand rather than pre-fetched, which is more efficient for expressions that don’t use all available resources.
• Deno (V8): The full V8-based JavaScript engine with broader compatibility.

On Community Edition, QuickJS is the default engine. To switch back to Deno, set the USE_DENO_FOR_FLOW_EVAL environment variable.

On Enterprise Edition, Deno remains the default. To opt into QuickJS, set the USE_QUICKJS_FOR_FLOW_EVAL environment variable.

Connecting flow steps

For each field, one has the option to write the JavaScript directly or to use the quick connect button if the field maps one to one with a field of the flow_input, a field of the previous_result or of any step.

From the editor, you can directly get:

• Static inputs: fixed values set directly in the step input fields.
• Flow env variables: flow-level constants that can be referenced in any step input using flow_env.VARIABLE_NAME or flow_env["VARIABLE_NAME"].
• Dynamic inputs:
  • using the id associated with the step
  • clicking on the plug logo that will let you pick flow inputs or previous steps’ results (after testing flow or step).

You can connect step inputs automatically using Orvanta AI.

Custom flow states

A state is an object stored as a resource of the resource type state which is meant to persist across distinct executions of the same Script. This is what enables Flows to watch for changes in most event-watching scenarios.

Custom flow states are a way to store data across steps in a flow. You can set and retrieve a value given a key from any step of a flow and it will be available from within the flow globally. That state will be stored in the flow state itself and thus has the same lifetime as the flow job itself.

It’s a powerful escape hatch when passing data as output/input is not feasible and using getResource/setResource has the issue of cluttering the workspace and inconvenient UX.

TypeScript example:

import * as orvanta from "[email protected]"
export async function main(x: string) {
  await orvanta.setFlowUserState("FOO", 42)
  return await orvanta.getFlowUserState("FOO")
}

Python example:

import orvanta
#extra_requirements:
#orvanta==1.297.0
def main(x: str):
    orvanta.set_flow_user_state("foobar", 43)
    return orvanta.get_flow_user_state("foobar")

Shared directory

By default, flows on Orvanta are based on a result basis (see above). A step will take as input the results of previous steps. This works fine for lightweight automation.

For heavier ETLs and any output that is not suitable for JSON, you might want to use the Shared Directory to share data between steps. Steps share a folder at ./shared in which they can store heavier data and pass them to the next step.

Get more details on the Persistent storage & databases dedicated page.