Pipeline

The Pipeline is the core state management system of Joynare Nexus. It is a dynamic data structure that carries state from the beginning of a flow execution to the end.

1. Core Concept

Think of the Pipeline as a shared "blackboard" for a Flow Service. When a flow starts, a new pipeline is created. As each step executes:

Visualization: The Data Journey

As each step executes:

1. It reads specific data it needs from the pipeline.
2. It performs its work (calls a service, computes a value, etc.).
3. It writes its results back into the pipeline.

Visualization: State Accumulation

This diagram shows how the internal state of the pipeline grows as each step completes.

---

2. Pipeline Lifecycle

Step Execution Cycle

Each step interaction with the pipeline follows this pattern:

Initialization

When a flow is invoked (via API or CLI), the input JSON is loaded into the pipeline.

• If a pipeline.input schema is defined in the flow YAML, only those fields are strictly validated, but all input data is available.

Execution

As the engine processes steps sequentially:

• Invoke Steps: Take data from the pipeline, call a service, and merge results back.
• Map Steps: Directly manipulate the pipeline by setting literal values or dropping variables.
• Branch/Loop Steps: Use pipeline data to decide execution paths.

Finalization

When the last step finishes, the engine looks at the pipeline's output schema. It filters the final state of the pipeline to return only the requested fields to the caller.

---

3. Variable Substitution (${field})

You can reference pipeline data anywhere in your step definitions using the ${} syntax.

Simple References

input:
  text: "${username}" # Reads the 'username' field from the pipeline

Template Strings (Interpolation)

You can mix variables with literal text:

input:
  url: "https://api.example.com/v1/users/${userId}/profile"

Numbers and Booleans

The engine is smart enough to preserve types for pure references:

• ${age} where age is 30 (int) -> remains an int in the service input.
• "Age is ${age}" -> becomes the string "Age is 30".

---

4. Input Mapping Modes

When calling a service (built-in or adapter), there are three ways to get data from the pipeline into the service's input fields.

A. Automatic Mapping

If the field name in the pipeline matches the field name expected by the service, it is mapped automatically.

# Pipeline has "text"
# Service 'Uppercase' expects "text"
steps:
  - type: "invoke"
    service: "system.string:Uppercase"
    # No input block needed!

B. Explicit Mapping

Use the ${} syntax to map a differently named pipeline field to a service input.

# Pipeline has "customerName"
# Service 'Uppercase' expects "text"
steps:
  - type: "invoke"
    service: "system.string:Uppercase"
    input:
      text: "${customerName}"

C. Literal Mapping

Pass a fixed value that doesn't change based on the pipeline data.

steps:
  - type: "invoke"
    service: "system.http:Request"
    input:
      method: "GET" # Fixed literal string
      timeout: 30    # Fixed literal integer

---

5. Output Merging

After an INVOKE step completes, the result is merged back into the pipeline.

Full Merge (Default)

By default, every key-value pair returned by a service is added to the pipeline. If a key already exists, it is overwritten.

Explicit Renaming

You can rename service outputs as they enter the pipeline to avoid collisions or match your output schema.

steps:
  - type: "invoke"
    service: "system.string:Uppercase"
    input: { text: "${firstName}" }
    output:
      result: "upperFirstName" # 'result' from service becomes 'upperFirstName' in pipeline

---

See Also

• Flow Service Definition
• INVOKE Step Details
• MAP Step Details