Expressions and templates

Pipo has one small expression language, used everywhere a pipeline makes a decision or builds a value. It's a safe subset of JavaScript expression syntax: it reads like JavaScript, but it can't loop, assign, call methods or reach anything outside the packet.

input:
  via: http
  with: { path: /people }
  validate:
    - exists(data.email) && matches(data.email, "^[^@\\s]+@[^@\\s]+$")
    - data.age >= 18
nodes:
  adults:
    from: input
    filter: default(data.country, "SE") in ["SE", "NO", "DK"]
  stamp:
    from: adults
    transform: map
    with:
      data:
        email: "${lower(trim(data.email))}"
        age: "${data.age}"
        seen: "${iso(meta.received_at)}"

Where they appear#

Expressions are plain strings evaluated for a value:

Templates are strings containing ${expr}. They appear in every with: block and every message:

Using a variable where it isn't available is an error with its exact position: P041 ('result' is not available here, with the list of what is). secrets outside a with: block is P051.

Context variables#

meta fields:

Syntax#

Allowed:

• Literals: strings in single or double quotes ('a', "b", with \n, \' escapes), numbers (1, 1.5, .5, 1e3), true, false, null, arrays ([1, 2]), objects ({a: 1, "b c": 2}, and shorthand {data}).
• Member and index access: data.a.b, data.items[0], data["first name"], optional chaining data?.a?.b.
• Arithmetic: + - * / %, unary - and +.
• Comparison: == != === !== < <= > >=.
• Logic: && || !, and ?? (use the right side when the left is null or missing).
• Membership: x in y.
• The ternary cond ? a : b, and parentheses.
• Calls to the helper functions.

Not allowed (each is a P040 error at check time):

• Assignment (=, +=), ++/--, statements and ;, the comma operator.
• Arrow functions, new, this, typeof, bitwise operators (| & ^ ~ << >>).
• Method calls on values: write len(x), not x.length or x.toUpperCase().
• Any name other than the context variables: undefined, Math, JSON, eval and so on are unknown.
• __proto__, constructor and prototype as members or object keys, and computed object keys ({[k]: 1}).

Semantics#

• Missing is safe. Member access on a missing or null value gives undefined and never throws, so data.a.b.c is fine when a is missing. In a template, undefined and null render as an empty string.

• Equality is strict and deep. == and === mean the same thing: no type coercion, and arrays and objects compare by content.

  Expression	Result
  36 == "36"	false
  true == 1	false
  [1, 2] == [1, 2]	true
  {a: 1, b: 2} == {b: 2, a: 1}	true
  null == data.missing	false (null and undefined differ)

• in tests membership. An array containing the value (compared deeply), a string containing the substring, or an object having the key:

  Expression	Result
  "a" in ["a", "b"]	true
  {a: 1} in [{a: 1}]	true
  "da" in "Ada"	true
  "x" in {x: 1}	true
  "toString" in {}	false (only own keys count)
  "a" in null	false

• Arithmetic and ordering follow JavaScript. "5" + 2 is "52", "5" * 2 is 10, data.missing + 1 is NaN, "10" < "9" is true (string comparison), "2" < 10 is true. Convert explicitly when types matter.

• Truthiness follows JavaScript. false, 0, "", null, undefined and NaN are false; everything else (including "0", [] and {}) is true. Validate rules, filters, routes and until use it.

• Only own properties are visible. Prototype members such as toString or length on an array are undefined. Use len().

Limits#

• An expression is at most 2,000 characters, its syntax tree at most 32 levels deep and 256 nodes. Expressions can't loop, so evaluation time is bounded by their size.
• matches() patterns are at most 200 characters.

Templates#

A template is a string with ${expr} parts:

message: "Received ${meta.packet_id} from ${meta.source} at ${iso(meta.received_at)}"

• Type preservation. When the whole value is a single ${expr}, the result keeps its type: a number stays a number, an object stays an object. Mixed text always gives a string.

  Copy

  with:
    data:
      age: "${data.age}"           # 36, a number
      tags: "${data.tags}"         # ["a", "b"], an array
      label: "age ${data.age}"     # "age 36", a string
      list: "tags=${data.tags}"    # 'tags=["a","b"]': objects and arrays become JSON
  

• Nested values. Templates are rendered inside every string of a with: block, at any depth in maps and lists.

• Escaping. $${ writes a literal ${: "cost: $${data.age}" renders as cost: ${data.age}.

• Errors. An unclosed ${ is a P040 error at check time. An expression that fails at run time (a helper given the wrong type, say) is a step error under that step's on_error.

Helper functions#

Notes:

• matches() runs on the runner's regular-expression engine (Rust's regex crate). It has no lookaround ((?=…), (?<!…)) and no backreferences (\1): such a pattern is an error ("Invalid regular expression…"). \d, \w and \s match ASCII digits and word characters, and Unicode whitespace. . doesn't match a newline. pipo check doesn't run patterns, so test a new one with pipo test. In YAML, a backslash inside double quotes needs doubling: "^\\d+$".
• len() counts UTF-16 code units, like JavaScript: len("😀") is 2.
• json() of a missing value is missing (renders as empty text). NaN and infinities become null.

Recipes#

# Required, non-empty string
- type(data.name) == "string" && len(trim(data.name)) > 0

# One of a fixed set
- data.status in ["new", "open", "closed"]

# A number in range
- type(data.age) == "number" && data.age >= 0 && data.age < 150

# An optional field, if present, must be a list (quoted: a leading ! is a YAML tag)
- "!exists(data.tags) || type(data.tags) == 'array'"

# Route by a header-like field with a fallback
route:
  urgent: lower(default(data.priority, "")) == "high"
  normal: else

# Drop packets older than an hour
filter: now() - meta.received_at < duration("1h")

# Stop a pipeline after 1,000 deliveries or 30 minutes of uptime
lifetime:
  until: stats.delivered >= 1000 || stats.uptime > 1800

# A file name per packet and day
path: "./out/${iso(meta.received_at)}-${meta.packet_id}.json"

# Pass a whole object into a prompt
prompt: |
  Classify this ticket:
  ${json(data)}

How it's checked and run#

pipo check parses every expression and template, and reports unknown helpers, unknown or unavailable variables, and syntax errors with their exact line and column (P040, P041, P051, P013 for an undeclared ${secrets.x}). The runner evaluates them with its own evaluator, which accepts only the syntax above. JavaScript's eval and Function are never used. The checker (TypeScript) and the runner (Rust) share a conformance suite of several hundred cases, so they agree on every result.

See also#

• Nodes and the graph: filters, routes and loops
• Inputs: validate and on_invalid
• Spec: §3.2 expressions and templates