Skip to content

Feature json to yaml - #11

Open
estansb-graycore wants to merge 4 commits into
mainfrom
feature-json-to-yaml
Open

estansb-graycore wants to merge 4 commits into
mainfrom
feature-json-to-yaml

Conversation

@estansb-graycore

@estansb-graycore estansb-graycore commented Oct 19, 2026 •

Copy link
Copy Markdown

Resolves #10

Feature description

Adds the reverse of build: a decompile executor (and decompileFile() in the library)
that turns an existing pipeline JSON into a *-pipeline.yaml, so pipelines written before
FTAC can be brought under it.

npx nx run pipelines:decompile --pattern 'legacy/*.json'

Every step is kept only if compiling it reproduces the original activity exactly
(firstDifference, key order ignored). If no action reproduces it, the activity is kept
verbatim as a fabric/raw step. The whole document is then compiled again and compared
with the input before anything is written. So the output is always correct, and parts
that aren't readable yet stay as raw JSON.

For examples/hello/hello.json the result has no raw steps, and building it gives back
the original file byte-for-byte.

What is read back

  • Every built-in action, via a new optional decompile hook next to each compile.
    Also if: groups and steps nested inside them and inside for-each.
  • Expressions back to ${{ }}: pipeline().parameters.x → inputs.x,
    variables('x') → vars.x, activity('x').output → steps.x.outputs,
    item() → item, and the escaped SQL literal → ${{ sql(...) }}.
  • {{ .<name>_id }} placeholders back to names. Policy back to timeout:/retry:,
    and userProperties back to labels:.
  • Lakehouse CSV and table datasets are collected under datasets:. A dataset used by
    several steps gets one entry.

Each raw step is reported with a reason, e.g.
nb: fabric/notebook: notebookId is "3333…", not a {{ .<name>_id }} token.

Changes to existing code

None of these change what build produces; existing specs and examples:check pass
unchanged.

  • executor/build.ts: the project-root lookup moved into executor/projectRoot.ts
    so both executors share it.
  • yaml/actions/types.ts: optional decompile on Action.
  • pack.ts: ships decompile.schema.json.

Known limitations

These stay fabric/raw, because current YAML can't express them:

  • Real GUIDs for notebooks, pipelines, lakehouses, warehouses or the workspace, as in
    a pipeline exported from the portal after deployment. A Copy keeps a GUID source or
    sink as raw: and the rest of the step stays readable.
  • An IfCondition with an else branch.
  • Exports that differ from what FTAC writes: userProperties: [],
    isSequential: false, plain-text Fail messages, Query script batches,
    secureInput: true.
  • Top-level properties such as description and annotations are dropped with a
    warning.
  • env: shortcuts and the original dataset names aren't in the JSON, so values are
    written out in full and datasets are named after their file or table.

Add a decompile executor, and decompileFile() in the library, that turns an
existing pipeline JSON into a *-pipeline.yaml. Each activity is read back
as the action that compiles to it exactly, with expressions as ${{ }},
{{ .<name>_id }} tokens as names and lakehouse datasets under datasets:.
Anything no action reproduces is kept verbatim as a fabric/raw step, with
the reason reported. The YAML is compiled again and must give the same
definition before anything is written, so the output is always correct.

Actions gain an optional decompile hook beside compile. The project-root
lookup moves from the build executor into executor/projectRoot.ts so both
executors share it. What build produces is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@estansb-graycore
estansb-graycore marked this pull request as ready for review October 20, 2026 15:24
@estansb-graycore
estansb-graycore marked this pull request as draft October 20, 2026 15:30
@damienwebdev

Copy link
Copy Markdown
Member

A few thoughts:

  1. The decompile naming is a bit odd. I think it makes more to just call it to-yaml. I also don't think it's safe to imply that we're making this tool perfectly invertible (which decompile somewhat implies).
  2. I also think that instead of making something convert to "raw", instead what we do is we throw a compile-time error indicating that the compiler doesn't support that conversion yet.
  3. We need tons of integration tests covering this behavior.

@estansb-graycore
estansb-graycore marked this pull request as ready for review October 22, 2026 15:42

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: JSON to YAML conversion for existing pipelines

3 participants