Skip to content

Processing Parameters and Asset Roles #46

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fmigneault wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Processing Parameters and Asset Roles #46

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5750027
add processing:expression results
fmigneault Oct 15, 2026
0d234a9
fix rendering
fmigneault Oct 15, 2026
e373874
fix names OAP
fmigneault Oct 15, 2026
778a1ca
fix lint md
fmigneault Oct 15, 2026
a50539b
proposal processing:expression parameters and asset roles
fmigneault Oct 20, 2026
e249cdb
update changelog
fmigneault Oct 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- New `parameters` field under `processing:expression` to provide a mapping of named input values employed.
- Asset roles `processing-input`, `processing-output`, `processing-parameters` and `processing-results`.
- Recommendation to employ `profile` in Asset using the above roles and examples for common applications.

### Changed

### Deprecated
Expand Down
65 changes: 53 additions & 12 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,18 +160,59 @@ This list is not exhaustive and can be extended with the processing level specif

### Expression Object

| Field Name | Type | Description |
| ---------- | ------ | ----------- |
| format | string | **REQUIRED** The type of the expression that is specified in the `expression` property. |
| expression | \* | **REQUIRED** An expression compliant with the `format` specified. The expression can be any data type and depends on the `format` given, e.g. string or object. |
| Field Name | Type | Description |
| ---------- | --------------- | ----------- |
| format | string | **REQUIRED** The type of the expression that is specified in the `expression` property (see [Format](#format) examples). |
| expression | \* | **REQUIRED** An expression compliant with the `format` specified. The expression can be any data type and depends on the `format` given, e.g. string or object. |
| parameters | Map<string, \*> | Optional parameters relevant for the context of the `format` specified (e.g., _openEO_ or _OGC API - Processes_ execution inputs, a variable name/file mapping for `gdal-calc`, etc. |

#### Format

Potential expression formats with examples:

| Format | Type | Description | Example |
| ----------- | ------ | ----------- | ------- |
| `gdal-calc` | string | A [`gdal_calc.py`](https://gdal.org/programs/gdal_calc.html) expression based on numpy syntax. | `A*logical_or(A<=177,A>=185)` |
| `openeo` | object | [openEO process](https://openeo.org/documentation/1.0/developers/api/reference.html#section/Processes) | [Example](https://raw.githubusercontent.com/Open-EO/openeo-processes/1.2.0/normalized_difference.json) |
| `rio-calc` | string | A [rio-calc](https://rasterio.readthedocs.io/en/latest/topics/calc.html) (RasterIO) expression | `(b4-b1)/(b4+b1)` |
| Format | Type | Description | Example |
| ------------------ | ------ | ----------- | ------- |
| `gdal-calc` | string | A [`gdal_calc.py`](https://gdal.org/programs/gdal_calc.html) expression based on numpy syntax. | `A*logical_or(A<=177,A>=185)` |
| `openeo` | object | An [openEO process](https://openeo.org/documentation/1.0/developers/api/reference.html#section/Processes) | [Example](https://raw.githubusercontent.com/Open-EO/openeo-processes/1.2.0/normalized_difference.json) |
| `ogcapi-processes` | object | An [OGC ProcessDescription](https://docs.ogc.org/is/18-062r2/18-062r2.html#ogc_process_description) | [Example](https://schemas.opengis.net/ogcapi/processes/part1/1.0/examples/json/ProcessDescription.json) |

@gadomski gadomski Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change (adding ogcapi-processes) might be easier to review in its own PR, since it's technically independent of processing parameters.

@fmigneault fmigneault Oct 28, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added it because the document already mentioned "OGC Process API" in the relation types, but didn't provide any corresponding value here.

@gadomski gadomski Oct 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess my point is, "Add ogciapi-processes to expression formats" could be its own, light pull request that could be easily reviewed and merged?

@fmigneault fmigneault Oct 29, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can cherry-pick it if desired.

| `rio-calc` | string | A [rio-calc](https://rasterio.readthedocs.io/en/latest/topics/calc.html) (RasterIO) expression | `(b4-b1)/(b4+b1)` |

> [!NOTE]
> To ensure portability, `format` may employ a specific profile URI from a registered naming authority
> that explicitly describes the represented `expression` without ambiguity.

Comment on lines +180 to +183

@gadomski gadomski Oct 27, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar with a "profile URI" — it could be helpful to link out to an explanation to help the reader understand how to use a profile?

@fmigneault fmigneault Oct 28, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree about adding an example for clarity.
Following is an example: https://www.opengis.net/def/profile/OGC/0/ogc-process-description

It can be any URI. Ideally it resolves to a concept/ontology, but it can technically be even just an well-defined string in a standard document. It can also be a JSON schema URI or some other identifiable resource that strictly characterizes the resource.

The idea of using profile URIs is mainly such that what they refer to is more interpretable and removes potential ambiguity in otherwise generic strings.

#### Parameters

The `parameters` object represents a mapping of named entities that can be interpreted by the `format` expression and
their corresponding values. The specific structure of those values and how they can be applied is specific to the `format`.

For example, an _OGC API - Processes_ or _openEO_ taking an input named `threshold` could supply their respective
_ProcessDecription_ or _Process Graph_ in `expression`, and along it, the `"parameters": {"threshold": 0.5}` inputs
it employed for its execution.

## Asset Roles

In certain cases, such as when large data is employed or produced by a processing expression, it may not be convenient
or reasonable to provide inputs and results inline of the STAC object. For this purpose, the following roles are defined
to annotate [Asset Objects](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#assets).

| Role | Description |
| --------------------- | ----------- |
| processing-input | An Asset that represents an individual input (e.g., a configuration file required by the processing expression). Note that an input STAC object should instead use the `derived_from` link [relation type](#relation-types). |
| processing-parameters | An Asset that embeds the entire processing parametrization contents that were submitted. This is similar to [Parameters](#parameters) but is not limited to `application/json` and can contain much more data. |
Comment on lines +201 to +202

@gadomski gadomski Oct 27, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't intuitively understand the different between the input and the parameters? In programming terms, aren't they both "function arguments"?

@fmigneault fmigneault Oct 28, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The difference would be as follows.
Given a process that takes {"a": <some-geotiff>, "b": <some-parquet>, "c": <some-float>}.

You would have multiple processing-input, each respectively pointing at the href of <some-geotiff> and <some-parquet>. The <some-float> can be embedded in an Asset file as text/plain, JSON, or similar if one really feels the need for it.

The processing-parameters instead represent the structure itself, as submitted to the process, i.e.:

{
  "a": {"href": <some-geotiff>},
  "b": {"href": <some-parquet>},
  "c": <some-float>
}

This is dependant of the specific encoding the process takes. Ideally it is also indicated as processing:expression to clarify, though not strictly enforced.

The processing-parameters also allows other formats, so if a multipart/related was used for examples instead of the JSON, it could also be provided as a whole.

I purposely used processing-parameters to align with the suggested (as discussed the meeting):

"processing:expression": {
  "format": "https://www.opengis.net/def/profile/OGC/0/ogc-process-description",
  "expression": { ... the big process description ... },
  "parameters": { ... same as above 'processing-parameters' content ... }
}

I think it is also a good idea to have distinct inputs/outputs and parameters/results names because they represent differ document contents. From experience with OGC Processes, "inputs" term as been used to refer to different concepts (description vs execution values), and people always get very confused about the structures. Different terms ensure a clear distinction.

@gadomski gadomski Oct 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The processing-parameters instead represent the structure itself, as submitted to the process, i.e.:

If I'm understanding (I'm probably not), the asset hrefs are duplicated in processing-input and processing-parameters? If so, this feels redundant.

Different terms ensure a clear distinction.

To me, it's less clear (inputs are inputs, that seems simplest to me). But again, I'm probably not the target audience for this extension, so I won't block.

@fmigneault fmigneault Oct 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optionally duplicated. It gives the flexibility. I would say it depends how specific the provenance wants to illustrate the concepts.

I can see at least 2 reasons why processing-parameters remains relevant:

  1. If a process only used a few floating point values, it is overkill to have it generate multiple distinct asset files with just a few bytes of data. Those files might not even exist also.
  2. If a provenance chain provides the original submission (in the context of how the openEO, OGC API, etc. produced it), it easy to re-execute it as is. Using processing-inputs would force a client to manually recompose it (maybe incorrectly) from the Assets.

| processing-output | An Asset that represents an invidivudal output (possibily amongst multiple) that was generated by the processing expression. These Assets should typically include `data` role as well, although not strictly required. |
| processing-results | An Asset that embeds the entire processing results contents generated by the processing expression (e.g., an _OGC API - Processes_ [Results](https://schemas.opengis.net/ogcapi/processes/part1/1.0/examples/json/Result.json) document. |
Comment on lines +203 to +204

@gadomski gadomski Oct 27, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly, what's the different between an "output" and a "result"? Aren't these both akin to "the return value of the function"?

@fmigneault fmigneault Oct 28, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Output/Results happens to align with the same OGC concepts of the big "Results" document containing all of the individual "Outputs".

Beside that, same reason as Inputs/Parameters for distinct naming for different concepts.


> [!NOTE]
> For each Asset that includes those roles, it is *STRONGLY RECOMMENDED* to include both a `profile` URI and a `type` field
> that will respectively describe the nature of the their contents. Alternate representations of those Assets could then
> be provided using other equivalent content negotiation by profile mechanisms.

> [!WARNING]
> The abscence of the `processing-output` role does **NOT** necessarily indicate that a certain Asset is not a result
> of the processing. This is to support previous versions of the extension and existing STAC definitions that did not
> supply it. Instead, it should be interpreted as *additional indication* to distinguish various Assets amongst
> themselves, notably when inputs and outputs are combined.

## Relation types

Expand All @@ -183,7 +224,7 @@ The following types should be used as applicable `rel` types in the
| derived_from | URL to a STAC Item that was used as input data in the creation of this Item. |
| source_for | URL to a STAC Item that was derived from this Item (forward provenance, inverse of `derived_from`). |
| processing-expression | A processing chain (or script) that describes how the data has been processed. |
| processing-execution | URL to any resource representing the processing execution (e.g. OGC Process API). |
| processing-execution | URL to any resource representing the processing execution (e.g. *OGC API - Processes*). |
| processing-software | URL to any resource that identifies the software and versions used for processing the data, e.g. a `Pipfile.lock` (Python) or `package-lock.json` (NodeJS). |
| processing-validation | URL to any kind of validation that has been applied after processing, e.g. a validation report or a script used for validation. |

Expand All @@ -197,10 +238,10 @@ for running tests are copied here for convenience.

### Running tests

The same checks that run as checks on PR's are part of the repository and can be run locally to verify that changes are valid.
The same checks that run as checks on PR's are part of the repository and can be run locally to verify that changes are valid.
To run tests locally, you'll need `npm`, which is a standard part of any [node.js installation](https://nodejs.org/en/download/).

First you'll need to install everything with npm once. Just navigate to the root of this repository and on
First you'll need to install everything with npm once. Just navigate to the root of this repository and on
your command line run:
```bash
npm install
Expand Down
Loading