Repository navigation
Fix WfFormat 1.6 compatibility - #168
Merged
Merged
Conversation
- Compute DAG level/width metrics on a task-ID-keyed graph, so that metrics are correct when task names differ from IDs (all recipe and benchmark outputs previously failed the WfFormat 1.6 validator), and writing no longer crashes on IDs without "_ID"/"_0" (e.g. "ID0000001", or IDs with ':' and '/'). - Emit integer byte totals only, gated on their own presence. - Write avgPowerInW (not avgPowerInKWh), non-empty string arguments, coreCount >= 1, RFC 3339 timestamps, unique file IDs, and the command object only when set; support specification-only instances. - Instance: accept every optional 1.6 property (description, createdAt, author, runtimeSystem, files, execution, machine cpu), keep tasks that have no execution entry, and round-trip energy, power, category, timestamps, author details, and unused machines. - SchemaValidator: use the 1.6 schema shipped with the package instead of a copy cached in the working directory, pick the validator from the schema's draft (2020-12), report format issues as warnings, and port the upstream semantic checks. - wfchef: key graph nodes by task ID and derive types from task names; compute task statistics before writing recipe files.
- Sanitize task and file IDs to the WfFormat character set (Makeflow commands such as g++, RO-Crate names, Pegasus/Snakemake/TaskVine file names), handle Makeflow "a->b" renames and variables, and give distinct files that share a name distinct IDs. - Pegasus: use the job's cores profile (default 1) instead of 0, string arguments, kickstart start times, maxrss/ram_total in bytes, and the planner version and DAGMan start time instead of the YAML format version and creation time. - RO-Crate: fix the doubling task counter, accept single input/output objects, and skip self-loops and cycles in inferred dependencies. - Nextflow: parse "ms" and "d" durations correctly. - Snakemake/TaskVine/Makeflow: real start times and makespans, unit conversions, None ignore list, quoted file names, exact name matching. - Emit machines (task.machines), pass constructor arguments by keyword, and replace exit() calls with exceptions. - Export the Snakemake, StreamFlow, and TaskVine parsers.
A task's executedAt is optional in WfFormat; tasks without a known start time no longer get the time at which the Task object was created.
- Translators: build the task graph from the workflow instead of regenerating its JSON, raise a clear error for tasks without a command, and map task IDs and names (which may contain ':', '/', '#', '.', '-') to unique identifiers valid in each target system (Parsl, Snakemake, Nextflow, CWL/StreamFlow, Airflow, Pegasus, PyCOMPSs, Swift/T); output is unchanged for workflows whose IDs are already valid identifiers. - Handle workflows without files (marker files keep the dependencies in Makeflow, Snakemake, and Nextflow), file IDs containing '/', and shell-unsafe --name values; replace Swift/T's exit(1) with an error. - WorkflowBenchmark: pass time/num_chunks in the right order, rewrite only the input/output file arguments when renaming files, derive output file names from file-system-safe IDs, avoid a division by zero when the workflow has no runtimes, clear measured execution data, and look up per-type parameters by category or task name.
The schema is no longer versioned in this repository. SchemaValidator fetches it from the WfFormat release that matches the schema version written by WfCommons (tag v1.6), caches it in the user cache directory (~/.cache/wfcommons), and ignores any wfcommons-schema.json file in the working directory. Downloaded schema files are git-ignored.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #167. WfCommons wrote
schemaVersion: "1.6", but most of its output failed the official WfFormat 1.6 validator, and it could not load many valid 1.6 instances.What was broken
compute_metricsbuilt its graph with nodes keyed by task name and edges keyed by parent id, sonumberOfLevels/minimumWidth/maximumWidthwere wrong whenever name ≠ id. Every recipe-generated instance (0 of 23 tested) and every benchmark failed validation._ID/_0(e.g.ID0000001, or the:and/newly allowed in 1.6) raisedValueErrorinwrite_json, and in all translators.Instancerequired optional properties (description,createdAt,author,runtimeSystem,files,execution, machinecpu) and dropped data on round trip (energy/power keys, author details, timestamps).Draft4Validatoragainst a draft 2020-12 schema, no handling of spec-only instances, and a schema cached in the current working directory (a stale 1.5 copy rejected every 1.6 instance).coreCount: 0, unsanitized file names), and wfbench swappedtime/num_chunksand corrupted arguments when renaming files.Changes
common/,wfinstances/instance.py,wfinstances/schema.py): metrics computed on an ID-keyed graph; integer byte totals;avgPowerInW; RFC 3339 timestamps; spec-only instances; all optional 1.6 properties accepted and round-tripped.SchemaValidatorfetches the schema from the WfFormat release matching the version WfCommons writes (v1.6), caches it in~/.cache/wfcommons, picks the validator from the schema's draft, and ports the upstream semantic checks.ms/ddurations, machines emitted, keyword constructor arguments.Behavior changes
Instancenow rejects what the official validator rejects (inconsistent metrics, duplicate IDs, dangling references, cycles).executedAt; an emptycommandis omitted.schema_file).jsonschema>=4.18.Testing
nextflow/legacy/fetchngs-dirt02-001.json) is itself invalid (maximumWidth19, actual 26) and needs regenerating in WfInstances.Known limitations
runName,repoin some Nextflow instances) are still dropped on round trip.#in task IDs is untested at runtime.maxrss/ram_total(KB) and Makeflow resource-monitor (MB) unit conversions were not checked against real logs.