diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 6380850..846eb2a 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -6,7 +6,7 @@ on: paths: - 'mapping-server.yml' - 'scripts/cli.py' - - 'transforms/**' + - 'registry.yml' schedule: - cron: "0 0 1 * *" workflow_dispatch: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2e07f87..feb523c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -54,38 +54,29 @@ To add a new mapping registry to the Mapping Commons: ## Add a new transform specification -This repository is the source of truth for linkml-map transforms that convert mapping formats into the FAIR Mappings Schema. Currently supported: +The linkml-map transforms that convert mapping formats into the FAIR Mappings Schema, and the slot-level SSSOM sets that document them, live in [fair-mappings-schema](https://github.com/mapping-commons/fair-mappings-schema) (`src/fair_mappings_schema/transformations/` and `src/fair_mappings_schema/mappings/`). That repository is their source of truth, and this one only registers them. Currently supported: -| Format | Transform file | Source type | -|--------|---------------|-------------| -| SSSOM | `transforms/sssom-to-fair.transformation.yaml` | `mapping set` | -| LinkML-Map | `transforms/linkmlmap-to-fair.transformation.yaml` | `TransformationSpecification` | +| Format | Transform | Source type | +|--------|-----------|-------------| +| SSSOM | | `mapping set` | +| LinkML-Map | | `TransformationSpecification` | To add support for a new mapping format: -1. **Create a transform file** in `transforms/` following the [linkml-map specification](https://linkml.io/linkml-map/). The transform should map your format's metadata to `MappingSpecification` fields. +1. **Add the transform** (and its slot-level SSSOM set) to fair-mappings-schema, following the [linkml-map specification](https://linkml.io/linkml-map/). The transform maps your format's metadata to `MappingSpecification` fields. -2. **Register it** in `registry.yml`: +2. **Give it a PURL** under `https://w3id.org/fair-map/transforms/` in the [w3id `fair-map` config](https://github.com/perma-id/w3id.org/tree/master/ids/fair-map). + +3. **Register it** in `registry.yml` by that PURL: ```yaml - transform_references: - - id: https://w3id.org/linkml/transformer/myformat-to-fair-mappings - title: MyFormat to FAIR Mappings Schema - description: Transforms MyFormat metadata to FAIR Mappings Schema + mapping_set_references: + - mapping_set_id: https://w3id.org/fair-map/transforms/myformat-to-fair + mapping_set_group: fair-mappings-transforms local_name: myformat-to-fair.transformation.yaml - source_format: myformat - source_schema_url: https://example.org/myformat-schema.yaml - source_type: MyRootClass ``` -3. **Update `scripts/cli.py`** to handle the new format if it requires different fetching/parsing logic. - -4. **Test locally**: - - ```bash - just transform-sssom my-input.yaml my-output.yaml # for SSSOM - just transform-linkmlmap my-input.yaml my-output.yaml # for LinkML-Map - ``` +4. **Update `scripts/cli.py`** to handle the new format if it requires different fetching/parsing logic. ## Local development diff --git a/README.md b/README.md index 4bf7731..8b94611 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ This repository: - Transforms SSSOM and LinkML-Map metadata into the FAIR Mappings Schema using [linkml-map](https://github.com/linkml/linkml-map) - Serves a static search/browse interface on GitHub Pages -The transform specifications in [`transforms/`](transforms/) are the source of truth for translating existing mapping formats into the FAIR Mappings Schema. See [`registry.yml`](registry.yml) for the full list. +The transform specifications that translate existing mapping formats into the FAIR Mappings Schema, and the slot-level SSSOM mapping sets that document them, live in [fair-mappings-schema](https://github.com/mapping-commons/fair-mappings-schema), which is their source of truth. [`registry.yml`](registry.yml) lists them by their PURLs, and the ETL fetches them from there. ## Local development diff --git a/mappings/linkmlmap-to-fair.sssom.tsv b/mappings/linkmlmap-to-fair.sssom.tsv deleted file mode 100644 index bbd182a..0000000 --- a/mappings/linkmlmap-to-fair.sssom.tsv +++ /dev/null @@ -1,29 +0,0 @@ -# curie_map: -# fair_mappings_schema: https://w3id.org/mapping-commons/fair-mappings-schema/ -# linkmlmap: https://w3id.org/mapping-commons/fair/mappingspecification/ -# owl: http://www.w3.org/2002/07/owl# -# rdf: http://www.w3.org/1999/02/22-rdf-syntax-ns# -# rdfs: http://www.w3.org/2000/01/rdf-schema# -# semapv: https://w3id.org/semapv/vocab/ -# skos: http://www.w3.org/2004/02/skos/core# -# sssom: https://w3id.org/sssom/ -# license: https://creativecommons.org/publicdomain/zero/1.0/ -# mapping_set_description: Slot-level correspondences between LinkML-Map TransformationSpecification -# and FAIR Mappings Schema MappingSpecification -# mapping_set_id: https://w3id.org/mapping-commons/fair/mappingspecification/linkml-map-to-fair-mappings/slot-mappings -# mapping_set_title: LinkML-Map to FAIR Mappings Schema Slot Mappings -subject_id subject_label predicate_id object_id object_label mapping_justification -linkmlmap:author author skos:exactMatch fair_mappings_schema:author author semapv:ManualMappingCuration -linkmlmap:content_url content_url skos:exactMatch fair_mappings_schema:content_url content_url semapv:ManualMappingCuration -linkmlmap:creator creator skos:exactMatch fair_mappings_schema:creator creator semapv:ManualMappingCuration -linkmlmap:description description skos:exactMatch fair_mappings_schema:description description semapv:ManualMappingCuration -linkmlmap:documentation documentation skos:exactMatch fair_mappings_schema:documentation documentation semapv:ManualMappingCuration -linkmlmap:id id skos:exactMatch fair_mappings_schema:id id semapv:ManualMappingCuration -linkmlmap:license license skos:exactMatch fair_mappings_schema:license license semapv:ManualMappingCuration -linkmlmap:mapping_method mapping_method skos:exactMatch fair_mappings_schema:mapping_method mapping_method semapv:ManualMappingCuration -linkmlmap:publication_date publication_date skos:exactMatch fair_mappings_schema:publication_date publication_date semapv:ManualMappingCuration -linkmlmap:reviewer reviewer skos:exactMatch fair_mappings_schema:reviewer reviewer semapv:ManualMappingCuration -linkmlmap:source_schema source_schema skos:narrowMatch fair_mappings_schema:subject_source subject_source semapv:ManualMappingCuration -linkmlmap:target_schema target_schema skos:narrowMatch fair_mappings_schema:object_source object_source semapv:ManualMappingCuration -linkmlmap:title title skos:exactMatch fair_mappings_schema:name name semapv:ManualMappingCuration -linkmlmap:version version skos:exactMatch fair_mappings_schema:version version semapv:ManualMappingCuration diff --git a/mappings/sssom-to-fair.sssom.tsv b/mappings/sssom-to-fair.sssom.tsv deleted file mode 100644 index bfa73a2..0000000 --- a/mappings/sssom-to-fair.sssom.tsv +++ /dev/null @@ -1,23 +0,0 @@ -# curie_map: -# fair_mappings_schema: https://w3id.org/mapping-commons/fair-mappings-schema/ -# owl: http://www.w3.org/2002/07/owl# -# rdf: http://www.w3.org/1999/02/22-rdf-syntax-ns# -# rdfs: http://www.w3.org/2000/01/rdf-schema# -# semapv: https://w3id.org/semapv/vocab/ -# skos: http://www.w3.org/2004/02/skos/core# -# sssom: https://w3id.org/sssom/ -# license: https://creativecommons.org/publicdomain/zero/1.0/ -# mapping_set_description: Slot-level correspondences between SSSOM Mapping Set and -# FAIR Mappings Schema MappingSpecification -# mapping_set_id: https://w3id.org/mapping-commons/fair/mappingspecification/sssom-to-fair-mappings/slot-mappings -# mapping_set_title: SSSOM to FAIR Mappings Schema Slot Mappings -subject_id subject_label predicate_id object_id object_label mapping_justification -sssom:license license skos:exactMatch fair_mappings_schema:license license semapv:ManualMappingCuration -sssom:mapping_set_description mapping_set_description skos:exactMatch fair_mappings_schema:description description semapv:ManualMappingCuration -sssom:mapping_set_id mapping_set_id skos:exactMatch fair_mappings_schema:id id semapv:ManualMappingCuration -sssom:mapping_set_title mapping_set_title skos:exactMatch fair_mappings_schema:name name semapv:ManualMappingCuration -sssom:mapping_set_version mapping_set_version skos:exactMatch fair_mappings_schema:version version semapv:ManualMappingCuration -sssom:mapping_tool mapping_tool skos:exactMatch fair_mappings_schema:mapping_method mapping_method semapv:ManualMappingCuration -sssom:object_source object_source skos:narrowMatch fair_mappings_schema:object_source object_source semapv:ManualMappingCuration -sssom:publication_date publication_date skos:exactMatch fair_mappings_schema:publication_date publication_date semapv:ManualMappingCuration -sssom:subject_source subject_source skos:narrowMatch fair_mappings_schema:subject_source subject_source semapv:ManualMappingCuration diff --git a/registry.yml b/registry.yml index 1f0315c..d206252 100644 --- a/registry.yml +++ b/registry.yml @@ -6,19 +6,23 @@ mapping_registry_description: >- homepage: https://github.com/mapping-commons/mapping-commons.github.io documentation: https://github.com/mapping-commons/mapping-commons.github.io/blob/main/README.md +# The source of truth for these four files is +# https://github.com/mapping-commons/fair-mappings-schema. Each mapping_set_id +# is a PURL that resolves to the file there; the ETL fetches it from that id. +# local_name is the file name, which tells the ETL how to read it. mapping_set_references: - - mapping_set_id: https://w3id.org/mapping-commons/fair/mappingspecification/sssom-to-fair-mappings + - mapping_set_id: https://w3id.org/fair-map/transforms/sssom-to-fair mapping_set_group: fair-mappings-transforms local_name: sssom-to-fair.transformation.yaml - - mapping_set_id: https://w3id.org/mapping-commons/fair/mappingspecification/linkml-map-to-fair-mappings + - mapping_set_id: https://w3id.org/fair-map/transforms/linkmlmap-to-fair mapping_set_group: fair-mappings-transforms local_name: linkmlmap-to-fair.transformation.yaml - - mapping_set_id: https://w3id.org/mapping-commons/fair/mappingspecification/sssom-to-fair-mappings/slot-mappings + - mapping_set_id: https://w3id.org/fair-map/mappings/sssom-to-fair mapping_set_group: fair-mappings-slot-mappings local_name: sssom-to-fair.sssom.tsv - - mapping_set_id: https://w3id.org/mapping-commons/fair/mappingspecification/linkml-map-to-fair-mappings/slot-mappings + - mapping_set_id: https://w3id.org/fair-map/mappings/linkmlmap-to-fair mapping_set_group: fair-mappings-slot-mappings local_name: linkmlmap-to-fair.sssom.tsv diff --git a/scripts/cli.py b/scripts/cli.py index b56db75..6df8201 100644 --- a/scripts/cli.py +++ b/scripts/cli.py @@ -137,47 +137,41 @@ def _process_sssom_mapping_set(mapping_set_id: str, mapping_set_uri: str) -> dic def _process_transform_registry(registry_path: str) -> list[dict]: - """Process local transform specs in registry.yml as browsable entries.""" + """Process the transform specs and slot mappings in registry.yml as browsable entries. + + The files live in mapping-commons/fair-mappings-schema; each one is fetched + from its mapping_set_id (a PURL) or mirror_from, not read from this repository. + """ with open(registry_path) as f: registry = yaml.safe_load(f) info = _registry_info(registry) - base_dir = os.path.dirname(registry_path) - search_dirs = ["transforms", "mappings"] specs = [] for ref in registry.get("mapping_set_references", []): local_name = ref.get("local_name") - if not local_name: - continue - - # Find the file in one of the search directories - input_file = None - for d in search_dirs: - candidate = os.path.join(base_dir, d, local_name) - if os.path.exists(candidate): - input_file = candidate - break - if not input_file: - click.echo(f"File not found: {local_name} (searched {search_dirs})", err=True) + url = ref.get("mirror_from", ref.get("mapping_set_id")) + if not local_name or not url: continue # Detect mapping type from file extension mapping_type = "sssom" if local_name.endswith(".sssom.tsv") else "linkml_map" try: + response = _session.get(url) + if response.status_code != 200: + raise RuntimeError(f"HTTP {response.status_code} fetching {url}") if mapping_type == "sssom": - with open(input_file) as f: - captured, columns = _capture_sssom_head(f) + captured, columns = _capture_sssom_head(response.text.splitlines()) meta = parse_sssom_tsv(iter(captured)) _inject_per_mapping_source(meta, columns) spec = transform_to_fair(meta, "sssom") else: - spec = load_mapping(input_file, mapping_type) + spec = transform_to_fair(yaml.safe_load(response.text), mapping_type) spec["registries"] = [info] specs.append(spec) except Exception as e: - click.echo(f"Transform failed for {local_name}: {e}", err=True) + click.echo(f"Transform failed for {local_name} ({url}): {e}", err=True) return specs diff --git a/transforms/linkmlmap-to-fair.transformation.yaml b/transforms/linkmlmap-to-fair.transformation.yaml deleted file mode 100644 index 948235c..0000000 --- a/transforms/linkmlmap-to-fair.transformation.yaml +++ /dev/null @@ -1,135 +0,0 @@ -id: https://w3id.org/mapping-commons/fair/mappingspecification/linkml-map-to-fair-mappings -title: LinkML-Map to FAIR Mappings Schema Transformation -description: |- - Transforms a LinkML-Map TransformationSpecification to a FAIR Mappings Schema - MappingSpecification. This preserves metadata elements while discarding the - actual transformation rules (class_derivations, slot_derivations, etc.) which - have no equivalent in the FAIR Mappings Schema. - -prefixes: - linkmlmap: https://w3id.org/mapping-commons/fair/mappingspecification/ - fair_mappings_schema: https://w3id.org/mapping-commons/fair-mappings-schema/ - -source_schema: tmp/linkml-map-schema.yaml -target_schema: tmp/fair-mappings-schema.yaml - -class_derivations: - - MappingSpecification: - populated_from: TransformationSpecification - description: |- - Maps TransformationSpecification metadata to MappingSpecification. - Note: class_derivations, slot_derivations, enum_derivations, prefixes, - copy_directives, and schema patches are not preserved as they have no - equivalent in the FAIR Mappings Schema. - slot_derivations: - id: - populated_from: id - name: - populated_from: title - description: Maps 'title' from source to 'name' in target - description: - populated_from: description - publication_date: - populated_from: publication_date - license: - populated_from: license - version: - populated_from: version - mapping_method: - populated_from: mapping_method - documentation: - populated_from: documentation - type: - value: "linkml_map" - content_url: - populated_from: content_url - creator: - populated_from: creator - author: - populated_from: author - reviewer: - populated_from: reviewer - subject_source: - expr: "{'name': source_schema}" - description: Maps source_schema string to a Source object with name field - object_source: - expr: "{'name': target_schema}" - description: Maps target_schema string to a Source object with name field - - Agent: - populated_from: Agent - description: Maps Agent class (abstract in both schemas) - slot_derivations: - id: - populated_from: id - name: - populated_from: name - type: - populated_from: type - - Person: - populated_from: Person - description: Maps Person (individual contributor) - slot_derivations: - id: - populated_from: id - name: - populated_from: name - type: - populated_from: type - orcid: - populated_from: orcid - affiliation: - populated_from: affiliation - - Organization: - populated_from: Organization - description: Maps Organization - slot_derivations: - id: - populated_from: id - name: - populated_from: name - type: - populated_from: type - ror_id: - populated_from: ror_id - url: - populated_from: url - - Software: - populated_from: Software - description: Maps Software agent - slot_derivations: - id: - populated_from: id - name: - populated_from: name - type: - populated_from: type - version: - populated_from: version - repository_url: - populated_from: repository_url - - Source: - populated_from: Source - description: Maps Source (data source metadata) - slot_derivations: - id: - populated_from: id - name: - populated_from: name - version: - populated_from: version - documentation: - populated_from: documentation - content_url: - populated_from: content_url - content_type: - populated_from: content_type - metadata_url: - populated_from: metadata_url - metadata_type: - populated_from: metadata_type diff --git a/transforms/sssom-to-fair.transformation.yaml b/transforms/sssom-to-fair.transformation.yaml deleted file mode 100644 index 4b6686f..0000000 --- a/transforms/sssom-to-fair.transformation.yaml +++ /dev/null @@ -1,55 +0,0 @@ -id: https://w3id.org/mapping-commons/fair/mappingspecification/sssom-to-fair-mappings -title: SSSOM to FAIR Mappings Schema Transformation -description: |- - Transforms an SSSOM Mapping Set to a FAIR Mappings Schema MappingSpecification. - This preserves metadata elements while discarding the individual mappings, - which have no equivalent in the FAIR Mappings Schema (which focuses on - mapping specification metadata, not individual mapping assertions). - - Key transformations: - - mapping_set_id → id - - mapping_set_title → name - - mapping_set_description → description - - subject_source → subject_source (Source with id) - - object_source → object_source (Source with id) - - Sets type to 'sssom' to indicate the mapping specification type - - Note: Creator/author/reviewer transformations are simplified since SSSOM - uses flat ID lists while FAIR Mappings uses structured Agent objects. - -prefixes: - sssom: https://w3id.org/sssom/ - fair_mappings_schema: https://w3id.org/mapping-commons/fair-mappings-schema/ - -source_schema: tmp/sssom-schema.yaml -target_schema: tmp/fair-mappings-schema.yaml - -class_derivations: - - MappingSpecification: - populated_from: "mapping set" - description: |- - Maps SSSOM Mapping Set metadata to MappingSpecification. - Note: Individual mappings are not preserved as they represent - actual mapping assertions, not metadata about the specification. - slot_derivations: - id: - populated_from: mapping_set_id - name: - populated_from: mapping_set_title - description: - populated_from: mapping_set_description - version: - populated_from: mapping_set_version - license: - populated_from: license - publication_date: - populated_from: publication_date - type: - value: "sssom" - mapping_method: - populated_from: mapping_tool - subject_source: - expr: "{'id': subject_source} if subject_source else None" - object_source: - expr: "{'id': object_source} if object_source else None"