fix: remote snakemake imports and align module with Modelblocks conventions

INTERFACE.yaml

@@ -1,19 +1,42 @@
-# Module Input-Output structure for automated docs generation
+convention_version: v1.0.0
 pathvars:
   snakemake_defaults:
     logs:
-      default: "<logs>"
-      description: location of snakemake log files.
+      default: <logs>
+      description: Location of module logs.
     resources:
-      default: "<resources>"
-      description: >-
-        Raw OSM files are written to retrieve/{country}_{feature}.json.
-        Clean features are written below clean; generic network outputs
-        below build (buses/lines/transformers as CSV under build/csv
-        and GeoJSON, including substation polygons, under build/geojson).
-        An interactive PyDeck map of the network, and the default target
-        of this workflow, is written to map.html.
+      default: <resources>
+      description: Location of downloaded PBFs, retrieved OSM features, and cleaning intermediates.
     results:
-      default: "<results>"
-      description: location of module result files.
+      default: <results>
+      description: Location of module results.
   user_resources: {}
+  results:
+    buses:
+      default: <results>/network/csv/buses.csv
+      description: Network buses.
+    lines:
+      default: <results>/network/csv/lines.csv
+      description: Network lines.
+    transformers:
+      default: <results>/network/csv/transformers.csv
+      description: Network transformers.
+    buses_geojson:
+      default: <results>/network/geojson/buses.geojson
+      description: Network buses geojson.
+    lines_geojson:
+      default: <results>/network/geojson/lines.geojson
+      description: Network lines geojson.
+    transformers_geojson:
+      default: <results>/network/geojson/transformers.geojson
+      description: Network transformers geojson.
+    stations_polygon:
+      default: <results>/network/geojson/stations_polygon.geojson
+      description: Network stations polygon.
+    buses_polygon:
+      default: <results>/network/geojson/buses_polygon.geojson
+      description: Network buses polygon.
+    map:
+      default: <results>/map.html
+      description: Network map.
+wildcards: {}

README.md

@@ -12,93 +12,212 @@ A modular Snakemake workflow for retrieving OpenStreetMap power infrastructure.
 
 ## About
 
-`grid-builder` is a modular `snakemake` workflow that retrieves OpenStreetMap power infrastructure and builds a generic high-voltage network. It can be imported into another `snakemake` workflow.
+`grid-builder` is a modular Snakemake workflow for retrieving OpenStreetMap power
+infrastructure and building a generic high-voltage AC network. It follows the
+[Modelblocks conventions](https://www.modelblocks.org/convention/) and can be
+imported into another Snakemake workflow or run on its own.
 
-The workflow retains AC substations, overhead lines, and cables at configured voltage levels, then creates generic buses, connected line segments, and voltage-pair transformers. The outputs preserve OSM provenance and geometry but contain no PyPSA-specific line types, capacities, or electrical-component assumptions.
-
-This module follows the Modelblocks conventions (https://www.modelblocks.org). For more information, consult the [integration example](./tests/integration/Snakefile) and the `snakemake` [modularisation documentation](https://snakemake.readthedocs.io/en/stable/snakefiles/modularization.html).
+For more information, consult the [Modelblocks documentation](https://modelblocks.readthedocs.io/en/latest/),
+the [integration example](./tests/integration/Snakefile), and the
+[Snakemake modularisation documentation](https://snakemake.readthedocs.io/en/stable/snakefiles/modularization.html).
 
 ## Overview
 
-Currently implemented:
+<p align="center">
+  <img src="./figures/rulegraph.png" width="900" alt="Rule graph: retrieve_osm_pbf → clean → build_network → build_interactive_map → all">
+</p>
 
-1. Retrieve OSM substations, lines, cables, and (optionally) circuit relations by country, either from a cached local Geofabrik PBF extract or the live Overpass API.
-2. Clean the raw retrieval output, filtering voltage, frequency, construction status, and future assets, and grouping relation member ways into one line per real-world circuit.
-3. Merge nearby stations and line endpoints into generic buses, AC lines, and transformers.
-4. Build a self-contained interactive map of the resulting network (`map.html`), with layer toggles, voltage/text filtering, and click-through OSM links — this is the workflow's default target.
+The rule graph shows the default Geofabrik backend. Selecting Overpass replaces
+`retrieve_osm_pbf` with `retrieve_osm_overpass`; the downstream rules are the same.
+
+Data processing steps:
+
+1. Retrieve OSM substations, overhead lines, cables, and optional circuit
+   relations by country, using Geofabrik PBF extracts or the Overpass API.
+   Downloaded PBF extracts are cached for reuse.
+2. Clean tags and geometries, filter by voltage and frequency, and combine
+   circuit relation member ways when enabled.
+3. Apply the configured construction-status and date filters, merge nearby
+   stations and line endpoints into buses, and construct AC lines and
+   transformers between voltage levels at the same station.
+4. Generate an interactive HTML map with layer controls, voltage and text
+   filters, and links to the source OSM objects. This is the default target
+   when running the workflow on its own.
+
+### Important assumptions
+
+- The workflow builds AC topology for now, DC lines and converters will be
+  added at a later stage.
+- Connections are inferred from geometry and OSM tags. Transformers are inferred
+  between every pair of voltage-level buses at the same real station, rather
+  than taken from an inventory of individual transformers.
+- Outputs include geometry and OSM references for buses and lines.
+- The map embeds the network data in an HTML file that opens without a local
+  server. Its JavaScript libraries and basemap require internet access.
 
 ## Configuration
 
-Configuration lives in [`config/config.yaml`](./config/config.yaml), validated against a generated JSON schema. See the configuration [README](./config/README.md) for the available controls, including retrieval backends, regional overrides, and personal/local settings.
+Consult the [configuration README](./config/README.md) and
+[default configuration](./config/config.yaml) for retrieval options, network
+settings, regional overrides, and local configuration.
+
+Configuration is validated with Pydantic. The same models generate the
+[JSON Schema in YAML format](./workflow/internal/config.schema.yaml) describing
+the available options.
 
 ## Input / output structure
 
-Please consult the [interface file](./INTERFACE.yaml) for more information.
-
-Raw retrieval outputs use `<resources>/retrieve/{country}_{feature}.json`, one
-file per country and feature (`lines_way`, `cables_way`, `substations_way`,
-`substations_node`, `substations_relation`, `routes_relation`). Both retrieval
-backends write the same raw-Overpass-JSON shape, so downstream cleaning doesn't
-need to know which one ran. Clean features use `<resources>/clean/*.geojson`;
-generic network components use `<resources>/build/csv/{buses,lines,transformers}.csv`
-and matching GeoJSON files under `<resources>/build/geojson/`, which also
-includes `stations_polygon.geojson` (clustered station shapes) and
-`buses_polygon.geojson` (substation polygons scoped to the buses in the output).
-An interactive map of the network is written to `<resources>/map.html`; it is
-a standalone HTML file (no server required) and the workflow's default target.
-Country logs use `<logs>/retrieve_osm_pbf/{country}.log` or
-`<logs>/retrieve_osm_overpass/{country}.log`, depending on `retrieve.source`. The
-integration example sets these roots to `resources/grid-builder` and
-`logs/grid-builder`. Downloaded PBF files (used for `retrieve.source: geofabrik`)
-are cached in `data/earth-osm` in this checkout.
-
-DC assets (links, converters, switching stations) are out of scope: this workflow
-builds a generic AC topology only, with no PyPSA-specific line types or capacities.
+No user-supplied data files are required; the workflow retrieves OSM data for
+the configured countries. Each public output has a pathvar documented in the
+[interface file](./INTERFACE.yaml).
+
+Main outputs:
+
+| Output | Default location |
+| --- | --- |
+| Network CSVs | `<results>/network/csv/{buses,lines,transformers}.csv` |
+| Network GeoJSONs | `<results>/network/geojson/*.geojson` |
+| Interactive map (default target) | `<results>/map.html` |
+
+Network GeoJSONs include the three network components plus
+`stations_polygon.geojson` and `buses_polygon.geojson`.
+
+Both retrieval backends write raw OSM JSON files to
+`<resources>/automatic/retrieve/{country}_{feature}.json`, where `country` is an
+identifier from the configured `countries`. The six feature names are: `lines_way`,
+`cables_way`, `substations_way`, `substations_node`, `substations_relation`, and
+`routes_relation`.
+
+Cleaning intermediates use `<resources>/automatic/clean/`. Downloaded PBFs are
+cached in `<resources>/automatic/earth-osm/` inside the consuming workflow's
+working directory. Logs use `<logs>/`, with a separate retrieval log per country.
+
+### Importing into another workflow
+
+The host environment needs Python 3.12, Snakemake >=9.27, `pydantic >=2`,
+`ruamel.yaml >=0.18`, `pyyaml >=6,<7`, and `earth-osm >=3.0.2` to load and validate
+the module. Rule dependencies are installed separately by `--use-conda`.
+
+Create `config/modules/grid_builder.yaml` with the module configuration:
+
+```yaml
+grid_builder:
+  countries: [BE]
+  retrieve:
+    source: geofabrik
+```
+
+Then import the module in your Snakefile:
+
+```python
+configfile: "config/modules/grid_builder.yaml"
+
+module grid_builder:
+    snakefile:
+        github(
+            "pypsa/grid-builder",
+            path="workflow/Snakefile",
+            tag="<release-or-commit>",
+        )
+    config:
+        config["grid_builder"]
+    pathvars:
+        logs="logs/grid-builder",
+        resources="resources/grid-builder",
+        results="results/grid-builder",
+        # Optional: rewire an individual result for a downstream module.
+        buses="results/shared/buses.csv",
+
+use rule * from grid_builder as grid_builder_*
+
+rule all_grid_builder:
+    default_target: True
+    input:
+        rules.grid_builder_build_interactive_map.output,
+```
+
+Run `snakemake --use-conda --cores 2 all_grid_builder`. Replace the tag
+placeholder with the published release or commit you want to use. For a local
+checkout, replace `github(...)` with the path to its `workflow/Snakefile`.
+
+Use the output pathvars in [INTERFACE.yaml](./INTERFACE.yaml) to rewire network
+files and the map. Retrieval intermediates can also be redirected: set
+`osm_retrieve` to change their directory, or set an individual pathvar such as
+`osm_lines_way="raw/{country}/lines.json"`. The six retrieval pathvars are
+`osm_lines_way`, `osm_cables_way`, `osm_substations_way`, `osm_substations_node`,
+`osm_substations_relation`, and `osm_routes_relation`.
+
+Retrieval-only consumers can request `rules.grid_builder_retrieve_osm_all.input`
+regardless of where those files are stored. Referencing imported rule outputs
+avoids hard-coding their locations.
 
 ## Development
 
-We use [`pixi`](https://pixi.sh/) as our package manager for development.
-Once installed, run the following to clone this repository and install all dependencies.
+We use [Pixi](https://pixi.sh/) to manage dependencies. Clone the repository and
+install the development environment:
 
 ```shell
-git clone git@github.com:PyPSA/grid-builder.git
+git clone https://github.com/pypsa/grid-builder.git
 cd grid-builder
 pixi install --locked
 ```
 
-For testing, simply run:
+This is a multi-environment project; see [pixi.toml](./pixi.toml):
+
+- `default`: development, validation, and testing tools, including the execution
+  dependencies needed by unit tests.
+- `module`: execution dependencies used by Snakemake rules.
+
+After changing dependencies, update the lock file and export the execution
+environment for module users:
 
 ```shell
-pixi run --locked lint
-pixi run --locked test
+pixi lock
+pixi run --locked export-snakemake-env module
 ```
 
-To test a minimal example of a workflow using this module:
+This writes `workflow/envs/module.yaml` and package pins from `pixi.lock` for
+Linux (`linux-64`), macOS (`osx-arm64`), and Windows (`win-64`). The export command
+also accepts an optional output directory after `module`.
+
+To regenerate the default configuration, schemas, and regional-file index after
+changing the Pydantic models or adding or removing regional config files:
 
 ```shell
-pixi shell                          # activate this project's environment
-cd tests/integration/               # navigate to the integration example
-snakemake --use-conda --cores 2      # run the workflow!
+pixi run --locked generate-config
 ```
 
-The Pixi environment supplies Snakemake and the configuration-validation
-dependencies. Snakemake installs the retrieval script's dependencies from
-`workflow/envs/retrieve.yaml` when `--use-conda` is enabled. A consuming workflow
-must also provide the host dependencies from `pixi.toml`; importing the module
-does not activate its Pixi environment automatically.
+## Testing
+
+Run the checks and test suite:
+
+```shell
+pixi run --locked lint
+pixi run --locked test
+```
 
-The integration test uses a fresh temporary output directory, runs the retrieval,
-cleaning, and generic network Conda environments, and checks the resulting
-components and country logging. It needs internet access on the first run to
-install dependencies and download Benin's OSM extract; subsequent runs can reuse
-those caches. Test logs are retained in `tests/integration/logs`.
+The suite covers configuration validation, network processing, local and HTTP
+module imports, output rewiring, and environment consistency. Remote-import
+tests use synthetic OSM data and a local mock Overpass server.
 
-Each retrieval job runs with one worker (`threads: 1`), so CPU allocation stays
-entirely under Snakemake's control. Snakemake can still run multiple country jobs
-in parallel using `--cores`.
+The live integration test retrieves Benin's OSM extract and runs the workflow
+in its exported Conda environment. It requires internet access for the data
+and, on the first run, environment installation. Conda environments are cached
+between runs. Test logs are retained in `tests/integration/logs`.
 
-If this checkout is moved and commands fail with a `bad interpreter` error,
-rebuild the installed environment with `pixi reinstall --locked`.
+To run only the integration checks:
+
+```shell
+pixi run --locked test-integration
+```
+
+To run the example consuming workflow manually:
+
+```shell
+pixi shell
+cd tests/integration/
+snakemake --use-conda --cores 2
+```
 
 ## License
 
@@ -109,3 +228,9 @@ rebuild the installed environment with `pixi reinstall --locked`.
 * Jonas Hörsch et al. 2018. PyPSA-Eur: An open optimisation model of the European transmission system, *Energy Strategy Reviews*, Volume 22. https://doi.org/10.1016/j.esr.2018.08.012
 * Maximilian Parzen et al. 2023. PyPSA-Earth: A new global open energy system optimization model demonstrated in Africa, *Applied Energy*, Volume 341. https://doi.org/10.1016/j.apenergy.2023.121096
 * Bobby Xiong et al. 2025. Modelling the high-voltage grid using open data for Europe and beyond. *Sci Data* 12, 277. https://doi.org/10.1038/s41597-025-04550-7
+
+## Contributors
+
+See [AUTHORS](./AUTHORS) for copyright attribution and the
+[contributor list](https://github.com/pypsa/grid-builder/graphs/contributors)
+for contributions to the project.

config/README.md

@@ -46,4 +46,13 @@ missing user agent risks being rate-limited or blocked. `retrieve.overpass_api.u
 also lets you point at your own or a faster mirror instance instead of the
 shared public endpoint, without touching the checked-in default.
 
-The generated [schema](./config.schema.json) describes every option.
+The generated [schema](../workflow/internal/config.schema.yaml) describes every
+option using JSON Schema in YAML format. Pydantic models remain the validation
+source of truth; `pixi run generate-config` updates the schema, default config,
+and the JSON copy at `config/config.schema.json` together.
+
+After adding or removing a `config/regions/config.<ISO>.yaml` file, run
+`pixi run generate-config`. It discovers the regional files and generates the
+sorted `config/regions/index.yaml` manifest automatically. Commit the generated
+index with the regional files so remote imports can discover them without a
+local checkout. Do not edit the index manually; a test checks that it is current.

config/config.yaml

@@ -1,6 +1,6 @@
 %YAML 1.1
 ---
-# yaml-language-server: $schema=./config.schema.json
+# yaml-language-server: $schema=../workflow/internal/config.schema.yaml
 countries:
 - BE
 

config/regions/index.yaml

@@ -0,0 +1,7 @@
+# Generated by pixi run generate-config; do not edit manually.
+- BE
+- BR
+- MX
+- NL
+- PH
+- US

mypy.ini

@@ -1,7 +1,7 @@
 [mypy]
 plugins = pydantic.mypy
 explicit_package_bases = True
-mypy_path = workflow
+mypy_path = workflow/scripts
 disable_error_code = import-untyped
 exclude = (^|/)\.(snakemake|pixi)(/|$)
 exclude_gitignore = True