Enable global applicability

CITATION.cff

@@ -10,3 +10,16 @@ authors:
   - given-names: "Bobby"
     family-names: "Xiong"
     orcid: "https://orcid.org/0000-0003-2854-0730"
+  - given-names: "Daniele"
+    family-names: "Lerede"
+    orcid: ""     
+  - given-names: "Davide"
+    family-names: "Fioriti"
+    orcid: ""
+  - given-names: "Ekaterina"
+    family-names: "Fedotova"
+    orcid: ""    
+  - given-names: "Emmanuel"
+    family-names: "Bolarinwa"
+    orcid: ""    
+

INTERFACE.yaml

@@ -9,7 +9,7 @@ pathvars:
       description: >-
         Raw OSM files are written to retrieve/{country}_{feature}.json.
         Clean features are written below clean; generic network outputs
-        below build (buses/lines/transformers as CSV under build/csv
+        below build (buses/lines/transformers/converters as CSV under build/csv
         and GeoJSON, including substation polygons, under build/geojson).
         An interactive PyDeck map of the network, and the default target
         of this workflow, is written to map.html.

README.md

@@ -1,6 +1,6 @@
 # grid-builder
 
-A modular Snakemake workflow for retrieving OpenStreetMap power infrastructure.
+A modular Snakemake workflow for building a model of transmission power grid for any country of the world. OpenStreetMap is used an a major source of original data and can be supplemented by custom inputs.
 
 <p align="center">
   <img src="./figures/example.png" width="75%">
@@ -12,9 +12,13 @@ A modular Snakemake workflow for retrieving OpenStreetMap power infrastructure.
 
 ## About
 
-`grid-builder` is a modular `snakemake` workflow that retrieves OpenStreetMap power infrastructure and builds a generic high-voltage network. It can be imported into another `snakemake` workflow.
+`grid-builder` is a modular `snakemake` workflow that retrieves [**OpenStreetMap**](https://osm.org) power infrastructure and builds a generic high-voltage network. Custom data can be injected by providing input files of unified structure. The `grid-builder` workflow can be imported into another `snakemake`based project.
 
-The workflow retains AC substations, overhead lines, and cables at configured voltage levels, then creates generic buses, connected line segments, and voltage-pair transformers. The outputs preserve OSM provenance and geometry but contain no PyPSA-specific line types, capacities, or electrical-component assumptions.
+You can help to impove quality of [**OpenStreetMap data**] by joining the [**MapYourGrid initiative**](https://mapyourgrid.org). Many resources as [video tutorials](https://www.youtube.com/channel/UC52jOcw_6_7iTMW-lXwLrQQ) or [starter-kit](https://mapyourgrid.org/starter-kit/) help to improve open data that is used by GridBuilder to build the grid topology.
+
+The GridBuilder workflow retains AC substations, overhead lines, and cables at configured voltage levels, then creates generic buses, connected line segments, and voltage-pair transformers. The outputs preserve OSM provenance and geometry but contain no PyPSA-specific line types, capacities, or electrical-component assumptions.
+
+Buses and lines carry the country they belong to, along with their construction status and planned start date. Country information is essential to resolve assign a correct line types which is strongly regional-specific.
 
 This module follows the Modelblocks conventions (https://www.modelblocks.org). For more information, consult the [integration example](./tests/integration/Snakefile) and the `snakemake` [modularisation documentation](https://snakemake.readthedocs.io/en/stable/snakefiles/modularization.html).
 
@@ -24,23 +28,21 @@ Currently implemented:
 
 1. Retrieve OSM substations, lines, cables, and (optionally) circuit relations by country, either from a cached local Geofabrik PBF extract or the live Overpass API.
 2. Clean the raw retrieval output, filtering voltage, frequency, construction status, and future assets, and grouping relation member ways into one line per real-world circuit.
-3. Merge nearby stations and line endpoints into generic buses, AC lines, and transformers.
-4. Build a self-contained interactive map of the resulting network (`map.html`), with layer toggles, voltage/text filtering, and click-through OSM links — this is the workflow's default target.
+3. Merge nearby stations and line endpoints into generic buses, AC and DC lines, and transformers.
+4. Build an interactive map of the resulting network (`map.html`).
 
 ## Configuration
 
 Configuration lives in [`config/config.yaml`](./config/config.yaml), validated against a generated JSON schema. See the configuration [README](./config/README.md) for the available controls, including retrieval backends, regional overrides, and personal/local settings.
 
 ## Input / output structure
 
-Please consult the [interface file](./INTERFACE.yaml) for more information.
-
 Raw retrieval outputs use `<resources>/retrieve/{country}_{feature}.json`, one
 file per country and feature (`lines_way`, `cables_way`, `substations_way`,
 `substations_node`, `substations_relation`, `routes_relation`). Both retrieval
 backends write the same raw-Overpass-JSON shape, so downstream cleaning doesn't
 need to know which one ran. Clean features use `<resources>/clean/*.geojson`;
-generic network components use `<resources>/build/csv/{buses,lines,transformers}.csv`
+generic network components use `<resources>/build/csv/{buses,lines,transformers,converters}.csv`
 and matching GeoJSON files under `<resources>/build/geojson/`, which also
 includes `stations_polygon.geojson` (clustered station shapes) and
 `buses_polygon.geojson` (substation polygons scoped to the buses in the output).
@@ -52,10 +54,9 @@ integration example sets these roots to `resources/grid-builder` and
 `logs/grid-builder`. Downloaded PBF files (used for `retrieve.source: geofabrik`)
 are cached in `data/earth-osm` in this checkout.
 
-DC assets (links, converters, switching stations) are out of scope: this workflow
-builds a generic AC topology only, with no PyPSA-specific line types or capacities.
+Please consult the [interface file](./INTERFACE.yaml) for more information.
 
-## Development
+## Dependency management
 
 We use [`pixi`](https://pixi.sh/) as our package manager for development.
 Once installed, run the following to clone this repository and install all dependencies.
@@ -106,6 +107,17 @@ rebuild the installed environment with `pixi reinstall --locked`.
 
 ## References & related work
 
+GridBuilder is built on top of other initiatives which created and improved open power infrastructure data, developed and ways to integrate those data into energy modelling workflows.
+
+The list bellow is contains a non-exaustive list of links and references, and can be absolutely expanded and improved.
+
+### Open data and open source projects
+
+* [OpenStreetMap](https://osmfoundation.org/) initiative which is the biggest crowd-sourced database of geospatial information
+* [MapYourGrid](https://mapyourgrid.org/) initiative that empower individuals, communities and nations around the world to map the electrical grid 
+
+### Academic publications
+
 * Jonas Hörsch et al. 2018. PyPSA-Eur: An open optimisation model of the European transmission system, *Energy Strategy Reviews*, Volume 22. https://doi.org/10.1016/j.esr.2018.08.012
 * Maximilian Parzen et al. 2023. PyPSA-Earth: A new global open energy system optimization model demonstrated in Africa, *Applied Energy*, Volume 341. https://doi.org/10.1016/j.apenergy.2023.121096
 * Bobby Xiong et al. 2025. Modelling the high-voltage grid using open data for Europe and beyond. *Sci Data* 12, 277. https://doi.org/10.1038/s41597-025-04550-7

config.CO.yaml

@@ -0,0 +1,5 @@
+# Colombia run used for the main vs integrate-composite-features comparison.
+# Everything else is left at the shipped defaults, so the two branches differ
+# only by code. Run with:
+#   snakemake --cores 2 --configfile config.CO.yaml
+countries: [CO]

config/README.md

@@ -1,49 +1,45 @@
-Set `countries` to the ISO country codes that define the retrieval scope. The
-workflow first loads the default `config/config.yaml`, then an optional
-`config/regions/config.<ISO>.yaml` file for every selected country. A `regions`
-mapping in the calling configuration overrides values from those regional files.
-
-`retrieve.source` picks the retrieval backend: `geofabrik` reads a cached local
-PBF extract (`retrieve_osm_pbf.py`), `overpass` queries the live Overpass API
-(`retrieve_osm_overpass.py`). Both produce the same output shape, so
-`clean` doesn't need to know which one ran. `network.include_relations`
-decides whether the network should consider `route=power`/`power=circuit`
-relations, grouping their member ways into one line per real-world circuit;
-retrieval respects this too, so relations aren't fetched at all when it's off.
-`network` also controls the minimum retained AC voltage, station merge buffer
-radius, construction filtering, and planned-asset cutoff date.
-
-The [BE+NL example](./examples/config.BE-NL.yaml) is a small European development
-scope. Country files under `config/regions` are intentionally small defaults for
-now; community-maintained local corrections belong there rather than in workflow
-code.
-
-`interactive_map` controls the size of `map.html`: `coordinate_decimals` rounds
-embedded coordinates, and `simplify_geometries` sets per-geometry-type
-Douglas-Peucker tolerances (in metres) for station polygons, bus polygons, and
-lines, or disables simplification entirely via `simplify_geometries.enable`.
+Set `countries` to the ISO country codes that define the retrieval scope. The workflow first loads the default `config/config.yaml`, then an optional `config/regions/config.<ISO>.yaml` file for every selected country. A `regions` mapping in the calling configuration overrides values from those regional files.
+
+`retrieve.source` picks the retrieval backend: `geofabrik` reads a cached local PBF extract (`retrieve_osm_pbf.py`), `overpass` queries the live Overpass API (`retrieve_osm_overpass.py`). Retrieven data are transferred to the cleaning phase and after that are used to build a topologically-clean network model.
+
+A parameter `network.include_relations` defines whether the network should consider OSM relations `route=power`/`power=circuit` , grouping their member ways into one line per real-world circuit. In the network-building phase, `network` scripts accepts custom values the minimum retained AC voltage, station merge buffer radius and a construction status.
+
+`network.station_merge_radius_m` is a buffer radius with the merge distance being *twice* as high.E.g.the default value of 500 m merges substations up to one kilometre apart.
+
+### Frequencies and DC lines
+
+Frequency tags are matched numerically within `network.frequency_tolerance_hz`, so `50.0` counts as 50 Hz and `0.0` as DC. Values in `network.accepted_ac_frequencies_hz` are treated as frequencies of a public-grid alternated current (AC) and normalised to the region's nominal AC frequency. Any other value, such as 16.7 Hz corresponding railway traction, is dropped, since it belongs to a separate grid. Where a line lists several circuits, e.g. `voltage=380000;110000` with `frequency=50;16.7`, each frequency is paired with the voltage in the same position.
+
+`network.dc_lines` controls DC lines and cables. With `keep`, the default, they carry `dc: true` and get their own buses at each station. `drop` removes DC lines, and `force_ac` keeps them relabelled as AC. DC has its own voltage floor, `network.minimum_voltage_dc_kv`, since HVDC links often run below the AC floor.
+
+HVDC links combine two approaches. As in PyPSA-Eur, a DC `route=power` relation becomes a single link: parallel poles collapse into one line, the member ways are replaced by it, and its `rating` tag is kept as `p_nom_mw`. DC ways outside any relation are kept too, as in PyPSA-Earth. Converters are written to `converters.csv` by two rules. A station holding both AC and DC buses pairs each DC bus with its AC bus of the closest voltage, as in PyPSA-Earth. A station tagged `substation=converter` with no AC bus of its own pairs with the highest-voltage bus of the nearest AC station within `network.converter_search_radius_m`, as in PyPSA-Eur. The `pairing` column records which rule applied.
+
+The [BE+NL example](./examples/config.BE-NL.yaml) is a small European development scope. Country files under `config/regions` are intentionally small defaults for now; community-maintained local corrections belong there rather than in workflow code.
+
+### Adding custom data
+
+The workflow provides an option to add custom data which can be handy to deal with inputs which are out of scope for OpenStreetMap, such as planned lines. To inject custom files into the worklow, a filed `custom_data.files` can be used:
+
+```yaml
+custom_data:
+  files:
+    - data/custom/BE_lines_way.json
+```
+
+Enabling this functionality makes `clean` read the custom files alongside the retrieved ones. The expected format correspond to raw elements and must be named `{country}_{feature}.json`. The feature has to be one the retrieval step produces: `lines_way`, `cables_way`, `substations_way`, `substations_node`, `substations_relation`, or `routes_relation`. 
+
+`interactive_map` controls the parameters of `map.html` with `coordinate_decimals` for displayed precision of coordinates, and `simplify_geometries` in meteres applied to station polygons, bus polygons, and lines.
 
 ### Personal settings and Overpass fair use
 
-Keep `config/config.yaml` as pure defaults — a test enforces that it matches the
-schema, and it's tracked in git, so it's not the place for anything
-environment- or person-specific. For local overrides (a custom Overpass
-endpoint, contact details, a smaller `countries` scope for development), create
-an untracked `config/config.local.yaml` and pass it alongside the default:
+Keep a git-tracked `config/config.yaml` as pure defaults. A test enforces that this file matches the schema. For local overrides, such as a custom Overpass endpoint, contact details, a smaller `countries` scope, please create an untracked `config/config.local.yaml` and pass it alongside the default:
 
 ```shell
 snakemake --configfile config/config.local.yaml ...
 ```
 
-Snakemake deep-merges it on top of `config/config.yaml`, so you only need to
-list the keys you're overriding.
+Snakemake deep-merges it on top of `config/config.yaml`, so you only need to list the keys you're overriding.
 
-If you use `retrieve.source: overpass`, set `retrieve.overpass_api.user_agent`
-to your own project name, contact email, and website. The [Overpass API fair
-use policy](https://wiki.openstreetmap.org/wiki/Overpass_API#Fair_use_policy)
-expects automated queries to be identifiable and reachable; a generic or
-missing user agent risks being rate-limited or blocked. `retrieve.overpass_api.url`
-also lets you point at your own or a faster mirror instance instead of the
-shared public endpoint, without touching the checked-in default.
+If you use `retrieve.source: overpass`, set `retrieve.overpass_api.user_agent` to your own project name, contact email, and website. The [Overpass API fair use policy](https://wiki.openstreetmap.org/wiki/Overpass_API#Fair_use_policy) expects automated queries to be identifiable and reachable; a generic or missing user agent risks being rate-limited or blocked. `retrieve.overpass_api.url` also lets you point at your own or a faster mirror instance instead of the shared public endpoint, without touching the checked-in default.
 
 The generated [schema](./config.schema.json) describes every option.