Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 20 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,10 @@ Every domain runs the same four stages (shared scripts in `etl/lib/`):

1. **fetch** — full dump from live API (Spinta UAPI CSV export with `select(...)`
incl. dereferenced parent keys, e.g. `apskritis.adm_kodas`; Seimas XML). Fails
loudly on empty results. Cache dirs are gitignored.
loudly on empty results. Cache dirs are gitignored. Child feeds keyed by a parent
id (Seimas sessions per term, sittings per session) go through
`etl/lib/fetch-xml-each.sh` (ids read with xmlstarlet, one document per id, merged
under a wrapper element before `xsltproc`).
2. **normalize** — CSV → CSV2RDF identity transform (docker), XML → XSLT 1.0
(`xsltproc`), both emit source-shaped RDF with `<{base}#column>` properties.
3. **graphify** — LDH-style quad `CONSTRUCT { GRAPH ?graph {…} }` mapping
Expand Down Expand Up @@ -107,13 +110,21 @@ since the rows differ in the image cell.
`ldh:ChildrenView` for its own `ldh:View` + `sp:Select` (`ac:GridMode`) so the
listing gets person filter/sort columns — see the next bullet.
- **1:N entity views**: cross-entity listings (county → municipalities, committee
→ members, party → nominees) are `ldh:inverseView` definitions in `app/ns.ttl`
→ members, party → nominees, Seimas → terms, term → sessions / members, session →
sittings, activity type → its sittings or sessions, person → terms) are `ldh:inverseView` definitions in `app/ns.ttl`
(the LDH namespace ontology, northwind-traders style): `<property>
ldh:inverseView <ldh:View>` + `spin:query` → `sp:Select` with `$about`. LDH
shows a view on every instance whose `rdf:type` matches the property's declared
`rdfs:range` — exact type match, no subsumption, so view targeting relies on
discriminating types in the data (`schema:PoliticalParty` for parties,
`cv:PublicOrganisation` for the Seimas). `app/import-ns.sh` (called by `make
`cv:PublicOrganisation` for the Seimas, `schema:EventSeries` / `schema:Event` for
sessions / sittings, which are both `eli-dl:Activity`). Views attach **per
property**: every `ldh:inverseView` of a property shows on every type in its
`rdfs:range`, so a new link needing its own view needs its own property — never
reuse one that already carries a view (`dct:isPartOf` carries `:SubUnits`, hence
session → term is `eli-dl:parliamentary_term`, sitting → session
`eli-dl:forms_part_of`). Views set `ldh:showWhenEmpty false`, so a view whose query
finds nothing for a given instance stays hidden. `app/import-ns.sh` (called by `make
install`) resets + POSTs the ontology into the admin `ontologies/namespace/`
document (served at `{base}ns`) and evicts the server-side ontology cache.
- **View filter/sort columns**: LDH builds a view's facet pills and its sort
Expand Down Expand Up @@ -227,6 +238,9 @@ since the rows differ in the image cell.
variable (e.g. `?graph`) silently unbinds and drops triples. Keep only triple
patterns inside OPTIONAL; do URI construction after it, guarded with
`IF(BOUND(…), …, ?undef)` (see `etl/seimas/mappings/persons.rq`).
- **An exported `JENA_HOME` beats `etl/config.mk`**: it pins 5.6.0 with `?=`, so a
shell that exports `JENA_HOME` (e.g. a 6.x install) silently overrides it and the
quad CONSTRUCT fails. Run the ETL with `JENA_HOME=…/apache-jena-5.6.0 make …`.
- **CSV2RDF must run via docker** — the local jar (`../CSV2RDF/target`) NPEs on
modern JDKs (tested Java 25).
- **Jena `shacl validate` ignores TriG named graphs** (validates the empty
Expand Down Expand Up @@ -326,7 +340,9 @@ since the rows differ in the image cell.

After changing a mapping: rebuild the domain, check the reported graph count
against source counts (10 counties / 60 municipalities / 584 elderships / 148 MPs),
then run the link-integrity pattern from `etl/queries/` (referenced `#this` targets
and the Seimas calendar (10 terms / 131 sessions / 4 771 plenary sittings since 1990,
195 in term 10; 8 scheduled sittings have no date), then run the link-integrity
pattern from `etl/queries/` (referenced `#this` targets
vs `foaf:primaryTopic` set — must be 0 dangling) and eyeball one entity graph.
`python3 etl/queries/render-examples.py` re-runs all example queries end-to-end.
SHACL shapes describe *current* data — if `make` fails the shapes check after a
Expand Down
11 changes: 8 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,9 +67,9 @@ kalba (kaip SQL reliacinėms DB). Pavyzdžiai — [žemiau](#ką-jau-galima-atsa
| Rinkinys | Objektai | Šaltinis |
|---|---|---|
| [`admin-units`](datasets/current/admin-units/) | 10 apskričių, 60 savivaldybių, 584 seniūnijos, 26 229 gyvenamosios vietovės ir 61 149 gatvės (`streets`) | Adresų registras per [get.data.gov.lt](https://get.data.gov.lt/) |
| [`seimas`](datasets/current/seimas/) | 148 Seimo nariai su ~3 000 pareigų/narysčių (galiojimo intervalai), 137 padaliniai (frakcijos, komitetai, komisijos, parlamentinės grupės), 11 partijų | [apps.lrs.lt](https://www.lrs.lt/) XML API |
| [`seimas`](datasets/current/seimas/) | 148 Seimo nariai su ~3 000 pareigų/narysčių (galiojimo intervalai), 139 padaliniai (frakcijos, komitetai, komisijos, parlamentinės grupės), 11 partijų; Seimo kalendorius nuo 1990 m.: 10 kadencijų, 131 sesija ir 4 771 plenarinis posėdis su protokolais ir stenogramomis | [apps.lrs.lt](https://www.lrs.lt/) XML API |
| [`legal-entities`](datasets/current/legal-entities/) | 6 025 valstybės ir savivaldybių biudžetinės įstaigos | JAR per get.data.gov.lt |
| [`taxonomies`](datasets/current/taxonomies/) | 8 SKOS klasifikatoriai: teisinės formos (iš JAR), statusai, pareigų/padalinių/vietovių/gatvių tipai | JAR + išvesta iš duomenų |
| [`taxonomies`](datasets/current/taxonomies/) | 9 SKOS klasifikatoriai: teisinės formos (iš JAR), statusai, pareigų/padalinių/vietovių/gatvių tipai, Seimo veiklos (sesijų ir posėdžių) tipai | JAR + išvesta iš duomenų |

Rinkiniai tarpusavyje **susieti natūraliais raktais** (Adresų registro kodai, JAR kodai, Seimo
asmenų ID) — iš bet kurio išorinio rakto galima sukonstruoti objekto URI. Pavyzdžiui, kiekvienos
Expand All @@ -92,6 +92,10 @@ graph LR
admin -->|geo:lat/long · gsp:asWKT| geo[koordinatės · geometrijos]
admin -->|owl:sameAs + herbai · foaf:img| wd[(Wikidata)]
seimas -->|owl:sameAs + nuotraukos| wd
seimas -->|epvoc:memberDuringParliamentaryTerm| terms[kadencijos]
sittings[plenariniai posėdžiai] -->|eli-dl:forms_part_of| sessions[sesijos]
sessions -->|eli-dl:parliamentary_term| terms
sittings -->|eli-dl:recorded_in_a_realization_of| eseimas[(e-seimas: protokolai, stenogramos)]
```

URI schema ir raktų taisyklės: [`etl/URI-SCHEME.md`](etl/URI-SCHEME.md).
Expand All @@ -105,9 +109,10 @@ Principas: **pirmiausia W3C standartai → tada srities žodynai (ES SEMIC, OP k
| Administraciniai vienetai, adresai | [SEMIC Core Location 2.1.1](https://semiceu.github.io/Core-Location-Vocabulary/releases/2.1.1/) (`cv:AdminUnit`, `cv:level`) + ES [ATU-type](https://op.europa.eu/en/web/eu-vocabularies/dataset/-/resource?uri=http://publications.europa.eu/resource/dataset/atu-type) klasifikatorius (`LTU_APS`, `LTU_SV`, `LTU_SEN`…) |
| Juridiniai asmenys | [RegOrg / Core Business](https://www.w3.org/TR/vocab-regorg/) (`rov:RegisteredOrganization`, `rov:companyType`, `rov:orgStatus`) |
| Organizacijų struktūra, pareigos, narystės | W3C [ORG](https://www.w3.org/TR/vocab-org/) (`org:Membership`, `org:role`, `org:memberDuring`) + [W3C Time](https://www.w3.org/TR/owl-time/) intervalai — **be reifikacijos ir be RDF-star** |
| Seimo kadencijos, sesijos, posėdžiai | ES [ELI-DL](https://data.europa.eu/eli/eli-draft-legislation-ontology) (`eli-dl:ParliamentaryTerm`, `eli-dl:Activity`, `eli-dl:forms_part_of`, `eli-dl:recorded_in_a_realization_of`) + Europos Parlamento [EPVOC](https://data.europarl.europa.eu/def/epvoc) (`epvoc:memberDuringParliamentaryTerm`) |
| Asmenys | FOAF (+ suderinama su [Core Person 2.0](https://semiceu.github.io/Core-Person-Vocabulary/releases/2.00/)) |
| Klasifikatoriai | [SKOS](https://www.w3.org/TR/skos-primer/) |
| Savi terminai | vienintelis `ltlod:nominatedBy` (`http://linkeddata.lt/ns#`) — „iškėlusi partija“, kuriam ES/W3C atitikmens nėra |
| Savi terminai | tik `ltlod:nominatedBy` — „iškėlusi partija“ — ir `ltlod:ElectoralDistrict` / `ltlod:electoralDistrict` — Seimo rinkimų apygarda (`http://linkeddata.lt/ns#`); ES/W3C atitikmenų nėra |

Pasirinkimų motyvacija ir žinomos modeliavimo skolos: [`etl/ONTOLOGY-NOTES.md`](etl/ONTOLOGY-NOTES.md).

Expand Down
10 changes: 5 additions & 5 deletions app/import-ns.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,23 +34,23 @@ ontology_doc="${admin_base}ontologies/namespace/"

printf "\n### Resetting namespace ontology document: %s\n" "$ontology_doc"
{ echo "BASE <${ontology_doc}>"; cat "$app_dir/patch-ontology.ru"; } | ldh patch \
-f "$cert_file" \
-c "$cert_file" \
-p "$cert_password" \
--proxy "$admin_proxy" \
"$ontology_doc"

printf "\n### Appending ns.ttl to the namespace ontology\n"
{ echo "@base <${base}ns> ."; cat "$app_dir/ns.ttl"; } | ldh post \
-f "$cert_file" \
-c "$cert_file" \
-p "$cert_password" \
--proxy "$admin_proxy" \
-t text/turtle \
"$ontology_doc"

printf "\n### Clearing ontology from server memory: %sns#\n" "$base"
ldh admin clear ontology \
-b "$admin_base" \
-f "$cert_file" \
-c "$cert_file" \
-p "$cert_password" \
--proxy "$admin_proxy" \
--ontology "${base}ns#"
--ontology "${base}ns#" \
"$admin_base"
4 changes: 2 additions & 2 deletions app/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ proxy="${4:-$base}"
app_dir="$(cd "$(dirname "$0")" && pwd)"

printf "\n### Creating authorization to make the dataspace public\n"
ldh admin make-public -b "$base" -c "$cert_file" -p "$cert_password" --proxy "$proxy"
ldh admin make-public -c "$cert_file" -p "$cert_password" --proxy "$proxy" "$base"

printf "\n### Pushing root and container documents\n"
ldh push -b "$base" -c "$cert_file" -p "$cert_password" --proxy "$proxy" --dir "$app_dir" "$base"
ldh push -c "$cert_file" -p "$cert_password" --proxy "$proxy" --dir "$app_dir" "$base"

printf "\n### Updating namespace ontology\n"
"$app_dir/import-ns.sh" "$base" "$cert_file" "$cert_password" "$proxy"
Loading
Loading