Skip to content

Faster loading and saving of large scenarios - #233

Merged
MarleneBusch merged 6 commits into
mainfrom
perf/fast-scenario-loading
Nov 3, 2026
Merged

MarleneBusch merged 6 commits into
mainfrom
perf/fast-scenario-loading

Conversation

@kalebphipps

Copy link
Copy Markdown
Member

Description

Most of the slow loading seems to be going into evaluating the NURBS surface of every heliostat one at a time (twice, once for points and once for normals), plus reading and copying every value of every heliostat to the device individually.

This PR aims to improve this without any changes for existing user code: the same functions are called with the same arguments.

NOTE: I have given this code a brief review, but we should test it on the cluster first and review in detail before merging

Key Changes

Loading

  • All heliostat data is read onto the CPU first, grouped, and moved to the device once per heliostat group.
  • Surface points and normals are computed in batched NURBS evaluations per group instead of per heliostat. The batch size can be set via the new optional surface_evaluation_chunk_size argument of Scenario.load_scenario_from_hdf5() (default 1024) if device memory is tight.
  • Warnings about missing parameters and prototype usage are summarised over all heliostats instead of logged once per heliostat.

New columnar layout for heliostats

  • H5ScenarioGenerator now stores the heliostats column-wise: one dataset per parameter for all heliostats, instead of one group per heliostat. The rest of the scenario file (tower, light sources, prototypes) is unchanged.
  • If the heliostats can't be stored this way (e.g. individual surfaces with different shapes), the generator logs a warning and falls back to the legacy layout.
  • Old scenarios still load. They produce a warning that the legacy layout is slower and can be converted with artist.io.convert_scenario_to_columnar(old_path, new_path).
  • ARTIST 2.0.0 and earlier can't read the columnar layout.

Type of change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

Reverts 50968be, which undid these changes on main after they were pushed
there by mistake. They now come back through a pull request.
@kalebphipps kalebphipps self-assigned this Oct 28, 2026
@kalebphipps kalebphipps added the enhancement New feature or request label Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Name Stmts Miss Cover Missing
artist\__init__.py 9 0 100%
artist\field\__init__.py 14 0 100%
artist\field\actuators.py 16 0 100%
artist\field\actuators_ideal.py 9 0 100%
artist\field\actuators_linear.py 57 0 100%
artist\field\heliostat_field.py 177 0 100%
artist\field\heliostat_group.py 52 0 100%
artist\field\heliostat_group_rigid_body.py 23 0 100%
artist\field\kinematics.py 12 0 100%
artist\field\kinematics_rigid_body.py 98 0 100%
artist\field\solar_tower.py 48 0 100%
artist\field\surface.py 22 13 41% 45-55, 88-121
artist\field\tower_target_areas.py 15 0 100%
artist\field\tower_target_areas_cylindrical.py 40 0 100%
artist\field\tower_target_areas_planar.py 32 0 100%
artist\flux\__init__.py 2 0 100%
artist\flux\bitmap.py 65 1 98% 116
artist\geometry\__init__.py 4 0 100%
artist\geometry\coordinates.py 95 1 99% 293
artist\geometry\rotations.py 31 1 97% 112
artist\geometry\transforms.py 97 1 99% 346
artist\io\__init__.py 4 0 100%
artist\io\calibration_parser.py 42 2 95% 197-202
artist\io\h5_columnar_heliostats.py 172 1 99% 161
artist\io\h5_scenario_parser.py 149 6 96% 43, 550, 554, 560, 565, 570
artist\io\paint_calibration_parser.py 66 0 100%
artist\io\paint_scenario_parser.py 193 0 100%
artist\io\stral_scenario_parser.py 42 0 100%
artist\nurbs\__init__.py 3 0 100%
artist\nurbs\surfaces.py 156 20 87% 210-241, 395, 399, 401-402, 553-554, 557-558
artist\nurbs\utils.py 24 0 100%
artist\optim\__init__.py 7 0 100%
artist\optim\aim_point_optimizer.py 186 10 95% 516, 695-702, 713-722
artist\optim\kinematics_reconstructor.py 207 15 93% 640-648, 677-700, 832, 1001
artist\optim\loss.py 85 0 100%
artist\optim\regularizers.py 24 0 100%
artist\optim\surface_reconstructor.py 212 12 94% 769-777, 817-835
artist\optim\training.py 67 1 99% 181
artist\raytracing\__init__.py 4 0 100%
artist\raytracing\blocking.py 283 9 97% 378, 380, 552-555, 592-593, 722, 737, 975
artist\raytracing\geometry.py 91 0 100%
artist\raytracing\heliostat_ray_tracer.py 142 1 99% 176
artist\raytracing\sampling.py 29 0 100%
artist\scenario\__init__.py 4 0 100%
artist\scenario\h5_scenario_generator.py 90 1 99% 162
artist\scenario\scenario.py 94 2 98% 219, 439
artist\scenario\surface_generator.py 89 0 100%
artist\scene\__init__.py 5 0 100%
artist\scene\light_source.py 11 0 100%
artist\scene\light_source_array.py 28 0 100%
artist\scene\rays.py 7 0 100%
artist\scene\sun.py 43 0 100%
artist\util\__init__.py 56 18 68% 101-135
artist\util\config.py 203 0 100%
artist\util\constants.py 315 0 100%
artist\util\env.py 110 23 79% 89-91, 186-228, 308
artist\util\indices.py 324 0 100%
artist\util\type_registry.py 11 0 100%
TOTAL 4496 138 97%

@codecov

codecov Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.69970% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.93%. Comparing base (50968be) to head (0366fad).

Files with missing lines Patch % Lines
artist/io/h5_columnar_heliostats.py 99.41% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #233      +/-   ##
==========================================
+ Coverage   96.41%   96.93%   +0.51%     
==========================================
  Files          57       58       +1     
  Lines        4245     4496     +251     
==========================================
+ Hits         4093     4358     +265     
+ Misses        152      138      -14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@MarleneBusch
MarleneBusch merged commit 66fab22 into main Nov 3, 2026
13 checks passed
@MarleneBusch
MarleneBusch deleted the perf/fast-scenario-loading branch November 3, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants