Repository navigation
Data Access: refuse dataset downloads with no time window - #849
Merged
Merged
Conversation
POST v1/data/:uuid read every resolved source over its own range, and a Sparkplug source or a union of them has none, so the request read the device's whole history. On production a machine dataset ran to hundreds of megabytes before the client gave up. The request body now takes an optional from and to, which narrow each source's range and never widen it. If any source is still open at either end, the service returns 422 with a reason that says how to bound it: pass from and to, or download a session dataset. A window outside the dataset's own, or with from not before to, is refused too. Sessions download as before.
The Data tab's download sends the time range in view for a dataset with no window of its own, so it keeps working now that Data Access refuses unbounded downloads. The other download buttons show the service's reason when it refuses, rather than a bare HTTP status. The Use tab's example request includes a window when the dataset needs one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
POST v1/data/:uuiddownloads every resolved source over its own time range. ASparkplug source, or a union of them, has no range, so downloading a machine dataset read every device's whole history. In testing, one machine dataset reached 660 MB before the client was stopped. This PR refuses that download and says how to bound it.What changes
fromandto(ISO 8601) go in the request body. They narrow each source's resolved range and never widen it.422with{"error": "This dataset has no time window, so downloading it would return its whole history. Add from and to to the request, or download a session dataset over the time you need."}. A window outside the dataset's own, an unparseable date, orfromnot beforetois refused the same way, with its own reason.Session limitsdownloads with no window in the body, exactly as before.docs/services/data-access.mddescribes the window and the refusal, and the release notes say what clients need to change.Who this affects
Any client that downloads a device or union dataset without a window now gets
422. That includesDataAccess.download_datain the JS service client, which sends no window. The fix is to download a session over the time needed, or to passfromandto.How to test
cd acs-data-access && npm test: 364 pass. The window tests are intests/current/download_filter.test.js.cd acs-admin && npm test: 844 pass.POST /v1/data/<a machine union>with{}returns422with the reason above, and Influx sees no query.{"from": "<an hour ago>", "to": "<now>"}returns a CSV covering only that hour.POST /v1/data/<a session>with{}returns the session's CSV, as before.Release notes
Data Access no longer downloads a dataset with no time window of its own, such as a machine's device or union dataset, because that read the device's whole history. It returns
422with a reason. Download a session dataset over the time you need, or passfromandtoin the request body. Sessions download as before.🤖 Generated with Claude Code