Remove the deprecated nwis module (without the STA8 copy edits) by thodson-usgs · Pull Request #458 · DOI-USGS/dataretrieval-python

thodson-usgs · 2026-10-27T19:08:28Z

This is #457 without its two STA8 copy-edit commits (a87052e5, a99c0763), which touch 71 files unrelated to the nwis removal and are hard to review alongside it. The other seven commits are cherry-picked unchanged. Apart from those copy edits, the tree is identical to #457.

Summary

This removes dataretrieval.nwis before its published 2027-05-06 horizon, as part of the next major release. USGS is decommissioning the waterservices endpoints the module called, and every getter in it already has a waterdata or nwdc replacement.

Breaking:

Importing dataretrieval.nwis in any form (import dataretrieval.nwis, from dataretrieval import nwis, from dataretrieval.nwis import get_dv) raises ImportError. The message points to waterdata (and nwdc for water use) and says how to keep using nwis while the endpoints are still online: pip install "dataretrieval<=1.4.0". nwis.py is now a stub that does nothing except raise that error. NEWS gives the same pin.
NoSitesError is removed, both dataretrieval.NoSitesError and exceptions.NoSitesError. Only the waterservices path raised it. The modern getters return an empty DataFrame when nothing matches.

Code

Removed the body of nwis.py, deleted tests/nwis_test.py, tests/waterservices_test.py, and 9 fixtures used only by those tests.
Deleted REMOVALS["nwis"], the "No sites/data" body check in _querying, and _validation.require_any_of, whose only caller was nwis.
.importlinter: dropped nwis from the layers contract and deleted the nwis-quarantine contract. Also updated the pyproject comment, the pre-commit comments, and the pyscn clone baseline.
Rewrote the generic tests that used nwis only to reach shared code (utils.query, the deprecation mechanism) so they call that code directly.
rdb.read_rdb's HTML-response message used to tell callers to migrate to waterdata. Only waterdata.get_ratings reaches it now, so the message asks them to report the moved endpoint instead.

Notebooks and demos

hydroshare/USGS_NWIS_WaterUse_Examples.ipynb was defunct and couldn't run. It is rewritten on nwdc.get_wateruse and renamed USGS_NWDC_WaterUse_Examples.ipynb.
nwqn_data_pull scripts now use waterdata.get_monitoring_locations and get_daily. Approval filtering happens on the server (approval_status="Approved") instead of matching the old NWIS _cd strings.
Copy-edited the notebooks that still called nwis current or said waterdata "will replace" it, and removed the unused nwis import from the R-vignette notebook.
Added nblinks for the two notebooks that weren't in the docs: USGS_WaterUse_Examples and hydroshare/USGS_NLDI_Examples. Updated examples/index.rst.

Prose

Updated README, NEWS (new 10/22/2026 entry with a migration map), CONTEXT (NWIS entry now marked retired), AGENTS, CONTRIBUTING, the architecture overview, and userguide/dataportals.rst.
ADR 0005 is marked completed. Where an ADR's Compliance section named the deleted contract or tests, I added a short dated note and left the historical reasoning alone.

Other deprecations evaluated

None of the other published horizons is earlier than May 2027, so they all stay. They are listed in the NEWS entry:

Surface	Horizon
`get_time_series_metadata` v0 filters	2027-06-01 (set by the service's v0 retirement)
`get_cql(service=)`	2027-08-09
`wateruse` alias	2027-08-11
`ogc.interruptions`	2027-08-25

The camelCase get_samples kwargs have no horizon published.

Remaining "NWIS" mentions (intentional)

Domain vocabulary: WQP providers=NWIS, NLDI nwissite/nwisgw, NGWMN organization_type="NWIS", the USGS NWIS time-zone code list, and docstrings saying site numbers are assigned by NWIS.
R readNWIS* calls quoted in the R-vignette notebook comments.
Historical record: earlier NEWS entries, ADR narrative, and the CONTEXT glossary entry.

Testing

Run on this branch:

ruff check, ruff format --check, lint-imports (7 kept, 0 broken), xenon, and complexipy all pass.
pytest: 1652 passed. Branch coverage is 99.07%, above fail_under 98.9.
mypy --python-version 3.12 reports no issues in 61 source files.
import dataretrieval.nwis raises the ImportError that names waterdata, nwdc, and the dataretrieval<=1.4.0 pin.

Run on #457 and not repeated on this branch:

make html succeeds and the executed notebooks render. The only cell errors are NetworkErrors on WQP calls in rvignettes and DiscreteSamples, caused by a local SSL problem that also reproduces with a bare httpx.get.
Ran against live services: the NWDC water-use notebook, WaterData_demo, Introduction, and peak_streamflow_trends. Ran retrieve_nwqn_streamflow.map_retrieval on one site with lithops and S3 stubbed out.
Not verified end to end: the full nwqn lithops/S3 pipeline.

🤖 Generated with Claude Code

Remove dataretrieval.nwis ahead of its 2027-05-06 horizon. USGS is decommissioning the waterservices endpoints it called, and every getter has a waterdata (or nwdc) replacement. - Delete dataretrieval/nwis.py, its tests and fixtures, and its reference page. - Remove NoSitesError (only the waterservices path raised it) and the "No sites/data" body check in _querying; remove REMOVALS["nwis"] and the now-unused _validation.require_any_of. - Drop nwis from .importlinter (layers + nwis-quarantine contract), pyproject, pre-commit comments, and the pyscn clone baseline. - Port demos: the defunct hydroshare water-use notebook is rewritten on nwdc.get_wateruse (USGS_NWDC_WaterUse_Examples); the nwqn_data_pull scripts use waterdata.get_monitoring_locations / get_daily; copy-edit notebooks that still described nwis as current. - Docs: add missing nblinks for USGS_WaterUse_Examples and USGS_NLDI_Examples; update README, NEWS, CONTEXT, AGENTS, CONTRIBUTING, the architecture overview, and mark ADR 0005 completed. BREAKING CHANGE: `import dataretrieval.nwis` raises ModuleNotFoundError, and dataretrieval.NoSitesError / exceptions.NoSitesError are removed.

- Drop the retry wrapper the NWQN demo put around waterdata.get_monitoring_locations, which already retries; carry USGS- IDs end to end instead of stripping and re-adding the prefix; delete the agency_code filter that no longer removes anything. - Delete the ignore-variadic-names ruff exemption: with it off, ARG001 reports nothing. - Use the glossary's "single-shot" instead of the new "single-request", and state shared-module rules without hand-written adapter lists (one said NGWMN never receives the key). - Replace the WQP-specific URL-too-long pseudo-code with a remedy that holds for every caller; make the format_datetime warning name the columns to check rather than the nonexistent datetime_index. - Move the ADR "Note (2026-10)" paragraphs into each record's Notes with an amended Status (ADR 0000); restore ADR 0010's decision-time survey; correct ADR 0003's note, which said rules still present were deleted. - Document that utils.query does not retry, the statement the removal deleted along with its NWIS rationale. - Remove a duplicate URL-too-long test, the NWIS payload in test_header, and the hard-coded wateruse horizon in the deprecation test. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

find_neighboring_sites already fetched every neighbor's monitoring location to filter by drainage area, then returned only the IDs, so the streamflow script requested the same rows again. It now returns the frame; only the BAD_NLDI_SITES branch fetches on its own. The streamflow get_daily call requests only the three columns the script reads, and the unused update_dataframe(overwrite=) parameter and project locals are gone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Restore dataretrieval/nwis.py as a stub that raises ImportError on import, so every spelling (import dataretrieval.nwis, from dataretrieval import nwis, from dataretrieval.nwis import get_dv) names the waterdata replacements and the last release that ships the module: pip install "dataretrieval==1.4.0". NEWS, CONTEXT, and ADR 0005 say the same, and .importlinter places the stub in the top layer. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Point to the waterdata and nwdc modules instead of listing each getter, matching the nwis ImportError. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

ehinman · 2026-10-28T02:23:11Z

-and ``dataretrieval.waterdata.ratings`` use. Callers layer their own
-post-processing (NWIS-specific datetime indexing, ratings-specific
-``df.attrs`` provenance, etc.) on top of the raw frame.
+This module exposes the parsing primitives. Callers layer their own


Suggested change

This module exposes the parsing primitives. Callers layer their own

This module exposes the parsing primitives used by ``dataretrieval.waterdata.get_ratings``. Callers layer their own

ehinman · 2026-10-28T02:30:50Z

    "# USGS dataretrieval Python Package Peak Streamflow Examples\n",
    "\n",
-    "This notebook provides examples of using the Python dataretrieval package to retrieve annual peak streamflow data for United States Geological Survey (USGS) monitoring locations using the **USGS Water Data API** via the `waterdata` module. The `waterdata` module is the recommended way to access USGS water data and replaces the deprecated `nwis` module."
+    "This notebook provides examples of using the Python dataretrieval package to retrieve annual peak streamflow data for United States Geological Survey (USGS) monitoring locations using the **USGS Water Data API** via the `waterdata` module. The `waterdata` module is the recommended way to access USGS water data."


Suggested change

"This notebook provides examples of using the Python dataretrieval package to retrieve annual peak streamflow data for United States Geological Survey (USGS) monitoring locations using the **USGS Water Data API** via the `waterdata` module. The `waterdata` module is the recommended way to access USGS water data."

"This notebook provides examples of using the Python dataretrieval package to retrieve annual peak streamflow data for United States Geological Survey (USGS) monitoring locations using the **USGS Water Data API** via the `waterdata` module."

ehinman · 2026-10-28T02:31:25Z

    "# USGS dataretrieval Python Package Rating Curve Examples\n",
    "\n",
-    "This notebook provides examples of using the Python dataretrieval package to retrieve stage–discharge rating curve data for United States Geological Survey (USGS) streamgages using the **USGS Water Data API** via the `waterdata` module. The `waterdata` module is the recommended way to access USGS water data and replaces the deprecated `nwis` module.\n",
+    "This notebook provides examples of using the Python dataretrieval package to retrieve stage–discharge rating curve data for United States Geological Survey (USGS) streamgages using the **USGS Water Data API** via the `waterdata` module. The `waterdata` module is the recommended way to access USGS water data.\n",


Suggested change

"This notebook provides examples of using the Python dataretrieval package to retrieve stage–discharge rating curve data for United States Geological Survey (USGS) streamgages using the **USGS Water Data API** via the `waterdata` module. The `waterdata` module is the recommended way to access USGS water data.\n",

"This notebook provides examples of using the Python dataretrieval package to retrieve stage–discharge rating curve data for United States Geological Survey (USGS) streamgages using the **USGS Water Data API** via the `waterdata` module.\n",

ehinman · 2026-10-28T02:31:57Z

    "# USGS dataretrieval Python Package Statistics Examples\n",
    "\n",
-    "This notebook provides examples of using the Python dataretrieval package to retrieve summary statistics for observed variables at a United States Geological Survey (USGS) monitoring location using the **USGS Water Data API** via the `waterdata` module. The `waterdata` module is the recommended way to access USGS water data and replaces the deprecated `nwis` module.\n",
+    "This notebook provides examples of using the Python dataretrieval package to retrieve summary statistics for observed variables at a United States Geological Survey (USGS) monitoring location using the **USGS Water Data API** via the `waterdata` module. The `waterdata` module is the recommended way to access USGS water data.\n",


Suggested change

"This notebook provides examples of using the Python dataretrieval package to retrieve summary statistics for observed variables at a United States Geological Survey (USGS) monitoring location using the **USGS Water Data API** via the `waterdata` module. The `waterdata` module is the recommended way to access USGS water data.\n",

"This notebook provides examples of using the Python dataretrieval package to retrieve summary statistics for observed variables at a United States Geological Survey (USGS) monitoring location using the **USGS Water Data API** via the `waterdata` module.\n",

ehinman · 2026-10-28T02:42:09Z

-The ``waterdata`` module will replace the ``nwis`` module as the primary
-set of data download functions for USGS water data. This Jupyter notebook
-covers a basic introduction to module functions and usage.
+The ``waterdata`` module is the primary set of data download functions for


Suggested change

The ``waterdata`` module is the primary set of data download functions for

The ``waterdata`` module contains the set of data download functions for

ehinman · 2026-10-28T02:45:56Z

    def test_key_excluded_for_external_host(self):
        """The key is not added for an external (non-USGS) host."""
-        url = "https://nwis.waterservices.usgs.gov/nwis/iv/"
+        url = "https://example.invalid/data/"


Why was this changed? Because nwis.waterservices.... is still in existence?

ehinman · 2026-10-28T02:48:02Z


 def test_query_nldi_opts_into_retry(monkeypatch):
-    """NLDI explicitly enables shared retry while NWIS remains unchanged."""
+    """NLDI explicitly enables shared retry while the default path stays unchanged."""


Does it make sense to refactor the architecture supporting these endpoints ("the default path") now that nwis is gone, or is the architecture still robust with the remaining endpoints?

ehinman · 2026-10-28T14:23:43Z

-        }
-        response = utils.query(url, payload)
+        httpx_mock.add_response(method="GET")
+        response = utils.query("https://example.invalid/x", {"a": "1"})


Is this really what we want/need? Seems like it used to test a "valid" url and now it's testing an invalid one?

ehinman · 2026-10-28T14:33:52Z

@@ -1,3 +1,5 @@
+**10/22/2026:** **Breaking change:** `dataretrieval.nwis` is removed, ahead of its 2027-05-06 horizon, because USGS is [decommissioning the legacy waterservices endpoints](https://waterdata.usgs.gov/blog/api-waterservices-decom/) the module called. `import dataretrieval.nwis` (or any import from it) now raises `ImportError`. Use `waterdata` instead, or `nwdc` for water use. To keep using `nwis` while the waterservices endpoints remain online, pin the last release that includes it: `pip install "dataretrieval<=1.4.0"`. **Breaking change:** `dataretrieval.exceptions.NoSitesError` (and its `dataretrieval.NoSitesError` re-export) is removed; only the waterservices path raised it, and the modern getters return an empty `DataFrame` when nothing matches. The other published deprecations are unchanged and remain on their horizons: the `get_time_series_metadata()` v0 filters (`begin_utc`, `end_utc`, `state`, `state_name`, `hydrologic_unit_code`) on or after 2027-06-01, `get_cql(service=)` on or after 2027-08-09, the `wateruse` module alias on or after 2027-08-11, and `ogc.interruptions` on or after 2027-08-25.


Suggested change

**10/22/2026:** **Breaking change:** `dataretrieval.nwis` is removed, ahead of its 2027-05-06 horizon, because USGS is [decommissioning the legacy waterservices endpoints](https://waterdata.usgs.gov/blog/api-waterservices-decom/) the module called. `import dataretrieval.nwis` (or any import from it) now raises `ImportError`. Use `waterdata` instead, or `nwdc` for water use. To keep using `nwis` while the waterservices endpoints remain online, pin the last release that includes it: `pip install "dataretrieval<=1.4.0"`. **Breaking change:** `dataretrieval.exceptions.NoSitesError` (and its `dataretrieval.NoSitesError` re-export) is removed; only the waterservices path raised it, and the modern getters return an empty `DataFrame` when nothing matches. The other published deprecations are unchanged and remain on their horizons: the `get_time_series_metadata()` v0 filters (`begin_utc`, `end_utc`, `state`, `state_name`, `hydrologic_unit_code`) on or after 2027-06-01, `get_cql(service=)` on or after 2027-08-09, the `wateruse` module alias on or after 2027-08-11, and `ogc.interruptions` on or after 2027-08-25.

**10/22/2026:** **Breaking change:** `dataretrieval.nwis` is removed, ahead of its 2027-02-22 horizon, because USGS is [decommissioning the legacy waterservices endpoints](https://waterdata.usgs.gov/blog/api-waterservices-decom/) the module called. `import dataretrieval.nwis` (or any import from it) now raises `ImportError`. Use `waterdata` instead, or `nwdc` for water use. To keep using `nwis` while the waterservices endpoints remain online, pin the last release that includes it: `pip install "dataretrieval<=1.4.0"`. **Breaking change:** `dataretrieval.exceptions.NoSitesError` (and its `dataretrieval.NoSitesError` re-export) is removed; only the waterservices path raised it, and the modern getters return an empty `DataFrame` when nothing matches. The other published deprecations are unchanged and remain on their horizons: the `get_time_series_metadata()` v0 filters (`begin_utc`, `end_utc`, `state`, `state_name`, `hydrologic_unit_code`) on or after 2027-06-01, `get_cql(service=)` on or after 2027-08-09, the `wateruse` module alias on or after 2027-08-11, and `ogc.interruptions` on or after 2027-08-25.

ehinman · 2026-10-28T16:21:11Z

Opened a PR that addresses a use case where someone might try loading the nwis module via dataretrieval.nwis.get_dv(). The package now produces a more informative attribute error.

thodson-usgs#9

thodson-usgs and others added 7 commits October 22, 2026 14:05

refactor: point the nwis ImportError at modules, not getters

7aff11e

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

refactor: pin nwis users to dataretrieval<=1.4.0

c6f713b

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs: consolidate the NEWS entry for the nwis removal

0bdb810

Point to the waterdata and nwdc modules instead of listing each getter, matching the nwis ImportError. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

thodson-usgs changed the title ~~Remove the deprecated nwis module (removal commit only, from #457)~~ Remove the deprecated nwis module (without the STA8 copy edits) Oct 27, 2026

ehinman reviewed Oct 28, 2026

View reviewed changes

Navigation Menu

Repository navigation

Remove the deprecated nwis module (without the STA8 copy edits) - #458

Remove the deprecated nwis module (without the STA8 copy edits)#458
thodson-usgs wants to merge 7 commits into
DOI-USGS:mainfrom
thodson-usgs:remove-nwis-commit

thodson-usgs commented Oct 27, 2026 •

edited

Loading

Uh oh!

ehinman Oct 28, 2026

Uh oh!

ehinman Oct 28, 2026

Uh oh!

ehinman Oct 28, 2026

Uh oh!

ehinman Oct 28, 2026

Uh oh!

ehinman Oct 28, 2026

Uh oh!

ehinman Oct 28, 2026

Uh oh!

ehinman Oct 28, 2026

Uh oh!

ehinman Oct 28, 2026

Uh oh!

ehinman Oct 28, 2026

Uh oh!

ehinman commented Oct 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	This module exposes the parsing primitives. Callers layer their own
	This module exposes the parsing primitives used by ``dataretrieval.waterdata.get_ratings``. Callers layer their own

	"This notebook provides examples of using the Python dataretrieval package to retrieve annual peak streamflow data for United States Geological Survey (USGS) monitoring locations using the USGS Water Data API via the `waterdata` module. The `waterdata` module is the recommended way to access USGS water data."
	"This notebook provides examples of using the Python dataretrieval package to retrieve annual peak streamflow data for United States Geological Survey (USGS) monitoring locations using the USGS Water Data API via the `waterdata` module."

	The ``waterdata`` module is the primary set of data download functions for
	The ``waterdata`` module contains the set of data download functions for

		@@ -1,3 +1,5 @@
		10/22/2026: Breaking change: `dataretrieval.nwis` is removed, ahead of its 2027-05-06 horizon, because USGS is [decommissioning the legacy waterservices endpoints](https://waterdata.usgs.gov/blog/api-waterservices-decom/) the module called. `import dataretrieval.nwis` (or any import from it) now raises `ImportError`. Use `waterdata` instead, or `nwdc` for water use. To keep using `nwis` while the waterservices endpoints remain online, pin the last release that includes it: `pip install "dataretrieval<=1.4.0"`. Breaking change: `dataretrieval.exceptions.NoSitesError` (and its `dataretrieval.NoSitesError` re-export) is removed; only the waterservices path raised it, and the modern getters return an empty `DataFrame` when nothing matches. The other published deprecations are unchanged and remain on their horizons: the `get_time_series_metadata()` v0 filters (`begin_utc`, `end_utc`, `state`, `state_name`, `hydrologic_unit_code`) on or after 2027-06-01, `get_cql(service=)` on or after 2027-08-09, the `wateruse` module alias on or after 2027-08-11, and `ogc.interruptions` on or after 2027-08-25.

Conversation

thodson-usgs commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Other deprecations evaluated

Remaining "NWIS" mentions (intentional)

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ehinman commented Oct 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

thodson-usgs commented Oct 27, 2026 •

edited

Loading