Skip to content

refactor: modernize to Python 3.11 typing syntax (PEP 585, PEP 604) - #347

Open
atrearyan wants to merge 7 commits into
google:mainfrom
atrearyan:refactor/py311
Open

atrearyan wants to merge 7 commits into
google:mainfrom
atrearyan:refactor/py311

Conversation

@atrearyan

Copy link
Copy Markdown

Summary

Modernizes the codebase to Python >=3.11 typing syntax as described in #344: PEP 585 built-in generics, PEP 604 X | Y unions, typing.Self where it applies, and consistent python_requires / classifiers.

The issue suggests one PR per package. This PR keeps that split as one commit per package, so it can be reviewed commit by commit, and I'm happy to break it into separate PRs if you prefer:

Commit Package Files
1 server/gti 10
2 server/secops 29
3 server/scc 3
4 server/secops-soar (core) 8
5 server/secops-soar/secops_soar_mcp/marketplace (generated) 294
6 run-with-google-adk 11

One deliberate exception: typing.Dict on tool return annotations

FastMCP (mcp 1.10 through 1.30) only builds an unwrapped <tool>DictOutput schema when a tool's return annotation is a built-in dict[...] (types.GenericAlias). typing.Dict[str, X] takes a different path and gets the wrapped {"result": ...} schema, and bare dict gets no outputSchema at all. Converting these annotations would therefore change outputSchema and the shape of structuredContent returned to clients for 62 tools (12 gti, 44 secops, 6 scc), and remove outputSchema from get_collection_mitre_tree.

To keep schemas unchanged, those 62 return annotations stay typing.Dict with # noqa: UP006 - typing.Dict keeps FastMCP {"result": ...} outputSchema (plus # noqa: UP035 on the matching import). Everything else is converted. A new gti test pins the wrapped schema so a future ruff --fix cannot silently change it. Upstream python-sdk is changing this check for mcp 2.x, so these markers should be removed together with an mcp 2.x upgrade.

Other changes

  • server/gti: except (asyncio.TimeoutError, TimeoutError) becomes except TimeoutError (they are the same class on 3.11+).
  • run-with-google-adk: the settings validator returns typing.Self; the typer floor is raised to >=0.12.4, because earlier versions cannot build CLI commands with X | None parameters.
  • python_requires=">=3.11" added to the four setup.py files, and 3.12/3.13/3.14 classifiers added to the four server pyproject.toml files.
  • No from __future__ import annotations was added anywhere, and nullable tool parameters keep X | None = None.

Testing

  • For every package, list_tools() output (inputSchema, outputSchema, descriptions) was compared before and after: byte-identical for all 2,258 tools (gti 36, secops 70, scc 6, SOAR core 17, marketplace 2,123, ADK sample servers 6).
  • call_tool() round-trips with mocked backends return identical structuredContent, including explicit null arguments.
  • pytest, on Python 3.11 through 3.14:
    • gti: 73 passed.
    • secops: 51 passed, plus 54 integration tests that need a live tests/config.json, unchanged from main.
    • secops-soar: 11 passed.
    • run-with-google-adk: 44 passed.
  • scc has no test suite. A smoke test of list_tools() and call_tool() matched main.
  • The ADK typer CLI options, FastAPI OpenAPI schema, and AgentSettings JSON schema are identical before and after.

Notes for reviewers

Closes #344

Replace typing.List/Dict/Optional/Union with built-in generics and
X | Y unions, and catch the builtin TimeoutError (asyncio.TimeoutError
is an alias of it on Python 3.11+).

Tool return annotations whose outermost type is typing.Dict stay
typing.Dict with `# noqa: UP006`. FastMCP builds an unwrapped
DictOutput schema only for built-in dict[...], so rewriting them would
drop the {"result": ...} wrapper from outputSchema and structuredContent
on 12 tools and remove outputSchema from get_collection_mitre_tree. A
new test pins the wrapped schema for those tools.

Also add python_requires=">=3.11" to setup.py and 3.12-3.14
classifiers to pyproject.toml.
Replace typing.List/Dict/Tuple/Optional/Union with built-in generics
and X | Y unions, and drop typing imports left unused.

Tool return annotations whose outermost type is typing.Dict stay
typing.Dict with `# noqa: UP006` (and `# noqa: UP035` on the import),
because FastMCP builds an unwrapped DictOutput schema only for built-in
dict[...]. Rewriting them would change outputSchema and
structuredContent on 44 tools.

Also add python_requires=">=3.11" to setup.py and 3.12-3.14
classifiers to pyproject.toml.
Use built-in dict[...] in the non-tool helper and drop the unused
typing.List import. The six tool return annotations stay typing.Dict
with `# noqa: UP006` (and `# noqa: UP035` on the import) to keep
FastMCP's {"result": ...} outputSchema unchanged.

Also add python_requires=">=3.11" to setup.py and 3.12-3.14
classifiers to pyproject.toml.
Replace typing.List/Dict/Type/Optional with built-in generics and
X | None in the core modules, utils, and tests, and drop typing imports
left unused. The generated marketplace modules are handled in a
separate commit.

Also add python_requires=">=3.11" to setup.py and 3.12-3.14
classifiers to pyproject.toml.
…odules

Replace typing.List/Dict/Optional/Union with built-in generics and
X | Y unions across the 294 generated marketplace integration modules,
and drop the typing imports left unused (only Annotated remains).

The change is mechanical and reproducible with
`ruff check --target-version py311 --select UP006,UP007,UP035,UP045 --fix`
plus removing the unused names from each `from typing import` line.
These modules are generated outside this repository, so the generator
should emit the same syntax or a regeneration will bring the legacy
aliases back.
Replace typing.List/Dict/Optional with built-in generics and X | None
across the agent, CLI, server routes, config, and sample servers and
agents, and drop typing imports left unused. Annotate the
AgentSettings model validator with typing.Self instead of a string
forward reference.

Raise the typer floor to >=0.12.4: earlier typer releases fail with
"RuntimeError: Type not yet supported: str | None" on the CLI's
X | None parameters.
… ADK import placement

Annotate the HttpClient get/post/patch params and req arguments as dict[str, Any] | None instead of relying on an implicit Optional default, and move the collections.abc import added in the ADK commit next to the other collections import. No runtime or schema change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Modernize codebase to Python >=3.11 syntax and typing (PEP 585, PEP 604)

1 participant