diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..6c540cd --- /dev/null +++ b/.gitignore @@ -0,0 +1,44 @@ +# Python +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# Virtual environments +venv/ +env/ +ENV/ +.venv + +# IDEs +.vscode/ +.idea/ +*.swp +*.swo +*~ + +# OS +.DS_Store +Thumbs.db + +# Test results +results/ +output/ +*.log \ No newline at end of file diff --git a/README.md b/README.md index c12bb06..b442e81 100644 --- a/README.md +++ b/README.md @@ -1,281 +1,257 @@ -# Python Packaging with Deephaven +# Deephaven Python packaging examples -This example demonstrates how to create and deploy Python packages that use Deephaven. It shows you how to package both command-line tools and reusable libraries using modern Python packaging standards. +This repository shows how to package Python code that uses [Deephaven Community Core](https://deephaven.io/community/) so that it can be installed with `pip`. It contains three small, self-contained example packages. Each example demonstrates exactly one packaging pattern: -This example accompanies the [Packaging custom code and dependencies](https://deephaven.io/core/docs/how-to-guides/sysadmin/setuptools-deployment/) guide in the Deephaven documentation. +| Example | Pattern | Installing it provides | +|---|---|---| +| [`my_dh_library/`](my_dh_library/) | Library only | Functions to import in Python code | +| [`my_dh_cli/`](my_dh_cli/) | Command line tool only | A `my-dh-query` terminal command | +| [`my_dh_toolkit/`](my_dh_toolkit/) | Library and command line tools combined | Importable functions plus `my-dh-toolkit-query` and `my-dh-toolkit-process` commands | -## What you'll learn +`my_dh_toolkit` is the other two patterns merged into a single package: its library modules play the same role as `my_dh_library`, and its commands play the same role as `my_dh_cli`. Its commands also call its own library functions, so the same code is reachable from Python and from the terminal. -This example shows you how to: +All three examples follow the [Python Packaging User Guide](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) conventions: a `pyproject.toml` file for metadata, dependencies, and entry points, and the src-layout for source code. This repository accompanies the [Packaging custom code and dependencies](https://deephaven.io/core/docs/how-to-guides/sysadmin/setuptools-deployment/) guide, which explains the underlying concepts in depth. -- Create installable Python packages with Deephaven dependencies -- Build command-line tools that process data with Deephaven -- Package reusable library code for other projects -- Manage dependencies with `pyproject.toml` -- Distribute packages as wheel archives +## Choose an example -## Project structure +- Start from **`my_dh_library`** to share reusable functions that other projects import. There is no command line interface. +- Start from **`my_dh_cli`** to ship a tool that users run from a terminal. No library code is exposed. +- Start from **`my_dh_toolkit`** to provide both: importable functions for Python users and commands for terminal users. -The example includes three complete packaging scenarios: +## Prerequisites -### 1. Library-only package (`my_dh_library/`) +- Python 3.9 or later. +- pip. +- Java 17 or later (required by `deephaven-server`, which each example installs as a dependency). +- The examples declare `deephaven-server` 0.35.0 or later as their dependency. For the latest Deephaven version, see [deephaven.io](https://deephaven.io/). -A reusable library with Deephaven query functions that other projects can import. +## Get the examples -``` -my_dh_library/ -├── src/ -│ └── my_dh_library/ -│ ├── __init__.py -│ ├── queries.py -│ └── utils.py -├── pyproject.toml -└── README.md +Clone the repository and work from its root directory. All commands below are run from the repository root. + +```bash +git clone https://github.com/deephaven-examples/deephaven-python-packaging.git +cd deephaven-python-packaging ``` -### 2. CLI-only package (`my_dh_cli/`) +## Sample data -Command-line tools for processing data with Deephaven. +The examples read the CSV files in the `data/` directory: -``` -my_dh_cli/ -├── src/ -│ └── my_dh_package/ -│ ├── __init__.py -│ ├── __main__.py -│ ├── cli.py -│ └── processor.py -├── pyproject.toml -├── data/ -│ └── sample.csv -└── README.md -``` +- `data/sample.csv`: a single 10-row file with `Name`, `Score`, `Value`, and `Category` columns. It is the input for the single-file examples: the library snippets, `my-dh-query`, and `my-dh-toolkit-query`. +- `data/batch/`: three smaller files (`file1.csv`, `file2.csv`, and `file3.csv`) with the same columns but different rows. It is the input for `my-dh-toolkit-process`, which processes every CSV file in a directory. -### 3. Combined package (`my_dh_toolkit/`) +## Example 1: `my_dh_library` — a library -Both reusable library code and command-line tools in one package. +**The story:** package reusable Deephaven query functions so that other projects can `pip install` the package and import the functions. ``` -my_dh_toolkit/ +my_dh_library/ ├── src/ -│ └── my_dh_package/ -│ ├── __init__.py -│ ├── __main__.py -│ ├── cli.py -│ ├── queries.py -│ └── utils.py -├── pyproject.toml +│ └── my_dh_library/ +│ ├── __init__.py # Exports the public API +│ ├── queries.py # Query functions: filter, compute, summarize +│ └── utils.py # Table validation helpers +├── pyproject.toml # Declares metadata and the deephaven-server dependency └── README.md ``` -## Prerequisites - -- Python 3.8 or later -- pip (Python package installer) -- Basic familiarity with Python packaging - -## Quick start - -Clone the repository: - -```shell -git clone https://github.com/deephaven-examples/python-packaging.git -cd python-packaging -``` - -Choose an example to try: - -### Try the CLI package +There is no `[project.scripts]` section in `pyproject.toml` and no `__main__.py` — this package is only ever imported. -```shell -cd my_dh_cli -pip install -e . -my-dh-query data/sample.csv --verbose -my-dh-process data/ --output results/ -``` +### Try it -### Try the library package +Install the package and start Python: -```shell -cd my_dh_library -pip install -e . +```bash +pip install -e ./my_dh_library python ``` -Then in Python: +A library that uses Deephaven needs a running server in the same process, so start one before importing `deephaven` modules: ```python +# A Deephaven server must be running before deephaven modules are imported. +from deephaven_server import Server +Server(port=10000, jvm_args=["-Xmx4g"]).start() + +# Import and use the installed library. from my_dh_library.queries import filter_by_threshold from deephaven import read_csv -data = read_csv("../data/sample.csv") +data = read_csv("data/sample.csv") filtered = filter_by_threshold(data, "Score", 75.0) -print(f"Filtered to {filtered.size} rows") +print(f"{filtered.size} of {data.size} rows have Score > 75") ``` -### Try the combined package +### What to study -```shell -cd my_dh_toolkit -pip install -e . +- [`pyproject.toml`](my_dh_library/pyproject.toml): the `dependencies` list installs `deephaven-server` automatically, and `[tool.setuptools.packages.find]` points setuptools at `src/`. +- [`queries.py`](my_dh_library/src/my_dh_library/queries.py): plain functions that take and return Deephaven tables. +- [`__init__.py`](my_dh_library/src/my_dh_library/__init__.py): re-exports the public functions. -# Use as a library -python -c "from my_dh_toolkit.queries import filter_by_threshold; print('Library imported successfully')" +## Example 2: `my_dh_cli` — a command line tool -# Use as CLI tools -my-dh-query ../data/sample.csv -my-dh-process ../data/ --output results/ +**The story:** package a Deephaven script as a terminal command. `pip install` creates a `my-dh-query` command that users run without writing any Python. + +``` +my_dh_cli/ +├── src/ +│ └── my_dh_cli/ +│ ├── __init__.py +│ ├── __main__.py # Enables `python -m my_dh_cli` during development +│ └── cli.py # The command implementation +├── pyproject.toml # Declares the my-dh-query entry point +└── README.md ``` -## What's included +The command comes from one line in `pyproject.toml`: -### Command-line tools +```toml +[project.scripts] +my-dh-query = "my_dh_cli.cli:app" +``` -The CLI examples demonstrate: +### Try it -- **Entry point scripts** - Commands installed to your PATH -- **Module execution** - Running with `python -m package_name` -- **Argument parsing** - Using Click for robust CLI interfaces -- **Multiple commands** - Single package with multiple tools -- **Verbose output** - Optional detailed logging +Install the package, then run the command on the sample data: -### Library modules +```bash +pip install -e ./my_dh_cli +my-dh-query data/sample.csv --verbose +``` -The library examples show: +The command starts its own Deephaven server, reads the CSV file, adds a computed `DoubleScore` column, and reports the row count. No separate setup is needed. -- **Reusable query functions** - Common Deephaven operations -- **Type hints** - Proper function signatures -- **Public API exports** - Clean import patterns -- **Documentation** - Docstrings for all functions +### What to study -### Configuration +- [`pyproject.toml`](my_dh_cli/pyproject.toml): the `[project.scripts]` section maps the command name to a function. +- [`cli.py`](my_dh_cli/src/my_dh_cli/cli.py): a [Click](https://click.palletsprojects.com/) command that starts the Deephaven server itself, so it works as a standalone tool. +- [`__main__.py`](my_dh_cli/src/my_dh_cli/__main__.py): allows `python -m my_dh_cli data/sample.csv` as an alternative during development. -All examples include: +## Example 3: `my_dh_toolkit` — a library and command line tools in one package -- **`pyproject.toml`** - Modern Python packaging configuration -- **Dependency management** - Automatic installation of Deephaven and other requirements -- **Version constraints** - Ensuring compatible package versions -- **Entry points** - Mapping command names to Python functions +**The story:** one package that provides both interfaces. Python users import its query functions, just as in `my_dh_library`; terminal users run its installed commands, just as in `my_dh_cli`. The commands call the package's own library functions, so there is one implementation behind both interfaces. + +``` +my_dh_toolkit/ +├── src/ +│ └── my_dh_toolkit/ +│ ├── __init__.py # Intentionally contains no imports (see "What to study") +│ ├── __main__.py # Enables `python -m my_dh_toolkit` (runs the query command) +│ ├── cli.py # Implements my-dh-toolkit-query (same pattern as my_dh_cli) +│ ├── processor.py # Implements my-dh-toolkit-process +│ ├── queries.py # Library query functions (same code as my_dh_library) +│ └── utils.py # Library table helpers (same code as my_dh_library) +├── pyproject.toml # Declares both commands +└── README.md +``` -## Building and distributing +### Try the commands -Each example can be built into a distributable wheel: +Install the package, then run each command: -```shell -cd my_dh_cli # or any example directory -pip install build -python -m build +```bash +pip install -e ./my_dh_toolkit +my-dh-toolkit-query data/sample.csv --verbose +my-dh-toolkit-process data/batch --output output --verbose ``` -This creates a `.whl` file in the `dist/` directory that can be: +`my-dh-toolkit-query` processes one CSV file. `my-dh-toolkit-process` processes every CSV file in a directory and writes one result file per input to the output directory. Both commands validate that the input has a `Value` column and add `DoubleValue` and `IsHigh` columns by calling the library's `validate_columns` and `add_computed_columns`. Like `my-dh-query` in the previous example, each command starts its own Deephaven server. -- Installed locally: `pip install dist/my_dh_cli-0.1.0-py3-none-any.whl` -- Distributed to others -- Published to PyPI: `python -m twine upload dist/*` +### Try the library -## Running the examples +The same installation also provides the library. In a Python session, start a Deephaven server, then import and use the query functions: -### Development mode +```python +# A Deephaven server must be running before deephaven modules are imported. +from deephaven_server import Server +Server(port=10000, jvm_args=["-Xmx4g"]).start() -Install in editable mode to make changes without reinstalling: +# Import and use the installed library. +from my_dh_toolkit.queries import filter_by_threshold +from deephaven import read_csv -```shell -pip install -e . +data = read_csv("data/sample.csv") +filtered = filter_by_threshold(data, "Score", 75.0) +print(f"{filtered.size} of {data.size} rows have Score > 75") ``` -### Regular installation +### What to study -Install from the built wheel: +- [`pyproject.toml`](my_dh_toolkit/pyproject.toml): a single `[project.scripts]` section defines both commands. +- [`__init__.py`](my_dh_toolkit/src/my_dh_toolkit/__init__.py): contains no imports, and that is deliberate. Importing any `deephaven` module fails unless a Deephaven server is already running in the process. When a command such as `my-dh-toolkit-query` starts, Python imports the `my_dh_toolkit` package before the command has started its server. If `__init__.py` imported the query functions, that import chain would reach `deephaven` and every command would fail at startup. Keeping `__init__.py` empty and importing the library from its submodules (`my_dh_toolkit.queries`, `my_dh_toolkit.utils`) avoids the problem. `my_dh_library` can safely re-export its functions from `__init__.py` because it has no commands: it is only ever imported after a server is running. +- [`cli.py`](my_dh_toolkit/src/my_dh_toolkit/cli.py) and [`processor.py`](my_dh_toolkit/src/my_dh_toolkit/processor.py): the commands import `my_dh_toolkit.queries` and `my_dh_toolkit.utils` *inside* the function that runs after the server has started, for the same reason. That is how a command module can reuse library code that depends on `deephaven`. -```shell -pip install dist/package_name-0.1.0-py3-none-any.whl -``` +## Adapt an example for your own project -### Without installation +Each example is a template. To turn one into your own package: -Run directly from source using module execution: +1. **Copy the example** that matches your scenario: -```shell -python -m my_dh_package input_data.csv -``` + ```bash + cp -r my_dh_cli my_tool + cd my_tool + ``` -## Sample data +2. **Rename the import package.** The directory under `src/` is the name used in `import` statements: -The `data/` directory contains sample CSV files for testing: + ```bash + mv src/my_dh_cli src/my_tool + ``` -- `sample.csv` - Small dataset with Name, Age, and Score columns -- `batch/` - Multiple CSV files for batch processing examples +3. **Update `pyproject.toml`.** Set your own `name`, `version`, and `description`, and point any `[project.scripts]` entries at the new package: -You can use your own CSV files with these examples. + ```toml + [project] + name = "my_tool" -## Key concepts + [project.scripts] + my-tool = "my_tool.cli:app" + ``` -### Entry point scripts vs module execution +4. **Update internal imports** to the new package name (for example, `from my_tool.cli import app` in `__main__.py`). -The examples demonstrate two ways to run Python packages: +5. **Replace the example logic** with your own code, and add any packages it needs to `dependencies` in `pyproject.toml`. Keep `deephaven-server` in the list so it installs automatically. -1. **Entry point scripts** - Commands defined in `[project.scripts]` that become available after installation - ```shell - my-dh-query data.csv - ``` +6. **Reinstall and test:** -2. **Module execution** - Running packages with `python -m` without installation - ```shell - python -m my_dh_package data.csv + ```bash + pip install -e . + my-tool --help ``` -See the [Execution patterns](https://deephaven.io/core/docs/how-to-guides/sysadmin/setuptools-deployment/#execution-patterns) section of the guide for when to use each method. +Three names must stay in sync: the package directory under `src/`, the module paths in `[project.scripts]`, and the package name in `import` statements. + +## Install and distribute -### Package structure +The examples above use editable installs (`pip install -e ./my_dh_cli`), which pick up source edits without reinstalling. This is ideal while developing. The other common options: -All examples use the **src-layout**, which is the recommended structure for Python packages. This keeps source code separate from tests and configuration files. +- **Regular install from source:** `pip install ./my_dh_cli` +- **Build and install a wheel** — the format to use when distributing a package to other machines or publishing to a package index: -### Dependencies + ```bash + pip install build + python -m build my_dh_cli + pip install my_dh_cli/dist/my_dh_cli-0.1.0-py3-none-any.whl + ``` -The examples show how to: + Wheels can be shared directly or published to PyPI with [`twine`](https://twine.readthedocs.io/). -- Specify required packages (like `deephaven-server`) -- Set version constraints -- Define optional dependencies for features like visualization or testing +## Troubleshooting + +- **Command not found after installation:** confirm the install succeeded (`pip show my_dh_cli`) and that the Python scripts directory is on `PATH`. Installing inside an activated virtual environment avoids most `PATH` issues. +- **`Address already in use` when a command or snippet starts:** the examples bind the Deephaven server to port 10000. If another process already uses that port (for example, a Deephaven server running in Docker), change the `port` value in the `Server(...)` call to a free port. +- **`deephaven` import errors:** the Deephaven server must be started (as shown in the library examples) before `deephaven` modules are imported, and Java 17 or later must be available. +- **Module not found after renaming:** check that the directory under `src/`, the `[project.scripts]` module paths, and the `import` statements all use the new package name, then reinstall with `pip install -e .`. ## Related documentation -- [Packaging custom code and dependencies](https://deephaven.io/core/docs/how-to-guides/sysadmin/setuptools-deployment/) - Complete guide +- [Packaging custom code and dependencies](https://deephaven.io/core/docs/how-to-guides/sysadmin/setuptools-deployment/), the guide this repository accompanies. - [Install and use Python packages](https://deephaven.io/core/docs/how-to-guides/install-and-use-python-packages/) - [Use the Deephaven Python package](https://deephaven.io/core/docs/how-to-guides/deephaven-python-package/) - [Python Packaging User Guide](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) - [Click documentation](https://click.palletsprojects.com/) -## Troubleshooting - -### Command not found after installation - -If your command isn't found after installation: - -- Ensure the installation completed without errors -- Check that the installation directory is in your PATH -- Try reinstalling: `pip install --force-reinstall .` - -### Import errors - -If you encounter import errors: - -- Verify all dependencies are installed: `pip list` -- Check that you're using Python 3.8 or later -- Ensure Deephaven is installed: `pip install deephaven-server` - -### Module not found errors - -If Python can't find your modules: - -- Verify `__init__.py` files exist in all package directories -- Check that package names in `[project.scripts]` match your directory structure -- Try reinstalling in editable mode: `pip install -e .` - -## Note - -The code in this repository is built for Deephaven Community Core v0.35.0 or later. For the latest Deephaven version, see [deephaven.io](https://deephaven.io/). - ## Contributing Have improvements or additional examples? Contributions are welcome! Please open an issue or pull request on GitHub. diff --git a/data/batch/file1.csv b/data/batch/file1.csv new file mode 100644 index 0000000..954231a --- /dev/null +++ b/data/batch/file1.csv @@ -0,0 +1,4 @@ +Name,Score,Value,Category +Kara,81,130,A +Liam,94,175,B +Mona,77,100,A diff --git a/data/batch/file2.csv b/data/batch/file2.csv new file mode 100644 index 0000000..6975da9 --- /dev/null +++ b/data/batch/file2.csv @@ -0,0 +1,4 @@ +Name,Score,Value,Category +Nina,86,115,C +Omar,90,155,B +Pria,74,80,A diff --git a/data/batch/file3.csv b/data/batch/file3.csv new file mode 100644 index 0000000..733a416 --- /dev/null +++ b/data/batch/file3.csv @@ -0,0 +1,5 @@ +Name,Score,Value,Category +Quinn,93,165,C +Rosa,84,125,B +Sam,79,95,A +Tara,88,145,C diff --git a/data/sample.csv b/data/sample.csv new file mode 100644 index 0000000..1d1b856 --- /dev/null +++ b/data/sample.csv @@ -0,0 +1,11 @@ +Name,Score,Value,Category +Alice,85,120,A +Bob,92,150,B +Charlie,78,95,A +Diana,88,110,C +Eve,95,180,B +Frank,72,85,A +Grace,91,160,C +Henry,83,105,B +Iris,89,140,A +Jack,76,90,C diff --git a/my_dh_cli/README.md b/my_dh_cli/README.md new file mode 100644 index 0000000..ee1b8e4 --- /dev/null +++ b/my_dh_cli/README.md @@ -0,0 +1,63 @@ +# My Deephaven CLI + +An example of packaging a Deephaven script as a command line tool. Installing this package creates one terminal command, `my-dh-query`. No library code is exposed; the package is used only through that command, so no Python needs to be written to use it. + +The command is defined by the `[project.scripts]` entry point in [`pyproject.toml`](pyproject.toml): + +```toml +[project.scripts] +my-dh-query = "my_dh_cli.cli:app" +``` + +## Installation + +From the repository root: + +```bash +pip install ./my_dh_cli +``` + +Or in editable mode for development: + +```bash +pip install -e ./my_dh_cli +``` + +## Usage + +Run the installed command on a CSV file. The command starts its own Deephaven server, so no separate setup is needed: + +```bash +my-dh-query data/sample.csv --verbose +``` + +It reads the file, adds a `DoubleScore` computed column, and reports the number of rows processed. + +The command binds its server to port 10000. If that port is already in use (for example, by Deephaven running in Docker), change the `port` value in [`cli.py`](src/my_dh_cli/cli.py). + +During development, the package also runs without an entry point via [`__main__.py`](src/my_dh_cli/__main__.py): + +```bash +python -m my_dh_cli data/sample.csv --verbose +``` + +## Command reference + +### my-dh-query + +Process a CSV file with Deephaven. The file must contain a `Score` column. + +**Arguments:** + +- `input_file` - Path to the CSV file to process. + +**Options:** + +- `--verbose, -v` - Enable verbose output. + +## Requirements + +- Python 3.9 or later +- Java 17 or later +- deephaven-server 0.35.0 or later (installed automatically as a dependency) +- Click 8.0.0 or later (installed automatically as a dependency) diff --git a/my_dh_cli/pyproject.toml b/my_dh_cli/pyproject.toml new file mode 100644 index 0000000..7890660 --- /dev/null +++ b/my_dh_cli/pyproject.toml @@ -0,0 +1,22 @@ +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_cli" +version = "0.1.0" +description = "Command line tool for data processing" +readme = "README.md" +requires-python = ">=3.9" +dependencies = [ + # deephaven-server also provides the deephaven module (through its deephaven-core dependency). + "deephaven-server>=0.35.0", + # click implements the command line interface. + "click>=8.0.0", +] + +[project.scripts] +my-dh-query = "my_dh_cli.cli:app" + +[tool.setuptools.packages.find] +where = ["src"] diff --git a/my_dh_cli/src/my_dh_cli/__init__.py b/my_dh_cli/src/my_dh_cli/__init__.py new file mode 100644 index 0000000..c1aefdf --- /dev/null +++ b/my_dh_cli/src/my_dh_cli/__init__.py @@ -0,0 +1,3 @@ +"""Command line tool that processes a CSV file with Deephaven.""" + +__version__ = "0.1.0" diff --git a/my_dh_cli/src/my_dh_cli/__main__.py b/my_dh_cli/src/my_dh_cli/__main__.py new file mode 100644 index 0000000..ff7364e --- /dev/null +++ b/my_dh_cli/src/my_dh_cli/__main__.py @@ -0,0 +1,4 @@ +from my_dh_cli.cli import app + +if __name__ == "__main__": + app() diff --git a/my_dh_cli/src/my_dh_cli/cli.py b/my_dh_cli/src/my_dh_cli/cli.py new file mode 100644 index 0000000..5c6fa6a --- /dev/null +++ b/my_dh_cli/src/my_dh_cli/cli.py @@ -0,0 +1,53 @@ +import click + + +def my_dh_query(input_file: str, verbose: bool = False): + """Read a CSV file and perform a simple query operation on the data.""" + from deephaven import read_csv + from pathlib import Path + + input_path = Path(input_file) + + if not input_path.exists(): + raise click.ClickException(f"Input file does not exist: '{input_path}'") + if not input_path.is_file(): + raise click.ClickException(f"Input path is not a file: '{input_path}'") + + if verbose: + click.echo(f"Processing {input_file}...") + + try: + source = read_csv(input_file) + except Exception as e: + raise click.ClickException(f"Failed to read CSV file '{input_file}': {e}") + + column_names = [col.name for col in source.columns] + if "Score" not in column_names: + raise click.ClickException( + f"File '{input_path.name}' is missing required column 'Score'. " + f"Available columns: {', '.join(column_names)}" + ) + + result = source.update(formulas=["DoubleScore = Score * 2"]) + + if verbose: + click.echo(f"Processed {result.size} rows") + + return result + + +@click.command() +@click.argument("input_file", type=click.Path(exists=True)) +@click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") +def app(input_file: str, verbose: bool) -> None: + """Process data with Deephaven.""" + from deephaven_server import Server + + Server(port=10000, jvm_args=["-Xmx4g"]).start() + + my_dh_query(input_file, verbose) + click.echo("Processing complete!") + + +if __name__ == "__main__": + app() diff --git a/my_dh_library/README.md b/my_dh_library/README.md new file mode 100644 index 0000000..56fdd27 --- /dev/null +++ b/my_dh_library/README.md @@ -0,0 +1,57 @@ +# My Deephaven Library + +An example of packaging reusable Deephaven query functions as a library. Installing this package makes its functions importable from any Python code. There are no command line tools; this package is only ever imported. + +## Installation + +From the repository root: + +```bash +pip install ./my_dh_library +``` + +Or in editable mode for development: + +```bash +pip install -e ./my_dh_library +``` + +## Usage + +> [!NOTE] +> All Deephaven functionality requires a running server in the same Python process. Start the server before importing `deephaven` modules. The snippet below binds the server to port 10000; if that port is already in use (for example, by Deephaven running in Docker), change the `port` value. + +From the repository root, start Python and use the library: + +```python +# A Deephaven server must be running before deephaven modules are imported. +from deephaven_server import Server +Server(port=10000, jvm_args=["-Xmx4g"]).start() + +# Import and use the installed library. +from my_dh_library.queries import filter_by_threshold, add_computed_columns +from deephaven import read_csv + +data = read_csv("data/sample.csv") +filtered = filter_by_threshold(data, "Score", 75.0) +enhanced = add_computed_columns(filtered) +``` + +## Available functions + +### Query functions (`my_dh_library.queries`) + +- `filter_by_threshold(table, column, threshold)` - Filter table rows where the column value exceeds the threshold. +- `add_computed_columns(table)` - Add commonly used computed columns to a table. +- `summarize_by_group(table, group_col, value_col)` - Create summary statistics grouped by a column. + +### Utility functions (`my_dh_library.utils`) + +- `validate_columns(table, required_columns)` - Check if a table has all required columns. +- `get_table_info(table)` - Get basic information about a table. + +## Requirements + +- Python 3.9 or later +- Java 17 or later +- deephaven-server 0.35.0 or later (installed automatically as a dependency) diff --git a/my_dh_library/pyproject.toml b/my_dh_library/pyproject.toml new file mode 100644 index 0000000..0e1f738 --- /dev/null +++ b/my_dh_library/pyproject.toml @@ -0,0 +1,17 @@ +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_library" +version = "0.1.0" +description = "Reusable Deephaven query functions" +readme = "README.md" +requires-python = ">=3.9" +dependencies = [ + # deephaven-server also provides the deephaven module (through its deephaven-core dependency). + "deephaven-server>=0.35.0", +] + +[tool.setuptools.packages.find] +where = ["src"] diff --git a/my_dh_library/src/my_dh_library/__init__.py b/my_dh_library/src/my_dh_library/__init__.py new file mode 100644 index 0000000..ada9859 --- /dev/null +++ b/my_dh_library/src/my_dh_library/__init__.py @@ -0,0 +1,7 @@ +"""Reusable Deephaven query functions and table utilities.""" + +__version__ = "0.1.0" + +from my_dh_library.queries import filter_by_threshold, add_computed_columns, summarize_by_group + +__all__ = ["filter_by_threshold", "add_computed_columns", "summarize_by_group"] diff --git a/my_dh_library/src/my_dh_library/queries.py b/my_dh_library/src/my_dh_library/queries.py new file mode 100644 index 0000000..eb0adfc --- /dev/null +++ b/my_dh_library/src/my_dh_library/queries.py @@ -0,0 +1,35 @@ +"""Reusable Deephaven query functions.""" + +from deephaven.table import Table +from deephaven import agg +from .utils import validate_columns + + +def filter_by_threshold(table: Table, column: str, threshold: float) -> Table: + """Filter table rows where column value exceeds threshold.""" + validate_columns(table, [column], raise_error=True) + return table.where(f"{column} > {threshold}") + + +def add_computed_columns(table: Table) -> Table: + """Add commonly used computed columns to a table.""" + validate_columns(table, ["Value"], raise_error=True) + return table.update( + [ + "DoubleValue = Value * 2", + "IsHigh = Value > 100", + ] + ) + + +def summarize_by_group(table: Table, group_col: str, value_col: str) -> Table: + """Create summary statistics grouped by a column.""" + validate_columns(table, [group_col, value_col], raise_error=True) + return table.agg_by( + [ + agg.sum_(f"Sum = {value_col}"), + agg.avg(f"Avg = {value_col}"), + agg.count_("Count"), + ], + by=[group_col], + ) diff --git a/my_dh_library/src/my_dh_library/utils.py b/my_dh_library/src/my_dh_library/utils.py new file mode 100644 index 0000000..c4d1abb --- /dev/null +++ b/my_dh_library/src/my_dh_library/utils.py @@ -0,0 +1,41 @@ +"""Utility functions for working with Deephaven tables.""" + +from __future__ import annotations + +from deephaven.table import Table + + +def validate_columns(table: Table, required_columns: list[str], raise_error: bool = False) -> bool: + """Check if table has all required columns. + + Args: + table: The table to validate + required_columns: List of column names that must be present + raise_error: If True, raises ValueError when columns are missing + + Returns: + True if all columns are present, False otherwise + + Raises: + ValueError: If raise_error is True and columns are missing + """ + table_columns = [col.name for col in table.columns] + missing = [col for col in required_columns if col not in table_columns] + + if missing: + if raise_error: + raise ValueError( + f"Column(s) {missing} not found in table. " + f"Available columns: {', '.join(table_columns)}" + ) + return False + return True + + +def get_table_info(table: Table) -> dict: + """Get basic information about a table.""" + return { + "num_rows": table.size, + "num_columns": len(table.columns), + "columns": [col.name for col in table.columns], + } diff --git a/my_dh_toolkit/README.md b/my_dh_toolkit/README.md new file mode 100644 index 0000000..1cb82a1 --- /dev/null +++ b/my_dh_toolkit/README.md @@ -0,0 +1,119 @@ +# My Deephaven Toolkit + +An example of one package with two interfaces: + +- **A library**: importable query functions, matching the [`my_dh_library`](../my_dh_library/) example. +- **Command line tools**: two terminal commands, following the same pattern as the [`my_dh_cli`](../my_dh_cli/) example. + +The commands call the package's own library functions (`validate_columns` and `add_computed_columns`), so Python users and terminal users share one implementation. + +The commands are defined by the `[project.scripts]` entry points in [`pyproject.toml`](pyproject.toml): + +```toml +[project.scripts] +my-dh-toolkit-query = "my_dh_toolkit.cli:app" +my-dh-toolkit-process = "my_dh_toolkit.processor:process" +``` + +## Installation + +From the repository root: + +```bash +pip install ./my_dh_toolkit +``` + +Or in editable mode for development: + +```bash +pip install -e ./my_dh_toolkit +``` + +## Usage as command line tools + +Run the installed commands on the sample data. Each command starts its own Deephaven server, so no separate setup is needed: + +```bash +my-dh-toolkit-query data/sample.csv --verbose +my-dh-toolkit-process data/batch --output output --verbose +``` + +`my-dh-toolkit-query` processes a single CSV file. `my-dh-toolkit-process` processes every CSV file in a directory and writes the results to the output directory. Both commands require a `Value` column and add `DoubleValue` and `IsHigh` columns. + +The commands bind their server to port 10000. If that port is already in use (for example, by Deephaven running in Docker), change the `port` value in [`cli.py`](src/my_dh_toolkit/cli.py) and [`processor.py`](src/my_dh_toolkit/processor.py). + +During development, the query command also runs via [`__main__.py`](src/my_dh_toolkit/__main__.py): + +```bash +python -m my_dh_toolkit data/sample.csv --verbose +``` + +## Usage as a library + +> [!NOTE] +> All Deephaven functionality requires a running server in the same Python process. Start the server before importing `deephaven` modules. + +From the repository root, start Python and use the library: + +```python +# A Deephaven server must be running before deephaven modules are imported. +from deephaven_server import Server +Server(port=10000, jvm_args=["-Xmx4g"]).start() + +# Import and use the installed library. +from my_dh_toolkit.queries import filter_by_threshold, add_computed_columns +from deephaven import read_csv + +data = read_csv("data/sample.csv") +filtered = filter_by_threshold(data, "Score", 75.0) +enhanced = add_computed_columns(filtered) +``` + +Import the library from its submodules (`my_dh_toolkit.queries`, `my_dh_toolkit.utils`), not from `my_dh_toolkit` itself. The package's [`__init__.py`](src/my_dh_toolkit/__init__.py) deliberately imports nothing that requires Deephaven, so that the commands can be started before a server is running. + +## Command reference + +### my-dh-toolkit-query + +Process a single CSV file with Deephaven. The file must contain a `Value` column. The result has two additional columns, `DoubleValue` and `IsHigh`. + +**Arguments:** + +- `input_file` - Path to the CSV file to process. + +**Options:** + +- `--verbose, -v` - Enable verbose output. + +### my-dh-toolkit-process + +Batch process every CSV file in a directory. Each file must contain a `Value` column. Each output file, named `processed_`, has two additional columns, `DoubleValue` and `IsHigh`. + +**Arguments:** + +- `directory` - Directory containing CSV files to process. + +**Options:** + +- `--output, -o` - Output directory (default: `./output`). Must differ from the input directory. +- `--verbose, -v` - Enable verbose output. + +## Available functions + +### Query functions (`my_dh_toolkit.queries`) + +- `filter_by_threshold(table, column, threshold)` - Filter table rows where the column value exceeds the threshold. +- `add_computed_columns(table)` - Add `DoubleValue` and `IsHigh` columns computed from `Value`. +- `summarize_by_group(table, group_col, value_col)` - Create summary statistics grouped by a column. + +### Utility functions (`my_dh_toolkit.utils`) + +- `validate_columns(table, required_columns)` - Check if a table has all required columns. +- `get_table_info(table)` - Get basic information about a table. + +## Requirements + +- Python 3.9 or later +- Java 17 or later +- deephaven-server 0.35.0 or later (installed automatically as a dependency) +- Click 8.0.0 or later (installed automatically as a dependency) diff --git a/my_dh_toolkit/pyproject.toml b/my_dh_toolkit/pyproject.toml new file mode 100644 index 0000000..ebdd348 --- /dev/null +++ b/my_dh_toolkit/pyproject.toml @@ -0,0 +1,23 @@ +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_toolkit" +version = "0.1.0" +description = "Deephaven library and CLI tools" +readme = "README.md" +requires-python = ">=3.9" +dependencies = [ + # deephaven-server also provides the deephaven module (through its deephaven-core dependency). + "deephaven-server>=0.35.0", + # click implements the command line interfaces. + "click>=8.0.0", +] + +[project.scripts] +my-dh-toolkit-query = "my_dh_toolkit.cli:app" +my-dh-toolkit-process = "my_dh_toolkit.processor:process" + +[tool.setuptools.packages.find] +where = ["src"] diff --git a/my_dh_toolkit/src/my_dh_toolkit/__init__.py b/my_dh_toolkit/src/my_dh_toolkit/__init__.py new file mode 100644 index 0000000..20bf420 --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/__init__.py @@ -0,0 +1,9 @@ +"""Deephaven query library and command line tools. + +This __init__ deliberately imports nothing that requires Deephaven: the CLI +entry points import this package before a Deephaven server is running, so the +package must be importable without one. The library API lives in the +`my_dh_toolkit.queries` and `my_dh_toolkit.utils` submodules. +""" + +__version__ = "0.1.0" diff --git a/my_dh_toolkit/src/my_dh_toolkit/__main__.py b/my_dh_toolkit/src/my_dh_toolkit/__main__.py new file mode 100644 index 0000000..c7407a9 --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/__main__.py @@ -0,0 +1,4 @@ +from my_dh_toolkit.cli import app + +if __name__ == "__main__": + app() diff --git a/my_dh_toolkit/src/my_dh_toolkit/cli.py b/my_dh_toolkit/src/my_dh_toolkit/cli.py new file mode 100644 index 0000000..86709cb --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/cli.py @@ -0,0 +1,56 @@ +import click + + +def my_dh_query(input_file: str, verbose: bool = False): + """Read a CSV file and add computed columns using the package's library functions.""" + # Imported here, not at module level: these modules import deephaven, which + # requires a running server. The entry point starts the server first, then + # calls this function. + from deephaven import read_csv + from my_dh_toolkit.queries import add_computed_columns + from my_dh_toolkit.utils import validate_columns + from pathlib import Path + + input_path = Path(input_file) + + if not input_path.exists(): + raise click.ClickException(f"Input file does not exist: '{input_path}'") + if not input_path.is_file(): + raise click.ClickException(f"Input path is not a file: '{input_path}'") + + if verbose: + click.echo(f"Processing {input_file}...") + + try: + source = read_csv(input_file) + except Exception as e: + raise click.ClickException(f"Failed to read CSV file '{input_file}': {e}") + + try: + validate_columns(source, ["Value"], raise_error=True) + except ValueError as e: + raise click.ClickException(f"File '{input_path.name}': {e}") + + result = add_computed_columns(source) + + if verbose: + click.echo(f"Processed {result.size} rows") + + return result + + +@click.command() +@click.argument("input_file", type=click.Path(exists=True)) +@click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") +def app(input_file: str, verbose: bool) -> None: + """Process data with Deephaven.""" + from deephaven_server import Server + + Server(port=10000, jvm_args=["-Xmx4g"]).start() + + my_dh_query(input_file, verbose) + click.echo("Processing complete!") + + +if __name__ == "__main__": + app() diff --git a/my_dh_toolkit/src/my_dh_toolkit/processor.py b/my_dh_toolkit/src/my_dh_toolkit/processor.py new file mode 100644 index 0000000..8f97b9c --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/processor.py @@ -0,0 +1,79 @@ +import click +from pathlib import Path + + +def batch_process(directory: str, output_dir: str, verbose: bool = False) -> None: + """Process multiple CSV files from a directory.""" + input_path = Path(directory) + output_path = Path(output_dir) + + if not input_path.exists(): + raise click.ClickException(f"Input directory does not exist: '{input_path}'") + if not input_path.is_dir(): + raise click.ClickException(f"Input path is not a directory: '{input_path}'") + if input_path.resolve() == output_path.resolve(): + raise click.ClickException( + f"Input and output directories must be different: '{input_path}'" + ) + + try: + output_path.mkdir(parents=True, exist_ok=True) + except PermissionError: + raise click.ClickException(f"Permission denied: Cannot create output directory '{output_path}'") + except OSError as e: + raise click.ClickException(f"Failed to create output directory '{output_path}': {e}") + + # Imported here, not at module level: these modules import deephaven, which + # requires a running server. The entry point starts the server first, then + # calls this function. + from deephaven import read_csv, write_csv + from my_dh_toolkit.queries import add_computed_columns + from my_dh_toolkit.utils import validate_columns + + csv_files = [path for path in input_path.glob("*.csv") if path.is_file()] + + if verbose: + click.echo(f"Found {len(csv_files)} CSV files to process") + + for csv_file in csv_files: + if verbose: + click.echo(f"Processing {csv_file.name}...") + + try: + table = read_csv(str(csv_file)) + except Exception as e: + raise click.ClickException(f"Failed to read CSV file '{csv_file}': {e}") + + try: + validate_columns(table, ["Value"], raise_error=True) + except ValueError as e: + raise click.ClickException(f"File '{csv_file.name}': {e}") + + processed = add_computed_columns(table) + + output_file = output_path / f"processed_{csv_file.name}" + try: + write_csv(processed, str(output_file)) + except Exception as e: + raise click.ClickException(f"Failed to write output file '{output_file}': {e}") + + if verbose: + click.echo(f" Processed {processed.size} rows -> {output_file.name}") + + +@click.command() +@click.argument("directory", type=click.Path(exists=True, file_okay=False)) +@click.option("--output", "-o", default="./output", help="Output directory") +@click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") +def process(directory: str, output: str, verbose: bool) -> None: + """Batch process CSV files with Deephaven.""" + from deephaven_server import Server + + Server(port=10000, jvm_args=["-Xmx4g"]).start() + + batch_process(directory, output, verbose) + click.echo("Batch processing complete!") + + +if __name__ == "__main__": + process() diff --git a/my_dh_toolkit/src/my_dh_toolkit/queries.py b/my_dh_toolkit/src/my_dh_toolkit/queries.py new file mode 100644 index 0000000..eb0adfc --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/queries.py @@ -0,0 +1,35 @@ +"""Reusable Deephaven query functions.""" + +from deephaven.table import Table +from deephaven import agg +from .utils import validate_columns + + +def filter_by_threshold(table: Table, column: str, threshold: float) -> Table: + """Filter table rows where column value exceeds threshold.""" + validate_columns(table, [column], raise_error=True) + return table.where(f"{column} > {threshold}") + + +def add_computed_columns(table: Table) -> Table: + """Add commonly used computed columns to a table.""" + validate_columns(table, ["Value"], raise_error=True) + return table.update( + [ + "DoubleValue = Value * 2", + "IsHigh = Value > 100", + ] + ) + + +def summarize_by_group(table: Table, group_col: str, value_col: str) -> Table: + """Create summary statistics grouped by a column.""" + validate_columns(table, [group_col, value_col], raise_error=True) + return table.agg_by( + [ + agg.sum_(f"Sum = {value_col}"), + agg.avg(f"Avg = {value_col}"), + agg.count_("Count"), + ], + by=[group_col], + ) diff --git a/my_dh_toolkit/src/my_dh_toolkit/utils.py b/my_dh_toolkit/src/my_dh_toolkit/utils.py new file mode 100644 index 0000000..c4d1abb --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/utils.py @@ -0,0 +1,41 @@ +"""Utility functions for working with Deephaven tables.""" + +from __future__ import annotations + +from deephaven.table import Table + + +def validate_columns(table: Table, required_columns: list[str], raise_error: bool = False) -> bool: + """Check if table has all required columns. + + Args: + table: The table to validate + required_columns: List of column names that must be present + raise_error: If True, raises ValueError when columns are missing + + Returns: + True if all columns are present, False otherwise + + Raises: + ValueError: If raise_error is True and columns are missing + """ + table_columns = [col.name for col in table.columns] + missing = [col for col in required_columns if col not in table_columns] + + if missing: + if raise_error: + raise ValueError( + f"Column(s) {missing} not found in table. " + f"Available columns: {', '.join(table_columns)}" + ) + return False + return True + + +def get_table_info(table: Table) -> dict: + """Get basic information about a table.""" + return { + "num_rows": table.size, + "num_columns": len(table.columns), + "columns": [col.name for col in table.columns], + } diff --git a/setuptools-deployment.md b/setuptools-deployment.md new file mode 100644 index 0000000..f0c7ba3 --- /dev/null +++ b/setuptools-deployment.md @@ -0,0 +1,378 @@ +--- +title: Packaging custom code and dependencies +sidebar_label: Python packaging +--- + +[Python packaging](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) enables you to create distributable packages containing custom code, command line tools, and managed dependencies. Deephaven's own packages are pip-installable, so a package that depends on them can be built and installed with the standard Python tooling. This guide walks through the concepts and patterns for packaging Deephaven-based Python projects. + +Python packaging with [`pyproject.toml`](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) provides: + +- **Reusable libraries** - Package query functions and utilities for import by other projects. +- **Command line tools** - Build executable scripts with entry point definitions. +- **Dependency management** - Automatically install Deephaven and required packages, with explicit compatibility constraints. +- **Distribution** - Share code as wheel archives via PyPI or direct distribution. + +## Example repository + +The examples in this guide use the [deephaven-python-packaging](https://github.com/deephaven-examples/deephaven-python-packaging) repository. It demonstrates three complete packaging scenarios with working code, sample data, and a README for each package. + +To explore the examples, clone the repository: + +```bash +git clone https://github.com/deephaven-examples/deephaven-python-packaging.git +cd deephaven-python-packaging +``` + +The repository contains three example packages: + +- `my_dh_library/` - Library-only package with reusable query functions. +- `my_dh_cli/` - CLI-only package with a command line tool. +- `my_dh_toolkit/` - Combined package with both library and CLI functionality. + +## Package structure + +The example packages in this guide use the **src-layout** described in the Python Packaging Authority's [src layout vs flat layout discussion](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/). This layout keeps source code separate from tests and configuration files: + +``` +my_dh_library/ +├── src/ +│ └── my_dh_library/ +│ ├── __init__.py +│ ├── queries.py +│ └── utils.py +├── pyproject.toml +└── README.md +``` + +### Key components + +- **`src/`** - Source directory containing the package code. +- **`my_dh_library/`** (under `src/`) - The Python package. Its directory name is the name used in `import` statements. +- **`__init__.py`** - Makes the directory importable and can export a public API. +- **`pyproject.toml`** - Defines package metadata, dependencies, and entry points. +- **Module files** - Python files containing your functions and classes. + +The package name under `src/` determines how users import your code. For example, with `src/my_dh_library/`, users import via `from my_dh_library import ...`. + +## Server initialization + +Deephaven requires a running server before using any Deephaven functionality. The server must be initialized in the same Python process that uses Deephaven: + +```python +from deephaven_server import Server + +# Initialize and start the server +server = Server(port=10000, jvm_args=["-Xmx4g"]) +server.start() + +# Now you can import and use Deephaven +from deephaven import read_csv +data = read_csv("data/sample.csv") +``` + +### Key points + +- Each Python process has its own JVM. +- Starting a server in one terminal doesn't help another terminal. +- Entry-point CLI commands should start their own server internally (see [CLI-only package](#cli-only-package)) so they work standalone; only functions imported directly need an already-running session. +- The examples size the JVM to 4 GB with `jvm_args=["-Xmx4g"]`; adjust this value to fit the workload. + +> [!NOTE] +> The examples bind the server to port 10000. If another process already uses that port (for example, a Deephaven server running in Docker), the server fails to start with `Address already in use`. Change the `port` value to a free port. + +### Keep `__init__.py` free of Deephaven imports + +Because `deephaven` modules cannot be imported until a server is running, the order of imports matters in any package that defines a command. When a command such as `my-dh-toolkit-query` starts, Python imports the package (`my_dh_toolkit/__init__.py`) before the command function has a chance to start the server. If `__init__.py` imported a module that imports `deephaven`, every command in the package would fail at startup. + +The rule that follows: + +- In a package that defines commands, keep every module imported *before* the command starts its server free of imports that reach `deephaven`. In practice, that means `__init__.py` and the top level of command modules should stay import-light. +- Import `deephaven` and any Deephaven-dependent library submodules lazily, inside the function that runs after the server has started. +- A library-only package such as `my_dh_library` can safely re-export its functions from `__init__.py`. It has no commands, so it is only ever imported after a server is running. + +## Packaging scenarios + +Different projects have different needs. The example repository demonstrates three common scenarios. The Python usage snippets below assume a running Deephaven server, as shown in [Server initialization](#server-initialization). + +### Library-only package + +Package reusable code without CLI tools. Other projects import your modules. + +**Structure:** + +``` +my_dh_library/ +├── src/ +│ └── my_dh_library/ +│ ├── __init__.py +│ ├── queries.py +│ └── utils.py +├── pyproject.toml +└── README.md +``` + +**Usage:** + +```python +from my_dh_library.queries import filter_by_threshold, add_computed_columns +from deephaven import read_csv + +data = read_csv("data/sample.csv") +filtered = filter_by_threshold(data, "Score", 75.0) +``` + +**Use when:** + +- Creating reusable utilities for other projects. +- You don't need a command line interface. +- Code will be imported, not executed directly. + +### CLI-only package + +Package an executable command line tool without exposing library code. The command starts its own Deephaven server, so it runs as a standalone terminal command. + +**Structure:** + +``` +my_dh_cli/ +├── src/ +│ └── my_dh_cli/ +│ ├── __init__.py +│ ├── __main__.py +│ └── cli.py +├── pyproject.toml +└── README.md +``` + +**Usage:** + +```bash +my-dh-query data/sample.csv --verbose +``` + +**Use when:** + +- Building command line tools for data processing. +- The tool is run from a terminal, with no Python code required from the user. +- You don't need to expose library code to other projects. + +### Combined package + +Package both reusable library code and command line tools. In `my_dh_toolkit`, the commands call the package's own library functions: `my-dh-toolkit-query` and `my-dh-toolkit-process` both use `validate_columns` and `add_computed_columns` from `my_dh_toolkit.utils` and `my_dh_toolkit.queries`. Python users and terminal users get two interfaces to one implementation. See the example files directly in the repository: [`pyproject.toml`](my_dh_toolkit/pyproject.toml), [`cli.py`](my_dh_toolkit/src/my_dh_toolkit/cli.py), and [`processor.py`](my_dh_toolkit/src/my_dh_toolkit/processor.py). + +**Structure:** + +``` +my_dh_toolkit/ +├── src/ +│ └── my_dh_toolkit/ +│ ├── __init__.py +│ ├── __main__.py +│ ├── cli.py +│ ├── processor.py +│ ├── queries.py +│ └── utils.py +├── pyproject.toml +└── README.md +``` + +**Usage:** + +```python +# As a library +from my_dh_toolkit.queries import filter_by_threshold +from deephaven import read_csv + +data = read_csv("data/sample.csv") +filtered = filter_by_threshold(data, "Score", 75.0) +``` + +```bash +# As CLI commands +my-dh-toolkit-query data/sample.csv --verbose +my-dh-toolkit-process data/batch --output output --verbose +``` + +**Use when:** + +- You need both library and CLI functionality. +- You want to provide multiple interfaces to the same code. +- Library functions are useful independently. + +## Configure `pyproject.toml` + +The [`pyproject.toml`](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) file defines your package configuration. + +### Configuration options + +Here's a detailed breakdown of `pyproject.toml` for a library-only package: + +```toml +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_library" +version = "0.1.0" +description = "Reusable Deephaven query functions" +readme = "README.md" +requires-python = ">=3.9" +dependencies = [ + # deephaven-server also provides the deephaven module (through its deephaven-core dependency). + "deephaven-server>=0.35.0", +] + +[tool.setuptools.packages.find] +where = ["src"] +``` + +### Key sections + +- **`[build-system]`** - Specifies setuptools as the build backend +- **`[project]`** - Package metadata and dependencies +- **`name`** - Project name (used for `pip install`) +- **`dependencies`** - Required packages, installed automatically +- **`[tool.setuptools.packages.find]`** - Tells setuptools to find packages in `src/` + +For CLI packages, add a `[project.scripts]` section: + +```toml +[project.scripts] +my-dh-query = "my_dh_cli.cli:app" +``` + +This creates a command line entry point that calls the `app` function from `my_dh_cli.cli`. A package can define any number of commands in this section; `my_dh_toolkit` defines two. + +## Manage dependencies + +Dependencies are specified in the `dependencies` field: + +```toml +[project] +dependencies = [ + "deephaven-server>=0.35.0", + "click>=8.0.0", + "pandas>=2.0.0", +] +``` + +Declaring `deephaven-server` is sufficient for Deephaven: it depends on a matching version of `deephaven-core`, which provides the `deephaven` module that packages import. + +### Version constraints + +Use version specifiers to control which versions are acceptable: + +- `>=0.35.0` - Minimum version (0.35.0 or higher) +- `>=2.0.0,<3.0.0` - Version range (2.x only) +- `~=1.24.0` - Compatible release (>=1.24.0, <1.25.0) +- `==1.0.0` - Exact version (useful for fully pinned applications, but usually too strict for libraries) + +Lower-bound constraints such as `deephaven-server>=0.35.0` are compatibility constraints, not reproducible locks. They say which versions the package supports. If you also need repeatable installs over time, add a separate lock file or other pinning step for the environment that installs your package. + +### Optional dependencies + +Define optional feature sets that users can install separately: + +```toml +[project.optional-dependencies] +visualization = [ + "matplotlib>=3.7.0", + "seaborn>=0.12.0", +] +dev = [ + "pytest>=7.0.0", + "black>=23.0.0", +] +``` + +Users can install optional dependencies: + +```bash +pip install my_dh_library[visualization] +pip install my_dh_library[visualization,dev] +``` + +## Install and distribute + +Install from source in editable mode for development. Editable installs pick up source edits without reinstalling: + +```bash +cd my_dh_library +pip install -e . +``` + +Or install normally: + +```bash +pip install . +``` + +After installation, use the package as shown in [Packaging scenarios](#packaging-scenarios): import a library's functions after starting a server, or run a CLI package's commands directly. + +To distribute a package to other machines or publish it to a package index, build a wheel: + +```bash +cd my_dh_library +pip install build +python -m build +``` + +This creates a `.whl` file in `dist/` that can be: + +- Installed locally: `pip install dist/my_dh_library-0.1.0-py3-none-any.whl` +- Distributed to others +- Published to PyPI: `python -m twine upload dist/*` + +## Best practices + +### Package structure + +- Prefer the src-layout for packages like these examples +- Keep package names lowercase with underscores +- Match the package directory name to the import name +- Include `__init__.py` in all package directories +- In packages that define entry-point commands, keep every module imported before server startup free of Deephaven-dependent imports, and import `deephaven` lazily inside the command path that runs after startup + +### Dependencies + +- Specify minimum versions for Deephaven and critical dependencies +- Use version ranges for flexibility +- Group related optional dependencies +- Document any system-level dependencies, such as the Java version + +### Documentation + +- Include a README.md with installation and usage instructions +- Document all public functions and classes +- Explain server initialization requirements +- Include sample data so users can try the package + +## Adapt an example for your own project + +The repository examples are meant to be copied and renamed, not retyped from a long tutorial. A practical workflow is: + +1. Choose the closest example: + - [`my_dh_library/`](my_dh_library/) for an importable library + - [`my_dh_cli/`](my_dh_cli/) for one installed command + - [`my_dh_toolkit/`](my_dh_toolkit/) for a package that exposes both library code and commands +2. Copy that directory and rename the package under `src/` to your import name. +3. Update the copied example's `pyproject.toml`: set `name`, `description`, dependencies, and any `[project.scripts]` entries to your own values. +4. Replace the example business logic in the package modules with your own code. +5. Reinstall the package and run the same kind of smoke test shown in the example README. + +The repository README has a concise adaptation checklist in [Adapt an example for your own project](README.md#adapt-an-example-for-your-own-project), and each example directory shows the exact file layout to copy. + +## Next steps + +The [deephaven-python-packaging](https://github.com/deephaven-examples/deephaven-python-packaging) repository provides complete, working examples of all three packaging scenarios, along with sample data in its `data/` directory. Clone it and adapt the example that matches your project. + +## Related documentation + +- [Install and use Python packages](https://deephaven.io/core/docs/how-to-guides/install-and-use-python-packages/) +- [Use the Deephaven Python package](https://deephaven.io/core/docs/how-to-guides/deephaven-python-package/) +- [Writing your `pyproject.toml`](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) +- [src layout vs flat layout](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/) +- [Creating and packaging command-line tools](https://packaging.python.org/en/latest/guides/creating-command-line-tools/) +- [Setuptools documentation](https://setuptools.pypa.io/) +- [Click documentation](https://click.palletsprojects.com/)