Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 34 additions & 18 deletions rust/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ transports.
| `transport` | `Transport` | `Default`, `Stdio`, `InProcess`, `Tcp`, or `External` |
| `extension_launch_provider` | `Option<Arc<dyn ExtensionLaunchProvider>>` | Connection-global extension launch resolver |

With the default `CliProgram::Resolve`, managed stdio and TCP transports resolve an explicit `CliProgram::Path(path)`, `COPILOT_CLI_PATH`, then the bundled `copilot-runtime` wrapper and adjacent `runtime.node`. In-process transport retains its CLI-entrypoint resolution. There is no PATH scanning.
With the default `CliProgram::Resolve`, managed stdio and TCP transports resolve an explicit `CliProgram::Path(path)`, `COPILOT_CLI_PATH`, then the bundled `copilot-runtime` wrapper and adjacent `runtime.node`. In-process transport loads the native runtime library adjacent to that resolved runtime bundle. There is no PATH scanning.

#### Extension launch provider

Expand Down Expand Up @@ -977,12 +977,13 @@ none of them are scheduled for removal.

## Bundled runtime artifacts

The SDK provisions its runtime at build time. By default the `bundled-cli`
feature embeds the verified `copilot-runtime` wrapper and adjacent
`runtime.node` in your compiled crate. The compatible CLI artifact remains
available separately for `install_bundled_cli` and in-process hosting.
Enable `bundled-in-process` to additionally embed the native runtime library
and use `Transport::InProcess`:
The SDK provisions two verified artifacts at build time. By default the
`bundled-cli` feature embeds both the full Copilot CLI/Node SEA and a separate
runtime bundle containing `copilot-runtime`, adjacent `runtime.node`, and its
required assets. Managed transports use only the runtime bundle; the full CLI
is available through `install_bundled_cli` for diagnostics and version probes.
Enable `bundled-in-process` to additionally include the native runtime library
in the runtime bundle and use `Transport::InProcess`:

```toml
github-copilot-sdk = { version = "1", features = ["bundled-in-process"] }
Expand Down Expand Up @@ -1017,17 +1018,24 @@ github-copilot-sdk = { version = "1", default-features = false }
### How it works

1. **Version pin.** `build.rs` reads the CLI version from one of two sources:
- `cli-version.txt` at the crate root (present in published crate tarballs and vendored slots).
- `cli-version.txt` and `cli-version-in-process.txt` at the crate root
(present in published crate tarballs and vendored slots).
- Otherwise, `../nodejs/package.json` (contributor build inside the github/copilot-sdk repo).

The resolved version is baked into the crate via `cargo:rustc-env=COPILOT_SDK_CLI_VERSION` regardless of mode. The runtime resolver consumes it to recompute the on-disk path by convention, so no absolute paths leak into the rlib.

2. **Build time:** `build.rs` downloads the platform-specific release archive and
verifies its SHA-256 against the release's `SHA256SUMS.txt` or the publish snapshot.
2. **Build time:** `build.rs` downloads the platform-specific full CLI archive
and runtime package, then verifies both SHA-256 hashes against the release's
`SHA256SUMS.txt` or the publish snapshots.
Then:
- **`bundled-cli` on (default):** creates and embeds a minimal archive containing the CLI executable, `copilot-runtime[.exe]`, and `runtime.node`.
- **`bundled-in-process` on:** the archive additionally contains the platform-native runtime library (`.dll`, `.so`, or `.dylib`).
- **`bundled-cli` off:** extracts the same artifacts directly into the platform cache using staging files and atomic renames.
- **`bundled-cli` on (default):** embeds the full CLI release archive and a
separately filtered runtime archive containing `copilot-runtime[.exe]`,
`runtime.node`, and required assets.
- **`bundled-in-process` on:** the runtime archive additionally contains the
platform-native runtime library (`.dll`, `.so`, or `.dylib`).
- **`bundled-cli` off:** downloads only the runtime package and extracts its
managed runtime artifacts directly into the platform cache using staging
files and atomic renames.

3. **Runtime:** in both modes the artifacts share one versioned directory:

Expand Down Expand Up @@ -1075,9 +1083,9 @@ For managed child-process transports, `Client::start` resolves the program in th
3. **`bundled-cli` on:** the embedded wrapper pair, lazily extracted on first call.
4. **`bundled-cli` off:** the build-time-extracted wrapper pair in the per-user cache.

In-process transport resolves the compatible CLI artifact from
`COPILOT_CLI_PATH`, the embedded archive, or the build-time cache. There is no
PATH scanning.
In-process transport loads the native runtime library adjacent to the runtime
wrapper selected from `COPILOT_CLI_PATH`, the embedded runtime archive, or the
build-time cache. There is no PATH scanning.

### Reaching the bundled binary without a `Client`

Expand Down Expand Up @@ -1118,11 +1126,19 @@ returns the wrapper path.

### Download cache (build-time, embed mode)

In embed mode `build.rs` re-downloads on every clean build by default. Set `BUNDLED_CLI_CACHE_DIR=<path>` to cache the verified archive between builds (CI keys this on `<os>-<version>` for ~zero-cost rebuilds on cache hits). With `bundled-cli` disabled there is no separate archive cache — the extracted binary itself is the cache.
In embed mode `build.rs` downloads both verified archives on every clean build
by default. Set `BUNDLED_CLI_CACHE_DIR=<path>` to cache them between builds (CI
keys this on `<os>-<version>` for near-zero-cost rebuilds on cache hits). For
Copilot CLI 1.0.83-5, the two upstream archives total roughly 132-157 MB per
platform before the runtime package is filtered. With `bundled-cli` disabled
there is no separate archive cache: the extracted runtime bundle is the cache.

### Platforms

Supported: `darwin-arm64`, `darwin-x64`, `linux-x64`, `linux-arm64`, `win32-x64`, `win32-arm64`. The target platform is auto-detected from `CARGO_CFG_TARGET_OS` and `CARGO_CFG_TARGET_ARCH` (cross-compilation works).
Supported: `darwin-arm64`, `darwin-x64`, `linux-x64`, `linux-arm64`,
`linuxmusl-x64`, `linuxmusl-arm64`, `win32-x64`, and `win32-arm64`. The target
platform is auto-detected from `CARGO_CFG_TARGET_OS`, `CARGO_CFG_TARGET_ARCH`,
and `CARGO_CFG_TARGET_ENV` (cross-compilation works).

## Features

Expand Down
171 changes: 130 additions & 41 deletions rust/build/in_process.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ pub(crate) fn main() {
println!("cargo:rerun-if-env-changed=BUNDLED_CLI_CACHE_DIR");
println!("cargo::rustc-check-cfg=cfg(has_bundled_cli)");
println!("cargo::rustc-check-cfg=cfg(has_extracted_cli)");
println!("cargo:rerun-if-changed=cli-version.txt");
println!("cargo:rerun-if-changed=cli-version-in-process.txt");

// Only declare the package metadata rerun when it actually exists.
Expand All @@ -19,7 +20,7 @@ pub(crate) fn main() {
// `nodejs/` (vendored slots, published crates) would force build.rs
// to re-run on every `cargo build` even when nothing has changed.
// The package file is only the source-of-truth in this repo's
// contributor builds; everywhere else `cli-version-in-process.txt` is canonical.
// contributor builds; everywhere else the snapshot files are canonical.
let manifest_dir = std::env::var("CARGO_MANIFEST_DIR").expect("CARGO_MANIFEST_DIR is set");
let package_json = Path::new(&manifest_dir)
.join("..")
Expand Down Expand Up @@ -92,12 +93,37 @@ pub(crate) fn main() {
let include_runtime = std::env::var_os("CARGO_FEATURE_BUNDLED_IN_PROCESS").is_some();

if std::env::var_os("CARGO_FEATURE_BUNDLED_CLI").is_some() {
let expected_hash = local_expected_hash
let runtime_expected_hash = local_expected_hash
.clone()
.unwrap_or_else(|| fetch_in_process_release_hash(&version, platform.package_name));
let archive = cached_download(&download_url, &cache_key, &expected_hash, &cache_dir);
verify_runtime_package(&archive, platform, &archive_name);
emit_embedded(out, &archive, platform, include_runtime);
let runtime_package = cached_download(
&download_url,
&cache_key,
&runtime_expected_hash,
&cache_dir,
);
verify_runtime_package(&runtime_package, platform, &archive_name);

let cli_asset_name = platform.cli_asset_name();
let cli_expected_hash = resolve_cli_hash(&version, &cli_asset_name);
let cli_archive = cached_download(
&format!(
"https://github.com/github/copilot-cli/releases/download/v{version}/{cli_asset_name}"
),
&format!("v{version}-{cli_asset_name}"),
&cli_expected_hash,
&cache_dir,
);
let cli_binary_size = verify_cli_archive(&cli_archive, platform, &cli_asset_name);

emit_embedded(
out,
&cli_archive,
cli_binary_size,
&runtime_package,
platform,
include_runtime,
);
println!("cargo:rustc-cfg=has_bundled_cli");
} else {
// With `bundled-cli` off the extracted runtime pair *is* the cache.
Expand Down Expand Up @@ -185,30 +211,43 @@ fn extracted_install_dir(version: &str) -> PathBuf {
}
}

/// Emit the `bundled_cli.rs` glue + `copilot_cli.archive` blob into `OUT_DIR`
/// for embed mode (`bundled-cli` cargo feature on). The version is exposed
/// crate-wide via the unconditional `cargo:rustc-env=COPILOT_SDK_CLI_VERSION`
/// emit; the binary name is OS-derived at runtime — so all we need to
/// generate here is the archive blob include.
fn emit_embedded(out: &Path, package: &[u8], platform: Platform, include_runtime: bool) {
let archive = build_embedded_archive(package, platform, include_runtime);
std::fs::write(out.join("copilot_cli.archive"), archive)
/// Emit separate full-CLI and runtime payloads into `OUT_DIR` for embed mode.
fn emit_embedded(
out: &Path,
cli_archive: &[u8],
cli_binary_size: u64,
runtime_package: &[u8],
platform: Platform,
include_runtime: bool,
) {
let runtime_archive =
build_embedded_runtime_archive(runtime_package, platform, include_runtime);
std::fs::write(out.join("copilot_cli.archive"), cli_archive)
.expect("failed to write copilot_cli.archive");
std::fs::write(out.join("copilot_runtime.archive"), runtime_archive)
.expect("failed to write copilot_runtime.archive");

let generated = r#"// Auto-generated by github-copilot-sdk build.rs. Do not edit.
let generated = format!(
r#"// Auto-generated by github-copilot-sdk build.rs. Do not edit.
pub(super) static CLI_ARCHIVE: &[u8] = include_bytes!("copilot_cli.archive");
"#;
pub(super) static RUNTIME_ARCHIVE: &[u8] = include_bytes!("copilot_runtime.archive");
pub(super) const CLI_BINARY_SIZE: u64 = {cli_binary_size};
"#
);

std::fs::write(out.join("bundled_cli.rs"), generated).expect("failed to write bundled_cli.rs");
}

fn build_embedded_archive(package: &[u8], platform: Platform, include_runtime: bool) -> Vec<u8> {
fn build_embedded_runtime_archive(
package: &[u8],
platform: Platform,
include_runtime: bool,
) -> Vec<u8> {
let encoder = flate2::GzBuilder::new()
.mtime(0)
.write(Vec::new(), flate2::Compression::default());
let mut archive = tar::Builder::new(encoder);
let (runtime, wrapper) = append_hostless_runtime_tree(&mut archive, package, platform);
append_archive_file(&mut archive, platform.binary_name, &wrapper, 0o755);
let runtime = append_hostless_runtime_tree(&mut archive, package, platform);
if include_runtime {
append_archive_file(
&mut archive,
Expand All @@ -229,11 +268,10 @@ fn append_hostless_runtime_tree<W: Write>(
archive: &mut tar::Builder<W>,
package: &[u8],
platform: Platform,
) -> (Vec<u8>, Vec<u8>) {
) -> Vec<u8> {
let decoder = flate2::read::GzDecoder::new(package);
let mut source = tar::Archive::new(decoder);
let mut runtime = None;
let mut wrapper = None;
for entry in source
.entries()
.unwrap_or_else(|e| panic!("failed to read npm package entries: {e}"))
Expand All @@ -257,9 +295,6 @@ fn append_hostless_runtime_tree<W: Write>(
if destination == Path::new("runtime.node") {
runtime = Some(bytes.clone());
}
if destination == Path::new(platform.runtime_wrapper_name()) {
wrapper = Some(bytes.clone());
}
append_archive_file(
archive,
destination
Expand All @@ -269,21 +304,12 @@ fn append_hostless_runtime_tree<W: Write>(
mode,
);
}
(
runtime.unwrap_or_else(|| {
panic!(
"package `{}` does not contain prebuilds/<platform>/runtime.node",
platform.package_name
)
}),
wrapper.unwrap_or_else(|| {
panic!(
"package `{}` does not contain prebuilds/<platform>/{}",
platform.package_name,
platform.runtime_wrapper_name()
)
}),
)
runtime.unwrap_or_else(|| {
panic!(
"package `{}` does not contain prebuilds/<platform>/runtime.node",
platform.package_name
)
})
}

fn hostless_runtime_path(source: &str, platform: Platform) -> Option<PathBuf> {
Expand Down Expand Up @@ -397,6 +423,25 @@ fn fetch_in_process_release_hash(version: &str, package_name: &str) -> String {
fetch_release_hash(version, &asset_name)
}

fn resolve_cli_hash(version: &str, asset_name: &str) -> String {
let manifest_dir = std::env::var("CARGO_MANIFEST_DIR").expect("CARGO_MANIFEST_DIR is set");
let snapshot = Path::new(&manifest_dir).join("cli-version.txt");
if snapshot.is_file() {
let contents = std::fs::read_to_string(&snapshot)
.unwrap_or_else(|e| panic!("failed to read {}: {e}", snapshot.display()));
let (snapshot_version, hash) = parse_snapshot(&contents, asset_name)
.unwrap_or_else(|e| panic!("invalid {}: {e}", snapshot.display()));
assert_eq!(
snapshot_version,
version,
"{} and the selected runtime version source must pin the same version",
snapshot.display()
);
return hash;
}
fetch_release_hash(version, asset_name)
}

fn marker_matches_version(contents: &str, version: &str) -> bool {
let mut lines = contents.lines();
lines.next() == Some(version)
Expand Down Expand Up @@ -471,6 +516,19 @@ struct Platform {
}

impl Platform {
fn cli_asset_name(&self) -> String {
let platform = self
.package_name
.strip_prefix("copilot-")
.expect("platform package name has copilot- prefix");
let extension = if self.package_name.contains("win32") {
"zip"
} else {
"tar.gz"
};
format!("copilot-{platform}.{extension}")
}

fn runtime_wrapper_name(&self) -> &'static str {
if self.package_name.contains("win32") {
"copilot-runtime.exe"
Expand Down Expand Up @@ -889,22 +947,53 @@ fn verify_runtime_package(archive: &[u8], platform: Platform, package_name: &str
}
}

fn verify_cli_archive(archive: &[u8], platform: Platform, archive_name: &str) -> u64 {
let binary_size = if platform.package_name.contains("win32") {
archive_zip_entry_size(archive, platform.binary_name)
} else {
archive_tar_entry_size(archive, platform.binary_name)
};
binary_size.unwrap_or_else(|| {
panic!(
"Copilot CLI archive `{archive_name}` does not contain an entry named `{}`",
platform.binary_name
)
})
}

fn archive_contains_tar_entry(targz: &[u8], binary_name: &str) -> bool {
archive_tar_entry_size(targz, binary_name).is_some()
}

fn archive_tar_entry_size(targz: &[u8], binary_name: &str) -> Option<u64> {
let gz = flate2::read::GzDecoder::new(targz);
let mut archive = tar::Archive::new(gz);
let Ok(entries) = archive.entries() else {
return false;
return None;
};
for entry in entries.flatten() {
let Ok(path) = entry.path() else {
continue;
};
let name = path.to_string_lossy();
if name == binary_name || name.ends_with(&format!("/{binary_name}")) {
return true;
return Some(entry.size());
}
}
false
None
}

fn archive_zip_entry_size(zip_bytes: &[u8], binary_name: &str) -> Option<u64> {
let reader = std::io::Cursor::new(zip_bytes);
let Ok(mut archive) = zip::ZipArchive::new(reader) else {
return None;
};
(0..archive.len()).find_map(|index| {
archive.by_index(index).ok().and_then(|entry| {
(entry.name() == binary_name || entry.name().ends_with(&format!("/{binary_name}")))
.then(|| entry.size())
})
})
}

fn verify_hash(data: &[u8], expected: &str) -> bool {
Expand Down
2 changes: 2 additions & 0 deletions rust/scripts/snapshot-bundled-cli-version.sh
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ ASSETS=(
"copilot-darwin-x64.tar.gz"
"copilot-linux-arm64.tar.gz"
"copilot-linux-x64.tar.gz"
"copilot-linuxmusl-arm64.tar.gz"
"copilot-linuxmusl-x64.tar.gz"
"copilot-win32-arm64.zip"
"copilot-win32-x64.zip"
)
Expand Down
Loading
Loading