From ec7fe0b138cc9b75478b9a11b38678fb2c9fa7f3 Mon Sep 17 00:00:00 2001 From: Matt Thalman Date: Thu, 19 Nov 2026 09:44:11 -0600 Subject: [PATCH] Clarify source-build dependency requirements Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- Documentation/README.md | 1 + Documentation/sourcebuild-in-repos/README.md | 5 + .../sourcebuild-in-repos/new-dependencies.md | 198 +++++++++++------- .../update-dependencies.md | 11 +- README.md | 2 + 5 files changed, 140 insertions(+), 77 deletions(-) diff --git a/Documentation/README.md b/Documentation/README.md index c4d44fb895..4865456216 100644 --- a/Documentation/README.md +++ b/Documentation/README.md @@ -14,6 +14,7 @@ | Document | Description | |---|---| +| [Adding Dependencies](sourcebuild-in-repos/new-dependencies.md) | Requirements and integration paths for direct and transitive dependencies used in the VMR source-build | | [CI Platform Coverage Guidelines](ci-platform-coverage-guidelines.md) | Guidelines for which platforms to test in CI | | [Feature Band Source Building](feature-band-source-building.md) | Guide for Linux distro maintainers on building feature band branches | | [Packaging and Installation](packaging-installation.md) | How to install or package a source-built .NET SDK | diff --git a/Documentation/sourcebuild-in-repos/README.md b/Documentation/sourcebuild-in-repos/README.md index 2b0ae7fb72..b0fc788dc4 100644 --- a/Documentation/sourcebuild-in-repos/README.md +++ b/Documentation/sourcebuild-in-repos/README.md @@ -3,6 +3,11 @@ These docs aim to be a collection of useful information for repo owners who work with and participate in source-build. +Before adding a NuGet dependency used in the VMR source-build, read the +[requirements for adding dependencies](new-dependencies.md). Adding a package +reference alone does not make the package and its transitive dependencies +available to source-build. + ## Index * [Adding new dependencies](new-dependencies.md) diff --git a/Documentation/sourcebuild-in-repos/new-dependencies.md b/Documentation/sourcebuild-in-repos/new-dependencies.md index 6ab970f008..b9afe2fb71 100644 --- a/Documentation/sourcebuild-in-repos/new-dependencies.md +++ b/Documentation/sourcebuild-in-repos/new-dependencies.md @@ -1,10 +1,121 @@ # Adding a new dependency +Adding a NuGet package reference is not enough to make a dependency available +to source-build. A normal repository build can restore a published binary from +NuGet. The VMR (Virtual Monolithic Repository) source-build must instead build +the product from source using prepared local inputs, without relying on online +package feeds. Having open-source code available does not by itself integrate +it into the build. + +These requirements apply to every package used in the VMR source-build, +whether it is produced by a VMR repository, +[Source Build Assets (SBA)][sba-repo], or supplied through another allowed +source-build input. They include **transitive dependencies** and packages used +as build tools, not just packages named directly in a project. For example, if +package A depends on package B, source-building A is not enough. B must also +come from an allowed source-build input. A version upgrade can introduce the +same problem without adding a new NuGet package reference. + +## Requirements for packages used by the VMR source-build + +Before making a product change depend on a package, work out how the VMR +source-build will obtain it. If the package or required version is not already +source-built, we recommend [opening a source-build +issue][source-build-issue] to coordinate +the integration with the source-build team. + +1. **Identify the complete dependency set and its use.** Check the package's + direct and transitive dependencies, including the tools needed to build its + source. Determine which packages provide executable implementation and + which are needed only as compile-time references or text assets. Check + whether the required package versions, not just their repositories, are + available in the VMR. A repository-level SourceBuild leg does not prove + that the VMR produces the specific package version. +1. **Build implementations from source in the VMR.** Packages that contain + executable code used by the product must be built from their source as part + of the VMR build. This includes + their transitive implementation dependencies. Build-time packages must + also come from the VMR build or an allowed source-build input so the build + runs without network access. The limited reference-only, text-only, and + bootstrap inputs are described + [below](#reference-only-text-only-and-bootstrap-inputs). + Before flowing implementation packages to consumers, check the produced + package and binary versions. Preserve strong names when the original + binaries have them. +1. **Check that the sources can be included in the VMR.** Included sources must + meet the VMR's [license and binary + policies](https://github.com/dotnet/dotnet/blob/main/docs/VMR-Permissible-Sources.md). + Public source hosting alone does not establish license compliance. + Checked-in binaries are subject to validation and source-build cleaning; + they are not a substitute for building the dependency. +1. **Connect the producer to its consumers.** For packages produced in the + VMR, make the source-built packages available before the consuming + repository builds, and align the requested versions with the produced + versions. Coordinate producer and consumer changes so the dependency does + not reach the VMR before its source-build support. See [package dependency + flow](../package-dependency-flow.md) for repository ordering and version + overrides. +1. **Validate in the VMR source-build.** Validate the integration with a [full + VMR source-build](https://github.com/dotnet/dotnet/blob/main/README.md#building), + not only a normal repository + build or a repository-level source-build leg. + +## Adding dependencies + +### Choose an integration path + +Choose how to provide the package based on its producer. The requirements +above apply to each path: + +- **A package already produced by the VMR:** Reuse that producer and arrange + the graph and version flow. +- **A repository onboarded directly into the VMR:** Follow the + [VMR Repository Onboarding + Guide](https://github.com/dotnet/dotnet/blob/main/docs/Repository-Onboarding.md). + This path requires Arcade onboarding, builds registered in BAR, and + source-build compatibility. +- **An external component built through [SBA][sba-repo]:** Use this option for + a repository that does not use Arcade and cannot be onboarded directly. + Repositories with few dependencies and simple builds are good candidates. + Pin the source submodule to the revision for the release tag corresponding to + the package version, and maintain its build project and patches. Coordinate + the + [SBA][sba-repo] source flow with consumer version updates so the package is + available when needed. Follow the [SBA][sba-repo] guidance for + [adding an external component](https://github.com/dotnet/source-build-assets/blob/main/README.md#adding-a-new-external-component). + +If none of these paths fits an implementation dependency, coordinate with the +source-build team before adding it. + +## Reference-only, text-only, and bootstrap inputs + +These inputs have specific uses and do not replace product implementations: + +- **Reference-only:** [SBA][sba-repo] reference packages contain API + signatures, not implementations. Targeting packages bundle reference + assemblies. Neither replaces code that executes during the build or ships + with the product, and [SBA][sba-repo] reference assemblies must not leak + into SDK output. See the + [reference package](https://github.com/dotnet/source-build-assets/blob/main/README.md#reference) + and [targeting package](https://github.com/dotnet/source-build-assets/blob/main/README.md#targeting) + guidance, including restrictions on current-major and preview versions. +- **Text-only:** Packages containing only text assets, such as props and + targets without executable code, can use [SBA][sba-repo]'s [text-only package + process](https://github.com/dotnet/source-build-assets/blob/main/README.md#text-only). + Any implementation dependencies those assets introduce still need to be + accounted for. +- **Bootstrap:** Previously source-built (PSB) packages can supply tools and + other build-time inputs, but PSB implementations must not leak into the + product. See + [previously source-built packages](../package-dependency-flow.md#previous-source-built-packages) + and [bootstrapping guidelines](../bootstrapping-guidelines.md). + ## Basics -When adding a new dependency, there are a few steps common between any type of -dependency. If you use darc, `darc add-dependency` will take care of this -process for you. Otherwise: +For dependencies managed through Arcade, declare the package and its producing +repository in the dependency metadata. `darc add-dependency` handles the normal +dependency entries; verify the source-build metadata and flow as well. +For manual changes: 1. Add the package and a default version to eng/Versions.props. The default version property should be named `PackageVersion`, e.g. @@ -24,10 +135,12 @@ process for you. Otherwise: 1. Set up dependency flow from the foreign repo to your repo. -In general, you should aim to use one version of each package. If you are using -a package as reference-only, it is possible to use multiple versions, but only -one implementation version of each package will be used - source-build will -override it to the version that is being built in this version of the SDK. +In general, aim to use one implementation version of each package and align +consumers with the version produced by the source-build graph. Multiple +reference-only versions can satisfy different compile-time requirements. +If multiple implementation versions are needed, coordinate that explicitly; +[SBA][sba-repo] documents how to [build multiple versions of an external +component](https://github.com/dotnet/source-build-assets/blob/main/README.md#building-multiple-versions-of-an-external-component). The source-build metadata is important - this tells source-build which repo package contains the specific nupkg you want. @@ -41,68 +154,6 @@ dependency](https://github.com/dotnet/installer/blob/ba1739a2363b1062f03ea386ec6 You can find the version needed by running `darc get-build` or using [BAR](https://aka.ms/bar). -## Adding dependencies - -Source build classifies dependencies in the following ways - -1. .NET - a dependency on a component from the .NET org - e.g. dotnet/runtime -1. Microsoft - a dependency on a component from the Microsoft org - e.g. - microsoft/vstest -1. External - a dependency on a component outside of Microsoft/.NET - e.g. - JamesNK/Newtonsoft.Json - -The following checklist can be used to determine how to handle each type of -dependency and the nuances it may have. - -1. Are you already using a package from the same repo and is the new dependency - already source-built? You can determine this by checking if the repo has a - SourceBuild leg building, e.g. runtime's is `runtime-dev-innerloop (Build - Linux x64 release SourceBuild)`. - - 1. This is the simplest case. You can add new dependencies from repos you - are already using freely. Note that you need to use the same version for - all packages from the repo. -1. Is the repo already built in source-build including the specific package you - want? - 1. Add the dependency using [`darc - add-dependency`](https://github.com/dotnet/arcade/blob/main/Documentation/Darc.md#add-dependency) - as normal, then add the [source-build metadata](#basics) as above. -1. Is the repo already built in source-build but the specific package is not? - 1. There's probably an issue with source-building this package. Please talk - to a [source-build team - member](https://github.com/orgs/dotnet/teams/source-build) about - why that is and whether we can fix it. -1. Is this a repo that uses Arcade to build? - 1. Does the foreign repo depend on your repo, directly or indirectly? i.e. - would adding the dependency create a cycle? - 1. This isn't necessarily a deal-breaker - it can sometimes be worked - around with reference-only packages. Please contact a [source-build - team - member](https://github.com/orgs/dotnet/teams/source-build) to - discuss. - 1. Does the foreign repo publish to BAR? - 1. If not, please contact them to get them publishing to BAR in an - appropriate channel. - 1. If neither of these caveats apply you should be in good shape. Follow the - instructions under "Basics" above. -1. Dependencies that have no code (e.g. SDKs with just props and targets) can - usually be added using the [source-build text-only-package - process](https://github.com/dotnet/source-build-reference-packages/tree/main/src/textOnlyPackages/src). - If the package is not already included there, you can [open a - PR](https://github.com/dotnet/source-build-reference-packages/pulls) or [file - an issue](https://github.com/dotnet/source-build/issues/new/choose) to include - it. -1. Source-build has in the past used Arcade shims to allow non-Arcade repos to - build appropriate packages for source-build. Please [log an - issue](https://github.com/dotnet/source-build/issues/new/choose) to determine - if this is a workable approach for your foreign repo. -1. We build some external dependencies in the - [dotnet/source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages/tree/main/src/externalPackages/src) - repo. Good targets for this generally have very few if any dependencies and - very simple build processes. [Please log an - issue](https://github.com/dotnet/source-build/issues/new/choose) to get the - process started. - ## Deciding what version to use - If you are using the package as reference-only and want the version to be @@ -112,9 +163,9 @@ dependency and the nuances it may have. `1.2.3` in addition to `4.5.6`. Also verify that the package is available in - [source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages), - and if not, [file a source-build - issue](https://github.com/dotnet/source-build/issues). + [Source Build Assets][sba-repo], + and if not, consider [opening a source-build + issue][source-build-issue]. - If you are using the package in the actual build or want the version to be updated whenever the foreign repo publishes to your channel, use the version number property set up in eng/Versions.props. When performing a source-build, @@ -124,3 +175,6 @@ dependency and the nuances it may have. essentially maintained as a separate concern, so something like repo A requiring Newtonsoft.Json 9.0.1 and repo B requiring 12.0.2 essentially doubles the source-build team's work. + +[sba-repo]: https://github.com/dotnet/source-build-assets +[source-build-issue]: https://github.com/dotnet/source-build/issues/new/choose diff --git a/Documentation/sourcebuild-in-repos/update-dependencies.md b/Documentation/sourcebuild-in-repos/update-dependencies.md index 2e7cbb7403..5cbf786692 100644 --- a/Documentation/sourcebuild-in-repos/update-dependencies.md +++ b/Documentation/sourcebuild-in-repos/update-dependencies.md @@ -26,12 +26,13 @@ Example Darc command: `darc update-dependencies --name MyCoolPackage -v 1.2.3` Updating a non-Microsoft (e.g. Newtonsoft.Json 9.0.1 to 13.0.1) or non-Arcade (e.g. ApplicationInsights) package is typically equivalent to adding the new -version all over again. Please [log an +version all over again. We recommend [opening a source-build issue](https://github.com/dotnet/source-build/issues/new/choose) to discuss -this. You can check if the external package you want is already included in -source-build in the -[dotnet/source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages/tree/main/src/externalPackages/src) -repo. +this. Review the [package requirements](new-dependencies.md#requirements-for-packages-used-by-the-vmr-source-build), +including any new or changed transitive dependencies. Check which external +repositories [Source Build Assets](https://github.com/dotnet/source-build-assets) includes in its +[external packages directory](https://github.com/dotnet/source-build-assets/tree/main/src/externalPackages/src), and verify that the +required package and version are produced for the VMR. ## Splitting, combining, and moving packages diff --git a/README.md b/README.md index 9fcafddc09..57aec11d45 100644 --- a/README.md +++ b/README.md @@ -94,6 +94,8 @@ to build the whole .NET SDK from source. ## Comprehensive Guidelines +* [Adding dependencies to source-build](Documentation/sourcebuild-in-repos/new-dependencies.md) - + Requirements to address before adding a NuGet dependency used in the VMR source-build * [Bootstrapping new distro and architecture guidelines](Documentation/bootstrapping-guidelines.md) * [Distribution packaging