Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Documentation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@

| Document | Description |
|---|---|
| [Adding Dependencies](sourcebuild-in-repos/new-dependencies.md) | Requirements and integration paths for direct and transitive dependencies used in the VMR source-build |
| [CI Platform Coverage Guidelines](ci-platform-coverage-guidelines.md) | Guidelines for which platforms to test in CI |
| [Feature Band Source Building](feature-band-source-building.md) | Guide for Linux distro maintainers on building feature band branches |
| [Packaging and Installation](packaging-installation.md) | How to install or package a source-built .NET SDK |
Expand Down
5 changes: 5 additions & 0 deletions Documentation/sourcebuild-in-repos/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
These docs aim to be a collection of useful information for repo owners who work
with and participate in source-build.

Before adding a NuGet dependency used in the VMR source-build, read the
[requirements for adding dependencies](new-dependencies.md). Adding a package
reference alone does not make the package and its transitive dependencies
available to source-build.

## Index

* [Adding new dependencies](new-dependencies.md)
Expand Down
198 changes: 126 additions & 72 deletions Documentation/sourcebuild-in-repos/new-dependencies.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,121 @@
# Adding a new dependency

Adding a NuGet package reference is not enough to make a dependency available
to source-build. A normal repository build can restore a published binary from
NuGet. The VMR (Virtual Monolithic Repository) source-build must instead build
the product from source using prepared local inputs, without relying on online
package feeds. Having open-source code available does not by itself integrate
it into the build.

These requirements apply to every package used in the VMR source-build,
whether it is produced by a VMR repository,
[Source Build Assets (SBA)][sba-repo], or supplied through another allowed
source-build input. They include **transitive dependencies** and packages used
as build tools, not just packages named directly in a project. For example, if
package A depends on package B, source-building A is not enough. B must also
come from an allowed source-build input. A version upgrade can introduce the
same problem without adding a new NuGet package reference.

## Requirements for packages used by the VMR source-build

Before making a product change depend on a package, work out how the VMR
source-build will obtain it. If the package or required version is not already
source-built, we recommend [opening a source-build
issue][source-build-issue] to coordinate
the integration with the source-build team.

1. **Identify the complete dependency set and its use.** Check the package's
direct and transitive dependencies, including the tools needed to build its
source. Determine which packages provide executable implementation and
which are needed only as compile-time references or text assets. Check
whether the required package versions, not just their repositories, are
available in the VMR. A repository-level SourceBuild leg does not prove
that the VMR produces the specific package version.
1. **Build implementations from source in the VMR.** Packages that contain
executable code used by the product must be built from their source as part
of the VMR build. This includes
their transitive implementation dependencies. Build-time packages must
Comment thread
mthalman marked this conversation as resolved.
Resolved
also come from the VMR build or an allowed source-build input so the build
runs without network access. The limited reference-only, text-only, and
bootstrap inputs are described
[below](#reference-only-text-only-and-bootstrap-inputs).
Before flowing implementation packages to consumers, check the produced
package and binary versions. Preserve strong names when the original
binaries have them.
1. **Check that the sources can be included in the VMR.** Included sources must
meet the VMR's [license and binary
policies](https://github.com/dotnet/dotnet/blob/main/docs/VMR-Permissible-Sources.md).
Public source hosting alone does not establish license compliance.
Checked-in binaries are subject to validation and source-build cleaning;
they are not a substitute for building the dependency.
1. **Connect the producer to its consumers.** For packages produced in the
VMR, make the source-built packages available before the consuming
repository builds, and align the requested versions with the produced
versions. Coordinate producer and consumer changes so the dependency does
not reach the VMR before its source-build support. See [package dependency
flow](../package-dependency-flow.md) for repository ordering and version
overrides.
1. **Validate in the VMR source-build.** Validate the integration with a [full
VMR source-build](https://github.com/dotnet/dotnet/blob/main/README.md#building),
not only a normal repository
build or a repository-level source-build leg.

## Adding dependencies

### Choose an integration path

Choose how to provide the package based on its producer. The requirements
above apply to each path:

- **A package already produced by the VMR:** Reuse that producer and arrange
the graph and version flow.
- **A repository onboarded directly into the VMR:** Follow the
[VMR Repository Onboarding
Guide](https://github.com/dotnet/dotnet/blob/main/docs/Repository-Onboarding.md).
This path requires Arcade onboarding, builds registered in BAR, and
source-build compatibility.
- **An external component built through [SBA][sba-repo]:** Use this option for
a repository that does not use Arcade and cannot be onboarded directly.
Repositories with few dependencies and simple builds are good candidates.
Pin the source submodule to the revision for the release tag corresponding to
the package version, and maintain its build project and patches. Coordinate
the
[SBA][sba-repo] source flow with consumer version updates so the package is
available when needed. Follow the [SBA][sba-repo] guidance for
[adding an external component](https://github.com/dotnet/source-build-assets/blob/main/README.md#adding-a-new-external-component).

If none of these paths fits an implementation dependency, coordinate with the
source-build team before adding it.

## Reference-only, text-only, and bootstrap inputs

These inputs have specific uses and do not replace product implementations:

- **Reference-only:** [SBA][sba-repo] reference packages contain API
signatures, not implementations. Targeting packages bundle reference
assemblies. Neither replaces code that executes during the build or ships
with the product, and [SBA][sba-repo] reference assemblies must not leak
into SDK output. See the
[reference package](https://github.com/dotnet/source-build-assets/blob/main/README.md#reference)
and [targeting package](https://github.com/dotnet/source-build-assets/blob/main/README.md#targeting)
guidance, including restrictions on current-major and preview versions.
- **Text-only:** Packages containing only text assets, such as props and
targets without executable code, can use [SBA][sba-repo]'s [text-only package
process](https://github.com/dotnet/source-build-assets/blob/main/README.md#text-only).
Any implementation dependencies those assets introduce still need to be
accounted for.
- **Bootstrap:** Previously source-built (PSB) packages can supply tools and
other build-time inputs, but PSB implementations must not leak into the
product. See
[previously source-built packages](../package-dependency-flow.md#previous-source-built-packages)
and [bootstrapping guidelines](../bootstrapping-guidelines.md).

## Basics

When adding a new dependency, there are a few steps common between any type of
dependency. If you use darc, `darc add-dependency` will take care of this
process for you. Otherwise:
For dependencies managed through Arcade, declare the package and its producing
repository in the dependency metadata. `darc add-dependency` handles the normal
dependency entries; verify the source-build metadata and flow as well.
For manual changes:

1. Add the package and a default version to eng/Versions.props. The default
version property should be named `<PackageName>PackageVersion`, e.g.
Expand All @@ -24,10 +135,12 @@ process for you. Otherwise:

1. Set up dependency flow from the foreign repo to your repo.

In general, you should aim to use one version of each package. If you are using
a package as reference-only, it is possible to use multiple versions, but only
one implementation version of each package will be used - source-build will
override it to the version that is being built in this version of the SDK.
In general, aim to use one implementation version of each package and align
consumers with the version produced by the source-build graph. Multiple
reference-only versions can satisfy different compile-time requirements.
If multiple implementation versions are needed, coordinate that explicitly;
[SBA][sba-repo] documents how to [build multiple versions of an external
component](https://github.com/dotnet/source-build-assets/blob/main/README.md#building-multiple-versions-of-an-external-component).

The source-build metadata is important - this tells source-build which repo
package contains the specific nupkg you want.
Expand All @@ -41,68 +154,6 @@ dependency](https://github.com/dotnet/installer/blob/ba1739a2363b1062f03ea386ec6
You can find the version needed by running `darc get-build` or using
[BAR](https://aka.ms/bar).

## Adding dependencies

Source build classifies dependencies in the following ways

1. .NET - a dependency on a component from the .NET org - e.g. dotnet/runtime
1. Microsoft - a dependency on a component from the Microsoft org - e.g.
microsoft/vstest
1. External - a dependency on a component outside of Microsoft/.NET - e.g.
JamesNK/Newtonsoft.Json

The following checklist can be used to determine how to handle each type of
dependency and the nuances it may have.

1. Are you already using a package from the same repo and is the new dependency
already source-built? You can determine this by checking if the repo has a
SourceBuild leg building, e.g. runtime's is `runtime-dev-innerloop (Build
Linux x64 release SourceBuild)`.

1. This is the simplest case. You can add new dependencies from repos you
are already using freely. Note that you need to use the same version for
all packages from the repo.
1. Is the repo already built in source-build including the specific package you
want?
1. Add the dependency using [`darc
add-dependency`](https://github.com/dotnet/arcade/blob/main/Documentation/Darc.md#add-dependency)
as normal, then add the [source-build metadata](#basics) as above.
1. Is the repo already built in source-build but the specific package is not?
1. There's probably an issue with source-building this package. Please talk
to a [source-build team
member](https://github.com/orgs/dotnet/teams/source-build) about
why that is and whether we can fix it.
1. Is this a repo that uses Arcade to build?
1. Does the foreign repo depend on your repo, directly or indirectly? i.e.
would adding the dependency create a cycle?
1. This isn't necessarily a deal-breaker - it can sometimes be worked
around with reference-only packages. Please contact a [source-build
team
member](https://github.com/orgs/dotnet/teams/source-build) to
discuss.
1. Does the foreign repo publish to BAR?
1. If not, please contact them to get them publishing to BAR in an
appropriate channel.
1. If neither of these caveats apply you should be in good shape. Follow the
instructions under "Basics" above.
1. Dependencies that have no code (e.g. SDKs with just props and targets) can
usually be added using the [source-build text-only-package
process](https://github.com/dotnet/source-build-reference-packages/tree/main/src/textOnlyPackages/src).
If the package is not already included there, you can [open a
PR](https://github.com/dotnet/source-build-reference-packages/pulls) or [file
an issue](https://github.com/dotnet/source-build/issues/new/choose) to include
it.
1. Source-build has in the past used Arcade shims to allow non-Arcade repos to
build appropriate packages for source-build. Please [log an
issue](https://github.com/dotnet/source-build/issues/new/choose) to determine
if this is a workable approach for your foreign repo.
1. We build some external dependencies in the
[dotnet/source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages/tree/main/src/externalPackages/src)
repo. Good targets for this generally have very few if any dependencies and
very simple build processes. [Please log an
issue](https://github.com/dotnet/source-build/issues/new/choose) to get the
process started.

## Deciding what version to use

- If you are using the package as reference-only and want the version to be
Expand All @@ -112,9 +163,9 @@ dependency and the nuances it may have.
`<PackageNameReferenceOnly>1.2.3</PackageNameReferenceOnly>` in addition to
`<PackageNamePackageVersion>4.5.6</PackageNamePackageVersion>`. Also verify
that the package is available in
[source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages),
and if not, [file a source-build
issue](https://github.com/dotnet/source-build/issues).
[Source Build Assets][sba-repo],
and if not, consider [opening a source-build
issue][source-build-issue].
- If you are using the package in the actual build or want the version to be
updated whenever the foreign repo publishes to your channel, use the version
number property set up in eng/Versions.props. When performing a source-build,
Expand All @@ -124,3 +175,6 @@ dependency and the nuances it may have.
essentially maintained as a separate concern, so something like repo A
requiring Newtonsoft.Json 9.0.1 and repo B requiring 12.0.2 essentially
doubles the source-build team's work.

[sba-repo]: https://github.com/dotnet/source-build-assets
[source-build-issue]: https://github.com/dotnet/source-build/issues/new/choose
11 changes: 6 additions & 5 deletions Documentation/sourcebuild-in-repos/update-dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,13 @@ Example Darc command: `darc update-dependencies --name MyCoolPackage -v 1.2.3`

Updating a non-Microsoft (e.g. Newtonsoft.Json 9.0.1 to 13.0.1) or non-Arcade
(e.g. ApplicationInsights) package is typically equivalent to adding the new
version all over again. Please [log an
version all over again. We recommend [opening a source-build
issue](https://github.com/dotnet/source-build/issues/new/choose) to discuss
this. You can check if the external package you want is already included in
source-build in the
[dotnet/source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages/tree/main/src/externalPackages/src)
repo.
this. Review the [package requirements](new-dependencies.md#requirements-for-packages-used-by-the-vmr-source-build),
including any new or changed transitive dependencies. Check which external
repositories [Source Build Assets](https://github.com/dotnet/source-build-assets) includes in its
[external packages directory](https://github.com/dotnet/source-build-assets/tree/main/src/externalPackages/src), and verify that the
required package and version are produced for the VMR.

## Splitting, combining, and moving packages

Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,8 @@ to build the whole .NET SDK from source.

## Comprehensive Guidelines

* [Adding dependencies to source-build](Documentation/sourcebuild-in-repos/new-dependencies.md) -
Requirements to address before adding a NuGet dependency used in the VMR source-build
* [Bootstrapping new distro and architecture
guidelines](Documentation/bootstrapping-guidelines.md)
* [Distribution packaging
Expand Down
Loading