Skip to content

fix(functions): keep sharing one source build across function deploy retries - #11260

Open
ajperel wants to merge 1 commit into
mainfrom
ajp/scraper-retries
Open

ajperel wants to merge 1 commit into
mainfrom
ajp/scraper-retries

Conversation

@ajperel

@ajperel ajperel commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Description

Functions deployed to the same region share one source build: the first function builds from source, and the rest wait for its source token and reuse that build. Since #11044, any retryable error on the first function (for example a quota error or a service account propagation delay) abandoned that shared build. Every waiting function then built from source in parallel, even when the retry succeeded, which slows down large deploys and uses more build quota.

Waiting functions now stay on the shared build while the first function retries. Only a terminal failure of the function that is building releases them. SourceTokenScraper.withToken now wraps the whole retried deploy, and the token is fetched at the start of each attempt, so tokens that expire during a backoff are still refreshed. This change also:

  • Stops a failed build from being treated as a finished shared build.
  • Gives the v2 Cloud Run quota delete-and-recreate the executor's backoff and retry limit instead of an unbounded loop.
  • Stops v2 deploys from waiting on a shared build when functionsv2deployoptimizations is disabled.

Scenarios Tested

  • Unit tests for SourceTokenScraper and Fabricator cover the paths that decide whether a build is shared:
    • The first function retrying and then succeeding, or exhausting its retries.
    • Failures from other functions while a build is in flight.
    • Token expiry during retries.
    • Failed operations with and without a token.
    • Cloud Run quota retries.
    • The disabled experiment.
  • Checked that each new test fails when the fix it covers is reverted.
  • npm run test.
  • Manually deployed a codebase with many functions in one region.

Sample Commands

firebase deploy --only functions

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request resolves an issue where retrying an initial Cloud Functions deployment could cause subsequent functions in the same region to build in parallel without a shared source token. This is achieved by swapping the order of scraper.withToken and this.functionExecutor.run in fabricator.ts, ensuring that waiting functions remain blocked on the scraper while the first function retries transient errors. Comprehensive unit tests have been added to fabricator.spec.ts to verify this behavior for both v1 and v2 functions during creation and updates. No review comments were provided, so there is no additional feedback.

@ajperel
ajperel force-pushed the ajp/scraper-retries branch from 783afad to e26c716 Compare October 10, 2026 00:52
@ajperel
ajperel requested review from joehan and shettyvarun268 and removed request for joehan and shettyvarun268 October 10, 2026 00:54
@ajperel
ajperel force-pushed the ajp/scraper-retries branch 2 times, most recently from bc3b246 to b243205 Compare October 17, 2026 00:17
@ajperel ajperel changed the title fix(functions): preserve source token scraping across function deploy retries fix(functions): keep sharing one source build across function deploy retries Oct 17, 2026
@ajperel
ajperel force-pushed the ajp/scraper-retries branch from b243205 to d738f17 Compare October 17, 2026 01:13
…retries

### Description

Functions deployed to the same region share one source build: the first function builds from source, and the rest wait for its source token and reuse that build. Since #11044, any retryable error on the first function (for example a quota error or a service account propagation delay) abandoned that shared build. Every waiting function then built from source in parallel, even when the retry succeeded, which slows down large deploys and uses more build quota.

Waiting functions now stay on the shared build while the first function retries. Only a terminal failure of the function that is building releases them. `SourceTokenScraper.withToken` now wraps the whole retried deploy, and the token is fetched at the start of each attempt, so tokens that expire during a backoff are still refreshed. This change also:

- Stops a failed build from being treated as a finished shared build.
- Gives the v2 Cloud Run quota delete-and-recreate the executor's backoff and retry limit instead of an unbounded loop.
- Stops v2 deploys from waiting on a shared build when `functionsv2deployoptimizations` is disabled.

### Scenarios Tested

- Unit tests for `SourceTokenScraper` and `Fabricator` cover the paths that decide whether a build is shared:
  - The first function retrying and then succeeding, or exhausting its retries.
  - Failures from other functions while a build is in flight.
  - Token expiry during retries.
  - Failed operations with and without a token.
  - Cloud Run quota retries.
  - The disabled experiment.
- Checked that each new test fails when the fix it covers is reverted.
- `npm run test`.
- Manually deployed a codebase with many functions in one region.

### Sample Commands

```bash
firebase deploy --only functions
```

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants