Skip to content

CI: Reduce workflow overhead #3962

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kevinjqliu wants to merge 4 commits into
base: main
Choose a base branch
from
Open

CI: Reduce workflow overhead #3962

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
09e311e
CI: Allow two concurrent merge-queue builds
kevinjqliu Sep 14, 2026
a04bb4c
CI: Cancel superseded pull request checks
kevinjqliu Sep 14, 2026
7aa5d08
CI: Use slim runners for lightweight jobs
kevinjqliu Sep 14, 2026
152f617
Docs: Skip default development dependencies
kevinjqliu Sep 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .asf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ github:
parameters:
check_response_timeout_minutes: 90
grouping_strategy: ALLGREEN
max_entries_to_build: 1
max_entries_to_build: 2
max_entries_to_merge: 1
merge_method: SQUASH
min_entries_to_merge: 1
Expand Down
6 changes: 5 additions & 1 deletion .github/workflows/asf-allowlist-check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,13 @@ on:
permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
asf-allowlist-check:
runs-on: ubuntu-24.04
runs-on: ubuntu-slim
Comment on lines 38 to +39

@jayceslesar jayceslesar Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to still be explicit about this versioning?

@rambleraptor rambleraptor Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The versioned Ubuntu images use 4 CPUs / 16 GB ram vs 1 CPU / 5 GB ram for slim. We're probably not using the additional resources, so might as well not waste them.

https://docs.github.com/en/actions/reference/runners/github-hosted-runners

@jayceslesar jayceslesar Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was just looking at apache/iceberg-verification#5 (comment) and thought it would make sense to see if possible is all

@jayceslesar jayceslesar Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

like is it possible to have a pinned slim 24.04

steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/check-md-link.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@ on:
permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
markdown-link-check:
runs-on: ubuntu-latest
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/codeql.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ on:
permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
analyze:
name: Analyze Actions
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/license_check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ on:
permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
rat:
runs-on: ubuntu-latest
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/nightly-pypi-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
Expand Down Expand Up @@ -30,7 +30,7 @@
jobs:
set-version:
if: github.repository == 'apache/iceberg-python' || github.event_name == 'workflow_dispatch' # Run on schedule for apache repo, or on manual dispatch from any repo
runs-on: ubuntu-latest
runs-on: ubuntu-slim
outputs:
VERSION: ${{ steps.set-version.outputs.VERSION }}
steps:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pypi-build-artifacts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
Expand Down Expand Up @@ -95,7 +95,7 @@
path: ./wheelhouse/*

pypi-merge-artifacts:
runs-on: ubuntu-latest
runs-on: ubuntu-slim
needs:
- pypi-build-artifacts
steps:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/python-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
Expand Down Expand Up @@ -118,7 +118,7 @@
python-ci-required:
if: ${{ always() }}
needs: [lint-and-unit-test, windows-unit-test, cibw-dev-env-smoke-test]
runs-on: ubuntu-latest
runs-on: ubuntu-slim
steps:
- name: Verify Python CI jobs succeeded
env:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/python-integration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
Expand Down Expand Up @@ -152,7 +152,7 @@
include-hidden-files: true

integration-coverage-report:
runs-on: ubuntu-latest
runs-on: ubuntu-slim
needs: [integration-test, integration-test-s3, integration-test-adls, integration-test-gcs]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand All @@ -178,7 +178,7 @@
python-integration-required:
if: ${{ always() }}
needs: [integration-test, integration-test-s3, integration-test-adls, integration-test-gcs, integration-coverage-report]
runs-on: ubuntu-latest
runs-on: ubuntu-slim
steps:
- name: Verify Python integration jobs succeeded
env:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/python-release-docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ concurrency:

jobs:
docs:
runs-on: ubuntu-latest
runs-on: ubuntu-slim
permissions:
contents: write

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/python-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ permissions:

jobs:
validate-inputs:
runs-on: ubuntu-latest
runs-on: ubuntu-slim
outputs:
VERSION: ${{ steps.validate-inputs.outputs.VERSION }} # e.g. 0.8.0
RC: ${{ steps.validate-inputs.outputs.RC }} # e.g. 1
Expand Down Expand Up @@ -96,7 +96,7 @@ jobs:
STEPS_VALIDATE_INPUTS_OUTPUTS_RC: ${{ steps.validate-inputs.outputs.RC }}

validate-library-version:
runs-on: ubuntu-latest
runs-on: ubuntu-slim
needs:
- validate-inputs
steps:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/stale.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ permissions:
jobs:
stale:
if: github.repository_owner == 'apache'
runs-on: ubuntu-latest
runs-on: ubuntu-slim
steps:
- uses: actions/stale@4391f3da665fdf50b6810c1a66712fb9ba21aa93 # v11.0.0
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/svn-build-artifacts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ jobs:
path: ./wheelhouse/*

svn-merge-artifacts:
runs-on: ubuntu-latest
runs-on: ubuntu-slim
needs:
- svn-build-artifacts
steps:
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/zizmor.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ on:

permissions: {}

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
zizmor:
name: Run zizmor 🌈
Expand Down
4 changes: 2 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,10 +165,10 @@ test-notebook: ## Run notebook tests (pyiceberg_example and spark_integration_ex
##@ Documentation

docs-serve: ## Serve local docs preview (hot reload)
uv run $(PYTHON_ARG) --group docs mkdocs serve -f mkdocs/mkdocs.yml --livereload
uv run $(PYTHON_ARG) --no-default-groups --group docs mkdocs serve -f mkdocs/mkdocs.yml --livereload

docs-build: ## Build the static documentation site
uv run $(PYTHON_ARG) --group docs mkdocs build -f mkdocs/mkdocs.yml --strict
uv run $(PYTHON_ARG) --no-default-groups --group docs mkdocs build -f mkdocs/mkdocs.yml --strict

# ========================
# Experimentation
Expand Down
Loading