Skip to content

Reduce IT test times - #1178

Open
gagan16k wants to merge 8 commits into
gardener:masterfrom
gagan16k:mcm_it
Open

gagan16k wants to merge 8 commits into
gardener:masterfrom
gagan16k:mcm_it

Conversation

@gagan16k

@gagan16k gagan16k commented Nov 2, 2026

Copy link
Copy Markdown
Member

What this PR does / why we need it:

This PR makes following changes to the integration tests:

  • New function IsMachineTerminatingOrDeleted instead of IsMachineDeleted makes it such that the tests do not wait on Terminating machines where it is not necessary, and the MCM process takes care of it in the background. This speeds up the suite significantly.
  • Reordered preservation specs to more effectively use machines from the MCD pool. This is so that they do not use all running machines from it and other specs have to wait for new machines to process creation and transition to Running.
  • The in-place ginkgo.By steps include the machine/node name (matching the convention already used by the preservation specs), making test output identify which machine each step acted on.
  • cleanTestResources tears down all test machine deployments and machines
  • The "orphaned resources" testcase runs the shared cleanup before querying the cloud provider, instead of deleting a single MCD.
  • Added a script that was used to test if the changes caused any flakiness/ to benchmark time. It could be used to check and reproduce if any run fails on CI.

Additionally, added a change to the machine controller health check code. Previously, any [Pending,InPlaceUpdate, Health] timeouts would have a hardcoded 1 minute re-enqueue time.
Replaced this with an bounded timeout [ShortRetry(5s), 1 Minute]. For the test code, this makes it such that health reconciliations happen much more deterministically and do not wait, in case kwok headbeat gets missed.

New test times:

Requested    : 10 run(s)
Executed     : 10 run(s)
Passed       : 10
Failed       : 0
Pass rate    : 100% (10/10)
Wall-clock   : 106m 10s (includes inter-run overhead)

Runtime per run (passing only):
  best       : 9m 36s
  worst      : 11m 01s
  average    : 10m 37s

Release note:

Reduce test times for IT by parallelizing machine deletion. Also, adjust machine health-check re-enqueue instead of hardcoded interval.

Signed-off-by: Gagan163264 <gagan.surathkal@gmail.com>
Signed-off-by: Gagan163264 <gagan.surathkal@gmail.com>
Signed-off-by: Gagan163264 <gagan.surathkal@gmail.com>
Signed-off-by: Gagan163264 <gagan.surathkal@gmail.com>
Signed-off-by: Gagan163264 <gagan.surathkal@gmail.com>
Signed-off-by: Gagan163264 <gagan.surathkal@gmail.com>
@gagan16k
gagan16k requested a review from a team as a code owner November 2, 2026 15:04
@gardener-prow gardener-prow Bot added do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 2, 2026
@gagan16k

gagan16k commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

/kind enhancement

@gardener-prow gardener-prow Bot added kind/enhancement Enhancement, improvement, extension and removed do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Nov 2, 2026
@elankath

Copy link
Copy Markdown
Member

Thanks for doing this. Looking at this

@elankath elankath self-assigned this Nov 12, 2026
WithArguments(ctx, helpers.McdName, controlClusterNamespace).
Should(gomega.BeTrue())
ginkgo.By("Ensuring the test resources are cleaned up")
c.cleanTestResources(ctx, 900)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels incorrect. There is no error checking for cleanTestResources. Is that intended ? Wont' that cause issues ie cleanup failure might produce false negatives in the orphan test. (at-least I think so)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in ec2dfe1

}
// Re-enqueue to re-check the timeout when it is due, bounded to [ShortRetry, 1m]
// Ensures that small timeouts do not wait too long, while larger timeouts are not polled too frequently.
sleepTime := min(max(timeOutDuration-timeElapsed, time.Duration(machineutils.ShortRetry)), time.Minute)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you move this into the else branch ?

// If timeout has not occurred, re-enqueue the machine
// after a specified sleep time
klog.V(4).Infof("Creation/Health Timeout hasn't occured yet , will re-enqueue after %s", time.Duration(sleepTime))
klog.V(4).Infof("Creation/Health Timeout hasn't occured yet , will re-enqueue after %s", sleepTime)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: occured -> occurred

func (c *IntegrationTestFramework) runControllersLocally() {
ginkgo.By("Starting Machine Controller ")
args := strings.Fields(
// buildMCStartArgs builds the `make start` command used to launch the machine controller locally.

@elankath elankath Nov 12, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lol, yes, we should have done this ages instead of duplicating startup logic - which might also need to change in future depending on environment type.

@@ -0,0 +1,178 @@
#!/usr/bin/env bash

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a FAQ note https://github.com/gardener/machine-controller-manager/blob/master/docs/faq.md#internals on how and when to use this script

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the FAQ note

Comment thread pkg/simulatedprovider/hack/stress.sh Outdated
Usage: $(basename "$0") [-n RUNS] [-o OUT_DIR] [-f FOCUS] [-s] [-v]

-n RUNS Number of times to run the integration test suite (default: ${RUNS})
-o OUT_DIR Directory to write per-run logs and the summary (default: ${BASE_DIR}/<timestamp>)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems some mismatch in usage/default. hack/logs/ is the default for BASE_DIR. It is relative to where it is invoked, not where the script lives.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in ec2dfe1

Signed-off-by: Gagan163264 <gagan.surathkal@gmail.com>
@gardener-prow

gardener-prow Bot commented Nov 12, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from elankath. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Gagan163264 <gagan.surathkal@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/enhancement Enhancement, improvement, extension size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants