Skip to content

Fix flaky appsec logging_memory_limit.phpt on PHP 7.0-7.2 - #4398

Draft
Leiyks wants to merge 1 commit into
masterfrom
leiyks/fix-appsec-logging-memory-limit-test
Draft

Leiyks wants to merge 1 commit into
masterfrom
leiyks/fix-appsec-logging-memory-limit-test

Conversation

@Leiyks

@Leiyks Leiyks commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Description

logging_memory_limit.phpt (#4361) failed randomly on PHP 7.0–7.2. The testing mlog() clamped messages to 500 bytes (and %.*s precision is capped at 500 before PHP 8.1). On 7.0–7.2 the resulting php_verror() buffer fits a small-bin slot, so hitting the memory limit depended on leftover heap state, which varies with the CI job's environment variables and the test path.

The test now logs a 4 KiB message, which needs a new page on every PHP version, and the testing mlog() formats it with %s so nothing truncates it. Assertions are unchanged.

Reviewer checklist

  • Test coverage seems ok.
  • Appropriate labels assigned.

The test fills every heap page and expects mlog() to hit the memory
limit inside php_verror(). The testing mlog() clamped messages to 500
bytes, and %.*s precision is capped at 500 before PHP 8.1 anyway. PHP
7.3+ grows the format buffer in whole pages, but 7.0-7.2 grow it by
len + 128, so the buffer came from a small bin. The limit was then only
hit when that bin had no free slot, which depends on the environment
and test path.

Format the testing message with %s and log 4 KiB. That is above the
largest small bin (3 KiB), so php_verror() needs new pages and reaches
the limit on every version.
@datadog-official

datadog-official Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 76.51% (+0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 91d5378 | Docs | View more details | Give us feedback!

@pr-commenter

pr-commenter Bot commented Oct 28, 2026

Copy link
Copy Markdown

Benchmarks [ appsec ]

Benchmark execution time: 2026-10-28 17:52:08

Comparing candidate commit 91d5378 in PR branch leiyks/fix-appsec-logging-memory-limit-test with baseline commit 8caedba in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants