Repository navigation
Reco: fix fetch-incidents skipping alerts during bursts - #3
Open
RecoLabs-YaronBratspiess wants to merge 464 commits into
Open
RecoLabs-YaronBratspiess wants to merge 464 commits into
RecoLabs-YaronBratspiess wants to merge 464 commits into
Conversation
…5879) * Prisma AIRS 1.0.1: brand spelling fix, DLP UX improvements (demisto#45850) * Prisma AIRS 1.0.1: brand spelling fix, DLP UX improvements, DLP Comprehensive test - Correct pack brand spelling "Prisma AIRs" -> "Prisma AIRS" across all display text and docs (identifiers/paths deliberately unchanged) - Runtime: dlp-dictionaries-create/-replace region_name is now a predefined pick list of valid tenant region labels (default United States), avoiding the HTTP 400 the API returns for AWS-style codes like us-west-2 - Runtime: dlp-patterns-delete surfaces an actionable error when the pattern is still referenced by an active data profile (HTTP 400) - Runtime: document the server-enforced 32-char DLP data-profile name limit (spec wrongly says 64) - Pack README: refresh Known Limitations with the accurate DLP constraints - Register the DLP Comprehensive test playbook (conf.json + Runtime tests list) - Bump pack to 1.0.1 with release notes Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Update Packs/PaloAltoNetworks_Prisma_AIRs/ReleaseNotes/1_0_1.md Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> * Update Packs/PaloAltoNetworks_Prisma_AIRs/ReleaseNotes/1_0_1.md Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> * Update Packs/PaloAltoNetworks_Prisma_AIRs/ReleaseNotes/1_0_1.md Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> * Update Packs/PaloAltoNetworks_Prisma_AIRs/ReleaseNotes/1_0_1.md Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> * Prisma AIRS 1.0.1: address pack reviewer feedback - Runtime: split the long DLP profile-name and dictionary region_name argument descriptions into shorter sentences (yml + README) - Runtime: use a period instead of a semicolon in the dlp-patterns-delete command description (yml + README) - ApiModule: rewrite the description in active voice describing what the module provides (yml + README) - Release notes: reword the 1.0.1 bullets per reviewer wording Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: epartington <epartington@paloaltonetworks.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> * Bump API modules pack version --------- Co-authored-by: epartington <epartington@users.noreply.github.com> Co-authored-by: epartington <epartington@paloaltonetworks.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> Co-authored-by: Kamal Qarain <kqarain@paloaltonetworks.com>
…ed after revert PR demisto#45814 (demisto#45836) This restores the identity.* migration for CiscoThousandEyes, ForeScoutCounterACT, IBMSecurityVerify, OneLogin, and ProofpointObserveIT that was unintentionally reverted by demisto#45814 (Revert ModelingRules changes across 88 packs). Rebuilt on top of current master; each pack_metadata.json bumped one patch beyond master's post-revert version. Modeling rule ids/names match the previously validated fix (clears BC107/MR108).
* CRTX-271525: add identity.* to Orca modeling rules * CRTX-271525: add identity.* to Netskope modeling rules * Modified rules names and ids in yml * CRTX-271525: add identity.* to NetBox modeling rules * Modified rule id for netbox * CRTX-271525: add identity.* to MySQLEnterprise modeling rules * Modified mysql id and name in yml * CRTX-271525: move AzureSecurityCenter and OracleCloudInfrastructure identity.* modeling rules to force-merge branch (IDs use legacy lowercase format) * CRTX-271525: move NGINXWebServer and InfobloxBloxOne identity.* modeling rules to force-merge branch (IDs use legacy format without ModelingRule suffix) * CRTX-271525: move CiscoASA and ManageEngine-ADManager identity.* modeling rules to force-merge branch (IDs use legacy lowercase format) * Modified yml version and names for packs * Modified RN * bump CiscoASA version to 2.0.8 and add release note * bump versions and add release notes for all identity.* modeling rule packs * Bump pack from version Orca to 3.1.5. --------- Co-authored-by: Content Bot <bot@demisto.com>
demisto#45881) * Update default XDM Mapping Contribution Reviewer in content_roles.json * Update content_roles.json
…demisto#45857) * CRTX-276239 - re-open vCenter vcsa-audit pid mapping fix (to_integer) Re-open PR demisto#45714 after it was reverted by demisto#45814. Re-applies the fix changing xdm.source.process.pid and xdm.target.process.pid from to_number() to to_integer() so 7-digit pids (>= 1,000,000) populate the INTEGER columns instead of landing as NULL. Pack: VMwareVCenter (1.0.26). * check of MR109 validation * trigger CI * added mr109 validation * added fix for source_process_pid
* gcp_compute_Operations * ai review * cr fixes * docs: apply doc review wording suggestions to GCP operations descriptions * add region handle * doc review
) * CRTX-271521: add identity.* to modeling rules (PR 7 of 7) Part of the XDM user.* -> identity.* alignment (epic CRTX-249617). For each of the 6 packs in batch 7, adds a new _3_19 modeling rule folder (fromversion 8.15.0) that maps xdm.<prefix>.identity.<field> alongside every existing xdm.<prefix>.user.<field> assignment. The previous modeling rule folder is capped with toversion 8.14.9 so existing tenants are unaffected. 13 assignments across: MicrosoftGraphSecurity (4), SymantecCloudSecureWebGateway (3), BarracudaWAF (2), CyberArkEPV (2), F5ASM (1), MicrosoftAdvancedThreatAnalytics (1) Schemas are copied unchanged (they describe raw source fields, not XDM). * CRTX-271521: address review - toversion 8.14.0 and normalize rule id/name - Cap previous modeling rule folders at toversion 8.14.0 (was 8.14.9). - Normalize the new _3_19 modeling rules to the <Pack>_ModelingRule / <Pack> Modeling Rule convention for id and name. - Realign the release note headings with the updated rule names. * Bump pack from version MicrosoftGraphSecurity to 2.5.8. * CRTX-271520 - correct rule version token 3_19 -> 2_11 3.19 is not a Cortex XSIAM version. The token for fromversion 8.15.0 is 2.11, following Salesforce_2_10 / SlackModelingRules_2_10 (8.14.0) and VMwareVcenter_2_9 (8.13.0) already in master. Renames the modeling rule folder and its .xif / .yml / _schema.json, and corrects the release note. The version window (fromversion 8.15.0, toversion 8.14.0 on the previous folder) is unchanged - only the label was wrong. * Bump pack from version MicrosoftGraphSecurity to 2.6.1. * Bump pack from version MicrosoftGraphSecurity to 2.6.2. * CRTX-271521: remove CyberArkEPV and MicrosoftGraphSecurity changes (moved to PR8) * CRTX-271521: remove BarracudaWAF, F5ASM and SymantecCloudSecureWebGateway changes (moved to PR8) These three packs' identity.* modeling rule additions are moved out of PR7 and into the force-merge branch crtx-271521-idenitity-pr8 (demisto#45805). * CRTX-271521 - fix run-validations: align MicrosoftAdvancedThreatAnalytics_2_11 id/name to base rule (BC107) + RN header (RN107) --------- Co-authored-by: Content Bot <bot@demisto.com> Co-authored-by: sfish99 <sfish@paloaltonetworks.com> Co-authored-by: sharonfi99 <147984773+sharonfi99@users.noreply.github.com>
) * CRTX-271521: add identity.* to modeling rules (PR 6 of 7) Part of the XDM user.* -> identity.* alignment (epic CRTX-271520). Adds a new _3_19 modeling rule folder (fromversion 8.15.0) per pack that maps xdm.<prefix>.identity.<field> alongside every existing xdm.<prefix>.user.<field> assignment. The previous folder is capped with toversion 8.14.0. AlibabaActionTrail (6), F5BigIPAWAF (4), ClearswiftDLP (3), MicrosoftWindowsAMSI (2), BeyondTrustEPM (1). * CRTX-271520 - correct rule version token 3_19 -> 2_11 3.19 is not a Cortex XSIAM version. The token for fromversion 8.15.0 is 2.11, following Salesforce_2_10 / SlackModelingRules_2_10 (8.14.0) and VMwareVcenter_2_9 (8.13.0) already in master. Renames the modeling rule folder and its .xif / .yml / _schema.json, and corrects the release note. The version window (fromversion 8.15.0, toversion 8.14.0 on the previous folder) is unchanged - only the label was wrong. * fix for yml naming and RN files * CRTX-271521: remove ClearswiftDLP changes (moved to PR8) * CRTX-271521 - fix run-validations: BeyondTrustEPM add 'platform' marketplace (BA132/BA130) + fix RN header name (RN107/RN114) * CRTX-271521 - fix BA131: revert platform marketplace + remove supportedModules from BeyondTrustEPM _2_11 MR * added supported modules in beyondtrustEPM pack metadata * added platfrom to pack_metadata --------- Co-authored-by: sfish99 <sfish@paloaltonetworks.com> Co-authored-by: sharonfi99 <147984773+sharonfi99@users.noreply.github.com>
* init * more * remove stuf * more * more * remove stuf * Bump pack from version CloudIncidentResponse to 1.0.36. * Bump pack from version CortexXDR to 6.3.34. * Bump pack from version CloudIncidentResponse to 1.0.37. * more * rn * rn * readme * rn * more * rn * RN * pre-commit fixes * validation fixes * validation fixes * pre-commit fixes * docker --------- Co-authored-by: Content Bot <bot@demisto.com>
…to#45886) * CIAC-17212: fix nightly run-pre-commit timeout in streaming+threaded tests The new streaming+multiple_threads tests spun up real 20-worker ThreadPoolExecutors and the inflight-bounds test released its blocked workers only AFTER send_data_to_xsiam returned. Since the producer blocks in the back-pressure loop waiting on in-flight futures, that release was unreachable -> deadlock that only unwound via per-worker timeout. On py2.7 (no cancel_futures) under the loaded nightly runner this consumed the 9h job budget. Fix (tests only): the bounds test now self-releases from inside the worker once the bound is reached (no post-return set, no fixed sleeps); all real-thread tests use tiny pools (NUM_OF_WORKERS patched to 2-4) and small data; error-path tests drain their executors so no worker thread outlives the test. Verified: py2.7 whole file 859 passed /201 skipped in 59s (was timing out), streaming_threaded 11 passed in <1s, 15x stress 0 fails. * CIAC-17212: address code-quality bot nits in streaming+threaded tests - Resolve mixed-import (import + import-from) of CommonServerPython in the backpressure, assets, and dispatch tests: drop 'from CommonServerPython import BaseClient' and use csp.BaseClient consistently. - Replace empty 'except Exception: pass' in the backpressure drain loop with an explanatory comment + continue (behavior unchanged: ignore drained-future errors, primary failure is asserted above).
… fetch pace (demisto#45877) * Improve MDA Event Collector fetch pace and reliability * Address AI reviewer comments * Add outgoing API call debug logs
…#45839) * CRTX-271524: re-add identity.* to modeling rules (PR 5 of 7) - reopened after revert PR demisto#45814 Restores the identity.* migration for GoogleChrome, HPEArubaClearPass, and TrendMicroTippingPoint after it was unintentionally reverted by demisto#45814. CiscoCatalyst, ForcepointSWG, and Infoblox are intentionally excluded (moved to the crtx-271524-identity-force-merge-packs branch, unaffected by this revert). pack_metadata.json bumped one patch beyond master's post-revert version. * CRTX-271524: bump packs to minor version for new identity.* modeling rule (PR 5 of 7) GoogleChrome 1.0.12 -> 1.1.0 HPEArubaClearPass 1.0.39 -> 1.1.0 TrendMicroTippingPoint 1.0.8 -> 1.1.0 Adding a new modeling rule version is a feature addition and requires a minor version bump per repo versioning guidelines, not a patch bump.
* CofenseTriage Release 2.2.0 * Updated as per provided comments * Updated argument descriptions as per provided comments --------- Co-authored-by: Crest Data <60967033+crestdatasystems@users.noreply.github.com> Co-authored-by: crestdatasystems <crestdatasystems@users.noreply.github.com>
…#45837) * CRTX-271524: re-add identity.* to modeling rules (PR 3 of 7) - reopened after revert PR demisto#45814 Restores the identity.* migration for Forcepoint, HPESwitch, LookoutMobileEndpointSecurity, and Vectra_AI after it was unintentionally reverted by demisto#45814. OktaOAG is intentionally excluded (moved to the crtx-271524-identity-force-merge-packs branch, unaffected by this revert). pack_metadata.json bumped one patch beyond master's post-revert version. * CRTX-271524: bump packs to minor version for new identity.* modeling rule (PR 3 of 7) Forcepoint 1.1.7 -> 1.2.0 HPESwitch 1.0.9 -> 1.1.0 LookoutMobileEndpointSecurity 1.0.9 -> 1.1.0 Vectra_AI 2.1.9 -> 2.2.0 Adding a new modeling rule version is a feature addition and requires a minor version bump per repo versioning guidelines, not a patch bump.
…#45838) * CRTX-271524: re-add identity.* to modeling rules (PR 4 of 7) - reopened after revert PR demisto#45814 Restores the identity.* migration for AtlassianConfluenceCloud, MicrosoftDefenderAdvancedThreatProtection, PrismaSaasSecurity, and SaviyntEIC after it was unintentionally reverted by demisto#45814. AristaSwitch and IronPort are intentionally excluded (moved to the crtx-271524-identity-force-merge-packs branch, unaffected by this revert). pack_metadata.json bumped one patch beyond master's post-revert version. * CRTX-271524: bump packs to minor version for new identity.* modeling rule (PR 4 of 7) AtlassianConfluenceCloud 1.2.4 -> 1.3.0 MicrosoftDefenderAdvancedThreatProtection 1.20.48 -> 1.21.0 PrismaSaasSecurity 2.1.4 -> 2.2.0 SaviyntEIC 1.0.11 -> 1.1.0 Adding a new modeling rule version is a feature addition and requires a minor version bump per repo versioning guidelines, not a patch bump. * Bump pack from version MicrosoftDefenderAdvancedThreatProtection to 1.21.0. --------- Co-authored-by: Content Bot <bot@demisto.com>
…#45840) * CRTX-271524: re-add identity.* to modeling rules (PR 6 of 7) - reopened after revert PR demisto#45814 Restores the identity.* migration for Bitbucket, DellEMCUnity, Exabeam, and GenetecSecurityCenter after it was unintentionally reverted by demisto#45814. FortiGate and BloodHoundEnterprise are intentionally excluded (moved to the crtx-271524-identity-force-merge-packs branch, unaffected by this revert). pack_metadata.json bumped one patch beyond master's post-revert version. * CRTX-271524: bump packs to minor version for new identity.* modeling rule (PR 6 of 7) Bitbucket 1.1.3 -> 1.2.0 DellEMCUnity 1.0.10 -> 1.1.0 Exabeam 2.4.15 -> 2.5.0 GenetecSecurityCenter 1.0.12 -> 1.1.0 Adding a new modeling rule version is a feature addition and requires a minor version bump per repo versioning guidelines, not a patch bump.
…#45841) * CRTX-271524: re-add identity.* to modeling rules (PR 7 of 7) - reopened after revert PR demisto#45814 Restores the identity.* migration for IBMStorageScale, MenloSecurity, OnePassword, RSASecureID, SemperisDSP, and WithSecure after it was unintentionally reverted by demisto#45814. Includes the previously fixed IBMStorageScale XQL bug (identity assignment no longer nested inside the preceding open if()). pack_metadata.json bumped one patch beyond master's post-revert version. * CRTX-271524: bump packs to minor version for new identity.* modeling rule (PR 7 of 7) IBMStorageScale 1.0.8 -> 1.1.0 MenloSecurity 1.0.6 -> 1.1.0 OnePassword 1.0.16 -> 1.1.0 RSASecureID 1.0.8 -> 1.1.0 SemperisDSP 1.0.9 -> 1.1.0 WithSecure 1.0.17 -> 1.1.0 Adding a new modeling rule version is a feature addition and requires a minor version bump per repo versioning guidelines, not a patch bump.
…to#45697) * fix(JiraCreateIssueExample): fix non-ASCII custom field drop and add mixed-args validation * update * deprecate(JiraCreateIssueExample): deprecate script in favor of jira-create-issue * revert(JiraCreateIssueExample): drop bug-fix code, keep deprecation only * Bump pack from version Jira to 3.3.43. --------- Co-authored-by: Content Bot <bot@demisto.com>
… from returned data (demisto#45907) * Orca: fix fetch-incidents losing alerts by taking the fetch watermark from returned data (demisto#45589) * INTG-4980: Take SOAR fetch watermark from returned alerts instead of clock time The fetch-incidents watermark was set to the current clock time after each cycle, permanently skipping alerts that became visible between the data read and the clock sample (e.g. due to replication delay). The watermark is now the max last_sync of the alerts actually received; an empty cycle keeps it in place. Boundary alerts (last_sync == watermark) are re-fetched next cycle and deduplicated by alert id unless their last_sync changed. * INTG-4980: Fix mypy type narrowing in advance_cycle_sync_state * INTG-4980: Address review - fix boundary dedup dropping alerts without last_sync An alert with no last_sync whose id was absent from the boundary map matched None == None and was silently dropped whenever a boundary set existed. Only a recorded (non-None) boundary entry counts as a match now. Also include the parse error in the last_sync log line and fix release-note command styling. --------- Co-authored-by: Kamal Qarain <45042524+kamalq97@users.noreply.github.com> * Update release notes and docker image --------- Co-authored-by: Viktor Poltorak <90267674+viktor-orca@users.noreply.github.com> Co-authored-by: Kamal Qarain <45042524+kamalq97@users.noreply.github.com> Co-authored-by: Kamal Qarain <kqarain@paloaltonetworks.com>
…o#45915) * Add Haseen Threat Intel pack - New partner feed integration ingesting Haseen's STIX 2.1 threat intel bundle - Id-keyed delta fetch with persistent watermark (integration context) to eliminate hourly re-reporting of unchanged indicators - SDO-aware STIX parsing: malware/attack-pattern/threat-actor surfaced as first-class indicators with canonical Cortex scores and relationship graph - Token query-param auth with optional Basic Auth credentials - Generic delta-in-feed playbook (Stage -> Hunt -> Block -> Close template) - Unit tests (17) plus SDK validate-clean pack * Address marketplace review: type hints, Cortex XSOAR prefix, drop hardcoded instance name - Add Client type hints to fetch_indicators_command/test_module and -> None to main - Add from __future__ import annotations for forward refs - ruff format: collapse implicit string concat, fix slice spacing - Replace bare 'XSOAR' with 'Cortex XSOAR' across pack docs - Remove hardcoded instance name in delta-in-feed playbook (sourceInstances) * Use sourceBrands (not sourceInstances) for delta-in-feed indicator staging The staging SearchIndicator query scopes by integration source instance. Replace the instance-name-based search with sourceBrands per review, so the query is: (sourceBrands:"Haseen Threat Intel") and (sourcetimestamp:>="24 hours ago" or lastSeen:>="24 hours ago"). * Address AI-reviewer comments: finalize playbook YAML/README, config sectioning, param types/descriptions, description header * Fix pylint W9011 (single return_error) and mypy type-narrowing at 261/577 * Address review feedback for Haseen Threat Intel pack - Add "do not append ?token" warning to URL param additionalinfo - Flip support from partner to community (author email unchanged) - Replace real STIX bundle with synthetic fixture (RFC-5737 IPs, example.com, fake hashes/timestamps); update test assertions - Restructure pack README: lead with "What does this pack do?" + capability list, move Haseen/DOI/M2M background below, use "Cortex XSOAR" prefix and US spelling consistently - Embed playbook screenshot + M2M image via doc_files/ relative paths - Rename playbook to "Haseen - Delta Feed - Generic" (match display name) - Add missing task descriptions; fix PB121 incident.id reference - Reconcile playbook README with task 47/50/12/52/53 descriptions (remove StixIndicators.value, justify 24h severity window + 500/500/1000 search limits) - Remove ReleaseNotes draft + IM111 ignore; resize image to 120x50 - mypy/pylint/ruff clean; 17/17 tests; validate + format pass --------- Co-authored-by: Mohamed Abdelrahman <60690190+Mohamed-Abdelrahman-7@users.noreply.github.com> Co-authored-by: Mohamed Abdelrahman <Mohamed-Abdelrahman-7@users.noreply.github.com> Co-authored-by: Moshe Eichler <78307768+MosheEichler@users.noreply.github.com>
…isto#45831) * CRTX-267799 - Zscaler Internet Access modeling rule field mapping additions * CRTX-267799 - re-trigger CI (empty commit after content-private branch cleanup)
* CRTX-229181 - add iZOOlogic modeling rule for incidents * change the modeling rule directory name * modeling rule file names were changed * CRTX-229181 - re-trigger CI (empty commit after content-private branch cleanup)
* Added modeling rules for koi * Modified files names * Modified modeling rule * Modified modeling rule * modified modeling rules, readme file and schema * Added RN * modify readme
) * CRTX-271521: add identity.* to modeling rules (PR 5 of 7) Part of the XDM user.* -> identity.* alignment (epic CRTX-271520). Adds a new _3_19 modeling rule folder (fromversion 8.15.0) per pack that maps xdm.<prefix>.identity.<field> alongside every existing xdm.<prefix>.user.<field> assignment. The previous folder is capped with toversion 8.14.0. RiverbedFlowGateways (8), Oracle_IAM (3), DeCYFIR (3), AzureWAF (1), BluecatAddressManager (1). * Bump pack from version BluecatAddressManager to 1.1.23. * Bump pack from version DeCYFIR to 1.1.7. * CRTX-271520 - correct rule version token 3_19 -> 2_11 3.19 is not a Cortex XSIAM version. The token for fromversion 8.15.0 is 2.11, following Salesforce_2_10 / SlackModelingRules_2_10 (8.14.0) and VMwareVcenter_2_9 (8.13.0) already in master. Renames the modeling rule folder and its .xif / .yml / _schema.json, and corrects the release note. The version window (fromversion 8.15.0, toversion 8.14.0 on the previous folder) is unchanged - only the label was wrong. * Bump pack from version DeCYFIR to 1.1.8. * Bump pack from version DeCYFIR to 1.1.9. * CRTX-271521: remove Oracle_IAM and DeCYFIR changes (moved to PR8) * fix for yml files and RN * CRTX-271521 - fix run-validations: add 'platform' marketplace to RiverbedFlowGateways (BA132/BA130) * CRTX-271521 - fix BA131: revert platform marketplace + remove supportedModules from RiverbedFlowGateways _2_11 MR * added supported modules into riverbed flow pack metadata * added platfrom to pack_metadata: * RiverbedFlowGateways _2_11: split packed identity_type mappings onto separate lines --------- Co-authored-by: Content Bot <bot@demisto.com> Co-authored-by: sfish99 <sfish@paloaltonetworks.com> Co-authored-by: sharonfi99 <147984773+sharonfi99@users.noreply.github.com>
) * CRTX-271521: add identity.* to modeling rules (PR 1 of 7) Part of the XDM user.* -> identity.* alignment (epic CRTX-271520). Adds a new _3_19 modeling rule folder (fromversion 8.15.0) per pack that maps xdm.<prefix>.identity.<field> alongside every existing xdm.<prefix>.user.<field> assignment. The previous folder is capped with toversion 8.14.0. VMwareVCenter (42) - the live folder VMwareVcenter_2_9; the remaining 29 user.* assignments live in VMwareVcenter_1_3 which is already capped at toversion 8.12.0 and therefore out of scope. * CRTX-271520 - correct rule version token 3_19 -> 2_11 3.19 is not a Cortex XSIAM version. The token for fromversion 8.15.0 is 2.11, following Salesforce_2_10 / SlackModelingRules_2_10 (8.14.0) and VMwareVcenter_2_9 (8.13.0) already in master. Renames the modeling rule folder and its .xif / .yml / _schema.json, and corrects the release note. The version window (fromversion 8.15.0, toversion 8.14.0 on the previous folder) is unchanged - only the label was wrong. * Bump pack from version VMwareVCenter to 1.0.25. * Bump pack from version VMwareVCenter to 1.0.26. * check MR109 validation * trigger CI * added mr109 validation * check for MR109 * revert for config files * revert for config files * Bump pack from version VMwareVCenter to 1.0.27. * VMwareVCenter _2_11: rebase modeling rule on master _2_9 base, keep identity mappings - Align vcsa_audit_format process.pid casts with master (to_integer) - Restore common_fields xdm.source.process.pid = to_integer(process_id) from master - Preserve all xdm.*.identity.* mappings * changes to yml file * updated RN --------- Co-authored-by: sfish99 <sfish@paloaltonetworks.com> Co-authored-by: sharonfi99 <147984773+sharonfi99@users.noreply.github.com> Co-authored-by: Content Bot <bot@demisto.com>
* CRTX-265425 - Huawei FW support USGxxxxF parsing * Update Packs/HuaweiFW/ReleaseNotes/1_0_8.md Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> --------- Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com>
…d in get_cloud_credentials (demisto#45880) * str account_id * rn * platform rn * ai review * Bump pack from version ApiModules to 2.4.52. * gcp version * Bump pack from version GCP to 1.16.0. --------- Co-authored-by: Content Bot <bot@demisto.com>
* Updated Docker Images for ZeroFox. * Updated Release Notes. * Bump pack from version ZeroFox to 1.4.5. * fux UT * Bump pack from version ZeroFox to 1.4.6. * Apply suggestion from @shmuel44 --------- Co-authored-by: content-bot <content-bot@users.noreply.github.com> Co-authored-by: Content Bot <bot@demisto.com> Co-authored-by: shmuel44 <skroizer@paloaltonetworks.com> Co-authored-by: Shmuel Kroizer <69422117+shmuel44@users.noreply.github.com>
* Fix / XSUP-77132 / Rasterize retry recursion tab leak * Bump pack from version rasterize to 2.1.51. * fix tests * ai review fixes --------- Co-authored-by: Content Bot <bot@demisto.com>
* fix bug * cr fixes * rn * Bump pack from version Mimecast to 3.1.3. * Revert all Mimecast pack changes to origin/master state * Mimecast Event Collector v2: use SIEM cursor within a single fetch cycle only - Add is_valid_cursor helper (invalid when all event-type offsets are 0-0) - Validate cursor before every use in get_siem_events - fetch_siem_events: persist only latest date + IDs, clamp date to now-23h beyond 24h - Migrate existing persisted cursors (reuse if valid, else fall back to date) - Persist clamped start date even when no events are returned - Add tests and release notes (3.1.3) * cr * Update Packs/Mimecast/ReleaseNotes/3_1_4.md Co-authored-by: JuliaH-TW <jhornik@paloaltonetworks.com> * add end date * Mimecast: is_valid_cursor considers :<page>:<token> suffix progress A SIEM cursor with all 0-0 offsets but a non-degenerate :<page>:<token> suffix (e.g. :3:<token>) carries real pagination progress and was wrongly classified invalid, causing early loop break and dropped pages. Now treat such cursors as valid; only the fully degenerate [...]:0:null sentinel is invalid. Updates tests for suffix-progress cases and the end_date kwarg. Addresses PR review comment by DanielTal87. * improve * add log * update the docker image * cr fixes * revert bad tests --------- Co-authored-by: Content Bot <bot@demisto.com> Co-authored-by: Daniel Tal <87daniel.tal@gmail.com> Co-authored-by: JuliaH-TW <jhornik@paloaltonetworks.com>
…ments (demisto#46340) * GoogleGemini ContentPack Update To Include Additional Generation Arguments (demisto#46028) * updated GoogleGemini Co-authored-by: Copilot <copilot@github.com> * added other new models * ran release notes update * Apply suggestions from code review pt 1 Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> Co-authored-by: julieschwartz18 <91824591+julieschwartz18@users.noreply.github.com> * Apply suggestions from code review pt 2 Co-authored-by: julieschwartz18 <91824591+julieschwartz18@users.noreply.github.com> * Implementing some of the AI suggestions * Update README.md with XSIAM specifics * Update GoogleGemini_description.md * precommit checks plus fixed test * linter changes * Updated command to allow temperature and input token limits, fixed bug in usage of arg_to_number for temperature, updated model list to current stable text models from 2.5 onward * readme fix * Fixed contributor naming convention and version mismatch * fixing readme and yml * fixing readme typo * added output token control * added additional test coverage * fixing old model ref in test suite * fixed imports * Updated fallback for temperature to client level (instance) * Apply batched suggestions from code review Accepting AI suggestions Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> * addressing code review * ran release note update * addressing return shortcoming * ruff formatter update * resolving mismatch in release version/notes / removed unneccesary details from description * doc update * camelcase * readme --------- Co-authored-by: breimers <breimers@deloitte.com> Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> Co-authored-by: julieschwartz18 <91824591+julieschwartz18@users.noreply.github.com> * fix yml fields * fix after Tech review * update README --------- Co-authored-by: Bradley Reimers <47170340+breimers@users.noreply.github.com> Co-authored-by: breimers <breimers@deloitte.com> Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> Co-authored-by: julieschwartz18 <91824591+julieschwartz18@users.noreply.github.com> Co-authored-by: lironcohen272 <lircohen@paloaltonetworks.com> Co-authored-by: test-split <test@test.com>
* Set ask/email tasks to use built-in mail sender --------- Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> Co-authored-by: JuliaH-TW <jhornik@paloaltonetworks.com>
… incident layouts (demisto#46343) * Fix/device security vulnerability fetch duplicates and enhancement to incident layouts (demisto#46225) * Enhance Device Security alert layout and field mappings * fix: device security fetching duplicates incidents * Remove embedded Python code from Device Security YAML. * Removed unintended changes * Updated with intended excludefromtightlycoupled * Update Packs/PaloAltoNetworks_DeviceSecurity/Integrations/PaloAltoNetworks_DeviceSecurity/README.md Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> * Fix Device Security release note prefixes and parameter formatting * Align Device Security pack version to 1.0.2 * Fix Device Security 1.0.2 release note prefixes and entity formatting and added unit tests for scripts * Fix: marketplace-ai-reviewer Fixes * Fix: marketplace-ai-reviewer Fixes password field * Fix Device Security credential parameter handling * Fix Device Security script error handling, metadata, and release notes * Fix Device Security creation metadata and bump pack version to 1.1.0 * Fix Device Security pre-commit failures in metadata, typing, and test formatting * Refactor client_secret handling to simplify extraction and update test cases for consistency * Add display label for Client Secret in configuration section * Updated the Vulnerability Layout and Mappings, also added an Automation to facilitate the Device History Layout * Delete Packs/PaloAltoNetworks_DeviceSecurity/ReleaseNotes/1_0_2.md * Added/Updated release notes with command demisto-sdk update-release-notes -i Packs/PaloAltoNetworks_DeviceSecurity * Updated Formatting * Remove unused 'runScriptAfterUpdate' field from incident fields and update layout for Device Security Alert * Re-added the "runScriptAfterUpdate" key and value as false * Added a check to check if history is coming in as a list to fix rendering the data * Fix Device Security validation issues, add custom site field mappings and Docker images * Align Device Security pack version with the existing 1.1.0 release notes * Formatting Update --------- Co-authored-by: Moshe Eichler <78307768+MosheEichler@users.noreply.github.com> Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> Co-authored-by: siddhantw-metron <siddhant.welinkar@metronlabs.com> * fix validation error * fix after tech review * update readme * fix readme --------- Co-authored-by: parthp-metron <parth.pujari@metronlabs.com> Co-authored-by: Moshe Eichler <78307768+MosheEichler@users.noreply.github.com> Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> Co-authored-by: siddhantw-metron <siddhant.welinkar@metronlabs.com> Co-authored-by: test-split <test@test.com> Co-authored-by: lironcohen272 <lircohen@paloaltonetworks.com>
* poetry files * Apply remove-all-data-source changes onto 1.39.13 * update integration name in release note --------- Co-authored-by: Content Bot <bot@demisto.com> Co-authored-by: yedidyacohenpalo <yecohen@paloaltonetworks.com>
* GoogleCloudSCC Release 2.2.0 (demisto#45532) * GoogleCloudSCC Release 2.2.0 * Updated the endpoints for the supported locations of findings * Updated the argument descriptions and removed the secrets * Updated as per provided comments * Updated Cortex Branding and README according to the suggestions --------- Co-authored-by: Harsh Gami <harsh.gami@crestdata.ai> Co-authored-by: Israel Lappe <79846863+ilappe@users.noreply.github.com> Co-authored-by: crestdatasystems <crestdatasystems@users.noreply.github.com> * Fix RN111 docker release note entry and clean up mixed test imports * Potential fix for pull request finding 'Module is imported with 'import' and 'import from'' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> --------- Co-authored-by: Crest Data <60967033+crestdatasystems@users.noreply.github.com> Co-authored-by: Harsh Gami <harsh.gami@crestdata.ai> Co-authored-by: Israel Lappe <79846863+ilappe@users.noreply.github.com> Co-authored-by: crestdatasystems <crestdatasystems@users.noreply.github.com> Co-authored-by: ilappe <ilappe@paloaltonetworks.com> Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
* Censys Release 3.2.2 * Review chanages --------- Co-authored-by: Crest Data <60967033+crestdatasystems@users.noreply.github.com> Co-authored-by: crestdatasystems <crestdatasystems@users.noreply.github.com>
* TAXIIServer: fix HTTP 500 on TAXII 1 poll service - Wrap SERVER.get_poll_response() in taxii_poll_service with a try/except so any synchronous failure (e.g. unknown collection name, STIX/CybOX serialization error) returns a handled 400 response instead of an unhandled HTTP 500. - Validate that File indicator values are proper MD5/SHA1/SHA256/SHA512 hashes in create_stix_hash_observable before serializing them into the CybOX File Hash field. File paths, file names, and fuzzy/SSDEEP hashes are now rejected with a ValueError (caught per-indicator by the existing streaming loop) instead of being silently serialized as invalid CybOX Hash type 'Other'. - Strip invalid XML control characters (e.g. NULL bytes) from all indicator values before they are embedded in STIX/CybOX observables (IP, Email, Domain, URL, File) via a new sanitize_for_xml helper. - Add regression tests covering the unguarded poll route, hash validation rejection, and control-character sanitization. - Bump pack version to 2.3.7 and add release notes. * Apply batched suggestions from code review Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> * update rn * Potential fix for pull request finding 'Module is imported with 'import' and 'import from'' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * Apply batched suggestions from code review Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> * add traceback import --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com>
* Initial release notes for cspm-em-actions-and-skills * empty * Add validations to commands wrapped by actions * RNs * RNs fix * Bump pack from version Azure to 1.10.1. * Bump pack from version Azure to 1.11.1. * revert AWS changes * Bump pack from version Azure to 1.12.1. * Bump pack from version Azure to 1.13.1. * Sync release notes from GitLab (e1a02821) * Sync release notes from GitLab (17708031) * added secret-ignore * RNs * Bump pack from version Azure to 1.14.1. * Bump pack from version Azure to 1.15.1. * Sync release notes from GitLab (1d719fb1) * Bump pack from version Azure to 1.16.1. * Sync release notes from GitLab (1ffae2de) * remove RNs * Sync release notes from GitLab (77607fda) * RNs * Bump pack from version Azure to 1.17.1. * RNs * Azure RN * Sync release notes from GitLab (22870d6d) * Bump pack from version Azure to 1.19.0. * Sync release notes from GitLab (14e923ee) * Bump pack from version Azure to 1.20.0. * Sync release notes from GitLab (fff4f60b) --------- Co-authored-by: CI Bot <ci@demisto.com> Co-authored-by: Mike Rizzo <mrizzo@paloaltonetworks.com> Co-authored-by: inbal <iapt@paloaltonetworks.com> Co-authored-by: inbalapt1 <164751454+inbalapt1@users.noreply.github.com> Co-authored-by: Content Bot <bot@demisto.com>
* Add Base URL parameter to ZscalerZIdentity integration and update documentation - Introduced a new *Base URL* parameter for configuring the ZIA API base URL. - Updated README, YAML configuration, and description files to reflect the new parameter. - Implemented logic to normalize the base URL in the integration code. - Added unit tests for the new base URL functionality. - Bumped version to 1.5.18 in metadata. * Rename 1_5_18.md to 1_6_0.md * Update pack_metadata.json * Update Packs/Zscaler/Integrations/ZscalerZIdentity/README.md Co-authored-by: cbalnave <cbalnave@paloaltonetworks.com> --------- Co-authored-by: cbalnave <cbalnave@paloaltonetworks.com>
* CRTX-284098_validate_next_token * cr fixes * update rn * update rn * update rn * Bump pack from version Azure to 1.21.0. --------- Co-authored-by: Content Bot <bot@demisto.com>
demisto#46272) * fix(MCPApiModule): flatten nullable types in list-tools schemas (XSUP-78007) * fix(MCPApiModule): leave multi-type schema types unspecified (XSUP-78007) * fix(MCPApiModule): ignore non-string output titles and make release notes customer facing (XSUP-78007) * docs(MCPApiModule): style literal values in release notes (XSUP-78007) * docs(MCPApiModule): name the module in consumer pack release notes (XSUP-78007) * docs(MCPApiModule): reference CRTX-289860 and clarify unknown type wording * refactor(MCPApiModule): make normalize_tool_schemas non-mutating
* Initial release notes for updatecase-old-action-remove * Bump pack from version Core to 3.5.99. --------- Co-authored-by: CI Bot <ci@demisto.com> Co-authored-by: Content Bot <bot@demisto.com>
* NodeZeroMCP Integration (demisto#45679) * Fix linting * update docs * add commands to documentation * add contributors doc * update docs * fix tags * add contributors to NodeZero pack * fix versioning * add auth code * fix formatting and mcp uri * fix auth code validation and doc URL format in NodeZero MCP pack - Add missing auth_code validation to validate_required_params; generate-login-url and test-module are exempt since they run before auth is established - Fix MCP server URLs in description.md and README.md to include /mcp path suffix - Bump pack version to 1.0.1 * Address review comments --------- Co-authored-by: Zach Shute <zachshute@ip-10-0-0-50.ec2.internal> * Apply tech writer suggestions and minor cleanups to NodeZeroMCP * Trigger clean GitLab CI pipeline run --------- Co-authored-by: Zach Shute <zach.shute@horizon3.ai> Co-authored-by: Zach Shute <zachshute@ip-10-0-0-50.ec2.internal> Co-authored-by: ibranski <ibranski@paloaltonetworks.com> Co-authored-by: branskiIdo <i.branski.1@gmail.com>
* implement lookback * ruff fixes * two more updates to the lookback * logs * fix test * change lookback to 10 minutes * revert code written only for testing * more logs * lookback to 11 minutes * cr fixes * cr fixes * fix look back * rn fix * missing events troubleshooting
…ility fetch (demisto#46171) * CIAC-17573: add streaming parse/send helpers to CommonServerPython * Qualys: stream host-list parse and adopt send_assets_to_xsiam * Qualys: stream host-list parse and adopt send_assets_to_xsiam * CIAC-17573: add Base release notes for CommonServerPython streaming helpers * CIAC-17573: rename send_assets_to_xsiam -> send_assets_and_vulnerabilities_to_xsiam (param assets -> data) * CIAC-17573: fix send_data_to_xsiam tests and restore master tests lost in merge * CIAC-18238: resolve send helper naming collision in QualysV2 * CIAC-18238: stream-parse the vulnerabilities fetch response * CIAC-18238: avoid O(N^2) deepcopy in get_detections_from_hosts * CIAC-18238: add release notes and bump version to 3.4.10 * Qualysv2: add fetch-assets memory logging and 1960 retry for vulnerabilities request * Qualysv2: keep only fetch-assets memory logging (strip concurrency-retry refactor) * Qualys VMDR: apply concurrency-limit retry to fetch-vulnerabilities * Qualysv2: stream by-date vulnerabilities send in bounded batches * Qualysv2: convert streamed XML elements directly, skip xml2json round-trip * Qualysv2: bump to 3.4.13 with memory/performance release note * CIAC-17573: address AI reviewer - add type hints to streaming helpers, fix RN entity styling, add decode_content to JSON example * Rename Qualys-local helper * Qualys: bump to 3.4.14 for fetch-assets/vulnerabilities performance improvements * Qualysv2: stream by-date assets send * Qualysv2: single snapshot per vulns cycle; rename publish helper * Remove memory-logging diagnostics from Qualysv2 * Qualysv2: remove streaming feature flags, make streaming the default * Qualysv2: apply assets limit reduction once in the caller * Qualysv2: resume assets fetch from last committed host * Qualysv2: address review comments (log prefixes, tracebacks) * Qualysv2: address review comments (shared send helper, batching, element_to_dict removal, stream error tests), bump to 3.5.0 * Qualysv2: document the assets and vulnerabilities recovery behavior * Qualys VMDR: add retry mechanism when sending assets and vulnerabilities to XSIAM (CIAC-18496) * Qualys VMDR: log successful first-attempt sends to XSIAM (CIAC-18496) * Qualysv2: stream the QIDs path per host batch, remove unused fetch code, extend tests, add temporary memory logs * Qualysv2: count concurrency-limit errors on both requests without halving the limit, count host list timeouts once the limit is 1, add logs * Qualysv2: remove temporary memory logs * Qualys V2: address review comments * Qualys V2: share host batching and last-run reading between assets flows * Qualys V2: add ASSETS_NEXT_TRIGGER and review tests * Qualys V2: fix flaky retry body test * Qualys V2: raise unrecoverable page errors immediately * Qualys V2: silence pylint false positive on ParseError * Qualys V2: release each vulnerabilities batch after it is sent
…o#46361) * Customizable incident name with nozomi alert fields * Customizable incident name with nozomi alert fields bump version release notes * Customizable incident name with nozomi alert fields bump version release notes * Customizable incident name with nozomi alert fields after review fix * Customizable incident name with nozomi after review fixes * Customizable incident name with nozomi after review fixes * Customizable incident name with nozomi after review fixes Co-authored-by: Nicolò <nicolo.ereni@nozominetworks.com>
* Modified parsing rule azure app service * Added RN * Update Packs/AzureAppService/ParsingRules/AzureAppService/AzureAppService.xif Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> --------- Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com>
…n params (demisto#46353) * Fix / XSUP-78452 / Workday: Update API version and enhance integration params * Enhance Workday integration documentation and configuration details * Enhance Workday integrations: update Docker images, add Fetch events parameter, and improve configuration sections * Update Packs/Workday/Integrations/Workday/README.md Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> * Enhance Workday integration: update API version, fix validation errors, and improve documentation * fix validate --------- Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com>
…ersion 2.4.0 (demisto#46365) * docs(PaloAltoDLP): update README and add release notes for version 2.4.0 breaking changes * remove outdated notes on incident attributes for version 2.4.0
* Deprecated expression search command (demisto#46224) * Remove the expression-search command * Add required newline * Deprecate the expression search command * Remove expression search import * Correct typo * Bump minor version * Add return type to normalize_url * Add unit test for normalize_url * Update Packs/ReversingLabs_Titanium_Cloud/ReleaseNotes/2_7_5.md Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> * Raise exception for the deprecated command * Put display and name at the top of the file * Add ReversingLabs to the keywords list * Rename release notes * Add newline at EoF --------- Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> * pre-commit fix * docker and RN * docker and RN * docker and RN --------- Co-authored-by: Mislav Sever <46045160+MislavReversingLabs@users.noreply.github.com> Co-authored-by: Marketplace AI reviewer <svc-mp-ai-reviewer@paloaltonetworks.com> Co-authored-by: Moshe Eichler <meichler@paloaltonetworks.com> Co-authored-by: Moshe Eichler <78307768+MosheEichler@users.noreply.github.com>
…emisto#46103) (demisto#46354) * AbnormalSecurity: make fetch-incidents resumable and time-bounded Each incident type (threats, abuse campaigns, Account Takeover cases) now keeps its own time window in last_run. A window only moves forward once every item in it has become an incident, so a run cut short by max_fetch, the time budget or an error is picked up by the next run instead of losing the rest. - Runs stop starting HTTP calls after a 150s budget and cap each call, including reading the body, at the time left. - A failing item is retried on later runs and skipped with a warning after failing on 3 runs. A 429, 5 failures in a row or the budget stop a type; 401/403 errors the instance and commits nothing. - Windows that overflow one 500-ID list page are halved rather than paginated, because list endpoints page by offset over live data. - 2.4.9's {"last_fetch": ...} migrates without a gap, and last_fetch is still written so a rollback to 2.4.9 resumes safely. * AbnormalSecurity: fetch item details concurrently under a rate limit Detail calls now run on a pool of 4 workers, each with its own Client, since a requests.Session isn't thread-safe. Results are applied to the window on the main thread in list order, so the output doesn't depend on timing. A thread-safe token bucket (2 requests/s, burst of 1) is acquired on every HTTP call in the run, including list calls and each threat detail page, to stay well under the SOAR API's 300 requests/min per customer. The first 429 stops the run; incidents other workers already completed are kept. * AbnormalSecurity: add case fetch mode and fetch tuning params, 2.4.10 - Case fetch mode: "Last modified time" (the default) keeps 2.4.9's behavior, where a modified, resolved or reopened Account Takeover case becomes a new incident; "Created time" emits each case once. Changing the mode restarts the case window from its current start. - Fetch time budget, maximum fetch window, concurrent detail requests and maximum requests per second are advanced params. Unset params fall back to code defaults that match the yml (150s, 60 min, 4, 2/s), since XSOAR doesn't apply yml defaults to existing instances. - max_fetch help text now says it's a total per run across all types. * AbnormalSecurity: simplify fetch, 1-day default window, review fixes - Shared per-run values (client, deadline, window bounds, concurrency, warnings) live in one FetchRun, and fetch_type is split into listing, building a batch and handling a failed list call. Detail calls always go through the executor, also with one worker. - Remove the get_paginated_*_list helpers, which only 2.4.9's fetch used. - Maximum fetch window defaults to 1 day. Busy windows still halve until they fit in one page, so catching up needs ~3 list calls per day of backlog instead of 72. - An error response's body is read within the deadline too; BaseClient read it in full, so a slow error body could hold a run past its budget. - 401/403 and 429 become AuthError and RateLimitedError in one place, Client._http_request. - A 402 (tenant not licensed) starts the window over at now each run, so licensing the tenant later doesn't replay a backlog or pin last_fetch. - Disabled types drop their window instead of keeping a flagged one. - The time-budget test server speaks HTTP/1.1 like the SOAR API, and new cases cover a body that stalls mid-way and a slow error body. - Release notes no longer claim faster fetches: the request rate is capped. * AbnormalSecurity: move to ContentClient, address review, 2.5.0 - Client inherits from ContentClient (demisto/fastapi image). Its token bucket replaces our RateLimiter, so there's no time.sleep. Retries and the client-wide circuit breaker are off: fetch retries on its next run and stops one type after 5 failures in a row. - Each fetch call runs under asyncio.wait_for with the budget left, which covers the rate-token wait, headers and body; the streamed-body reader and per-worker client clones are gone. - List loops are bounded: halving runs while the window has another page and spans 2s or more; pagination stops at 100 pages. - Invalid JSON and non-numeric params raise DemistoException; an unknown case fetch mode lists the accepted values. - Param descriptions follow the docs style; README rows match the yml. - Release notes move to 2.5.0 with approved prefixes and italic params. - The fake SOAR API is an httpx handler with safe parsing. * AbnormalSecurity: reword 2.5.0 release notes to pass doc-review spelling * AbnormalSecurity: use datetime.fromisoformat directly in the fake SOAR API --------- Co-authored-by: Ivana Irene Thomas <ivanaairenee@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Moshe Eichler <78307768+MosheEichler@users.noreply.github.com> Co-authored-by: MosheEichler <meichler@paloaltonetworks.com>
* Removed unnecessary arraymap from modeling rule * Added RN
yanivblumReco
force-pushed
the
reco-fetch-incidents-pagination
branch
from
October 23, 2026 12:39
87b6ff9 to
e307a76
Compare
…and improve IOC handling for Sandbox (demisto#46291) * [ANY.RUN pack update] Add Cortex XDR playbooks, fix the Using input, and improve IOC handling for Sandbox (demisto#45770) * feat(general): update integrations and playbooks * fix pre-commit issues * feat: add Using parameter to playbooks * add new xdr playbooks and fix get_analysis_report * feat(sandbox): add opt_automated_interactivity parameter for actions * feat: add opt_automated_interactivity, adjust playbooks, fix no iocs message * fix: delete opt_automated_interactivity for linux and android * fix: final adjustments and add release notes * fix: address pre-commit issues * fix: address after-review comments * fix: address pre-commit issues * fix: address pre-commit issues * fix: address review comment issues * fix: address review comment issues * fix after docs review * fix validation errors * fix release notes * Ignore PB106 error --------- Co-authored-by: ANY.RUN Integrations <166487206+anyrun-integrations@users.noreply.github.com> Co-authored-by: test-split <test@test.com> Co-authored-by: lironcohen272 <lircohen@paloaltonetworks.com>
…low Critical - Send multiple severities as `eq` clauses joined with `or`; the External API returns HTTP 500 on the SCIM `in` operator. - Raise API errors from get_alerts instead of returning [], so failures are visible in XSOAR rather than looking like "no new alerts". - Make the Test button query alerts with the same filter fetch-incidents uses. - Bump pack to 1.8.2 with release notes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
yanivblumReco
force-pushed
the
reco-fetch-incidents-pagination
branch
from
October 25, 2026 22:41
e307a76 to
ef26dee
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
During an alert flood,
fetch-incidentscould permanently skip the majority of alerts in a single time window. Separately,fetch-incidentswas also ingesting alerts belonging to Policies in Preview state, which should never leave Reco.Fix 1 - burst pagination:
Root cause:
get_alertsfetched one page per cycle using the External API's default sort (createdAtdescending), thenfetch_incidentsadvanced thelastRuncursor to the newest timestamp seen in that page. When a burst exceededMax fetchin one window, the returned page was the newest slice of the burst, so the cursor jumped past all the older alerts that didn't make it into that page — silently dropping them forever (reported: 669/1069 alerts lost in a one-minute burst, Iron Mountain support ticket).get_alertsnow requests ascendingcreatedAtorder and forwardsstart_index.fetch_incidentsonly advanceslastRunonce a page comes back smaller thanmax_fetch(i.e. the window is fully drained). A full page instead advances astartIndexoffset persisted inlast_run, so a burst larger thanmax_fetchis paginated across multiple fetch cycles instead of being truncated.Fix 2 - exclude Preview-state alerts:
Root cause: a Policy in Preview state is meant to keep its alerts out of the customer's SOAR while it's being evaluated. Reco's webhook and share-service delivery paths already enforce this, but
fetch-incidents(which polls the External API directly) had no equivalent guard, so Preview alerts leaked into XSOAR. Reported in the CAA Slack thread after theTriggerShareguard fix turned out not to cover the content-pack polling path.get_alertsnow always addsstatus ne "ALERT_STATUS_PREVIEW"to its SCIM filter.Version bumped 1.8.0 → 1.8.1.
Test plan
pytest Packs/Reco/Integrations/Reco/Reco_test.py— 72/72 pass (Docker unavailable locally, sodemisto-sdk pre-commit's ruff/xsoar-lint/pylint/mypy steps will run via CI on the upstream PR rather than locally)startIndexwithout skipping;get_alertssends ascending sort + pagination params;get_alertsexcludesALERT_STATUS_PREVIEWalongside other filtersstatus ne "ALERT_STATUS_PREVIEW"verified against the live External API (confirmed a known Preview alert is excluded by this exact filter)Now opened against
demisto/contentas demisto#45577.