Terminate message queries that exceed a configurable time limit

[ramonsmits]

Customer debug packages showed audit message queries against MessagesViewIndexWithFullTextSearch running for up to ~65 minutes. RavenDB's own Databases.QueryTimeoutInSec (default 300) does not stop them because the server renews the operation deadline while a query keeps making progress, so large scans/sorts run for hours and their sorted results spill into the server's Temp folder, filling the disk.

Query time limit

The message view and search queries now run under a linked CancellationTokenSource that cancels after a configurable time limit, default 1 minute, maximum 1 hour. Cancelling the client request aborts the call to the database server, which does terminate the query server-side. Invalid values fall back to the default.

One QueryTimeLimit helper in ServiceControl.Infrastructure serves all persisters. Its rule is that whatever exception surfaces after the deadline fired is the timeout: RavenDB and Npgsql raise OperationCanceledException, Microsoft.Data.SqlClient raises SqlException("Operation cancelled by user"). The SQL Server path is covered by a test against the real container that slows the command down server-side.

The limit applies to exactly these queries:

• Audit RavenAuditDataStore: GetMessages, QueryMessages, QueryMessagesByReceivingEndpoint, QueryMessagesByReceivingEndpointAndKeyword, QueryMessagesByConversationId, read from ServiceControl.Audit/QueryTimeoutInSeconds. QueryAuditCounts (licensing throughput) and QuerySagaHistoryById are not under the limit.
• Primary IMessagesViewDataStore on RavenDB, SQL Server and PostgreSQL: GetAllMessages, GetAllMessagesForEndpoint, GetAllMessagesByConversation, GetAllMessagesForSearch, SearchEndpointMessages, read from ServiceControl/QueryTimeoutInSeconds. On SQL Server and PostgreSQL these queries also use the limit as their per-command timeout, so Database/CommandTimeout cannot undercut it.

Other paged index queries (GetFailedMessages, groups, event log, ...) are unchanged.

What a caller sees

• A query that runs out of its time is answered with 504 Gateway Timeout and a application/problem+json body whose detail names the setting, on both hosts.
• The primary's scatter-gather keeps partial data: a local timeout is absorbed the way a remote failure already was, a remote 504 or error is a missing instance rather than an instance with no data. The response lists every missing instance in the X-Particular-Incomplete-Results header (instanceId:timeout|unavailable|error, exposed through CORS) and carries no ETag. Only when no queried instance answered and one of them timed out does the composite itself answer 504.
• The in-process audit counts used for licensing throughput fail when any instance is missing, instead of recording a partial sum as the day's throughput.
• The remote instance HttpClient timeout is ServiceControl/QueryTimeoutInSeconds plus 30 seconds instead of the 100 second default, so raising the limit on both instances does not make the primary give up on the audit instance first.

ServicePulse follow-up: the audit list already treats a non-2xx answer (the new 504) as a failed query; reading X-Particular-Incomplete-Results on a 200 is still to be added.

Docs: Particular/docs.particular.net#8486

[mauroservienti]

@ramonsmits, I have not finished reviewing this yet. However, we also have this CustomCheck that probably needs to be adapted to incorporate the query timeouts. Also, we have this concept of an instance being temporarily unavailable if it fails to respond to the mentioned custom check. When that property is true, the ScatterGather API skips the unavailable instance until it comes back.

[mauroservienti]

I feel that the TemporaryUnavailable instances, I mentioned, should be counted here

[ramonsmits]

They are. ScatterGatherApi.Execute adds every skipped remote to the results with QueryFailure.Unavailable, AggregateResults turns those into IncompleteInstances, and this header renders them as instanceId:unavailable. IncompleteResultsTests.A_temporarily_unavailable_remote_is_reported_as_missing pins it.

[ramonsmits]

Looked at CheckRemotes against the query timeouts. The check probes /api, which runs no query, so a remote whose queries time out never fails the health check, and that is intended: a 504 means the remote is up, so TemporarilyUnavailable stays false and the next query asks it again (A_remote_query_timeout_keeps_the_local_data_and_reports_the_remote_instance asserts that).

There was a real interaction though, caused by this PR: the probe used the remote's named HttpClient, whose timeout is now the query time limit. With a short limit a healthy but slow remote would fail the health check, and the failure message would still claim 10 seconds. While in there I also fixed a pre-existing problem: the probe's own timeout never applied, because its cancellation was swallowed by a no-op branch, so a hanging remote passed the check and was never disabled, and a shutdown waited out the full budget.

79d19e9 gives the probe its own budget independent of the query limit, makes a remote that doesn't answer in time fail the check and become temporarily unavailable, and lets a shutdown abort the probe without a verdict. CheckRemotesTests covers the four cases.

Since your last pass I also dropped the 30 second margin (22e6109): the remote client now times out at this instance's limit, so ServiceControl/QueryTimeoutInSeconds bounds the whole composite. An audit instance that is slow, hung, or configured with a larger limit is reported as missing rather than stretching the response. The docs PR is updated accordingly.