Skip to content

feat(storage): Add replicated keyspace drop - #1643

Merged
gtema merged 1 commit into
mainfrom
raft-single-use-storage
Nov 13, 2026
Merged

gtema merged 1 commit into
mainfrom
raft-single-use-storage

Conversation

@gtema

@gtema gtema commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

Add Mutation::DropKeyspace, committed through Raft and applied on
every node, and StorageApi::list_keyspaces(prefix) for cleanup
discovery (ADR 0044 step 3).

DropKeyspace is accepted only for names matching
single_use_{use_case}_{bucket} with a numeric bucket, and only in a
transaction made of drops. Apply runs after the batch commits and
removes the keyspace, its per-record metadata in the meta keyspace
and any ephemeral entry. A missing keyspace is a no-op.

The real storage reported the first violation of any type as
Conflict, so a rate limit, quarantine or refused drop looked like
a replay. Violations other than CONFLICT now surface as the new
StoreError::Rejected, which keeps the violation type.

Move require_no_conflict from the scim, service-grant and
oauth2-client drivers into storage-api. It maps only a CONFLICT
violation or error to the caller's conflict and returns Rejected for
any other violation. The oauth2-key JTI retry loop uses it and no
longer retries a rejected write.

Signed-off-by: Artem Goncharov artem.goncharov@gmail.com

Add Mutation::DropKeyspace, committed through Raft and applied on
every node, and StorageApi::list_keyspaces(prefix) for cleanup
discovery (ADR 0044 step 3).

DropKeyspace is accepted only for names matching
single_use_{use_case}_{bucket} with a numeric bucket, and only in a
transaction made of drops. Apply runs after the batch commits and
removes the keyspace, its per-record metadata in the meta keyspace
and any ephemeral entry. A missing keyspace is a no-op.

The real storage reported the first violation of any type as
Conflict, so a rate limit, quarantine or refused drop looked like
a replay. Violations other than CONFLICT now surface as the new
StoreError::Rejected, which keeps the violation type.

Move require_no_conflict from the scim, service-grant and
oauth2-client drivers into storage-api. It maps only a CONFLICT
violation or error to the caller's conflict and returns Rejected for
any other violation. The oauth2-key JTI retry loop uses it and no
longer retries a rejected write.

Signed-off-by: Artem Goncharov <artem.goncharov@gmail.com>
@gtema
gtema enabled auto-merge November 13, 2026 19:00
@github-actions

Copy link
Copy Markdown

🦢 Load Test Results

Goose Attack Report

Plan Overview

Action Started Stopped Elapsed Users
Increasing 26-11-13 19:02:42 26-11-13 19:03:04 00:00:22 0 → 45
Maintaining 26-11-13 19:03:04 26-11-13 19:03:34 00:00:30 45
Decreasing 26-11-13 19:03:34 26-11-13 19:03:36 00:00:02 0 ← 45

Request Metrics

Method Name # Requests # Fails Average (ms) Min (ms) Max (ms) RPS Failures/s
DELETE DELETE /SCIM/v2/:domain_id/Users/:id (sync offboard) 1 0 3.9 3.9 3.9 0.03 0.00
DELETE DELETE /v3/OS-TRUST/trusts/:id 24 0 240 6.07 398 0.80 0.00
DELETE DELETE /v3/auth/tokens 89 0 194 104 289 2.97 0.00
DELETE DELETE /v3/auth/tokens (churn) 25 0 235 5.7 380 0.83 0.00
DELETE DELETE /v3/credentials/:id 24 0 175 4.91 282 0.80 0.00
DELETE DELETE /v3/endpoints/:id 33 0 170 107 246 1.10 0.00
DELETE DELETE /v3/groups/:id 37 0 208 25 343 1.23 0.00
DELETE DELETE /v3/groups/:id/users/:id 37 0 255 43 414 1.23 0.00
DELETE DELETE /v3/projects/:id (churn) 25 0 300 11 401 0.83 0.00
DELETE DELETE /v3/projects/:id (teardown) 1 0 178 178 178 0.03 0.00
DELETE DELETE /v3/projects/:id/users/:id/roles/:id (churn) 25 0 200 8.09 333 0.83 0.00
DELETE DELETE /v3/regions/:id 33 0 178 71 294 1.10 0.00
DELETE DELETE /v3/roles/:id (teardown) 1 0 139 139 139 0.03 0.00
DELETE DELETE /v3/services/:id 33 0 175 104 274 1.10 0.00
DELETE DELETE /v3/users/:id (churn) 48 0 291 7.31 446 1.60 0.00
DELETE DELETE /v3/users/:id (teardown) 2 0 208 192 223 0.07 0.00
DELETE DELETE /v4/mappings/rulesets/:mapping_id (cc teardown) 1 0 9.87 9.87 9.87 0.03 0.00
DELETE DELETE /v4/mappings/rulesets/:mapping_id (teardown) 1 0 170 170 170 0.03 0.00
DELETE DELETE /v4/oauth2/:domain_id/clients/:provider_id (cc teardown) 1 0 7.54 7.54 7.54 0.03 0.00
DELETE DELETE /v4/oauth2/:domain_id/clients/:provider_id (teardown) 1 0 85 85 85 0.03 0.00
GET 105 0 171 104 298 3.50 0.00
GET GET /SCIM/v2/:domain_id/Users (sync reconcile list) 15 0 314 2.2 587 0.50 0.00
└─ 200 (73.3%) 11 427 328 587
└─ 429 (26.7%) 4 3.36 2.2 4.53
GET GET /SCIM/v2/:domain_id/Users/:id (sync reconcile show) 16 0 202 1.39 453 0.53 0.00
└─ 200 (50.0%) 8 400 361 453
└─ 429 (50.0%) 8 4.4 1.39 14
GET GET /v3/OS-TRUST/trusts (by trustor) 24 0 174 5.34 302 0.80 0.00
GET GET /v3/OS-TRUST/trusts/:id 24 0 179 4.33 298 0.80 0.00
GET GET /v3/auth/tokens (churn validate) 25 0 206 5.28 319 0.83 0.00
GET GET /v3/auth/tokens (trustor) 24 0 209 159 287 0.80 0.00
GET GET /v3/auth/tokens (validate new) 89 0 170 106 279 2.97 0.00
GET GET /v3/auth/tokens (validate revoked) 89 0 174 115 295 2.97 0.00
GET GET /v3/auth/tokens (validate token) 275 0 219 121 384 9.17 0.00
GET GET /v3/credentials (by user) 24 0 150 4.11 223 0.80 0.00
GET GET /v3/credentials/:id 24 0 148 3.68 251 0.80 0.00
GET GET /v3/endpoints (by service) 32 0 137 92 196 1.07 0.00
GET GET /v3/endpoints/:id 32 0 135 88 201 1.07 0.00
GET GET /v3/groups 37 0 131 17 206 1.23 0.00
GET GET /v3/groups/:id 36 0 149 95 245 1.20 0.00
GET GET /v3/groups/:id/users 35 0 213 144 326 1.17 0.00
GET GET /v3/projects/:id 206 0 147 89 270 6.87 0.00
GET GET /v3/projects/:id (catalog) 178 0 147 85 277 5.93 0.00
GET GET /v3/projects?domain_id 285 0 190 116 341 9.50 0.00
GET GET /v3/regions/:id 32 0 138 86 263 1.07 0.00
GET GET /v3/role_assignments 101 0 157 93 296 3.37 0.00
GET GET /v3/role_assignments?role.id 101 0 147 90 277 3.37 0.00
GET GET /v3/role_assignments?scope.domain.id 102 0 144 81 287 3.40 0.00
GET GET /v3/role_assignments?user.id 101 0 150 93 260 3.37 0.00
GET GET /v3/roles 361 0 167 92 300 12.03 0.00
GET GET /v3/roles/:id 246 0 123 65 293 8.20 0.00
GET GET /v3/services 32 0 132 91 198 1.07 0.00
GET GET /v3/services/:id 32 0 139 100 235 1.07 0.00
GET GET /v3/users/:id 397 0 147 87 281 13.23 0.00
GET GET /v3/users/:id (catalog) 176 0 146 89 240 5.87 0.00
GET GET /v3/users?domain_id 284 0 196 123 352 9.47 0.00
GET GET /v4/api-keys 117 0 129 76 248 3.90 0.00
GET GET /v4/api-keys/:client_id 117 0 130 82 278 3.90 0.00
GET GET /v4/mappings/rulesets 117 0 128 78 229 3.90 0.00
GET GET /v4/mappings/rulesets/:mapping_id 118 0 130 82 277 3.93 0.00
GET GET /v4/oauth2/:domain_id/.well-known/openid-configuration 3152 0 9.64 1.71 164 105.07 0.00
GET GET /v4/oauth2/:domain_id/clients 117 0 133 74 311 3.90 0.00
GET GET /v4/oauth2/:domain_id/clients/:provider_id 117 0 125 80 236 3.90 0.00
GET GET /v4/oauth2/:domain_id/jwks 3152 0 9.41 1.63 84 105.07 0.00
GET GET /v4/scim_realms 75 0 130 82 243 2.50 0.00
GET GET /v4/scim_realms/:domain_id/:provider_id 75 0 124 72 247 2.50 0.00
HEAD HEAD /v3/groups/:id/users/:id 35 0 195 135 304 1.17 0.00
PATCH PATCH /SCIM/v2/:domain_id/Users/:id (sync attribute drift) 16 0 377 2.8 697 0.53 0.00
└─ 200 (75.0%) 12 502 385 697
└─ 429 (25.0%) 4 4.24 2.8 6.1
PATCH PATCH /v3/credentials/:id 24 0 240 6.31 388 0.80 0.00
PATCH PATCH /v3/groups/:id 36 0 189 48 309 1.20 0.00
PATCH PATCH /v3/services/:id 32 0 193 121 317 1.07 0.00
POST POST /v3/OS-TRUST/trusts 24 0 253 6.4 394 0.80 0.00
POST POST /v3/auth/tokens 89 0 130 76 267 2.97 0.00
POST POST /v3/auth/tokens (churn user) 25 0 1338 875 2262 0.83 0.00
POST POST /v3/auth/tokens (password, invalid) 25 0 1275 802 1923 0.83 0.00
POST POST /v3/auth/tokens (password, project-scoped) 24 0 1429 946 2361 0.80 0.00
POST POST /v3/auth/tokens (password, unscoped) 25 0 1249 695 1827 0.83 0.00
POST POST /v3/auth/tokens (rescope to system) 357 0 169 92 308 11.90 0.00
POST POST /v3/auth/tokens (rescope, invalid project) 203 0 149 93 275 6.77 0.00
POST POST /v3/auth/tokens (system-scoped, bootstrap admin) 48 0 1313 920 2265 1.60 0.00
POST POST /v3/credentials 24 0 153 5.87 230 0.80 0.00
POST POST /v3/endpoints 32 0 169 112 258 1.07 0.00
POST POST /v3/groups 35 0 146 108 223 1.17 0.00
POST POST /v3/projects (churn) 23 0 145 93 202 0.77 0.00
POST POST /v3/regions 32 0 159 99 261 1.07 0.00
POST POST /v3/services 32 0 158 91 275 1.07 0.00
POST POST /v3/users (churn) 48 0 1362 948 2277 1.60 0.00
POST POST /v3/users (setup) 1 0 2126 2126 2126 0.03 0.00
POST POST /v4/api-keys/:client_id/revoke (teardown) 1 0 153 153 153 0.03 0.00
POST POST /v4/oauth2/:domain_id/device_authorization 1337 0 22 5.81 173 44.57 0.00
POST POST /v4/oauth2/:domain_id/token (client_credentials) 17 0 383 262 575 0.57 0.00
PUT PUT /v3/groups/:id/users/:id 35 0 211 137 351 1.17 0.00
PUT PUT /v3/projects/:id/users/:id/roles/:id (churn) 24 0 185 130 270 0.80 0.00
PUT PUT /v4/scim_realms/:domain_id/:provider_id 75 0 147 88 320 2.50 0.00
Aggregated 13818 0 97 1.39 2361 460.60 0.00

Response Time Metrics

Method Name 50%ile (ms) 60%ile (ms) 70%ile (ms) 80%ile (ms) 90%ile (ms) 95%ile (ms) 99%ile (ms) 100%ile (ms)
DELETE DELETE /SCIM/v2/:domain_id/Users/:id (sync offboard) 3.9 3.9 3.9 3.9 3.9 3.9 3.9 3.9
DELETE DELETE /v3/OS-TRUST/trusts/:id 237 245 281 301 326 375 398 398
DELETE DELETE /v3/auth/tokens 189 196 208 241 261 285 289 289
DELETE DELETE /v3/auth/tokens (churn) 241 245 253 264 334 354 380 380
DELETE DELETE /v3/credentials/:id 181 185 196 208 224 230 282 282
DELETE DELETE /v3/endpoints/:id 161 171 189 196 222 224 246 246
DELETE DELETE /v3/groups/:id 202 222 232 255 272 297 343 343
DELETE DELETE /v3/groups/:id/users/:id 255 264 272 281 297 383 414 414
DELETE DELETE /v3/projects/:id (churn) 326 326 338 346 383 387 401 401
DELETE DELETE /v3/projects/:id (teardown) 178 178 178 178 178 178 178 178
DELETE DELETE /v3/projects/:id/users/:id/roles/:id (churn) 200 206 218 230 272 309 333 333
DELETE DELETE /v3/regions/:id 171 179 183 228 268 268 294 294
DELETE DELETE /v3/roles/:id (teardown) 139 139 139 139 139 139 139 139
DELETE DELETE /v3/services/:id 173 179 185 206 232 243 274 274
DELETE DELETE /v3/users/:id (churn) 301 317 338 367 395 412 446 446
DELETE DELETE /v3/users/:id (teardown) 192 192 192 223 223 223 223 223
DELETE DELETE /v4/mappings/rulesets/:mapping_id (cc teardown) 9.87 9.87 9.87 9.87 9.87 9.87 9.87 9.87
DELETE DELETE /v4/mappings/rulesets/:mapping_id (teardown) 170 170 170 170 170 170 170 170
DELETE DELETE /v4/oauth2/:domain_id/clients/:provider_id (cc teardown) 7.54 7.54 7.54 7.54 7.54 7.54 7.54 7.54
DELETE DELETE /v4/oauth2/:domain_id/clients/:provider_id (teardown) 85 85 85 85 85 85 85 85
GET 161 169 185 204 230 264 293 298
GET GET /SCIM/v2/:domain_id/Users (sync reconcile list) 371 383 416 465 569 569 587 587
GET GET /SCIM/v2/:domain_id/Users/:id (sync reconcile show) 14 379 387 391 395 449 453 453
GET GET /v3/OS-TRUST/trusts (by trustor) 177 181 185 200 253 272 302 302
GET GET /v3/OS-TRUST/trusts/:id 175 191 216 222 249 285 298 298
GET GET /v3/auth/tokens (churn validate) 208 228 232 234 272 293 319 319
GET GET /v3/auth/tokens (trustor) 196 202 220 247 268 285 287 287
GET GET /v3/auth/tokens (validate new) 163 169 183 206 224 247 272 279
GET GET /v3/auth/tokens (validate revoked) 167 177 187 202 226 261 281 295
GET GET /v3/auth/tokens (validate token) 212 224 234 253 289 317 342 384
GET GET /v3/credentials (by user) 144 163 183 191 204 208 223 223
GET GET /v3/credentials/:id 130 142 155 191 243 245 251 251
GET GET /v3/endpoints (by service) 136 142 144 161 179 179 196 196
GET GET /v3/endpoints/:id 136 138 146 155 161 171 201 201
GET GET /v3/groups 126 136 144 157 169 196 206 206
GET GET /v3/groups/:id 142 151 157 185 196 210 245 245
GET GET /v3/groups/:id/users 218 226 230 237 281 281 326 326
GET GET /v3/projects/:id 140 146 157 177 191 218 257 270
GET GET /v3/projects/:id (catalog) 142 153 163 175 191 208 228 277
GET GET /v3/projects?domain_id 177 189 206 220 255 289 317 341
GET GET /v3/regions/:id 132 136 142 157 163 165 263 263
GET GET /v3/role_assignments 146 153 161 185 224 249 285 296
GET GET /v3/role_assignments?role.id 138 148 161 173 189 208 228 277
GET GET /v3/role_assignments?scope.domain.id 134 142 153 171 191 228 255 287
GET GET /v3/role_assignments?user.id 142 155 163 173 196 220 253 260
GET GET /v3/roles 161 171 179 198 220 241 276 300
GET GET /v3/roles/:id 115 125 134 146 167 177 232 293
GET GET /v3/services 121 129 144 157 167 177 198 198
GET GET /v3/services/:id 129 136 155 157 169 187 235 235
GET GET /v3/users/:id 138 146 159 177 194 220 259 281
GET GET /v3/users/:id (catalog) 138 144 155 175 202 218 230 240
GET GET /v3/users?domain_id 189 198 210 228 264 276 313 352
GET GET /v4/api-keys 120 132 140 155 179 189 226 248
GET GET /v4/api-keys/:client_id 122 131 142 155 171 185 214 278
GET GET /v4/mappings/rulesets 116 125 138 153 181 202 226 229
GET GET /v4/mappings/rulesets/:mapping_id 122 130 138 159 171 179 243 277
GET GET /v4/oauth2/:domain_id/.well-known/openid-configuration 8.03 9.02 10 12 16 19 37 164
GET GET /v4/oauth2/:domain_id/clients 126 132 142 153 179 214 241 311
GET GET /v4/oauth2/:domain_id/clients/:provider_id 118 124 136 144 167 181 222 236
GET GET /v4/oauth2/:domain_id/jwks 7.97 9.02 10 12 16 19 32 84
GET GET /v4/scim_realms 121 130 140 155 179 198 222 243
GET GET /v4/scim_realms/:domain_id/:provider_id 117 123 132 148 163 169 189 247
HEAD HEAD /v3/groups/:id/users/:id 183 204 218 249 259 264 304 304
PATCH PATCH /SCIM/v2/:domain_id/Users/:id (sync attribute drift) 449 489 494 518 528 610 697 697
PATCH PATCH /v3/credentials/:id 243 245 249 255 338 358 388 388
PATCH PATCH /v3/groups/:id 183 196 208 226 249 268 309 309
PATCH PATCH /v3/services/:id 183 189 208 237 261 264 317 317
POST POST /v3/OS-TRUST/trusts 261 264 281 326 350 387 394 394
POST POST /v3/auth/tokens 121 126 138 163 183 212 251 267
POST POST /v3/auth/tokens (churn user) 1,352 1,384 1,434 1,466 1,712 1,745 2,262 2,262
POST POST /v3/auth/tokens (password, invalid) 1,237 1,286 1,303 1,352 1,516 1,892 1,923 1,923
POST POST /v3/auth/tokens (password, project-scoped) 1,368 1,384 1,417 1,516 1,712 2,089 2,361 2,361
POST POST /v3/auth/tokens (password, unscoped) 1,188 1,204 1,253 1,352 1,778 1,810 1,827 1,827
POST POST /v3/auth/tokens (rescope to system) 161 173 181 200 224 251 281 308
POST POST /v3/auth/tokens (rescope, invalid project) 142 151 165 179 196 220 241 275
POST POST /v3/auth/tokens (system-scoped, bootstrap admin) 1,253 1,286 1,335 1,417 1,548 1,860 2,265 2,265
POST POST /v3/credentials 148 165 191 196 214 222 230 230
POST POST /v3/endpoints 161 167 171 194 232 232 258 258
POST POST /v3/groups 140 146 155 169 194 198 223 223
POST POST /v3/projects (churn) 144 151 153 157 177 183 202 202
POST POST /v3/regions 155 159 167 181 226 232 261 261
POST POST /v3/services 140 146 173 191 239 253 275 275
POST POST /v3/users (churn) 1,319 1,352 1,417 1,450 1,499 2,146 2,277 2,277
POST POST /v3/users (setup) 2,126 2,126 2,126 2,126 2,126 2,126 2,126 2,126
POST POST /v4/api-keys/:client_id/revoke (teardown) 153 153 153 153 153 153 153 153
POST POST /v4/oauth2/:domain_id/device_authorization 20 22 24 28 35 42 64 173
POST POST /v4/oauth2/:domain_id/token (client_credentials) 367 379 395 444 485 502 575 575
PUT PUT /v3/groups/:id/users/:id 198 204 222 255 285 301 351 351
PUT PUT /v3/projects/:id/users/:id/roles/:id (churn) 177 189 206 224 232 247 270 270
PUT PUT /v4/scim_realms/:domain_id/:provider_id 136 148 165 171 189 206 237 320
Aggregated 22 108 136 163 204 247 1,188 2,361

Status Code Metrics

Method Name Status Codes
DELETE DELETE /SCIM/v2/:domain_id/Users/:id (sync offboard) 1 [429]
DELETE DELETE /v3/OS-TRUST/trusts/:id 24 [204]
DELETE DELETE /v3/auth/tokens 89 [204]
DELETE DELETE /v3/auth/tokens (churn) 25 [204]
DELETE DELETE /v3/credentials/:id 24 [204]
DELETE DELETE /v3/endpoints/:id 33 [204]
DELETE DELETE /v3/groups/:id 37 [204]
DELETE DELETE /v3/groups/:id/users/:id 37 [204]
DELETE DELETE /v3/projects/:id (churn) 25 [204]
DELETE DELETE /v3/projects/:id (teardown) 1 [204]
DELETE DELETE /v3/projects/:id/users/:id/roles/:id (churn) 25 [204]
DELETE DELETE /v3/regions/:id 33 [204]
DELETE DELETE /v3/roles/:id (teardown) 1 [204]
DELETE DELETE /v3/services/:id 33 [204]
DELETE DELETE /v3/users/:id (churn) 48 [204]
DELETE DELETE /v3/users/:id (teardown) 2 [204]
DELETE DELETE /v4/mappings/rulesets/:mapping_id (cc teardown) 1 [204]
DELETE DELETE /v4/mappings/rulesets/:mapping_id (teardown) 1 [204]
DELETE DELETE /v4/oauth2/:domain_id/clients/:provider_id (cc teardown) 1 [204]
DELETE DELETE /v4/oauth2/:domain_id/clients/:provider_id (teardown) 1 [204]
GET 105 [200]
GET GET /SCIM/v2/:domain_id/Users (sync reconcile list) 11 [200], 4 [429]
GET GET /SCIM/v2/:domain_id/Users/:id (sync reconcile show) 8 [429], 8 [200]
GET GET /v3/OS-TRUST/trusts (by trustor) 24 [200]
GET GET /v3/OS-TRUST/trusts/:id 24 [200]
GET GET /v3/auth/tokens (churn validate) 25 [200]
GET GET /v3/auth/tokens (trustor) 24 [200]
GET GET /v3/auth/tokens (validate new) 89 [200]
GET GET /v3/auth/tokens (validate revoked) 89 [404]
GET GET /v3/auth/tokens (validate token) 275 [200]
GET GET /v3/credentials (by user) 24 [200]
GET GET /v3/credentials/:id 24 [200]
GET GET /v3/endpoints (by service) 32 [200]
GET GET /v3/endpoints/:id 32 [200]
GET GET /v3/groups 37 [200]
GET GET /v3/groups/:id 36 [200]
GET GET /v3/groups/:id/users 35 [200]
GET GET /v3/projects/:id 206 [200]
GET GET /v3/projects/:id (catalog) 178 [200]
GET GET /v3/projects?domain_id 285 [200]
GET GET /v3/regions/:id 32 [200]
GET GET /v3/role_assignments 101 [200]
GET GET /v3/role_assignments?role.id 101 [200]
GET GET /v3/role_assignments?scope.domain.id 102 [200]
GET GET /v3/role_assignments?user.id 101 [200]
GET GET /v3/roles 361 [200]
GET GET /v3/roles/:id 246 [200]
GET GET /v3/services 32 [200]
GET GET /v3/services/:id 32 [200]
GET GET /v3/users/:id 397 [200]
GET GET /v3/users/:id (catalog) 176 [200]
GET GET /v3/users?domain_id 284 [200]
GET GET /v4/api-keys 117 [200]
GET GET /v4/api-keys/:client_id 117 [200]
GET GET /v4/mappings/rulesets 117 [200]
GET GET /v4/mappings/rulesets/:mapping_id 118 [200]
GET GET /v4/oauth2/:domain_id/.well-known/openid-configuration 3,152 [200]
GET GET /v4/oauth2/:domain_id/clients 117 [200]
GET GET /v4/oauth2/:domain_id/clients/:provider_id 117 [200]
GET GET /v4/oauth2/:domain_id/jwks 3,152 [200]
GET GET /v4/scim_realms 75 [200]
GET GET /v4/scim_realms/:domain_id/:provider_id 75 [200]
HEAD HEAD /v3/groups/:id/users/:id 35 [204]
PATCH PATCH /SCIM/v2/:domain_id/Users/:id (sync attribute drift) 4 [429], 12 [200]
PATCH PATCH /v3/credentials/:id 24 [200]
PATCH PATCH /v3/groups/:id 36 [200]
PATCH PATCH /v3/services/:id 32 [200]
POST POST /v3/OS-TRUST/trusts 24 [201]
POST POST /v3/auth/tokens 89 [201]
POST POST /v3/auth/tokens (churn user) 25 [201]
POST POST /v3/auth/tokens (password, invalid) 25 [401]
POST POST /v3/auth/tokens (password, project-scoped) 24 [201]
POST POST /v3/auth/tokens (password, unscoped) 25 [201]
POST POST /v3/auth/tokens (rescope to system) 357 [201]
POST POST /v3/auth/tokens (rescope, invalid project) 203 [401]
POST POST /v3/auth/tokens (system-scoped, bootstrap admin) 48 [201]
POST POST /v3/credentials 24 [201]
POST POST /v3/endpoints 32 [201]
POST POST /v3/groups 35 [201]
POST POST /v3/projects (churn) 23 [201]
POST POST /v3/regions 32 [201]
POST POST /v3/services 32 [201]
POST POST /v3/users (churn) 48 [201]
POST POST /v3/users (setup) 1 [201]
POST POST /v4/api-keys/:client_id/revoke (teardown) 1 [200]
POST POST /v4/oauth2/:domain_id/device_authorization 1,337 [200]
POST POST /v4/oauth2/:domain_id/token (client_credentials) 17 [200]
PUT PUT /v3/groups/:id/users/:id 35 [204]
PUT PUT /v3/projects/:id/users/:id/roles/:id (churn) 24 [204]
PUT PUT /v4/scim_realms/:domain_id/:provider_id 75 [200]
Aggregated 12,130 [200], 17 [429], 819 [201], 228 [401], 89 [404], 535 [204]

Transaction Metrics

Transaction # Times Run # Fails Average (ms) Min (ms) Max (ms) RPS Failures/s
ReadHeavy
0.0 1 0 2213 2213 2213 0.03 0.00
0.1 105 0 199 132 339 3.50 0.00
0.2 105 0 189 116 341 3.50 0.00
0.3 105 0 171 105 298 3.50 0.00
TokenLifecycle
1.0 1 0 2189 2189 2189 0.03 0.00
1.1 89 0 669 475 903 2.97 0.00
ValidateToken
2.0 1 0 1869 1869 1869 0.03 0.00
2.1 275 0 219 121 384 9.17 0.00
UserCRUD
3.0 0 0 0 0 0 0.00 0.00
3.1 1 0 2126 2126 2126 0.03 0.00
3.2 397 0 147 87 281 13.23 0.00
3.3 2 0 208 192 223 0.07 0.00
ProjectCRUD
4.0 0 0 0 0 0 0.00 0.00
4.1 0 0 0 0 0 0.00 0.00
4.2 206 0 147 89 270 6.87 0.00
4.3 1 0 178 178 178 0.03 0.00
RoleCRUD
5.0 0 0 0 0 0 0.00 0.00
5.1 0 0 0 0 0 0.00 0.00
5.2 246 0 123 65 293 8.20 0.00
5.3 1 0 139 139 139 0.03 0.00
RoleList
6.0 0 0 0 0 0 0.00 0.00
6.1 361 0 167 92 300 12.03 0.00
RoleAssignmentList
7.0 0 0 0 0 0 0.00 0.00
7.1 100 0 158 101 296 3.33 0.00
7.2 101 0 150 93 260 3.37 0.00
7.3 101 0 145 81 287 3.37 0.00
7.4 101 0 147 90 277 3.37 0.00
UserRead
8.0 0 0 0 0 0 0.00 0.00
8.1 178 0 195 123 352 5.93 0.00
8.2 176 0 146 90 240 5.87 0.00
ProjectRead
9.0 0 0 0 0 0 0.00 0.00
9.1 180 0 191 119 332 6.00 0.00
9.2 178 0 147 85 277 5.93 0.00
PasswordAuth
10.0 23 0 1199 695 1773 0.77 0.00
10.1 24 0 1429 946 2361 0.80 0.00
PasswordAuthNegative
11.0 25 0 1276 802 1924 0.83 0.00
Rescope
12.0 0 0 0 0 0 0.00 0.00
12.1 357 0 169 92 308 11.90 0.00
RescopeNegative
13.0 0 0 0 0 0 0.00 0.00
13.1 203 0 149 93 275 6.77 0.00
SystemScopeAuth
14.0 48 0 1313 920 2265 1.60 0.00
GroupChurn
15.0 0 0 0 0 0 0.00 0.00
15.1 37 0 1682 886 1952 1.23 0.00
CatalogChurn
16.0 0 0 0 0 0 0.00 0.00
16.1 33 0 1877 1428 2090 1.10 0.00
CredentialChurn
17.0 0 0 0 0 0 0.00 0.00
17.1 24 0 2569 1356 3633 0.80 0.00
TrustChurn
18.0 0 0 0 0 0 0.00 0.00
18.1 24 0 2657 1589 3746 0.80 0.00
IdentityChurn
19.0 0 0 0 0 0 0.00 0.00
19.1 25 0 2603 1285 3370 0.83 0.00
ApiKeyCRUD
20.0 0 0 0 0 0 0.00 0.00
20.1 0 0 0 0 0 0.00 0.00
20.2 116 0 130 82 278 3.87 0.00
20.3 117 0 129 76 248 3.90 0.00
20.4 1 0 153 153 153 0.03 0.00
MappingCRUD
21.0 0 0 0 0 0 0.00 0.00
21.1 0 0 0 0 0 0.00 0.00
21.2 117 0 130 82 277 3.90 0.00
21.3 117 0 128 78 229 3.90 0.00
21.4 1 0 170 170 170 0.03 0.00
OAuth2ClientCRUD
22.0 0 0 0 0 0 0.00 0.00
22.1 0 0 0 0 0 0.00 0.00
22.2 116 0 126 80 236 3.87 0.00
22.3 117 0 133 74 311 3.90 0.00
22.4 1 0 85 85 85 0.03 0.00
OAuth2ClientCredentials
23.0 0 0 0 0 0 0.00 0.00
23.1 0 0 0 0 0 0.00 0.00
23.2 17 0 383 262 575 0.57 0.00
23.3 1 0 17 17 17 0.03 0.00
OAuth2DeviceAuthorization
24.0 1337 0 23 5.82 173 44.57 0.00
OAuth2Discovery
25.0 3151 0 9.42 1.64 84 105.03 0.00
25.1 3152 0 9.66 1.72 164 105.07 0.00
ScimRealmRead
26.0 0 0 0 0 0 0.00 0.00
26.1 75 0 124 72 247 2.50 0.00
26.2 75 0 130 82 243 2.50 0.00
26.3 75 0 147 89 320 2.50 0.00
ScimSync
27.0 0 0 0 0 0 0.00 0.00
27.1 0 0 0 0 0 0.00 0.00
27.2 0 0 0 0 0 0.00 0.00
27.3 16 0 607 328 1006 0.53 0.00
27.4 16 0 703 361 1014 0.53 0.00
27.5 16 0 628 385 1008 0.53 0.00
27.6 1 0 1005 1005 1005 0.03 0.00
Aggregated 12470 0 109 1.64 3746 415.67 0.00

Scenario Metrics

Scenario # Users # Times Run Average (ms) Min (ms) Max (ms) Scenarios/s Iterations
ReadHeavy 2 104 559 398 751 3.47 52.00
TokenLifecycle 2 87 671 475 903 2.90 43.50
ValidateToken 2 273 219 121 384 9.10 136.50
UserCRUD 2 395 147 87 281 13.17 197.50
ProjectCRUD 1 205 147 89 270 6.83 205.00
RoleCRUD 1 245 122 65 293 8.17 245.00
RoleList 2 359 167 92 300 11.97 179.50
RoleAssignmentList 2 100 599 388 867 3.33 50.00
UserRead 2 176 341 232 502 5.87 88.00
ProjectRead 2 178 338 227 513 5.93 89.00
PasswordAuth 2 23 2676 2074 4189 0.77 11.50
PasswordAuthNegative 1 24 1278 802 1924 0.80 24.00
Rescope 2 355 169 92 308 11.83 177.50
RescopeNegative 1 202 149 93 275 6.73 202.00
SystemScopeAuth 2 46 1320 920 2265 1.53 23.00
GroupChurn 2 35 1708 1256 1952 1.17 17.50
CatalogChurn 2 31 1892 1428 2090 1.03 15.50
CredentialChurn 2 22 2649 2042 3633 0.73 11.00
TrustChurn 2 22 2752 2316 3746 0.73 11.00
IdentityChurn 2 23 2663 2211 3370 0.77 11.50
ApiKeyCRUD 1 116 259 163 417 3.87 116.00
MappingCRUD 1 117 258 177 407 3.90 117.00
OAuth2ClientCRUD 1 116 258 163 451 3.87 116.00
OAuth2ClientCredentials 1 16 1850 1503 2311 0.53 16.00
OAuth2DeviceAuthorization 1 1336 23 5.82 173 44.53 1336.00
OAuth2Discovery 2 3150 19 4.75 181 105.00 1575.00
ScimRealmRead 1 74 402 265 574 2.47 74.00
ScimSync 1 15 1946 1780 2463 0.50 15.00
Aggregated 45 7845 171 4.75 4189 261.50 174.33

Error Metrics

Method Name # Error
POST POST /v3/auth/tokens (password, invalid) 50 401 Unauthorized: POST /v3/auth/tokens (password, invalid)
POST POST /v3/auth/tokens (rescope, invalid project) 406 401 Unauthorized: POST /v3/auth/tokens (rescope, invalid project)
GET GET /v3/auth/tokens (validate revoked) 178 404 Not Found: GET /v3/auth/tokens (validate revoked)
DELETE DELETE /SCIM/v2/:domain_id/Users/:id (sync offboard) 2 429 Too Many Requests: DELETE /SCIM/v2/:domain_id/Users/:id (sync offboard)
GET GET /SCIM/v2/:domain_id/Users (sync reconcile list) 8 429 Too Many Requests: GET /SCIM/v2/:domain_id/Users (sync reconcile list)
GET GET /SCIM/v2/:domain_id/Users/:id (sync reconcile show) 16 429 Too Many Requests: GET /SCIM/v2/:domain_id/Users/:id (sync reconcile show)
PATCH PATCH /SCIM/v2/:domain_id/Users/:id (sync attribute drift) 8 429 Too Many Requests: PATCH /SCIM/v2/:domain_id/Users/:id (sync attribute drift)

View full report

@github-actions

Copy link
Copy Markdown

🧪 Tempest Identity Compatibility Results (advisory, non-blocking)

rust

Totals
======
Ran: 133 tests in 34.0689 sec.
 - Passed: 89
 - Skipped: 4
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 40
Sum of execute time for each test: 30.4417 sec.
Failed test IDs
===== FAILED TESTS (target=rust) =====
tempest.api.identity.admin.v3.test_credentials.CredentialsTestJSON.test_credentials_create_get_update_delete[id-7cd59bf9-bda4-4c72-9467-d21cab278355,smoke]
tempest.api.identity.admin.v3.test_default_project_id.TestDefaultProjectId.test_default_project_id[id-d6110661-6a71-49a7-a453-b5e26640ff6d]
tempest.api.identity.admin.v3.test_domain_configuration.DomainConfigurationTestJSON.test_create_domain_config_and_show_config_groups_and_options[id-9e3ff13c-f597-4f01-9377-d6c06c2a1477]
tempest.api.identity.admin.v3.test_domain_configuration.DomainConfigurationTestJSON.test_create_update_and_delete_domain_config_groups_and_opts[id-c7510fa2-6661-4170-9c6b-4783a80651e9]
tempest.api.identity.admin.v3.test_domain_configuration.DomainConfigurationTestJSON.test_show_default_group_config_and_options[id-11a02bf0-6f94-4380-b3b0-c8dc18fc0d22]
tempest.api.identity.admin.v3.test_inherits.InheritsV3TestJSON.test_inherit_assign_check_revoke_roles_on_projects_group[id-26021436-d5a4-4256-943c-ded01e0d4b45]
tempest.api.identity.admin.v3.test_inherits.InheritsV3TestJSON.test_inherit_assign_check_revoke_roles_on_projects_user[id-18b70e45-7687-4b72-8277-b8f1a47d7591]
tempest.api.identity.admin.v3.test_inherits.InheritsV3TestJSON.test_inherit_assign_list_check_revoke_roles_on_domains_group[id-c7a8dda2-be50-4fb4-9a9c-e830771078b1]
tempest.api.identity.admin.v3.test_inherits.InheritsV3TestJSON.test_inherit_assign_list_check_revoke_roles_on_domains_user[id-4e6f0366-97c8-423c-b2be-41eae6ac91c8]
tempest.api.identity.admin.v3.test_inherits.InheritsV3TestJSON.test_inherit_assign_list_revoke_user_roles_on_domain[id-3acf666e-5354-42ac-8e17-8b68893bcd36]
tempest.api.identity.admin.v3.test_inherits.InheritsV3TestJSON.test_inherit_assign_list_revoke_user_roles_on_project_tree[id-9f02ccd9-9b57-46b4-8f77-dd5a736f3a06]
tempest.api.identity.admin.v3.test_domains_negative.DomainsNegativeTestJSON.test_delete_active_domain[gate,id-1f3fbff5-4e44-400d-9ca1-d953f05f609b,negative]
setUpClass (tempest.api.identity.admin.v3.test_endpoint_groups.EndPointGroupsTest)
tempest.api.identity.admin.v3.test_projects.ProjectsTestJSON.test_create_is_domain_project[id-a7eb9416-6f9b-4dbb-b71b-7f73aaef59d5]
tempest.api.identity.admin.v3.test_projects.ProjectsTestJSON.test_project_create_with_parent[id-1854f9c0-70bc-4d11-a08a-1c789d339e3d]
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON.test_assignments_for_domain_roles[id-3859df7e-5b78-4e4d-b10e-214c8953842a]
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON.test_assignments_for_implied_roles_create_delete[id-c8828027-df48-4021-95df-b65b92c7429e]
tempest.api.identity.admin.v3.test_oauth_consumers.OAUTHConsumersV3Test.test_create_and_show_consumer[id-c8307ea6-a86c-47fd-ae7b-5b3b2caca76d]
tempest.api.identity.admin.v3.test_oauth_consumers.OAUTHConsumersV3Test.test_delete_consumer[id-fdfa1b7f-2a31-4354-b2c7-f6ae20554f93]
tempest.api.identity.admin.v3.test_oauth_consumers.OAUTHConsumersV3Test.test_list_consumers[id-09ca50de-78f2-4ffb-ac71-f2254036b2b8]
tempest.api.identity.admin.v3.test_oauth_consumers.OAUTHConsumersV3Test.test_update_consumer[id-080a9b1a-c009-47c0-9979-5305bf72e3dc]
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON.test_implied_domain_roles[id-eb1e1c24-1bc4-4d47-9748-e127a1852c82]
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON.test_implied_roles_create_check_show_delete[id-c90c316c-d706-4728-bcba-eb1912081b69]
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON.test_role_create_update_show_list[id-18afc6c0-46cf-4911-824e-9989cc056c3a,smoke]
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_get_trusts_all[id-4773ebd5-ecbf-4255-b8d8-b63e6f72b65d,smoke]
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_get_trusts_query[id-6268b345-87ca-47c0-9ce3-37792b43403a]
tempest.api.identity.admin.v3.test_list_projects.ListProjectsTestJSON.test_list_projects_with_parent[id-6edc66f5-2941-4a17-9526-4073311c1fac]
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_trust_expire[id-0ed14b66-cefd-4b5c-a964-65759453e292]
tempest.api.identity.admin.v3.test_users_negative.UsersNegativeTest.test_create_user_for_non_existent_domain[id-e75f006c-89cc-477b-874d-588e4eab4b17,negative]
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_trust_impersonate[id-5a0a91a4-baef-4a14-baba-59bf4d7fcace]
tempest.api.identity.v3.test_application_credentials.ApplicationCredentialsV3Test.test_create_application_credential_expires[id-852daf0c-42b5-4239-8466-d193d0543ed3]
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_trust_noimpersonate[id-ed2a8779-a7ac-49dc-afd7-30f32f936ed2]
tempest.api.identity.admin.v3.test_tokens.TokensV3TestJSON.test_get_available_domain_scopes[id-ec5ecb05-af64-4c04-ac86-4d9f6f12f185]
tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_media_types[id-657c1970-4722-4189-8831-7325f3bc4265,smoke]
tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_version_resources[id-b9232f5e-d9e5-4d97-b96c-28d3db4de1bd,smoke]
tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_identity_v3_existence[id-79aec9ae-710f-4c54-a4fc-3aa25b4feac3]
tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_list_api_versions[id-721f480f-35b6-46c7-846e-047e6acea0dc,smoke]
tempest.api.identity.admin.v3.test_tokens.TokensV3TestJSON.test_rescope_token[id-565fa210-1da1-4563-999b-f7b5b67cf112]
tempest.api.identity.v3.test_catalog.IdentityCatalogTest.test_catalog_standardization[id-56b57ced-22b8-4127-9b8a-565dfb0207e2]
tempest.api.identity.v3.test_tokens.TokensV3Test.test_validate_token[id-a9512ac3-3909-48a4-b395-11f438e16260]
===== END FAILED TESTS (target=rust) =====

View full run

@github-actions

Copy link
Copy Markdown

🐰 Bencher Report

Projectkeystone
Branchraft-single-use-storage
Testbedubuntu-latest
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
nanoseconds (ns)
(Result Δ%)
Upper Boundary
nanoseconds (ns)
(Limit %)
Command_Serde/apply/remove📈 view plot
🚷 view threshold
131,550.00 ns
(+4.61%)Baseline: 125,748.58 ns
206,265.13 ns
(63.78%)
Command_Serde/apply/set📈 view plot
🚷 view threshold
153,070.00 ns
(+16.41%)Baseline: 131,487.23 ns
201,357.98 ns
(76.02%)
Command_Serde/pack/delete📈 view plot
🚷 view threshold
107.00 ns
(-7.03%)Baseline: 115.10 ns
165.07 ns
(64.82%)
Command_Serde/pack/delete_index📈 view plot
🚷 view threshold
94.22 ns
(-8.69%)Baseline: 103.19 ns
144.72 ns
(65.11%)
Command_Serde/pack/set📈 view plot
🚷 view threshold
160.80 ns
(-14.74%)Baseline: 188.60 ns
274.41 ns
(58.60%)
Command_Serde/pack/set_index📈 view plot
🚷 view threshold
95.50 ns
(-7.41%)Baseline: 103.15 ns
144.62 ns
(66.03%)
Command_Serde/unpack/delete📈 view plot
🚷 view threshold
149.05 ns
(+3.59%)Baseline: 143.89 ns
215.91 ns
(69.03%)
Command_Serde/unpack/delete_index📈 view plot
🚷 view threshold
122.66 ns
(+0.61%)Baseline: 121.91 ns
181.25 ns
(67.67%)
Command_Serde/unpack/set📈 view plot
🚷 view threshold
208.76 ns
(-6.45%)Baseline: 223.16 ns
340.62 ns
(61.29%)
Command_Serde/unpack/set_index📈 view plot
🚷 view threshold
121.56 ns
(+0.39%)Baseline: 121.09 ns
178.13 ns
(68.24%)
Payload_encryption/pack/remove_cmd📈 view plot
🚷 view threshold
101.25 ns
(-5.45%)Baseline: 107.09 ns
159.45 ns
(63.50%)
Payload_encryption/pack/set_cmd📈 view plot
🚷 view threshold
184.71 ns
(-4.49%)Baseline: 193.40 ns
295.85 ns
(62.43%)
Payload_encryption/unpack/remove_cmd📈 view plot
🚷 view threshold
143.76 ns
(-1.51%)Baseline: 145.96 ns
216.94 ns
(66.27%)
Payload_encryption/unpack/set_cmd📈 view plot
🚷 view threshold
203.37 ns
(-10.47%)Baseline: 227.15 ns
346.88 ns
(58.63%)
Raft_1Node_Latency/prefix/1node📈 view plot
🚷 view threshold
3,691,400.00 ns
(+1.38%)Baseline: 3,641,239.22 ns
7,759,280.23 ns
(47.57%)
Raft_1Node_Latency/read/1node📈 view plot
🚷 view threshold
23,699.00 ns
(-26.95%)Baseline: 32,441.62 ns
50,099.89 ns
(47.30%)
Raft_1Node_Latency/remove/1node📈 view plot
🚷 view threshold
305,100.00 ns
(-13.62%)Baseline: 353,221.09 ns
658,973.62 ns
(46.30%)
Raft_1Node_Latency/write/1node📈 view plot
🚷 view threshold
329,540.00 ns
(-9.37%)Baseline: 363,625.94 ns
596,980.65 ns
(55.20%)
build_snapshot/default📈 view plot
🚷 view threshold
1,802.80 ns
(-13.97%)Baseline: 2,095.53 ns
2,833.17 ns
(63.63%)
evaluate allow (identity/role/show)📈 view plot
🚷 view threshold
70,994.00 ns
(-21.18%)Baseline: 90,076.67 ns
124,752.67 ns
(56.91%)
evaluate deny (identity/role/show)📈 view plot
🚷 view threshold
9,429.70 ns
(-19.21%)Baseline: 11,672.34 ns
15,509.14 ns
(60.80%)
fernet token/project📈 view plot
🚷 view threshold
1,181.80 ns
(-6.19%)Baseline: 1,259.80 ns
1,745.35 ns
(67.71%)
get_data_keyspace📈 view plot
🚷 view threshold
0.27 ns
(-7.10%)Baseline: 0.29 ns
0.37 ns
(73.17%)
get_db📈 view plot
🚷 view threshold
0.27 ns
(-7.22%)Baseline: 0.29 ns
0.37 ns
(73.25%)
get_fernet_token_timestamp/project📈 view plot
🚷 view threshold
114.94 ns
(-11.98%)Baseline: 130.58 ns
180.03 ns
(63.85%)
get_keyspace📈 view plot
🚷 view threshold
3.54 ns
(-49.12%)Baseline: 6.96 ns
16.28 ns
(21.76%)
load all policies📈 view plot
🚷 view threshold
360,400,000.00 ns
(-15.29%)Baseline: 425,462,222.22 ns
530,006,220.01 ns
(68.00%)
🐰 View full continuous benchmarking report in Bencher

@gtema
gtema added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit f8ac6ac Nov 13, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant