Skip to content

feat(tempest): Add tempest identity compatibility verify test#998

Merged
gtema merged 1 commit into
mainfrom
claude/skaffold-identity-tempest-test-kxtqhj
Jul 16, 2026
Merged

feat(tempest): Add tempest identity compatibility verify test#998
gtema merged 1 commit into
mainfrom
claude/skaffold-identity-tempest-test-kxtqhj

Conversation

@gtema

@gtema gtema commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Add a third skaffold module that runs the tempest identity v3 test
suite against both keystone-rs and keystone-py (already deployed
side by side by the keystone module), packaged as a new
tempest-identity image built from tools/Dockerfile.tempest.

Not every v3 operation is implemented by keystone-rs yet, so this
runs and reports independently of the main verify suite
(skaffold verify -m tempest) and is invoked as a non-blocking,
continue-on-error step in CI. Failing test IDs are printed per
target in the pod logs and, in CI, collected into a job summary and
an uploaded log artifact so compatibility gaps and regressions can
be tracked over time without gating merges.

Signed-off-by: Claude noreply@anthropic.com

@github-actions

github-actions Bot commented Jul 11, 2026

Copy link
Copy Markdown

🦢 Load Test Results

Goose Attack Report

Plan Overview

Action Started Stopped Elapsed Users
Increasing 26-07-16 19:42:39 26-07-16 19:42:54 00:00:15 0 → 30
Maintaining 26-07-16 19:42:54 26-07-16 19:43:25 00:00:31 30
Decreasing 26-07-16 19:43:25 26-07-16 19:43:25 00:00:00 0 ← 30

Request Metrics

Method Name # Requests # Fails Average (ms) Min (ms) Max (ms) RPS Failures/s
DELETE DELETE /v3/auth/tokens 510 0 107.35 18 176 17.00 0.00
DELETE DELETE /v3/projects/:id (teardown) 2 0 52.00 49 55 0.07 0.00
DELETE DELETE /v3/users/:id (teardown) 3 0 55.33 41 76 0.10 0.00
GET 4682 0 100.47 71 195 156.07 0.00
GET GET /v3/auth/tokens (validate new) 510 0 106.91 72 139 17.00 0.00
GET GET /v3/projects/:id 731 0 81.99 70 117 24.37 0.00
GET GET /v3/projects/:id (catalog) 726 0 82.13 71 114 24.20 0.00
GET GET /v3/users/:id 1020 0 88.25 78 120 34.00 0.00
GET GET /v3/users/:id (catalog) 802 0 88.03 77 123 26.73 0.00
POST POST /v3/auth/tokens 508 0 80.53 70 104 16.93 0.00
Aggregated 9494 0 94.91 18 195 316.47 0.00

Response Time Metrics

Method Name 50%ile (ms) 60%ile (ms) 70%ile (ms) 80%ile (ms) 90%ile (ms) 95%ile (ms) 99%ile (ms) 100%ile (ms)
DELETE DELETE /v3/auth/tokens 110 110 110 110 110 120 130 176
DELETE DELETE /v3/projects/:id (teardown) 49 49 49 55 55 55 55 55
DELETE DELETE /v3/users/:id (teardown) 49 49 49 49 76 76 76 76
GET 93 97 100 110 160 160 160 195
GET GET /v3/auth/tokens (validate new) 110 110 110 110 110 120 130 139
GET GET /v3/projects/:id 81 83 84 85 87 90 97 117
GET GET /v3/projects/:id (catalog) 81 83 84 86 88 90 99 110
GET GET /v3/users/:id 88 89 90 92 94 96 100 120
GET GET /v3/users/:id (catalog) 87 89 90 91 94 96 100 120
POST POST /v3/auth/tokens 80 81 82 84 86 88 93 100
Aggregated 88 92 98 100 110 160 160 195

Status Code Metrics

Method Name Status Codes
DELETE DELETE /v3/auth/tokens 510 [204]
DELETE DELETE /v3/projects/:id (teardown) 2 [204]
DELETE DELETE /v3/users/:id (teardown) 3 [204]
GET 4,682 [200]
GET GET /v3/auth/tokens (validate new) 510 [200]
GET GET /v3/projects/:id 731 [200]
GET GET /v3/projects/:id (catalog) 726 [200]
GET GET /v3/users/:id 1,020 [200]
GET GET /v3/users/:id (catalog) 802 [200]
POST POST /v3/auth/tokens 508 [201]
Aggregated 508 [201], 515 [204], 8,471 [200]

Transaction Metrics

Transaction # Times Run # Fails Average (ms) Min (ms) Max (ms) RPS Failures/s
ReadHeavy
0.0 0 0 0.00 0 0 0.00 0.00
0.1 791 0 100.14 86 133 26.37 0.00
0.2 797 0 82.71 71 105 26.57 0.00
0.3 795 0 81.75 71 119 26.50 0.00
TokenLifecycle
1.0 0 0 0.00 0 0 0.00 0.00
1.1 510 0 295.89 179 370 17.00 0.00
ValidateToken
2.0 0 0 0.00 0 0 0.00 0.00
2.1 773 0 156.11 112 195 25.77 0.00
UserCRUD
3.0 0 0 0.00 0 0 0.00 0.00
3.1 0 0 0.00 0 0 0.00 0.00
3.2 1020 0 88.29 78 120 34.00 0.00
3.3 3 0 55.67 41 76 0.10 0.00
ProjectCRUD
4.0 0 0 0.00 0 0 0.00 0.00
4.1 0 0 0.00 0 0 0.00 0.00
4.2 731 0 82.03 70 117 24.37 0.00
4.3 2 0 52.00 49 55 0.07 0.00
UserRead
5.0 0 0 0.00 0 0 0.00 0.00
5.1 801 0 99.22 86 132 26.70 0.00
5.2 802 0 88.06 77 123 26.73 0.00
ProjectRead
6.0 0 0 0.00 0 0 0.00 0.00
6.1 725 0 83.19 72 112 24.17 0.00
6.2 726 0 82.16 71 114 24.20 0.00
Aggregated 8476 0 106.30 41 370 282.53 0.00

Scenario Metrics

Transaction # Users # Times Run Average (ms) Min (ms) Max (ms) Scenarios/s Iterations
ReadHeavy 7 790 265.60 252 303 26.33 112.86
TokenLifecycle 5 505 296.30 280 370 16.83 101.00
ValidateToken 4 769 156.23 139 195 25.63 192.25
UserCRUD 3 1017 88.29 78 120 33.90 339.00
ProjectCRUD 2 729 82.04 70 117 24.30 364.50
UserRead 5 799 187.82 170 223 26.63 159.80
ProjectRead 4 723 165.88 148 202 24.10 180.75
Aggregated 30 5332 168.64 70 370 177.73 1450.16

View full report

@gtema gtema marked this pull request as draft July 13, 2026 17:52
@gtema gtema force-pushed the claude/skaffold-identity-tempest-test-kxtqhj branch 2 times, most recently from f6a0b5e to 400a4e8 Compare July 16, 2026 18:11
@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown

🐰 Bencher Report

Branchclaude/skaffold-identity-tempest-test-kxtqhj
Testbedubuntu-latest
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
nanoseconds (ns)
(Result Δ%)
Upper Boundary
nanoseconds (ns)
(Limit %)
Command_Serde/apply/remove📈 view plot
🚷 view threshold
151,490.00 ns
(-26.19%)Baseline: 205,234.59 ns
669,749.02 ns
(22.62%)
Command_Serde/apply/set📈 view plot
🚷 view threshold
142,990.00 ns
(-45.17%)Baseline: 260,780.84 ns
1,244,421.28 ns
(11.49%)
Command_Serde/pack/delete📈 view plot
🚷 view threshold
121.52 ns
(-0.55%)Baseline: 122.20 ns
144.21 ns
(84.27%)
Command_Serde/pack/delete_index📈 view plot
🚷 view threshold
109.55 ns
(+0.33%)Baseline: 109.20 ns
128.16 ns
(85.48%)
Command_Serde/pack/set📈 view plot
🚷 view threshold
191.14 ns
(-3.76%)Baseline: 198.61 ns
238.72 ns
(80.07%)
Command_Serde/pack/set_index📈 view plot
🚷 view threshold
109.57 ns
(+0.44%)Baseline: 109.09 ns
127.91 ns
(85.66%)
Command_Serde/unpack/delete📈 view plot
🚷 view threshold
204.11 ns
(+5.86%)Baseline: 192.81 ns
235.89 ns
(86.53%)
Command_Serde/unpack/delete_index📈 view plot
🚷 view threshold
162.72 ns
(+0.91%)Baseline: 161.25 ns
197.41 ns
(82.43%)
Command_Serde/unpack/set📈 view plot
🚷 view threshold
275.71 ns
(+1.59%)Baseline: 271.39 ns
324.90 ns
(84.86%)
Command_Serde/unpack/set_index📈 view plot
🚷 view threshold
158.99 ns
(-0.26%)Baseline: 159.41 ns
193.75 ns
(82.06%)
Payload_encryption/pack/remove_cmd📈 view plot
🚷 view threshold
129.55 ns
(+11.58%)Baseline: 116.10 ns
139.21 ns
(93.06%)
Payload_encryption/pack/set_cmd📈 view plot
🚷 view threshold
199.78 ns
(-1.65%)Baseline: 203.13 ns
254.23 ns
(78.58%)
Payload_encryption/unpack/remove_cmd📈 view plot
🚷 view threshold
212.77 ns
(+3.64%)Baseline: 205.29 ns
255.33 ns
(83.33%)
Payload_encryption/unpack/set_cmd📈 view plot
🚷 view threshold
281.77 ns
(-1.31%)Baseline: 285.52 ns
349.14 ns
(80.70%)
Raft_1Node_Latency/prefix/1node📈 view plot
🚷 view threshold
3,523,400.00 ns
(+28.97%)Baseline: 2,732,010.78 ns
6,071,718.11 ns
(58.03%)
Raft_1Node_Latency/read/1node📈 view plot
🚷 view threshold
41,170.00 ns
(+3.81%)Baseline: 39,659.20 ns
53,331.24 ns
(77.20%)
Raft_1Node_Latency/remove/1node📈 view plot
🚷 view threshold
391,660.00 ns
(-35.69%)Baseline: 609,032.81 ns
2,461,581.69 ns
(15.91%)
Raft_1Node_Latency/write/1node📈 view plot
🚷 view threshold
409,490.00 ns
(-36.02%)Baseline: 640,057.50 ns
2,954,730.62 ns
(13.86%)
build_snapshot/default📈 view plot
🚷 view threshold
112,550.00 ns
(-2.24%)Baseline: 115,127.22 ns
158,486.28 ns
(71.02%)
fernet token/project📈 view plot
🚷 view threshold
1,483.10 ns
(+4.16%)Baseline: 1,423.84 ns
1,652.46 ns
(89.75%)
get_data_keyspace📈 view plot
🚷 view threshold
0.31 ns
(-1.86%)Baseline: 0.32 ns
0.37 ns
(84.90%)
get_db📈 view plot
🚷 view threshold
0.31 ns
(-1.96%)Baseline: 0.32 ns
0.37 ns
(84.72%)
get_fernet_token_timestamp/project📈 view plot
🚷 view threshold
136.07 ns
(-4.73%)Baseline: 142.82 ns
174.54 ns
(77.96%)
get_keyspace📈 view plot
🚷 view threshold
4.36 ns
(-9.02%)Baseline: 4.79 ns
9.62 ns
(45.37%)
🐰 View full continuous benchmarking report in Bencher

@gtema gtema force-pushed the claude/skaffold-identity-tempest-test-kxtqhj branch 2 times, most recently from 42b33a7 to 19af9fc Compare July 16, 2026 18:46
Add a third skaffold module that runs the tempest identity v3 test
suite against both keystone-rs and keystone-py (already deployed
side by side by the keystone module), packaged as a new
tempest-identity image built from tools/Dockerfile.tempest.

Not every v3 operation is implemented by keystone-rs yet, so this
runs and reports independently of the main verify suite
(skaffold verify -m tempest) and is invoked as a non-blocking,
continue-on-error step in CI. Failing test IDs are printed per
target in the pod logs and, in CI, collected into a job summary and
an uploaded log artifact so compatibility gaps and regressions can
be tracked over time without gating merges.

Assisted-by: Claude <noreply@anthropic.com>
Signed-off-by: Artem Goncharov <artem.goncharov@gmail.com>
@gtema gtema force-pushed the claude/skaffold-identity-tempest-test-kxtqhj branch from 19af9fc to 976225c Compare July 16, 2026 19:24
@gtema gtema marked this pull request as ready for review July 16, 2026 19:49
@gtema gtema enabled auto-merge July 16, 2026 19:50
@gtema gtema added this pull request to the merge queue Jul 16, 2026
Merged via the queue into main with commit 2fab90e Jul 16, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant