TC-3071 scale test for put and patch advisory lables #78

bxf12315 · 2025-11-03T14:45:13Z

No description provided.

ctron

Running that change over and over again, it would always set the same value on the same entity.

I think we should have this a bit more random. At least when it comes to the values. One idea could be to use a timestamp as a value.

Another idea could be to pick a random advisory, instead of using only a single (configured) one.

bxf12315 · 2025-11-12T06:01:38Z

Running that change over and over again, it would always set the same value on the same entity.

I think we should have this a bit more random. At least when it comes to the values. One idea could be to use a timestamp as a value.

Another idea could be to pick a random advisory, instead of using only a single (configured) one.

I chose the second approach. Please review it again.

ctron · 2025-11-12T08:13:56Z

Another idea could be to pick a random advisory, instead of using only a single (configured) one.

I chose the second approach. Please review it again.

Great. Sorry for not being clear enough. I think in this case the values should not be configured. There seems to be no benefit of doing that. But simply always fetch it from the database.

bxf12315 · 2025-11-26T12:19:20Z

The corresponding pull request. Thanks
guacsec/trustify-scale-test-runs#30

ctron · 2025-11-26T15:55:36Z

The corresponding pull request. Thanks guacsec/trustify-scale-test-runs#30

I am not sure how this relates to the request of not having the value configured, but evaluated from the database?

bxf12315 · 2025-11-27T00:46:00Z

The corresponding pull request. Thanks guacsec/trustify-scale-test-runs#30

I am not sure how this relates to the request of not having the value configured, but evaluated from the database?

If put_advisory_labels and patch_advisory_labels are empty in the scenario, these two methods will be skipped. Does trustify-scale-test-run regenerate the scenario file on every run? Do you mean removing put_advisory_labels and patch_advisory_labels from the scenario and reading them from the database each time we run the scale test? However, in this case, no database is specified when the test is executed.
At the moment, the logic in this PR is to query 20 IDs from the database for put_advisory_labels and patch_advisory_labels. Then, here (https://github.com/guacsec/trustify-scale-testing/pull/78/files#diff-6991c4c6d61c5d119cd2983b5f8f30c3b5681dddb103d2978c7ab4175b2567f2R153), these IDs are read and used. As the scale test currently reads its data from the scenario, it seems logical to write those IDs into the scenario file. Would this approach be considered incorrect? If not, could you please provide some more detailed clarification? Thanks.

ctron · 2025-11-27T07:13:41Z

Another idea could be to pick a random advisory, instead of using only a single (configured) one.

I chose the second approach. Please review it again.

Great. Sorry for not being clear enough. I think in this case the values should not be configured. There seems to be no benefit of doing that. But simply always fetch it from the database.

Right, this is what I tried to say here. Those values should not be configured. But just picked randomly from the database, each time it runs. I think this could be done by using the API. Or it could be done by directly accessing the database. I think using the API might make it easier to run, as no DB connection would be required.

It may be beneficial to split this up into multiple steps.

bxf12315 · 2025-11-27T09:11:14Z

Another idea could be to pick a random advisory, instead of using only a single (configured) one.

I chose the second approach. Please review it again.

Great. Sorry for not being clear enough. I think in this case the values should not be configured. There seems to be no benefit of doing that. But simply always fetch it from the database.

Right, this is what I tried to say here. Those values should not be configured. But just picked randomly from the database, each time it runs. I think this could be done by using the API. Or it could be done by directly accessing the database. I think using the API might make it easier to run, as no DB connection would be required.

It may be beneficial to split this up into multiple steps.

Does this mean that before performing each put or patch action, an ID needs to be queried first, just like it is done when uploading and deleting advisories?
Do we need to retain the put_advisory_labels and patch_advisory_labels attributes in the scenario? If so, should the corresponding advisory IDs be randomly fetched via REST API at the start of the scale test and assigned to these attributes? Is that correct?

bxf12315 · 2025-11-27T15:01:05Z

The latest commit randomly selects an advisory ID and then performs put or patch operations on it. However, it results in two requests being made.

That seems reasonable as a benchmark, don’t you think?

ctron · 2025-11-28T10:01:07Z

scenarios/full-20250604.json5

  "download_advisory": "24ae57c3-4b57-4f4e-82c1-83ae26059a89",
-  "get_advisory": "24ae57c3-4b57-4f4e-82c1-83ae26059a89"
+  "get_advisory": "24ae57c3-4b57-4f4e-82c1-83ae26059a89",
+  "put_advisory_lables": [


Please remove those arrays.

ctron · 2025-11-28T10:02:41Z

src/restapi.rs

+    user: &mut GooseUser,
+) -> Result<String, Box<TransactionError>> {
+    // Generate random 3-digit offset
+    let offset = rand::thread_rng().gen_range(0..=999);


Instead of assuming a maximum number, you can (on startup) query the total number.

please check again.

ctron · 2025-12-10T08:34:07Z

src/main.rs

                            delete_counter.clone()
-                        ), name: format!("delete_sbom_from_pool_sequential[{} SBOMs]", pool.len()))
+                        ),
+                         name: format!("delete_sbom_from_pool_sequential[{} SBOMs]", pool.len()))


I am not sure "pool" is the right term now.

ctron · 2025-12-10T08:34:29Z

src/restapi.rs

-use goose::goose::{GooseUser, TransactionResult};
+use goose::goose::{GooseMethod, GooseRequest, GooseUser, TransactionError, TransactionResult};
+use reqwest::{Client, RequestBuilder};
+


Please remove that empty lone and have it sorted.

ctron · 2025-12-10T08:35:59Z

src/restapi.rs

 use urlencoding::encode;

+// Static variable to store advisory total count
+static ADVISORY_TOTAL: OnceCell<u64> = OnceCell::const_new();


Instead of having a global static, wouldn't it be possible to perform this lookup once during the start of the application and pass it in to functions?

ctron · 2025-12-10T08:40:50Z

src/restapi.rs

+
+/// Send Advisory labels request using PUT method
+pub async fn put_advisory_labels(user: &mut GooseUser) -> TransactionResult {
+    let advisory_id = list_advisory_random_single(user).await?;


Having this as part of the function will pollute the measurements for this operation. as it does add more: the loading of the advisory and possibly the loading of all advisories.

To my understanding, this could be split into two steps and information be passed along as GooseUserData.

As for Goose, its transactions seem to be executed in a random order — may I ask how we can ensure they are executed sequentially? Wouldn’t this add more complexity?
The best approach would actually be to read a certain number of IDs from the database and then perform operations on them randomly, just like in the first version of the PR.

As mentioned (with a link) on the other PR, I don't think that's the case. The round robin schedule (which is default) keeps the order we define.

The idea of the tests is to see how operations change over time. If we combine operations, we might not be aware of changes, or which operation is the cause for a change. As Goose seems to support splitting it up and supporting handover of state, we should leverage that.

Do you mean exchanging data through session? Session values are only valid for the current user, but these actions should be callable by different users, right?

Where is the link you mentioned earlier? Could you share it.

I'm a bit confused about a few points and would appreciate your clarification:

1. This PR does not involve uploading advisories; the advisory upload is handled in a separate PR.

Yes it is. It's unfortunate, as it could go together with this one.

2. If we do not use `GooseScheduler::Serial`, how can we ensure that looking up the advisory ID is always executed before the PUT or PATCH request? In my testing, after setting parameters via `GooseUserData`, reading them back is not guaranteed to happen on the same `GooseUser`.（`GooseScheduler:: RoundRobin `）

To my understanding of https://book.goose.rs/config/scheduler.html#round-robin-scheduler ... this is what would work for as, and is the default. Serial would work too, but it not the default.

3. In this PR, there are three operations: * Compute the total number of advisories * Randomly select one advisory ID * Perform a PUT or PATCH on the labels for this ID Should these three operations be grouped into a dedicated `Scenario`?

The compute part is only required to limit the test. It should only be executed once. Therefore, there is no need to have this as part of the metrics. Randomly selecting an advisory is required with every operation. However its time to run should not pollute the time of the actual operation.

And in the "upload advisory" case, would there then be two operations: * Upload the advisory file * Delete this file

If using GooseUserData is not appropriate for the second point, what would be the recommended way to pass parameters between these steps?

I think that is the appropriate way.

However, GooseUserData does not solve this problem, because it is only valid for the current GooseUser, and in different transactions there may be different GooseUsers executing the steps.

However, GooseUserData does not solve this problem, because it is only valid for the current GooseUser, and in different transactions there may be different GooseUsers executing the steps.

I am not sure how you come to this conclusion. I created an example which should show how to use it and which should prove it does what we require: https://github.com/ctron/goose-session-example

Please let me know if there are any flaws in this example.

Okay, give me a bit of time to make the changes.

After rechecking the previous code related to GooseUserData, I found that set_on_start is the key factor. If GooseUserData is not set in a set_on_start transaction, it is very hard to ensure that the data is later read from the same GooseUser.

I have already made a new revision following the logic we discussed earlier; could you please review it? If it looks good to you, I will update the upload Advisory PR accordingly.

ctron · 2025-12-15T07:53:45Z

src/restapi.rs

 use urlencoding::encode;

+/// Get advisory total count
+pub async fn get_advisory_total(user: &mut GooseUser) -> TransactionResult {


Please don't do this is a transaction, but once on the startup of the application (if required) and pass in the value as u64 to functions.

I have already moved get_advisory_total out of the transaction, but running it in the transaction startup should be a more reasonable approach. This is the result of running get_advisory_total in the transaction startup.get_advisory_total is not included in the metrics list.

New version code, a lot of additional logic checks were introduced, and a new environment variable is also required. Do you think this level of change is really worth it?

ctron · 2025-12-15T07:55:32Z

src/restapi.rs

+/// List advisory with random offset and limit=1
+pub async fn list_advisory_random_single(user: &mut GooseUser) -> TransactionResult {


Suggested change

/// List advisory with random offset and limit=1

pub async fn list_advisory_random_single(user: &mut GooseUser) -> TransactionResult {

/// Get a random advisory ID

pub async fn find_random_advisory(user: &mut GooseUser) -> TransactionResult {

ctron · 2025-12-15T07:56:49Z

src/restapi.rs

+            if let Some(id) = first_item.get("uuid").and_then(|u| u.as_str()) {
+                log::info!("Listing advisory with offset {}: {}", offset, id);
+
+                let goose_user_data =


If possible, please re-use the user data from the top of the function.

ctron · 2025-12-15T07:58:29Z

src/restapi.rs

+pub async fn list_advisory_random_single(user: &mut GooseUser) -> TransactionResult {
+    // Get the total count first
+    let total = {
+        let goose_user_data = user


The code for getting the user data is used quite often in the same way. Please extract this into an own function. As part of the GooseUserData struct.

src/utils.rs

ctron · 2025-12-16T08:57:45Z

src/restapi.rs

-    let response = user.get("/api/v2/advisory").await?;
-    let json_data = response.response?.json::<serde_json::Value>().await?;
+/// Fetch advisory total count once at application startup
+pub async fn get_advisory_total(host: Option<String>) -> Result<u64, anyhow::Error> {


To me it looks like the value is never None, can't we just use host: &str?

ctron · 2025-12-16T08:58:18Z

src/restapi.rs

+
+    log::info!("Fetching advisory total from: {}", url);
+
+    let response = client


I think it you don't re-use the client, you can also use request::get directly.

src/main.rs

ctron · 2025-12-16T09:56:14Z

src/main.rs

+            .set_weight(5)?;
+            // Register advisory label transactions if host is available.
+            // Since the scenario object doesn't provide host information, we use environment
+            let host = s.host.clone().or_else(|| std::env::var("HOST").ok());


Is there maybe a way to get this from the Goose configuration?

I have tried for a long time and so far have not found a way to obtain it from the Goose configuration.

bxf12315 requested a review from ctron November 3, 2025 14:47

ctron reviewed Nov 11, 2025

View reviewed changes

TC-3071 scale test for put and patch advisory lables

459689b

bxf12315 force-pushed the TC-3071-modify branch from daeefa1 to e0e85e5 Compare November 12, 2025 05:59

bxf12315 force-pushed the TC-3071-modify branch from e0e85e5 to 2f72b0a Compare November 12, 2025 07:04

bxf12315 mentioned this pull request Nov 24, 2025

TC-3071 scale test for put and patch advisory lables guacsec/trustify-scale-test-runs#30

Open

Add a feature to randomly retrieve IDs from the ID pool.

7e12e20

bxf12315 force-pushed the TC-3071-modify branch from 2f72b0a to 7e12e20 Compare November 27, 2025 01:08

ctron reviewed Nov 28, 2025

View reviewed changes

TC-3071 randomly fetch advisory ID

e195f7d

bxf12315 force-pushed the TC-3071-modify branch from 63d075b to e195f7d Compare November 28, 2025 14:36

ctron mentioned this pull request Dec 1, 2025

add scale test for sbom #79

Draft

ctron reviewed Dec 10, 2025

View reviewed changes

bxf12315 force-pushed the TC-3071-modify branch from 5332deb to 5c953b9 Compare December 14, 2025 13:34

Use GooseUserData to pass data

bd3bdbb

bxf12315 force-pushed the TC-3071-modify branch from 5c953b9 to bd3bdbb Compare December 14, 2025 14:54

ctron reviewed Dec 15, 2025

View reviewed changes

ctron reviewed Dec 16, 2025

View reviewed changes

src/main.rs Outdated Show resolved Hide resolved

ctron reviewed Dec 16, 2025

View reviewed changes

src/main.rs Outdated Show resolved Hide resolved

ctron reviewed Dec 16, 2025

View reviewed changes

src/main.rs Outdated Show resolved Hide resolved

ctron reviewed Dec 16, 2025

View reviewed changes

bxf12315 force-pushed the TC-3071-modify branch from 6614a9a to e44f4b0 Compare December 16, 2025 10:19

Do not run get_advisory_total within a single transaction

8cd204e

bxf12315 force-pushed the TC-3071-modify branch from e44f4b0 to 8cd204e Compare December 16, 2025 10:23

		/// List advisory with random offset and limit=1
		pub async fn list_advisory_random_single(user: &mut GooseUser) -> TransactionResult {


		log::info!("Fetching advisory total from: {}", url);

		let response = client

TC-3071 scale test for put and patch advisory lables #78

Are you sure you want to change the base?

TC-3071 scale test for put and patch advisory lables #78

Uh oh!

Conversation

bxf12315 commented Nov 3, 2025

Uh oh!

ctron left a comment

Choose a reason for hiding this comment

Uh oh!

bxf12315 commented Nov 12, 2025

Uh oh!

ctron commented Nov 12, 2025

Uh oh!

bxf12315 commented Nov 26, 2025

Uh oh!

ctron commented Nov 26, 2025

Uh oh!

bxf12315 commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ctron commented Nov 27, 2025

Uh oh!

bxf12315 commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bxf12315 commented Nov 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bxf12315 commented Nov 27, 2025 •

edited

Loading

bxf12315 commented Nov 27, 2025 •

edited

Loading