investigate performance degradation at scale, with api/v1/vulnerability and api/v1/product

[Jim Fuller]

loadtest revealed

 Slowest page load within specified percentile of requests (in ms):
 ------------------------------------------------------------------------------
 Name                     |    50% |    75% |    98% |    99% |  99.9% | 99.99%
 ------------------------------------------------------------------------------
 GET /advisories          |      1 |      2 |      3 |      4 |      6 |     11
 GET /api/v1/advisory     |      5 |      5 |      9 |      9 |     10 |     10
 GET /api/v1/importer     |      3 |      4 |      6 |      7 |      9 |      9
 GET /api/v1/organization |      4 |      5 |      9 |     10 |     13 |     13
 GET /api/v1/product      |     73 |     80 |     97 |    100 |    120 |    120
 GET /api/v1/purl         |      5 |      5 |      8 |      9 |     19 |     19
 GET /api/v1/purl?q=curl  |      5 |      6 |      9 |     10 |     11 |     11
 GET /api/v1/sbom         |      4 |      5 |      9 |     11 |     11 |     11
 GET /api/v1/vulnerabil.. |  1,000 |  1,000 |  1,000 |  1,000 |  1,000 |  1,000
 GET /importers           |      1 |      2 |      3 |      4 |      6 |     10
 GET /index               |      1 |      2 |      3 |      4 |      6 |     10
 GET /openapi             |      3 |      4 |      6 |      7 |     10 |     16
 GET /packages            |      1 |      2 |      3 |      4 |      6 |     11
 GET /sboms               |      1 |      2 |      3 |      4 |      6 |      9
 POST query advisory wi.. |      4 |      5 |      7 |      8 |     12 |     18
 -------------------------+--------+--------+--------+--------+--------+-------
 Aggregated               |      2 |      4 |      6 |      7 |  1,000 |  1,000
 ------------------------------------------------------------------------------

we should perform code/data analysis why api/v1/product and api/v1/vulnerability degrade

[Bob McWhirter]

For sure. They are very complex queries, and definitely are succumbing to a 1+N queries that we could probably optimize a bit.

[Jim Fuller]

it looks like we are looping through some queries, for example with api/v1/product

    pub async fn from_entities(
        products: &[product::Model],
        tx: &ConnectionOrTransaction<'_>,
    ) -> Result<Vec<Self>, Error> {
        let mut summaries = Vec::new();

        for product in products {
            summaries.push(ProductSummary::from_entity(product, tx).await?);
        }

        Ok(summaries)
    }

even if from_entity takes advantage of find_related, the top level ensures multiple calls and pushes.

[Jim Fuller]

same with api/v1/vulnerability

impl VulnerabilitySummary {
    pub async fn from_entities(
        vulnerabilities: &[vulnerability::Model],
        tx: &ConnectionOrTransaction<'_>,
    ) -> Result<Vec<Self>, Error> {
        let advisories = vulnerabilities
            .load_many_to_many(advisory::Entity::find(), advisory_vulnerability::Entity, tx)
            .await?;

        let mut summaries = Vec::new();

        for (vuln, advisories) in vulnerabilities.iter().zip(advisories.iter()) {
            let cvss3 = cvss3::Entity::find()
                .filter(cvss3::Column::VulnerabilityId.eq(&vuln.id))
                .all(tx)
                .await?;

            let total_score = cvss3
                .iter()
                .map(|e| {
                    let base = Cvss3Base::from(e.clone());
                    base.score().value()
                })
                .reduce(|accum, e| accum + e);

            let average_score = total_score.map(|total| Score::new(total / cvss3.len() as f64));

            summaries.push(VulnerabilitySummary {
                head: VulnerabilityHead::from_vulnerability_entity(vuln, tx).await?,
                average_severity: average_score.map(|score| score.severity().to_string()),
                average_score: average_score.map(|score| score.value()),
                advisories: VulnerabilityAdvisoryHead::from_entities(vuln, advisories, tx).await?,
            });
        }

        Ok(summaries)
    }

[Jim Crossley]

same with api/v1/vulnerability

This is currently on my plate relating to #411

[Jim Fuller]

the past few weeks of work have addressed - will open a new issue based on more testing if required