investigate performance degradation at scale, with api/v1/vulnerability and api/v1/product
JimFuller-RedHat opened this issue · comments
loadtest revealed
Slowest page load within specified percentile of requests (in ms):
------------------------------------------------------------------------------
Name | 50% | 75% | 98% | 99% | 99.9% | 99.99%
------------------------------------------------------------------------------
GET /advisories | 1 | 2 | 3 | 4 | 6 | 11
GET /api/v1/advisory | 5 | 5 | 9 | 9 | 10 | 10
GET /api/v1/importer | 3 | 4 | 6 | 7 | 9 | 9
GET /api/v1/organization | 4 | 5 | 9 | 10 | 13 | 13
GET /api/v1/product | 73 | 80 | 97 | 100 | 120 | 120
GET /api/v1/purl | 5 | 5 | 8 | 9 | 19 | 19
GET /api/v1/purl?q=curl | 5 | 6 | 9 | 10 | 11 | 11
GET /api/v1/sbom | 4 | 5 | 9 | 11 | 11 | 11
GET /api/v1/vulnerabil.. | 1,000 | 1,000 | 1,000 | 1,000 | 1,000 | 1,000
GET /importers | 1 | 2 | 3 | 4 | 6 | 10
GET /index | 1 | 2 | 3 | 4 | 6 | 10
GET /openapi | 3 | 4 | 6 | 7 | 10 | 16
GET /packages | 1 | 2 | 3 | 4 | 6 | 11
GET /sboms | 1 | 2 | 3 | 4 | 6 | 9
POST query advisory wi.. | 4 | 5 | 7 | 8 | 12 | 18
-------------------------+--------+--------+--------+--------+--------+-------
Aggregated | 2 | 4 | 6 | 7 | 1,000 | 1,000
------------------------------------------------------------------------------
we should perform code/data analysis why api/v1/product and api/v1/vulnerability degrade
For sure. They are very complex queries, and definitely are succumbing to a 1+N queries that we could probably optimize a bit.
it looks like we are looping through some queries, for example with api/v1/product
pub async fn from_entities(
products: &[product::Model],
tx: &ConnectionOrTransaction<'_>,
) -> Result<Vec<Self>, Error> {
let mut summaries = Vec::new();
for product in products {
summaries.push(ProductSummary::from_entity(product, tx).await?);
}
Ok(summaries)
}
even if from_entity takes advantage of find_related, the top level ensures multiple calls and pushes.
same with api/v1/vulnerability
impl VulnerabilitySummary {
pub async fn from_entities(
vulnerabilities: &[vulnerability::Model],
tx: &ConnectionOrTransaction<'_>,
) -> Result<Vec<Self>, Error> {
let advisories = vulnerabilities
.load_many_to_many(advisory::Entity::find(), advisory_vulnerability::Entity, tx)
.await?;
let mut summaries = Vec::new();
for (vuln, advisories) in vulnerabilities.iter().zip(advisories.iter()) {
let cvss3 = cvss3::Entity::find()
.filter(cvss3::Column::VulnerabilityId.eq(&vuln.id))
.all(tx)
.await?;
let total_score = cvss3
.iter()
.map(|e| {
let base = Cvss3Base::from(e.clone());
base.score().value()
})
.reduce(|accum, e| accum + e);
let average_score = total_score.map(|total| Score::new(total / cvss3.len() as f64));
summaries.push(VulnerabilitySummary {
head: VulnerabilityHead::from_vulnerability_entity(vuln, tx).await?,
average_severity: average_score.map(|score| score.severity().to_string()),
average_score: average_score.map(|score| score.value()),
advisories: VulnerabilityAdvisoryHead::from_entities(vuln, advisories, tx).await?,
});
}
Ok(summaries)
}
same with api/v1/vulnerability
This is currently on my plate relating to #411
the past few weeks of work have addressed - will open a new issue based on more testing if required