Giters

Kinto / kinto-signer

Digital signatures to guarantee integrity and authenticity of collections of records.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Invalidate fewer things

glasserc opened this issue · comments

commented

@autrilla noticed that the CPU/request spikes we're encountering on Remote Settings seem to line up unusually well with the refresh_signature lambda from mozilla-services/remote-settings-lambdas. He noticed that each time the lambda runs, it causes a bunch of Cloudfront invalidations. Each Cloudfront invalidation is for /v1/*, which would naturally lead to an increase in requests. Since the refresh_signature lambda causes each collection to be resigned, and each resignature invalidates /v1/*, we end up needlessly recomputing far too many requests.

Is it possible to only invalidate paths under the collection? The only other path that I think could be affected is /buckets/monitor/collection/changes, which we could explicitly add. Are there other exceptions?

commented

We had this in the past

We rolled this back because we were seeing many TooManyInvalidationsInProgressexceptions. But we still have with /v1/* anyway so... we could give it a try again. It makes a lot more sense to invalidate specific paths.

Also, note that we invalidate cache only for Firefox 62 users (ie. Firefox with broadcast enabled without cache busting).

What about disabling caching invalidation and use Normandy to disable broadcast on Firefox 62?

commented

Talking about this more with Mat online, he realized that actually, cache invalidation isn't necessary for resigning collections -- the data is the same and matches both the old signature and the new signature. We only need to invalidate caches during change approval. If we don't invalidate signatures during resign, the problem should become much less drastic. Then we can combine that with only invalidating signatures on collections with changes, and the additional load during a change should be manageable. So that's our plan.