Investigate installation performance bugs with many resources

[L. Fernando De Pombo]

Investigate installation performance bugs with many resources

[David Ellis]

Two major sources:

1. At large scale, the diff algorithm scales poorly (roughly O(n^2) normally, but becomes much worse at large n when the GC goes into overdrive to keep the memory usage below ~512MB (despite us telling V8 to let it go up to 8GB...) This has been fixed in this commit by reworking the algorithm in use to be O(n) and reduce recomputation of set information.
2. At all scales, the TypeORM read logic is not optimized for batch reads, and sub-entities are queried one-at-a-time with no caching. There's nothing we can do here, according to the official docs, besides writing our own batch read mechanism, either completely from scratch or digging into the TypeORM internals, both of which have significant downsides. There also appears to be a lot of GC action during this phase, so we might be able to speed it up some if we can figure out the V8 flags we need to set, but it appears to be mostly IO-bound (by splitting the Postgres queries up the way it does), so that would only be a minor fix on that front.

[David Ellis]

Third issue:

3. I haven't fully debugged it, yet, but there appears to be a similar slowdown as iasql_install within the sync path of the iasql_commit logic.

[David Ellis]

Strangely couldn't reproduce the number 3 slowdown when I manually did the same sorts of things in staging with a local IaSQL.

It takes more time than I'd like (the extra checks our apply and sync loops do slow down the operation), but the flamegraphs look reasonable to me.

[David Ellis]

An annotated flame graph of my findings. It all looks reasonable to me?