Support returning the full dependency tree for packages

[Aron Carroll]

The problem

Build a dependencies web service to provide a full list of all transitive dependencies for a given package, the same way npm builds a dependency tree when a user does npm install.

Background

Developers working with NodeJS use packages in their code. A package is a functional NodeJS module that includes versioning, documentation, dependencies (in the form of other packages), and more. NodeJS has a managed packages environment called npm, which regularly gets updated with new packages and new versions of existing packages.

Snyk scans NodeJS packages to identify and assist developers in remediating vulnerabilities prior to merging their code back with its project.

In order for Snyk to identify these vulnerabilities in certain packages, this is what happens:

1. The user provides the name of the package for analysis.
2. We fetch the overall set of dependencies from the relevant package manager, for the given package.
3. We compare the set of dependencies that we retrieve with our database of vulnerable packages in order to identify whether any of the dependencies are vulnerable.
4. For any package that we identify as vulnerable, we then list all available remediation paths (upgrades and/or patches for vulnerable packages) for the user.
5. The user chooses their preferred remediation actions from the list, and we apply them by creating a PR for the relevant repository.

Details

The web-server should return the full package dependency tree based on a given package name and version (user input), which we could then later use for stage 3 above.

Considerations

• There are currently over 1M packages on npmjs.com, and the number is growing all the time.
• The packages update from time to time, just as their dependencies do too.
• There are many different packages, with different edge cases. Some packages to look out for are:
  • express
  • npm
  • trucolor
  • @snyk/snyk-docker-plugin
• A user may provide invalid input (e.g. a non-existing package), causing the web-server to crash if there is no error handling in place.
• Vulnerabilities may exist in older versions of some package, but fixed in newer versions.

Design and Implementation Details

1. Extend the working web-server that, given an HTTP request containing the name of a published npm package and version, returns the entire set of dependencies for the package.
2. Present the dependencies in a tree view (e.g. JSON).
3. Account for asynchronous fetching of dependencies as you see fit.
4. Consider caching relevant data so that repeated requests resolve with minimum latency.
5. Code should be reasonably easy to read and understand.
6. Ensure there are tests of any kind (integration, unit etc.) that cover core functionality and ensure requirements are implemented correctly.
7. Consider how to handle errors.