erdogant / d3graph

Creation of interactive networks using d3 Javascript

Home Page:https://erdogant.github.io/d3graph

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Support sparse data types / make this the internal data structure for the graph

aaronzo opened this issue · comments

Summary

When loading a graph into d3graph users must provide a dense adjacency matrix. This is inconvenient because most real-world graphs are sparse, and the dense representation is memory-ineffiecient.

Furthermore, the d3graph class saves a copy of the adjacency matrix for its internal representation of the data when the .graph() method is called - but everytime this self.adjmat is used, it seems to have been more natural to have stored it in a different structure.
E.g set_edge_properties first converts the matrix into a dict (which is a sparse representation) and get_cluste_color effectively converts to an edgelist as far as I can tell (also sparse). Both of these operations are fairly inefficient and could have been avoided had a different structure been used to begin with. Adjacency matrices of graphs tend to be most useful when the algorithm we want to apply can be expressed as linear algebra, e.g. shortest paths, pagerank or traversals. Even in most of these situations, sparse matrices are usually the go-to for scalable solutions.

Impact

  • d3graph is limiting the sizes of graphs it supports by enforcing an inefficient input type.
  • d3graph is increasing its runtime due applying inefficient operations on this dense data structure when generating a figure

Possible Solutions

My preferred suggestion: since we already have networkx as a dependency, it would be wise to delegate the responsibility of storing the graph structure to that package, since:

  • it's a trusted, respected and mature project - no doubt they have applied more thought on efficiency than we could here. It's also unlikey to find bugs
  • its API is very stable
  • users will be familiar with the API
  • NetworkX already has great conversion functions to/from other data structures, meaning less inefficient conversions and boilerplate here.
  • would make the python portion of the code much shorter and more readable (adjmat2* methods could be removed)

Great suggestions! I could use a little bit of help doing this ;-)