lyft / cartography

Cartography is a Python tool that consolidates infrastructure assets and the relationships between them in an intuitive graph view powered by a Neo4j database.

Home Page:https://lyft.github.io/cartography/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Cardinality Issue on new datamodel

jychp opened this issue · comments

For the Hexnode source I have a situation that can be handled with the current data model:

  • A device can belong to several groups
  • A group can contain several devices

So I can't set target_node_matcher in my relationships because on none of the nodes I have a unique "foreign_id" on which I can match.

I want to define a CartographyRelSchema with a matcher like this one

target_node_matcher: TargetNodeMatcher = make_target_node_matcher(
    {'myproperty': PropertyRef('related.ids', multiple_values = True)},
)

This multiple target node should generate a cypher like this :

MATCH (n:NodeLabel) WHERE n.myproperty IN $data.related.ids [..] MERGE (parent)-[:MYREL]->(n)

Instead of this one (which I assume is used for single node target)

MATCH (n:NodeLabel {myproperty: $data.related.id}) [..] MERGE (parent)-[:MYREL]->(n)

My initial thought is to use a transform function to reshape the data into single node targets so that it works with the model. I don't forsee any scaling issues as the neo4j load is actually pretty fast. This is the way that the code currently behaves in many (most?) places.

To be more illustrative, let's use your example of

A device can belong to several groups
A group can contain several devices

If we were to use no data model I think we would implement this in raw cypher like

unwind devices as device
  merge (d:Device{id: device.Id})
  ...
  match (g:Group{id:d.group})
  merge(d)-[:ASSOCIATION]->(g)

and

unwind groups as group
  merge (g:Group{id: group.Id})
  ...
  match (d:Device{id:g.device})
  merge(d)-[:ANOTHER_ASSOCIATION]->(g)

In these cases, I think it'd be cleanest using the single node target style anyway.

Happy to talk about this on Slack if that's easier too.