nmervaillie/fix-your-microservices

Get the sample application and build it

git clone https://github.com/sqshq/PiggyMetrics
mvn clean package -DskipTests -f PiggyMetrics/pom.xml

Install jQAssistant

curl -O https://jqassistant.org/wp-content/uploads/2018/12/jqassistant-commandline-neo4jv3-1.6.0-distribution.zip
unzip jqassistant-commandline-neo4jv3-1.6.0-distribution.zip
mv jqassistant-commandline-neo4jv3-1.6.0 jqassistant

Scan the example application

./jqassistant/bin/jqassistant.sh scan -p config/scan.properties -f PiggyMetrics/account-service/target/account-service.jar,PiggyMetrics/auth-service/target/auth-service.jar,PiggyMetrics/notification-service/target/notification-service.jar,PiggyMetrics/statistics-service/target/statistics-service.jar,PiggyMetrics/config/target/config.jar

By default, jQAssistant will create an embedded database, which can be started with ./jqassistant/bin/jqassistant.sh server

If you want to import to you locally running database, add these arguments to the command line when scanning -storeUri bolt://localhost -storeUsername neo4j -storePassword password

We’ll use the latter in our examples with a local neo4j DB accessible on http://localhost:7474, with the APOC library installed.

Running some rules

jQAssistant comes out of the box with some plugins that help us, such as the Spring plugin, so we can use it to enrich our existing database to create some additional concepts like REST controllers, repositories, etc

./jqassistant/bin/jqassistant.sh analyze -concepts classpath:Resolve -storeUri bolt://localhost -storeUsername neo4j -storePassword password
./jqassistant/bin/jqassistant.sh analyze -groups spring-boot:Default -storeUri bolt://localhost -storeUsername neo4j -storePassword password

We can explore the graph and check spring repositories are present with

MATCH (r:Spring:Repository)
WHERE r.fqn STARTS WITH 'com'
RETURN *

From now on, you can run the queries below manually, or use the script runAnalysis.sh to run them automatically.

Apply some higher level concepts

Create the microservice concept and compute a user friendly artifact name for it

Tags jars containing REST controllers as microservices

MATCH (a:Artifact)--(cls:Class)-[:ANNOTATED_BY]->(ann:Annotation)-[:OF_TYPE]->(:Type{name:"RestController"})
SET a:Microservice
SET a.serviceName = reverse(split(a.fileName, '/'))[0]
RETURN a

Tags spring controller methods with the Endpoint concept, and adds URL and HTTP method information

MATCH (cls:Class)-[:DECLARES]->(endpoint)-[:ANNOTATED_BY]->(ann:Annotation)-[:OF_TYPE]->(:Type{name:"RequestMapping"})
  WHERE cls.fqn starts with 'com.'
OPTIONAL MATCH (ann)-[:HAS]->(:Value{name:"value"})-[:CONTAINS]->(url:Value)
OPTIONAL MATCH (ann)-[:HAS]->(:Value{name:"path"})-[:CONTAINS]->(path:Value)
OPTIONAL MATCH (ann)-[:HAS]->(:Value{name:"method"})-[:CONTAINS]->()-[:IS]->(httpMethod:Field)
OPTIONAL MATCH (cls)-[:ANNOTATED_BY]->(classMapping:Annotation)-[:OF_TYPE]->(Type{name:"RequestMapping"}),(classMapping)-[:HAS]->(:Value{name:"value"})-[:CONTAINS]->(classLevelUrl:Value)
SET endpoint:Endpoint
SET endpoint.method=split(httpMethod.signature, " ")[1]
SET endpoint.url=coalesce(classLevelUrl.value, '') + coalesce(url.value, '') + coalesce(path.value, '')
RETURN cls.fqn, endpoint.url, endpoint.method

Note	Notice that the URL can come from both class level and method level RequestMapping

Tags service client methods with the FeignMethod concept, and adds URL and HTTP method information

MATCH (client:Interface)-[:DECLARES]->(m:Method)
  WHERE client.fqn STARTS WITH "com."
  AND (client)-[:ANNOTATED_BY]->()-[:OF_TYPE]->(:Type{fqn:"org.springframework.cloud.openfeign.FeignClient"})
MATCH (m)-[:ANNOTATED_BY]->(ann:Annotation)-[:HAS]->(:Value{name:"value"})-[:CONTAINS]->(url:Value)
MATCH (m)-[:ANNOTATED_BY]->(ann:Annotation)-[:HAS]->(:Value{name:"method"})-[:CONTAINS]->()-[:IS]->(httpMethod:Field)
SET m:FeignClient
SET m.url = apoc.text.regreplace(url.value, '\\{.*\\}', '{}')
SET m.httpMethod = split(httpMethod.signature, ' ')[1]
return m.name, m.httpMethod, m.url

Our services are deployed under a specific URL defined in the config service (in YAML configuration files). So we need to add this to our controllers this part of the path to get the full URL

Enriches the endpoints URLs with the deployment URLs parts of the services

MATCH (configJar:Artifact) WHERE configJar.fileName CONTAINS 'config.jar'
MATCH (configJar)-[:CONTAINS]->(f:File:YAML)-[*]->(k:YAML:Key{fqn: 'server.servlet.context-path'})--(path:Value)
WITH reverse(split(replace(f.fileName, '.yml', ''), '/'))[0] as serviceName, path.value as urlPrefix
MATCH (serviceJar:Artifact)-[*]->(sn:YAML:Key{fqn: 'spring.application.name'})--(appName:Value)
WHERE appName.value = serviceName
MATCH (serviceJar)-[:CONTAINS|DECLARES*..2]->(endpoint:Endpoint)
SET endpoint.fullUrl = urlPrefix + apoc.text.regreplace(endpoint.url, '\\{.*\\}', '{}')
RETURN distinct serviceName, endpoint.url, endpoint.fullUrl

Now we have a the Endpoint and FeignClient concepts, we can link them together

Creates a relationship between the services and their clients based on URLs and HTTP methods

MATCH (client:FeignClient), (endpoint:Endpoint)
WHERE client.url=endpoint.fullUrl AND client.httpMethod=endpoint.method
MERGE (client)-[:INVOKES_REMOTE]->(endpoint)
RETURN client.url, endpoint.fullUrl

Now we can do cross service dependency analysis with:

MATCH (sa:Artifact)-[:CONTAINS]-(caller:Type)-[:DECLARES]-(client)-[:INVOKES_REMOTE]->(endpoint:Endpoint)
MATCH (endpoint)-[:DECLARES]-(ctrl:Type)-[:CONTAINS]-(ta:Artifact)
RETURN *

Interface / implementation missing links

To go further in the analysis, we need to interface/dependency method invocation info missing by adding VIRTUAL_INVOKES rels

Links methods on interfaces to corresponding methods in implem classes with a VIRTUAL_INVOKES rel

MATCH (itf:Interface)<-[:IMPLEMENTS]-(impl:Type)
MATCH (itf)-[:DECLARES]->(m1:Method)
MATCH (impl)-[:DECLARES]->(m2:Method)
WHERE itf.fqn STARTS WITH 'com.piggy'
AND m1.signature = m2.signature
MERGE (m1)-[:VIRTUAL_INVOKES]-(m2)
RETURN m1.signature, m2.signature

Entity analysis

Applies Entity and MongoDb labels on mongo entities, adds the collectionName on the entity

MATCH (entity:Type)-[:ANNOTATED_BY]->(ann:Annotation)
MATCH (ann)-[:OF_TYPE]-(:Type{fqn:'org.springframework.data.mongodb.core.mapping.Document'})
MATCH (ann)-[:HAS]->(collection:Value{name:"collection"})
SET entity:Entity:MongoDb
SET entity.collectionName=collection.value
RETURN entity.fqn

Which collections are used by service?

MATCH (entity:MongoDb:Class)--(a:Artifact)
RETURN entity.fqn as class, entity.collectionName as collection, a.serviceName as usedBy
ORDER by collection

Endpoints using repository methods

MATCH p=(ep:Endpoint)-[:INVOKES|VIRTUAL_INVOKES|INVOKES_REMOTE*]->(m)<--(r:Repository)
RETURN r.name, m.signature, collect(ep.method +' '+ ep.fullUrl) as usedBy
ORDER BY r.name

MATCH p=(ep:Endpoint)-[:INVOKES|VIRTUAL_INVOKES*..10]->(m)<--(r:Repository)
RETURN ep.fullUrl, ep.method, collect(r.name+'::'+m.signature)
ORDER BY ep.fullUrl

MATCH p=(ep:Endpoint)-[:INVOKES|VIRTUAL_INVOKES*..10]->(m)<--(r:Repository)
RETURN r.name, m.signature, collect(ep.method +' '+ ep.fullUrl) as endpoints
ORDER BY r.name

Fallbacks

Do my HTTP clients declare fallbacks?

MATCH (client:Interface)-[:ANNOTATED_BY]->(a)-[:OF_TYPE]->(t:Type{fqn:"org.springframework.cloud.openfeign.FeignClient"})
OPTIONAL MATCH (a)-[:HAS]-(v:Value{name:'fallback'})--(fb:Type)
RETURN client.fqn as client, fb.fqn as fallback

Documentation

Let’s add some sample documentation to the application (it has no doc out of the box)

cp test-files/api-docs.yml PiggyMetrics/statistics-service/src/main/resources

and rebuild the services and rescan the app as done before

Now we can check if my services have some documentation

Do the services have api specifications?

MATCH (a:Artifact)
OPTIONAL MATCH (a)-[:CONTAINS]->(f:File)--(doc:Document:YAML)--(key:Key{name:'openapi'})
RETURN distinct a.serviceName, f.fileName

Extract endpoints and parameters from apidoc

MATCH (a:Artifact)-[:CONTAINS]->(f:File)--(doc:Document:YAML)--(key:Key{name:'openapi'})
MATCH (doc)-->(:Key{name:'paths'})-->(path:Key)--(method:Key)
OPTIONAL MATCH (method)-[*2]-(:Key{name:'name'})--(val:Value)
RETURN path.name, method.name, collect(val.value) as params

Get the controller parameters and return values

MATCH (ep:Endpoint)-[:RETURNS]->(returnType:Type)
OPTIONAL MATCH (ep)-[:HAS]->(param:Parameter)-[:ANNOTATED_BY]->(:Annotation)
OPTIONAL MATCH (param)-[:OF_TYPE]->(type:Type)
RETURN ep.fullUrl, ep.method, count(param), collect(type.name)

Export as GraphML

Creates a GraphML report for artifact dependencies.

MATCH (source:Artifact)-[*]->(c:FeignClient)
MATCH (c)-[:INVOKES_REMOTE]->(:Endpoint)<-[*]-(target:Artifact)
RETURN distinct source,
    { role: "relationship", type: "DEPENDS_ON",
        startNode: source, endNode: target, properties: {test: "blah"}
    } as rel, target

nmervaillie / fix-your-microservices