Please add this working germlin server conf to the documentation
veqryn opened this issue · comments
I need to access the graph database with Golang and other languages, which means I need to have gremlin server running.
After banging my head against the wall for several hours, I finally got a working config.
Can you please confirm it works, and add it to the documentation.
Would be great if it was supported going forward as well.
gremlin-server-sqlg-postgres.yaml
host: gremlin-server
port: 8182
evaluationTimeout: 30000
# channelizer: org.apache.tinkerpop.gremlin.server.channel.HttpChannelizer
graphs: {
graph: /path/to/gremlin-server-sqlg-postgres.properties
}
scriptEngines: {
gremlin-groovy: {
plugins: { org.apache.tinkerpop.gremlin.server.jsr223.GremlinServerGremlinPlugin: {},
org.umlg.sqlg.groovy.plugin.SqlgPostgresGremlinPlugin: {},
org.apache.tinkerpop.gremlin.groovy.jsr223.GroovyCompilerGremlinPlugin: {enableThreadInterrupt: true},
org.apache.tinkerpop.gremlin.jsr223.ImportGremlinPlugin: {classImports: [java.lang.Math], methodImports: [java.lang.Math#*]},
org.apache.tinkerpop.gremlin.jsr223.ScriptFileGremlinPlugin: {files: [/path/to/gremlin-server-sqlg-postgres.groovy]}
}
}
}
serializers:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { ioRegistries: [org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerIoRegistryV3d0] }} # application/vnd.gremlin-v3.0+gryo
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { serializeResultToString: true }} # application/vnd.gremlin-v3.0+gryo-stringd
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0, config: { ioRegistries: [org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerIoRegistryV3d0] }} # application/json
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1 } # application/vnd.graphbinary-v1.0
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1, config: { serializeResultToString: true }} # application/vnd.graphbinary-v1.0-stringd
metrics: {
consoleReporter: {enabled: false, interval: 180000},
csvReporter: {enabled: false, interval: 180000, fileName: /tmp/gremlin-server-metrics.csv},
jmxReporter: {enabled: false},
slf4jReporter: {enabled: true, interval: 180000}
}
strictTransactionManagement: false
idleConnectionTimeout: 0
keepAliveInterval: 0
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 65536
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64gremlin-server-sqlg-postgres.properties
jdbc.url=jdbc:postgresql://hostaddress:5432/databaseschema
jdbc.username=username
jdbc.password=password
gremlin.graph=org.umlg.sqlg.structure.SqlgGraphgremlin-server-sqlg-postgres.groovy
def globals = [:]
globals << [g : graph.traversal()]To run:
/opt/gremlin-server/bin/gremlin-server.sh install org.umlg sqlg-postgres 2.1.5
/opt/gremlin-server/bin/gremlin-server.sh /path/to/gremlin-server-sqlg-postgres.yaml
Yes I can add it to the documentation.
Generally I am hesitant about gremlin-server as so many features are only accessible from java at present. In particular bulk loading.
Does this not concern you before investing too much time into Sqlg?
I guess if I need bulk loading, I will have to find some script in java for that. But everything else is in golang, and that isn't about to change.
I think it would be a great addition to sqlg, even if one or two features aren't supported.
Ok, I'll have a look at gremlin-server, in particular how it might be possible to allow the non jvm languages access to Sqlg's non standard features. Especially on postgresql there are many of them, multiple bulk loading options, partitions, citus support and now recently postgres_fdw support.
thank you so much!
Ok made some progress, needed to add support for the GraphBinaryMessageSerializerV1. I am able to get gremlin server to work with that but not with GraphSONMessageSerializerV3d0. Need to check a bit more what's going there.
Ok all the tests passed on the test server.
Here is the gremlin-server.yaml file I used,
host: localhost
port: 8182
graphs: {
graph: src/test/resources/conf/sqlg.properties}
scriptEngines: {
gremlin-groovy: {
plugins: { org.apache.tinkerpop.gremlin.server.jsr223.GremlinServerGremlinPlugin: {},
org.umlg.sqlg.groovy.plugin.SqlgPostgresGremlinPlugin: {},
org.apache.tinkerpop.gremlin.jsr223.ScriptFileGremlinPlugin: {files: [src/test/resources/conf/gremlin-server-sqlg-postgres.groovy]} } } }
serializers:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1, config: { ioRegistries: [org.umlg.sqlg.structure.SqlgIoRegistryV3] } } # application/vnd.graphbinary-v1.0
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0, config: { ioRegistries: [org.umlg.sqlg.structure.SqlgIoRegistryV3] } } # application/json
gremlin-server-sqlg-postgres.groovy
def globals = [:]
globals << [g : graph.traversal()]
sqlg.properties
jdbc.url=jdbc:postgresql://localhost:5432/sqlgraphdb
jdbc.username=username
jdbc.password=password
gremlin.graph=org.umlg.sqlg.structure.SqlgGraph
And the java test client code, maybe it will help with the golang
@Test
public void testGraphBinarySerializer() {
IoRegistry registry = SqlgIoRegistryV3.instance();
TypeSerializerRegistry typeSerializerRegistry = TypeSerializerRegistry.build().addRegistry(registry).create();
GraphBinaryMessageSerializerV1 graphBinaryMessageSerializerV1 = new GraphBinaryMessageSerializerV1(typeSerializerRegistry);
Cluster cluster = Cluster.build().
serializer(graphBinaryMessageSerializerV1).
create();
Client client = cluster.connect();
GraphTraversalSource g = AnonymousTraversalSource.traversal().withRemote(DriverRemoteConnection.using(client, "g"));
g.addV("Person").property("name", "John").iterate();
List<Vertex> list = g.V().hasLabel("Person").toList();
Assert.assertEquals(1, list.size());
List<Map<Object, String>> valueMaps = g.V().hasLabel("Person").<String>valueMap().by(__.unfold()).toList();
Assert.assertEquals(1, valueMaps.size());
Assert.assertEquals("John", valueMaps.get(0).get("name"));
}
@Test
public void testGraphJsonSerializer() {
IoRegistry registry = SqlgIoRegistryV3.instance();
GraphSONMapper.Builder builder = GraphSONMapper.build().addRegistry(registry).addCustomModule(GraphSONXModuleV3d0.build().create(false));
GraphSONMessageSerializerV3d0 graphSONMessageSerializerV3d0 = new GraphSONMessageSerializerV3d0(builder);
Cluster cluster = Cluster.build().
serializer(graphSONMessageSerializerV3d0).
create();
Client client = cluster.connect();
GraphTraversalSource g = AnonymousTraversalSource.traversal().withRemote(DriverRemoteConnection.using(client, "g"));
for (int i = 0; i < 10; i++) {
g.addV("Person").property("name", "John").iterate();
}
List<Vertex> persons = g.V().hasLabel("Person").toList();
Assert.assertEquals(10, persons.size());
}
The only change I made was to support the bimary serializer. The json serializer required no changes.
I have deployed Sqlg `2.1.6-SNAPSHOT` to Sonatypes snapshot repositories.
Cool. I can confirm that yaml also works for me, in my limited testing/usage of it (except for the visualizer tools, see other issue).
Add gremlin-server yaml config to documentation