Error when running the stixtoneodb-5.0.jar .

Question

Error when running the stixtoneodb-5.0.jar .

limengmingx opened this issue 5 years ago · comments

I got an Error as below

PS F:\StixToNeoDB\target\scala-2.12> java -jar stixtoneodb-5.0.jar -f "F:\StixToNeoDB\stix-files\testfull.json"
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(Unknown Source)
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at java.io.BufferedReader.read1(Unknown Source)
at java.io.BufferedReader.read(Unknown Source)
at java.io.Reader.read(Unknown Source)
at scala.io.BufferedSource.mkString(BufferedSource.scala:98)
at com.kodekutters.neo4j.Neo4jFileLoader.loadBundleFile(Neo4jFileLoader.scala:67)
at com.kodekutters.StixToNeoDB$.main(StixToNeoDB.scala:50)
at com.kodekutters.StixToNeoDB.main(StixToNeoDB.scala)

After adding an lib floder containing the slf4j-simple-1.7.26.jar and changing the build.sbt by adding the codes below

unmanagedJars in Compile ++= Seq(
  Attributed.blank[File](file(baseDirectory.value + "/lib/slf4j-simple-1.7.26.jar")),
)

I get another error

$ java -jar stixtoneodb-5.0.jar -f "F:\StixToNeoDB\stix-files\testfull.json"
[main] INFO StixToNeoDB - connected to Neo4j community at: F:\StixToNeoDB\target\scala-2.12/stixdb
[main] INFO StixToNeoDB - processing file: F:\StixToNeoDB\stix-files\testfull.json
Exception in thread "main" java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(Unknown Source)
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at java.io.BufferedReader.read1(Unknown Source)
at java.io.BufferedReader.read(Unknown Source)
at java.io.Reader.read(Unknown Source)
at scala.io.BufferedSource.mkString(BufferedSource.scala:98)
at com.kodekutters.neo4j.Neo4jFileLoader.loadBundleFile(Neo4jFileLoader.scala:67)
at com.kodekutters.StixToNeoDB$.main(StixToNeoDB.scala:50)
at com.kodekutters.StixToNeoDB.main(StixToNeoDB.scala)

and I am working on solving the error

limengmingx · Answer 1 · Mon Mar 11 2019 16:49:55 GMT+0800 (China Standard Time)

The error is connected with the code :

stixtoneolib\src\main\scala\com\kodekutters\neo4j\Neo4jFileLoader.scala

line 67 , where:
val jsondoc = Source.fromFile(inFile).mkString

I am fresh to JAVA ,this maybe the most job I can do .
Looking forward to fix the error .

workingDog · Answer 2 · Mon Mar 11 2019 18:48:49 GMT+0800 (China Standard Time)

Hi limengmingx,

I will investigate your error (although I don't have a Windows PC).
Could you try:

java -Dfile.encoding=UTF8 -jar stixtoneodb-5.0.jar -f "F:\StixToNeoDB\stix-files\testfull.json"

limengmingx · Answer 3 · Mon Mar 11 2019 20:39:59 GMT+0800 (China Standard Time)

Thanks ! The new command has make it . Though an Error has been through

java -Dfile.encoding=UTF8 -jar stixtoneodb-5.0.jar -f "F:\StixToNeoDB\stix-files\testfull.json"

[main] INFO StixToNeoDB - connected to Neo4j community at: F:\neo4j-community-3.5.3\TEST/stixdb
[main] INFO StixToNeoDB - processing file: F:\StixToNeoDB\stix-files\testfull.json
[main] ERROR StixToNeoDB - could not process relation: relationship--bdcef81d-9dfa-4f5d-a495 from: threat-actor--aaaaaaaaaaaaaaaa to: identity--bbbbbbbbbbbbbbbbbbbb
[main] ERROR StixToNeoDB - could not process relation: sighting--834006833 from: indicator--xxxxxzzzzzxxxxx to: indicator--xxxxxzzzzzxxxxx
[main] INFO StixToNeoDB - StixObj: 8
[main] INFO StixToNeoDB - SRO: 141
[main] INFO StixToNeoDB - SDO: 148
[main] INFO StixToNeoDB - total: 297

after try document testfull.json , I try an easy json document ,which only contains a bundle file

{
    "type": "bundle",
    "id": "bundle--f83477e5-f853-47e1-a267-43f3aa1bd5b0",
    "spec_version": "2.0",
    "objects": [
        {
            "type": "indicator",
            "id": "indicator--548af3be-39d7-4a3e-93c2-1a63cccf8951",
            "created": "2018-04-05T18:32:24.193Z",
            "modified": "2018-04-05T18:32:24.193Z",
            "name": "File hash for malware variant",
            "pattern": "[file:hashes.md5 = 'd41d8cd98f00b204e9800998ecf8427e']",
            "valid_from": "2018-04-05T18:32:24.193659Z",
            "labels": [
                "malicious-activity"
            ]
        },
        {
            "type": "malware",
            "id": "malware--3d7f0c1c-616a-4868-aa7b-150821d2a429",
            "created": "2018-04-05T18:32:46.584Z",
            "modified": "2018-04-05T18:32:46.584Z",
            "name": "Poison Ivy",
            "labels": [
                "remote-access-trojan"
            ]
        },
        {
            "type": "relationship",
            "id": "relationship--34ddc7b4-4965-4615-b286-1c8bbaa1e7db",
            "created": "2018-04-05T18:32:49.474Z",
            "modified": "2018-04-05T18:32:49.474Z",
            "relationship_type": "indicates",
            "source_ref": "indicator--548af3be-39d7-4a3e-93c2-1a63cccf8951",
            "target_ref": "malware--3d7f0c1c-616a-4868-aa7b-150821d2a429"
        }
    ]
}

and no error occurs:

java -Dfile.encoding=UTF8 -jar stixtoneodb-5.0.jar -f "F:\StixToNeoDB\stix-files\1.json"
[main] INFO StixToNeoDB - connected to Neo4j community at: F:\neo4j-community-3.5.3\TEST/stixdb
[main] INFO StixToNeoDB - processing file: F:\StixToNeoDB\stix-files\1.json
[main] INFO StixToNeoDB - StixObj: 0
[main] INFO StixToNeoDB - SRO: 1
[main] INFO StixToNeoDB - SDO: 2
[main] INFO StixToNeoDB - total: 3

Can you describe it in more details about how to change the Neo4j database path in neo4j.conf , I have try many times ,only to find no data in browser.

workingDog · Answer 4 · Tue Mar 12 2019 07:33:20 GMT+0800 (China Standard Time)

Hi limengmingx,

I'm glad you got it working.

The errors you get with testfull.json are "normal". It means there is something wrong with:
relationship--bdcef81d-9dfa-4f5d-a495.
This relationship refers to source_ref or target_ref that are not present in the data, they do not exist. So stixtoneodb gives you information about those errors. Same for the sighting relationship.

I am not familiar with the Neo4j settings. It is best to consult the neo4j documentations at:
https://neo4j.com/docs/operations-manual/current/reference/configuration-settings/
I will have a look too.

In the coming weeks I will update stixtoneodb and stixtoneolib to include the slf4j logging jar and fix other bugs.

limengmingx · Answer 5 · Tue Mar 12 2019 10:25:52 GMT+0800 (China Standard Time)

complex may it look like ,but here is the method what i do :
first, change the neo4j.conf by adding such codes:

dbms.active_database=stixdb
dbms.directories.data = F:/neo4j-community-3.5.3/TEST/

after restarting my neo4j , there would be database floder generated .

TEST/
databases/
stixdb/
store_lock

and then I running the command to generate stix database

java -Dfile.encoding=UTF8 -jar stixtoneodb-5.0.jar -f "F:\StixToNeoDB\stix-files\testfull.json"

the I repalced the stixdb floder generated by neo4j command with the new stix database .
after restart neo4j database again , Finaly , I can view the sitx data in my chrome ! (LOL)