NGMON LogTranslator git repository is located [here] (https://github.com/michalxo/LogTranslator).
Project for changing classic log statements from Log4j, Slf4j, Commons Logging frameworks into NGMON's syntax. For now, it works only with maven projects.
-
Set appropriately all the necessary settings in logtranslator.properties files and execute
-
mvn install
-
mvn exec:exec
-
Install by maven generated source project
-
Add maven project settings for NGMON LogTranslator in parent's pom.xml file. You can find them [here][mvn-settings].
-
Recompile your application of choice (Apache Hadoop in our case)
-
Execute hadoop job with added classpath to ngmon-logger jar.
[How to add custom jar to MapReduce] (http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/)
-
Add StringUtils.java file path to exclude-list file
-
Syntax errors complaining mainly about "diamonds" and their type mismatch.
- Fix: Make sure you compile and run with JDK 1.7 and language level is set to "Diamonds, ARM, Multicatch". (Project settings in your IDE)
-
[WARNING] bad version number found in .m2/repository/org/aspectj/aspectjrt/1.7.2/aspectjrt-1.7.2.jar expected 1.6.11 found 1.7.2
- Fix: Change in "aspectj-maven-plugin" artifact .m2/repository/org/codehaus/mojo/aspectj-maven-plugin/1.4/aspectj-maven-plugin-1.4.pom.xml file version 1.7.2 from 1.6.11. This might help to solve a problem.
-
[WARNING] antlr & ngmon logger are having some problems during compilation. We pass '-XX:-UseSplitVerifier' argument to JVM. This behaviour and workaround is tested with OpenJDK 1.7.51 and OracleJDK 1.7.17.
-
All variables should be declared before usage. LogTranslator is not as powerful as compiler and makes only 1 pass over each java file. (fix hadoop-common/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java )
[mvn-deps]: You also need to build ngmon-logger-java by yourself.
Sources are available to download from https://github.com/ngmon/ngmon-logger-java/
You have to add logtranslator as dependency for this project in appropriate pom.xml file to understand logtranslator's directory. (Create dependencies part)
org.ngmon.logger logtranslator 1.0-SNAPSHOT compile###==== Apache Hadoop ====
-
Install these tools on your machine: a) package depending on your system protobuf-c (rpm)
- OR - protobuf-compiler (deb) b) maven's invocator of protoc compiler hawtbuf-protoc (rpm package)
- OR - maven-protoc-plugin (mvn install from https://github.com/usmanismail/maven-protoc-plugin) c) make sure you have build-essential, cmake, pkg-config, zlib1g (zlib1g-dev) and libssl (libssl-dev) packages
-
Download findbugs (http://findbugs.sourceforge.net/downloads.html) environment properties settings: set FINDBUGS_HOME set proper JAVA_HOME - must point to javah executable
-
Compilation git clone https://github.com/apache/hadoop-common.git
We have used 'commit 33a47d90022f8c6611e89f2da0b6f72e008ed529' for testing purposes.
mvn package -Pdist,native,docs -Dtar -DskipTests
- Running Hadoop a) Add log4j2 to apache hadoop's classpath first HADOOP_CLASSPATH=/home/mtoth/.m2/repository/org/apache/logging/log4j/log4j-core/2.0-beta4/log4j-core-2.0-beta4.jar:/home/mtoth/Downloads/apache-log4j-2.0-rc1-bin/log4j-api-2.0-rc1.jar
b) run example application ./hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/bin/hadoop jar ./hadoop-mapreduce-project/hadoop-mapreduce-examples/target/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar teragen 3 /tmp/teragen2
Note: When translating hadoop-common, make sure you do following:
Move fields definitions from back of the file, to the beginning before first constructor/ method. hadoop-common/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
Default log4j output is set to /tmp/Hadoop-LogTranslator.log file