Pigeon is an open source, lightweight Java client that asynchronously listens for and fetches JMS messages sent by Apache Falcon via an MQ service. Built for use with Apache Hadoop and Apache Falcon. Ideal if you're using Falcon to schedule jobs on Hadoop, and want a monitoring solution.
Falcon has workflows. Those workflows send JMS message alerts upon completion to indicate their success or failure, and other cross-referential information that's very good for monitoring and debugging. Pigeon simply retrieves those alerts and writes them to a file. I use this on my Hadoop team to power a monitoring dashboard.
Because this retrieves alerts for every job that runs on the cluster, Pigeon can easily build a job history and lineage on your Hadoop cluster. Give it a shot if you like!
Easy. Pull this code (it's a Maven project). Change the hostname variable to the server where Falcon is installed, and change the filename variable to give Pigeon a text file to write to. Package the jar and place it on your cluster (can be an edge node). Run it with a hadoop jar command and presto! It will write results to a file in that directory.
Have questions, ideas, or feedback? Email me at landonrobinson92 [@] gmail.com.
I appreciate, but do not require, formal attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Hadoopsters: Tutorials, Tips and Tricks for Hadoop (https://github.com/landonrobinson/ApacheFalconJMSListener).”
If you feel your use of code examples falls outside fair use or the permission given here, feel free to contact us at hadoopsters@gmail.com.