Capture stdout
shadyb opened this issue · comments
Hello,
I have been using resqueboard with resque-ex for awhile now. I have noticed that it is extremely difficult to debug jobs. Doing an echo on a variable does not displace output in resque-board. It would be nice to have the ability to do echos and then have a section in job (maybe under "Job Arguments") to show any output.
I know resqueboard will capture and handler exceptions (to a degree) but I would like to capture all output (stderr and stdout) and output it to the user. This would make diagnosing bugs and problematic code MUCH easier.
I can implement this on your behalf, in which case it would be nice for you to outline a possible strategy.
regards
Using an echo will not display anything on Resqueboard. Resqueboard is a 'listener', used in parallel of your php-resque-ex setup.
All exceptions and stdout are written directly in php-resque output.log, and can not be read by resqueboard. Resqueboard can only read output sent via Monolog, and that monolog instance is available only on the workers, not on the jobs.
Or maybe do you already have something in mind ?
I have implemented functionality in my ResqueBoard branch to show logs for processed jobs here: https://github.com/pwhelan/ResqueBoard/tree/display-extended-logs. I have not made it a PR yet since it still lacks cosmetic touches.
Which commit exactly ? It's a little hard to find since there's a lot of merge and the commit does not seems to be among the lastest one
I believe the work is in this commit:
I see that the log is saved in the log key. But where does it come from ?
Here is the BaseWorker class I use and extend for my workers. It includes the error handling code to create the log entries.
<?php if ( ! defined('RESQUEWORKERS')) die('No direct script access');
/// CONFIRMATION STOP GAPS:
// reviews, sms_tracking
class BaseWorker
{
private $_startTime;
private $_ci = null;
private $_php_to_resque_error_level = [
E_ERROR => Resque_Worker::LOG_TYPE_ERROR,
E_WARNING => Resque_Worker::LOG_TYPE_WARNING,
E_NOTICE => Resque_Worker::LOG_TYPE_INFO,
E_CORE_ERROR => Resque_Worker::LOG_TYPE_ERROR,
E_CORE_WARNING => Resque_Worker::LOG_TYPE_WARNING,
E_COMPILE_ERROR => Resque_Worker::LOG_TYPE_ERROR,
E_COMPILE_WARNING => Resque_Worker::LOG_TYPE_WARNING,
E_USER_ERROR => Resque_Worker::LOG_TYPE_ERROR,
E_USER_WARNING => Resque_Worker::LOG_TYPE_WARNING,
E_USER_NOTICE => Resque_Worker::LOG_TYPE_INFO,
E_STRICT => Resque_Worker::LOG_TYPE_WARNING,
E_RECOVERABLE_ERROR => Resque_Worker::LOG_TYPE_ERROR,
E_DEPRECATED => Resque_Worker::LOG_TYPE_WARNING,
E_USER_DEPRECATED => Resque_Worker::LOG_TYPE_WARNING
];
public static $currentJob = null;
public function log_worker_error($errno, $errstr, $errfile, $errline, $errcontext)
{
if (in_array($errno, [E_ERROR, E_CORE_ERROR, E_USER_ERROR])) {
throw new Exception($errstr, $errno);
}
// Log all other notices and errors as process messages...
// Must find a type for logging things...
// Why RB oh Why?!?!?!
$worker_errno =
in_array($errno, array_keys($this->_php_to_resque_error_level)) ?
$this->_php_to_resque_error_level[$errno] :
Resque_Worker::LOG_TYPE_CRITICAL;
if (isset($this->job) && isset($this->job->worker)) {
print "LOGGING ({$errfile}:{$errline}) {$this->job->payload['id']}: {$errstr}\n";
$this->job->worker->log([
'message' => $errstr,
'data' => [
'type' => 'log',
'log' => $errstr." ({$errfile}:{$errline})",
'job_id'=> (string)$this->job->payload['id'],
'time' => round(microtime(true) - $this->_startTime, 3) * 1000
]
], $worker_errno);
}
else {
print "LOG: {$errno}: {$errstr} ($errfile}:{$errline}\n";
}
}
public function log_fatal_error()
{
$error = error_get_last();
if ($error == null) {
return;
}
if (!in_array($error['type'], [E_ERROR, E_CORE_ERROR, E_USER_ERROR])) {
return;
}
debug_print_backtrace();
$worker_errno =
in_array($error['type'], array_keys($this->_php_to_resque_error_level)) ?
$this->_php_to_resque_error_level[$error['type']] :
Resque_Worker::LOG_TYPE_CRITICAL;
if (isset(BaseWorker::$currentJob->worker)) {
BaseWorker::$currentJob->worker->log([
'message' => $error['message']." ({$error['file']}:{$error['line']})",
'data' => [
'type' => 'fail',
'log' => $error['message']." ({$error['file']}:{$error['line']})",
'job_id'=> (string)BaseWorker::$currentJob->payload['id'],
'time' => round(microtime(true) - $this->_startTime, 3) * 1000
]
], $worker_errno);
}
print "Fatal Error({$error['file']}:{$error['line']}): {$error['message']}\n";
}
public function __construct()
{
$this->_ci = &get_instance();
// ResqueBoard does not really display anything besides the
// last failure... We could drag along all the log lines
// like Fingers crossed perhaps...
//$this->_startTime = microtime(TRUE);
set_error_handler([$this, 'log_worker_error'], E_ALL);
register_shutdown_function([$this, "log_fatal_error"]);
$this->_startTime = microtime();
if (isset($this->job)) {
self::$currentJob = $this->job;
}
}
public function setUp()
{
if (isset($this->job)) {
self::$currentJob = $this->job;
}
}
}How are you using that class ?
I use it as the base class for my workers.
Workers are defined inside php-resque. How do you make it use your base class ? Are you editing file inside php-resque package ?
It's the base class for my own worker (the one with the perform method), not an actual Resque_Worker class instance.
Ah, it's the Job class
@Kamisama exactly! excuse my confusing use of terminology there.
I'll review it when I'll be less busy.
I get the part about printing the log in ResqueBoard. Maybe someone could make a PR about the changes in php-resque-ex side, to catch and log job's error in the worker class ?
I've developed some of my own work on this with changes to both Resque Board and php-resque-ex
At this point all stdout is captured and stored to Mongo and is available at the end of the script.
At the moment I'm working on capturing stdout as the output buffer is flushed to offer real-time updates in Resque Board via websockets.
Should only be a day or two. (We're using this in our corporate environment so I get a full 8 hours a day to work on this).
@paco3346 Any update on this one ?
@Techbrunch Sorry, apparently I don't check my github notifications very often.
I haven't touched this code in a long time but it still works well. Check out the following repos to see what I did. (I should mention- I'm no longer with the company that's using this so I don't actually have access to where it's running)
https://github.com/ussignal/ResqueBoard/commits/master
https://github.com/ussignal/Fresque/commits/master
https://github.com/ussignal/cube/commits/master
https://github.com/ussignal/php-resque-ex/commits/master
If you have specific questions please let me know and I'd be happy to assist. This code not only captures all stdout but even streams it to the browser in real-time.