State-Management Error - Memory leak
Parsoolak opened this issue · comments
Hi dear lovely community
I Have a problem with my ReactPHP Project.
I Have Created an application to save/update crypto price every second in Redis and use it on my another related project.
I use a state-Manager variable to avoid Re-Running at same time.
but now i have a problem! My State-Manager Variable Get stuck!
I Also Add Promise-Timer to my project but it still stop!
I use last version of reactPHP-reactMySQL-reactRedis-reactTimeout.
Here is My Code:
// Setup EventLoop
$loop = React\EventLoop\Loop::get();
// Database Factory
$databaseFactory = new Factory($loop);
// Async Redis
$redisFactory = new Clue\React\Redis\Factory();
$asyncRedis = $redisFactory->createLazyClient(// My Cred);
// Database
$database = new DatabaseHelper($databaseFactory->createLazyConnection(// My Cred));
// Create Feed Generator // Thats Generate My Feed
$feedGenerator = new FeedGenerator();
// Do Update
$isActive = false;
$loop->addPeriodicTimer(1, function () use ($feedGenerator, $asyncRedis, &$isActive) {
Logger::console(date("Y-m-d H:i:s") . " | Start Running | " . intval($isActive));
if ($isActive) {
Logger::console(date("Y-m-d H:i:s") . " | Error | Service Already Active");
return false;
}
$isActive = true;
return timeout($feedGenerator->getCryptoFeed(), 5)
->then(function ($result) use ($asyncRedis, &$isActive) {
if (!$result['result']) {
$isActive = false;
Logger::console(date("Y-m-d H:i:s") . " | Error | Generating | {$result['error']}");
return false;
}
return $asyncRedis->set("CryptoPrices", json_encode($result['content']))
->then(function ($result) use (&$isActive) {
$isActive = false;
if ($result !== 'OK') {
Logger::console(date("Y-m-d H:i:s") . " | Error | Redis | {$result}");
return false;
} else {
Logger::console(date("Y-m-d H:i:s") . " | Success");
return true;
}
});
}, function ($error) use (&$isActive) {
$isActive = false;
if ($error instanceof TimeoutException) {
Logger::console(date("Y-m-d H:i:s") . " | Exception | Timed-Out | Restore State");
} else {
Logger::console(date("Y-m-d H:i:s") . " | Exception | State Manager | Restore State");
}
return false;
});
});
$loop->run();and after 1-2 days Running it Stuck and something like this happeend:
Please Help me fix this!
I have tried to fix it for 20 Days.
Its pleasure to hear your help @clue and other lovely community.
Hey @Parsoolak since you have a bunch of logging in there could you change it to include the memory usage at the time of logging? Something like:
Logger::console(date("Y-m-d H:i:s") . " | " . (memory_get_usage(false) / 1024 / 1024) . "MB | Start Running | " . intval($isActive));Because currently it's hard to spot the baseline memory usage from your image, and by how much it creeps up every cycle.
And by judging that output, and I can only guess at best, something happens in FeedGenerator that causes failures. No clue what tho
Hi Dear @WyriHaximus
I`v Logged the ram usage and now i have a report for you.
My codes got stuck at 19:55 and start increasing ram usage from 6 MB to 128 MB at 20:18 and my ram exhausted.
Also I wanna say something wierd! My code always stop on some exact time like 19:55 and 14:55 and 7:55 !!
Also i wanna attached that i use remote mysql server but i dont have any error logs there.
Screenshot of when my codes start stucking:
Screenshot of when my codes ram exhausted:
as you said it could be from FeedGenerator, Here it my codes on that part:
class FeedGenerator
{
protected DatabaseHelper $db1;
protected DatabaseHelper $db2
protected DatabaseHelper $db3
public function __construct(
DatabaseHelper $db1,
DatabaseHelper $db2,
DatabaseHelper $db3)
{
$this->db1 = $db1;
$this->db2 = $db2;
$this->db3 = $db3;
}
public function getCryptoFeed(): PromiseInterface|Promise
{
return all([
$this->db3->createSelectQuery("Market",['service' => 'IRT'],null,"ORDER BY date DESC LIMIT 30"),
$this->db1->createSelectQuery("Cryptos",['isListed' => 1],null,"ORDER BY sarafRank ASC")
])->then(function ($results) {
foreach ($results as $result) {
if (!$result['result'])
return ['result' => false, 'error' => $result['error']];
if ($result['count'] == 0)
return ['result' => false, 'error' => "Zero Count"];
}
$historicalTetherPrice = [];
foreach ($results[0]['rows'] as $usdtRow) {
$historicalTetherPrice[$usdtRow['date']] = doubleval($usdtRow['close']);
}
return map($results[1]['rows'], function ($cryptoRow) use ($historicalTetherPrice) {
$symbol = $cryptoRow['symbol'];
$name = $cryptoRow['name'];
$exchanges = json_decode($cryptoRow['exchanges'], true);
$pairedSymbol = $symbol . "-USDT";
if ($symbol == "USDT")
return [
'result' => true,
's' => $symbol,
'n' => $name,
'h' => array_values($historicalTetherPrice),
'hu' => [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
];
if ($exchanges[0] != 'kucoin')
return ['result' => false];
return $this->db2
->createSelectQuery(
"Market",
['pair' => $pairedSymbol],
null,
"ORDER BY date DESC LIMIT 30"
)->then(function ($result) use ($symbol, $name, $historicalTetherPrice) {
if (!$result['result'] || $result['count'] == 0)
return ['result' => false];
$historicalIrtPrice = [];
$historicalUsdPrice = [];
foreach ($result['rows'] as $priceRow) {
$historicalUsdPrice[] = GlobalHelper::internalRounder($priceRow['close']);
if (isset($historicalTetherPrice[$priceRow['date']])) {
$historicalIrtPrice[] = round($priceRow['close'] * $historicalTetherPrice[$priceRow['date']]);
} else {
$historicalIrtPrice[] = round($priceRow['close'] * array_values($historicalTetherPrice)[0]);
}
}
return [
'result' => true,
's' => $symbol,
'n' => $name,
'h' => $historicalIrtPrice,
'hu' => $historicalUsdPrice
];
});
})->then(function ($rows) {
$finalRows = [];
foreach ($rows as $row) {
if (!$row['result'])
continue;
$currentPrice = $row['hu'][0];
if (isset($row['h'][1]))
$changePercentage = round((($row['h'][0] - $row['h'][1]) / $row['h'][1]) * 100, 2);
else
$changePercentage = 0;
$irtPrice = $row['h'][0];
$pairWithName = "IRToman";
$finalRows[] = [
's' => $row['s'],
'n' => $row['n'],
'p' => $irtPrice,
'd' => $currentPrice,
'q' => $pairWithName,
'h' => $row['h'],
'hu' => $row['hu'],
'c' => $changePercentage
];
}
return [
'result' => true,
'content' => [
'lastUpdateTime' => GlobalHelper::getCurrentMicroTime(),
'Image' => GlobalHelper::$cryptoAssetImageBaseEndpoint,
'Items' => $finalRows
]
];
});
});
} GlobalHelper::internalRounder is this:
public static function internalRounder($price): float|int
{
ini_set('serialize_precision', 14);
if ($price > 10000) {
$precision = 0;
} else if ($price > 1000) {
$precision = 1;
} else if ($price > 10) {
$precision = 2;
} else if ($price > 0.1) {
$precision = 3;
} else if ($price > 0.01) {
$precision = 5;
} else {
$precision = 8;
}
return round($price, $precision, PHP_ROUND_HALF_DOWN);
}The Whole Process Get Crypto Market Rates From Exchanges and Get Internal-Country Price From a database and then merge them and calculate crypto prices in national Prices.
Much Thanks Dear Cees-Jan
There are not big weird things in there. That leads to two questions:
- What happens at XX55 every day?
- Why store everything in a single Redis key instead of multiple or a list? Because if there is a big burst of data you get more to process and might run out of memory
I checked every service and nothing runs at XX55 But I think somehow I found the issue about your first question. every XX55 there's a release and renew happening from my DHCP client to renew the IP address..When this thing happens the whole timeout, MYSQL lazy connection and the react-mq stops working and it can't cancel the Promise nor Reject them...Whole applications stops working...
What I'm trying to say is that when the MYSQL Lazy Connection wants to create a new underlying connection for its own query queue this happens and fails the process of creating connection because of this exception other things fails too...
I mean maybe reactphp can't handle this type of exceptions!? or do you have any thoughts about these kind of situations...
Before diving into the what and why @Parsoolak. Do you get a new IP address every renew? If that is the case, your existing mysql/redis connection becomes invalid and the source/destination address changes. That isn't a ReactPHP issue perse, that is a networking issue.