CurlEasy causes a large memory leak.
http200it opened this issue · comments
Hello again, faced a problem on Linux Mono 3.0 CurlEasy causes a large memory leak.
My test code:
public static void DoTestCurlEasy()
{
bool wykonuj = true;
int nrpetli = 0;
while (wykonuj)
{
using (var easy = new CurlEasy())
{
easy.Url = "http://www.xxxxxxx.com/dotest.html";
easy.WriteData = null;
easy.WriteFunction = OnWriteData;
easy.Perform();
}
Console.WriteLine("nrpetli: {0}", nrpetli);
nrpetli++;
Thread.Sleep(200);
}
}
public static Int32 OnWriteData(Byte[] buf, Int32 size, Int32 nmemb, Object extraData)
{
//var userData = (string)extraData;
//Console.Write(Encoding.UTF8.GetString(buf));
return size * nmemb;
}
Linux htop
Running is 0.9 MEM%
- 10 minutes
1.1 MEM%
1.2 MEM%
...
2.0 MEM%
And all the time the leak is getting bigger and RAM occupancy increases :(
Please check at the time, it may also fail to fix this :)
Thank you
It seems that the problem lies in the linux Momo though I have the latest version:
mono -V
Mono JIT compiler version 3.10.0 (tarball Mon Oct 13 14:05:11 EDT 2014)
Application
if ((nrpetli % 10) == 0)
{
GC.Collect();
Console.WriteLine("GC.Collect");
}
does not help
@http200it Indeed, it seems to be a Mono-specific problem. I tweaked your sample code to make it multi-threaded:
private static void Main(string[] args)
{
for (var i = 0; i < 8; i++)
{
var th = new Thread(DoTestCurlEasy);
th.Start();
}
Curl.GlobalCleanup();
}
public static void DoTestCurlEasy()
{
Curl.GlobalInit(CurlInitFlag.All);
var wykonuj = true;
var nrpetli = 0;
while (wykonuj)
{
using (var easy = new CurlEasy())
{
easy.Url = "http://localhost:88/data.txt";
easy.WriteData = null;
easy.WriteFunction = OnWriteData;
easy.Perform();
}
Console.WriteLine("nrpetli: {0}", nrpetli);
nrpetli++;
//Thread.Sleep(200);
}
}
public static Int32 OnWriteData(Byte[] buf, Int32 size, Int32 nmemb, Object extraData)
{
return size*nmemb;
}The target URL is a 10+ MB data file, so any memory leaks should crop up fairly easily. Even with the above muti-threaded example I was unable to detect memory leaks in .net.
Here's a screenshot after several minutes:
Memory usage is still not increased, it's hovering around 8.5 - 9 MB after running for several minutes.
Environment for the above screenshot is: Windows 8.1 x64 + .net 4.5.2 x64 + libcurl-64.
That's right, after many tests, I noticed that the problem regards the Linux platform Mono on Windows is not. Sorry for the inconvenience.
@http200it: Can you kindly experiment with the functions CurlEasy._curlWriteCallback(), CurlEasy._curlReadCallback() and see if explicitly releasing the memory buffer after invoking the callback delegates (_pfCurlWrite & _pfCurlRead respectively) will prevent the memory leaks?
Unfortunately I don't have access to a Linux/OS X box right now.
https://github.com/masroore/CurlSharp/blob/master/CurlSharp/CurlEasy.cs#L2090
later, see :)
@http200it I have tested the sample project under OS X (Yosemite) and Mono 3.10 and confirm the memory leak. I modified the source code to create 20 concurrent threads. On startup, memory usage hovered between 10-20 MB. After running the app for 3+ hours, I found the memory usage has crept up to ~140MB.
@masroore Thank you for confirming tests :)
I tested an application written in C ++ uses libcurl - Linux debian 7 64 Bit:
#curl -V
curl 7.38.0 (x86_64-unknown-linux-gnu) libcurl/7.38.0 NSS/3.14.5.0 zlib/1.2.7 c-ares/1.10.0 libidn/1.25 libssh2/1.4.2 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile NTLM NTLM_WB SSL libz
I say a big memory leak for HTTP (S) HTTP leak is not. Application pseudo code:
whiel(true)
{
curl_global_init(CURL_GLOBAL_ALL);
// app code
curl_global_cleanup();
}
reduces memory leak for HTTPS, but not suitable for threads.
In MONO 3.10 is also a leak for HttpWebRequest.
My test code seems to be correct are:
public static void OdpalaWatekTestowyDoHttps()
{
ServicePointManager.ServerCertificateValidationCallback = delegate(object s, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
{ return true; };
for (int jjj = 0; jjj < 10; jjj++)
{
Thread MyThreadGZd = new Thread(() => DoTestWebRequestGet(jjj));
MyThreadGZd.Start();
}
}
public static void DoTestWebRequestGet(int nrwataka)
{
bool wykonuj = true;
int nrpetli = 0;
while (wykonuj)
{
HttpWebResponse response = null;
StreamReader reader = null;
HttpWebRequest request = null;
request = (HttpWebRequest) WebRequest.Create("https://www.domain.com/gfx/wyswietldate.php?randettsos=" + nrpetli.ToString());
request.Credentials = CredentialCache.DefaultCredentials;
request.ServicePoint.Expect100Continue = false;
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
request.KeepAlive = false;
request.UserAgent = "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0";
try
{
response = (HttpWebResponse) request.GetResponse();
Stream dataStream = response.GetResponseStream();
reader = new StreamReader(dataStream);
string html = reader.ReadToEnd().Trim();
lock (lockThisOb)
{
Console.WriteLine("Status:{0} html: {1} nrpetli: {2} nrwataka: {3} ilewywolal: {4}", ((HttpWebResponse)response).StatusDescription, html, nrpetli, nrwataka, ClassPomocnicze.ilewywolal);
ClassPomocnicze.ilewywolal++;
}
}
catch(Exception exe)
{
Console.WriteLine("Exception {0} ", exe.Message);
}
finally
{
if (response != null)
{
response.Close();
response.Dispose();
response = null;
}
if (reader != null)
{
reader.Close();
reader.Dispose();
reader = null;
}
if (request != null)
{
request = null;
}
}
nrpetli++;
Thread.Sleep(1000);
}
}
/// end
However, the leakage is much smaller but increasing RAM busy hours
Leakage is not a similar test in JAVA applications.
Regards