Memory Leaks

[Danilo Bargen]

This project seems to have a small memory leak: RAM usage increased by ~300MiB over the last 7 days on a production server with 4 parallel SaltyRTC processes.

Nothing terrible though.

[Lenny]

Could be related to #46

[Danilo Bargen]

After 70 days:

[Lenny]

Now, that is quite a lot. Can you send me more graphs (users connecting, disconnecting), so I can look at the full picture? Via Threema of course if you don't want to reveal that here. :)

[Lenny]

Running the tests and tracking memory with pympler and objgraph revelead no memory leaks, so I cannot reproduce this. @dbrgn If you want, I can push the code to a branch and you can track memory for a period of time on one of your servers. This should reveal the leak.

[Danilo Bargen]

Yeah, we could do that sometime.

Does pympler handle cases where a Python object is freed but the memory cannot be reclaimed by the operating system?

[Lenny]

I don't think it does. These tools all look at referenced objects rather than tracking raw memory.

[Danilo Bargen]

I think that's actually where the problem is, in which case tracking live objects won't help unfortunately.

This talk is interesting (although it won't really give a solution for that issue): https://archive.org/details/EuroPython_2014_YRUIVJC7

[Danilo Bargen]

Can we add a signal handler to the server, so that the process dumps its heap for example on SIGUSR1?

[Lenny]

Sure but what do you want to dump exactly?

[Danilo Bargen]

Anything that will help us to find the source of these allocations. What about current heap allocations along with types?

[Lenny]

I'm really busy atm. I can send you my code though. It's not ready for GH as there may be some print('<insert toilet humour here>') lines 'n stuff. 😉

[Lenny]

I will close this since I couldn't ever reproduce this. It would require further investigation on Threema's side.

[Lenny]

Might be related: python-websockets/websockets#533

[marcor4]

I have the same problem. I run the following testing code on Raspbian (python 3.5.3) and memory is constantly increasing:

import asyncio
import websockets
import time

CLIENTS=set()

async def server_handler(websocket_server,path):
	CLIENTS.add(websocket_server)
	print("client connected")

	try:
		server_receiver_task=asyncio.ensure_future(server_receiver(websocket_server))
		server_sender_task=asyncio.ensure_future(server_sender(websocket_server))
		done,pending=await asyncio.wait([server_receiver_task,server_sender_task],
			return_when=asyncio.FIRST_COMPLETED,)

		for task in pending:
			task.cancel()

	finally:
		CLIENTS.remove(websocket_server)

async def server_receiver(websocket_server):
	try:
		while True:
			message=await websocket_server.recv()
			print("Message received from the client: "+message)
			await asyncio.sleep(0.1)
	except:
		print("client diconnected")

async def server_sender(websocket_server):
	while True:
		if CLIENTS:
			await asyncio.wait([client.send("server time: "+str(time.time())) for client in CLIENTS])
		await asyncio.sleep(1)


async def client_handler():

	while True:
		async with websockets.connect("ws://localhost:8765") as websocket_client:

			try:
				client_receiver_task=asyncio.ensure_future(client_receiver(websocket_client))
				client_sender_task=asyncio.ensure_future(client_sender(websocket_client))
				done,pending=await asyncio.wait([client_receiver_task,client_sender_task],
					return_when=asyncio.FIRST_COMPLETED,)

				for task in pending:
					task.cancel()

			finally:
				await websocket_client.close()


async def client_receiver(websocket_client):
	message=await websocket_client.recv()
	print("Message received from the server: "+message)
	await asyncio.sleep(5)

async def client_sender(websocket_client):
	while True:
		await websocket_client.send("client time: "+str(time.time()))
		await asyncio.sleep(1)


start_server=websockets.serve(server_handler,"",8765)
loop=asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(start_server,client_handler()))
loop.close

[Lenny]

Interesting. Thanks for letting us know. So, this might be an issue with websockets (or asyncio) after all.