Seeking assistance in optimizing program efficiency for simulation

[Jerry Yang]

Hi there,

I'm hoping to create an offline simulation tool that allows us to analyze alphas without using the WorldQuant BRAIN platform. However, my program's efficiency needs significant improvement in order to achieve this goal.

Specifically, I'm seeking help with optimizing the program's performance by speeding up the data processing time and minimizing the resources required (mainly memory overhead).

• The program costs about 1min to load data from the local sqlite3 database (data from June 2015 to March 2022), and also about 1min to sort the DataFrame by symbol and timestamp.
• The simulation part utilizes multiprocessing, and it works fine with 8 cores on my machine if I just load one year of data. However, the memory limitation seems to cause broken pipe errors and crash the running program if I try to load more at the beginning.

I will later try fixing this problem by dropping unnecessary columns in the original DataFrame, but I truly hope to get help from anyone proficient in Python programming in these computing tasks.

If you have experience in optimizing Python program efficiency, I would greatly appreciate your assistance with this project. Feel free to leave a comment below, and I'll be happy to provide you with more details.

Thanks in advance for your help!

Jerry Yang