Use raw text files as Jetbrains did rather than in memory module to reduce memory overhead and startup time.
elonzh opened this issue · comments
elonzh commented
James Meakin commented
I chose for runtime speed over startup time, memory usage is 5MB for a speedup of 37x. Versions prior to 0.2 used your suggested approach. Some profiling:
Our approach:
pyswot version: 0.2.1
Filename: test.py
Line # Mem usage Increment Occurrences Line Contents
=============================================================
6 20.1 MiB 20.1 MiB 1 @profile
7 def fun():
8 25.2 MiB 5.1 MiB 1 import pyswot
10000 loops for user@gmcc.tnua.edu.tw, best of 5: 0.00031309701799998944 s
10000 loops for user@nope.tnua.edu.tw, best of 5: 0.0003268956379999963 s
Raw text file approach:
pyswot version: 0.1.3
Filename: test.py
Line # Mem usage Increment Occurrences Line Contents
=============================================================
6 20.0 MiB 20.0 MiB 1 @profile
7 def fun():
8 20.0 MiB 0.0 MiB 1 import pyswot
10000 loops for user@gmcc.tnua.edu.tw, best of 5: 0.011768581818000012 s
10000 loops for user@nope.tnua.edu.tw, best of 5: 0.013157385768000003 s
test.py
:
from memory_profiler import profile
from importlib.metadata import version
print(f"pyswot version: {version('pyswot')}")
@profile
def fun():
import pyswot
fun()
import timeit
from statistics import mean
def time_profile(email):
number = 10000
timing = timeit.repeat(f"is_academic('{email}')", "from pyswot import is_academic")
print(f"{number} loops for {email}, best of {len(timing)}: {mean(timing) / number} s")
time_profile("user@gmcc.tnua.edu.tw")
time_profile("user@nope.tnua.edu.tw")