Spark MLlib GBT 4d (for demos)
szilard opened this issue · comments
100 trees, depth 10, 100K data, 32 cores (1 node) runs for 17 mins. Ugh!
How can we make Spark MLlib GBT work fast enough for a demo?
Smaller data? Less trees? Less depth? More cores? More nodes? Let's help the Spark fans...
We actually want it fast even on a laptop. So we'll work with m5.xlarge (4 cores, 16GB RAM)
Less data (10K records):
runs 200 sec, AUC=0.670, peak RAM usage 7.8GB - too slow!
Less trees (10 trees)
runs 65 sec, AUC=0.711, RAM 6.5GB - nice, we are getting faster!
Less depth (depth 4)
runs 360 sec, AUC=0.725, RAM 6.4GB - oh no, way too slow!
Even less depth? depth 1 maybe?
runs 290 sec, AUC=0.704, RAM 6.2GB - argh!
We need a combination! Less trees (10) and less depth (4)
runs 40sec, AUC=0.708 - maybe OK for a demo! but can we get even faster?
Maybe 5 trees and depth 3?
runs 20sec, AUC=0.694 - dunno... I guess training sophisticated machine learning algorithms (sorry, I mean AI) just takes time
Oh, yeah, 1 tree and depth 1 is the ultimate solution! Thanks @daroczig for the idea!
runs 6 sec, AUC=0.634
btw lightgbm 10 trees depth 4 take 0.2 sec (vs Spark 40 sec)