Question about decision trees being universal function approximators.

[Rohan R Paleja]

Hi,

I hope this is the right place to post a question about the HTML materials! I was reviewing this and am having some trouble understanding why a decision tree or regression tree would be a universal function approximator. Would you be able to provide some clarification for why a regression tree would be a universal function approximator? If there is a pointer to a chapter within Machine Learning Refined that goes deeply into this, I would greatly appreciate a reference!

Thanks,

[Jeremy Watt]

Hi! Apologies for the delayed response!

Broadly speaking - a tree is a universal function approximator for the same reason polynomials, Fourier series, or fully connected networks are - using them you can approximate (reasonable) functions to any tolerance you desire. Basically it means you can find some combination of the basis functions to approximate just about any "real world" function you'd ever encounter.

Check out the video above Example 11.6 in the link you shared. There you'll see a little animation showing the progressive approximation to a sine wave of successive combinations of polynomials (left panel), fully connected networks (middle panel), and trees (right panel).

This is just a single example of course, but you can imagine (or test!) how you could perform the same experiment on more complicated functions.

Hope that helps!