This thesis aims to present key results in the approximation theory of artificial neural networks assuming only undergraduate mathematics. This research field studies necessary and sufficient conditions under which neural networks can approximate an arbitrary function belonging to a particular family. The approximation is formalized within a function space. Theorems addressing those issues are known as the universal approximation theorems. This thesis provides a comprehensive survey of approximation theory of neural networks, including detailed proofs of various universal approximation theorems for continuous functions on compact sets. Those results were generalized to spaces of Lebesgue integrable and square-integrable functions. The thesis also discusses the universal approximation of Borel measurable functions in a probabilistic sense, concluding with an experimental study of the relationship between established theoretical results and practical applications.
The thesis was submitted for a BSc in Computer Science and Mathematics at The University of Edinburgh and was graded an exceptional A1 (92%).
Here is a compiled thesis.