What is a neural network and how does it successfully work? This article provides an introduction of neural networks, an overview of the optimization algorithms for training and some main theory questions regarding Deep Neural Networks. First, we start with some basic intuitions of a neural network and some detailed implementations about forward propagation and backward propagation. Second, we review the obstacle in training Deep Neural Networks and the corresponding solution including initialization and generic optimization algorithms. Third, we discuss the three main theory questions about Deep Neural Networks.