One lesson I took away from my time in graduate school was the importance of understanding things from first principles. So for learning about neural networks, I really enjoyed reading Grokking Deep Learning, which implements neural networks using only numpy, keeping the reader from being bogged down with implementation details. Another excellent introduction is this video video from Grant Sanderson*, which also goes into mathematical details. After that, Michael Nielsen’s Neural Networks and Deep Learning is a good followup.

A bit of history

The idea of artificial neural networks has been around since at least 1940s in the form of Hebbian theory. But neural networks really took off after Geoffrey Hinton’s famous 1986 paper on backpropagation combined with the availability of big datasets and increased computational power.

Network tuning

One of the interesting and frustrating problems in modeling complex data is overfitting. The solution often involves a combination of picking the right tools, then knowing how to interpret their output. In a neural network context once you’ve picked the appropriate architecture, the right activation functions, you may need to implement dropout.

Neural Smithing covers some network tuning issues.

*_Vlogging on YouTube as 3Blue1Brown, Sanderson has an excellent series where he explained math concepts visually.