# Neural Network Learning Resources

## Mathematics Fundamentals

### Linear Algebra
#### Books
- [Linear Algebra and Its Applications](https://math.mit.edu/~gs/linearalgebra/) - Gilbert Strang
- [Linear Algebra Done Right](https://linear.axler.net/) - Sheldon Axler
- [Matrix Analysis and Applied Linear Algebra](http://www.matrixanalysis.com/) - Carl D. Meyer

#### Video Courses
- [MIT OpenCourseWare Linear Algebra](https://ocw.mit.edu/courses/18-06sc-linear-algebra-fall-2011/)
- [3Blue1Brown Essence of Linear Algebra](https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab)

#### Research Papers
- "Fast Linear Algebra is Stable" - [arXiv:1405.7903](https://arxiv.org/abs/1405.7903)
- "Randomized Numerical Linear Algebra: Foundations & Algorithms" - [arXiv:2002.01387](https://arxiv.org/abs/2002.01387)

### Calculus
#### Books
- [Calculus: Early Transcendentals](https://www.stewartcalculus.com/) - James Stewart
- [Calculus Made Easy](https://www.gutenberg.org/ebooks/33283) - Silvanus P. Thompson
- [Advanced Calculus: A Differential Forms Approach](https://www.maa.org/press/maa-reviews/advanced-calculus-a-differential-forms-approach) - Harold M. Edwards

#### Video Courses
- [MIT Single Variable Calculus](https://ocw.mit.edu/courses/18-01sc-single-variable-calculus-fall-2010/)
- [3Blue1Brown Essence of Calculus](https://www.youtube.com/playlist?list=PLZHQObOWTQDMsr9K-rj53DwVRMYO3t5Yr)

#### Research Papers
- "The Calculus of Deep Learning" - [arXiv:2102.01454](https://arxiv.org/abs/2102.01454)

### Statistics & Probability
#### Books
- [Introduction to Probability](https://www.probabilitycourse.com/) - Dimitri P. Bertsekas
- [All of Statistics](https://www.stat.cmu.edu/~larry/all-of-statistics/) - Larry Wasserman
- [Probability Theory: The Logic of Science](https://bayes.wustl.edu/etj/prob/book.pdf) - E. T. Jaynes

#### Video Courses
- [Harvard Statistics 110](https://projects.iq.harvard.edu/stat110/home)
- [MIT Probabilistic Systems Analysis](https://ocw.mit.edu/courses/6-041sc-probabilistic-systems-analysis-and-applied-probability-fall-2013/)

## Programming Fundamentals

### C Programming
#### Books
- [Modern C](https://modernc.gforge.inria.fr/) - Jens Gustedt
- [Expert C Programming: Deep C Secrets](https://www.pearson.com/en-us/subject-catalog/p/expert-c-programming-deep-c-secrets/P200000009159) - Peter van der Linden
- [C Programming: A Modern Approach](http://knking.com/books/c2/) - K.N. King
- [21st Century C](https://shop.oreilly.com/product/0636920033677.do) - Ben Klemens

#### Online Resources
- [C Programming Language Documentation](https://devdocs.io/c/)
- [GNU C Reference Manual](https://www.gnu.org/software/gnu-c-manual/)
- [C Programming Wikibook](https://en.wikibooks.org/wiki/C_Programming)

## Neural Network Implementation

### Foundational Resources
#### Books
- [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/) - Michael Nielsen
- [Deep Learning](https://www.deeplearningbook.org/) - Ian Goodfellow, Yoshua Bengio, Aaron Courville
- [Pattern Recognition and Machine Learning](https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf) - Christopher Bishop

#### Research Papers
- "Training Deep Nets with Sublinear Memory Cost" - [arXiv:1604.06174](https://arxiv.org/abs/1604.06174)
- "Deep Learning Hardware: Past, Present, and Future" - [arXiv:1907.09423](https://arxiv.org/abs/1907.09423)
- "Efficient BackProp" - [Research Paper Link](http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf)

### Advanced Implementation
#### Online Courses
- [Stanford CS231n](http://cs231n.stanford.edu/)
- [Fast.ai Deep Learning Course](https://course.fast.ai/)
- [CMU Deep Learning Course](http://deeplearning.cs.cmu.edu/)

#### Code Repositories
- [Tiny-DNN](https://github.com/tiny-dnn/tiny-dnn)
- [Neural Networks in C](https://github.com/codeplea/genann)
- [CNet - Neural Network in C](https://github.com/hughperkins/cNet)
- [NNPACK](https://github.com/Maratyszcza/NNPACK)

#### Research Papers
- "Random Search for Hyper-Parameter Optimization" - [Random Search for Hyper-Parameter Optimization](https://jmlr.org/papers/v13/bergstra12a.html)
- "A Disciplined Approach to Neural Network Hyper-Parameters" - [arXiv:1803.09820](https://arxiv.org/abs/1803.09820)
- "An Analysis of Deep Neural Network Models for Practical Applications" - [arXiv:1605.07678](https://arxiv.org/abs/1605.07678)

## Performance Optimization

### System Level Optimization
#### Books
- [High Performance Computing](https://www.netlib.org/utk/papers/advanced-computers/mpi-book/mpi-book.html)
- [Optimizing C++](https://www.agner.org/optimize/)
- [Computer Systems: A Programmer's Perspective](http://csapp.cs.cmu.edu/) - Bryant and O'Hallaron

#### Research Papers
- "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference" - [arXiv:1712.05877](https://arxiv.org/abs/1712.05877)
- "Mixed Precision Training" - [arXiv:1710.03740](https://arxiv.org/abs/1710.03740)

### Architecture Design
#### Books
- [Clean Code](https://www.oreilly.com/library/view/clean-code-a/9780136083238/)
- [The Art of UNIX Programming](http://www.catb.org/~esr/writings/taoup/)
- [System Design Interview](https://www.amazon.com/System-Design-Interview-insiders-Second/dp/B08CMF2CQF)

#### Research Papers
- "A Hardware Architecture for Deep Learning" - [arXiv:1908.07748](https://arxiv.org/abs/1908.07748)
- "System Design for Large Scale Machine Learning" - [Paper Link](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2016/EECS-2016-1.pdf)

## Additional Resources

### Developer Tools
- [GDB Documentation](https://www.gnu.org/software/gdb/documentation/)
- [Valgrind User Manual](https://valgrind.org/docs/manual/manual.html)
- [Perf Wiki](https://perf.wiki.kernel.org/index.php/Main_Page)

### Community Resources
- [Papers With Code](https://paperswithcode.com/)
- [ArXiv Machine Learning](https://arxiv.org/list/cs.LG/recent)
- [Reddit r/MachineLearning](https://www.reddit.com/r/MachineLearning/)

---
