Preface
1. The Neural Network
Building Intelligent Machines
The Limits of Traditional Computer Programs
The Mechanics of Machine Learning
The Neuron
Expressing Linear Perceptrons as Neurons
Feed-Forward Neural Networks
Linear Neurons and Their Limitations
Sigmoid, Tanh, and ReLU Neurons
Softmax Output Layers
Looking Forward
2. Training Feed-Forward Neural Networks
The Fast-Food Problem
Gradient Descent
The Delta Rule and Learning Rates
Gradient Descent with Sigmoidal Neurons
The Backpropagation Algorithm
Stochastic and Minibatch Gradient Descent
Test Sets, Validation Sets, and Overfitting
Preventing Overfitting in Deep Neural Networks
Summary
3. Implementing Neural Networks in TensorFIow
What Is TensorFlow?
How Does TensorFlow Compare to Alternatives?
Installing TensorFlow
Creating and Manipulating TensorFlow Variables
TensorFlow Operations
Placeholder Tensors
Sessions in TensorFlow
Navigating Variable Scopes and Sharing Variables
Managing Models over the CPU and GPU
Specifying the Logistic Regression Model in TensorFlow
Logging and Training the Logistic Regression Model
Leveraging TensorBoard to Visualize Computation Graphs and Learning
Building a Multilayer Model for MNIST in TensorFlow
Summary
4. Beyond Gradient Descent
The Challenges with Gradient Descent
Local Minima in the Error Surfaces of Deep Networks
Model Identifiability
How Pesky Are Spurious Local Minima in Deep Networks?
Flat Regions in the Error Surface
When the Gradient Points in the Wrong Direction
Momentum-Based Optimization
A Brief View of Second-Order Methods
Learning Rate Adaptation
AdaGrad--Accumulating Historical Gradients
RMSProp--Exponentially Weighted Moving Average of Gradients
Adam--Combining Momentum and RMSProp