# Introduction to Deep Learning

You can read the notes from the previous lecture from Andrew Ng's CS229 course on Kernal Methods here.
We now begin our study of deep learning. In this set of notes, we give an overview of neural networks, discuss vectorization and discuss training neural networks with backpropagation.

## 1. Supervised Learning with Non-linear Models

In the supervised learning setting (predicting $y$$y$yy from the input $x$$x$xx), suppose our model/hypothesis is ${h}_{\theta }\left(x\right)$${h}_{\theta }\left(x\right)$h_(theta)(x)h_{\theta}(x). In the past lectures, we have considered the cases when ${h}_{\theta }\left(x\right)={\theta }^{\mathrm{\top }}x$${h}_{\theta }\left(x\right)={\theta }^{\mathrm{\top }}x$h_(theta)(x)=theta^(TT)xh_{\theta}(x)=\theta^{\top} x (in linear regression or logistic regression) or ${h}_{\theta }\left(x\right)=$${h}_{\theta }\left(x\right)=$h_(theta)(x)=h_{\theta}(x)= ${\theta }^{\mathrm{\top }}\varphi \left(x\right)$${\theta }^{\mathrm{\top }}\varphi \left(x\right)$theta^(TT)phi(x)\theta^{\top} \phi(x) (where $\varphi \left(x\right)$$\varphi \left(x\right)$phi(x)\phi(x) is the feature map). A commonality of these two models is that they are linear in the parameters $\theta$$\theta$theta\theta. Next we will consider learning general family of models that are non-linear in both the parameters $\theta$$\theta$theta\theta and the inputs $x$$x$xx. The most common non-linear models are neural networks, which we will define staring from the next section. For this section, it suffices to think ${h}_{\theta }\left(x\right)$${h}_{\theta }\left(x\right)$h_(theta)(x)h_{\theta}(x) as an abstract non-linear model.[1]
Suppose ${\left\{\left({x}^{\left(i\right)},{y}^{\left(i\right)}\right)\right\}}_{i=1}^{n}$${\left\{\left({x}^{\left(i\right)},{y}^{\left(i\right)}\right)\right\}}_{i=1}^{n}${(x^((i)),y^((i)))}_(i=1)^(n)\left\{\left(x^{(i)}, y^{(i)}\right)\right\}_{i=1}^{n} are the training examples. For simplicity, we start with the case where ${y}^{\left(i\right)}\in \mathbb{R}$${y}^{\left(i\right)}\in \mathbb{R}$y^((i))inRy^{(i)} \in \mathbb{R} and ${h}_{\theta }\left(x\right)\in \mathbb{R}$${h}_{\theta }\left(x\right)\in \mathbb{R}$h_(theta)(x)inRh_{\theta}(x) \in \mathbb{R}.
Cost/loss function. We define the least square cost function for the $i$$i$ii-th example $\left({x}^{\left(i\right)},{y}^{\left(i\right)}\right)$$\left({x}^{\left(i\right)},{y}^{\left(i\right)}\right)$(x^((i)),y^((i)))\left(x^{(i)}, y^{(i)}\right) as
$\begin{array}{}\text{(1.1)}& {J}^{\left(i\right)}\left(\theta \right)=\frac{1}{2}{\left({h}_{\theta }\left({x}^{\left(i\right)}\right)-{y}^{\left(i\right)}\right)}^{2}\end{array}$$\begin{array}{}\text{(1.1)}& {J}^{\left(i\right)}\left(\theta \right)=\frac{1}{2}{\left({h}_{\theta }\left({x}^{\left(i\right)}\right)-{y}^{\left(i\right)}\right)}^{2}\end{array}${:(1.1)J^((i))(theta)=(1)/(2)(h_(theta)(x^((i)))-y^((i)))^(2):}$$J^{(i)}(\theta)=\frac{1}{2}\left(h_{\theta}\left(x^{(i)}\right)-y^{(i)}\right)^{2}\tag{1.1}$$
and define the mean-square cost function for the dataset as
$\begin{array}{}\text{(1.2)}& J\left(\theta \right)=\frac{1}{n}\sum _{i=1}^{n}{J}^{\left(i\right)}\left(\theta \right)\end{array}$$\begin{array}{}\text{(1.2)}& J\left(\theta \right)=\frac{1}{n}\sum _{i=1}^{n} {J}^{\left(i\right)}\left(\theta \right)\end{array}${:(1.2)J(theta)=(1)/(n)sum_(i=1)^(n)J^((i))(theta):}$$J(\theta)=\frac{1}{n} \sum_{i=1}^{n} J^{(i)}(\theta) \tag{1.2}$$
which is same as in linear regression except that we introduce a constant $1/n$$1/n$1//n1/n in front of the cost function to be consistent with the convention. Note that multiplying the cost function with a scalar will not change the local minima or global minima of the cost function. Also note that the underlying parameterization for ${h}_{\theta }\left(x\right)$${h}_{\theta }\left(x\right)$h_(theta)(x)h_{\theta}(x) is different from the case of linear regression, even though the form of the cost function is the same mean-squared loss. Throughout the notes, we use the words "loss" and "cost" interchangeably.
Optimizers (SGD). Commonly, people use gradient descent (GD), stochastic gradient (SGD), or their variants to optimize the loss function $J\left(\theta \right)$$J\left(\theta \right)$J(theta)J(\theta). GD's update rule can be written as [2]
$\begin{array}{}\text{(1.3)}& \theta :=\theta -\alpha {\mathrm{\nabla }}_{\theta }J\left(\theta \right)\end{array}$$\begin{array}{}\text{(1.3)}& \theta :=\theta -\alpha {\mathrm{\nabla }}_{\theta }J\left(\theta \right)\end{array}${:(1.3)theta:=theta-alphagrad_(theta)J(theta):}$$\theta:=\theta-\alpha \nabla_{\theta} J(\theta) \tag{1.3}$$
where $\alpha >0$$\alpha >0$alpha > 0\alpha>0 is often referred to as the learning rate or step size. Next, we introduce a version of the SGD (Algorithm $1$$1$11), which is lightly different from that in the first lecture notes.
 Algorithm 1 Stochastic Gradient Descent 1: Hyperparameter: learning rate $\alpha$$\alpha$alpha\alpha$\alpha$, number of total iteration ${n}_{\text{iter}}$${n}_{\text{iter}}$n_("iter")n_{\text{iter}}${n}_{\text{iter}}$. 2: Initialize $\theta$$\theta$theta\theta$\theta$ randomly. 3: for $i=1$$i=1$i=1i = 1$i=1$ to ${n}_{\text{iter}}$${n}_{\text{iter}}$n_("iter")n_{\text{iter}}${n}_{\text{iter}}$ do 4: $\phantom{\rule{1em}{0ex}}$$\phantom{\rule{1em}{0ex}}$quad\quad$\phantom{\rule{1em}{0ex}}$ Sample $j$$j$jj$j$ uniformly from $\left\{1,\dots ,n\right\}$$\left\{1,\dots ,n\right\}${1,dots,n}\{1,\ldots,n\}$\left\{1,\dots ,n\right\}$, and update $\theta$$\theta$theta\theta$\theta$ by $\begin{array}{}\text{(1.4)}& \theta :=\theta -\alpha {\mathrm{\nabla }}_{\theta }{J}^{j}\left(\theta \right)\end{array}$$\begin{array}{}\text{(1.4)}& \theta :=\theta -\alpha {\mathrm{\nabla }}_{\theta }{J}^{j}\left(\theta \right)\end{array}${:(1.4)theta:=theta-alphagrad_(theta)J^(j)(theta):}$$\theta := \theta - \alpha \nabla_{\theta}J^{j}(\theta) \tag{1.4}$$$\begin{array}{}\text{(1.4)}& \theta :=\theta -\alpha {\mathrm{\nabla }}_{\theta }{J}^{j}\left(\theta \right)\end{array}$
Oftentimes computing the gradient of $B$$B$BB examples simultaneously for the parameter $\theta$$\theta$theta\theta can be faster than computing $B$$B$BB gradients separately due to hardware parallelization. Therefore, a mini-batch version of SGD is most commonly used in deep learning, as shown in Algorithm $2$$2$22. There are also other variants of the SGD or mini-batch SGD with slightly different sampling schemes.
 Algorithm 2 Mini-batch Stochastic Gradient Descent 1: Hyperparameters: learning rate $\alpha$$\alpha$alpha\alpha$\alpha$, batch size $B,\mathrm{#}$$B,\mathrm{#}$B,#B, \#$B,\mathrm{#}$ iterations ${n}_{\text{iter}}$${n}_{\text{iter}}$n_("iter")n_{\text{iter}}${n}_{\text{iter}}$. 2: Initialize $\theta$$\theta$theta\theta$\theta$ randomly 3: for $i=1$$i=1$i=1i = 1$i=1$ to ${n}_{\text{iter}}$${n}_{\text{iter}}$n_("iter")n_{\text{iter}}${n}_{\text{iter}}$ do 4: $\phantom{\rule{1em}{0ex}}$$\phantom{\rule{1em}{0ex}}$quad\quad$\phantom{\rule{1em}{0ex}}$ Sample $B$$B$BB$B$ examples ${j}_{1},\dots ,{j}_{B}$${j}_{1},\dots ,{j}_{B}$j_(1),dots,j_(B)j_{1},\ldots,j_{B}${j}_{1},\dots ,{j}_{B}$ (without replacement) uniformly from $\left\{1,\dots ,n\right\}$$\left\{1,\dots ,n\right\}${1,dots,n}\{1,\ldots, n \}$\left\{1,\dots ,n\right\}$, and update $\theta$$\theta$theta\theta$\theta$ by $\begin{array}{}\text{(1.5)}& \theta :=\theta -\frac{\alpha }{B}\sum _{k=1}^{B}{\mathrm{\nabla }}_{\theta }{J}^{\left({j}_{k}\right)}\left(\theta \right)\end{array}$$\begin{array}{}\text{(1.5)}& \theta :=\theta -\frac{\alpha }{B}\sum _{k=1}^{B} {\mathrm{\nabla }}_{\theta }{J}^{\left({j}_{k}\right)}\left(\theta \right)\end{array}${:(1.5)theta:=theta-(alpha )/(B)sum_(k=1)^(B)grad_(theta)J^((j_(k)))(theta):}$$\theta := \theta - \frac{\alpha}{B} \sum^{B}_{k=1}\nabla_{\theta}J^{(j_{k})}(\theta) \tag{1.5}$$$\begin{array}{}\text{(1.5)}& \theta :=\theta -\frac{\alpha }{B}\sum _{k=1}^{B}{\mathrm{\nabla }}_{\theta }{J}^{\left({j}_{k}\right)}\left(\theta \right)\end{array}$
With these generic algorithms, a typical deep learning model is learned with the following steps. 1. Define a neural network parametrization ${h}_{\theta }\left(x\right)$${h}_{\theta }\left(x\right)$h_(theta)(x)h_{\theta}(x), which we will introduce in Section 2, and 2. write the backpropagation algorithm to compute the gradient of the loss function ${J}^{\left(j\right)}\left(\theta \right)$${J}^{\left(j\right)}\left(\theta \right)$J^((j))(theta)J^{(j)}(\theta) efficiently, which will be covered in Section 3, and 3. run SGD or mini-batch SGD (or other gradient-based optimizers) with the loss function $J\left(\theta \right)$$J\left(\theta \right)$J(theta)J(\theta).

## 2. Neural Networks

Neural networks refer to broad type of non-linear models/parametrizations ${h}_{\theta }\left(x\right)$${h}_{\theta }\left(x\right)$h_(theta)(x)h_{\theta}(x) that involve combinations of matrix multiplications and other entrywise non-linear operations. We will start small and slowly build up a neural network, step by step.
A Neural Network with a Single Neuron. Recall the housing price prediction problem from before: given the size of the house, we want to predict the price. We will use it as a running example in this subsection.
Previously, we fit a straight line to the graph of size vs. housing price. Now, instead of fitting a straight line, we wish to prevent negative housing prices by setting the absolute minimum price as zero. This produces a "kink" in the graph as shown in Figure 1. How do we represent such a function with a single kink as ${h}_{\theta }\left(x\right)$${h}_{\theta }\left(x\right)$h_(theta)(x)h_{\theta}(x) with unknown parameter? (After doing so, we can invoke the machinery in Section 1.)
We define a parameterized function ${h}_{\theta }\left(x\right)$${h}_{\theta }\left(x\right)$h_(theta)(x)h_{\theta}(x) with input $x$$x$xx, parameterized by $\theta$$\theta$theta\theta, which outputs the price of the house $y$$y$yy. Formally, ${h}_{\theta }:x\to y$${h}_{\theta }:x\to y$h_(theta):x rarr yh_{\theta}: x \rightarrow y. Perhaps one of the simplest parametrization would be
$\begin{array}{}\text{(2.1)}& {h}_{\theta }\left(x\right)=max\left(wx+b,0\right)\text{, where}\theta =\left(w,b\right)\in {\mathbb{R}}^{2}\end{array}${:(2.1)h_(theta)(x)=max(wx+b","0)", where "theta=(w","b)inR^(2):}$$h_{\theta}(x)=\max (w x+b, 0) \text {, where } \theta=(w, b) \in \mathbb{R}^{2} \tag{2.1}$$
Here ${h}_{\theta }\left(x\right)$${h}_{\theta }\left(x\right)$h_(theta)(x)h_{\theta}(x) returns a single value: $\left(wx+b\right)$$\left(wx+b\right)$(wx+b)(w x+b) or zero, whichever is greater. In the context of neural networks, the function $max\left\{t,0\right\}$$max\left\{t,0\right\}$max{t,0}\max \{t, 0\} is called a ReLU (pronounced "ray-lu"), or rectified linear unit, and often denoted by $\mathrm{ReLU}\left(t\right)\triangleq$$\mathrm{ReLU}\left(t\right)\triangleq$ReLU(t)≜\operatorname{ReLU}(t) \triangleq $max\left\{t,0\right\}$$max\left\{t,0\right\}$max{t,0}\max \{t, 0\}.
Generally, a one-dimensional non-linear function that maps $\mathbb{R}$$\mathbb{R}$R\mathbb{R} to $\mathbb{R}$$\mathbb{R}$R\mathbb{R} such as ReLU is often referred to as an activation function. The model ${h}_{\theta }\left(x\right)$${h}_{\theta }\left(x\right)$h_(theta)(x)h_{\theta}(x) is said to have a single neuron partly because it has a single non-linear activation function. (We will discuss more about why a non-linear activation is called neuron.)
When the input $x\in {\mathbb{R}}^{d}$$x\in {\mathbb{R}}^{d}$x inR^(d)x \in \mathbb{R}^{d} has multiple dimensions, a neural network with a single neuron can be written as
$\begin{array}{}\text{(2.2)}& {h}_{\theta }\left(x\right)=\mathrm{ReLU}\left({w}^{\mathrm{\top }}x+b\right)\text{, where}w\in {\mathbb{R}}^{d},b\in \mathbb{R}\text{, and}\theta =\left(w,b\right)\end{array}${:(2.2)h_(theta)(x)=ReLU(w^(TT)x+b)", where "w inR^(d)","b inR", and "theta=(w","b):}$$h_{\theta}(x)=\operatorname{ReLU}\left(w^{\top} x+b\right) \text {, where } w \in \mathbb{R}^{d}, b \in \mathbb{R} \text {, and } \theta=(w, b) \tag{2.2}$$
The term $b$$b$bb is often referred to as the "bias", and the vector $w$$w$ww is referred to as the weight vector. Such a neural network has $1$$1$11 layer. (We will define what multiple layers mean in the sequel.)
Stacking Neurons. A more complex neural network may take the single neuron described above and "stack" them together such that one neuron passes its output as input into the next neuron, resulting in a more complex function.
Let us now deepen the housing prediction example. In addition to the size of the house, suppose that you know the number of bedrooms, the zip code and the wealth of the neighborhood. Building neural networks is analogous to Lego bricks: you take individual bricks and stack them together to build complex structures. The same applies to neural networks: we take individual neurons and stack them together to create complex neural networks.
Given these features (size, number of bedrooms, zip code, and wealth), we might then decide that the price of the house depends on the maximum family size it can accommodate. Suppose the family size is a function of the size of the house and number of bedrooms (see Figure $2$$2$22). The zip code may provide additional information such as how walkable the neighborhood is (i.e., can you walk to the grocery store or do you need to drive everywhere). Combining the zip code with the wealth of the neighborhood may predict the quality of the local elementary school. Given these three derived features (family size, walkable, school quality), we may conclude that the price of the home ultimately depends on these three features.
Formally, the input to a neural network is a set of input features ${x}_{1},{x}_{2},{x}_{3},{x}_{4}$${x}_{1},{x}_{2},{x}_{3},{x}_{4}$x_(1),x_(2),x_(3),x_(4)x_{1}, x_{2}, x_{3}, x_{4}. We denote the intermediate variables for "family size", "walkable", and "school quality" by ${a}_{1},{a}_{2},{a}_{3}$${a}_{1},{a}_{2},{a}_{3}$a_(1),a_(2),a_(3)a_{1}, a_{2}, a_{3} (these ${a}_{i}$${a}_{i}$a_(i)a_{i}'s are often referred to as
Figure 1: Housing prices with a "kink" in the graph.
Figure 2: Diagram of a small neural network for predicting housing prices.
"hidden units" or "hidden neurons"). We represent each of the ${a}_{i}$${a}_{i}$a_(i)a_{i} 's as a neural network with a single neuron with a subset of ${x}_{1},\dots ,{x}_{4}$${x}_{1},\dots ,{x}_{4}$x_(1),dots,x_(4)x_{1}, \ldots, x_{4} as inputs. Then as in Figure 1, we will have the parameterization:
$\begin{array}{rl}& {a}_{1}=\mathrm{ReLU}\left({\theta }_{1}{x}_{1}+{\theta }_{2}{x}_{2}+{\theta }_{3}\right)\\ & {a}_{2}=\mathrm{ReLU}\left({\theta }_{4}{x}_{3}+{\theta }_{5}\right)\\ & {a}_{3}=\mathrm{ReLU}\left({\theta }_{6}{x}_{3}+{\theta }_{7}{x}_{4}+{\theta }_{8}\right)\end{array}$$\begin{array}{r}{a}_{1}=\mathrm{ReLU}\left({\theta }_{1}{x}_{1}+{\theta }_{2}{x}_{2}+{\theta }_{3}\right)\\ {a}_{2}=\mathrm{ReLU}\left({\theta }_{4}{x}_{3}+{\theta }_{5}\right)\\ {a}_{3}=\mathrm{ReLU}\left({\theta }_{6}{x}_{3}+{\theta }_{7}{x}_{4}+{\theta }_{8}\right)\end{array}${:[a_(1)=ReLU(theta_(1)x_(1)+theta_(2)x_(2)+theta_(3))],[a_(2)=ReLU(theta_(4)x_(3)+theta_(5))],[a_(3)=ReLU(theta_(6)x_(3)+theta_(7)x_(4)+theta_(8))]:}\begin{aligned} &a_{1}=\operatorname{ReLU}\left(\theta_{1} x_{1}+\theta_{2} x_{2}+\theta_{3}\right) \\ &a_{2}=\operatorname{ReLU}\left(\theta_{4} x_{3}+\theta_{5}\right) \\ &a_{3}=\operatorname{ReLU}\left(\theta_{6} x_{3}+\theta_{7} x_{4}+\theta_{8}\right) \end{aligned}
where $\left({\theta }_{1},\cdots ,{\theta }_{8}\right)$$\left({\theta }_{1},\cdots ,{\theta }_{8}\right)$(theta_(1),cdots,theta_(8))\left(\theta_{1}, \cdots, \theta_{8}\right) are parameters. Now we represent the final output ${h}_{\theta }\left(x\right)$${h}_{\theta }\left(x\right)$h_(theta)(x)h_{\theta}(x) as another linear function with ${a}_{1},{a}_{2},{a}_{3}$${a}_{1},{a}_{2},{a}_{3}$a_(1),a_(2),a_(3)a_{1}, a_{2}, a_{3} as inputs, and we get[3]
$\begin{array}{}\text{(2.3)}& {h}_{\theta }\left(x\right)={\theta }_{9}{a}_{1}+{\theta }_{10}{a}_{2}+{\theta }_{11}{a}_{3}+{\theta }_{12}\end{array}$$\begin{array}{}\text{(2.3)}& {h}_{\theta }\left(x\right)={\theta }_{9}{a}_{1}+{\theta }_{10}{a}_{2}+{\theta }_{11}{a}_{3}+{\theta }_{12}\end{array}${:(2.3)h_(theta)(x)=theta_(9)a_(1)+theta_(10)a_(2)+theta_(11)a_(3)+theta_(12):}$$h_{\theta}(x)=\theta_{9} a_{1}+\theta_{10} a_{2}+\theta_{11} a_{3}+\theta_{12} \tag{2.3}$$
where $\theta$$\theta$theta\theta contains all the parameters $\left({\theta }_{1},\dots ,{\theta }_{12}\right)$$\left({\theta }_{1},\dots ,{\theta }_{12}\right)$(theta_(1),dots,theta_(12))(\theta_{1},\ldots,\theta_{12}).
Now we represent the output as a quite complex function of $x$$x$xx with parameters $\theta$$\theta$theta\theta. Then you can use this parametrization ${h}_{\theta }$${h}_{\theta }$h_(theta)h_{\theta} with the machinery of Section 1 to learn the parameters $\theta$$\theta$theta\theta.
Inspiration from Biological Neural Networks. As the name suggests, artificial neural networks were inspired by biological neural networks. The hidden units ${a}_{1},\dots ,{a}_{m}$${a}_{1},\dots ,{a}_{m}$a_(1),dots,a_(m)a_{1}, \ldots, a_{m} correspond to the neurons in a biological neural network, and the parameters ${\theta }_{i}$${\theta }_{i}$theta_(i)\theta_{i}'s correspond to the synapses. However, it's unclear how similar the modern deep artificial neural networks are to the biological ones. For example, perhaps not many neuroscientists think biological neural networks could have 1000 layers, while some modern artificial neural networks do (we will elaborate more on the notion of layers.) Moreover, it's an open question whether human brains update their neural networks in a way similar to the way that computer scientists learn artificial neural networks (using backpropagation, which we will introduce in the next section.).
Two-layer Fully-Connected Neural Networks. We constructed the neural network in equation (2.3) using a significant amount of prior knowledge/belief about how the "family size", "walkable", and "school quality" are determined by the inputs. We implicitly assumed that we know the family size is an important quantity to look at and that it can be determined by only the "size" and "# bedrooms". Such a prior knowledge might not be available for other applications. It would be more flexible and general to have a generic parameterization. A simple way would be to write the intermediate variable ${a}_{1}$${a}_{1}$a_(1)a_{1} as a function of all ${x}_{1},\dots ,{x}_{4}$${x}_{1},\dots ,{x}_{4}$x_(1),dots,x_(4)x_{1}, \ldots, x_{4}:
$\begin{array}{}\text{(2.4)}& {a}_{1}=\mathrm{ReLU}\left({w}_{1}^{\mathrm{\top }}x+{b}_{1}\right),\text{where}{w}_{1}\in {\mathbb{R}}^{4}\text{and}{b}_{1}\in \mathbb{R}\end{array}${:(2.4)a_(1)=ReLU(w_(1)^(TT)x+b_(1))","" where "w_(1)inR^(4)" and "b_(1)inR:}$$a_{1}=\operatorname{ReLU}\left(w_{1}^{\top} x+b_{1}\right), \text { where } w_{1} \in \mathbb{R}^{4} \text { and } b_{1} \in \mathbb{R} \tag{2.4}$$
${a}_{2}=\mathrm{ReLU}\left({w}_{2}^{\mathrm{\top }}x+{b}_{2}\right),\text{where}{w}_{2}\in {\mathbb{R}}^{4}\text{and}{b}_{2}\in \mathbb{R}$a_(2)=ReLU(w_(2)^(TT)x+b_(2))," where "w_(2)inR^(4)" and "b_(2)inRa_{2}=\operatorname{ReLU}\left(w_{2}^{\top} x+b_{2}\right), \text { where } w_{2} \in \mathbb{R}^{4} \text { and } b_{2} \in \mathbb{R}
${a}_{3}=\mathrm{ReLU}\left({w}_{3}^{\mathrm{\top }}x+{b}_{3}\right),\text{where}{w}_{3}\in {\mathbb{R}}^{4}\text{and}{b}_{3}\in \mathbb{R}$a_(3)=ReLU(w_(3)^(TT)x+b_(3))," where "w_(3)inR^(4)" and "b_(3)inRa_{3}=\operatorname{ReLU}\left(w_{3}^{\top} x+b_{3}\right), \text { where } w_{3} \in \mathbb{R}^{4} \text { and } b_{3} \in \mathbb{R}
We still define ${h}_{\theta }\left(x\right)$${h}_{\theta }\left(x\right)$h_(theta)(x)h_{\theta}(x) using equation (2.3) with ${a}_{1},{a}_{2},{a}_{3}$${a}_{1},{a}_{2},{a}_{3}$a_(1),a_(2),a_(3)a_{1}, a_{2}, a_{3} being defined as above. Thus we have a so-called fully-connected neural network as visualized in the dependency graph in Figure 2 because all the intermediate variables ${a}_{i}$${a}_{i}$a_(i)a_{i}'s depend on all the inputs ${x}_{i}$${x}_{i}$x_(i)x_{i}'s.
For full generality, a two-layer fully-connected neural network with $m$$m$mm hidden units and $d$$d$dd dimensional input $x\in {\mathbb{R}}^{d}$$x\in {\mathbb{R}}^{d}$x inR^(d)x \in \mathbb{R}^{d} is defined as
$\begin{array}{}\text{(2.5)}& \mathrm{\forall }j\in \left[1,\dots ,m\right],\phantom{\rule{1em}{0ex}}{z}_{j}={w}_{j}^{\left[1{\right]}^{\mathrm{\top }}}x+{b}_{j}^{\left[1\right]}\text{where}{w}_{j}^{\left[1\right]}\in {\mathbb{R}}^{d},{b}_{j}^{\left[1\right]}\in \mathbb{R}\end{array}${:(2.5)AA j in[1","dots","m]","quadz_(j)=w_(j)^([1]^(TT))x+b_(j)^([1])" where "w_(j)^([1])inR^(d)","b_(j)^([1])inR:}$$\forall j \in[1, \ldots, m], \quad z_{j}=w_{j}^{[1]^{\top}} x+b_{j}^{[1]} \text { where } w_{j}^{[1]} \in \mathbb{R}^{d}, b_{j}^{[1]} \in \mathbb{R} \tag{2.5}$$
Figure 3: Diagram of a two-layer fully connected neural network. Each edge from node ${x}_{i}$${x}_{i}$x_(i)x_{i} to node ${a}_{j}$${a}_{j}$a_(j)a_{j} indicates that ${a}_{j}$${a}_{j}$a_(j)a_{j} depends on ${x}_{i}$${x}_{i}$x_(i)x_{i}. The edge from ${x}_{i}$${x}_{i}$x_(i)x_{i} to ${a}_{j}$${a}_{j}$a_(j)a_{j} is associated with the weight ${\left({w}_{j}^{\left[1\right]}\right)}_{i}$${\left({w}_{j}^{\left[1\right]}\right)}_{i}$(w_(j)^([1]))_(i)\left(w_{j}^{[1]}\right)_{i} which denotes the $i$$i$ii-th coordinate of the vector ${w}_{j}^{\left[1\right]}$${w}_{j}^{\left[1\right]}$w_(j)^([1])w_{j}^{[1]}. The activation ${a}_{j}$${a}_{j}$a_(j)a_{j} can be computed by taking the ReLUof the weighted sum of ${x}_{i}$${x}_{i}$x_(i)x_{i} 's with the weights being the weights associated with the incoming edges, that is, ${a}_{j}=\mathrm{ReLU}\left(\sum _{i=1}^{d}{\left({w}_{j}^{\left[1\right]}\right)}_{i}{x}_{i}\right)$${a}_{j}=\mathrm{ReLU}\left(\sum _{i=1}^{d} {\left({w}_{j}^{\left[1\right]}\right)}_{i}{x}_{i}\right)$a_(j)=ReLU(sum_(i=1)^(d)(w_(j)^([1]))_(i)x_(i))a_{j}=\operatorname{ReLU}\left(\sum_{i=1}^{d}\left(w_{j}^{[1]}\right)_{i} x_{i}\right).
$\begin{array}{rl}{a}_{j}& =\mathrm{ReLU}\left({z}_{j}\right),\\ a& ={\left[{a}_{1},\dots ,{a}_{m}\right]}^{\mathrm{\top }}\in {\mathbb{R}}^{m}\\ \text{(2.6)}& {h}_{\theta }\left(x\right)& ={w}^{\left[2{\right]}^{\mathrm{\top }}}a+{b}^{\left[2\right]}\text{where}{w}^{\left[2\right]}\in {\mathbb{R}}^{m},{b}^{\left[2\right]}\in \mathbb{R},\end{array}${:[a_(j)=ReLU(z_(j))","],[a=[a_(1),dots,a_(m)]^(TT)inR^(m)],(2.6)h_(theta)(x)=w^([2]^(TT))a+b^([2])" where "w^([2])inR^(m)","b^([2])inR",":}\begin{align} a_{j} &=\operatorname{ReLU}\left(z_{j}\right),\nonumber \\ a &=\left[a_{1}, \ldots, a_{m}\right]^{\top} \in \mathbb{R}^{m}\nonumber \\ h_{\theta}(x) &=w^{[2]^{\top}} a+b^{[2]} \text { where } w^{[2]} \in \mathbb{R}^{m}, b^{[2]} \in \mathbb{R}, \tag{2.6} \end{align}
Note that by default the vectors in ${\mathbb{R}}^{d}$${\mathbb{R}}^{d}$R^(d)\mathbb{R}^{d} are viewed as column vectors, and in particular $a$$a$aa is a column vector with components ${a}_{1},{a}_{2},\dots ,{a}_{m}$${a}_{1},{a}_{2},\dots ,{a}_{m}$a_(1),a_(2),dots,a_(m)a_{1}, a_{2}, \ldots, a_{m}. The indices ${}^{\left[1\right]}$${}^{\left[1\right]}$^([1]){ }^{[1]} and ${}^{\left[2\right]}$${}^{\left[2\right]}$^([2]){ }^{[2]} are used to distinguish two sets of parameters: the ${w}_{j}^{\left[1\right]}$${w}_{j}^{\left[1\right]}$w_(j)^([1])w_{j}^{[1]},s (each of which is a vector in ${\mathbb{R}}^{d}$${\mathbb{R}}^{d}$R^(d)\mathbb{R}^{d} ) and ${w}^{\left[2\right]}$${w}^{\left[2\right]}$w^([2])w^{[2]} (which is a vector in ${\mathbb{R}}^{m}$${\mathbb{R}}^{m}$R^(m)\mathbb{R}^{m} ). We will have more of these later.
Vectorization. Before we introduce neural networks with more layers and more complex structures, we will simplify the expressions for neural networks with more matrix and vector notations. Another important motivation of vectorization is the speed perspective in the implementation. In order to implement a neural network efficiently, one must be careful when using for loops. The most natural way to implement equation (2.5) in code is perhaps to use a for loop. In practice, the dimensionalities of the inputs and hidden units are high. As a result, code will run very slowly if you use for loops. Leveraging the parallelism in GPUs is/was crucial for the progress of deep learning.
This gave rise to vectorization. Instead of using for loops, vectorization takes advantage of matrix algebra and highly optimized numerical linear algebra packages (e.g., BLAS) to make neural network computations run quickly. Before the deep learning era, a for loop may have been sufficient on smaller datasets, but modern deep networks and state-of-the-art datasets will be infeasible to run with for loops.
We vectorize the two-layer fully-connected neural network as below. We define a weight matrix ${W}^{\left[1\right]}$${W}^{\left[1\right]}$W^([1])W^{[1]} in ${\mathbb{R}}^{m×d}$${\mathbb{R}}^{m×d}$R^(m xx d)\mathbb{R}^{m \times d} as the concatenation of all the vectors ${w}_{j}^{\left[1\right],s}$${w}_{j}^{\left[1\right],s}$w_(j)^([1],s)w_{j}^{[1], s} in the following way:
$\begin{array}{}\text{(2.7)}& {W}^{\left[1\right]}=\left[\begin{array}{c}-{w}_{1}^{\left[1{\right]}^{\mathrm{\top }}}-\\ -{w}_{2}^{\left[1{\right]}^{\mathrm{\top }}}-\\ ⋮\\ -{w}_{m}^{\left[1{\right]}^{\mathrm{\top }}}-\end{array}\right]\in {\mathbb{R}}^{m×d}\end{array}$$\begin{array}{}\text{(2.7)}& {W}^{\left[1\right]}=\left[\begin{array}{c}-{w}_{1}^{\left[1{\right]}^{\mathrm{\top }}}-\\ -{w}_{2}^{\left[1{\right]}^{\mathrm{\top }}}-\\ ⋮\\ -{w}_{m}^{\left[1{\right]}^{\mathrm{\top }}}-\end{array}\right]\in {\mathbb{R}}^{m×d}\end{array}${:(2.7)W^([1])=[[-w_(1)^([1]^(TT))-],[-w_(2)^([1]^(TT))-],[vdots],[-w_(m)^([1]^(TT))-]]inR^(m xx d):}$$W^{[1]}=\left[\begin{array}{c} -w_{1}^{[1]^{\top}}- \\ -w_{2}^{[1]^{\top}}- \\ \vdots \\ -w_{m}^{[1]^{\top}}- \end{array}\right] \in \mathbb{R}^{m \times d} \tag{2.7}$$
Now by the definition of matrix vector multiplication, we can write $z=$$z=$z=z= ${\left[{z}_{1},\dots ,{z}_{m}\right]}^{\mathrm{\top }}\in {\mathbb{R}}^{m}$${\left[{z}_{1},\dots ,{z}_{m}\right]}^{\mathrm{\top }}\in {\mathbb{R}}^{m}$[z_(1),dots,z_(m)]^(TT)inR^(m)\left[z_{1}, \ldots, z_{m}\right]^{\top} \in \mathbb{R}^{m} as
$\begin{array}{}\text{(2.8)}& \underset{z\in {\mathbb{R}}^{m×1}}{\underset{⏟}{\left[\begin{array}{c}{z}_{1}\\ ⋮\\ ⋮\\ {z}_{m}\end{array}\right]}}=\underset{{W}^{\left[1\right]}\in {\mathbb{R}}^{m×d}}{\underset{⏟}{\left[\begin{array}{c}-{w}_{1}^{\left[1{\right]}^{\mathrm{\top }}}-\\ -{w}_{2}^{\left[1{\right]}^{\mathrm{\top }}}-\\ ⋮\\ -{w}_{m}^{\left[1{\right]}^{\mathrm{\top }}}-\end{array}\right]}}\underset{x\in {\mathbb{R}}^{d×1}}{\underset{⏟}{\left[\begin{array}{c}{x}_{1}\\ {x}_{2}\\ ⋮\\ {x}_{d}\end{array}\right]}}+\underset{{b}^{\left[1\right]}\in {\mathbb{R}}^{m×1}}{\underset{⏟}{\left[\begin{array}{c}{b}_{1}^{\left[1\right]}\\ {b}_{2}^{\left[1\right]}\\ ⋮\\ {b}_{m}^{\left[1\right]}\end{array}\right]}}\end{array}$$\begin{array}{}\text{(2.8)}& \underset{z\in {\mathbb{R}}^{m×1}}{\underset{⏟}{\left[\begin{array}{c}{z}_{1}\\ ⋮\\ ⋮\\ {z}_{m}\end{array}\right]}}=\underset{{W}^{\left[1\right]}\in {\mathbb{R}}^{m×d}}{\underset{⏟}{\left[\begin{array}{c}-{w}_{1}^{\left[1{\right]}^{\mathrm{\top }}}-\\ -{w}_{2}^{\left[1{\right]}^{\mathrm{\top }}}-\\ ⋮\\ -{w}_{m}^{\left[1{\right]}^{\mathrm{\top }}}-\end{array}\right]}}\underset{x\in {\mathbb{R}}^{d×1}}{\underset{⏟}{\left[\begin{array}{c}{x}_{1}\\ {x}_{2}\\ ⋮\\ {x}_{d}\end{array}\right]}}+\underset{{b}^{\left[1\right]}\in {\mathbb{R}}^{m×1}}{\underset{⏟}{\left[\begin{array}{c}{b}_{1}^{\left[1\right]}\\ {b}_{2}^{\left[1\right]}\\ ⋮\\ {b}_{m}^{\left[1\right]}\end{array}\right]}}\end{array}${:(2.8)ubrace([[z_(1)],[vdots],[vdots],[z_(m)]])_(z inR^(m xx1))=ubrace([[-w_(1)^([1]^(TT))-],[-w_(2)^([1]^(TT))-],[vdots],[-w_(m)^([1]^(TT))-]])_(W^([1])inR^(m xx d))ubrace([[x_(1)],[x_(2)],[vdots],[x_(d)]])_(x inR^(d xx1))+ubrace([[b_(1)^([1])],[b_(2)^([1])],[vdots],[b_(m)^([1])]])_(b^([1])inR^(m xx1)):}$$\underbrace{\left[\begin{array}{c} z_{1} \\ \vdots \\ \vdots \\ z_{m} \end{array}\right]}_{z \in \mathbb{R}^{m \times 1}}=\underbrace{\left[\begin{array}{c} -w_{1}^{[1]^{\top}}- \\ -w_{2}^{[1]^{\top}}- \\ \vdots \\ -w_{m}^{[1]^{\top}}- \end{array}\right]}_{W^{[1]} \in \mathbb{R}^{m \times d}} \underbrace{\left[\begin{array}{c} x_{1} \\ x_{2} \\ \vdots \\ x_{d} \end{array}\right]}_{x \in \mathbb{R}^{d \times 1}}+\underbrace{\left[\begin{array}{c} b_{1}^{[1]} \\ b_{2}^{[1]} \\ \vdots \\ b_{m}^{[1]} \end{array}\right]}_{b^{[1]} \in \mathbb{R}^{m \times 1}} \tag{2.8}$$
Or succinctly,
$\begin{array}{}\text{(2.9)}& z={W}^{\left[1\right]}x+{b}^{\left[1\right]}\end{array}$$\begin{array}{}\text{(2.9)}& z={W}^{\left[1\right]}x+{b}^{\left[1\right]}\end{array}${:(2.9)z=W^([1])x+b^([1]):}$$z=W^{[1]} x+b^{[1]} \tag{2.9}$$
We remark again that a vector in ${\mathbb{R}}^{d}$${\mathbb{R}}^{d}$R^(d)\mathbb{R}^{d} in this notes, following the conventions previously established, is automatically viewed as a column vector, and can also be viewed as a $d×1$$d×1$d xx1d \times 1 dimensional matrix. (Note that this is different from numpy where a vector is viewed as a row vector in broadcasting.)
Computing the activations $a\in {\mathbb{R}}^{m}$$a\in {\mathbb{R}}^{m}$a inR^(m)a \in \mathbb{R}^{m} from $z\in {\mathbb{R}}^{m}$$z\in {\mathbb{R}}^{m}$z inR^(m)z \in \mathbb{R}^{m} involves an elementwise non-linear application of the ReLU function, which can be computed in parallel efficiently. Overloading ReLU for element-wise application of ReLU (meaning, for a vector $t\in {\mathbb{R}}^{d}$$t\in {\mathbb{R}}^{d}$t inR^(d)t \in \mathbb{R}^{d}, $\mathrm{ReLU}\left(t\right)$$\mathrm{ReLU}\left(t\right)$ReLU(t)\operatorname{ReLU}(t) is a vector such that $\mathrm{ReLU}\left(t{\right)}_{i}=$$\mathrm{ReLU}\left(t{\right)}_{i}=$ReLU(t)_(i)=\operatorname{ReLU}(t)_{i}= $\mathrm{ReLU}\left({t}_{i}\right)\right)$$\mathrm{ReLU}\left({t}_{i}\right))${: ReLU(t_(i)))\left.\operatorname{ReLU}\left(t_{i}\right)\right), we have
$\begin{array}{}\text{(2.10)}& a=\mathrm{ReLU}\left(z\right)\end{array}$$\begin{array}{}\text{(2.10)}& a=\mathrm{ReLU}\left(z\right)\end{array}${:(2.10)a=ReLU(z):}$$a=\operatorname{ReLU}(z) \tag{2.10}$$
Define ${W}^{\left[2\right]}=\left[{w}^{\left[2{\right]}^{\mathrm{\top }}}\right]\in {\mathbb{R}}^{1×m}$${W}^{\left[2\right]}=\left[{w}^{\left[2{\right]}^{\mathrm{\top }}}\right]\in {\mathbb{R}}^{1×m}$W^([2])=[w^([2]^(TT))]inR^(1xx m)W^{[2]}=\left[w^{[2]^{\top}}\right] \in \mathbb{R}^{1 \times m} similarly. Then, the model in equation (2.6) can be summarized as
$\begin{array}{rl}a& =\mathrm{ReLU}\left({W}^{\left[1\right]}x+{b}^{\left[1\right]}\right)\\ \text{(2.11)}& {h}_{\theta }\left(x\right)& ={W}^{\left[2\right]}a+{b}^{\left[2\right]}\end{array}$$\begin{array}{r}a=\mathrm{ReLU}\left({W}^{\left[1\right]}x+{b}^{\left[1\right]}\right)\\ \text{(2.11)}& {h}_{\theta }\left(x\right)& ={W}^{\left[2\right]}a+{b}^{\left[2\right]}\end{array}${:[a=ReLU(W^([1])x+b^([1]))],(2.11)h_(theta)(x)=W^([2])a+b^([2]):}\begin{align} a &=\operatorname{ReLU}\left(W^{[1]} x+b^{[1]}\right)\nonumber \\ h_{\theta}(x) &=W^{[2]} a+b^{[2]} \tag{2.11} \end{align}
Here $\theta$$\theta$theta\theta consists of ${W}^{\left[1\right]},{W}^{\left[2\right]}$${W}^{\left[1\right]},{W}^{\left[2\right]}$W^([1]),W^([2])W^{[1]}, W^{[2]} (often referred to as the weight matrices) and ${b}^{\left[1\right]},{b}^{\left[2\right]}$${b}^{\left[1\right]},{b}^{\left[2\right]}$b^([1]),b^([2])b^{[1]}, b^{[2]} (referred to as the biases). The collection of ${W}^{\left[1\right]},{b}^{\left[1\right]}$${W}^{\left[1\right]},{b}^{\left[1\right]}$W^([1]),b^([1])W^{[1]}, b^{[1]}