aNeural Networks

Table of Content

1. Neural Networks (NN) 1.1. What is a neuron (in NN)? 1.2. Why do we need the bias term? 1.3. How a NN learns non-linear patterns? 1.4. How does a NN learn? 1.5. Chain Rule 1.6. How do we update the weights? 1.7. Stochastic Gradient Descent (SGD) 1.8. Momentum 1.9. AdaGrad 1.10. Adam 1.11. RMSProp 1.12. AdaDelta 1.13. Vanishing and exploding gradients 1.13.1. Initialization 1.14. ReLU and Leaky ReLU 1.15. tanh 1.16. Loss Functions 1.17. Avoid Overfitting 1.17.1. Regularization 1.17.2. Dropout 1.18. How to determine the number of layers and neurons?

1. Neural Networks (NN) 1.1. What is a neuron (in NN)? • Sometimes called perceptron, is a graphical representation of the smallest part of a NN that takes an input, multiply it by a weight.

x1${\displaystyle x_1}$x2${\displaystyle x_2}$x3${\displaystyle x_3}$y1${\displaystyle y_1}$∑${\displaystyle \sum }$𝜎${\displaystyle 𝜎}$𝜎(w1x1+w2x2+w3x3+b) = y1${\displaystyle 𝜎(w_1{x}_1+w_2{x}_2+w_3{x}_3+b)=y_1}$𝜎(WT.X+1.b) = Y${\displaystyle 𝜎({\mathbf{W}}^T.\mathbf{X}+1.\mathbf{b})=\mathbf{Y}}$𝜎(WT.X+1.b) = 1

1 + e-(WT.X+1.b)${\displaystyle 𝜎({\mathbf{W}}^T.\mathbf{X}+1.\mathbf{b})=\frac{1}{1+e^{-({\mathbf{W}}^T.\mathbf{X}+1.\mathbf{b})}}}$ • The 𝜎(WT.X+b)${\displaystyle 𝜎({\mathbf{W}}^T.\mathbf{X}+\mathbf{b})}$ is the same as logistic regression and the loss is exactly the same as the one in logistic regression:(1)L(y,y) = 1

Nn∑i=1yilogyi + (1-yi)log(1 - yi)$$L(\widehat{y},y)=\frac{1}{N}\sum _{i=1}^n{y}_i\log {\widehat{y}}_i+(1-y_i)\log (1-{\widehat{y}}_i)$$• Note: There are other non-linear functions used as well, such as Relu, tanh, etc.• We can update the weights by taking gradients of the loss function with respect to the weights, (2)∇L=[a

∂L ∂b

∂L ∂w1

...

∂L ∂wn

] (3)wt+1=wt - r∇L $$\begin{array}{c}\nabla L=\left[\begin{array}{c}\frac{\partial L}{\partial b}\\ \frac{\partial L}{\partial w_1}\\ ...\\ \frac{\partial L}{\partial w_n}\end{array}\right]\\ \\ w^{t+1}=w^t-r\nabla L\\ \\ \end{array}$$• Note: According to equation (3), in order to update weights, we move in the opposite direction of loss gradient adjusted by the learning rate (r${\displaystyle r}$). 1.2. Why do we need the bias term?• Bias is like the intercept added in a linear equation. It is an additional parameter in the Neural Network which is used to adjust the output along with the weighted sum of the inputs to the neuron. Thus, Bias is a constant which helps the model in a way that it can fit best for the given data.• The bias term helps in cases where all the wixi${\displaystyle w_i{x}_i}$ terms are 0${\displaystyle 0}$, which means that the model cannot be trained. Adding a bias terms let the model be trained in such cases. 1.3. How a NN learns non-linear patterns?• Each neuron in a NN learns decision boundary.• Since NN has many neurons, the combined learned decision boundaries creates a non-linear decision boundary. • In the example below, there's no way to separate the data with one line. There are feature engineering methods (or other algorithms) that can handle this. But, how a NN can separate these two classes?

y1${\displaystyle y_1}$∑${\displaystyle \sum }$x3${\displaystyle x_3}$x2${\displaystyle x_2}$x1${\displaystyle x_1}$𝜎${\displaystyle 𝜎}$𝜎${\displaystyle 𝜎}$∑${\displaystyle \sum }$𝜎${\displaystyle 𝜎}$∑${\displaystyle \sum }$b${\displaystyle b}$• The above picture is a simple example of how a NN can capture non-linear patterns. In practice, NN have more than one hidden layers and more neurons per layer.– Note: Usually the number of neurons in each hidden layer decreases as we move forward through the network. 1.4. How does a NN learn?• Let's explain this through a small NN below.

𝜎${\displaystyle 𝜎}$∑${\displaystyle \sum }$𝜎${\displaystyle 𝜎}$∑${\displaystyle \sum }$𝜎${\displaystyle 𝜎}$∑${\displaystyle \sum }$x2${\displaystyle x_2}$x1${\displaystyle x_1}$yout${\displaystyle {\widehat{y}}_{out}}$w1${\displaystyle w_1}$w2${\displaystyle w_2}$w3${\displaystyle w_3}$w4${\displaystyle w_4}$w5${\displaystyle w_5}$w6${\displaystyle w_6}$h1in${\displaystyle h_{in}^1}$h2in${\displaystyle h_{in}^2}$h1out${\displaystyle h_{out}^1}$h2out${\displaystyle h_{out}^2}$yin${\displaystyle y_{in}^{}}$h1in = w1x1+w3x2${\displaystyle h_{in}^1=w_1{x}_1+w_3{x}_2}$h2in= w2x1+w4x2${\displaystyle h_{in}^2=w_2{x}_1+w_4{x}_2}$h1out = 𝜎(h1in)${\displaystyle h_{out}^1=𝜎\left(h_{in}^1\right)}$h2out = 𝜎(h2in)${\displaystyle h_{out}^2=𝜎\left(h_{in}^2\right)}$yin = w5h1out + w6h2out${\displaystyle y_{in}^{}=w_5{h}_{out}^1+w_6{h}_{out}^2}$yout = 𝜎(yin)${\displaystyle {\widehat{y}}_{out}=𝜎(y_{in}^{})}$L(y,y) = 1

Nn∑i=1yilogyi + (1-yi)log(1 - yi)$$L(\widehat{y},y)=\frac{1}{N}\sum _{i=1}^n{y}_i\log {\widehat{y}}_i+(1-y_i)\log (1-{\widehat{y}}_i)$$Loss for one example: yout = 0.33, y = 1${\displaystyle {\widehat{y}}_{out}=0.33,y=1}$aL = log(yout) = log(𝜎(yin)) = log(𝜎(w5h1out + w6h2out)) = log(𝜎(w5𝜎(h1in) + w6𝜎(h2in))) = log(𝜎(w5𝜎(w1x1+w3x2) + w6𝜎(w2x1+w4x2)))${\displaystyle \begin{array}{l}L=\log ({\widehat{y}}_{out})\\ =\log (𝜎(y_{in}^{}))\\ =\log \left(𝜎\right(w_5{h}_{out}^1+w_6{h}_{out}^2\left)\right)\\ =\log \left(𝜎\right(w_5𝜎\left(h_{in}^1\right)+w_6𝜎\left(h_{in}^2\right)\left)\right)\\ =\log (𝜎(w_5𝜎(w_1{x}_1+w_3{x}_2)+w_6𝜎(w_2{x}_1+w_4{x}_2)))\end{array}}$The loss function is →1.5. Chain Rule

* Now, we have to take the gradient of L.${\displaystyle L.}$ Since loss function is a complex functionit's hard to derive the analytical gradient. * Since the loss function is essentially a function of functions, we use the chain rule tocompute the derivatives. For example,∂L

∂w1 = ∂L

∂yout.∂yout

∂yin.∂yin

∂h1out.∂h1out

∂h1in.∂h1in

∂w1${\displaystyle \frac{\partial L}{\partial w_1}=\frac{\partial L}{\partial {\widehat{y}}_{out}}.\frac{\partial {\widehat{y}}_{out}}{\partial y_{in}}.\frac{\partial y_{in}}{\partial h_{out}^1}.\frac{\partial h_{out}^1}{\partial h_{in}^1}.\frac{\partial h_{in}^1}{\partial w_1}}$∂L

∂w6 = ∂L

∂yout.∂yout

∂yin.∂yin

∂w6${\displaystyle \frac{\partial L}{\partial w_6}=\frac{\partial L}{\partial {\widehat{y}}_{out}}.\frac{\partial {\widehat{y}}_{out}}{\partial y_{in}}.\frac{\partial y_{in}}{\partial w_6}}$* Note that the first two terms of ∂L

∂w6${\displaystyle \frac{\partial L}{\partial w_6}}$ and ∂L

∂w1${\displaystyle \frac{\partial L}{\partial w_1}}$ are the same. This means that we can use dynamic programming + chain rule to calculate derivatives. * We start by first calculating ∂L

∂w6${\displaystyle \frac{\partial L}{\partial w_6}}$ and work our way to ∂L

∂w1${\displaystyle \frac{\partial L}{\partial w_1}}$. * This gives us something called backpropagation. Backpropagation is the standard wayto train NN. * For training we need: * Forward Pass → To figure out how far our predictions are from the actual value * Backpropagation → Once we have the loss, we can backpropagate those gradients to update all of the weights in our NN.1.6. How do we update the weights?

wt+1 = wt - r∇L${\displaystyle w^{t+1}=w^t-r\nabla L}$* If we plot the average loss obtained from all of the trained examples against a particularparameter, say w1${\displaystyle w_1}$, we get a function like below,L(y,y)${\displaystyle L(\widehat{y},y)}$w1${\displaystyle w_1}$* The difference between this function and the logistic regression function is that herewe have local optima. In logistic regression, we were guaranteed to have one minimum. * That's because we stacked up neurons and added layers, so we opened up ourselves tolocal optima. 1.7. Stochastic Gradient Descent (SGD)

* There are some techniques that increases the chance of not getting stuck in the local optima. The most popular method is Stochastic Gradient Descent (SGD). * SGD's characteristic of not getting stuck in local optima is just a by-product of takingrandom examples and updating the weights with just that single example. This randomnessin the weight updates, can increase the chances that we don't get stuck in a local optima. * The problem with SGD is that it's slow to converge. * One idea to speed up convergence is by incorporating momentum. * The idea of momentum is to keep track of the previous updates. 1.8. Momentum

* The problem with SGD is that it's slow to converge. * One idea to speed up convergence is by incorporating momentum. * The idea of momentum is to keep track of the previous updates. awt+1 = wt - r∇Lt - 𝛾 r (∇Lt-1+∇Lt-2+…+∇Lt-n) or wt+1 = wt - Vt Vt = 𝛾Vt-1 - r∇L${\displaystyle \begin{array}{l}{w}^{t+1}=w^t-r\nabla L^t-𝛾r(\nabla L^{t-1}+\nabla L^{t-2}+\dots +\nabla L^{t-n})\\ \\ \text{or}\\ w^{t+1}=w^t-{\mathbf{V}}^t\\ {\mathbf{V}}^t=𝛾{\mathbf{V}}^{t-1}-r\nabla L\end{array}}$* The 𝛾${\displaystyle 𝛾}$ parameter is usually set to 0.9${\displaystyle 0.9}$ so that the previous gradient doesn't matter as much as the current gradient. * The problem with momentum is that sometimes we could build so much momentum that we pass the global optima.1.9. AdaGrad

* There's another method called AdaGrad which adjusts the learning rate per parameter.*Note: rgeneral${\displaystyle r_{general}}$ is typically set to 0.001${\displaystyle 0.001}$.wt+11 = wt1 - rt1∂Lt

∂w1${\displaystyle w_1^{t+1}=w_1^t-r_1^t\frac{\partial L^t}{\partial w_1}}$rt1=rgeneral

(∂Lt-1

∂w1)2 + … + (∂Lt-n

∂w1)2+𝜀${\displaystyle r_1^t=\frac{r_{general}}{\sqrt{(\frac{\partial L^{t-1}}{\partial w_1}{)}^2+\dots +(\frac{\partial L^{t-n}}{\partial w_1}{)}^2}+𝜀}}$* Why would you want to do something like this? * It balances the update value at each step such that when gradient is high it lowers the learning rate and when the gradient is low it increase the learning rate. * This way it moderates the steps we take at each update (and for each parameter). * Note: The 𝜀${\displaystyle 𝜀}$ term in the denominator is set to a small value to avoid dividing by 0${\displaystyle 0}$. * Note: AdaGrad really helps in the case of sparse features, because if we have sparse features, that means that the weights associated with those features will be updated less, and therefore the learning rate will be higher.1.10. Adam

* The other method is Adam.* Adam combines momentum and adaptive learning rate.wt+1 = wt - rgeneral

Vt+𝜀mt${\displaystyle w_{}^{t+1}=w_{}^t-\frac{r_{general}}{\sqrt{{\widehat{V}}_t}+𝜀}{\widehat{m}}_t}$mt = 𝛽1mt-1+(1-𝛽1)∇tloss${\displaystyle m_t={𝛽}_1{m}_{t-1}+(1-{𝛽}_1){\nabla }_{loss}^t}$Vt = 𝛽2Vt-1+(1-𝛽2)(∇tloss)2${\displaystyle V_t={𝛽}_2{V}_{t-1}+(1-{𝛽}_2)({\nabla }_{loss}^t{)}^2}$* Note: 𝛽1${\displaystyle {𝛽}_1}$ and 𝛽2${\displaystyle {𝛽}_2}$ are hyperparameters.* Note: The only difference between mt${\displaystyle m_t}$ and Vt${\displaystyle V_t}$ is the squared gradient loss term.* Note: Notice that the mt${\displaystyle m_t}$ and Vt${\displaystyle V_t}$ are adjusted (i.e. mt${\displaystyle {\widehat{m}}_t}$ and Vt${\displaystyle {\widehat{V}}_t}$). The reason is because these terms are technically moments of a function, and in order to get an unbiased moment on these functions we have to adjust them by the 𝛽${\displaystyle 𝛽}$ parameters.* Note: Adam looks like a ball rolling down a hill with momentum, but the ball also has friction. The idea is that the friction helps the parameters settle in the global optima, while the momentum helps the parameters escape the local minimum.mt = mt

1-𝛽t1, Vt = Vt

1-𝛽t2${\displaystyle {\widehat{m}}_t=\frac{m_t}{1-{𝛽}_1^t},{\widehat{V}}_t=\frac{V_t}{1-{𝛽}_2^t}}$ 1.11. RMSProp• Will add the note later 1.12. AdaDelta• Will add the note later 1.13. Vanishing and exploding gradients• Once we have the gradients, from whatever optimizer we use, multiplying these gradients together can result in a problem.– Let's say if we use the sigmoid activation function, the maximum value of the gradients are 0.25${\displaystyle 0.25}$.– Now, if we multiply a lot of 0.25${\displaystyle 0.25}$s together, this final gradient (based on chain rule) will brace towards 0${\displaystyle 0}$ → this result in underflow.– This is called → vanishing gradient.• Also, some of the gradient terms include the value of weights. – If the weights are extremely large, by multiplying them together, we can end up getting an extremely large value → This is called an exploding gradient.• There are a few methods to mitigate these problems. 1.13.1. Initialization• One of the method to tackle the vanishing/exploding gradients is to initializing the weights of the NN in a particular way.• A bad way to initialize the weights is just to use a uniform distribution between 0${\displaystyle 0}$ and 1${\displaystyle 1}$. • Another bad way to just to initialize these parameters with a normal distribution in which the mean is 0${\displaystyle 0}$ and the standard deviation is 1${\displaystyle 1}$.• What we can do instead is to initialize the weights from a normal distribution in which the mean is 0${\displaystyle 0}$ but the standard deviation is 𝜎=2⁄(fi+fo)${\displaystyle 𝜎=\sqrt{2⁄(f_i+f_o)}}$. • fi${\displaystyle f_i}$ is fan in and fo${\displaystyle f_o}$ is fan out. • Fan in is the number of inputs to a particular layer and fan out is the number of outputs for that layer.• This way, we can initialize the weights for each layer of NN.• This is called Xavier or Glorot initialization.• The reason why this helps is because we're shrinking the standard deviation by how many ever times we will be multiplying these variables together per layer. – Not doing this makes the variances of each layer multiply together and that causes the variance to grow exponentially. – So, if we can shrink down the standard deviation early, these the other multiplications, hopefully, won't result in exponential growth (or shrinkage) of the gradients.• This works best when we use something called a symmetric activation function. Example of such functions → sigmoid function.• 1.14. ReLU and Leaky ReLU • What if we want to use a non-symmetric activation function?– Example of such functions is ReLU (Rectified Linear Unit) function.

ReLU = max(0, x)${\displaystyle ReLU=\max (0,x)}$• Why do we want to use ReLU?– More computationally efficient → All the negative values take on the value of 0${\displaystyle 0}$, and all positive values take on the value itself → When taking derivatives, the derivatives of 0${\displaystyle 0}$ is 0${\displaystyle 0}$ and the derivative of any value is just 1${\displaystyle 1}$.– Tends to produce better model performance– Sparsity → reduce overfitting → not all neurons will output a value (negative values → 0${\displaystyle 0}$). – • What are the downsides of ReLU?– It has an uncapped activation. With sigmoid, we'd have something called saturation, where the output of the neuron could be no larger than the value of 1${\displaystyle 1}$.– However, the ReLU can output any value, which means that we could be susceptible to exploding gradients more often.– As well, we can even now be susceptible to exploding forward passes where by simply doing multiplications in the forward pass all the way through the NN, we can also get unreasonably large numbers that overflow.– Another problem called dying ReLU problem.* It comes from the fact that a neuron that takes on a value of 0${\displaystyle 0}$ will be 0${\displaystyle 0}$ forever.* That means that the neuron will be completely dead and never output another value except 0${\displaystyle 0}$.– Even with these problems, ReLU activation functions are used often in practice.– • Initialization for ReLU– Instead of Xavier initialization, we use the Kaiming initialization → 𝜎=2⁄fi${\displaystyle 𝜎=\sqrt{2⁄f_i}}$– The Kaiming initialization can be used for other asymmetric activation functions like Leaky ReLU → f(x)= aa

x	if x>0
0.01x	otherwise

${\displaystyle f(x)=\{\begin{array}{cc}x& ifx>0\\ 0.01x& \text{otherwise}\end{array}}$– Leaky ReLU tries to get around the dead neuron problem by adding a slight angle to the slope.

(x)= aa

x	if x>0
0.01x	otherwise

${\displaystyle (x)=\{\begin{array}{cc}x& ifx>0\\ 0.01x& \text{otherwise}\end{array}}$• Another thing to do (in addition to the initialization) is feature scaling.• 1.15. tanh• tanh${\displaystyle tanh}$ is very similar to the sigmoid function, but instead of being in the range of [0,1]${\displaystyle [0,1]}$, it lies in the range of [-1,1]${\displaystyle [-1,1]}$. • The idea to cross validate between all the activation functions to see which works best for your data.• Note: Different activation functions can be used at different layers of NN.• Note: The last neuron will dictate what the output looks like. sigmoid → binary classification, softmax → multi-class classification (the maximum value of softmax function is your prediction), linear regression → linear activation function 1.16. Loss Functions• Regression → Mean Squared Error (MSE) → L(y, y) = ∑N(yi - yi)2

N${\displaystyle L(y,\widehat{y})=\frac{\sum _N^{}(y_i-{\widehat{y}}_i{)}^2}{N}}$– N${\displaystyle N}$ is usually the batch_size.• Regression → Mean Absolute Error (MAE) → L(y, y) = ∑N|yi - yi|2

N${\displaystyle L(y,\widehat{y})=\frac{\sum _N^{}|y_i-{\widehat{y}}_i{|}^2}{N}}$• Classification → Cross Entropy (sometimes called logloss) → L(y, y) = -(ylogyi + (1-y)log(1-yi)${\displaystyle L(y,\widehat{y})=-(y\log {\widehat{y}}_i+(1-y)\log (1-{\widehat{y}}_i)}$• Classification → Cross Entropy for K${\displaystyle K}$ classes → L(y, y) = - ∑kyilog(yi)${\displaystyle L(y,\widehat{y})=-\sum _k^{}{y}_i\log ({\widehat{y}}_i)}$1.17. Avoid Overfitting1.17.1. Regularization• We can do regularization by adding L1${\displaystyle L_1}$ or L2${\displaystyle L_2}$ term to the loss function → L(y, y) = - ∑kyilog(yi) + 𝜆 ∑w|wi|${\displaystyle L(y,\widehat{y})=-\sum _k^{}{y}_i\log ({\widehat{y}}_i)+𝜆\sum _w^{}|w_i|}$1.17.2. Dropout• Dropout is when you have, per layer of the NN, a particular neuron in that layer that will have some probability of sticking around. The others will be dropped out for this training iteration.• For each layer we assign a dropout probability (e.g. P = 0.5${\displaystyle P=0.5}$).• The problem with dropout is that during training, the dropped out neurons (during training) will not drop out during prediction → so, all of the sudden, the last node summation will be a lot higher (because we have all the neurons).– To solve this problem, we can use inverted dropout. During training (after every mini-batch), they'll take the output of the layers and divide by the dropout rate → output

dropout rate${\displaystyle \frac{\text{output}}{\text{dropout rate}}}$ .– This ensures that the total sum coming into the last node will match on average the total sum coming to it during prediction time. 1.18. How to determine the number of layers and neurons?• If your data is linearly separable, you don't need any hidden layer at all.• Beyond that, it's safe to start with a single hidden layer, and the number of neurons in that single hidden layer should be the average of input and output.• Another alternative is to start with more layers or units than you need, and then go examine the weights of your connections.– The weights that are close to 0${\displaystyle 0}$, should allow you to prune the surrounding neuron.– Once, you drop the neuron, you run the cross validation to see how much the NN model performance is affected. Back to Top