aLogistic Regression

Table of Content

1. Logistic Regression 1.1. Coefficient interpretation 1.2. Multinomial regression 1.3. Regularization 1.3.1. Why Lasso regularization induce model sparsity? 1.4. Early stopping 1.5. Other considerations

1. Logistic Regression• Logistic regression is based on the same idea as linear regression in the way that we still use a line to designate our model. The only difference is that we now want y${\displaystyle y}$ to be a probability.• The probability equation, which is a sigmoid function, is:P(y|X)=1

1+e-(𝛽0+𝛽X)$$P(y|X)=\frac{1}{1+e^{-({𝛽}_0+\mathbf{𝛽}X)}}$$• Unlike linear regression, there's no closed form solution for logistic regression.• The loss function for logistic regression is the log loss (cross-entropy loss):• Loss(y,y)=- ∑n[yilogyi +(1-yi)log(1-yi)]$$Loss(y,\widehat{y})=-\sum _n^{}[y_i\log {\widehat{y}}_i+(1-y_i)\log (1-{\widehat{y}}_i)]$$ • Note: Log Loss is a slight twist on the likelihood function. In fact → log loss = -1 × log(likelihood function)${\displaystyle logloss=-1\times \log (likelihoodfunction)}$.• Note: The likelihood function of logistic regression is:L(𝛽0,𝛽)=n∏i=1p(xi)yi(1-p(xi))1-yi$$L({𝛽}_0,𝛽)=\prod _{i=1}^{n}p(x_i{)}_{}^{y_i}(1-p(x_i){)}^{1-y_i}$$• To minimize the loss function, we take derivatives w.r.t. coefficients:• dLoss

d𝛽= ∑n[yi-yi]xi dLoss

d𝛽0= ∑n[yi-yi].1 𝛻𝛽=[a

dLoss d𝛽

dLoss d𝛽0

] 𝛽t+1i=𝛽ti-r𝛻𝛽i → until coefficient gradients converge to 0 r → learning rate → usually [10-6,0.1]$$\begin{array}{c}\frac{dLoss}{d𝛽}=\sum _n^{}[{\widehat{y}}_i-y_i]x_i\\ \\ \frac{dLoss}{d{𝛽}_0}=\sum _n^{}[{\widehat{y}}_i-y_i].1\\ \\ {𝛻}_{𝛽}=\left[\begin{array}{c}\frac{dLoss}{d𝛽}\\ \frac{dLoss}{d{𝛽}_0}\end{array}\right]\\ \\ {𝛽}_i^{t+1}={𝛽}_i^t-r{𝛻}_{{𝛽}_i}\to \text{until coefficient gradients converge to 0}\\ \\ r\to \text{learning rate}\to \text{usually}[{10}^{-6},0.1]\end{array}$$1.1. Coefficient interpretation• In order to interpret the impact of coefficient 𝛽${\displaystyle 𝛽}$ on the probability, we have to exponentiate it, e𝛽${\displaystyle e^{𝛽}}$, to get something called the odds ratio.– 1-e𝛽${\displaystyle 1-e^{𝛽}}$ gives the % change in the odds. 1.2. Multinomial regression• We use multinomial regression when we want to predict more than two classes.• Instead of sigmoid, we're going to use softmax function. P(y=k|xi)=e𝛽0+𝛽xi

K∑j=1e𝛽0+𝛽xj$$P(y=k|x_i)=\frac{e^{{𝛽}_0+𝛽x_i}}{\sum _{j=1}^K{e}^{{𝛽}_0+𝛽x_j}}$$ • A softmax function is generalized sigmoid such that it produces the probability among K${\displaystyle K}$ classes.– The predicted value will the be class with the maximum predicted probability. 1.3. Regularization• Regularization is a techniques used to avoid overfitting which involves adding a term to the loss function which is the sum of all coefficients. There are two main types of regularization:– L1${\displaystyle L1}$, Lasso, or Laplace → ∑j|𝛽j|${\displaystyle \sum _j^{}|{𝛽}_j|}$* Typically results in more zero-valued coefficients, which means fewer features will be used.– L2${\displaystyle L2}$, ridge, Gaussian → ∑j𝛽2j${\displaystyle \sum _j^{}{𝛽}_j^2}$* Usually results in small weights for many of the features (that would've been out by L1${\displaystyle L1}$).– Note: Both L1${\displaystyle L1}$ and L2${\displaystyle L2}$ usually have a coefficient, 𝜆${\displaystyle 𝜆}$, multiplied to them which allows to control the degree of regularization.* Two high 𝜆${\displaystyle 𝜆}$ can result in under-fitting and too low can result in overfitting.* 𝜆${\displaystyle 𝜆}$ is best tuned in cross validation.– Note: When using regularization, it's better to scale our data. Scaling data can also help the model to converge faster.1.3.1. Why Lasso regularization induce model sparsity?• First off, note that– L1 norm: ||w||1=|w1|+|w2|+|w3|+…+|wn|${\displaystyle ||w|{|}_1=|w_1|+|w_2|+|w_3|+\dots +|w_n|}$– L2 norm: ||w||2=w21+w22+w23+…+w2n${\displaystyle ||w|{|}_2=\sqrt{w_1^2+w_2^2+w_3^2+\dots +w_n^2}}$• • When optimizing the cost function, we use gradient descent and update our weights by → wt+1=wt-r∇w${\displaystyle w^{t+1}=w^t-r{\nabla }_w}$– Convergence occurs when the value of wt${\displaystyle w^t}$ doesn't change much with further iterations → i.e. 𝜕Loss

𝜕w≈0${\displaystyle \frac{𝜕Loss}{𝜕w}\approx 0}$ → i.e. wt+1≈wt${\displaystyle w^{t+1}\approx w^t}$. • L1 norm: The derivative is → 𝜕|w|

𝜕w=1${\displaystyle \frac{𝜕|w|}{𝜕w}=1}$, therefore → wt+1=wt-r.1${\displaystyle w^{t+1}=w^t-r.1}$.– We can see that our loss derivative becomes a constant, so the condition of convergence occurs faster because we only have r${\displaystyle r}$ in the subtraction terms and it's not being multiplied by any smaller value of w${\displaystyle w}$.– Therefore, wt${\displaystyle w^t}$ tends towards zero in a few iterations. • • L2 norm: The derivative → 𝜕w2

𝜕w=2w${\displaystyle \frac{𝜕w^2}{𝜕w}=2w}$, therefore → wt+1=wt-2.r.w${\displaystyle w^{t+1}=w^t-2.r.w}$.– We can see that our loss derivative term is not constant and thus for smaller values of w${\displaystyle w}$, our condition of convergence will not occur faster (or maybe at all) because we have a smaller value of w${\displaystyle w}$ getting multiplied with r${\displaystyle r}$ and thus making the whole term to be subtracted even smaller. – Therefore, after a few iterations, our wt${\displaystyle w^t}$ becomes a very small constant value but not zero. – Hence, not contributing to the sparsity of the weight vector. 1.4. Early stopping• Another technique to avoid overfitting is early stopping.• Simply, it means to stop training somewhere before reaching the absolute minimum [of loss function] to avoid overfitting the training examples. 1.5. Other considerations• We can't use the same R2${\displaystyle R^2}$ from the linear regression. For logistic regression we use something called McFadden's pseudo R2${\displaystyle R^2}$. which also lies between 0${\displaystyle 0}$ and 1${\displaystyle 1}$.– Its value is usually smaller than R2${\displaystyle R^2}$. A value of 0.2${\displaystyle 0.2}$ and 0.4${\displaystyle 0.4}$ usually indicates an excellent fitting model.• Logistic regression → discriminative model– Naive Bayes → generative model Back to Top