Machine learning and statistics are like Chandler and Joey, best combination – you see?
So today let’s talk about another supervised learning algorithm which is absolutely based on statistics.
Enter Logistic Regression.
It is the go-to method for binary classification problems (problems with two class values). Hence it is used when the target is categorical.
For example: To predict whether a student passed (1) or not (0)
Therefore this is a classification algorithm.
The name logistic regression comes from the function used at the core of the method, the logistic function.
The logistic function, (a.k.a sigmoid function) was developed by statisticians to describe properties of population growth in ecology, rising quickly and maxing out at the carrying capacity of the environment. It is an S-shaped curve (you can see the graph below) which can take any real-value number and map it into a value between 0 and 1, but never exactly at those limits.
1 / (1 + e^-value)
e : base of the natural logarithms (Euler’s number) and value is the actual numerical value that you want to transform. Below is a plot of the numbers between -5 and 5 transformed into the range 0 and 1 using the logistic function.
Let’s discuss the mathematical expression used to describe Logistic Regression.
Input values (x) are combined linearly using weights or coefficient values to predict an output value (y). A key difference from linear regression is that the output value being modeled is a binary values (0 or 1) rather than a numeric value.
An example logistic regression equation:
y = e^(b0 + b1*x) / (1 + e^(b0 + b1*x))
y : predicted output,
b0 : bias or intercept term
b1 : coefficient for the single input value (x).
Each column in your input data has an associated b coefficient (a constant real value) that must be learned from your training data.
How does Logistic Regression work?
(let’s discuss with a real world example, also when to use linear regression and when logistic regression)
Let’s go with the good ol’ student example. To predict whether a student passed (1) or not (0). In this case, if we use simple linear regression, we will need to specify a threshold on which classification can be done.
Let say the actual class is the students who pass, and predicted continuous value is 0.85 and the threshold we have considered is 0.9, then this data point will be considered as the students who will fail and this will lead to the wrong prediction.
So we conclude that we can not use linear regression for this type of classification problem. As we know linear regression is bounded, So here comes logistic regression where value strictly ranges from 0 to 1.
Other examples could be:
- To predict whether a person will buy a car (1) or (0)
- To know whether the tumor is malignant (1) or (0)
How to implement logistic regression from scratch?
This is an implementation using numpy in Python.
Link to the source code: https://github.com/smriti-mishra/Logistic_regression
Tusen tack! (thousand thanks – in Swedish)