AI & Python #24: Create a Linear Regression using Python

Learn the core concepts of machine learning while building a linear regression model in Python

In a previous article, we’ve learned machine learning algorithms any AI enthusiast should know. Today, we’ve learned how to implement one of the most basic algorithms in Python.

P.S. If you’re not into coding, go to settings and turn off notifications for “AI & Python” (leave the rest the same to keep receiving my other emails)

One of the first machine learning algorithms we should learn is linear regression. This simple model helps us grasp core concepts of machine learning such as recognizing the dependent and independent variables, building a model, and understand the math and statistics behind a model.

There are 2 common ways to make linear regression in Python — using the statsmodel and sklearn libraries. Both are great options and have their pros and cons. In this guide, I will show you how to make a linear regression using both of them, and also we will learn all the core concepts behind a linear regression model.

Table of Contents
1. What is Linear Regression?
2. Linear Regression in Python
 - The Data
3. Linear Regression with Statsmodels
 - Simple Linear Regression
 - Multiple Linear Regression
4. Linear Regression with sklearn

What is Linear Regression?

Linear regression is an approach for modeling the relationship between two (simple linear regression) or more variables (multiple linear regression). In simple linear regression, one variable is considered the predictor or independent variable, while the other variable is viewed as the outcome or dependent variable.

Here’s the linear regression equation:

where y is the dependent variable (target value), x1, x2, … xn the independent variable (predictors), b0 the intercept, b1, b2, ... bn the coefficients and n the number of observations.

If the equation isn’t clear, the picture below might help.

Credit: Quora

In the picture, you can see a linear relationship. That is, if one independent variable increases or decreases, the dependent variable will also increase or decrease.

Linear regression can be used to make simple predictions such as predicting exams scores based on the number of hours studied, the salary of an employee based on years of experience, and so on.

Enough theory! Let’s learn how to make a linear regression in Python.

Linear Regression in Python

There are different ways to make linear regression in Python. The 2 most popular options are using the statsmodels and scikit-learn libraries.

First, let’s have a look at the data we’re going to use to create a linear model.

The Data

To make a linear regression in Python, we’re going to use a dataset that contains Boston house prices. The original dataset comes from the sklearn library, but I simplified it, so we can focus on building our first linear regression.

You can download this dataset on my Github or on Google Drive. Make sure to leave this CSV file in the same directory where your Python script is located.

Let’s have a look at this dataset. To do so, import pandas and run the code below.

import pandas as pd
df_boston = pd.read_csv('Boston House Prices.csv')
df_boston

Image by author

There are 3 columns. The “Value” column contains the median value of owner-occupied homes in $1000’s (this is what we want to predict, that is, our target value). The “Rooms” and “Distance” columns contain the average number of rooms per dwelling and weighted distances to five Boston employment centers (both are the predictors)

To sum it up, we want to predict home values based on the number of rooms a home has and its distance to employment centers.

Linear Regression with Statsmodels

Statsmodels is a module that helps us conduct statistical tests and estimate models. It provides an extensive list of results for each estimator.

If you have installed Python through Anaconda, you already have statsmodels installed. If not, you can install it either with conda or pip.

# pip
pip install statsmodels

# conda
conda install -c conda-forge statsmodels

Once you have statsmodel installed, import it using the following line of code.

import statsmodels.api as sm

The first thing to do before creating a linear regression is to define the dependent and independent variables. We’ve already discussed them in the previous section. The dependent variable is the value we want to predict and is also known as the target value. On the other hand, the independent variable(s) is the predictor.

In our dataset we have 2 predictors, so we can use any or both of them.

Let’s start with a simple linear regression. A simple linear regression estimates the relationship between one independent variable and one dependent variable.

Simple Linear Regression

For this example, I’ll choose “Rooms” as our predictor/independent variable.

• Dependent variable: “Value”

• Independent variable: “Rooms”

Let’s define the dependent and independent variables in our code as well.

y = df_boston['Value'] # dependent variable
x = df_boston['Rooms'] # independent variable

Throughout this guide, I’ll be using linear algebra notation — lower case letters will be used for vectors and upper case letters will be used for matrices.

Fitting the model
Now it’s time to fit the model. To explain to you what fitting a model means, consider the following generic equation used for simple linear regression.

𝑦 = 𝑎𝑥 + 𝑏

Credit: Quora

Fitting the model means finding the optimal values of a and b, so we obtain a line that best fits the data points. A model that is well-fitted produces more accurate outcomes, so only after fitting the model, we can predict the target value using the predictors.

Now let’s fit a model using statsmodels. First, we add a constant before fitting a model (sklearn adds it by default) and then we fit the model using the .fit() method.

x = sm.add_constant(x1) # adding a constant
lm = sm.OLS(y,x).fit() # fitting the model

“lm” stands for linear model and represents our fitted model. This variable will help us predict our target value.

>>> lm.predict(x)

0      25.232623
1      24.305975
2      31.030253
3      29.919727
4      31.231138
         ...    
501    24.603318
502    20.346831
503    27.822178
504    26.328552
505    19.661029

The code above predicts home values (printed output) based on the data inside the “Room” column.

The Regression Table

Although we could predict the target values, the analysis isn’t done yet. We need to know how this linear model performs. The regression table can help us with that. This table provides an extensive list of results that reveal how good/bad is our model.

To obtain the regression table run the code below:

lm.summary()

You will obtain this table:

Image by author

The table is titled “OLS Regression Results.” OLS stands for Ordinary Least Squares and this is the most common method to estimate linear regression.

Let’s have a look at some important results in the first and second tables.

• Dep. Variable: This is the dependent variable (in our example “Value” is our target value)

• R-squared: Takes values from 0 to 1. R-squared values close to 0 correspond to a regression that explains none of the variability of the data, while values close to 1 correspond to a regression that explains the entire variability of the data. The r-squared obtained is telling us that the number of rooms explains 48.4% of the variability in house values.

• Coef: These are the coefficients (a, b) we’ve seen in the model equation before.

• Std error: Represents the accuracy of the prediction. The lower the standard error, the better prediction.

• t, P>t (p-value): The t scores and p-values are used for hypothesis test. The “Rooms” variable has a statistically significant p-value. Also, we can say at a 95% percent confidence level that the value of “Rooms” is between 8.279 to 9.925.

Linear Regression Equation
From the table above, let’s use the coefficients (coef) to create the linear equation and then plot the regression line with the data points.

# Rooms coef: 9.1021
# Constant coef: - 34.6706

# Linear equation: 𝑦 = 𝑎𝑥 + 𝑏
y_pred = 9.1021 * x['Rooms'] - 34.6706

where y_pred (also known as yhat) is the predicted value of y (the dependent variable) in the regression equation.

Linear Regression Plot
To plot the equation let’s use seaborn.

import seaborn as sns
import matplotlib.pyplot as plt

# plotting the data points
sns.scatterplot(x=x['Rooms'], y=y)

#plotting the line
sns.lineplot(x=x['Rooms'],y=y_pred, color='red')

#axes
plt.xlim(0)
plt.ylim(0)
plt.show()

The code above produces the following plot.

Image by author

The red plot is the linear regression we built using Python. We can say that this is the line that best fits the blue data points.

Congratulation! You’ve just built your first simple linear regression in Python. If you’re up for a challenge, check how to make a multiple linear regression.

Multiple Linear Regression

Now that you already know the core concepts of linear regression, we can easily create a multiple linear regression.

Let’s start by setting the dependent and independent variables. In this case, we’re going to use 2 independent variables.

• Dependent variable: “Value”

• Independent variable: “Rooms” and “Distance”

Let’s define the dependent and independent variables in our code as well.

y = df_boston['Value'] # dependent variable
X = df_boston[['Rooms', 'Distance']] # independent variable

Now let’s add a constant and fit the model.

X = sm.add_constant(X) # adding a constant
lm = sm.OLS(y, X).fit() # fitting the model

Let’s have a look at the results.

lm.summary()

Image by author

The r-squared increased a bit. Also, there’s a new line in the second table that represents the parameters for the “Distance” variable. The analysis of this table is similar to the simple linear regression, but if you have any questions, feel free to let me know in the comment section.

Linear Regression with sklearn

Scikit-learn is the standard machine learning library in Python and it can also help us make either a simple linear regression or a multiple linear regression.

Since we deeply analyzed the simple linear regression using statsmodels before, now let’s make a multiple linear regression with sklearn.

First, let’s install sklearn. If you have installed Python through Anaconda, you already have sklearn installed. If not, you can install it either with conda or pip.

# pip
pip install scikit-learn

# conda
conda install -c conda-forge scikit-learn

Now let’s import linear_model from the sklearn library.

from sklearn import linear_model

The dependent and independent variables will be the following.

y = df_boston['Value'] # dependent variable
X = df_boston[['Rooms', 'Distance']] # independent variable

Now we have to fit the model (note that the order of arguments in the fit method using sklearn is different from statsmodels)

lm = linear_model.LinearRegression()
lm.fit(X, y) # fitting the model

Similarly to statsmodels, we use the predict method to predict the target value in sklearn.

lm.predict(X)

However, unlike statsmodels we don’t get a summary table using .summary(). Instead, we have to call each element one by one.

>>> lm.score(X, y)
0.495

>>> lm.coef_
array([8.80141183, 0.48884854])

>>> lm.intercept_
-34.636050175473315

The results are the same as the table we obtained with statsmodels.

Note that we didn’t split the data into training and test for the sake of simplicity. Splitting the data before building the model is a popular approach to avoid overfitting. That’ll be a topic for a future article, so stay tuned!

That’s it! You’ve just learned how to make a simple and multiple linear regression in Python. You can find all the code written in this guide on my Github.