Two years ago, Microsoft announced Python in Excel.

Microsoft said it was a significant evolution in the analytical capabilities available within Excel. They combined the power of Python with the flexibility of Excel.

This integration is another good reason to learn Python in 2025.

With this integration, you can write Python code in Excel cells, create advanced visualizations using libraries such as matplotlib and seaborn, and even apply machine learning techniques using libraries like scikit-learn and statsmodels.

Python in Excel will change the way Excel analysts work. Just a few months ago, it became generally available to eligible Enterprise and Business users. If you’ve recently taken one of my Python courses or are unsure whether to learn Python in 2025, keep reading.

Artificial Corner is a reader-supported publication. Please, consider becoming a free or paid subscriber👇

Subscribe now

Excel analysts will have to learn to code

Excel is probably the most popular tool for data analysis because of its usability and flexibility.

Excel users don’t need to know how to code to clean data or create visualizations and macros. With a couple of formulas and some clicks, we can wrangle data and create pivot tables and charts in Excel.

Excel alone was great for doing basic data analysis, but its limitations didn’t allow data analysts to do complex data transformations and create advanced visualizations (let alone apply machine learning techniques). In contrast, programming languages like Python can handle complex computations.

Now, Excel analysts will need to learn Python to future-proof their careers.

But will they adapt? Well, the closest most Excel users have been to a programming language was Visual Basic for Applications (VBA), but even those who write VBA code don’t call themselves “programmers.” That’s why most Excel users saw learning to code as something either complex or unnecessary (why learn to code when you could get a pivot table with one click?)

Hopefully, Excel analysts will adapt. The good news is that Python is an easy language to learn. Excel users don’t even need to install Python on their computers and download a code editor to start writing Python code. In fact, there’s a new PY function in Excel that allows users to write Python code in an Excel cell.

Amazing, isn’t it? Now we can write Python code in a cell to get a dataframe, and visualizations inside our worksheet.

This is definitely an evolution in the analytical capabilities in Excel.

Subscribe now

Python libraries for data analysis are available in Excel. This benefits both Python and Excel analysts

Now it’s possible to use powerful Python libraries such as pandas, seaborn, and scikit-learn in an Excel workbook. These libraries will help us do advanced analytics, create amazing visualizations, and apply machine learning, predictive analytics, and forecasting techniques in Excel.

Excel analysts who can’t write Python code will have to settle for pivot tables, formulas, and Excel charts, but those who adapt will take their analytical skills to the next level.

Here are some examples of how doing data analysis looks with Python in Excel.

With Python in Excel, we’ll be able to use regular expressions (regex) to locate specific strings or patterns of text in cells. In the example below, a regex is used to extract dates from text.

Advanced visualizations such as heatmaps, violin maps, and swarm plots are now possible in Excel with Seaborn. Here’s the typical pair plot we’d create with Seaborn, but now displayed in an Excel worksheet.

Last, but not least, now you can use machine learning models like a DecisionTreeClassifier in an Excel worksheet and fit the model using pandas dataframes.

Subscribe now

Python in Excel will bridge the gap between Python and Excel analysts

The days when Python and Excel analysts struggled to work together are nearly over.

Excel analysts will need to adapt to these new changes, not only to have Python as a new skill in their curriculum but to future-proof their careers. Learning VBA won’t be as relevant for Excel analysts as learning Python libraries such as Pandas and Numpy.

Python calculations will be run in the Microsoft Cloud, so even analysts running computers with limited resources will experience faster processing for complex computations.

On the other hand, Python analysts will be able to collaborate more easily with Excel analysts, bridging the gap between them.

Learning Python in 2025 becomes more relevant thanks to this integration.

Learning Python is a fundamental first step in becoming a data scientist. Python’s simplicity and ecosystem have made it the most popular programming language for data science.

That’s why I created a free 17-hour video course to learn it from scratch!

This complete course for absolute beginners goes from learning basic Python to applying machine learning on a text classification problem. Each topic builds on the previous ones, reflecting the real workflow of data science projects.

Let’s see each module in more detail!

To get weekly articles like this, subscribe 👇 After subscribing, check my welcome email to download my Python, ChatGPT, and more cheat sheets :)

Subscribe now

Module 1: Python Basics for Data Science

For those new to programming, a crash course in core Python is crucial before diving into data science tools.

In this introductory module, we’ll learn basic Python syntax, data types (numbers, strings, lists, dictionaries), control structures (loops and conditionals), and functions. Mastering these basics will provide a strong foundation to write and understand code for data analysis tasks.

What does coding in Python look like?

# Basic Python operations

x = 5
y = 3
print("Sum:", x + y)

# Looping through a list

fruits = ["Apple", "Banana", "Cherry"]
for fruit in fruits:
    print("I like", fruit)

Here’s the link to watch the first module of this course

Link: Module 1 - Python Basics for Data Science

Module 2: Introduction to Pandas and NumPy

With core Python skills in place, the next step is learning NumPy and pandas, two libraries that form the backbone of data science in Python. These libraries provide high-performance data structures and functions that make it vastly easier to work with data.

Why are Pandas and NumPy essential? Together, they allow data scientists to load, manipulate, and analyze data efficiently. For example, with pandas one can read a CSV file into a DataFrame in one line of code and then easily filter or summarize it. NumPy provides the numerical foundation, enabling operations like vectorized calculations (applying a formula to an entire array) that are both convenient and fast.

The snippet below shows how one might use NumPy and pandas in practice. We create a NumPy array and compute a statistic, and we create a pandas DataFrame to perform an operation on a column:

import numpy as np
import pandas as pd

# Using NumPy for numerical computations
data = np.array([1, 2, 3, 4])
print("NumPy mean:", data.mean())

# Using pandas for tabular data manipulation
df = pd.DataFrame({'A': [10, 20, 30],
                   'B': [2, 4, 6]})
print("DataFrame sum of A:", df['A'].sum())

Here’s the link to watch the second module of this course:

Link: Module 2 - Introduction to Pandas and NumPy

Module 3: Web Scraping with Pandas (Project #1)

The goal of this first project is to learn how to gather data from the web and import it into Python for analysis using pandas.

In many real-world situations, data is not neatly provided as a file – it might be embedded in web pages (HTML tables, lists, etc.) or available through web APIs. Web scraping is the technique of extracting data from websites, and it’s an essential skill for data scientists to build their own datasets. This project teaches beginners how to perform basic web scraping and load the data into a pandas DataFrame, which they can then analyze.

Pandas provides convenient functions for simple scraping tasks. For example, if a Wikipedia page contains a table, read_html can often fetch it directly. For more complex web scraping tasks – such as handling JavaScript, or scraping non-tabular data, we would need more powerful libraries like Selenium or Scrapy. That said, pandas will give you a gentle introduction to web scraping.

Web scraping with pandas is sometimes as easy as this:

import pandas as pd

url = "site link"
tables = pd.read_html(url)
df = tables[0]                    
print(df.head())    

Here’s the link to watch the third module of this course:

Link: Module 3 - Web Scraping with Pandas (Project #1)

Module 4: Filtering Data & Data Extraction

Once data is loaded into a pandas DataFrame, a fundamental operation is filtering the data. “Filtering” means selecting only those subsets of data (rows or columns) that meet certain criteria

Why is filtering essential? Real datasets often contain more information than you need for a particular analysis. For example, if you have a dataset of worldwide economic indicators but you are only studying data for a specific country, you would filter the dataset to that country. Similarly, if you have collected time-series data for a range of dates but only need the last year of data, filtering by date is necessary. Also, in data cleaning (a module we’ll see later), filtering is used to remove outliers or invalid entries.

Here’s what filtering data looks like:

import pandas as pd

df = pd.DataFrame({'Name': ['Alice', 'Bob', 'Cathy'],
                   'Age':  [24, 27, 19]})

adults = df[df['Age'] > 20]   # filter rows where Age > 20
print(adults)

Here’s the link to watch the fourth module of this course:

Link: Module 4 - Filtering Data & Data Extraction

Module 5: Making Data Visualizations (Project #2)

Data visualization is an important part of data science. It allows us to translate complex datasets into visual representations that are easier for the human brain to understand. By plotting data, we can quickly identify patterns, trends, and outliers that might not be obvious from raw numbers. Visualization is also key to communicating results to others. A well-crafted chart can convey insights that would be hard to glean from tables.

Python offers different libraries for data visualization, such as Matplotlib and Seaborn. That said, probably the easiest way to make visualizations with Python is using the pandas library.

This second project consists in making visualizations such as barplots, histograms, scatterplots, and more with pandas.

Link: Module 5 - Making Data Visualizations (Project #2)

Subscribe now

Module 6: GroupBy, Aggregate Function, and Concatenation

Many analytical questions are of the form “what is the value by some category?” For example, “What is the average income by country?”, “How many transactions per day of the week?” or “What is the total sales for each product category?“

GroupBy allows you to answer these by aggregating data within each subgroup.

Common aggregation methods include .sum(), .mean(), and .count(), among others. For example, if you want to get the average income for each country present in a dataframe, you’d do something like this:

df.groupby('Country')['Income'].mean() 

Here’s the link to watch the fourth module of this course:

Link: Module 6 - GroupBy and Aggregate Function

Module 7: Regular Expressions

Data scientists often work with textual data, whether it’s extracting specific information from strings or cleaning and preprocessing raw text for analysis. Regular expressions (regex) are a powerful tool for such tasks.

A regular expression is essentially a pattern that describes a set of strings. With regex, you can test if a string matches a pattern, find all substrings that match, or replace parts of a string that match a pattern.

Here’s an example of how to use regex to filter data inside a dataframe.

import pandas as pd

# Small dataset
data = {
    'Name': ['Alan', 'Bob', 'Ann'],
    'Age': [25, 22, 27]
}

df = pd.DataFrame(data)

# Using regex to filter names that start with 'A' and end with 'n'
filtered_df = df[df['Name'].str.contains(r'^A.*n$', regex=True)]

print(filtered_df)

Here’s the link to watch the seventh module of this course:

Link: Module 7 - Regular Expressions

Module 8: Data Cleaning with Pandas (Project #3)

“Garbage in, garbage out” is a saying that highlights the importance of data cleaning.

Real-world data is often noisy, incomplete, or inconsistent. Before any analysis or modeling, this raw data needs to be cleaned and structured properly. Data professionals spend a large portion of their time on data cleaning – by some estimates, around 80% of a data scientist’s time is spent preparing and cleaning data. Data cleaning is thus one of the essential steps in the data science process, as the quality of insights you can draw is directly linked to the quality of your data.

In this third project, we will use pandas to perform common data cleaning tasks on a messy dataset.

Link: Module 8 — Data Cleaning with Pandas (Project #3)

Module 9: Machine Learning with Python

After cleaning and exploring data, the next major step in a data science journey is often to build predictive models – in other words, to apply machine learning (ML).

Machine learning allows data scientists to extract patterns from data and make predictions or decisions without being explicitly programmed with rules. In practical terms, this is how we go from analyzing what has happened (descriptive analytics) to predicting what might happen (predictive analytics) or making automated decisions (like classifying an email as spam or not spam). Understanding ML opens up a huge range of applications: you can train a model to predict housing prices from past data, classify images, cluster customers into groups, etc.

In the video below, we’ll learn the core concepts of machine learning while building a linear regression model in Python. We'll use 2 basic machine learning libraries in Python: statsmodels and sklearn.

Link: Module 9 - Machine Learning with Python

Module 10: Text Classification (Project #4)

The goal of this last project is to create a machine learning model that will predict whether a movie review is positive or negative. This is known as binary text classification and will help us explore the scikit-learn library while building a basic machine learning model from scratch.

Text classification is a supervised learning task where the input features are not numeric measurements but text strings. Thus, it introduces the concept of feature extraction from text. Typically, one must convert text into a numerical representation before feeding it to a machine learning model. The most common approach is the Bag-of-Words model or its variant TF-IDF:

• Bag-of-Words: We create features from word occurrences: for each document (text example), we count how often each word from a vocabulary appears. This transforms the text into a vector of numbers (word counts). For example, if our vocabulary is {"bad", "good", "great", "terrible"}, a review "This product is good and great" might transform into [0, 1, 1, 0]

• TF-IDF: It’s a refinement of word counts that downweights common words and upweights rare but potentially informative words.

These and more new concepts are covered in the project below.

Link: Module 10 - Text Classification (Project #4)

That’s it! Let your Python for data science journey begin

Subscribe now

This is an article written by me that was published on AI Supremacy some weeks ago. Today I’m making this article free for all of you.

If you’re new, subscribe to get weekly articles like this👇

Subscribe now

Tools we know well like ChatGPT and Gemini can make learning to code easier and less frustrating. Also, less-known tools like NotebookLM are changing how we learn new subjects and coding is one of them.

In this article, we’ll explore how to make the most of these AI tools for learning to code. Since I often write about the Python programming language, I’ll use Python as an example. However, everything discussed here can be applied to other programming languages as well.

ChatGPT Canvas: Run and debug your code within ChatGPT

Canvas is a new interface for working with ChatGPT on writing and coding projects. Thanks to Canvas, now you can use ChatGPT as if it was your AI code editor.

This is a feature that I’d definitely use if I had to learn Python again because it allows us to do many things within ChatGPT, such as running Python code, debugging code, and asking in-line feedback and suggestions.

To use Canvas, just ask ChatGPT to write a simple “Hello World” script, and then click on view tools and select “Canvas.” Once you press enter, you’ll see a panel on the right. That’s your Python environment where you can test your code. Just click on run every time you modify your code, and the results will be printed on the console below.

If you’re new to coding, things like installing Python, and setting up your code editor and environment might seem intimidating and frustrating at first. Fortunately, now you can skip these steps and focus on learning to code thanks to ChatGPT Canvas.

You can also import Python libraries using ChatGPT Canvas. In the example below, Canvas imports matplotlib and numpy to do a task common in data analysis, which is plotting visualizations. You can import many other libraries, but remember that there’s a limited number of libraries available.

Canvas has a button located in the bottom-right corner that displays many features useful for developers and software engineers. These features are:

• Review code

• Port to a language

• Fix bugs

• Add logs

• Add comments to your code

Besides Canvas, it’s worth mentioning other ways to learn Python with ChatGPT. My favorite is brainstorming unique projects with ChatGPT. Why projects? It’s the best way to put the concepts you learned into practice.

We can use ChatGPT to come up with cool and unique project ideas based on the programming knowledge we have and the things we like.

Here’s the prompt I use for this.

Act as a programming mentor. I’m learning Python and have already learned [topic 1, topic 2, …]

What Python projects can I solve? Consider that I like [thing 1, thing 2, …]

Here’s an example.

Once you select a project, you can ask for the steps to solve it (without making ChatGPT write the code for you).

In a previous article, I covered other ways to use ChatGPT to learn Python. While some of those suggestions can now be accomplished with just one click using Canvas, others remain relevant.

Subscribe now

Turn Gemini into your real-time AI tutor

Gemini is Google’s AI model. One cool feature of Gemini 2.0 is real-time screen sharing. This feature allows us to share our screens with Gemini, enabling it to analyze on-screen content and provide real-time assistance.

As programmers, we can allow Gemini 2.0 to view our screen and provide real-time guidance for solving any coding problem we encounter.

Anyone with a Google account can use this feature for free. To use it, go to this site, click on “stream realtime” on the left panel, and select “share screen.” After that, Gemini will see your screen and you can talk with it in real-time.

You can open your code editor, start writing code, and ask for guidance from Gemini whenever you need it. Here’s a demo where I asked Gemini for help to create a Python script that generates ten random numbers.

This is like having an AI tutor that can see whatever you’re working on 24/7. As a beginner, you can use Gemini to write a script, remember functions, learn terminal commands, set up your code environment, and more!

Improve the way you learn to code with NotebookLM

NotebookLM is an AI-powered research and note-taking tool developed by Google. You can use NotebookLM to:

• Convert long documents (PDFs, slides, links) into easy-to-understand summaries

• Build your own study guides effortlessly

• Create a podcast where an AI host walks you through complex material in minutes

This is game-changing for learning new subjects. That’s why I think this tool will change the way you learn coding.

How do I use NotebookLM to learn Python? I use it to better understand any Python tutorial or guide I’m following and even go beyond that by asking questions about the content in the video or guide. To do this, go to NotebookLM, and then provide the link to a YouTube tutorial or Python guide, book, etc. Once you upload your file, NotebookLM will analyze its content and then we can start a chat with it.

In the example below, I gave NotebookLM a link to my tutorial on how to send emails with Python. Once it analyzed the video’s transcript, I could ask it questions like the libraries used in the video or what I should keep into account before writing the script to send emails with Python.

If you want to go beyond the tutorial, just generate a podcast from the document uploaded. To generate the podcast, just go to the Audio Overview section and click on Generate. After some minutes, NotebookLM will generate an engaging audio discussion between two AI-generated hosts. The discussion is a summary of the material with some extra information that will help you deep diver into the topic and even learn new things.

Here’s the podcast NotebookLM generated from my YouTube video.

I find the podcast a good complement to my video tutorial. While my video mostly focuses on the Python code to send the emails, the podcast dives into concepts such as SSL, SMTP, secure passwords, and more!

You can find another nice AI-generated podcast about my Python book recommendation, in this article.

Subscribe now

GitHub Copilot: A general-purpose AI coding assistant

Here’s a bonus for those with some coding experience: GitHub Copilot.

GitHub's AI tool is a game-changer for real-time code writing. What's cool about it is that it offers an interaction style similar to ChatGPT, but it's focused on coding. Plus, it can keep building out a program you're working on in the editor.

One feature that really stands out to me is the chat function. You can literally quiz it about constructing functions or clear up any doubts about specific bits of your code. It's like having a coding guru on standby, ready to jump in with insights or solutions.

Something I like about Copilot is that it doesn't just stop after giving a response. It keeps the ball rolling, offering up possible questions I might want to ask next or things to consider. Below is how Copilot completed a Fibonacci function and the follow-up questions it generated.

It's like having an ongoing conversation with your code!

Plus, there's a feature that lets you move the suggested code into your editor. This creates a seamless flow, letting you bounce between the chat and the editor.

If you want to multitask within the editor, you can get the Copilot chat by pressing Ctrl + i.

Here are some of my favorite commands:

• Diving deeper into the code: /explain

• Getting unstuck or fixing code snags: /fix

• Conducting tests on the code: /tests

I have to say Copilot is one of my favorite tools. It's like having the best of ChatGPT, but baked right into your IDE, making coding effortless and more efficient. You can add the GitHub Copilot extension to Visual Studio Code, Visual Studio, JetBrains, and Neovim.

GitHub Copilot is my favorite general-purpose coding assistant, but other tools excel in specific scenarios. In this article, I reviewed all the AI coding assistants I tried.

Are you using any other AI tool to learn to code? Let me know in the comments!

Subscribe now

Give a gift subscription

Before we start today’s article, I have a recommendation for anyone into Java and system design: Javin’s Substack, Javarevisited.

Javin has a Substack called Javarevisited where he helps thousands of tech enthusiasts, developers, and software professionals to learn Java, Programming, and System Design and prepare for tech interviews. He has been writing blogs since 2010 and published Grokking the Java Interview book and courses on Udemy.

If you're into Java or System Design, check out his Substack!

The Mac terminal is quite a useful tool for programmers, yet it doesn’t have a friendly interface.

If you’re a programmer, chances are you spend a lot of time using the terminal. That’s why you should invest some minutes to upgrade it in order to become more productive when working with it.

Here’s how to upgrade your Mac terminal so it looks less like the first image and more like the second image below.

Subscribe now

Install Homebrew

In this guide, we’ll use the brew command a lot, so the first thing to do is install Homebrew. Open up a terminal and paste the command below.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Install iTerm2

Then we’re going to install iTerm 2. That’s a replacement for Terminal and has features you wouldn’t find on the default terminal. iTerm2 works on Macs with macOS 10.14 or newer. To install it run the command below in the terminal.

brew install --cask iterm2

Once you get the message “iterm2 was successfully installed!” a new iTerm icon will be added (press F4 to see it in launchpad)

Open iTerm 2. From now on, we’ll work with it. Now, let’s install some extra packages.

Install Git

Git will help us easily clone themes for our terminal a bit later.

brew install git

Install Oh-My-ZSh

Oh My Zsh is a framework that comes bundled with thousands of helpful functions, helpers, plugins, and themes that will help us transform our terminal.

sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Install Themes

There are many themes you can install to give a different look to your terminal. One cool theme I found on the internet is PowerLevel10K

First, install PowerLevel10K theme.

git clone https://github.com/romkatv/powerlevel10k.git $ZSH_CUSTOM/themes/powerlevel10k

Then set the theme in configuration. To do so, go to the home folder and open ~/.zshrc with the command below (you can use nano or vim for that).

nano ~/.zshrc
vim ~/.zshrc

Once ~/.zshrc is opened, locate the word “ZSH_THEME” and change its value to the name of the theme we installed (powerlevel10k/powerlevel10k)

ZSH_THEME=”powerlevel10k/powerlevel10k”

Then exit ~/.zshrc (esc > :wq or ^x > enter) and apply the changes running the command below.

source ~/.zshrc

Then you’ll see the window below. If you want to install Meslo Nert Font (which I recommend you), press Y.

The font will be downloaded and then we have to quit iTerm (use Cmd+Q).

Open iTerm again. You should see the image below (if not run “p10k configure”). These are a series of questions to verify everything is fine. If everything is as indicated in the questions, press Y.

Then you’ll see some preferences to configure the prompt style, heads, tails, spacing, etc. You can choose the options you like the most.

After the setup is done, iTerm2 should look like the image below.

Subscribe now

Extra Customizations

ITerm 2 can be further customized. We can increase the font size, install useful plugins, and more.

1. Change iTerm2 font size

1. Open iTerm2 preferences (cmd + ,)

2. Go to Profiles and click on Text

3. In the Font section, you can increase the font size and select other fonts

2. Install Plugins

Here are some useful plugins I recommend you to install.

a. Autosuggestions

First, install autosuggestions to get suggestions based on the previous commands you run on the terminal.

git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions

After installing a plugin, you go to the home folder, open ~/.zshrc, and modify the line below.

nano ~/.zshrc
vim ~/.zshrc

Add this plugin inside the parenthesis.

plugins=(git zsh-autosuggestions)

Exit ~/.zshrc and apply the changes by running the command below.

source ~/.zshrc

Now you should have auto suggestions enabled. Every time you type a command, you’ll see the suggestions in gray.

b. Syntax-highlighting

If you want to add syntax and highlighting to the commands we run (like in a code editor), you should install this plugin with the command below.

git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting

As we did before, after the installation, we need to add this new plugin to the plugin variable inside ~/.zshrc.

plugins=(git zsh-autosuggestions zsh-syntax-highlighting)

Exit ~/.zshrc and apply the changes by running the command below.

source ~/.zshrc

Now you should have auto syntax-highlighting enabled.

That’s it! You’ve just upgraded your Mac terminal.

3. Bonus: Themes for Vim

In case you’re using Vim as your text editor, you can also change the themes used for Vim.

To do so, open ~/.vimrc

vim ~/.vimrc

Now, to see a list of ready-to-use themes, type the following

:colorscheme [space] [Ctrl+d]

Now you should see all the themes available.

If you want to set the default vim theme, type i (to enable insert mode) and then type the following to set the default color scheme to the color you want and also enable the syntax highlight

colorscheme [colorscheme_name]
syntax on

Here’s how your ~/.vimrc should look like (in this case, I chose the desert theme).

Now press escape and then :wq to save and quit. Now vim will have the theme you chose!

Subscribe now

Give a gift subscription

There’s no perfect course in this world, but do you know what most of them don’t teach you?

The not-so-obvious skills and tools you’ll need at work!

Most courses are great for getting you started with a programming language and refreshing your knowledge of tech skills, but they lack a couple of modules dedicated to the skills you need to have at work to work efficiently with other programmers and non-technical coworkers.

Here are three things you probably didn’t learn in an online course (plus resources to learn them on your own).

Software Engineering Practices

As a data scientist, you’ll be writing code in programming languages such as SQL, Python, and R. Although most data science courses will help you get started with coding, they won’t teach you or focus on good practices.

Good practices such as writing clean code, modular code, and optimizing your code are very popular in software engineering. Believe it or not, these good practices will help you become a better data scientist.

Here’s a simple example of how to optimize your code:

# Imagine we have an array of random exam scores and we want to get the average score of those who failed the exam (score<70)
# Below are two way of solving this problem (one using loops and the other using vector operations)

import time
import numpy as np

random_scores = np.random.randint(1, 100, size=10000001)

# SOL1: solving problem using a for loop
start = time.time()

count_failed = 0
sum_failed = 0
for score in random_scores:
    if score < 70:
        sum_failed += score
        count_failed += 1

print(sum_failed/count_failed)
print(f'Duration: {time.time() - start} seconds')


# SOL2: solving problem using vector operations
start = time.time()

mean_failed = (random_scores[random_scores < 70]).mean()
print(mean_failed)

print(f'Duration: {time.time() - start} seconds')

That’s only a basic example with a few lines of code! The longer the script, the more important it is to follow these good practices.

What can you do to learn good software engineering practices?

You can search for guides or tutorials to learn good software engineering practices for data science, and read a book that covers all these points and gives a big picture on the topic.

Here are 5 Python books that every beginner should read to go beyond the basics. I’d recommend checking the book “Clean Code in Python” in particular because it covers most of the points mentioned before.

Subscribe now

Read more

Online courses helped me learn much of the Python stuff I know. I like going at my own pace and having the flexibility to study and work, so I finished most of them without any problem.

However, there were also times when I joined a course, took a couple of lectures, and never returned. Sometimes the course was too advanced for me, the instructor was too boring, or the concepts were poorly explained. After taking many online courses, I’d like to share with you my strategy for self-studying using online courses.

This guide covers things like how to pick the right Python course, how to make the most of online courses, and more.

Subscribe now

First Things First — Answer These 3 Vital Questions

One of the key points to choosing the right course(s) for you is to know what exactly you’re looking for, why you want to learn that, and how much time you have to learn it.

Ask yourself these questions:

1. What (exactly) do I wanna learn?
2. Why do I wanna learn it?
3. How much time available do I have to learn that?

Questions #1 and #3 help you narrow down the courses, while question #2 supports your decision (keep this in mind whenever you feel you make little progress).

Say you know the basics of Python and want to learn more. There are many fields where Python can be used (data science, web development, AI programming, machine learning, etc), so you need to know what exactly you want to learn. Also, if you have little time available, you should stay away from those 50+ hour courses that you might never finish.

Note: If you don’t know how to answer question 1, jump to the next section (there’s a course that might help you make up your mind).

Here is how I answered these questions when I was an absolute beginner.

1. Data Science with Python
2. I like Python and working with data.
3. All day!

Here is how I answered these questions after learning Python and data science for some years:

1. Machine Learning with TensorFlow
2. I love ML and TensorFlow is popular among developers (it has Google’s support)
3. 1 hour on weekdays

I gave these 2 examples, so the advice in the next section can fit most situations.

Great! Now is your time to answer these questions. Once you know the answers, choosing the right course will be way easier.

How to choose the right course for you

There are hundreds of courses out there, so choosing the right course might be overwhelming.

You might ask for advice from your friends and colleagues, but, the truth is, a course defined as “perfect” by your colleague might not be perfect for you. Why? Well, he/she might have a different background, experience, and goals!

Here are some rules of thumb that you should follow when picking a course as a beginner or advanced student:

• The course should have tasks, exercises, and projects. This helps put into practice what you learn

• Keep an eye on the instructor(s). Smart people aren’t always the best teachers

• If you want to join a course that involves coding, make sure the instructor doesn’t write code right away, but, whenever possible, makes some time to explain concepts with slides, an online whiteboard, etc.

If you’re an absolute beginner

If you’ve just started your journey, take courses that focus on core concepts, so you can build a strong foundation. Also, if you still don’t know what exactly you want to learn, take courses that cover different topics.

Say you want to start your data science journey with Python. Here’s what I get after searching “Data Science with Python” on Udemy,

After checking them out and considering the previous advice, course #3 looks like a good candidate. Why? It focuses on Python for data science and has plenty of exercises and projects.

Course #2 is good too, but isn’t only about Python but also math and statistics (this goes against my answer to question 1), On the other hand, course #4 is a bit advanced and not only focuses on Python but on R, while course #1 is kind of outdated.

If you have previous knowledge

If you already know some stuff and want to specialize in a topic, you should stay away from those “ultimate courses” that cover a lot of stuff but don’t delve into the topic you’re interested in.

Here’s what I get after searching “Machine Learning with TensorFlow” on Udemy.

This time is harder to choose a good course.

Course #1 is too broad and basic. Course #3 lacks the projects and exercises, while course #4 is more about math for ML.

If I had to pick one of those courses I’d choose course #5 (it has a couple of good projects) and course #2. But they don’t convince me completely. Course #2 has some exercises, but not so many projects and I’m afraid it can be too broad (63 hours of content!).

The good news is that there are also other platforms we can check out. I like Udacity’s courses because they’re taught not only by 1 but many industry experts who are also good at teaching. Coursera is another great option.

How to make the most of a course

Now that you selected the right course, it’s time to get your hands dirty.

Subscribe now

Read more

Before paying for a course, it’s a good idea to check what has been published out there — there’s a decent amount of high-quality free content. This is what I did when I started learning Python for data science. I checked the curriculum of paid data science courses and then searched all the stuff related to Python.

I selected the best 4 free courses I took to learn Python for data science. The 4 courses cover some of the stuff in “The 4 Stages of Learning Python for AI & ML.” Although you won’t get a nice certificate after completing any of these courses, the knowledge you will acquire is priceless.

Subscribe now

#1 Learn Python — Full Course for Beginners

This 4-hour Python beginners course covers all the basic Python stuff you need before learning libraries used in data science. This is a crash course in Python — when I took this course I had zero knowledge about programming. The course will show you how to set up Python, teach you how to print your first “Hello World”, and explain all the core concepts in Python.

Please, do not skip the fundamentals of Python and jump to the essentials of Python for Data Science. Remember that Python is a programming language with applications not only in data science but in many fields. Some Python for Data Science courses might skip core Python concepts that will be necessary in the future.

Some of the topics covered in this course that are frequently used in data science are:

• Variables & Data Types

• Lists, Dictionaries, Tuples, Nested Lists

• Functions, Return Statement, If Statements

• For Loops, While Loop, Nested Loops

• Try / Except, Reading and Writing to Files

• Python Interpreter, Modules, Pip Classes

• Objects, Object Functions, Inheritance

Plus, there are practical exercises where you would put all the concepts learned into practice. You will build a basic calculator, guessing name, translator, and a multiple-choice quiz. On top of that, the instructor has good knowledge of Python, clear explanations, and engaging delivery.

#2 Python for Data Analysis Course

This one is more like a playlist than a course; however, you will find more useful lectures in this playlist than in some paid courses.

The first 8 videos in the playlist make a 10-hour data analysis course. It starts with the most popular data collection technique that data scientists use to build datasets — web scraping. Then you will learn data analysis libraries such as Pandas, Numpy, Matplotlib, and Scikit-learn. On top of that, the playlist includes cool projects that will help you get hands-on experience. All of this is carried out on Jupyter Notebooks, which is data scientists’ computational notebook of choice.

Some of the datasets used in this course to teach the libraries mentioned before are:

• Pokemon Dataset

• FIFA 20 Dataset

• Amazon Product Review Dataset

• Sales Data

So if you want to have fun analyzing Pokemon and FIFA datasets or build a machine-learning model using product review, consider watching this playlist.

I watched the first 8 videos of the playlist and that helped me build a strong foundation on Python for data analysis.

Subscribe now

#3 Natural Language Processing Course in Python

This is a short yet useful 2-hour NLP course for anyone interested in the field of Natural Language Processing. NLP is a branch of artificial intelligence that allows machines to understand human language. Data scientists use NLP techniques to interpret text data for analysis.

This introduction to NLP covers the following topics:

• Pre-processing techniques (tokenization, text normalization, and data cleaning)

• Machine learning techniques (topic modeling, word embeddings, and text generation)

• Python libraries for NLP (NLTK, TextBlob, spaCy, and gensim)

I like this short course in particular because it’s highly engaging and includes cool exercises like spam email classification and sentiment analysis of tweets. Of course, to do all of this you will need to have at least a basic understanding of data analysis libraries such as Pandas and Scikit-learn.

#4 Machine Learning & Artificial Intelligence with Tensorflow 2.0 Course

This is a 7-hour Tensorflow course designed for Python programmers who wish to learn machine learning (ML) and artificial intelligence (AI) with TensorFlow. TensorFlow is one of the best libraries available for working with Machine Learning in Python. It makes machine learning model building easy for beginners.

In this course, you will find 8 modules that cover fundamental topics in ML & AI. To name a few:

• Core Learning Algorithms

• Neural Networks with TensorFlow

• Deep Computer Vision — Convolutional Neural Networks

• Natural Language Processing with RNNs

• Reinforcement Learning with Q-Learning

I like this course because the instructor has good knowledge of machine learning and artificial intelligence and has a clear explanation. Also, the course is engaging and contains valuable information and coding examples. All of this makes learning TensowFlow easier.

That’s it! Hope these 4 courses help you enhance your knowledge of Python for data science and AI.

Share

Give a gift subscription

We’ll see how this Python library works in detail in this guide.

Subscribe now

Install the libraries

To easily create interactive visualizations, we need to install Cufflinks. This is a library that connects Pandas with Plotly, so we can create visualizations directly from Pandas (in the past you had to learn workarounds to make them work together, but now it’s simpler)

First, make sure you install Pandas and Plotly by running the following commands on the terminal:

pip install pandas
pip install plotly

Note that you can also use conda to install Plotly: conda install -c plotly.

Once you have Plotly installed, run the following commands to install Cufflinks:

pip install cufflinks

Import the libraries

To start working, import the following libraries:

import pandas as pd
import cufflinks as cf
from IPython.display import display,HTML

cf.set_config_file(sharing='public',theme='ggplot',offline=True) 

In this case, I’m using the ‘ggplot’ theme, but feel free to choose any theme you want. Run the command cf.getThemes() to get all the themes available.

To make interactive visualizations with Pandas in the following sections, we only need to use the syntax dataframe.iplot().

The data

For this guide, we’ll use a population dataframe. First, download the CSV file available on Google Drive or Github, move the file where your Python script is located, and then read it in a Pandas dataframe as shown below.

df_population = pd.read_csv('population_total.csv')

The dataframe contains the population throughout the years for most countries around the world and looks like this:

This dataframe is almost ready for plotting, we just have to drop null values, reshape it, and then select a couple of countries to test our interactive plots. The code shown below does all of this.

# dropping null values
df_population = df_population.dropna()

# reshaping the dataframe
df_population = df_population.pivot(index='year', columns='country',
                                    values='population')

# selecting 5 countries
df_population = df_population[['United States', 'India', 'China', 
                               'Indonesia', 'Brazil']]

Now the dataframe looks like the picture below and is ready for plotting.

If you’d like to know how a data scientist collects real-world data like this “population_total.csv”, check this or this guide I made (you won’t always have a Kaggle dataset available for a data science project)

Now let’s start making interactive visualization!

Subscribe now

Lineplot

Let’s make a lineplot to compare how much the population has grown from 1955 to 2020 for the 5 countries selected.

As mentioned before, we will use the syntax df_population.iplot(kind='name_of_plot') to make plots as shown below.

df_population.iplot(kind='line',xTitle='Years', yTitle='Population',
                    title='Population (1955-2020)')

At first glance, it’s easy to see that the population in India has grown faster than the other countries.

Barplot

We can make a single barplot on barplots grouped by categories. Let’s have a look.

Single Barplot

Let’s create a barplot that shows the population of each country by the year 2020. To do so, first, we select the year 2020 from the index and then transpose rows with columns to get the year in the column. We’ll name this new dataframe df_population_2020 (we’ll use this dataframe again when plotting piecharts)

df_population_2020 = df_population[df_population.index.isin([2020])]
df_population_2020 = df_population_2020.T

Now we can plot this new dataframe with .iplot(). In this case, I’m going to set the bar color to light-green using the color argument.

df_population_2020.iplot(kind='bar', color='lightgreen',
                         xTitle='Years', yTitle='Population',
                         title='Population in 2020')

Barplot grouped by “n” variables

Now let’s see the evolution of the population at the beginning of each decade.

# filter years out
df_population_sample = df_population[df_population.index.isin([1980, 1990, 2000, 2010, 2020])]

# plotting
df_population_sample.iplot(kind='bar', xTitle='Years',
                           yTitle='Population')

Naturally, all of them increased their population throughout the years, but some did it at a faster rate.

Boxplot

Boxplots are useful when we want to see the distribution of the data. The boxplot will reveal the minimum value, first quartile (Q1), median, third quartile (Q3), and maximum value. The easiest way to see those values is by creating an interactive visualization.

Let’s see the population distribution of the USA.

df_population['United States'].iplot(kind='box', color='green', 
                                     yTitle='Population')

Let’s now get the same distribution but for all the selected countries.

df_population.iplot(kind='box', xTitle='Countries',
                    yTitle='Population')

As we can see, we can also filter out any country by clicking on the legends on the right.

Histogram

A histogram represents the distribution of numerical data. Let’s see the population distribution of the USA and Indonesia.

df_population[['United States', 'Indonesia']].iplot(kind='hist',
                                                xTitle='Population')

Piechart

Let’s compare the population by the year 2020 again but now with a piechart. To do so, we’ll use the df_population_2020 dataframe created in the “Single Barplot” section.

However, to make a piechart we need the “country” as a column and not as an index, so we use .reset_index() to get the column back. Then we transform the 2020 into a string.

# transforming data
df_population_2020 = df_population_2020.reset_index()
df_population_2020 =df_population_2020.rename(columns={2020:'2020'})

# plotting
df_population_2020.iplot(kind='pie', labels='country',
                         values='2020',
                         title='Population in 2020 (%)')

Scatterplot

Although population data is not suitable for a scatterplot (the data follows a common pattern), I would make this plot for the purposes of this guide.

Making a scatterplot is similar to a line plot, but we have to add the mode argument.

df_population.iplot(kind='scatter', mode='markers')

That’s it! Now you’re ready to make your own beautiful interactive visualization with Pandas. If you want to learn other visualization libraries in Python such as Matplotlib and Seaborn, check out this guide.

Subscribe now

Give a gift subscription

In this article, I will show you 3 books that helped me automate tasks such as sending emails, collecting data, creating reports, and more!

1. Automate the Boring Stuff with Python

This is a book that I think many have heard of. Automate the Boring Stuff with Python teaches you how to write programs that can accomplish in minutes what would take hours to do manually. Unlike some books listed in this article, this book doesn’t require any prior programming experience.

The book starts with some Python basics and then explains how to automate tedious tasks such as searching for text in a file, creating files, renaming files, searching the web, downloading content from the web (also known as web scraping), and sending email responses and text notifications.

The fact that Automate the Boring Stuff has a section that covers web scraping, makes this book worth it for data scientists. This skill will allow you to build your own dataset for your next data science project.

Also, I consider automation projects essential for those who have to deal with lots of repetitive tasks at work. If you’re quite familiar with the basics of Python, I would recommend you skip Part I and pick a project that you find interesting in Part II. By solving these real-world problems you will increase your knowledge of Python while automating your life.

Subscribe now

2. Python Automation Cookbook: 75 Python automation ideas

This book is for Python beginners who want to expand their knowledge by automating everyday tasks. That said, you should at least know the basics of Python before reading this book.

The book contains 75 Python automation ideas with applications in the areas of marketing, sales, data science, and more. You will learn how to automate the web, generate automatic spreadsheet reports with graphs, communicate with automatically generated emails, get notifications via text message, and schedule tasks on your computer.

In the latest edition of this book, two new chapters were added: Cleaning & Processing Data and Machine Learning for Automation. Both are great chapters for those who want to automate data science tasks.

In Cleaning & Processing Data, you will learn how to batch process to speed up working with large data. I liked this project idea because cleaning and processing data usually takes a lot of a data scientist’s time.

On the other hand, the Machine Learning for Automation chapter shows how to use Google’s Machine Learning APIs to extract text from images. At the end of the chapter, you will create and train a model that detects which department an email should be assigned using text as input.

Subscribe now

3. Python for Excel: A Modern Environment for Automation and Data Analysis

Microsoft Excel is probably the most popular tool used for data analysis; however, its automation language, VBA, stopped evolving a long time ago. This is why some Excel users adopted Python to automate their everyday tasks.

In this book, the creator of xlwings (a package to automate Excel with Python) shows how to integrate both Python and Excel efficiently. The book starts with some Python core concepts and then teaches you all the stuff you need to learn for automation and data analysis. You will learn how to use Pandas to clean and analyze data, learn xlwings to build interactive Excel tools (using Python under the hood), and automate tedious tasks like consolidating Excel workbooks and producing reports.

By the end of the book, you will replace VBA, Power Query, and Power Pivot with Python.

Give a gift subscription

Share

Everyone has been in a situation where they don’t know what to learn next or what to specialize in.

You might like Python but that doesn’t mean you’ll learn every single library out there. You also don’t want to learn libraries that you may never use, so what should you do?

Well, there are a few Python libraries that will come in handy regardless of the field you’re in. Whether you’re into web development or data analysis, you’d need these libraries to some degree.

Here are some Python modules, libraries, and packages that you should learn whenever you don’t know what to learn next.

Subscribe now

OS/Pathlib

Working with paths is something that anyone does sooner or later. Whether you’d like to export a dataset in a specific directory or read static files for your web application, you’d need to learn the OS or Pathlib modules.

The OS and Pathlib are Python modules that allow us to do file system operations such as creating directories, listing directory content, working with paths, and more.

But which one should you learn? OS or Pathlib?

The OS and Pathlib modules have some similarities. With the OS module, you can manage paths, create folders, and more, but this module represents paths as string values. This limitation may sometimes lead you to search for workarounds that could be avoided by using the Pathlib module.

Obtaining the parts of a path using Pathlib is as simple as calling an attribute. Let’s have a look.

from pathlib import Path

# obtaining the current working directory
my_path = Path.cwd()

>>> my_path
PosixPath('/Users/frankandrade')

>>> my_path.parent
PosixPath('/Users')

>>> my_path.name
'frankandrade'

Unlike OS, Pathlib works with objects that have properties and internal methods that make working with paths simpler.

Requests

Whenever you need to interact with a server, you’ll have to use the Requests library. This is a Python library that you can use to send HTTP requests. HTTP is a request-response protocol between a client and a server that is the foundation of data exchange on the web.

Why do you need that? Nowadays it’s very common to connect to third-party services using APIs, so we can get access to data such as tweets and the weather or add functionality to your web app like adding payment, translation, and email. If those APIs didn’t exist, you’d have to create those services from scratch!

Fortunately, with APIs, we only need some minutes to get access to such services. That said, if you want to work with an API, you’ll need to send HTTP requests and receive responses. As you might guess, the Requests library is the perfect tool for this.

Here’s an example of how I used the requests library to perform speech-to-text using this API.

import requests

...

headers = {'authorization': api_key,
           'content-type': 'application/json'}

response = requests.post(upload_endpoint,
                         headers=headers,
                         data=read_file(filename))

audio_url = response.json()['upload_url']

But that’s not all! If you add a web scraping library like Beautiful Soup, you can even extract data from websites using requests.

Subscribe now

Selenium

Every job has tasks that can be automated, and unless you want to waste time doing it manually, you should learn Selenium to automate most websites out there.

Selenium is an automation testing tool that you can use to automate websites with Python. With Selenium, you can perform web actions such as clicking on buttons, navigating through pages, selecting elements within a dropdown, and even extracting data!

You can also customize the bot further by adding waits and rotating proxies, so the bot behaves like a “regular person.”

Here’s some Python code to open a browser, go to a website, and then quit it using Selenium.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service

web = 'https://www.example.com'
path = '/.../chromedriver'

# Creating the driver
driver_service = Service(executable_path=path)
driver = webdriver.Chrome(service=driver_service)
driver.get(web)

# Quit driver
driver.quit()

And here’s a demo.

The best thing about learning Selenium is that once you master it, you can automate not only one but most websites out there.

Here’s a video tutorial to get you started with Selenium in Python.

PyTest

Anyone learning Python should be able to prepare their code for an industry setting.

How to do that? Testing your code!

PyTest is a framework that allows us to write test codes in Python. With Pytest, we can write small, readable tests that can scale to support complex functional testing for apps and libraries.

But why do we need to test our code? Well, we can make little mistakes when writing code that aren’t easy to detect simply because they don’t throw an error or crash a program. Your code may run fine once, but it could give you some headaches in the long run. This is why we need to test our code to avoid unexpected results and ensure we can rely on our output.

There are different ways to test code, but Pytest makes testing simple and also allows us to see which tests failed and which succeeded.

Here’s a test I made:

If you know a library that can be added to this list, let me know in the comments 👇

Give a gift subscription

Share

Hi!

Before we start with today’s article, I’d like to tell you about something that might be of interest to those interested in Python and data science. DataCamp has an incredible certification program that’s perfect for you: The Python Data Associate Certification.

This certification is designed to help you:

• Prove your Python skills are ready for real-world application

• Build credibility in the competitive data industry

• Land your dream job in the data field

If you’re serious about standing out in the job market, this certification is the perfect step forward. To get the certification, you need to register, and then you’ll have 30 days to conquer a timed exam and practical exam. If you need to give it a second try, Datacamp can help you fill the gaps in your knowledge before you try again.

What I like about Datacamp is that they offer interactive lessons, practical exercises, and real-world projects. Projects are very useful to put the concepts we learn into practice, and today we’ll see 5 data science projects you can do with Python.

I love solving projects to learn data science and Python.

Why? A project will help you put into practice all the knowledge you’ve acquired from math, statistics, and programming. It’s like putting the pieces of a puzzle together to solve a real-world problem.

In this article, I listed some data science projects you can do with Python. The projects are listed by difficulty starting with the beginner projects. Knowledge of libraries such as Pandas, Numpy, and Scikit-learn is necessary to solve most of these projects. I’ll leave links to tutorials to help you solve these projects.

Subscribe now

1. Sentiment Analysis

The first project of this list is to build a machine-learning model that predicts the sentiment of a movie review. Sentiment analysis is an NLP technique used to determine whether data is positive, negative, or neutral. It’s really helpful for businesses because it helps them understand the overall opinions of their customers.

For this project, you will use an IMDB dataset that contains 50k movie reviews. with 2 columns (review and sentiment). The goal is to build the best machine learning model that predicts the sentiment given a movie review.

To make this project beginner friendly you only have to predict whether a movie review is positive or negative. This is known as binary text classification because there are only two possible outcomes.

One of the things that make this first project special is that you will explore the scikit-learn library while building a basic machine-learning model from scratch.

• Libraries: Pandas, Scikit-learn

• Tutorial: Sentiment Analysis in Python (Text Classification)

2. Fake News Detection

The most beginner-friendly detection project is probably Fake News Detection. Fake news is spread everywhere on the internet. This generates confusion and panic among the population. This is why is important to identify the authenticity of the information. Fortunately, we can use Python to tackle this data science project.

• Libraries: Scikit learn (TfidfVectorizer and PassiveAggressiveClassifier), Pandas and Numpy

• Tutorial: Detecting Fake News

The goal of this project is to separate real news from fake news. To do so, we will use sklearn’s tools such as TfidfVectorizer and PassiveAggressiveClassifier.

Subscribe now

3. Credit card fraud detection

Credit card fraud costs both consumers and companies billions of dollars while fraudsters keep trying to find new ways to commit these illegal actions. This is why fraud detection systems have become essential for banks to minimize losses.

In this project, you should analyze customer’s spending behavior from a dataset that contains transaction history. Variables like the location will help you identify fraudulent transactions.

• Libraries: Pandas, Numpy, Matplolib, Scikit-learn, Machine Learning Algorithms (XGBoost, Random forest, KNN, Logistic regression, SVM, and Decision tree )

• Source Code: Credit Card Fraud Detection With Machine Learning in Python

4. Chatbots

A chatbot is just a program that simulates human conversation through voice commands or text chats. Advanced chatbots are built using artificial intelligence and used in most messaging applications you have on your phone.

Although creating voice assistants like Siri and Alexa are too complex, we can still create a basic chatbot using Python and deep learning. In this project, you will have to train a chatbot with a dataset using data science techniques. As these chatbots process more interactions, their intelligence and accuracy will increase.

Building a simple chatbot will expose you to a variety of useful skills for data science and programming

• Libraries: Keras, NLTK, Numpy

• Source Code: Python Chatbot Project (Option 1, Option 2)

5. Customer Churn Prediction

Customer churn is the rate at which customers stop doing business with a company. This represents the percentage of subscribers who discontinue their subscriptions within a given time period. This is a good project to test your data science skills. I even had to solve it in hackathons!

The main goal of this project is to classify if a customer is going to churn or not. To do so, you will use a dataset that has financial data about a bank’s customers. Information such as credit score, tenure, number of products, and estimated salary will be used to build this prediction model.

This project and the credit card fraud detection project are the most complete data science projects listed in this article. It includes exploratory data analysis, feature engineering, data preparation, model fitting, and model selection.

• Libraries: Pandas, Matplolib, Scikit-learn, Machine Learning Algorithms (XGBoost, Random forest, KNN, Logistic regression, SVM, and Decision tree)

• Source Code: Bank Customer Churn Prediction

Good luck solving these projects!

Give a gift subscription

As a beginner, I used to come across some roadblocks that slowed down my workflow.

Sometimes I forgot the format needed to create a cron job, a metacharacter needed to build the right regular expression, etc. The tools listed in this article helped me easily deal with these issues.

Regex101: Test your regular expressions

Have you been in a situation where you need to match a specific search pattern but can’t build the regular expression (regex) that does the job?

You know regex but you just don’t remember that metacharacter that does the trick or something goes wrong when you build a long regular expression.

I used to have this issue … until I found Regex101.

Regex101 is a tool that we can use to test our regular expressions. You only need to type a regex and the text that matches will be highlighted in blue.

Say we want to identify valid emails with a regular expression.

As you can see in the gif above, only the first email is highlighted in blue because that’s the only valid email.

Crontab Guru

Every time we need to schedule a task on macOS and Linux we create a cron job. This allows us to automate a script to complete a task any time we wish.

The problem is that if we don’t create the right cron expressions, the job could never start or fail.

That’s why we need to use Crontab Guru.

Crontab Guru is a quick and simple editor for cron schedule expressions.

You only need to type the minute, hour, day, and month to see what that expression means in plain English and know when’s the next time the script is going to be executed.

Subscribe now

Strftime

Datetime is a very useful module that allows us to work with dates and times in Python. To convert a date/time object to string we need to use the strftime() method as shown below.

from datetime import datetime
now = datetime.now()
now.strftime(<format>)

Simple, right? The thing is, I sometimes forget the format used to convert the date to their string representation.

Fortunately, there’s a website called strftime where you can find a Python strftime cheatsheet.

Now I can easily get today’s date in the format I want.

>>> now.strftime("%B %d, %Y")
'October 28, 2022'

HTML Editor

HTML Editor is a tool that allows us to create HTML documents as if we were typing text in a Word document.

Say we want to create an HTML document that has a title, paragraph, and list that looks like the one below.

Well, we can easily do that with HTML Editor.

On this website, we only need to type text in the visual editor on the left and it automatically generates its HTML code on the right.

Note that you can use the toolbar to add different elements to the document such as lists, lines, images, links, etc.

Although there are other alternatives, HTML editor is the best online software with visual preview, inline editor, and markup clean-up feature.

Read more

Probably your current Python skills are enough for your personal projects or to keep your boss happy, but there’s a lot more to learn! Now you might not pay attention to scalability, readability, and organization when writing code, but if you want to move from beginner to advanced Python programmer, these are some of the topics you should learn next.

Here are 4 books that cover intermediate and advanced programming concepts that you should learn to go beyond the basic Python stuff.

Subscribe now

Object-Oriented Python

If you have a basic knowledge of Python, you probably write code following a sequence of steps line by line to get things done. As a result, you end up with multiple sections that don’t have a strong logical connection. This is known as procedural programming, which isn’t bad for general-purpose programming, but this creates code that is neither reusable nor easy to scale.

If you want to take your Python skills to the next level you should learn object-oriented programming (OOP). With OOP, we can break a program into reusable blueprint-like components. In OOP we use classes, which are the foundation of Python objects — if we master classes, we can create any type of Python objects, therefore any type of program.

A good book for learning OOP in Python is Object-Oriented Python by Irv Kalb. This book covers cover the basics of OOP and put theory into practice using the pygame package. You’ll learn key concepts of OOP such as encapsulation, polymorphism, and inheritance. In the end, you’ll build a complete video game with animations and sounds.

Clean Code in Python: Develop maintainable and efficient code

As a beginner, you probably write code to get things done fast at the expense of organization, readability, and testability, which might lead you to write spaghetti code.

How to overcome this? Writing clean code.

If you learn Python and want to develop maintainable and efficient code, you should read Clean Code in Python by Mariano Anaya. The book covers topics such as Pythonic code, the SOLID principles, decorators, generators, iterators, asynchronous programming, unit testing, refactoring, and more! To sum it up, many concepts you need to become an advanced programmer.

By the end of the book, you’ll be able to apply industry-approved coding practices to design clean, sustainable, and readable real-world Python code.

Note: Another good option is “Beyond the Basic Stuff with Python” by Al Sweigart. In that book, you’ll also learn how to use tools like the command line, code formatters, type checkers, linters, and version control.

Subscribe now

Serious Python: Black-Belt Advice on Deployment, Scalability, Testing, and More

Serious Python is a book written for developers and experienced programmers that brings together over 15 years of Python experience to teach you how to avoid common mistakes, write code more efficiently, and build better programs in less time.

If you ever wanted to get advice from experts and sharpen your Python skills, this is the perfect book for you. This book is a collection of practical tips and real-world advice for tackling common Python problems.

Here are some of the things covered in this book:

• What you should consider before you take a project

• Python modules, libraries, and frameworks (and how they work under the hood)

• How to document your project and manage an API as your project grows

• How to handle time zones in your programs

• Packaging and distribution standards

• Methods and decorators

• Functional programming

• Performance and optimization

• Scaling and architecture

• How to manage relational databases.

The best part of the book is that at the end of each chapter, there’s an interview with expert Python developers who share real-world advice from years of experience.

Fluent Python: Clear, Concise, and Effective Programming

Everyone likes Python because of its simplicity. That’s good when you start learning the language, but once you master the basics, you should unlock the full potential of Python. To do so, you must learn how to write effective, idiomatic Python code by leveraging its best features

In the book Fluent Python, you’ll learn how to make your code shorter, faster, and more readable at the same time.

Here are some of the topics that the book covers:

• Python data model

• Data structures: An array of sequences, dictionaries and sets, and text vs bytes.

• Functions as objects

• Object-oriented idioms: References, mutability, interfaces, operator overloading, and multiple inheritance.

• Control flow: Context managers, generators, coroutines, and concurrency with futures and asyncio.

• Metaprogramming: Dynamic attributes and properties, attribute descriptors, and class metaprogramming.

Any Python book recommendation? Share it in the comments!

Give a gift subscription

Let’s face it, we sometimes don’t pay attention to things like writing efficient code, code structure, and maintainability.

But we should!

One day you’ll be part of projects that involve working with other people. This is why we need to write robust code and follow some good practices that software engineers have.

Here are some software engineering best practices that you should know.

Subscribe now

Write Clean Code

Writing clean code means writing readable, simple, and concise code. Clean code is the foundation of a script that is easy to maintain.

Believe me, you’ll make the lives of your teammates easier by writing code that is easy to understand. Simple is better than complex, so don’t write complex code that even you might struggle to understand.

Here’s an example of writing clean code:

# Imagine we want to write a program that categorizes a task based on its execution time
# Below are two ways of writing this code (one is more clean than the other)

# bad
t = end - start_time  # get execution time
c = category(t)  # get category of task
print(f'Task Duration: {t} seconds, Category: {c}')

# good
execution_time = end_time - start_time  # get execution time
category = categorize_task(t)  # get category of task
print(f'Task Duration: {execution_time} seconds, Category: {category}')

Below are the good practices followed in the previous example:

• Use meaningful variable names: Make your variable names explanatory and descriptive. A variable named end isn't as explanatory as end_time. A boolean variable single isn’t as descriptive as is_single.

• Don’t use abbreviations in a variable name that no one will understand (e.g.,t and c).

• Don’t write too long variable names that no one will remember.

• Use indentation and whitespaces properly: There are many conventions here like using four spaces for indentation or separating sections with a blank line. They might be hard to remember; fortunately, IDEs like Pycharm will suggest you follow such conventions and show you how to do it.

• Follow PEP8 conventions as much as possible when naming objects (e.g., how to use case convention, when to use an underscore, etc)

Write Modular Code

Modular code means writing code that can be separated into functions and modules.

A program that can be broken into modules helps when debugging. As a program grows in size, it’s a good practice to split your code into modules. This lets you easily pinpoint the source of errors.

Also, modular code will help you avoid repetition and write efficient and reusable code.

Here‘s some advice to start writing modular code:

• Don’t repeat yourself: If you’re using the same function/method to accomplish a single task, consider creating a function or a for loop instead.

# Imagine we have a list named "numbers" and we want to sum a random number to each element of the list and store it in a new list
# Below are two ways of writing this code (one avoids repetition)

numbers = [10, 20, 30, 40, 50]

# bad
numbers1 = []
for number in numbers:
    numbers1.append(number+1)

numbers5 = []
for number in numbers:
    numbers5.append(number+5)


# better
def sum_number(my_list, n):
    return [number + n for number in my_list]

sum_1 = sum_number(numbers, 1)
sum_5 = sum_number(numbers, 5)

• Minimize the number of functions, classes, and modules

• Single responsibility principle: A class/function should have only one responsibility. If they do more than one thing, consider refactoring them to two or more classes/functions.

• Use modules

Optimize Your Code

Writing code that works is good, but you know what’s better? Writing efficient code that runs fast and consumes little memory and storage. This is why you should optimize your code (even if it already does the job).

Writing efficient code isn’t easy. This is a skill that you’ll learn over time. That said, here’s some advice that will help you start writing more efficient code today.

• Vectorize your operations: Use vector operations (Numpy) over loops whenever possible.

• Inspect the running time of every operation: This will help you find bottlenecks in your script.

• Know your data structures and which methods are faster

Let’s see an example of how vectorizing your operations can help optimize the performance of your script:

# Imagine we have an array of random exam scores and we want to get the average score of those who failed the exam (score<70)
# Below are two way of solving this problem (one using loops and the other using vector operations)

import time
import numpy as np

random_scores = np.random.randint(1, 100, size=10000001)

# SOL1: solving problem using a for loop
start = time.time()

count_failed = 0
sum_failed = 0
for score in random_scores:
    if score < 70:
        sum_failed += score
        count_failed += 1

print(sum_failed/count_failed)
print(f'Duration: {time.time() - start} seconds')


# SOL2: solving problem using vector operations
start = time.time()

mean_failed = (random_scores[random_scores < 70]).mean()
print(mean_failed)

print(f'Duration: {time.time() - start} seconds')

If you run the snippet above, you’ll see that both solutions get the same result; however, it takes around 2.57 seconds for solution 1 to get the job done, while solution 2 only needs 0.06 seconds.

Although this little tweak barely saved us 2 seconds, at a larger scale, it can make a big difference in performance.

Subscribe now

Read more

There have never been so many free resources online to learn technical subjects.

Do you want to learn Python? There are thousands of tutorials on Youtube
Do you want to practice SQL? You can google SQL exercises with solutions 
Do you want to learn statistics? Khan Academy has many lectures you need for data science

But when it comes to learning all of this on our own, things get a bit complicated. You might wonder where to start, how to study less and learn more, how to retain more information, how to reach your peak performance when studying, and how to make consistent progress.

In this article, I’ll show you the 5 steps I follow to learn Python and other technical stuff.

Subscribe now

Read more

Hi!

Before we start with today’s article, I have great news for all of you!

From November 4th to 10th, DataCamp is offering unlimited FREE access to their entire learning platform! It’s the perfect chance to level up your Python, data and AI skills.

This includes:

• Access to 510+ courses, career tracks, and certifications

• Learn Python, ChatGPT, Power BI, SQL, and more

• Land your dream job with data and AI skills

Don’t miss out on this exclusive opportunity. Click here to get free access to Datacamp. In case you don’t know where to start, I recommend you to try this Python Data Fundamentals course.

When we start learning Python, we usually write code that gets the job done without paying attention to the readability of our code and how concise and efficient it is.

That’s fine, but there are ways to make our Python code shorter without neglecting readability. This is what one-liners are all about (if you use them properly).

Here are 8 one-liners that anyone learning Python should know.

1. If — Else Statement

The if-else statement is one of the first statements we learn in Python. It’s used to execute the true and false parts of a given condition.

We use this statement quite often, but do you know it can be simplified in one line of code? In this case, both the if and else statements would be on the same line.

age = 18

valid = "You're an adult"
invalid = "You're NOT an adult"

print(valid) if age >= 18 else print(invalid)

Subscribe now

2. Create a list based on existing lists

Lists are a common way to store data, but do you know you can create a new list based on existing lists with just one line of code?

That’s true! It’s called list comprehension and it offers a short syntax to create a list based on the values of an existing list. List comprehension is more compact than functions and loops for making lists.

Here’s the syntax, we’ll use.

[expression for item in list]

Here’s an example:

words = ['united states', 'brazil', 'united kingdom']

capitalized = [word.title() for word in words]

>>> capitalized
['United States', 'Brazil', 'United Kingdom']

It looks better, isn’t it?

Remember that we should keep the code user-friendly, so don’t write very long list comprehensions in a single line.

3. Dictionary Comprehension

Similar to list comprehension, there’s also dictionary comprehension in Python. The idea is the same. Dictionary comprehension offers a short syntax to create a dictionary in one line of code.

Here’s the syntax, we’ll use:

{key: value for key, value in iterable}

Here’s an example:

dict_numbers = {x:x*x for x in range(1,6) }

>>> dict_numbers
{1: 1, 2: 4, 3: 9, 4: 16, 5:25}

4. Join Dictionaries

There are different ways to join dictionaries. You can use the update() method, merge() operator, and even dictionary comprehension.

That said, there’s a simpler way to join dictionaries in Python. This is by using the unpack operator **. We only need to add the ** in front of each dictionary we wish to combine and use an additional dictionary to store the output.

dict_1 = {'a': 1, 'b': 2}
dict_2 = {'c': 3, 'd': 4}

merged_dict = {**dict_1, **dict_2}

>>> merged_dict
{'a': 1, 'b': 2, 'c': 3, 'd': 4}

After we apply the ** operator to the dictionaries, both will expand their content and combine to create a new dictionary.

Subscribe now

5. Remove Duplicates in a List

Sometimes we need to make sure there isn’t any duplicate value in our list. Although there isn’t a method that can take care of it easily, you can use sets to get rid of duplicates.

A set is an unordered collection of items where every element is unique. This means that if we turn our list into a set, we can remove duplicates. Then we only need to convert the set into a list again.

Let’s see a basic example to get the hang of it.

numbers = [1,1,1,2,2,3,4,5,6,7,7,8,9,9,9]

>>> list(set(numbers))
[1, 2, 3, 4, 5, 6, 7, 8, 9]

6. Filter values from a list

Say we want to filter some values from a list. You can use many approaches to do this, but a simple way to do it is by using the filter() function.

Here’s the syntax of the filter function:

filter(function, iterable)

If you add a lambda function inside of the filter function, things get even better!

Let’s get the hang of it by filtering even numbers from a list.

my_list = [10, 11, 12, 13, 14, 15]

>>> list(filter(lambda x: x%2 == 0, my_list ))
[10, 12, 14]

7. Sort dictionary by keys

Sorting dictionaries isn’t as simple as sorting lists — we can’t use either sort() or sorted() to sort dictionaries as we do it with lists.

The good news is that we can combine dictionary comprehension with the sorted() function to sort dictionaries by keys.

Let’s have a look. In the example below, we’ll sort the dictionaries by the name of the products.

product_prices = {'Z': 9.99, 'Y': 9.99, 'X': 9.99}

>>{key:product_prices[key] for key in sorted(product_prices.keys())}
{'X': 9.99, 'Y': 9.99, 'Z': 9.99}

8. Sort dictionary by values

Similarly to sorting dictionaries by keys, we need to use the sorted() function and list comprehension to sort dictionaries by values. But, in addition to that, we also need to add a lambda function.

First, let’s see all the parameters that the sorted() function has.

sorted(iterable, key=None, reverse=False)

To sort a dictionary by values, we need to use the key parameter. This parameter accepts a function that serves as a key for the sort comparison. Here we can use a lambda function to make things simpler.

Let’s say we have a dictionary with population values and we want to sort it by values.

population = {'USA':329.5, 'Brazil': 212.6, 'UK': 67.2}

>>> sorted(population.items(), key=lambda x:x[1])
[('UK', 67.2), ('Brazil', 212.6), ('USA', 329.5)]

Now the only thing left is to add dictionary comprehension.

population = {'USA':329.5, 'Brazil': 212.6, 'UK': 67.2}

>>> {k:v for k, v in sorted(population.items(), key=lambda x:x[1])}
{'UK': 67.2, 'Brazil': 212.6, 'USA': 329.5}

That’s it! If you have another one-liner that you use frequently, share it in the comments!

Subscribe now

Learning Python can be difficult. You might spend a lot of time watching videos and reading books; however, if you can’t put all the concepts learned into practice, that time will be wasted.

This is why you should get your hands dirty with Python projects. A project will help you bring together everything you’ve learned, stay motivated, build a portfolio, and come up with ways of approaching problems and solving them with code.

In this article, I listed some projects that helped me level up my Python code and hopefully will help you too. The projects are listed by difficulty, so the beginner projects are in the beginning, while the advanced projects are at the end of the article.

Subscribe now

Read more

Jupyter Notebook is the computational notebook of choice for data scientists and AI enthusiasts. It allows you to create documents containing not only live code but also equations, visualizations, and text. However, by default, Jupyter Notebook lacks several useful features, such as autocompletion, a table of contents, code folding, etc.

This is why I made a list of useful Jupyter Notebook extensions that will make your life easier and increase your productivity when writing code.

Subscribe now

How to Install Extensions

To install extensions, run the following code in the command prompt or terminal

pip install jupyter_contrib_nbextensions

Then run the code below to add the nbextensions files into the Jupyter server’s search directory.

jupyter contrib nbextension install

Now open Jupyter Notebooks. There should be a new tab called “Nbextensions.” Click on it and you’ll see a bunch of extensions you can use to increase your productivity on Jupyter Notebooks.

The following are the extensions I find the most useful.

Move selected cell

This is an extremely useful extension that will allow you to move selected cell(s) using keyboard shortcuts Alt-up and Alt-down. Just check the “Move selected cell” box inside Nbextensions. Then, refresh the notebook and you will be able to move cells via simple keystrokes.

Enable autocompletion (Hinterland)

This is an extremely useful extension for those who struggle writing code on Jupyter Notebooks because there’s no autocompletion. By checking the ‘Hinterland’ box, you’ll enable autocompletion on Jupyter Notebooks and would be able to write code like in your favorite editor.

Explore the complete list of extensions handpicked by me by becoming a paid subscriber :)

Read more

I’ve made a list of 5 books I read some years ago.

I recommend reading them if you’ve been dabbling in data science, ML, or AI and would like to learn them from scratch. Also, the books are great for those with some previous knowledge of Python, statistics, or math and would like to know how that knowledge can be used in data science, ML, and AI.

Subscribe now

#1 “Data Science from Scratch: First Principles with Python” by Joel Grus

This book aims to help you develop the hacking skills necessary to get started doing data science. After reading the book, you’d be comfortable with the maths and statistics that are at the core of data science. Although there’s a chapter in this book called “A Crash Course in Python,” the book’s goal is not to teach you Python from scratch, so, in case you have never written code in Python, you should supplement this book with one of my recommended Python books for beginners.

The book is packed with topics that every data scientist should be familiar with in the fields of statistics (probability, hypothesis, and inference), linear algebra (vectors and matrices), and machine learning (theory and concepts behind machine learning models).

Keep in mind that this first book mostly gives an overview of the topics mentioned before. To dive deeper into these topics, consider reading the books listed below in no particular order.

Read more

Python is known for being a slow programming language. Although it’s a fact that Python is slower than other languages, there are some ways to speed up our Python code.

How? Simple, optimize your code.

If we write code that consumes little memory and storage, we’ll not only get the job done but also make our Python code run faster.

Here’s a fast and also a super-fast way to loop in Python.

The average loop

Say we want to sum the numbers from 1 to 100000000 (we might never do that but that big number will help me make my point).

A typical approach would be to create a variable total_sum=0, loop through a range and increment the value of total_sum by i on every iteration.

import time

start = time.time()

total_sum = 0
for i in range(100000000):
    total_sum += i

print(f'Sum: {total_sum}')
print(f'For loop: {time.time() - start} seconds')

This gets the job done, but it takes around 6.58 seconds.

Although that doesn’t look so slow now, it’ll get slower as you add more 0s to the number inside the range.

Let’s speed this up!

Subscribe now

Read more