For many of us, it wouldn’t be surprising if, in the not-so-distant future, the global center of gravity for AI shifted from Silicon Valley to a Chinese city like Shenzhen or Beijing.

A series of telling developments in 2025 suggests that China is mounting a serious challenge to the United States' long-standing dominance in artificial intelligence.

The Chinese AI lab DeepSeek captured global attention in early 2025 when its chatbot app surged to the number one spot on both the Apple App Store and Google Play. This “DeepSeek moment” took technologists and Wall Street analysts by surprise. The company—relatively unknown at the time and based in Hangzhou—delivered an AI model on par with its Western counterparts, but developed at a significantly lower cost.

But the rise of DeepSeek has broader implications.

First, it validated China’s AI strategy focused on computational efficiency: training large-scale models using less advanced hardware.

Second, by open-sourcing its …

Read more

Whether you’re new to programming or an experienced developer, Python has one or two applications that might interest you.

The best thing is that you don’t need to be an expert programmer to get started with Python. Python’s syntax makes Python code similar to natural language, which makes it even easier for beginners to learn.

In this article, I’ll list all the things you can do with Python from simple applications that don’t require previous knowledge to advanced stuff that requires knowledge in other fields besides programming.

I’ll leave links to free tutorials to get you started.

Subscribe now

Automation

Probably the most fun and easiest application of Python is automation. You don’t need to become an expert in Python to build simple automation that will save you from doing repetitive tasks.

The only thing you need to do is find something worth automating and then learn the Python libraries that will help you automate this task.

What can you automate with Python?

What’s worth automating?…

Read more

Python is the language of choice in AI. It offers a large number of libraries that provide great functionality in mathematics, statistics, and scientific functions.

However, Python has applications beyond AI, so there are a good number of Python libraries that you will never use in an AI project.

In this article, I will show you the top Python libraries any AI enthusiast should know. and I will share resources to help you learn them.

Note: I added some lines of code to complement the explanation. That said, you don’t need to be an expert programmer to follow this article.

Subscribe now

Python libraries for Data Collection

Every data project starts with data collection. Sometimes the data is available in a CSV format or needs to be extracted from a database. However, when the data isn’t available, you could get public data from the biggest database in the world — the internet. The libraries below help you extract data from the internet with a technique called web scraping.

Request & Beautiful Soup

By using both the Request and Beautiful Soup library we can extract data from websites that don’t run JavaScript.

The requests library helps us make HTTP requests in Python. Thanks to this, we can get the content of a website. Then we use a parser (e.g., html.parser, lxml, etc) and Beautiful Soup to extract any data within the website.

Let’s see an example:

import requests
from bs4 import BeautifulSoup

# sending request and parsing
website = requests.get('https://example.com').text
soup = BeautifulSoup(website, 'html.parser')

# extracting data
headlines = soup.find_all('span', class_='class-example')
data = [headline.text for headline in headlines]

Selenium/Scrapy

Selenium and Scrapy do the same job as Beautiful Soup; however, they’re more powerful.

Both of them can extract data from JavaScript-driven websites. Selenium can be also used for web automation, while Scrapy is fast, allows you to easily export data to a database, and have other functionalities that make it the most complete tool.

Below you will find guides to start learning these libraries from scratch:

• ​Beautiful Soup Guide

• ​Selenium Guide

• 4 Web Scraping Project to Automate Your Life

Python libraries for Data Cleaning & Wrangling

Once you have the data in a readable format (CSV, JSON, etc), it’s time to clean it. The Pandas and Numpy libraries can help with it.

Pandas

Pandas is a powerful tool that offers a variety of ways to manipulate and clean data. Pandas work with dataframes that structures data in a table similar to an Excel spreadsheet, but faster and with all the power of Python.

This is how you create a Pandas dataframe:

import pandas as pd

# data used for the example (stored in lists)
states = ["California", "Texas", "Florida", "New York"]
population = [39613493, 29730311, 21944577, 19299981]

# Storing lists within a dictionary
dict_states = {'States': states, 'Population': population}

# Creating the dataframe
df_population = pd.DataFrame.from_dict(dict_states)

print(df_population)

Numpy

Numpy is a Python library with math functionalities. It allows us to work with multi-dimensional arrays, matrices, generate random numbers, linear algebra routines, and more.

When it comes to wrangling and transforming data, some Numpy methods such as np.where and np.select are often used. In addition to that, other libraries such as Matplotlib, and Scikit-learn depend on NumPy to some extent.

Let’s see how to create a two-dimensional array with NumPy.

import numpy as np

b = np.array([[1.5,2,3], [4,5,6]],dtype=float)

IN [0]: print(b)
IN [1]: print(f'Dimension: {b.ndim}')

OUT [0]: [[1.5  2.  3. ]
          [4.   5.  6. ]]
OUT [1]: Dimension: 2

Imbalanced-learn

Imbalanced-learn is a library that helps us deal with imbalanced data. Imbalanced data happens when the number of observations per class is not equally distributed. For example, in the review section of an Amazon product, you will typically see a high number of positive reviews (the majority class) and a low number of negative reviews (the minority class).

We use the imbalanced-learn (imblearn) library to resample our data. For example, you can undersample positive reviews or oversample negative reviews.

Below you will find guides to start learning these libraries from scratch:

• Pandas & Numpy Guide for Excel users

Subscribe now

Python libraries for Data Visualization

Plots such as pie charts, bar plots, boxplots, and histograms are often used in Exploratory Data Analysis and also when presenting results. Python libraries allow us to make traditional as well as interactive plots.

Matplotlib/Seaborn

Matplotlib is a library that allows us to make basic plots, while Seaborn specializes in statistics visualization.

The main difference is in the lines of code you need to write to create a plot. Seaborn is easier to learn, has default themes, and makes better-looking plots than Matplotlib by default.

Let’s create a barplot of the df_population dataframe we created in the Pandas section.

import matplotlib.pyplot as plt

plt.bar(x=df_population['States'],
        height=df_population['Population'])

plt.xlabel('States')
plt.ylabel('Population')
plt.show()

Now let’s create the same plot with Seaborn.

import seaborn as sns

sns.barplot(x=df_population['States'],
            y=df_population['Population'],
            palette='deep')
plt.show()

As you can see, we don’t need to specify the axes names in Seaborn (it takes it from the dataframe columns), while Matplotlib needs more lines of code and the plots aren’t good looking at all by default.

Plotly/Bokeh (Pandas integration)

If you want to go to the next level, you should try making interactive visualization with Plotly or Bokeh. Both allow creating a good number of interactive plots and the coolest thing is that you can use any of them to plot directly with Pandas plotting syntax.

Both make it easy to plot interactive visualization, but in my opinion, Plotly creates better-looking plots by default.

Here’s an example of how to create interactive plots with Plotly using Pandas plotting syntax.

import pandas as pd
import cufflinks as cf
from IPython.display import display,HTML
cf.set_config_file(sharing='public',theme='white',offline=True) 

df_population = df_population.set_index('States')
df_population.iplot(kind='bar', color='red',
                    xTitle='States', yTitle='Population')

Wordcloud/Stylecloud

Wordclouds allows us to identify keywords in a piece of text. Python has two libraries for this type of graphs —wordclouds and stylecloud.

The first makes basic wordclouds and even allows us to upload our own image as a mask for the wordcloud, while the second creates gorgeous wordclouds with a few lines of codes and offers a good number of high-quality icons that you can use in your wordcloud.

Let’s make a wordcloud of the famous Steve Job’s speech at Standford.

import stylecloud

stylecloud.gen_stylecloud(file_path='SJ-Speech.txt',
                          icon_name= "fas fa-apple-alt")

That’s all you need to make this wordcloud! You can remove stopwords and use other functionalities. For more details, check my wordcloud guide.

Below you will find guides to start learning these libraries from scratch:

• Matplotlib & Seaborn Guide

Python libraries used in AI & ML models

Subscribe now

Read more

As we’ve seen in previous articles Python is the language of choice for AI. In this article, I present you the 4 stages to learn Python for AI & Machine Learning. I left many links to resources that will come in handy for you.

Python is the most popular language in the AI community due to its simplicity, flexibility, and data science libraries such as Pandas, Numpy, and Scikit-learn. This is why, in this article, we will see the Python stuff you need for AI and Machine Learning and discover what stage you’re in.

I’ll describe each stage and give you tips on how to master them so that you can move to the next stage.

Subscribe now

Stage 1: The Basics of Python

This stage is for anyone who is learning the basics of Python.

At this level, you should at least know basic concepts such as data types and variables. Knowing the most popular options to store data (lists, dictionaries, and tuples.) is a must at this level. Also, you should be able to use conditional statements and control flow tools. This includes the if/else statements, boolean operations, and different types of loops (for, while, and nested).

Conditional statements, control flow, and loops open the door for a large variety of things you can do with Python, so use them and stay curious to develop a strong foundation necessary for the next stage.

One last important thing at this level is to start getting familiar with Jupyter Notebook. Jupyter allows us to create not only code but equations, visualizations, and text.

Topics: Data types, variables, lists, dictionaries, tuples, conditions, operators, control flow (if / else), loops, iterables, functions, file I/O operations (read, write to text files), and common methods.

How to master this level? As I mentioned before, solving problems that involve conditional statements, control flow, and loops will help you master stage 1.

Projects for beginners:

• Automate 4 Boring Tasks in Python with 5 Lines of Code

• How to Automate Emails with Python

Stage 2: Python for Data Analysis

This is what I call the “essential Python stuff to work with data.” This means having at least a basic understanding of libraries used for data analysis such as Pandas, NumPy, Matplotlib, and Seaborn.

Using those libraries to solve common data analysis tasks such as data cleaning, exploratory data analysis (EDA) through visualizations, and feature engineering is important at this level.

Data analysis tutorials:

• Pandas & Numpy for beginners

• Data Cleaning

• Data Visualization

If you’re able to understand the code in the tutorials above, then you’re at this stage.

Regarding the stuff you already knew in stage 1, there’s still room for improvement — especially for the stuff you would frequently use for data analysis. Some of them are list comprehension, lambda, zip(), f-string, and the with statement.

Last but not least, acquiring skills necessary for data collection like web scraping will come in handy. Below are guides to learning web scraping from scratch.

Web scraping guides:

• Web scraping with pandas

• Web scraping with Beautiful Soup

• Web scraping with Selenium

Topics: Most of the methods/functions used in Pandas, NumPy, Matplotlib, Seaborn, and web scraping libraries (Selenium and Scrapy). List comprehension, lambda, zip(), f-string, the with statement, and any other stuff that helps you write better code.

How to master this level? Solving Python projects. At this stage, projects usually involve all the data analysis libraries mentioned before. Make sure you start projects that have topics you’re interested in (that’s more fun!)

Projects for stage 2:

• 4 Web Scraping Projects That Will Help Automate Your Life

• Predicting Football Games With a Simple Model

Subscribe now

Stage 3: Python for Statistics & Math

In Stage 3 different fields get together, so your Python project will become an ML project. You already know how to clean data and conduct EDA from stage 2, but also you’re supposed to know all the fundamental statistics and math behind ML.

Statistics is crucial to make sure the data you are using to train a model is not biased. For example, using Matplotlib and Seaborn to plot histograms and boxplots will help you identify outliers. In addition to that, you should know how to apply most statistical concepts to a project. You should know how to deal with imbalanced data, segment train/test data, and formulate a problem and hypothesis.

Some topics in math you should know are functions and matrices. This stuff is implemented in Python through Numpy. Numpy has support for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays.

Another important thing you should understand is how machine learning algorithms work. There’s a lot of math and statistics behind those algorithms, so make sure you understand them before learning the Python code that lets you build them.

Guides for stage 3:

• Machine Learning Algorithms Any AI Enthusiast Should Know (Part 1)

• Machine Learning Algorithms Any AI Enthusiast Should Know (Part 2)

Topics: Imbalanced data, segment train/test data, machine learning algorithms, arrays/matrices (Numpy), data visualization (Matplotlib/Seaborn). Above all, you should know how to apply statistics and math to a project.

How to master this level? Solving projects such as sentiment analysis, credit card fraud detection, and customer churn prediction.

Stage 4: Python for Machine Learning

The last stage is all about developing machine learning models. The scikit-learn library is a good start to this. Some basic things you should be able to do with this library are text representation (BOW, Count Vectorizer, TF-IDF), model selection, evaluation, and parameter tuning. This project covers all these topics. If you’re able to understand the code, then you’re at this level.

Other important libraries for data scientists at this level are Keras and TensorFlow. Keras features several of the building blocks and tools necessary for creating a neural network such as neural layers, activation and cost functions, objectives, etc. TensorFlow is one of the best library available for working with Machine Learning on Python. It makes machine learning model building easy for beginners and professionals alike.

Guides for stage 4:

• Building a Basic Machine Learning Model in Python

• What is NLP And Why is Important in AI (7 NLP Techniques)

Topics: Text representation, model selection, evaluation, and parameter tuning, among others.

How to master this level and beyond? This will depend on the area you’re interested in. Find an area you like and learn the necessary libraries you need for it. For example, if you’re into NLP, learning NLTK and solving projects like building a movie recommender system or a chatbot would help you get started in this area.

Subscribe now

Share

Give a gift subscription

In this second part, we will continue exploring how some algorithms used in Machine Learning work. As we discussed in the first part, there are both supervised and unsupervised algorithms.

Let’s see 6 more algorithms you should know.

1.K-Nearest Neighbors (kNN)

KNN algorithm is a supervised learning technique used for both classification and regression. It’s a simple yet practical method for making predictions. The goal is to find the "k" nearest observations in the feature space and make a decision based on the labels of these neighbors.

Let's break down the key concepts to get a better understanding of how this algorithm works:

• The value of "k" is an integer that defines how many neighboring points will be considered.

• Distance: To measure how close two points are, a distance metric is usually used. The most common is Euclidean distance, though others like Manhattan, Minkowski, and more can also be applied.

• If we use it for classification, the new point is assigned t…

Read more

Natural language processing (NLP) is a subfield of AI that uses machine learning to enable computers to understand and communicate with human language. NLP is used in various AI applications like chabots.

Today, we’ll learn some common NLP techniques. We’ll focus on the concepts. That said, I added some lines of Python code to better understand the techniques with examples (you don’t need to know coding to understand the examples though).

Natural Language Processing (NLP) is focused on enabling computers to understand and process human language. Computers are great at working with structured data like spreadsheets; however, a lot of the data we generate is unstructured.

We can implement many NLP techniques with just a few lines of Python code thanks to open-source libraries such as spaCy and NLTK. In this article, we’ll dive into the world of NLP by learning some common techniques.

Table of Contents
1. Sentiment Analysis
2. Named Entity Recognition (NER)
3. Stemming
4. Lemmatization
5. Bag of Words (BoW)
6. Term Frequency–Inverse Document Frequency (TF-IDF)
7. Bonus: Wordcloud

Subscribe now

1. Sentiment Analysis

Sentiment Analysis is a popular NLP technique that involves taking a piece of text (e.g., a comment, review, or a document) and determining whether it’s positive, negative, or neutral. It has many applications in healthcare, customer service, banking, etc.

Python Implementation

For simple cases, in Python, we can use VADER (Valence Aware Dictionary for Sentiment Reasoning) which is available in the NLTK package and can be applied directly to unlabeled text data.

As an example, let’s get all sentiment scores of the lines spoken by characters in a TV show. First, we wrangle a dataset available on Kaggle named ‘avatar.csv’, and then with VADER we calculate the score of each line spoken. All of this is stored in the df_character_sentiment dataframe.

import pandas as pd
import nltk
from nltk.sentiment.vader import SentimentIntensityAnalyzer

# reading and wragling data
df_avatar = pd.read_csv('avatar.csv', engine='python')
df_avatar_lines = df_avatar.groupby('character').count()
df_avatar_lines = df_avatar_lines.sort_values(by=['character_words'], ascending=False)[:10]
top_character_names = df_avatar_lines.index.values

# filtering out non-top characters
df_character_sentiment = df_avatar[df_avatar['character'].isin(top_character_names)]
df_character_sentiment = df_character_sentiment[['character', 'character_words']]

# calculating sentiment score
sid = SentimentIntensityAnalyzer()
df_character_sentiment.reset_index(inplace=True, drop=True)
df_character_sentiment[['neg', 'neu', 'pos', 'compound']] = df_character_sentiment['character_words'].apply(sid.polarity_scores).apply(pd.Series)
df_character_sentiment

In the df_character_sentiment below, we can see that every sentence receives a negative, neutral, and positive score.

We could group the scores by character and calculate the mean to obtain the sentiment score for a character and then represent it with horizontal bar plots as shown below.

Note: VADER is optimized for social media text, so we should take the results with a grain of salt. You can use a more complete algorithm or develop your own with machine learning libraries. In the link below, there’s a complete guide on how to create one from scratch with Python using the sklearn library.

2. Named Entity Recognition (NER)

Named Entity Recognition is a technique used to locate and classify named entities in text into categories such as persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. It‘s used for optimizing search engine algorithms, recommendation systems, customer support, content classification, etc.

Python Implementation

In Python, we can use SpaCy’s named entity recognition that supports the following entity types.

To see it in action, we first import spacy, and then create a nlp variable that will store the en_core_web_sm pipeline. This is a small English pipeline trained on written web text (blogs, news, comments), that includes vocabulary, vectors, syntax, and entities. To find the entities, we apply nlp to a sentence.

Let’s do a test with the following sentence "Biden invites Ukrainian president to White House this summer."

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Biden invites Ukrainian president to White House this summer")

print([(X.text, X.label_) for X in doc.ents])

Here are the entities we get.

[('Biden', 'PERSON'), ('Ukrainian', 'GPE'), ('White House', 'ORG'), ('this summer', 'DATE')]

Spacy found that “Biden” is a person, “Ukranian” is GPE (countries, cities, states, “White House” is an organization, and “this summer” is a date.

Subscribe now

3. Stemming & Lemmatization

Stemming and lemmatization are 2 popular techniques in NLP. Both normalize a word but in different ways.

• Stemming: It truncates a word to its stem word. For example, the words “friends,” “friendship,” and “friendships” will be reduced to “friend.” Stemming may not give us a dictionary, or grammatical word for a particular set of words.

• Lemmatization: Unlike the stemming technique, lemmatization finds the dictionary word instead of truncating the original word. Lemmatization algorithms extract the correct lemma of each word, so they often require a dictionary of the language to be able to categorize each word correctly.

Both techniques are widely used and you should choose them wisely based on your project’s goals. Lemmatization has a lower processing speed, compared to stemming so if accuracy is not the project’s goal but speed, then stemming is an appropriate approach; however. if accuracy is crucial, then consider using lemmatization.

Python’s library NLTK makes it easy to work with both techniques. Let’s see it in action.

Python Implementation (Stemming)

For the English language, there are two popular libraries available in nltk — Porter Stemmer and LancasterStemmer.

from nltk.stem import PorterStemmer
from nltk.stem import LancasterStemmer

# PorterStemmer
porter = PorterStemmer()
# LancasterStemmer
lancaster = LancasterStemmer()

print(porter.stem("friendship"))
print(lancaster.stem("friendship"))

PorterStemmer algorithm doesn’t follow linguistics, but a set of 5 rules for different cases that are applied in phases to generate stems. The print(porter.stem(“friendship”)) code will print the word friendship

LancasterStemmer is simple, but heavy stemming due to iterations and over-stemming may occur. This causes the stems to be not linguistic, or they may have no meaning. The print(lancaster.stem(“friendship”)) code will print the word friend.

You can try any other word to see how both algorithms differ. In the case of other languages, you can import SnowballStemme from nltk.stem

Python Implementation (Lemmatization)

We’ll use NLTK again, but this time we import WordNetLemmatizer as shown in the code below.

from nltk import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()
words = ['articles', 'friendship', 'studies', 'phones']

for word in words:
    print(lemmatizer.lemmatize(word))

Lemmatization generates different outputs for different Part Of Speech (POS) values. Some of the most common POS values are verb (v), noun (n), adjective (a), and adverb (r). The default POS value in lemmatization is a noun, so the printed values for the previous example will be article, friendship, study and phone.

Let’s change the POS value to verb (v).

from nltk import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()
words = ['be', 'is', 'are', 'were', 'was']

for word in words:
    print(lemmatizer.lemmatize(word, pos='v'))

In this case, Python will print the word be for all the values in the list.

Subscribe now

5. Bag of Words

The Bag of Words (BoW) model is a representation that turns text into fixed-length vectors. This helps us represent text into numbers so we can use it for machine learning models. The model doesn’t care about the word order, but it’s only concerned with the frequency of words in the text. It has applications in NLP, information retrieval from documents, and classifications of documents.

The typical BoW workflow involves cleaning raw text, tokenization, building a vocabulary, and generating vectors.

Python Implementation

Python’s library sklearn contains a tool called CountVectorizer that takes care of most of the BoW workflow.

Let’s use the following 2 sentences as examples.

Sentence 1: “I love writing code in Python. I love Python code”

Sentence 2: “I hate writing code in Java. I hate Java code”

Both sentences will be stored in a list named text. Then we’re going to create a dataframe df to store this text list. After this, we‘ll initiate an instance of CountVectorizer(cv), and then we’ll fit and transform the text data to obtain the numeric representation. This will be stored in a document-term matrix df_dtm.

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer

text = ["I love writing code in Python. I love Python code",
        "I hate writing code in Java. I hate Java code"]

df = pd.DataFrame({'review': ['review1', 'review2'], 'text':text})
cv = CountVectorizer(stop_words='english')
cv_matrix = cv.fit_transform(df['text'])
df_dtm = pd.DataFrame(cv_matrix.toarray(),
                      index=df['review'].values,
                      columns=cv.get_feature_names())
df_dtm

The BoW representation made with CountVectorizer stored in df_dtm looks like the picture below. Keep in mind that words with 2 letters or fewer are not taken into account by the CountVectorizer.

As you can see the numbers inside the matrix represent the number of times each word was mentioned in each review. Words like “love,” “hate,” and “code” have the same frequency (2) in this example.

Overall, we can say that CountVectorizer does a good job tokenizing text, building a vocabulary, and generating vectors.

6. Term Frequency–Inverse Document Frequency (TF-IDF)

Unlike the CountVectorizer, the TF-IDF computes “weights” that represent how relevant a word is to a document in a collection of documents (aka corpus). The TF-IDF value increases proportionally to the number of times a word appears in the document and is offset by the number of documents in the corpus that contain the word. Simply put, the higher the TF-IDF score, the rarer or unique or valuable the term and vice versa. It has applications in information retrieval like search engines that aim to deliver results that are most relevant to what you’re searching for.

Before we see the Python implementation, let’s see an example so you have an idea of how the TF and IDF are calculated. For the following example, we’ll use the same sentences used for the CountVectorizer example.

Sentence 1: “I love writing code in Python. I love Python code”

Sentence 2: “I hate writing code in Java. I hate Java code”

Term Frequency (TF)

There are different ways to define the term frequency. One suggests the raw count itself (i.e., what the Count Vectorizer does), but others suggest it’s the frequency of the word in the sentence divided by the total number of words in the sentence.

For this simple example, we’ll use the first criteria. The term frequency is shown in the following table.

As you can see, the values are the same as the ones calculated for the CountVectorizer before. Also, words with 2 letters or fewer are not taken into account.

Inverse Document Frequency (IDF)

The IDF is also calculated in different ways. Although standard textbook notation defines the IDF as idf(t) = log [ n / (df(t) + 1), the sklearn library we’ll use later in Python calculates the formula by default as follows.

Also, sklearn assumes natural logarithm ln instead of log and smoothing (smooth_idf=True). Let’s calculate the IDF values for each word as sklearn will do it.

TF-IDF

Once we have the TF and IDF values, we can obtain the TF-IDF by multiplying both values (TF-IDF = TF * IDF). The values are shown in the table below.

Python Implementation

Calculating the TF-IDF shown in the table above in Python requires a few lines of code thanks to the sklearn library.

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer

text = ["I love writing code in Python. I love Python code",
        "I hate writing code in Java. I hate Java code"]

df = pd.DataFrame({'review': ['review1', 'review2'], 'text':text})
tfidf = TfidfVectorizer(stop_words='english', norm=None)
tfidf_matrix = tfidf.fit_transform(df['text'])
df_dtm = pd.DataFrame(tfidf_matrix.toarray(),
                      index=df['review'].values,
                      columns=tfidf.get_feature_names())
df_dtm

The TF-IDF representation stored in df_dtm is presented below.

Note: By default TfidfVectorizer() uses l2 normalization, but to use the same formulas shown above we set norm=None as a parameter. For more details of the formulas used by default in sklearn and how you can customize it check its documentation.

Subscribe now

Bonus: Wordcloud

Wordcloud is a popular technique that helps us identify the keywords in a text. It’s not considered as a NLP technique but it still uses some of the techniques explained in this article.

In a wordcloud, more frequent words have a larger and bolder font, while less frequent words have smaller or thinner fonts. In Python, you can make simple wordclouds with the wordcloud library and nice-looking wordclouds with the stylecloudlibrary.

Below you can find the code to make a wordcloud in Python. I’m using a text file of a Steve Jobs speech.

import stylecloud

stylecloud.gen_stylecloud(file_path='SJ-Speech.txt',
                          icon_name= "fas fa-apple-alt")

This is the result of the code above.

Wordclouds are popular because they’re engaging, easy to understand, and easy to create.

You can take customization even further by changing the colors, removing stopwords, choosing your image, or even adding your own image as a mask of the wordcloud.

Share

Leave a comment

In the previous article of our Behind AI series, we explained in plain English what databases are and compared the top DBMS. Today, we’ll focus on a type of database that has become increasingly important in AI: vector databases.

The internet contains a huge amount of data in different forms. In the past, this data was mostly structured, but as the internet grew, unstructured data such as photos, audio, text, and video files became more common.

Analysts estimate that 80-90% of any organization’s data is unstructured, so how can we deal with this unstructured data?

Traditional relational databases and NoSQL databases struggle to analyze unstructured data especially when it comes to doing it in real-time. Here’s when vector databases can help. They were built to manage massive embeddings vectors converted from unstructured data.

In this article, we’ll learn more about vector databases, how they can help us manage unstructured data, and the solutions they offer.

Vector Databases vs Tradition…

Read more

Hi!

Data is an important resource in AI. Companies store data in different types of databases and today we’ll learn everything about them (in the next article, we’ll focus on a type of database that is specially used in AI, so stay tuned!)

No matter what your job is probably you’ve ever heard of the word database.

Companies out there use different types of databases to store all the information they collected throughout the years. Although all these databases might seem the same, they have some functionalities that make them more suitable for certain situations, so it’s worth learning more about them.

In this article, we’ll see what databases are and the most commonly used in companies.

Table of Contents
1. What is a Database?
2. Types of Databases
 - Relational databases
 - Non-Relational databases
3. Top databases

Subscribe now

What is a Database?

A database is a collection of data typically stored electronically in a computer system and controlled by a database management system (DBMS). The data, the DBMS, and the applications associated with them are referred to as a database system (or just “database”).

The words database management system and database are often used interchangeably, but technically they’re not the same.

To distinguish them consider the case of a social media app that stores different information about its users such as messages, photos, comments, etc. The database stores this big collection of data, but that’s pretty much what it does. If you want to edit, update, or delete data, you need a DBMS that does the talking for you. Some of the most popular DBMS are Oracle, MySQL, SQL Server, and PostgreSQL.

Again, people often refer to both the DBMS and DB simply as “database” but now you know how this actually works.

Another way to think of a database is as a big spreadsheet with many rows and columns. That’s a good comparison, but a database goes beyond that. Both the database and spreadsheet are good for storing information, but they mainly differ in the following aspects:

• How the data is stored and manipulated: Databases allow complex data manipulation, while spreadsheets aren't meant for users who need much data manipulation.

• Who can access the data: Databases allow multiple users to quickly access and query the data, while spreadsheets were designed only for a single user or a small number of users.

• The amount of data that can be stored: Databases are designed to store larger collections of data, while spreadsheets have a limitation.

Last but not least, a database cannot only store data in tables and rows. That’s how relational databases typically work, but there’s also another type of database called non-relational database. This leads us to our next point.

Subscribe now

Types of Databases

Databases are typically divided into relational and non-relational databases. Among the top 10 databases, you’ll see both relational and non-relational databases. One type of database is not better than the other, but they suit different needs.

Relational databases

A relational database (aka SQL database), stores data in tables and rows also referred to as records. This type of database links information from different tables through keys.

A key is a unique value in a table that is also known as the “primary key”. When this key is added to a record located in another table, it’s called “foreign key” in this second table. This connection between primary and foreign keys creates a relationship between records within both tables.

Some popular relational database management systems (RDBMS) are Oracle, MySQL, SQL Server, and PostgreSQL.

Here’s a basic schema that shows how a relational database works.

To query data in a RDBMS, we use Structured Querying Language (SQL). With SQL we can create new records, update them, and more. This makes the RDBMS good for apps that need transactional functionality, data mining, and complex reporting.

Non-Relational databases

A non-relational database (aka NoSQL database), stores data without tables, rows, or keys. In other words, a non-relational database stores data in a non-tabular form. This adds some flexibility and helps satisfy specific requirements of the type of data being stored.

You can think of a non-relational database as a collection of documents. A document can contain a lot of detailed information about a customer. Each customer can have different types of information, but they can be stored in the same document.

The ability to process and organize different types of information makes non-relational databases more flexible than relational databases.

There are four popular non-relational types: document data store, column-oriented database, key-value store, and graph database. One of the most popular NoSQL databases is MongoDB.

Subscribe now

What are the top databases?

It’s hard to rank the database based on their functionality because they suit different needs and can be more convenient in certain scenarios than in others. That said, it’s possible to rank the database management systems according to their popularity.

In fact, DB-Engine ranks DBMS by their current popularity. To do so, they calculate scores following different parameters.

Here are the top 10 databases by their popularity.

Now let’s see more about them, compare them, and see their pros and cons.

1. Oracle

Oracle Database is a widely used RDBMS across industries. In fact, it has the largest market share of around 30.2% in the RDBMS market.

Oracle Database supports SQL language to interact with the database. It’s considered one of the best databases because it supports all data types involving relational, graph, structured, and unstructured information. In addition to that, Oracle Database is preferred for its flexible standards, scalability, high availability, and strong security.

Pros

• It’s highly compatible with different apps and platforms

• Helps with scalability

• It offers good privacy and security

Cons

• The license is expensive

• Users might need extensive SQL knowledge to use Oracle Database

Popularity

Google Trends shows more interest in Oracle than in MySQL over the past 5 years. The graph also reveals the same ups and downs for both databases.

2. MySQL

MySQL is one of the most popular databases. It’s open-source so any person or company can use MySQL for free, but if the code needs to be integrated into a commercial application, you need to purchase a license. That said, this database is still worth it for anyone who wants to experiment with a friendly yet powerful database.

MySQL was developed by Oracle and it’s a relational database management system. As explained before, the relation model consists in organizing data in tables with rows and columns, while the relationship between elements follows a logical structure. Companies such as Facebook, Twitter, Wikipedia, and YouTube employ MySQL backends.

Pros

• It’s open source: Unlike other options, you don’t have to pay to use most features of MySQL

• It’s cross-platform: Runs on Linus, Solaris, and Windows and supports platforms with programming languages such as C, C++, Java, Python, etc.

• Reliable data security: MySQL is known for being a secure database management system. This is why so many well-known companies use it in their applications.

• It’s easy to use: Anyone can download, install and start to use MySQL in a few minutes.

Cons

• It’s not for large-sized data

• It doesn’t support SQL check constraints

• It doesn’t have a good debugging tool compared to paid databases

• It doesn’t handle transactions very efficiently

Popularity

Google Trends shows that the interest in MySQL has slightly decreased over the past 5 years, but has suddenly risen compared to other databases like SQL Server.

Subscribe now

3. SQL Server

SQL Server was developed by Microsoft and it’s considered a great RDBMS for both on-premise and cloud environments. It has a Database Engine component that allows storing, processing, and securing data. The database engine is divided into two segments — the relational and storage engine. The first is used to process commands and queries, while the second is used to manage features such as tables, pages, files, indexes, and transactions.

Besides SQL language, SQL Server also includes Transact-SQL (T-SQL), which is Microsoft’s extension to the SQL used to interact with relational databases. SQL Server is a good option for businesses that want to scale the performance, availability, and security seamlessly based on their requirements.

Pros

• It has various supported editions (enterprise, standard, express, and developer). The express SQL server edition is free of cost.

• It has an online documentation

• On-premise and cloud database support

• It offers different tools and apps

Cons

• Expensive enterprise edition

• It’s available for Windows, Linux, and macOS, but the steps to install it on a Mac aren’t as straightforward as on a Windows machine.

Popularity

Google Trends shows more interest over time in SQL Server than in PostgreSQL. In fact, the interest in PostgreSQL hasn’t changed that much over the past 5 years.

4. PostgreSQL

PostgreSQL is known as the world’s most advanced open source object-relational database management system (ORDBMS). Part of this reputation is due to its architecture, reliability, robustness, and extensibility.

PostgreSQL comes with many features that help build apps, protect data integrity and help manage data no matter how big or small the data is. It’s also highly extensible in many areas. To name a few:

• Stored functions and procedures

• PL/PGSQL, Perl, Python

• SQL/JSON path expressions

• Additional functionality such as PostGIS (spatial database extender for PostgreSQL)

These extensions help us process data right from PostgreSQL, so we don’t need to find workarounds to implement them.

Pros

• It’s extremely programmable: You can extend PostgreSQL thanks to its directory-based operation and dynamic loading

• It’s highly extensible

• It has a very rich set of indexing options

Cons

• Performance: PostgreSQL is sometimes less efficient than other RDBMS such as MySQL (at least for simple intensive reading operations)

• It might be difficult to troubleshoot PostgreSQL

Popularity

Google Trends shows similar interest over time in PostgreSQL and MongoDB. That said, we should consider that PostgreSQL was initially released in 1996, while MongoDB was released in 2009.

5. MongoDB

MongoDB is an open-source document database that uses a flexible schema for storing data. Unlike SQL databases that store data in tables of rows and columns, NoSQL database programs like MongoDB use JSON-like documents with optional schemas.

MongoDB is great for those who build internet and business applications and need to evolve and scale quickly. Some of the advantages of MongoDB for developers are the power of document-oriented databases (documents can be retrieved directly in JSON format, which developers find easy to work with), user experience, scalability and transactionality, and its thriving community.

Overall, MongoDB is good if you’re looking for a database that:

• Supports rapid iterative development

• Enables the scale to high levels of read and write traffic

• Stores, manages, and searches data when creating apps

Pros

• It offers a flexible schema that it’s not possible to get in a RDBMS

• Scalability: MongoDB uses sharding, which allows the database to use horizontal scalability.

• It’s free and supports Windows, macOS, and Linux

Cons

• High memory usage: The data size in MongoDB is higher than in other databases

• Less flexibility with querying: It fails to support joins as a relational database

Give a gift subscription

Hi!

I created a new series called “AI & Python” which will be focused on learning Python, coding concepts, automation, and creating AI apps. From now on, “Behind AI“ will be focused on what's behind the AI products you love. No programming knowledge will be required. Tech concepts (if any) will be explained in plain English.

Examples of a Behind AI piece are this article and the one you’re reading right now.

P.S. In case you’re not interested in the Python articles, go to settings and turn off notifications for “AI & Python” (leave the rest the same to keep receiving my other emails)

Machine learning (ML) is the field behind all the magic in AI products. If you’re new to ML, you probably must’ve heard of the words “algorithm” or “model” without knowing how they’re related to machine learning.

Here’s a brief explanation in plain English.

Machine learning algorithms are categorized as supervised or unsupervised. Supervised learning algorithms model the relationship between labeled input and output data (aka target). This model is then used to predict the label of new observations using new labeled input data. If the target variable is discrete, we’re dealing with a classification problem, while if the target variable is continuous we’re dealing with a regression problem. In contrast, unsupervised learning doesn’t rely on labeled input/output data but processes unlabeled data.

Here are 6 supervised learning algorithms that you should know.

Subscribe now

1. Linear Regression

Linear regression is the simplest algorithm used in machine learning. This algorithm is used for modeling the relationship between two or more variables. There are two types of linear regression — simple and multiple linear regression.

In simple linear regression, there’s one independent variable and one dependent variable, while in multiple linear regression there are multiple independent variables and one dependent variable.

Here’s the multiple linear regression equation:

where y is the dependent variable (target value), x1, x2, … xn the independent variables (predictors), b0 the intercept, b1, b2, ... bn the coefficients and n the number of observations.

In the picture below, you’ll see a simplified version of the linear regression equation.

As you can see in the picture above, there’s a linear relationship, so if one variable increases or decreases, the other variable will also increase or decrease.

We can use linear regression to predict scores, salaries, house prices, etc. That said, the prediction accuracy isn’t as good as those you’d get with other algorithms.

2. SVM

A Support Vector Machine (SVM) is a supervised learning algorithm that is mostly used in classification problems. We usually feed the SVM model with labeled training data to categorize new text.

SVM is a good choice when we have a limited number of samples and speed is a priority. This is why it’s used when we work with a dataset that has a few thousand of tagged samples in text classification.

To understand much better how SVM works let’s see an example.

In the picture below, we have two tags (green and yellow) and two features (x and y). Say we want to build a classifier that finds whether our text data is either green or yellow. If that’s the case, we will plot each observation (aka data point) in an n-dimensional space, where “n” is the number of features used.

We only have two features, so the observations are plotted in 2-dimensional space as shown in the picture below.

SVM takes the data points and makes a hyperplane that best separates the classes. Since the observations are plotted in 2-dimensional space, the hyperplane is a line.

This red line is also known as the decision boundary. The decision boundary determines whether a data point belongs to one class or to another. In our example, if the data point falls on the left side, it will be classified as green, while if it falls on the right side, it will be classified as yellow.

Subscribe now

3. Decision Tree

If you know nothing about machine learning, you might still know about decision trees.

A decision tree is a model used in planning, statistics, and machine learning that uses a tree-like structure of decisions/consequences to evaluate the possible events involved in a particular problem.

Here’s a decision tree that evaluates scenarios where people want to play football.

Each square is called a node. The last nodes of the decision tree are called the leaves of the tree. To make predictions we start from the root of the tree (first node). Each node in the decision tree will be evaluated. Then we follow the branch that agrees with the evaluation and jump to the next node.

The decision tree algorithm can be used for solving both regression and classification problems. We use a decision tree to build a model that can predict the class or value of the target variable by learning decision tree rules inferred from the training data.

4. Random Forest

Random forest is an ensemble of many decision trees. It combines the simplicity of a decision tree with flexibility resulting in an improvement in accuracy.

To make a random forest, first, we need to create a “bootstrapped” dataset. Bootstrapping is randomly selecting samples from original data (we can even choose the same sample more than once). Then, we use the bootstrapped dataset to create a decision tree.

This method is known as “bagging.” If we repeat the previous steps multiple times, we get a good number of trees. This variety of trees is what makes random forests more effective than a single decision tree.

If the random forest is used for a classification task, the model selects the mode of the predictions of each decision tree. For a regression task, the model selects the mean value of the results from the decision trees.

5. Naive Bayes

Naive Bayes is a supervised learning algorithm that uses conditional probability to predict a class.

The Naive Bayes algorithm is based on the Bayes theorem:

p(A|B): Probability of event A given event B has already occurred
p(B|A): Probability of event B given event A has already occurred
p(A): Probability of event A
p(B): Probability of event B

Naive Bayes assumes that every feature is independent of each other, which isn’t always the case, so we should examine our data before choosing this algorithm.

The assumption that features are independent of each other makes Naive Bayes fast compared to more complex algorithms; however, it also makes this algorithm less accurate.

We can use Naive Bayes to predict weather forecasting, fraud detection, and more.

Subscribe now

6. Logistic Regression

Logistic regression is a supervised learning algorithm that is commonly used for binary classification problems. This means we can use logistic regression to predict whether a customer will churn or not, and to find whether a mail is spam or not.

The logistic regression is based on the logistic function (aka the sigmoid function), which takes in a value and assigns a probability between 0 and 1.

Here’s the graph of the logistic regression:

To understand much better how Logistic Regression works, consider a scenario where we need to classify whether an email is spam or not.

In the graph, if Z goes to infinity, Y (our target value) will become 1, which means the email is spam. However, if Z goes to negative infinity, Y will become 0, which means the email is not spam.

The output value is a probability, so if we obtain a value of 0.64, this means that there’s a 64% chance that an email will be spam.

That’s it for now! Let me know in the comments if you want a 2nd part of this.

Share

Subscribe now

Hi!

The other day I was thinking that many people still don’t know the value of data, which is surprising given that data is what makes AI possible. Companies like OpenAI have been collecting data for many years to train their models, creating the tools we all know today.

In the coming articles, I’ll show you the techniques these companies use to collect data and what you can do with this data but, first, let’s see why data is so important nowadays.

The concept of data as a strategic asset has been gaining momentum in the past years, however, regular people aren’t able to see the real value in data.

We know big tech companies have been collecting data for a long time. We know that year after year new regulations about the use of data are created. That said, most of us still don’t understand the impact data has on our society.

A few years ago, The Economist published an article called “The world’s most valuable resource is no longer oil, but data.” However, for regular…

Read more