For many of us, it wouldn’t be surprising if, in the not-so-distant future, the global center of gravity for AI shifted from Silicon Valley to a Chinese city like Shenzhen or Beijing.

A series of telling developments in 2025 suggests that China is mounting a serious challenge to the United States' long-standing dominance in artificial intelligence.

The Chinese AI lab DeepSeek captured global attention in early 2025 when its chatbot app surged to the number one spot on both the Apple App Store and Google Play. This “DeepSeek moment” took technologists and Wall Street analysts by surprise. The company—relatively unknown at the time and based in Hangzhou—delivered an AI model on par with its Western counterparts, but developed at a significantly lower cost.

But the rise of DeepSeek has broader implications.

First, it validated China’s AI strategy focused on computational efficiency: training large-scale models using less advanced hardware.

Second, by open-sourcing its model, DeepSeek helped catalyze a new wave of “open AI” in China. Almost overnight, dozens of Chinese companies began integrating DeepSeek’s models into their own products.

What did that spark?

A rapid acceleration of domestic AI innovation. What began as a viral chatbot—with millions of downloads in just a few weeks—is now evolving into a full-fledged ecosystem.

And this is only part of the story. There are several other reasons why China is beginning to reshape the future of the global AI industry.

Subscribe now

Building the Silicon Great Wall: Chips, Chips, Chips

In May 2024, China launched its third state-backed semiconductor fund, totaling 344 billion yuan (about $47.5 billion), bringing its total “Big Fund” investment to roughly $100 billion across three phases. Backed by the Ministry of Finance and major state-owned banks, these funds aim to support everything from chip fabrication plants to advanced lithography R&D.

This effort is part of President Xi’s broader push for “semiconductor self-sufficiency.” Under the Made in China 2025 initiative, the official goal is for 70% of China’s chip demand to be met domestically by 2025.

So, what impact is this having on the chip industry?

For starters, China’s chip production is booming in the low- and mid-range segments, flooding the market with affordable chips for cars, home appliances, and Internet of Things (IoT) devices. Despite US sanctions, Chinese firms have also made impressive strides in high-end chips.

At Tsinghua University, researchers developed ACCEL, a light-based analog AI chip that, in lab tests, completed an image recognition task 3,000 times faster than Nvidia’s A100 GPU—while using far less power.

In another breakthrough, a fully optical AI chip called Taichi-II demonstrated a one-million-fold improvement in energy efficiency for certain tasks by training directly with photons.

Of course, heavy investment doesn’t guarantee success.

China still trails in producing cutting-edge chipmaking equipment, such as extreme ultraviolet (EUV) lithography machines, which are essential for the latest CPUs and GPUs. Still, with billions flowing into startups, new fabrication facilities, subsidies, and talent pipelines, the momentum is hard to ignore.

Manus vs the Machines: AI Agents Get Real

Manus is a general-purpose AI agent developed by Chinese startup Monica. It made headlines by claiming to be the world’s first truly autonomous AI assistant—a system capable of planning and carrying out complex tasks with minimal input.

Unlike traditional chatbots that respond to one prompt at a time, Manus can browse the web, book appointments, write code, analyze data, and string together actions to achieve a broader objective.

It’s no surprise that some observers dubbed Manus’s launch “another DeepSeek moment,” marking what they saw as a significant step forward in AI capabilities.

What makes Manus especially noteworthy is how it fits into China’s growing AI ecosystem.

On one hand, it’s the product of a small, homegrown startup. But it quickly gained international attention and racked up a waitlist of 2 million users—accessible by invitation only.

By March 2025, China’s state broadcaster CCTV had begun airing reports comparing Manus to DeepSeek’s chatbot, and Beijing’s city government fast-tracked regulatory approval for Manus to enter the domestic market.

Clearly, Chinese authorities see Manus as a potential new flagship for the country’s AI ambitions—just as DeepSeek has been—particularly if it can compete with leading Western products. In fact, Manus recently partnered with Alibaba to integrate with its Qwen AI models, signaling a broader trend of collaboration between China’s tech giants and up-and-coming startups in the race to build advanced AI agents.

Manus may be China’s answer to OpenAI—or, given the timing, the question that inspired OpenAI’s answer.

After going viral on X and impressing analysts with its live demos, Manus sent a clear signal to the West: China could soon lead not only in conversational AI, but in task-oriented AI as well.

While it’s not without flaws—early users have noted that it can be slow or prone to the usual AI errors—being first to market with a general-purpose agent gives China a potentially powerful edge.

Subscribe now

A Quantum Leap: Zuchongzhi-3 vs. Google’s Willow

While China continues to invest heavily in traditional silicon-based chips, it’s also making significant progress in quantum computing—a field that could eventually solve problems far beyond the capabilities of even today’s most advanced supercomputers.

In 2024, researchers at the University of Science and Technology of China (USTC) introduced Zuchongzhi 3.0, a superconducting quantum processor with 105 qubits, putting it roughly on par with Google’s latest quantum chip, known as “Willow.”

So, why does this matter?

Zuchongzhi-3 completed a complex task called random circuit sampling in just a few hundred seconds—a process that would take Frontier, the world’s second-fastest supercomputer, an estimated 5.9 billion years to complete.

To put this into perspective, Google’s Willow achieved a similar milestone in late 2024 using its own 105-qubit processor, marking a significant step toward quantum supremacy. China has now matched that breakthrough—and may have even raised the bar.

What does this mean for AI?

In the near term, quantum computers remain experimental tools, not yet practical for training the next generation of AI models. However, the future of AI could be accelerated by technologies like quantum optimization and quantum neural networks. By establishing an early lead in quantum hardware, China is positioning itself to take advantage of this convergence when it arrives.

As one Chinese researcher put it, this progress “lays the groundwork for a new era where quantum processors play an essential role in tackling real-world challenges.”

Practically speaking, China’s advances in quantum computing could give its scientists a powerful edge in solving encryption, search, and simulation problems—capabilities that not only strengthen its research ecosystem but also pose new questions for the future of AI.

Final Thoughts

China has rolled out record-setting AI models, embraced open-source approaches that challenge the closed ecosystems favored by many Western firms, invested heavily in semiconductor independence, demonstrated parity in quantum computing, and even taken the lead in developing new AI agent paradigms.

At the same time, US tech giants remain firmly in the race—continuing to produce many of the world’s most advanced models and innovations, with strong advantages in chip technology (think Nvidia) and foundational research.

Rather than declaring a winner too soon, it may be more prudent to keep asking the right questions. After all, AI leadership isn’t a zero-sum game. Progress in one part of the world can lift the entire field—but if one side pulls too far ahead, it could also create a widening global divide in AI capabilities.

Subscribe now

Whether you’re new to programming or an experienced developer, Python has one or two applications that might interest you.

The best thing is that you don’t need to be an expert programmer to get started with Python. Python’s syntax makes Python code similar to natural language, which makes it even easier for beginners to learn.

In this article, I’ll list all the things you can do with Python from simple applications that don’t require previous knowledge to advanced stuff that requires knowledge in other fields besides programming.

I’ll leave links to free tutorials to get you started.

Subscribe now

Automation

Probably the most fun and easiest application of Python is automation. You don’t need to become an expert in Python to build simple automation that will save you from doing repetitive tasks.

The only thing you need to do is find something worth automating and then learn the Python libraries that will help you automate this task.

What can you automate with Python?

What’s worth automating? It can be anything. Here are my favorite automation scripts I built so far:

• Automating Morning News

• Sending Emails

• Automating Boring Excel Reports

• Whatsapp Messages

• Tinder

Here’s how I used Python to send messages on WhatsApp (believe it or not the message is sent by a bot).

After you write a script to automate a task, you can schedule it to be run at any time you want. Forget about doing any boring task manually!

If you’re still unsure of where to start, I recommend you start with web automation. There are millions of websites out there that you can automate with a single Python library: Selenium.

Free Tutorials: 3 automation projects

Web Scraping

Web scraping is all about collecting data from websites. This is a skill that data analysts and data scientists should have, but it can be used in any field where there’s a need for data.

Web scraping is a technique that consists in creating scrapers (aka automated bots) that extract millions of data points from the internet. This is similar to copying data from a website and then pasting it into a spreadsheet. That said, it can take us hours to do this task manually, however, a web scraper will do this in a couple of minutes (or seconds!).

In this video, you can find everything you need to know about web scraping explained in 3 minutes.

In Python, we can scrape websites using libraries such as Beautiful Soup, Selenium, and Scrapy. Beautiful Soup is an easy-to-learn library but has lots of limitations. Selenium is a web automation library that can scrape JavaScript-driven websites but is a bit slow. In contrast, Scrapy is a robust framework that has multiple functionalities and is faster than the other 2 libraries.

You can scrape most websites out there. To name a few:

• Twitter

• Amazon

• Airbnb

• News Sites

Free tutorials: Beautiful Soup, Selenium, Scrapy

Subscribe now

Data Analysis & Data Science

Data is one of the most valuable things on the internet. Some even call data “the new oil of the 21st century.”

Regardless of whether that’s true or not, the large amount of data available on the internet and possessed by big organizations is undeniable. According to IBM, the global volume of data was predicted to reach 35 zettabytes in 2020.

With so much data out there, there’s a demand for professionals to work with this data and get value from it. Here’s when data analysts and data scientists come into play.

In data analysis, we use Python to clean data, wrangle data and create visualizations. We use Python libraries such as Pandas, Numpy, Matplotlib, and Seaborn to do that.

In data science, we usually go one step further and use Python to develop a machine learning model that predicts outputs based on data we feed to our model.

Here is all the Python stuff you need to learn for data science explained in 5 minutes.

Machine learning is often used in data science, so at least you should learn sklearn, which is the most basic library to do machine learning in Python. This library is the foundation for other advanced Python libraries such as TensorFlow and Keras.

Some models you can build with machine learning are:

• Fake News Detection

• Credit card fraud detection

• Customer Churn Prediction

Free Course: Data Analysis with Python

Web Development

If you ever dreamt of building your own website, I have good news for you — you can do it with Python!

Python has some frameworks like Flask and Django that allows us to build the back end of a website with Python. Of course, to build the front end you still need to use HTML, CSS, and JavaScript. That said, you have nothing to worry about since HTML is a very simple language and you could use Bootstrap to avoid implementing CSS and JavaScript code from scratch.

What websites were built using Python?

There are many websites out there that use Python for the back-end of their websites. Here are some of them.

Some of these websites used Python in early stages, while others still use it to this date. In fact, Instagram currently features the world’s largest deployment of the Django web framework (written entirely in Python).

Below you’ll find more about how Python powers these websites:

• How Python Powers Dropbox

• How we use Python at Spotify

• Why Reddit uses Python

Here are some free tutorials to get you started with web development.

Free Tutorials: Python Website Tutorial with Flask, Django For Beginners 

Machine Learning & Artificial Intelligence

Machine learning is a branch of artificial intelligence that allows a machine to automatically learn from past data without programming explicitly.

This isn’t something that you can learn solely with Python but requires knowledge in linear algebra, calculus, and more. With Python, we can use scientific libraries such as Numpy, Pandas, PyTorch, and TensorFlow to take care of all the math behind a machine learning model. Your job will be to understand the results and make the best decision using your analytical skills.

Here are some popular machine-learning applications:

• Google’s self-driving car

• Recommendation systems from Amazon, YouTube, and Netflix

• Fraud detection

Of course, those are advanced applications. Before you dive into Machine Learning and AI you need to have a solid foundation in calculus, and linear Algebra and know Python libraries such as NumPy, pandas, Matplotlib, and PyTorch.

Once you familiarize yourself with the basics of machine learning you can learn more advanced Python libraries like TensorFlow and Keras.

Below are free tutorials that you can take to learn these libraries.

Free Tutorial: TensorFlow 2.0 — Python Neural Networks for Beginners Tutorial

Once you acquired the previous skills, the sky is the limit!

You can use this knowledge to specialize in more advanced topics like NLP, AI, and DL.

That’s it! Let your Python journey begin!

Give a gift subscription

Share

Python is the language of choice in AI. It offers a large number of libraries that provide great functionality in mathematics, statistics, and scientific functions.

However, Python has applications beyond AI, so there are a good number of Python libraries that you will never use in an AI project.

In this article, I will show you the top Python libraries any AI enthusiast should know. and I will share resources to help you learn them.

Note: I added some lines of code to complement the explanation. That said, you don’t need to be an expert programmer to follow this article.

Subscribe now

Python libraries for Data Collection

Every data project starts with data collection. Sometimes the data is available in a CSV format or needs to be extracted from a database. However, when the data isn’t available, you could get public data from the biggest database in the world — the internet. The libraries below help you extract data from the internet with a technique called web scraping.

Request & Beautiful Soup

By using both the Request and Beautiful Soup library we can extract data from websites that don’t run JavaScript.

The requests library helps us make HTTP requests in Python. Thanks to this, we can get the content of a website. Then we use a parser (e.g., html.parser, lxml, etc) and Beautiful Soup to extract any data within the website.

Let’s see an example:

import requests
from bs4 import BeautifulSoup

# sending request and parsing
website = requests.get('https://example.com').text
soup = BeautifulSoup(website, 'html.parser')

# extracting data
headlines = soup.find_all('span', class_='class-example')
data = [headline.text for headline in headlines]

Selenium/Scrapy

Selenium and Scrapy do the same job as Beautiful Soup; however, they’re more powerful.

Both of them can extract data from JavaScript-driven websites. Selenium can be also used for web automation, while Scrapy is fast, allows you to easily export data to a database, and have other functionalities that make it the most complete tool.

Below you will find guides to start learning these libraries from scratch:

• ​Beautiful Soup Guide

• ​Selenium Guide

• 4 Web Scraping Project to Automate Your Life

Python libraries for Data Cleaning & Wrangling

Once you have the data in a readable format (CSV, JSON, etc), it’s time to clean it. The Pandas and Numpy libraries can help with it.

Pandas

Pandas is a powerful tool that offers a variety of ways to manipulate and clean data. Pandas work with dataframes that structures data in a table similar to an Excel spreadsheet, but faster and with all the power of Python.

This is how you create a Pandas dataframe:

import pandas as pd

# data used for the example (stored in lists)
states = ["California", "Texas", "Florida", "New York"]
population = [39613493, 29730311, 21944577, 19299981]

# Storing lists within a dictionary
dict_states = {'States': states, 'Population': population}

# Creating the dataframe
df_population = pd.DataFrame.from_dict(dict_states)

print(df_population)

Numpy

Numpy is a Python library with math functionalities. It allows us to work with multi-dimensional arrays, matrices, generate random numbers, linear algebra routines, and more.

When it comes to wrangling and transforming data, some Numpy methods such as np.where and np.select are often used. In addition to that, other libraries such as Matplotlib, and Scikit-learn depend on NumPy to some extent.

Let’s see how to create a two-dimensional array with NumPy.

import numpy as np

b = np.array([[1.5,2,3], [4,5,6]],dtype=float)

IN [0]: print(b)
IN [1]: print(f'Dimension: {b.ndim}')

OUT [0]: [[1.5  2.  3. ]
          [4.   5.  6. ]]
OUT [1]: Dimension: 2

Imbalanced-learn

Imbalanced-learn is a library that helps us deal with imbalanced data. Imbalanced data happens when the number of observations per class is not equally distributed. For example, in the review section of an Amazon product, you will typically see a high number of positive reviews (the majority class) and a low number of negative reviews (the minority class).

We use the imbalanced-learn (imblearn) library to resample our data. For example, you can undersample positive reviews or oversample negative reviews.

Below you will find guides to start learning these libraries from scratch:

• Pandas & Numpy Guide for Excel users

Subscribe now

Python libraries for Data Visualization

Plots such as pie charts, bar plots, boxplots, and histograms are often used in Exploratory Data Analysis and also when presenting results. Python libraries allow us to make traditional as well as interactive plots.

Matplotlib/Seaborn

Matplotlib is a library that allows us to make basic plots, while Seaborn specializes in statistics visualization.

The main difference is in the lines of code you need to write to create a plot. Seaborn is easier to learn, has default themes, and makes better-looking plots than Matplotlib by default.

Let’s create a barplot of the df_population dataframe we created in the Pandas section.

import matplotlib.pyplot as plt

plt.bar(x=df_population['States'],
        height=df_population['Population'])

plt.xlabel('States')
plt.ylabel('Population')
plt.show()

Now let’s create the same plot with Seaborn.

import seaborn as sns

sns.barplot(x=df_population['States'],
            y=df_population['Population'],
            palette='deep')
plt.show()

As you can see, we don’t need to specify the axes names in Seaborn (it takes it from the dataframe columns), while Matplotlib needs more lines of code and the plots aren’t good looking at all by default.

Plotly/Bokeh (Pandas integration)

If you want to go to the next level, you should try making interactive visualization with Plotly or Bokeh. Both allow creating a good number of interactive plots and the coolest thing is that you can use any of them to plot directly with Pandas plotting syntax.

Both make it easy to plot interactive visualization, but in my opinion, Plotly creates better-looking plots by default.

Here’s an example of how to create interactive plots with Plotly using Pandas plotting syntax.

import pandas as pd
import cufflinks as cf
from IPython.display import display,HTML
cf.set_config_file(sharing='public',theme='white',offline=True) 

df_population = df_population.set_index('States')
df_population.iplot(kind='bar', color='red',
                    xTitle='States', yTitle='Population')

Wordcloud/Stylecloud

Wordclouds allows us to identify keywords in a piece of text. Python has two libraries for this type of graphs —wordclouds and stylecloud.

The first makes basic wordclouds and even allows us to upload our own image as a mask for the wordcloud, while the second creates gorgeous wordclouds with a few lines of codes and offers a good number of high-quality icons that you can use in your wordcloud.

Let’s make a wordcloud of the famous Steve Job’s speech at Standford.

import stylecloud

stylecloud.gen_stylecloud(file_path='SJ-Speech.txt',
                          icon_name= "fas fa-apple-alt")

That’s all you need to make this wordcloud! You can remove stopwords and use other functionalities. For more details, check my wordcloud guide.

Below you will find guides to start learning these libraries from scratch:

• Matplotlib & Seaborn Guide

Python libraries used in AI & ML models

Subscribe now

Read more

As we’ve seen in previous articles Python is the language of choice for AI. In this article, I present you the 4 stages to learn Python for AI & Machine Learning. I left many links to resources that will come in handy for you.

Python is the most popular language in the AI community due to its simplicity, flexibility, and data science libraries such as Pandas, Numpy, and Scikit-learn. This is why, in this article, we will see the Python stuff you need for AI and Machine Learning and discover what stage you’re in.

I’ll describe each stage and give you tips on how to master them so that you can move to the next stage.

Subscribe now

Stage 1: The Basics of Python

This stage is for anyone who is learning the basics of Python.

At this level, you should at least know basic concepts such as data types and variables. Knowing the most popular options to store data (lists, dictionaries, and tuples.) is a must at this level. Also, you should be able to use conditional statements and control flow tools. This includes the if/else statements, boolean operations, and different types of loops (for, while, and nested).

Conditional statements, control flow, and loops open the door for a large variety of things you can do with Python, so use them and stay curious to develop a strong foundation necessary for the next stage.

One last important thing at this level is to start getting familiar with Jupyter Notebook. Jupyter allows us to create not only code but equations, visualizations, and text.

Topics: Data types, variables, lists, dictionaries, tuples, conditions, operators, control flow (if / else), loops, iterables, functions, file I/O operations (read, write to text files), and common methods.

How to master this level? As I mentioned before, solving problems that involve conditional statements, control flow, and loops will help you master stage 1.

Projects for beginners:

• Automate 4 Boring Tasks in Python with 5 Lines of Code

• How to Automate Emails with Python

Stage 2: Python for Data Analysis

This is what I call the “essential Python stuff to work with data.” This means having at least a basic understanding of libraries used for data analysis such as Pandas, NumPy, Matplotlib, and Seaborn.

Using those libraries to solve common data analysis tasks such as data cleaning, exploratory data analysis (EDA) through visualizations, and feature engineering is important at this level.

Data analysis tutorials:

• Pandas & Numpy for beginners

• Data Cleaning

• Data Visualization

If you’re able to understand the code in the tutorials above, then you’re at this stage.

Regarding the stuff you already knew in stage 1, there’s still room for improvement — especially for the stuff you would frequently use for data analysis. Some of them are list comprehension, lambda, zip(), f-string, and the with statement.

Last but not least, acquiring skills necessary for data collection like web scraping will come in handy. Below are guides to learning web scraping from scratch.

Web scraping guides:

• Web scraping with pandas

• Web scraping with Beautiful Soup

• Web scraping with Selenium

Topics: Most of the methods/functions used in Pandas, NumPy, Matplotlib, Seaborn, and web scraping libraries (Selenium and Scrapy). List comprehension, lambda, zip(), f-string, the with statement, and any other stuff that helps you write better code.

How to master this level? Solving Python projects. At this stage, projects usually involve all the data analysis libraries mentioned before. Make sure you start projects that have topics you’re interested in (that’s more fun!)

Projects for stage 2:

• 4 Web Scraping Projects That Will Help Automate Your Life

• Predicting Football Games With a Simple Model

Subscribe now

Stage 3: Python for Statistics & Math

In Stage 3 different fields get together, so your Python project will become an ML project. You already know how to clean data and conduct EDA from stage 2, but also you’re supposed to know all the fundamental statistics and math behind ML.

Statistics is crucial to make sure the data you are using to train a model is not biased. For example, using Matplotlib and Seaborn to plot histograms and boxplots will help you identify outliers. In addition to that, you should know how to apply most statistical concepts to a project. You should know how to deal with imbalanced data, segment train/test data, and formulate a problem and hypothesis.

Some topics in math you should know are functions and matrices. This stuff is implemented in Python through Numpy. Numpy has support for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays.

Another important thing you should understand is how machine learning algorithms work. There’s a lot of math and statistics behind those algorithms, so make sure you understand them before learning the Python code that lets you build them.

Guides for stage 3:

• Machine Learning Algorithms Any AI Enthusiast Should Know (Part 1)

• Machine Learning Algorithms Any AI Enthusiast Should Know (Part 2)

Topics: Imbalanced data, segment train/test data, machine learning algorithms, arrays/matrices (Numpy), data visualization (Matplotlib/Seaborn). Above all, you should know how to apply statistics and math to a project.

How to master this level? Solving projects such as sentiment analysis, credit card fraud detection, and customer churn prediction.

Stage 4: Python for Machine Learning

The last stage is all about developing machine learning models. The scikit-learn library is a good start to this. Some basic things you should be able to do with this library are text representation (BOW, Count Vectorizer, TF-IDF), model selection, evaluation, and parameter tuning. This project covers all these topics. If you’re able to understand the code, then you’re at this level.

Other important libraries for data scientists at this level are Keras and TensorFlow. Keras features several of the building blocks and tools necessary for creating a neural network such as neural layers, activation and cost functions, objectives, etc. TensorFlow is one of the best library available for working with Machine Learning on Python. It makes machine learning model building easy for beginners and professionals alike.

Guides for stage 4:

• Building a Basic Machine Learning Model in Python

• What is NLP And Why is Important in AI (7 NLP Techniques)

Topics: Text representation, model selection, evaluation, and parameter tuning, among others.

How to master this level and beyond? This will depend on the area you’re interested in. Find an area you like and learn the necessary libraries you need for it. For example, if you’re into NLP, learning NLTK and solving projects like building a movie recommender system or a chatbot would help you get started in this area.

Subscribe now

Share

Give a gift subscription

In this second part, we will continue exploring how some algorithms used in Machine Learning work. As we discussed in the first part, there are both supervised and unsupervised algorithms.

Let’s see 6 more algorithms you should know.

1.K-Nearest Neighbors (kNN)

KNN algorithm is a supervised learning technique used for both classification and regression. It’s a simple yet practical method for making predictions. The goal is to find the "k" nearest observations in the feature space and make a decision based on the labels of these neighbors.

Let's break down the key concepts to get a better understanding of how this algorithm works:

• The value of "k" is an integer that defines how many neighboring points will be considered.

• Distance: To measure how close two points are, a distance metric is usually used. The most common is Euclidean distance, though others like Manhattan, Minkowski, and more can also be applied.

• If we use it for classification, the new point is assigned to the most common class among its "k" neighbors (majority voting). If there is a tie, it can be resolved using additional criteria, such as weighting the distances.

If we use it for regression, the prediction is made by averaging the values of the neighbors.

Here’s a representation of how the algorithm performs a classification with a hyperparameter of k = 5.

The advantage of this algorithm is that it’s easy to interpret and implement. Additionally, it doesn't make assumptions about the data distribution. However, it’s important to consider the characteristics of the data since this algorithm is distance-based. Another point to keep in mind is its high computational cost for large datasets. It’s also very sensitive to noise or outliers when k is small, while a very large k can overly smooth the decision.

Subscribe now

2. Gradient Boosted Decision Trees (GBDT)

GBDT is a supervised learning algorithm used for both classification and regression tasks. It’s a boosting technique that sequentially combines multiple decision trees, with each tree trying to correct the errors of the one before it. The idea is to add weak models (trees) together to build a more robust model.

The goal is to minimize error by combining several weak models (decision trees) into a strong model. For regression tasks, the loss function often uses Mean Squared Error (MSE), while for classification, it uses Log-loss.

One of the advantages of GBDT is its strong performance in both classification and regression tasks. It is also a robust model when dealing with outliers, thanks to its ability to handle non-linear data. However, a drawback is that since it uses boosting by adding more trees, the training process can be slower compared to other algorithms. Here, we face the trade-off between performance and speed.

3. K-Means Clustering

K-Means is an unsupervised clustering algorithm that groups data into k clusters based on their features. The goal is to minimize the distances between the points within a cluster and its centroid, aiming to reduce internal variance. This is an iterative process that essentially forms groups of data points with similar characteristics, distinguishing them from other, more distant groups.

Let’s break down how this algorithm works:

1. Initialization: Randomly select k centroids, which represent the centers of the initial clusters.

2. Assign Points to Clusters: Assign each data point to the cluster whose centroid is closest (using Euclidean distance).

3. Recalculate Centroids: Once all points are assigned, recalculate the centroids of each cluster as the mean of all points within that cluster.

4. Reassign Points: Reassign each data point to the cluster with the nearest updated centroid.

5. Repeat: Repeat steps 3 and 4 until the centroids no longer change significantly (or until a maximum number of iterations is reached).

Final Clusters: The data points will then be organized into k clusters, with their respective centroids stabilized.

Now, let’s see how we can use this algorithm to perform the following classification:

4. Hierarchical Clustering

Hierarchical clustering creates a tree-like structure (dendrogram) to group the data. Clusters are formed by either merging (agglomerative) or splitting (divisive) the data iteratively. The goal is to create a hierarchy of clusters that allows data to be grouped at different levels of similarity.

Here’s a step-by-step breakdown of how this algorithm works:

1. Initialization: Each data point starts as its own individual cluster.

2. Calculate Distances: Calculate the distances between all clusters, typically using Euclidean distance.

3. Merge Clusters: In each iteration, merge the two closest clusters (those with the smallest distance).

4. Update Distances: After merging two clusters, recalculate the distances between the newly formed cluster and the remaining clusters.

5. Build the Dendrogram: As clusters are merged, the dendrogram visually represents how clusters are combined at each level. The nodes in the tree indicate cluster merges at different heights.

6. Determine Final Clusters: The final number of clusters can be set by "cutting" the dendrogram at a certain level, corresponding to the desired number of clusters.

This will lead us to the final result:

Hierarchical clustering always produces the same clusters. In contrast, K-Means clustering can yield different clusters depending on the initial placement of the centroids (cluster centers). However, hierarchical clustering is slower compared to K-Means. It takes a significant amount of time to run, especially with large datasets.

5. DBSCAN Clustering

DBSCAN is a density-based clustering algorithm that groups nearby points and labels isolated points as "noise" or outliers. Unlike K-Means, it does not require you to define the number of clusters in advance. The goal is to identify high-density clusters and mark isolated points.

Here’s a closer look at how this algorithm works:

1. Parameter Definition: Two key parameters are set: ε (epsilon), which is the radius of the neighborhood around a point, and minPts, the minimum number of points needed to form a cluster.

2. Initialization: A random point is selected from the dataset.

3. Neighborhood Density: If there are at least minPts within the ε radius around the selected point, it is considered a core point, and a cluster is formed. If not, the point is marked as noise.

4. Cluster Expansion: The cluster expands by searching for all points within the ε radius of the points already included in the cluster. If any of these points also have at least minPts neighbors within their ε radius, the cluster continues to grow.

5. Repeat: This process repeats for each unvisited point. Points that do not meet the density criteria are marked as noise and are not included in any cluster.

6. Final Clusters: The algorithm concludes by forming clusters of densely packed points, while isolated points are labeled as noise.

In some cases, finding an appropriate neighborhood distance (ε) can be challenging and may require domain expertise.

6. Principal Component Analysis (PCA)

PCA is a dimensionality reduction technique used to transform a dataset with many features (variables) into a new set of uncorrelated variables called principal components. These components capture most of the variability in the original data while using the fewest possible dimensions.

Principal components are linear combinations of the original variables that capture the maximum variance in the data. They are calculated by finding the eigenvectors and eigenvalues, which define the direction of the components and indicate how much variance each component explains.

Here is a graphical representation before applying the PCA algorithm. After applying PCA, we select 2 components, as they account for the most explained variance. Finally, we project the original data onto these first 2 components.

One advantage of dimensionality reduction is that it simplifies the visualization and analysis of complex data by reducing the number of dimensions. It also eliminates multicollinearity since using linear combinations of the original variables removes correlations between them. However, a key disadvantage is the loss of interpretability and the sensitivity to variable scaling (if the data is not standardized).

Subscribe now

Give a gift subscription

Natural language processing (NLP) is a subfield of AI that uses machine learning to enable computers to understand and communicate with human language. NLP is used in various AI applications like chabots.

Today, we’ll learn some common NLP techniques. We’ll focus on the concepts. That said, I added some lines of Python code to better understand the techniques with examples (you don’t need to know coding to understand the examples though).

Natural Language Processing (NLP) is focused on enabling computers to understand and process human language. Computers are great at working with structured data like spreadsheets; however, a lot of the data we generate is unstructured.

We can implement many NLP techniques with just a few lines of Python code thanks to open-source libraries such as spaCy and NLTK. In this article, we’ll dive into the world of NLP by learning some common techniques.

Table of Contents
1. Sentiment Analysis
2. Named Entity Recognition (NER)
3. Stemming
4. Lemmatization
5. Bag of Words (BoW)
6. Term Frequency–Inverse Document Frequency (TF-IDF)
7. Bonus: Wordcloud

Subscribe now

1. Sentiment Analysis

Sentiment Analysis is a popular NLP technique that involves taking a piece of text (e.g., a comment, review, or a document) and determining whether it’s positive, negative, or neutral. It has many applications in healthcare, customer service, banking, etc.

Python Implementation

For simple cases, in Python, we can use VADER (Valence Aware Dictionary for Sentiment Reasoning) which is available in the NLTK package and can be applied directly to unlabeled text data.

As an example, let’s get all sentiment scores of the lines spoken by characters in a TV show. First, we wrangle a dataset available on Kaggle named ‘avatar.csv’, and then with VADER we calculate the score of each line spoken. All of this is stored in the df_character_sentiment dataframe.

import pandas as pd
import nltk
from nltk.sentiment.vader import SentimentIntensityAnalyzer

# reading and wragling data
df_avatar = pd.read_csv('avatar.csv', engine='python')
df_avatar_lines = df_avatar.groupby('character').count()
df_avatar_lines = df_avatar_lines.sort_values(by=['character_words'], ascending=False)[:10]
top_character_names = df_avatar_lines.index.values

# filtering out non-top characters
df_character_sentiment = df_avatar[df_avatar['character'].isin(top_character_names)]
df_character_sentiment = df_character_sentiment[['character', 'character_words']]

# calculating sentiment score
sid = SentimentIntensityAnalyzer()
df_character_sentiment.reset_index(inplace=True, drop=True)
df_character_sentiment[['neg', 'neu', 'pos', 'compound']] = df_character_sentiment['character_words'].apply(sid.polarity_scores).apply(pd.Series)
df_character_sentiment

In the df_character_sentiment below, we can see that every sentence receives a negative, neutral, and positive score.

We could group the scores by character and calculate the mean to obtain the sentiment score for a character and then represent it with horizontal bar plots as shown below.

Note: VADER is optimized for social media text, so we should take the results with a grain of salt. You can use a more complete algorithm or develop your own with machine learning libraries. In the link below, there’s a complete guide on how to create one from scratch with Python using the sklearn library.

2. Named Entity Recognition (NER)

Named Entity Recognition is a technique used to locate and classify named entities in text into categories such as persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. It‘s used for optimizing search engine algorithms, recommendation systems, customer support, content classification, etc.

Python Implementation

In Python, we can use SpaCy’s named entity recognition that supports the following entity types.

To see it in action, we first import spacy, and then create a nlp variable that will store the en_core_web_sm pipeline. This is a small English pipeline trained on written web text (blogs, news, comments), that includes vocabulary, vectors, syntax, and entities. To find the entities, we apply nlp to a sentence.

Let’s do a test with the following sentence "Biden invites Ukrainian president to White House this summer."

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Biden invites Ukrainian president to White House this summer")

print([(X.text, X.label_) for X in doc.ents])

Here are the entities we get.

[('Biden', 'PERSON'), ('Ukrainian', 'GPE'), ('White House', 'ORG'), ('this summer', 'DATE')]

Spacy found that “Biden” is a person, “Ukranian” is GPE (countries, cities, states, “White House” is an organization, and “this summer” is a date.

Subscribe now

3. Stemming & Lemmatization

Stemming and lemmatization are 2 popular techniques in NLP. Both normalize a word but in different ways.

• Stemming: It truncates a word to its stem word. For example, the words “friends,” “friendship,” and “friendships” will be reduced to “friend.” Stemming may not give us a dictionary, or grammatical word for a particular set of words.

• Lemmatization: Unlike the stemming technique, lemmatization finds the dictionary word instead of truncating the original word. Lemmatization algorithms extract the correct lemma of each word, so they often require a dictionary of the language to be able to categorize each word correctly.

Both techniques are widely used and you should choose them wisely based on your project’s goals. Lemmatization has a lower processing speed, compared to stemming so if accuracy is not the project’s goal but speed, then stemming is an appropriate approach; however. if accuracy is crucial, then consider using lemmatization.

Python’s library NLTK makes it easy to work with both techniques. Let’s see it in action.

Python Implementation (Stemming)

For the English language, there are two popular libraries available in nltk — Porter Stemmer and LancasterStemmer.

from nltk.stem import PorterStemmer
from nltk.stem import LancasterStemmer

# PorterStemmer
porter = PorterStemmer()
# LancasterStemmer
lancaster = LancasterStemmer()

print(porter.stem("friendship"))
print(lancaster.stem("friendship"))

PorterStemmer algorithm doesn’t follow linguistics, but a set of 5 rules for different cases that are applied in phases to generate stems. The print(porter.stem(“friendship”)) code will print the word friendship

LancasterStemmer is simple, but heavy stemming due to iterations and over-stemming may occur. This causes the stems to be not linguistic, or they may have no meaning. The print(lancaster.stem(“friendship”)) code will print the word friend.

You can try any other word to see how both algorithms differ. In the case of other languages, you can import SnowballStemme from nltk.stem

Python Implementation (Lemmatization)

We’ll use NLTK again, but this time we import WordNetLemmatizer as shown in the code below.

from nltk import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()
words = ['articles', 'friendship', 'studies', 'phones']

for word in words:
    print(lemmatizer.lemmatize(word))

Lemmatization generates different outputs for different Part Of Speech (POS) values. Some of the most common POS values are verb (v), noun (n), adjective (a), and adverb (r). The default POS value in lemmatization is a noun, so the printed values for the previous example will be article, friendship, study and phone.

Let’s change the POS value to verb (v).

from nltk import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()
words = ['be', 'is', 'are', 'were', 'was']

for word in words:
    print(lemmatizer.lemmatize(word, pos='v'))

In this case, Python will print the word be for all the values in the list.

Subscribe now

5. Bag of Words

The Bag of Words (BoW) model is a representation that turns text into fixed-length vectors. This helps us represent text into numbers so we can use it for machine learning models. The model doesn’t care about the word order, but it’s only concerned with the frequency of words in the text. It has applications in NLP, information retrieval from documents, and classifications of documents.

The typical BoW workflow involves cleaning raw text, tokenization, building a vocabulary, and generating vectors.

Python Implementation

Python’s library sklearn contains a tool called CountVectorizer that takes care of most of the BoW workflow.

Let’s use the following 2 sentences as examples.

Sentence 1: “I love writing code in Python. I love Python code”

Sentence 2: “I hate writing code in Java. I hate Java code”

Both sentences will be stored in a list named text. Then we’re going to create a dataframe df to store this text list. After this, we‘ll initiate an instance of CountVectorizer(cv), and then we’ll fit and transform the text data to obtain the numeric representation. This will be stored in a document-term matrix df_dtm.

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer

text = ["I love writing code in Python. I love Python code",
        "I hate writing code in Java. I hate Java code"]

df = pd.DataFrame({'review': ['review1', 'review2'], 'text':text})
cv = CountVectorizer(stop_words='english')
cv_matrix = cv.fit_transform(df['text'])
df_dtm = pd.DataFrame(cv_matrix.toarray(),
                      index=df['review'].values,
                      columns=cv.get_feature_names())
df_dtm

The BoW representation made with CountVectorizer stored in df_dtm looks like the picture below. Keep in mind that words with 2 letters or fewer are not taken into account by the CountVectorizer.

As you can see the numbers inside the matrix represent the number of times each word was mentioned in each review. Words like “love,” “hate,” and “code” have the same frequency (2) in this example.

Overall, we can say that CountVectorizer does a good job tokenizing text, building a vocabulary, and generating vectors.

6. Term Frequency–Inverse Document Frequency (TF-IDF)

Unlike the CountVectorizer, the TF-IDF computes “weights” that represent how relevant a word is to a document in a collection of documents (aka corpus). The TF-IDF value increases proportionally to the number of times a word appears in the document and is offset by the number of documents in the corpus that contain the word. Simply put, the higher the TF-IDF score, the rarer or unique or valuable the term and vice versa. It has applications in information retrieval like search engines that aim to deliver results that are most relevant to what you’re searching for.

Before we see the Python implementation, let’s see an example so you have an idea of how the TF and IDF are calculated. For the following example, we’ll use the same sentences used for the CountVectorizer example.

Sentence 1: “I love writing code in Python. I love Python code”

Sentence 2: “I hate writing code in Java. I hate Java code”

Term Frequency (TF)

There are different ways to define the term frequency. One suggests the raw count itself (i.e., what the Count Vectorizer does), but others suggest it’s the frequency of the word in the sentence divided by the total number of words in the sentence.

For this simple example, we’ll use the first criteria. The term frequency is shown in the following table.

As you can see, the values are the same as the ones calculated for the CountVectorizer before. Also, words with 2 letters or fewer are not taken into account.

Inverse Document Frequency (IDF)

The IDF is also calculated in different ways. Although standard textbook notation defines the IDF as idf(t) = log [ n / (df(t) + 1), the sklearn library we’ll use later in Python calculates the formula by default as follows.

Also, sklearn assumes natural logarithm ln instead of log and smoothing (smooth_idf=True). Let’s calculate the IDF values for each word as sklearn will do it.

TF-IDF

Once we have the TF and IDF values, we can obtain the TF-IDF by multiplying both values (TF-IDF = TF * IDF). The values are shown in the table below.

Python Implementation

Calculating the TF-IDF shown in the table above in Python requires a few lines of code thanks to the sklearn library.

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer

text = ["I love writing code in Python. I love Python code",
        "I hate writing code in Java. I hate Java code"]

df = pd.DataFrame({'review': ['review1', 'review2'], 'text':text})
tfidf = TfidfVectorizer(stop_words='english', norm=None)
tfidf_matrix = tfidf.fit_transform(df['text'])
df_dtm = pd.DataFrame(tfidf_matrix.toarray(),
                      index=df['review'].values,
                      columns=tfidf.get_feature_names())
df_dtm

The TF-IDF representation stored in df_dtm is presented below.

Note: By default TfidfVectorizer() uses l2 normalization, but to use the same formulas shown above we set norm=None as a parameter. For more details of the formulas used by default in sklearn and how you can customize it check its documentation.

Subscribe now

Bonus: Wordcloud

Wordcloud is a popular technique that helps us identify the keywords in a text. It’s not considered as a NLP technique but it still uses some of the techniques explained in this article.

In a wordcloud, more frequent words have a larger and bolder font, while less frequent words have smaller or thinner fonts. In Python, you can make simple wordclouds with the wordcloud library and nice-looking wordclouds with the stylecloudlibrary.

Below you can find the code to make a wordcloud in Python. I’m using a text file of a Steve Jobs speech.

import stylecloud

stylecloud.gen_stylecloud(file_path='SJ-Speech.txt',
                          icon_name= "fas fa-apple-alt")

This is the result of the code above.

Wordclouds are popular because they’re engaging, easy to understand, and easy to create.

You can take customization even further by changing the colors, removing stopwords, choosing your image, or even adding your own image as a mask of the wordcloud.

Share

Leave a comment

In the previous article of our Behind AI series, we explained in plain English what databases are and compared the top DBMS. Today, we’ll focus on a type of database that has become increasingly important in AI: vector databases.

The internet contains a huge amount of data in different forms. In the past, this data was mostly structured, but as the internet grew, unstructured data such as photos, audio, text, and video files became more common.

Analysts estimate that 80-90% of any organization’s data is unstructured, so how can we deal with this unstructured data?

Traditional relational databases and NoSQL databases struggle to analyze unstructured data especially when it comes to doing it in real-time. Here’s when vector databases can help. They were built to manage massive embeddings vectors converted from unstructured data.

In this article, we’ll learn more about vector databases, how they can help us manage unstructured data, and the solutions they offer.

Vector Databases vs Traditional Databases

Relational databases store items with pre-defined relationships between them. These databases typically organize data in tables with columns and rows. This is known as structured data and is generally easy to search and analyze. In contrast, unstructured data is more complex and requires more work to understand.

Machine learning and deep learning models can help us understand this unstructured data by transforming it into vector embeddings. These embeddings are high dimensional vectors that describe complex data as numerical values in different dimensions.

Vector databases are designed for search and data mining.

This search involves querying data as we’d do with a relational database. However, a relational database retrieves results that are an exact match, while a vector database offers more complex search capabilities.

One of the types of search that vector databases excel at is similarity search (aka vector search). Similarity search consists in finding the most similar item to the one we’re looking for. This task is called nearest-neighbor search because the similar items we get are actually near-neighbor matches. This list of items wouldn’t exist if we weren’t performing a similarity search.

To sum it up, traditional databases return results that are an exact match, while a vector database returns near-neighbor matches. In addition to that, vector databases offer good speed, accuracy, and flexibility.

This is great for product search. Say we want to buy black Nike shoes, so we go to our favorite online store and search “black Nike shoes.” If the search engine was built with similarity search functionalities, we’ll get similar items, in case there aren’t any black Nike shoes in the inventory.

This is just one of the applications that vector databases have. Let’s see some common use cases.

Vector Databases Use Cases

Recommendation Engines

Recommendation systems are everywhere. They’re used to suggest items similar to past videos, movies, purchases, and more.

Vector databases are great for building recommender systems. Its similarity search functionality makes vector databases a good option for suggesting relevant items to users.

We could use vector databases to build a recommender system that would suggest movies that a user might like based on the user’s historical movie rating data. We could also build a system that recommends products based on past purchases a customer made.

Semantic Search

Semantic search is a data searching technique that not only allows us to find keywords but seeks to understand natural language as a person would do.

How does it do that? It puts our search query into context.

We can use vector databases to index vector embeddings from NLP models in order to understand the context of the text. This provides more accurate search results.

If you remember all the things you search on Google, you’ll realize that we often use natural language in our queries. Semantic search is then necessary for a search engine to provide relevant results.

Similarity Search

Unstructured data such as images, video, and audio are hard to classify in relational databases.

There are some workarounds to deal with this using relational databases, but different people would use different criteria, which makes this more complicated.

In contrast, vector databases can analyze large-scale data in real-time. They’re behind image search technologies such as autonomous cars that are able to recognize objects and phones that can recognize the face of a user. They’re also present in audio search. This helps our devices identify the name of a song or a user’s voice.

A case study of a vector database

There is an emerging market for vector databases out there. Let’s take Milvus as an example and look at its features and advantages.

Milvus is an open-source vector database that was created with the purpose of storing, indexing, and managing embedding vectors generated by machine learning models.

Unlike relational databases that handle structured data using a pre-defined pattern, Milvus can handle embedding vectors converted from unstructured data.

Here’s why Milvus is a good option:

• It offers high performance when conducting vector search on massive datasets.

• Cloud scalability and high reliability even in the event of a disruption.

• Hybrid search achieved by pairing scalar filtering with vector similarity search.

Here’s the Milvus workflow.

Hi!

Data is an important resource in AI. Companies store data in different types of databases and today we’ll learn everything about them (in the next article, we’ll focus on a type of database that is specially used in AI, so stay tuned!)

No matter what your job is probably you’ve ever heard of the word database.

Companies out there use different types of databases to store all the information they collected throughout the years. Although all these databases might seem the same, they have some functionalities that make them more suitable for certain situations, so it’s worth learning more about them.

In this article, we’ll see what databases are and the most commonly used in companies.

Table of Contents
1. What is a Database?
2. Types of Databases
 - Relational databases
 - Non-Relational databases
3. Top databases

Subscribe now

What is a Database?

A database is a collection of data typically stored electronically in a computer system and controlled by a database management system (DBMS). The data, the DBMS, and the applications associated with them are referred to as a database system (or just “database”).

The words database management system and database are often used interchangeably, but technically they’re not the same.

To distinguish them consider the case of a social media app that stores different information about its users such as messages, photos, comments, etc. The database stores this big collection of data, but that’s pretty much what it does. If you want to edit, update, or delete data, you need a DBMS that does the talking for you. Some of the most popular DBMS are Oracle, MySQL, SQL Server, and PostgreSQL.

Again, people often refer to both the DBMS and DB simply as “database” but now you know how this actually works.

Another way to think of a database is as a big spreadsheet with many rows and columns. That’s a good comparison, but a database goes beyond that. Both the database and spreadsheet are good for storing information, but they mainly differ in the following aspects:

• How the data is stored and manipulated: Databases allow complex data manipulation, while spreadsheets aren't meant for users who need much data manipulation.

• Who can access the data: Databases allow multiple users to quickly access and query the data, while spreadsheets were designed only for a single user or a small number of users.

• The amount of data that can be stored: Databases are designed to store larger collections of data, while spreadsheets have a limitation.

Last but not least, a database cannot only store data in tables and rows. That’s how relational databases typically work, but there’s also another type of database called non-relational database. This leads us to our next point.

Subscribe now

Types of Databases

Databases are typically divided into relational and non-relational databases. Among the top 10 databases, you’ll see both relational and non-relational databases. One type of database is not better than the other, but they suit different needs.

Relational databases

A relational database (aka SQL database), stores data in tables and rows also referred to as records. This type of database links information from different tables through keys.

A key is a unique value in a table that is also known as the “primary key”. When this key is added to a record located in another table, it’s called “foreign key” in this second table. This connection between primary and foreign keys creates a relationship between records within both tables.

Some popular relational database management systems (RDBMS) are Oracle, MySQL, SQL Server, and PostgreSQL.

Here’s a basic schema that shows how a relational database works.

To query data in a RDBMS, we use Structured Querying Language (SQL). With SQL we can create new records, update them, and more. This makes the RDBMS good for apps that need transactional functionality, data mining, and complex reporting.

Non-Relational databases

A non-relational database (aka NoSQL database), stores data without tables, rows, or keys. In other words, a non-relational database stores data in a non-tabular form. This adds some flexibility and helps satisfy specific requirements of the type of data being stored.

You can think of a non-relational database as a collection of documents. A document can contain a lot of detailed information about a customer. Each customer can have different types of information, but they can be stored in the same document.

The ability to process and organize different types of information makes non-relational databases more flexible than relational databases.

There are four popular non-relational types: document data store, column-oriented database, key-value store, and graph database. One of the most popular NoSQL databases is MongoDB.

Subscribe now

What are the top databases?

It’s hard to rank the database based on their functionality because they suit different needs and can be more convenient in certain scenarios than in others. That said, it’s possible to rank the database management systems according to their popularity.

In fact, DB-Engine ranks DBMS by their current popularity. To do so, they calculate scores following different parameters.

Here are the top 10 databases by their popularity.

Now let’s see more about them, compare them, and see their pros and cons.

1. Oracle

Oracle Database is a widely used RDBMS across industries. In fact, it has the largest market share of around 30.2% in the RDBMS market.

Oracle Database supports SQL language to interact with the database. It’s considered one of the best databases because it supports all data types involving relational, graph, structured, and unstructured information. In addition to that, Oracle Database is preferred for its flexible standards, scalability, high availability, and strong security.

Pros

• It’s highly compatible with different apps and platforms

• Helps with scalability

• It offers good privacy and security

Cons

• The license is expensive

• Users might need extensive SQL knowledge to use Oracle Database

Popularity

Google Trends shows more interest in Oracle than in MySQL over the past 5 years. The graph also reveals the same ups and downs for both databases.

2. MySQL

MySQL is one of the most popular databases. It’s open-source so any person or company can use MySQL for free, but if the code needs to be integrated into a commercial application, you need to purchase a license. That said, this database is still worth it for anyone who wants to experiment with a friendly yet powerful database.

MySQL was developed by Oracle and it’s a relational database management system. As explained before, the relation model consists in organizing data in tables with rows and columns, while the relationship between elements follows a logical structure. Companies such as Facebook, Twitter, Wikipedia, and YouTube employ MySQL backends.

Pros

• It’s open source: Unlike other options, you don’t have to pay to use most features of MySQL

• It’s cross-platform: Runs on Linus, Solaris, and Windows and supports platforms with programming languages such as C, C++, Java, Python, etc.

• Reliable data security: MySQL is known for being a secure database management system. This is why so many well-known companies use it in their applications.

• It’s easy to use: Anyone can download, install and start to use MySQL in a few minutes.

Cons

• It’s not for large-sized data

• It doesn’t support SQL check constraints

• It doesn’t have a good debugging tool compared to paid databases

• It doesn’t handle transactions very efficiently

Popularity

Google Trends shows that the interest in MySQL has slightly decreased over the past 5 years, but has suddenly risen compared to other databases like SQL Server.

Subscribe now

3. SQL Server

SQL Server was developed by Microsoft and it’s considered a great RDBMS for both on-premise and cloud environments. It has a Database Engine component that allows storing, processing, and securing data. The database engine is divided into two segments — the relational and storage engine. The first is used to process commands and queries, while the second is used to manage features such as tables, pages, files, indexes, and transactions.

Besides SQL language, SQL Server also includes Transact-SQL (T-SQL), which is Microsoft’s extension to the SQL used to interact with relational databases. SQL Server is a good option for businesses that want to scale the performance, availability, and security seamlessly based on their requirements.

Pros

• It has various supported editions (enterprise, standard, express, and developer). The express SQL server edition is free of cost.

• It has an online documentation

• On-premise and cloud database support

• It offers different tools and apps

Cons

• Expensive enterprise edition

• It’s available for Windows, Linux, and macOS, but the steps to install it on a Mac aren’t as straightforward as on a Windows machine.

Popularity

Google Trends shows more interest over time in SQL Server than in PostgreSQL. In fact, the interest in PostgreSQL hasn’t changed that much over the past 5 years.

4. PostgreSQL

PostgreSQL is known as the world’s most advanced open source object-relational database management system (ORDBMS). Part of this reputation is due to its architecture, reliability, robustness, and extensibility.

PostgreSQL comes with many features that help build apps, protect data integrity and help manage data no matter how big or small the data is. It’s also highly extensible in many areas. To name a few:

• Stored functions and procedures

• PL/PGSQL, Perl, Python

• SQL/JSON path expressions

• Additional functionality such as PostGIS (spatial database extender for PostgreSQL)

These extensions help us process data right from PostgreSQL, so we don’t need to find workarounds to implement them.

Pros

• It’s extremely programmable: You can extend PostgreSQL thanks to its directory-based operation and dynamic loading

• It’s highly extensible

• It has a very rich set of indexing options

Cons

• Performance: PostgreSQL is sometimes less efficient than other RDBMS such as MySQL (at least for simple intensive reading operations)

• It might be difficult to troubleshoot PostgreSQL

Popularity

Google Trends shows similar interest over time in PostgreSQL and MongoDB. That said, we should consider that PostgreSQL was initially released in 1996, while MongoDB was released in 2009.

5. MongoDB

MongoDB is an open-source document database that uses a flexible schema for storing data. Unlike SQL databases that store data in tables of rows and columns, NoSQL database programs like MongoDB use JSON-like documents with optional schemas.

MongoDB is great for those who build internet and business applications and need to evolve and scale quickly. Some of the advantages of MongoDB for developers are the power of document-oriented databases (documents can be retrieved directly in JSON format, which developers find easy to work with), user experience, scalability and transactionality, and its thriving community.

Overall, MongoDB is good if you’re looking for a database that:

• Supports rapid iterative development

• Enables the scale to high levels of read and write traffic

• Stores, manages, and searches data when creating apps

Pros

• It offers a flexible schema that it’s not possible to get in a RDBMS

• Scalability: MongoDB uses sharding, which allows the database to use horizontal scalability.

• It’s free and supports Windows, macOS, and Linux

Cons

• High memory usage: The data size in MongoDB is higher than in other databases

• Less flexibility with querying: It fails to support joins as a relational database

Give a gift subscription

Hi!

I created a new series called “AI & Python” which will be focused on learning Python, coding concepts, automation, and creating AI apps. From now on, “Behind AI“ will be focused on what's behind the AI products you love. No programming knowledge will be required. Tech concepts (if any) will be explained in plain English.

Examples of a Behind AI piece are this article and the one you’re reading right now.

P.S. In case you’re not interested in the Python articles, go to settings and turn off notifications for “AI & Python” (leave the rest the same to keep receiving my other emails)

Machine learning (ML) is the field behind all the magic in AI products. If you’re new to ML, you probably must’ve heard of the words “algorithm” or “model” without knowing how they’re related to machine learning.

Here’s a brief explanation in plain English.

Machine learning algorithms are categorized as supervised or unsupervised. Supervised learning algorithms model the relationship between labeled input and output data (aka target). This model is then used to predict the label of new observations using new labeled input data. If the target variable is discrete, we’re dealing with a classification problem, while if the target variable is continuous we’re dealing with a regression problem. In contrast, unsupervised learning doesn’t rely on labeled input/output data but processes unlabeled data.

Here are 6 supervised learning algorithms that you should know.

Subscribe now

1. Linear Regression

Linear regression is the simplest algorithm used in machine learning. This algorithm is used for modeling the relationship between two or more variables. There are two types of linear regression — simple and multiple linear regression.

In simple linear regression, there’s one independent variable and one dependent variable, while in multiple linear regression there are multiple independent variables and one dependent variable.

Here’s the multiple linear regression equation:

where y is the dependent variable (target value), x1, x2, … xn the independent variables (predictors), b0 the intercept, b1, b2, ... bn the coefficients and n the number of observations.

In the picture below, you’ll see a simplified version of the linear regression equation.

As you can see in the picture above, there’s a linear relationship, so if one variable increases or decreases, the other variable will also increase or decrease.

We can use linear regression to predict scores, salaries, house prices, etc. That said, the prediction accuracy isn’t as good as those you’d get with other algorithms.

2. SVM

A Support Vector Machine (SVM) is a supervised learning algorithm that is mostly used in classification problems. We usually feed the SVM model with labeled training data to categorize new text.

SVM is a good choice when we have a limited number of samples and speed is a priority. This is why it’s used when we work with a dataset that has a few thousand of tagged samples in text classification.

To understand much better how SVM works let’s see an example.

In the picture below, we have two tags (green and yellow) and two features (x and y). Say we want to build a classifier that finds whether our text data is either green or yellow. If that’s the case, we will plot each observation (aka data point) in an n-dimensional space, where “n” is the number of features used.

We only have two features, so the observations are plotted in 2-dimensional space as shown in the picture below.

SVM takes the data points and makes a hyperplane that best separates the classes. Since the observations are plotted in 2-dimensional space, the hyperplane is a line.

This red line is also known as the decision boundary. The decision boundary determines whether a data point belongs to one class or to another. In our example, if the data point falls on the left side, it will be classified as green, while if it falls on the right side, it will be classified as yellow.

Subscribe now

3. Decision Tree

If you know nothing about machine learning, you might still know about decision trees.

A decision tree is a model used in planning, statistics, and machine learning that uses a tree-like structure of decisions/consequences to evaluate the possible events involved in a particular problem.

Here’s a decision tree that evaluates scenarios where people want to play football.

Each square is called a node. The last nodes of the decision tree are called the leaves of the tree. To make predictions we start from the root of the tree (first node). Each node in the decision tree will be evaluated. Then we follow the branch that agrees with the evaluation and jump to the next node.

The decision tree algorithm can be used for solving both regression and classification problems. We use a decision tree to build a model that can predict the class or value of the target variable by learning decision tree rules inferred from the training data.

4. Random Forest

Random forest is an ensemble of many decision trees. It combines the simplicity of a decision tree with flexibility resulting in an improvement in accuracy.

To make a random forest, first, we need to create a “bootstrapped” dataset. Bootstrapping is randomly selecting samples from original data (we can even choose the same sample more than once). Then, we use the bootstrapped dataset to create a decision tree.

This method is known as “bagging.” If we repeat the previous steps multiple times, we get a good number of trees. This variety of trees is what makes random forests more effective than a single decision tree.

If the random forest is used for a classification task, the model selects the mode of the predictions of each decision tree. For a regression task, the model selects the mean value of the results from the decision trees.

5. Naive Bayes

Naive Bayes is a supervised learning algorithm that uses conditional probability to predict a class.

The Naive Bayes algorithm is based on the Bayes theorem:

p(A|B): Probability of event A given event B has already occurred
p(B|A): Probability of event B given event A has already occurred
p(A): Probability of event A
p(B): Probability of event B

Naive Bayes assumes that every feature is independent of each other, which isn’t always the case, so we should examine our data before choosing this algorithm.

The assumption that features are independent of each other makes Naive Bayes fast compared to more complex algorithms; however, it also makes this algorithm less accurate.

We can use Naive Bayes to predict weather forecasting, fraud detection, and more.

Subscribe now

6. Logistic Regression

Logistic regression is a supervised learning algorithm that is commonly used for binary classification problems. This means we can use logistic regression to predict whether a customer will churn or not, and to find whether a mail is spam or not.

The logistic regression is based on the logistic function (aka the sigmoid function), which takes in a value and assigns a probability between 0 and 1.

Here’s the graph of the logistic regression:

To understand much better how Logistic Regression works, consider a scenario where we need to classify whether an email is spam or not.

In the graph, if Z goes to infinity, Y (our target value) will become 1, which means the email is spam. However, if Z goes to negative infinity, Y will become 0, which means the email is not spam.

The output value is a probability, so if we obtain a value of 0.64, this means that there’s a 64% chance that an email will be spam.

That’s it for now! Let me know in the comments if you want a 2nd part of this.

Share

Subscribe now

Hi!

The other day I was thinking that many people still don’t know the value of data, which is surprising given that data is what makes AI possible. Companies like OpenAI have been collecting data for many years to train their models, creating the tools we all know today.

In the coming articles, I’ll show you the techniques these companies use to collect data and what you can do with this data but, first, let’s see why data is so important nowadays.

The concept of data as a strategic asset has been gaining momentum in the past years, however, regular people aren’t able to see the real value in data.

We know big tech companies have been collecting data for a long time. We know that year after year new regulations about the use of data are created. That said, most of us still don’t understand the impact data has on our society.

A few years ago, The Economist published an article called “The world’s most valuable resource is no longer oil, but data.” However, for regular folks, it’s still hard to understand how data can be the new oil.

Data and oil have some similarities, but also some differences. Here are some of them.

Subscribe now

1. Data and oil need to be refined

Data and oil are rarely used in their raw state.

If oil is unrefined, it cannot be used. For oil to be useful, it has to be extracted, refined, and distributed. The same happens with data. We don’t use the data as soon as it’s extracted, but we have to process it first before it’s ready for analysis.

Here’s how Clive Humby, the data science entrepreneur who coined the phrase “data is the new oil,” compares oil and data.

“Data is the new oil. Like oil, data is valuable, but if unrefined, it cannot really be used. It has to be changed into gas, plastic, chemicals, etc. to create a valuable entity that drives profitable activity. So, must data be broken down, analysed for it to have value.”

This is true. Once data is collected, it needs to be cleaned and transformed to get it in the desired format. Why? Well, real-world data is messy, so there might be inaccurate or missing data that we need to deal with.

To put it simply, imagine you have collected data from a survey. You can be confident that the results obtained from the multiple-choice questions don’t need much preprocessing, but things change with the open-ended questions because people can answer whatever they want (sometimes without following a common pattern) and even leave an answer blank.

Real-world data is sometimes as messy as those open-ended questions.

This is why raw data isn’t enough. Only after the data is “refined” we can make the most of it by making reports, doing analysis, and creating something valuable.

2. Oil is a finite resource, while more and more data is created every day

One of the things that makes oil so valuable is the concept of scarcity. There might be undiscovered oil reserves out there, but, the thing is, oil is a finite resource. One day there won’t be any oil on this planet and we have to find some other forms of energy.

That doesn’t happen with data.

There isn’t only plenty of data possessed by companies and even publicly available on the Internet, but more and more data is being created by people every day. How? Every time you watch a movie on Netflix, buy a product on Amazon or listen to a song on Spotify, a new data point is created.

These data points are created every second around the world!

Thanks to millions of data points, big tech companies can develop a good recommender system that can predict what movie or song you might like or suggest products to buy based on purchase history.

In addition to that, unlike oil, data can be reused without losing its quality. One engineer can use a dataset for one purpose, while another can use the same dataset for a completely different purpose.

But if data is infinite, how can it be so valuable? The value depends on the eye of the beholder. A dataset about sports statistics can be utterly useless for an e-commerce company but can be extremely valuable for a football club.

3. Data and oil are not always available to everyone

Yes, data is infinite, but it’s not available to everyone.

There isn’t a company that would share data that probably took them years to collect (at least not for free). Something similar happens with data available on websites. The data is there and you could somehow extract it, but it’s protected by privacy guidelines and terms and conditions. This means that, although you can extract the data, you should think twice about how you use this data.

Let’s consider the HiQ and LinkedIn case as an example.

HiQ extracted publicly-available data from LinkedIn. LinkedIn invoked the CFAA in a cease-and-desist letter to HiQ. Although the US Court of Appeals denied LinkedIn’s request, this didn’t grant HiQ the freedom to use the data extracted for commercial purposes.

As you can see, there are ethical issues around data collection and they’re quite different from oil extraction.

Even companies that collect data from their customers can’t use it as they want. As an example, there’s the General Data Protection Regulation (GDPR) that imposes obligations on organizations that collect data related to people in the European Union. Those who violate its privacy and security standards could pay fines that reach tens of millions of euros.

Let’s sum up this oil and data comparison in a few sentences.

• Like oil, data needs to be refined. Otherwise, it can’t be used as it’s not so valuable.

• Unlike oil, data is an infinite resource that is created every day by people. It can even be reused and it rarely loses its quality.

• Data is everywhere, but it isn’t always available to everyone. You could extract data, but then you should think twice about how you use it.

How companies get all this data

Data serves as the foundation upon which AI systems are built and operate. Recommendation systems in products like YouTube, Amazon, and Netflix are great examples of apps that use large amounts of data. These companies have generated data from users’ interactions with their products. Every time we rate a product on Amazon, like a YouTube video, or dislike a Netflix movie, we’re giving the system information about our likes and dislikes. These are data points that we, the users, have created. Big datasets are created thanks to the millions of users these companies have.

But that’s only one of the ways companies get data. After all, not every company has millions of customers.

The other way is through web scraping or web crawling. OpenAI has been doing this for some time to collect public data all over the internet. Although OpenAI won’t confirm if it got its data through social media posts, copyrighted works, or anywhere else, it’s clear that the internet provided much of the training data for OpenAI’s GPT models.

Public data is all over the internet and it seems only big companies are making the most of them. That said, in this AI revolution, you don’t need to have a big company to benefit from public data. In the coming articles, we’ll see the techniques these companies use to collect data and the things you can do with data scraped from the internet.

If you’d like to learn how to scrape data from the internet, you can take my Web Scraping Course in Python (or redeem it for free if you’re a paid annual subscriber). To take the course you only need to know the basic concepts of Python, which you can learn on this 1-hour crash course (video at the end of the article).

Subscribe now