Understanding Distance Metrics Used in Machine Learning

Clustering is an important part of data cleaning used in artificial intelligence, deep learning, and data science. Today, we will discuss distance metrics, the backbone of clustering. Distance metrics deal with finding the proximity or distance between data points and determining if they can be clustered together. In this article, we will walk through 4 types of distance metrics in machine learning and understand how they work in Python.

Objectives

• Understand the use cases of various distance metrics in machine learning.
• Gain an understanding about the different types of distance metrics in machine learning.
• Learn about the importance of distance metrics in data mining.

What Are Distance Metrics?

Distance metrics are a key part of several machine learning algorithms. They are used in supervised and unsupervised learning to calculate the similarity between data points. An effective distance metric improves the performance of our machine learning model, whether that’s for classification tasks or clustering.

It would be best if you created clusters using a clustering algorithm such as K-Means Clustering or k-nearest neighbour algorithm (knn), which uses nearest neighbours to solve a classification or regression problem. How will you define the similarity between different observations? How can we say that two points are similar to each other? This will happen if their features are similar, right? When we plot these points, they will be closer to each other by distance.

Hence, we can calculate the distance between points and then define their similarity. Here’s the million-dollar question: How do we calculate this distance, and what are the different distance metrics in machine learning? Also, are these metrics different for different learning problems? Do we use any special theorem for this? These are all questions we are going to answer in this article.

Types of Distance Metrics in Machine Learning

Let us now look at different types of distance metrics in machine learning.

1. Euclidean Distance
2. Manhattan Distance
3. Minkowski Distance
4. Hamming Distance

Let’s start with the most commonly used distance measures in machine learning – Euclidean Distance.

Euclidean Distance

Euclidean Distance represents the shortest distance between two vectors. It is the square root of the sum of squares of differences between corresponding elements.

The Euclidean distance metric corresponds to the L2-norm of a difference between vectors and vector spaces. The cosine similarity is proportional to the dot product of two vectors and inversely proportional to the product of their magnitudes.

Most machine learning algorithms, including K-Means use this distance metric to measure the similarity between observations. Let’s say we have two points, as shown below:

So, the Euclidean Distance between these two points, A and B, will be:

Formula for Euclidean Distance

We use this formula when we are dealing with 2 dimensions. We can generalize this for an n-dimensional space as:

Where,

• n = number of dimensions
• pi, qi = data points

Let’s code Euclidean Distance in Python. This will help you better understand how this distance metric works.

We will first import the required libraries. I will be using the SciPy library that contains pre-written codes for most of the distance functions used in Python:

These are the two sample points that we will be using to calculate the different distance functions. Let’s now calculate the Euclidean Distance between these two points:

Python Code

# importing the library
from scipy.spatial import distance

# defining the points
point_1 = (1, 2, 3)
point_2 = (4, 5, 6)
point_1, point_2

# computing the euclidean distance
euclidean_distance = distance.euclidean(point_1, point_2)
print('Euclidean Distance b/w', point_1, 'and', point_2, 'is: ', euclidean_distance)

This is how we can calculate the Euclidean Distance between two points in Python. Let’s now understand the second distance metric, Manhattan Distance.

Manhattan Distance

Manhattan Distance is the sum of absolute differences between points across all the dimensions.

We can represent Manhattan Distance as:

Formula for Manhattan Distance

Since the above representation is 2 dimensional, to calculate Manhattan Distance, we will take the sum of absolute distances in both the x and y directions. So, the Manhattan distance in a 2-dimensional space is given as:

And the generalized formula for an n-dimensional space is given as:

Where,

• n = number of dimensions
• pi, qi = data points

Now, we will calculate the Manhattan Distance between the two points:

# computing the manhattan distance
manhattan_distance = distance.cityblock(point_1, point_2)
print('Manhattan Distance b/w', point_1, 'and', point_2, 'is: ', manhattan_distance)

Note that Manhattan Distance is also known as city block distance. SciPy has a cityblock function that returns the Manhattan Distance between two points.

Let’s now look at the next distance metric – Minkowski Distance.

Minkowski Distance

Minkowski Distance is the generalized form of Euclidean and Manhattan Distance.

Formula for Minkowski Distance

Here, p represents the order of the norm. Let’s calculate the Minkowski Distance formula of order 3:

# computing the minkowski distance
minkowski_distance = distance.minkowski(point_1, point_2, p=3)
print('Minkowski Distance b/w', point_1, 'and', point_2, 'is: ', minkowski_distance)

The p parameter of the Minkowski Distance metric of SciPy represents the order of the norm. When the order(p) is 1, it will represent Manhattan Distance and when the order in the above formula is 2, it will represent Euclidean Distance.

Python Code

# minkowski and manhattan distance
minkowski_distance_order_1 = distance.minkowski(point_1, point_2, p=1)
print('Minkowski Distance of order 1:',minkowski_distance_order_1, '\nManhattan Distance: ',manhattan_distance)

Here, you can see that when the order is 1, both Minkowski and Manhattan Distance are the same. Let’s verify the Euclidean Distance as well:

# minkowski and euclidean distance
minkowski_distance_order_2 = distance.minkowski(point_1, point_2, p=2)
print('Minkowski Distance of order 2:',minkowski_distance_order_2, '\nEuclidean Distance: ',euclidean_distance)

When the order is 2, we see that Minkowski and Euclidean distances are the same.

So far, we have covered the distance metrics used when dealing with continuous or numerical variables. But what if we have categorical variables? How can we decide the similarity between categorical variables? This is where we can use another distance metric called Hamming Distance.

Hamming Distance

In machine learning, the hamming distance measures the similarity between two strings of the same length. It is the number of positions at which the corresponding characters are different.

Let’s understand the concept using an example. Let’s say we have two strings:

“euclidean” and “manhattan”

Since the length of these strings is equal, we can calculate the Hamming Distance. We will match the strings character by character. The first character of both strings (e and m, respectively) differs. Similarly, the second character of both strings (u and a) is different, and so on.

Look carefully – seven characters are different, whereas two characters (the last two characters) are similar:

Hence, the hamming distance here will be 7. Note that the larger the hamming distance between two strings, the more dissimilar those strings will be (and vice versa).

Python Code

Let’s see how we can compute the Hamming Distance of two strings in Python. First, we’ll define two strings that we will be using:

# defining two strings
string_1 = 'euclidean'
string_2 = 'manhattan'

These are the two strings “Euclidean” and “Manhattan”, which we have seen in the example. Let’s now calculate the Hamming distance between these two strings:

# computing the hamming distance
hamming_distance = distance.hamming(list(string_1), list(string_2))*len(string_1)
print('Hamming Distance b/w', string_1, 'and', string_2, 'is: ', hamming_distance)

As the example above shows, the Hamming Distance between “Euclidean” and “Manhattan” is 7. We also saw that the Hamming Distance only works when we have strings of the same length.

Let’s see what happens when we have strings of different lengths:

# strings of different shapes
new_string_1 = 'data'
new_string_2 = 'science'
len(new_string_1), len(new_string_2)

You can see that the lengths of both the strings are different. Let’s see what will happen when we try to calculate the Hamming Distance between these two strings:

# computing the hamming distance
hamming_distance = distance.hamming(list(new_string_1), list(new_string_2))

This throws an error saying that the lengths of the arrays must be the same. Hence, Hamming distance only works when we have strings or arrays of the same length.

These are some of the most commonly used similarity measures or distance matrices in Machine Learning.

Checkout this article about How KNN Uses Distance Measures

Conclusion

Distance metrics are a key part of several machine learning algorithms. They are used in supervised and unsupervised learning to calculate the similarity between data points. Therefore, understanding distance measures in machine learning is more important than you realize. Take k-NN, for example—a technique often used for supervised learning. By default, it often uses Euclidean distance, a great distance measure in machine learning, for clustering.

By grasping the concept of distance metrics and their mathematical properties, data scientists can make informed decisions in selecting the appropriate metric for their specific problem. Our BlackBelt program provides comprehensive training in machine learning concepts, including distance metrics, empowering learners to become proficient in this crucial aspect of data science. Enroll in our BlackBelt program today to enhance your skills and take your data science expertise to the next level.

Key Takeaways

• Distance metrics are used in supervised and unsupervised learning to calculate similarity in data points.
• Distance measures in machine learning improve performance, whether for classification tasks or clustering.
• The four types of distance metrics in machine learning are Euclidean Distance, Manhattan Distance, Minkowski Distance, and Hamming Distance.

Frequently Asked Questions

Pulkit Sharma

My research interests lies in the field of Machine Learning and Deep Learning. Possess an enthusiasm for learning new skills and technologies.