Compute vector and matrix norm using NumPy norm

In linear algebra, a norm is a function that assigns a strictly positive length or size to each vector in a vector space. It’s a measure of the magnitude of the vector.

Mathematically, for a vector x, the p-norm is defined as ||x||p = (sum |xi|^p)^(1/p).

In this tutorial, we’ll use numpy.linalg.norm() function, which is a key function used to compute norms in Python.

 

 

Syntax and parameters of numpy.linalg.norm()

The function used to compute the norm in NumPy is numpy.linalg.norm(). Here is its syntax:

numpy.linalg.norm(x, ord=None, axis=None, keepdims=False)

The parameters are as follows:

• x: Input array. It could be a vector or a matrix.
• ord: Order of the norm. It could be any positive number, np.inf, -np.inf, 0, 1, or 2.
• axis: Axis or axes along which the norm is computed. If this is a tuple of ints, the norm is computed on multiple dimensions.
• keepdims: If this is set to True, the axes which are normed over are left in the result as dimensions with size one.

Here’s an example of its use:

import numpy as np

# Define a vector
vector = np.array([1, -2, 3, -4, 5])

# Compute L1 norm
l1_norm = np.linalg.norm(vector, ord=1)

print("L1 Norm: ", l1_norm)

Output:

L1 Norm: 15.0

In the code above, we define a vector and calculate its L1 norm using numpy.linalg.norm(). We pass the vector and ord=1 to calculate the L1 norm, and the result is 15.0.

 

Normalization vs taking the norm

Normalization scales the data so that it falls within a specific range, typically between 0 and 1.

Taking the norm of a vector computes the magnitude or length of the vector in the vector space.
Here’s an example to illustrate the difference:

import numpy as np
vector = np.array([1, 2, 3, 4, 5])

# Normalize the vector
normalized_vector = vector / np.linalg.norm(vector)

# Compute the l2 norm (Euclidean distance)
l2_norm = np.linalg.norm(vector)

print("Normalized Vector: ", normalized_vector)
print("L2 Norm: ", l2_norm)

Output:

Normalized Vector:  [0.13483997 0.26967994 0.40451992 0.53935989 0.67419986]
L2 Norm:  7.416198487095663

In this code, we first define a vector, then normalize it by dividing every element in the vector by the L2 norm of the vector.

We also compute the L2 norm of the original vector. As we can see, the normalized vector has its data rescaled between 0 and 1, while the L2 norm outputs a single number that represents the length or magnitude of the original vector.

 

Types of vector norms

There are different types of vector norms, including:

1. L1 Norm: Also known as Manhattan distance, it is the sum of the absolute values of the elements in the vector.
2. L2 Norm: Also known as Euclidean distance, it is the square root of the sum of the square of each element in the vector.
3. Infinity Norm: This norm gives the maximum absolute value among the elements of the vector.
4. Negative Infinity Norm: This norm gives the minimum absolute value among the elements of the vector.

Let’s computer each type of these norms.

 

L1 norm (Manhattan distance)

The L1 norm, also known as Manhattan distance, is computed as the sum of the absolute values of the elements in the vector.

It’s particularly useful when differences of absolute values matter more. Here’s how you can compute the L1 norm:

import numpy as np
vector = np.array([1, -2, 3, -4, 5])

# Compute L1 norm
l1_norm = np.linalg.norm(vector, ord=1)
print("L1 Norm: ", l1_norm)

Output:

L1 Norm:  15.0

In the code above, we first define a vector. Then we compute the L1 norm of the vector using np.linalg.norm() with ord=1. The result is 15.0, which is the sum of the absolute values of the elements in the vector.

 

L2 norm (Euclidean distance)

The L2 norm, also known as the Euclidean distance, is the square root of the sum of the squares of the elements in the vector.

It’s commonly used to measure the distance between points in a space. Here’s how you can compute the L2 norm:

import numpy as np
vector = np.array([1, -2, 3, -4, 5])

# Compute L2 norm
l2_norm = np.linalg.norm(vector, ord=2)
print("L2 Norm: ", l2_norm)

Output:

L2 Norm:  7.416198487095663

In the code above, we first define a vector. We then compute the L2 norm of the vector using np.linalg.norm() with ord=2.

The result is 7.416198487095663, which is the square root of the sum of the squares of the elements in the vector.

 

Infinity norm (Maximum norm)

The Infinity norm, also known as the Maximum norm, returns the maximum absolute value among the elements of the vector.

Here’s how you can compute the Infinity norm:

import numpy as np
vector = np.array([1, -2, 3, -4, 5])

# Compute Infinity norm
infinity_norm = np.linalg.norm(vector, ord=np.inf)
print("Infinity Norm: ", infinity_norm)

Output:

Infinity Norm:  5.0

In the code above, we first define a vector. We then compute the Infinity norm of the vector using np.linalg.norm() with ord=np.inf.

The result is 5.0, which is the maximum absolute value among the elements of the vector.

 

Negative Infinity norm (Minimum norm)

The Negative Infinity norm, also known as the Minimum norm, returns the minimum absolute value among the elements of the vector.

Here’s how you can compute the Negative Infinity norm:

import numpy as np
vector = np.array([1, -2, 3, -4, 5])

# Compute Negative Infinity norm
negative_infinity_norm = np.linalg.norm(vector, ord=-np.inf)
print("Negative Infinity Norm: ", negative_infinity_norm)

Output:

Negative Infinity Norm:  1.0

In the code above, we first define a vector. We then compute the Negative Infinity norm of the vector using np.linalg.norm() with ord=-np.inf.

The result is 1.0, which is the minimum absolute value among the elements of the vector.

 

Computing other p-norms

Besides the L1, L2, Infinity, and Negative Infinity norms, you can compute any other p-norm using numpy.linalg.norm().

The p-norm is defined as (sum |x_i|^p)^(1/p).
Here’s an example where we compute the 3-norm of a vector:

import numpy as np
vector = np.array([1, -2, 3, -4, 5])

# Compute 3-norm
p3_norm = np.linalg.norm(vector, ord=3)
print("3-Norm: ", p3_norm)

Output:

3-Norm:  6.082201995573399

In the code above, we first define a vector. We then compute the 3-norm of the vector using np.linalg.norm() with ord=3.

The result is 6.082201995573399, which is the cube root of the sum of the cubes of the absolute values of the elements in the vector.

 

Explanation of matrix norms

Matrix norms are extensions of vector norms. For matrices, we have several types of norms. The most popular ones are:

1. Frobenius Norm: The square root of the sum of the absolute squares of its elements, similar to L2 norm for vectors.
2. Spectral Norm (2-Norm): The largest singular value of the matrix (i.e., the square root of the largest eigenvalue of the matrix’s Hermitian).
3. 1-Norm: The maximum absolute column sum of the matrix.
4. ∞-Norm: The maximum absolute row sum of the matrix.

Matrix norms can be computed using the same numpy.linalg.norm() function as vector norms, but we must input a 2-dimensional array instead of a 1-dimensional array.

Let’s compute these matrix norms in the next sections.

 

Frobenius norm

Here’s how you can compute the Frobenius norm:

import numpy as np
matrix = np.array([[1, -2, 3], [-4, 5, -6], [7, -8, 9]])

# Compute Frobenius norm
frobenius_norm = np.linalg.norm(matrix, ord='fro')
print("Frobenius Norm: ", frobenius_norm)

Output:

Frobenius Norm:  16.881943016134134

In the code above, we first define a matrix. We then compute the Frobenius norm of the matrix using np.linalg.norm() with ord='fro'.

The result is 16.881943016134134, which is the square root of the sum of the squares of the elements in the matrix.

 

Spectral norm (largest singular value)

Here’s how you can compute the Spectral norm:

import numpy as np

# Define a matrix
matrix = np.array([[1, -2, 3], [-4, 5, -6], [7, -8, 9]])

# Compute Spectral norm
spectral_norm = np.linalg.norm(matrix, ord=2)

print("Spectral Norm: ", spectral_norm)

Output:

Spectral Norm:  16.84810335261421

In the code above, we first define a matrix. We then compute the Spectral norm of the matrix using np.linalg.norm() with ord=2.

The result is 16.84810335261421, which is the largest singular value of the matrix.

 

Computing other induced norms (1-norm, ∞-norm)

Here’s how you can compute 1-norm and ∞-norm norms:

import numpy as np
matrix = np.array([[1, -2, 3], [-4, 5, -6], [7, -8, 9]])

# Compute 1-Norm
one_norm = np.linalg.norm(matrix, ord=1)

# Compute ∞-Norm
inf_norm = np.linalg.norm(matrix, ord=np.inf)

print("1-Norm: ", one_norm)
print("∞-Norm: ", inf_norm)

Output:

1-Norm: 18.0
∞-Norm: 24.0

In the code above, we first define a matrix. We then compute the 1-Norm and ∞-Norm of the matrix using np.linalg.norm() with ord=1 and ord=np.inf respectively.

The 1-Norm result is 18.0, which is the maximum absolute column sum of the matrix.

The ∞-Norm result is 24.0, which is the maximum absolute row sum of the matrix.

 

Performance Comparison

To compare the performance, let’s calculate the L1 norm and L2 norm for a large vector and measure the computation time:

import numpy as np
import time

# Create a large vector
vector = np.random.rand(1000000)

# Compute L1 norm and measure time
start_time = time.time()
l1_norm = np.linalg.norm(vector, ord=1)
end_time = time.time()

print("L1 norm: ", l1_norm)
print("Time for L1 norm: %s seconds" % (end_time - start_time))

# Compute L2 norm and measure time
start_time = time.time()
l2_norm = np.linalg.norm(vector, ord=2)
end_time = time.time()

print("L2 norm: ", l2_norm)
print("Time for L2 norm: %s seconds" % (end_time - start_time))

Note: The exact output will vary each time you run it because we are creating a random vector:

L1 norm: 500205.0234115845
Time for L1 norm: 0.006276130676269531 seconds
L2 norm: 577.4774120713894
Time for L2 norm: 0.00099945068359375 seconds

In this case, computing the L2 norm was faster than computing the L1 norm.

 

Common mistakes while using numpy.linalg.norm()

Mistaking the default norm: By default, numpy.linalg.norm() computes the Frobenius norm for matrices and the 2-norm (Euclidean distance) for vectors.

Always specify the ord parameter if you want a different norm.

# Wrong usage for L1 norm
vector = np.array([1, -2, 3, -4, 5])
wrong_l1_norm = np.linalg.norm(vector) # This computes L2 norm, not L1
print("Wrong L1 Norm: ", wrong_l1_norm)

# Correct usage for L1 norm
correct_l1_norm = np.linalg.norm(vector, ord=1)
print("Correct L1 Norm: ", correct_l1_norm)

Output:

Wrong L1 Norm:  7.416198487095663
Correct L1 Norm:  15.0

Mistaking matrix norms for vector norms: If you’re dealing with matrices, ensure you’re using a matrix norm.

For example, using ord=1 with a matrix will compute the maximum absolute column sum, not the sum of absolute values of all elements.

# Wrong usage for matrix
matrix = np.array([[1, -2, 3], [-4, 5, -6], [7, -8, 9]])
wrong_l1_norm = np.linalg.norm(matrix, ord=1) # This computes max column sum, not sum of absolute values
print("Wrong L1 Norm: ", wrong_l1_norm)

# Correct usage for matrix
correct_fro_norm = np.linalg.norm(matrix, ord='fro') # Frobenius norm gives the square root of sum of squares of all elements
print("Correct Frobenius Norm: ", correct_fro_norm)

Output:

Wrong L1 Norm:  18.0
Correct Frobenius Norm:  16.881943016134134

Not considering negative infinity norm: The Negative Infinity norm returns the smallest absolute value in the array, not the smallest value itself.

This means negative numbers will be made positive before the smallest is selected.

vector = np.array([1, -2, 3, -4, 5])
neg_inf_norm = np.linalg.norm(vector, ord=-np.inf)
print("Negative Infinity Norm: ", neg_inf_norm)

Output:

Negative Infinity Norm:  1.0

 

Real-world applications for numpy.linalg.norm()

Norms play an integral role in various real-world applications, particularly in machine learning and data science.

Here’s an example of how we can use L2 norm in Natural Language Processing (NLP) for text vector normalization.

In this example, we’ll use the TfidfVectorizer from sklearn, which transforms text to feature vectors that can be used as input to a machine learning model.

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

# Sample corpus
corpus = [
    'This is the first document.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?',
]

vectorizer = TfidfVectorizer()

# Fit and transform the corpus
X = vectorizer.fit_transform(corpus)

# Get the first document's vector
first_vector = X[0].toarray()

# Before normalization
print("Before normalization: ", first_vector)

# Compute L2 norm
l2_norm = np.linalg.norm(first_vector, ord=2)

# After normalization
normalized_vector = first_vector / l2_norm
print("After normalization: ", normalized_vector)

Output:

Before normalization:  [[0.         0.46979139 0.58028582 0.38408524 0.         0.
  0.38408524 0.         0.38408524]]
After normalization:  [[0.         0.46979139 0.58028582 0.38408524 0.         0.
  0.38408524 0.         0.38408524]]

 

In the code above, we first define a corpus of four documents. We then initialize a TfidfVectorizer, which converts a collection of raw documents to a matrix of TF-IDF features.

We use the vectorizer to fit and transform our corpus. We then select the first document’s vector.
Before normalization, the vector’s values can range widely, depending on the length of the document and the frequency of each word.

After we compute the L2 norm of the vector and normalize the vector by dividing it by the L2 norm, the resulting vector has a length (or magnitude) of 1.

 

Further Reading

https://numpy.org/doc/stable/reference/generated/numpy.linalg.norm.html