As we mentioned in the last post, we will now explore linear algebra and its use in AI. Here, we are not going to cover the full university level linear algebra instead, we will focus on the parts that are actually used in machine learning and deep learning. In AI, data is represented as vectors and matrices, and neural networks operate on these structures. In this guide, we will cover these concepts minimally and with examples, so everyone can understand them. These topics are very broad, but our goal is to provide a simple and practical understanding of the parts that matter most in AI.
Scaler
A scalar is the simplest type of data in linear algebra. it is just a single number. You can think of it as a point on a number line. In AI, scalars are everywhere, they represent individual values like a bias, a weight, or the loss of a model. Scalars are also used for simple constants that do not change during a calculation. For example, a value like temperature = 37.5 or speed = 10 is a scalar. Even though scalars are simple, they are the building blocks for more complex structures like vectors, matrices, and tensors in AI computations.
Vector
A vector is not like a scalar, it has two perspectives, in mathematics and in AI, which often causes confusion.
In Mathematics:
A vector is a list of numbers that represents a point or an arrow in space. The number of components determines its geometric dimension. For example, [5, 3, 1] has three components, so it is a 3-dimensional (3D) vector in space. Vectors in math are used to describe positions, directions, or magnitudes in 2D, 3D, or higher-dimensional space.
In AI / Machine Learning:
In AI, a vector is also a list of numbers, but we usually treat it as a 1-dimensional tensor. Each number represents a feature of a data point. For example, a point in 2D space (x=2, y=3) can be stored as [2, 3], which is a vector of shape (2,). In machine learning, each row of a dataset is often a vector of features, and in natural language processing (NLP), words are converted into vectors (embeddings) so that computers can process and compare their meaning.
Key Point:
Many beginners get confused because they assume vectors in AI work the same way as in math. In AI, vectors are primarily data structures used for computation, not geometric arrows in space. Understanding this difference is crucial for working with machine learning and deep learning models.
Matrices
A matrix is a collection of numbers arranged in a 2 dimensional grid, with rows and columns. You can think of a matrix like a table, a spreadsheet, or a collection of multiple vectors stacked together. For example, a 3×3 matrix has 3 rows and 3 columns. In AI, matrices are extremely common because most datasets and model parameters can be represented this way. A dataset with many samples can be stored as a matrix where each row is a sample and each column is a feature. The weights in neural networks are also stored as matrices, and even images can be represented as matrices of pixel values. For example, if we have 1000 grayscale images where each image is 28×28 pixels, we can reshape them into a matrix with shape (1000, 784) so that AI models can work with them.
Tensors
A tensor is a generalization of scalars, vectors, and matrices to higher dimensions. If a scalar is a single number (0D), a vector is a list of numbers (1D), and a matrix is a 2D grid of numbers, then a tensor can extend to 3D, 4D, or even higher. Tensors are important in deep learning because they allow us to represent complex data. For example, an image can be stored as a 3D tensor with height, width, and color channels. A video can be stored as a 4D tensor where each frame is like an image in sequence. Deep learning frameworks like PyTorch and TensorFlow use tensors as their main data structure because models need to operate on multi-dimensional data efficiently.
| Concept | In Math / Geometry | In AI / Machine Learning |
|---|---|---|
| Scalar | A single number. Example: 5 → a point on a number line | A single number. Example: bias = 0.5 or loss = 0.01 |
| Vector | An arrow in space with magnitude and direction. [5, 3, 1] → 3D vector | A 1D list of numbers representing features. [5, 3, 1] → shape (3,) |
| Matrix | A 2D grid of numbers (rows × columns). Can represent transformations. Example: rotation matrix | 2D array of numbers. Example: dataset with samples as rows, features as columns. Shape (num_samples, num_features) |
| Tensor | Generalization of matrices to higher dimensions (less common in basic math). | Multi-dimensional array. Example: image → 3D tensor (H, W, C); video → 4D (Frames, H, W, C) |
| Dimension / Rank | Number of geometric axes a vector exists in. [5,3,1] → 3D vector | Number of axes a tensor has. [5,3,1] → 1D tensor with shape (3,) |
| Shape | Usually not mentioned in basic math | Shape tells how many numbers exist in each axis. Example: matrix [[1,2],[3,4]] → shape (2,2) |
Now that we’ve discussed the theory, let’s look at some example calculations to deepen understanding and provide additional practical knowledge.
1️⃣ Vector Addition / Subtraction
- Addition: Add corresponding elements of two vectors
- Subtraction: Subtract corresponding elements
v1 = [2, 3, 5]
v2 = [1, 4, 2]
v1 + v2 = [2+1, 3+4, 5+2] = [3, 7, 7]
v1 = [7, 5, 3]
v2 = [2, 4, 1]
v1 - v2 = [7-2, 5-4, 3-1] = [5, 1, 2]
2️⃣ Matrix Addition / Subtraction
- Addition/Subtraction: Add/subtract corresponding elements of two matrices of the same size
A = [[1, 2],[3, 4]]
B = [[5, 6],[7, 8]]
A + B = [[1+5, 2+6],[3+7, 4+8]]
= [[6, 8],[10, 12]]
A = [[9, 4],[2, 7]]
B = [[3, 1],[5, 2]]
A - B = [[9-3, 4-1],[2-5, 7-2]]
= [[6, 3],[-3, 5]]
3️⃣ Scalar Multiplication (Vector / Matrix)
- Multiply each element of a vector or matrix by a scalar (single number)
v = [2, 5, 3]
scalar = 3
scalar * v = [3*2, 3*5, 3*3] = [6, 15, 9]
A = [[1, 2],[3, 4]]
scalar = 2
scalar * A = [[2*1, 2*2],[2*3, 2*4]]
= [[2, 4],[6, 8]]Today we covered a lot of foundational concepts in linear algebra scalars, vectors, matrices, tensors, shapes, and basic operations. To strengthen your understanding and build confidence, it’s important to practice regularly. Here are some beginner-friendly resources with clear explanations and examples:
- Khan Academy – Linear Algebra Basics https://www.khanacademy.org/math/linear-algebra - Learn scalars, vectors, and matrices with interactive exercises.
- 3Blue1Brown – Essence of Linear Algebra (YouTube) https://www.youtube.com/playlist?list=PLZHQObOWTQDNPOjrT6KVlfJuKtYTftq3e - Visual, intuitive explanations of vectors, matrices, and transformations.
- GeeksforGeeks – Linear Algebra for Machine Learning https://www.geeksforgeeks.org/linear-algebra-for-machine-learning/ - Clear examples of operations like vector addition, matrix multiplication, and scalar multiplication.
- Towards Data Science – Beginner’s Guide https://towardsdatascience.com/linear-algebra-for-machine-learning-7f3c5c3efc44 - Short explanations with Python examples, perfect for AI beginners.