Tech Tuesday

Introduction

Continuing my reflection on the Vector Database learning that I did back in 2025. I’m feeling the need to review everything before I move forward on my current research project. I wrote an article two weeks ago or so – on June 27, 2026, titled “Reviewing My Generative AI & Vector Database Notes”.

In that article I started why I wanted to start looking into vector databases. I wanted to learn the working of A.I. and thought a good place to start – in my hands on fashion – was to create an AI chatbot from scratch. In the process of my learning I came across a wonderful step-by-step article, titled: “9 components you will need to build your own custom AI Chat Bot”. One of the steps/components was “vector database”.

With my love for Linear Algebra, vector databases piqued my interest.

In this 2023 article it is stated that:

 “This [vector database] is a new technology that is essential to the chat bot stack. Vector databases are optimized for storing and retrieving vectors. 

Vectors are how embeddings are stored. And embeddings are the numerical representation of your text data, which is how the AI understands the words that are given to it.”

It also mentions two of the most popular Vector databases: Pinecone, and Chroma.

I have only briefly looked into these databases but first I wanted to understand the process. What “embeddings” mean, etc. That is where I focused my attention.

Again an overview:

The chart below shows where Chatbot sits in the whole Artificial Intelligence picture. Then within Chatbot is the Vector Database component and the steps taken to search and retrieve data.

The general steps within the Vector Database are as follows:

1. Take some unstructured data

2. Use a Transformer Model to transform it into a vector embedding, which is a ‘numerical’ representation of that data

3. Add it to the vector database

4. Then you can do a ‘similarity search’ on it

To go into detail on each of these steps I took a close look at the diagram & steps above..

Vector Database Overview

Traditional databases store structured data, in rows and columns. Structured data is data organized in a predefined format. Thinking in terms of an Excel spreadsheet, your username can be stored in the first box (row1,column1) and its format is character; your password can be stored in the second box (row1, column2) and its format is varchar (variable character), etc. 

A vector database, on the other hand, takes a massive dataset of unstructured data, information that doesn't have a set format or structure, and allows you to store, index, and search it. This is done by leveraging embeddings from machine learning models. 

1. Unstructured data can be data like text, documents,  audio files, video files, images, and emails. This data can be taken as a whole. 

2. With machine learning models, such as Image Transformer, Audio Transformer, and NLP Text Transformer, an embedding (also known as a vector) is created. Below are a few transformers that can be used:

   1. HuggingFace - sentence transformers

   2. Img2Vec - image transformers

   3. Facebook Kats - time series data transformer

3. A vector (or embedding) is a mathematical entity with both magnitude and direction. In terms of data, vectors are used to represent various types of information. This vector is then stored in the Vector Database. Below are a few Vector Databases, ones that I have heard about (in no particular order)

   1. Milvus

   2. Pinecone

   3. Qdrant 

   4. MongoDB Atlas

4. Now Similarity Search can be performed

This is a big-picture view of Vector databases. I previously downloaded a few Vector Databases and will download any open-source models/transformers. Also, it looks like SQL can be used with Vector Databases, but I'm not sure about that. As I continue on my journey I will be documenting what I learn. They say that "the best way to learn is to teach" so these articles are to help me learn and I hope others can get something from them as well. 😊

Detailed Vector Database Process

Overview of a basic Vector Database workflow is to take the unstructured dataset (text, image, video, audio, etc) and embed it (turning it into a vector). This embedding (vector) is used in the Vector Database for a Similarity Search. 

Vectors are used because most machine learning algorithms, including neural networks, cannot process plain text in its raw form. Once the similar vector(s) is located… it’s returned and then run through a language model to convert it back to text. 

I think the steps are different for audio, test, and video. So, I decided to start with ‘text embedding’ and will use the “Does snow absorb sound?” text. 

Text model embedding is a process that converts text into a numerical representation that computers can understand. This representation is a vector of numbers or a high-dimensional point in a latent space.

Now I know there are already some open-source text models, such as HuggingFace, so why reinvent the wheel? It’s just my nature to want to know what goes on behind the scenes, I want to know why we are doing this and that.

General Steps

Taking information from online and ChatGPT, I believe these are the general steps that involve working with a Vector Database. I say believe because I am still trying to understand all the specifics.

1. Tokenization

The sentence is broken into tokens (words and punctuation). Before text embedding can begin, the input text is split into smaller units called tokens (e.g., words, subwords, or characters).

EXAMPLE: 

Tokens: ["Does", "snow", "absorb", "sound", "?"]

_________________________________

2. Mapping Tokens to Vectors, or encoding. 

Each token is mapped to an initial numerical representation, often a one-hot encoding or an index. These representations are sparse and high-dimensional, so they need to be transformed into dense, meaningful vectors.

EXAMPLE: Using One-Hot Encoding..Each token is represented as a one-hot vector. For a vocabulary size of 5 (the tokens themselves), the one-hot vectors are:

One-Hot Vectors: 

I need to look into this further. Are vectors just been assigned or are there calculations going on?

_________________________________

3. Word Embedding Models

An embedding layer is a type of neural network layer that maps discrete inputs (like words or tokens) into dense, continuous vector representations. It's learned during training (or pre-trained in models like Word2Vec or BERT). 

Embedding layers or pre-trained embedding models (like Word2Vec, GloVe, or embeddings from transformer models) transform tokens into dense, low-dimensional vectors. 

Embedding Matrix:

Each token's one-hot vector is multiplied by the embedding matrix to retrieve its corresponding dense embedding. In practice, this is done via direct indexing into the embedding matrix.

Another step I need to look into further. Is the embedding matrix a static matrix or is there some calculation that needs to be done to get the embedding matrix.

_________________________________

4. Contextual Embeddings (Advanced Models like BERT)

Contextual embeddings consider the meaning of a word in context. For example, "sound" in "absorb sound" differs from "sound reasoning."

Transformers use attention mechanisms to calculate these embeddings:

1. Input Embedding

2. Linear Transformations

3. Attention Scores

4. Contextualized Embedding

_________________________________

5. Fine-Tuning or Training

If the embeddings are part of a larger model (e.g., GPT or BERT), the embedding vectors are fine-tuned during training using gradient descent and backpropagation to minimize a loss function (like cross-entropy loss).

_________________________________

6. Math Summary of Embedding Generation

At its core, text embedding involves:

• Matrix multiplication: Transforming sparse vectors into dense ones.

• Linear transformations: Projecting embeddings into different spaces.

• Attention mechanisms: Computing relationships between tokens.

• Softmax: Normalizing scores for probabilistic interpretation.

_________________________________

7. Dimensionality of the Embedding Space

The final embedding vector is typically d-dimensional 

In Conclusion

It’s been a year since I looked at all of this, so this is basically a general review article.

I am not sure I want to focus on Vector Database, even though I still like Linear Algebra, but I think it’s still good to know all of this as I continue with my A.I. learning.

I’m still thinking about it but I might create an article on each step… going into as much detail as I can. For now, I believe these are the steps taken for the text embedding model.

I ultimately want to research how to make Data Centers more environmentally friendly and maybe know how Vector Database processes work, someone can come up with a way that doesn’t need so much energy & water.