# Semantic Search using OpenAI Embedding and Postgres Vector DB in NodeJS

This project is about implementing semantic search using OpenAI embedding and Postgres vector database in NodeJS. Semantic search is a search technique that uses natural language processing to understand the meaning of the query and returns results that are semantically related to the query. OpenAI embedding is an API that will convert text into a numerical vector representation that can be used for semantic search. Combining it with the Postgres vector database we can use this to make a semantic search of any text or article we want.

For the example text, I choose [https://jamesclear.com/why-facts-dont-change-minds](https://jamesclear.com/why-facts-dont-change-minds) for the article.

# Diagram

Here is the diagram for the API that we want to create. It is split into two functions.

1. Create and store the embedding based on the article text
    

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1679386644638/bf2196bc-eeca-4965-b15d-c774ad1f2cca.jpeg align="center")

1. Search the queried text
    
    ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1679386660035/7bc0af46-4ead-4a83-b885-1b8cb43836e8.jpeg align="center")
    

# Prerequisite

## Database

For the Postgres database, we will be using [Supabase](https://supabase.com/). Supabase has a free tier, and we can create and use it instantly.

Let's install the extension for Postgres, open the Supabase SQL editor, and run:

```sql
create extension pgvector;
```

Notes: If this extension needs another extension to install, install that also.

## Library

For this project, we use some of the node-js libraries:

```javascript
"@supabase/supabase-js" // supabase node-js client
"dotenv"// to read .env file
"express" // we use express for rest api framework
"gpt-3-encoder" // we need this for chunk the text
"openai" // openai node-js client
```

```javascript
npm install @supabase/supabase-js dotenv express gpt-3-encoder openai
```

## ENV

For the environment variable, we need these 3 for this project:

```javascript
SUPABASE_PROJECT_URL= 
SUPABASE_SECRET_KEY=
OPENAI_API_KEY=
```

Get `SUPABASE_PROJECT_URL` and `SUPABASE_SECRET_KEY (service_role secret)` from creating a database from Supabase first, and go to API Settings.

The `OPENAI_API_KEY` you can get it [here](https://platform.openai.com/account/api-keys).

# Code

Our Project structure will look like this:

```less
node_modules
.env
article.txt
embed.js
openAi.js
package.json
server.js
supabase.js
```

`article.txt` is the content of the article that we want to search for. So make sure to copy the content of the article there first.

Let's create and store the article chunk's embedding first:

## 1.Get Article Data

We need to get the text from the `article.txt`

Before that, let's import the necessary library first.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1679383167141/f840fa92-57ce-41a1-bf70-4a3d3b25e8de.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1679382238545/4fce2eef-0607-444f-905e-dc48901e595e.png align="center")

## 2.Chunk the text

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1679382277010/bbb7765b-7bd8-424e-8adc-1be4650c9d12.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1679382338943/2e278d8f-c77e-4be9-aec3-704823edff5f.png align="center")

### What do we do here?

OpenAI has a token system. 1 token means +- 1 word. What we want now is we chunk the article, and split it into limited `CHUNK_LIMIT` (200 tokens) each. We chunk it, so we can search for it later.

The function first checks if the length of the encoded article is greater than the `CHUNK_LIMIT`. If so, it splits the article into individual sentences and then concatenates them into chunks that are less than or equal to the `CHUNK_LIMIT`. It also ensures that a sentence is not split across different chunks by checking if the last character of a sentence is alphanumeric.

Each chunk is represented as an object with four properties: `content` (the text content of the chunk), `content_length` (the length of the text content), `content_tokens` (the number of tokens in the content), and `embedding`. `embedding` will be added later.

If the resulting chunks are smaller than the `CHUNK_MINIMAL` (100 Tokens ), they are merged with the previous chunk. Finally, the function returns an array of the resulting chunks.

## 3.Create Chunk's Embedding

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1679383478586/4962895d-0c74-4c98-86fd-a6865cca2da6.png align="center")

```javascript
const { Configuration, OpenAIApi } = require('openai');
const configuration = new Configuration({
    apiKey: process.env.OPENAI_API_KEY,
});
const openAi = new OpenAIApi(configuration);

const createEmbedding = async (input) => {
    const embeddingRes = await openAi.createEmbedding({
        model: 'text-embedding-ada-002',
        input: input
    });

    const [{embedding}] = embeddingRes.data.data;
    return embedding
}

module.exports = {
    createEmbedding
}
```

We need OpenAI's API Create Embedding to create the embedding of the chunked text.

## 4.Store them in Postgres DB

Before we store them, let's create the table first via the Supabase SQL editor:

```sql
create table semantic_search_poc (
  id bigserial primary key,
  content text,
  content_tokens bigint,
  embedding vector (1536)
);
```

You can name it whatever you want. I named it `semantic_search_poc` so our query will be like this:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1679384053645/4ebcbb14-b1fd-42ef-a9e0-1318a8dc0160.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1679384098136/7403d6b0-9c9e-4b3b-89a7-5c8b91d2b1f0.png align="center")

Now run the script

```bash
node embed.js
```

If the process is going smoothly, we will able to see the data in the Supabase table view, something like this:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1679384322839/473557f2-9de6-4d30-b471-c6e22620d7d1.png align="center")

Done! You have success to build the most crucial part of this project!

Now to the next step, queried the data.

## 5.Get Query from User

We use `Express JS` to create Rest API Server to get the query from the user. Create simple Express js server code:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1679384708688/ce759a09-0885-4bd8-8d36-0958ac43452b.png align="center")

## 6.Create embedding based on query

```javascript
//server.js
app.get('/', async (req, res) => {
    const { q } = req.query;
    const embedding = await openAiHelper.createEmbedding(q);
})
```

## 7.Search it in Postgres

We use the Postgres function for this search function. So let's head out to the Supabase function editor.

```sql
create or replace function semantic_search (
  query_embedding vector(1536),
  similiarity_threshold float,
  match_count int
)

returns table (
  id bigint,
  content text,
  content_tokens bigint,
  similiarity float
)
language plpgsql
as $$
begin
  return query
  select
    semantic_search_poc.id,
    semantic_search_poc.content,
    semantic_search_poc.content_tokens,
    1 - (semantic_search_poc.embedding <=> query_embedding) as similiarity
  from semantic_search_poc
  where 1 - (semantic_search_poc.embedding <=> query_embedding) > similiarity_threshold
  order by semantic_search_poc.embedding <=> query_embedding
  limit match_count;
end;
$$;
```

We create a Postgres script that creates a function called "semantic\_search". The function takes in a query embedding, similarity threshold, and match count as inputs and returns a table with columns for ID, content, content tokens, and similarity.

The function uses the "semantic\_search\_poc" table to find rows where the similarity between the query embedding and the content embedding is greater than the similarity threshold. It then orders the results by similarity and limits the output to the specified match count.

The Postgres function is done, now call it on the code

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1679385248833/9a75e48b-58a8-44ff-9688-777f88253c51.png align="center")

Run the server:

```javascript
node server.js
```

Now try to hit GET `/` with `q` params something that you want to ask.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1679385456731/010b25e6-0a25-4248-a740-6b3e7a5a38f6.png align="center")

Voila! The API returns the related content ordered by the most similar one.

I hope this article helps to understand how we can create semantic-search apps easily today.

# Resources

GitHub for this project: [https://github.com/fandyaditya/semantic-search-poc](https://github.com/fandyaditya/semantic-search-poc)

References: [https://github.com/mckaywrigley/paul-graham-gpt](https://github.com/mckaywrigley/paul-graham-gpt)