The motivation for creating Hashnode Canva Integration is bringing Hashnode Article to wider audiences by converting it to a different format. I think many of you sometimes see PowerPoint-like content on social media like Instagram, TikTok, or LinkedIn. Something like this:

You have been working hard on publishing a good article here. Why stop there? Bring your Hashnode's article to more audiences by start distributing it as Microblog or Digital Booklet format.

Introducing: Hashnode Canva Integration

Why Canva

Canva is one of many graphic design platforms, if not the best one right now. The easy-to-use, to-the-point, rich template and elements make Canva good for every use case. I believe we at least used Canva once to create cover image of our article

Integration capabilities of Canva, and then GraphQL API from Hashnode, make this possibility become reality. And here we are, Hashnode Canva Integraion is Born.

On Development

I want to make this as simple as possible but keep the functionality intact. The main function is to get article data that can be added as an element to Canva Design. The final application flow is like this:

1. Get data from the URL using Hashnode's GraphQL API

2. Mapping the content

3. Drag to Canvas as Design Element

4. Done!

Simple.

From GraphQL API we can get the content on html markdown or text format. I choose markdown because it is easier to parse. Also, many markdown parsers out there are ready to use.

Canva has SDK that makes it easy to develop on their side. Turning the article content into Canva Element is quite straightforward using it.

The apps extract the content of the article and group them into 5 separate tabs that you can use as elements on canvas design. The tabs are:

• Meta

  This tab contains metadata such as:

  • Background Image

  • Author Name

  • Author Profile Picture

  • Published Date

  • Title

  • Subtitle

• Content

  This tab contains the content of the article. Including the header, text content, code, and images.

• Links

  This tab contains links that appear on the article

• Image

  This tab contains all images in the article

• Code

  This tab contains codes that appear in the article

Every element of the tab can be used as an element of Canva design.

Obstacle

The actual obstacle is the time limit. 15 Days. With full-day working on weekdays, so I need a little sacrifice on my sleep. Also, publishing new Integration to Canva takes a long process of submission. This app is still in review and hopefully, it can accepted as soon as possible.

I will update the status of the Apps here. So Stay Tune!

In Closing

As Always, Thank you to Hashnode for organizing the hackathon! Makes my coding habit still going, and bringing little impact here and there.

Hackathon is a great source of push if you want to build something but always procrastinate. Given timebox, our mind will try to find the most efficient to finish what we started. Driving by incentives, and your motivation is hard to deplet. Give it a try.

Thank you for reading!

#APIHackaton

About MindsDB Hackathon

I've been following MindsDB since the last hackathon on April 2023. But, when at that time, I underestimated the true challenge of the hackathon. The true challenge is to finish. Because every time I see, hundreds or thousands that registered, only few that submit. So, I didn't want to miss it again, this hackathon is a chance to actually build and ship.

What I expect from this hackathon is two things. Building the idea that I already have in mind, and exploring & contributing to the hottest topic of the year: Generative AI, and the tools that support to make it easy and possible.

MindsDB, The Experience

MindsDB is an open source, AI Engine, that makes it easy to build AI tools. With hundreds of databases, and apps integration ready to use using SQL - Like Syntax and HTTP Rest APIs. In today's era, where AI is the hottest topic you can't miss MindsDB. Making AI Tools is easy because you can leave the heavy work to MindsDB. Also, the team and communities are so active in adding new integrations every day.

The experience is new. This is my first time exploring deeply an open source and actually understand what I read haha. Finding something that is not documented, finding a gem, a new way to do something that actually possible by only reading the code and trial-error so we can get. This is really new experience.

And of course, MindsDB makes the heavy part of building the YoutubeAsk easy. Almost 70% of the work is done with the MindsDB feature and integration. What I did was glue it all together and build things. The possibilities are endless. Like, you can create thousands of combinations of apps and functionalities, only with MindsDB. Pretty cool!

My flow when working on this project is like this:

• Read the documentation on the web.

• If not find what I want, go to the source code, to the integration handler module

• Read README on the integration handler

• If not satisfied, read the unit test

• If not available, directly read the handler code.

The deeper I find, the higher the satisfaction of finding something. Like finding a hidden gem.

Prerequisite

MinsdDB is the core of YoutubeAsk. But there are some other tools needed.

Other Tools

• Express JS

  This project uses Express JS as a simple web server.

• Telegram bot

  Why Telegram Bot?

  1. I use Telegram,

  2. I think Telegram has good bot APIs

Create a Telegram Bot first. What we need is:

• Register via Bot Father

• Save the bot token

Follow this tutorial on how to register a telegram bot.

• Ngrok

  For development, we need to register webhook to telegram bot APIs, so telegram knows, where it "forwards" the chat message. We can get what chat data is sent to the bot, and process it in our server.

  In the local machine, we need some kind of tunneling. After a little research, I found this tool: Ngrok.

  With ngrok, we can create a secure tunnel from my local laptop to the internet. With this, I can register my local development server as the webhook to the telegram bot.

  Follow the instructions here

MindsDB

MindsDB will be the core of the apps that we want to make. MindsDB currently can create a Knowledge Base. It's like (Correct Me If I'm wrong Lol) AI storage that can answer any Natural Language query based on the knowledge that is inserted to them. So this is the core of the YoutubeAsk. We need a Knowledge Base based on video YouTube content that asked/registered. Next, where i can get the YouTube data to give feed to the Knowledge Base? Magically, Mindsdb is already have the integrations!

Youtube Integration

MindsDB have YouTube API Integrations that already can get comments, video info, and channel info based on the id given. For YoutubeAsk, what I want to get is video info, to feed the knowledgebase. Gladly MindDB already has it. So convenient!

Oh, also you need a YouTube API key. Follow this tutorial

Vector Databases

Vector Databases is the core of Knowledge Base. Because every data inserted, is inserted as vector, and queried as matching vector to understand the queries and find the related content. This process called Semantic Search. Knowledge Base wrap this into APIs that easy to use. So we just like query to some SQL databases. There's some vector databases that can be integrated to MindDB. I choose the familiar one: ChromaDB.

OpenAI Integration is used in model on this project. Go to OpenAI to get one.

Installing MindsDB on a Local Machine

I'm quite familiar with docker. So when there are Docker images of MindsDB, I just use it on my local machine. But the original images do not come with the dependency of YouTube and Chromadb integration, so I need to do a little tweak. Creating custom Dockerfile, with MindsDB images as bases and adding extra dependencies:

# Use the mindsdb/mindsdb base image
FROM mindsdb/mindsdb

# install mindsdb youtube plugin
RUN pip install mindsdb[youtube]

# install mindsdb chromadb plugin
RUN pip install mindsdb[chromadb]
# RUN pip install pysqlite3-binary

# This create folder is needed for chromadb integration can run
RUN mkdir -p /usr/local/lib/python3.10/site-packages/google/colab

# Set the entry point with the installation and MindsDB command
ENTRYPOINT ["sh", "-c", "python -m mindsdb --config=/root/mindsdb_config.json --api=http,mysql,mongodb"]

Yes, installing YouTube and ChromaDB Mindsdb integration plugin. There some mkdir process, that will explained on the next section.

To build the images, run:

# Build docker
docker build -t my_mindsdb .

Run the images

docker run -p 47334:47334 -p 47335:47335 my_mindsdb

Go to browser and access localhost:47734, it will take some time after docker container successfully runs. You will see Mindsdb Admin UI.

For more info on docker installation, go here.

Next, we go to the development.

Development

I want to make it pretty basic and simple, yet deliver. So the bot becomes like this:

• /ask [youtube-link]: to set the video that you want to ask

• /session: to check the current video

After ready then start asking like regular chat. Simple.

How the apps work

With the help of MindsDB, building something like this is pretty straightforward. Here is the flow.

Let's dive into it

• Get Youtube Transcript

  First, we need to initialize YouTube integration.

    CREATE DATABASE youtubedb
    WITH ENGINE = 'youtube',
    PARAMETERS = {
      "youtube_api_token": "AIzxxx" --  your youtube api token
    };
  

  Then to get the video transcript, like this:

    SELECT transcript FROM youtubedb.videos WHERE video_id = '${id}'
  

  Done! We get the transcript. The data are like this:

    [
        {
            "text": "This is some transcript"
            "start": 1.0,
            "duration": 3.6,
        },
        {
            "text": "This is another transcript"
            "start": 4.7,
            "duration": 3.9,
        },
        ...
    ]
  

• Chunk the transcript

  We need to chunk the transcript so we can search it. Because it is already on a JSON array, I chunk it based on 10 items. Every item is an average of 5 seconds clip. So if we chunk every 10 items, we get ~50 seconds of clip for each chunk.

• Insert to Knowledge Base

  Before we can insert into it, we need some preparation. Create the ML Engine, Model, and Knowledge Base:

--Setup ml engine
CREATE ML_ENGINE embedding FROM langchain_embedding;

CREATE ML_ENGINE openai_engine
FROM openai
USING
    api_key = 'sk-xxxx'; -- your openai api key


--Create embedding model
CREATE MODEL openai_embedding_model
PREDICT embeddings
USING
    engine = "embedding",
    class = "openai",
    api_key = "sk-xxxx", -- your openai api key
    input_columns = ["content"];


-- Create qna model using openai
CREATE MODEL q_n_a_model
PREDICT answer
USING
    engine = 'openai_engine',
    prompt_template = 'Use the following pieces of video context to answer the question at the end. If you do not know the answer, just say that you do not know, do not try to make up an answer. Dont answer outside of the context given.
                        Video context: {{context}}
                        Question: {{question}}
                        Helpful Answer:',
    model_name= 'gpt-3.5-turbo-16k',
    mode = 'default',
    max_tokens = 1500;

-- Connect to Chroma
CREATE DATABASE chromadb
WITH ENGINE = "chromadb",
PARAMETERS = {
    "persist_directory": "chromadb/"
};

-- create youtube_ask table, we need to spesify init dummy data so chromadb collection can be created
CREATE TABLE chromadb.youtube_ask (
  select content, '{"data": "init data"}' as metadata, embeddings 
  from openai_embedding_model
  where content = 'init content'
);


--Create knowledge base
CREATE KNOWLEDGE_BASE youtube_ask_kb
USING
    model = openai_embedding_model,
    storage = chromadb.youtube_ask;

• I followed Knowledge Base and Q&A documentation for preparing The Knowledge Base, and Q&A model, with a little tweak:

  • Changing Embedding Model using OpenAI

  • Add parameter model_name of Q&A Model to gpt-3.5-turbo-16k, max_tokens to 1500 and mode to default . Changing model name to that, so we can send 16k token context to the OpenAI. Max token to 1500 so the response is not truncated. And it can all work, when I use mode=default.

Now we can insert to the knowledge base using:

    INSERT INTO youtube_ask_kb (content, metadata) 
    VALUES ('chunked content', '{"link": "http://you.tube/link"}')

With content is the chunked content, and the metadata.link is the YouTube video URL. We need the link so we can query the context based on the video given.

• Query the knowledge base

  Querying knowledge base is pretty straightforward:

    SELECT content 
    FROM youtube_ask_kb 
    WHERE content = 'Question from user' 
    AND youtube_ask_kb.metadata.link = 'http://you.tube/link'
  

• Give the queried result to q_n_a_model

  Now with data that we get from Knowledge Base, we want to feed LLM so it can give a natural response to the users. Based on the question, and context from the knowledge base data.

    SELECT answer 
    FROM q_n_a_model 
    WHERE context = 'Data from queried knowledge base' 
    AND question = 'Question from user'
  

  We will get something like this:

  Q: What kind of technique the video suggest to mapping out my week?

  A: The video suggests the technique of writing down every single thing you are going to do for the next week. This includes your morning routine, focused work routine, other tasks and meetings, and nightly routine. By writing everything down, you can reduce the friction of making better decisions and spot problems in your days. The video also mentions the importance of experimenting and iterating on your plan week after week to refine your system.

  Nice!

When Things Didn't Go as Planned

I can guarantee that when building things, there are some something that didn't go as planned. For this project, here they are.

ChromaDB Installation

Something happened on ChromaDB Installation. After some research, I found the issue. The default Sqlite from Python 3.10 is not compatible with the ChromaDB. So it needs to be installed separately.

I found this post, and scrolling to the comment, and solved it like this:

#Dockerfile

# This create folder is needed for chromadb integration can run
RUN mkdir -p /usr/local/lib/python3.10/site-packages/google/colab

By creating an empty Google Collab folder via Dockerfile, it downloaded separate Sqlite that is needed by the ChromaDB.

Q&A Model

The problem with Knowledge Base is when we query something general like: What is the video about? it can't get any data because it doesn't get any matching criteria for that kind of question. So, my fix is, just send all of the transcript to Q&A Model and, let it consume all of the content of the video, and answer based on that.

The second problem occurs. When we use the default configuration for the Q&A Model, it only supports 4k context windows, because it uses text-davinci OpenAI model. When we just send all of the transcript to the model, it error because pass over 4k context. We need to change the model. Because it isn't documented anywhere, I need to dive into the code to find if this is possible or not. Of course, it is possible, and changing to a model gpt-3.5-turbo-16k makes it work. Handling 20~ pages of context is possible now.

In Closing

Participating in this hackathon gave me new experience and knowledge. Hackaton gives a sense of urgency, with a driving incentive that gives me kind of motivation to explore new things, and actually build an idea that has been urged to be checked on my idea notes. Exploring new tools to create something awesome. Building something becomes easier and easier nowadays. The same applies to MindsDB, creating AI Apps so much easier and faster.

Oh, and also contributing to open source. This is my first time creating PR for Open Source.

• Update Documentation of YouTube Integration

• Update Documentation of OpenAI Integration

Well, maybe that's not much hahaha. But it's ok, extra documentation may save hours for others.

Thank you to MindsDB and Hashnode for the amazing hackathon. Can't wait for the next hackathon and bringing the next idea a life!

You can find GitHub repo here

Also thank you for reading! Have a nice day!

#MindsDB #MindsDBHackathon

Generated AI is one of the latest tech. The progress is swift, and it actually helpful for many jobs in every aspect of people's lives. The most groundbreaking tech discovery after spreadsheet :p. Everyone uses it.

LLM like ChatGpt, trains using data on the Internet. But what if it also learns from the data prompt that we give them? And if that is true, sending sensitive data to it is not good because our sensitive data can be exploited, something that we don't want. Please don't risk it.

Maybe sending sensitive data to third parties is not really a problem for people from different backgrounds and demographics, because they need another important problem to take care of.

"We are on the Internet! No data is really private!"

But if we can slightly prevent it, by ourselves, why not?

The simplest way we can prevent giving any sensitive data to LLMs is to hide the specific sensitive data. Or on another term, redact it.

Yes, redact the sensitive information from the provided prompt, and un-redact the LLMs response to get the actual response back.

Just like that, I think the simplest way. And the simplest way to develop that is to use Pangea Cloud

What is Pangea Cloud

Couple weeks ago, I discovered Pangea Cloud. Based on the description on their landing page:

Security Services for Developers

Get the APIs you need to deliver a secure customer experience, all in one place!

Yes, it is security as a service, so we, developers can deliver apps with high-quality security out of the box.

Based on that, now we can create simple web apps to redact prompts with the Redact API provided by Pangea Cloud.

Introducing: redactd

redactd works really simple, like diagram below:

1. Redact the prompt using Pangea Redact API

2. Got redacted prompt

3. Copy-paste the redacted prompt to LLM

4. Got response

   1. If the response is redacted, then Un-redacted it first

   2. Got actual response

   3. Done!

5. Done!

Now let's try it!

The prompt:

Budi Dena have savings in banks $5000 with 3% yearly interest 

Selene Sofiro have savings on index funds $3000 with 10% interest yearly 

Within 10 years, who have more savings?

It will redacting the person's name:

Now copy-paste the redacted prompt to ChatGpt

As we can see, ChatGPT keeps mentioning the redacted text, now we want the un-redact version of the response. Copy the response to un-redact input field:

Voila, it works!

Using the formula for compound interest:

\[ \text{Future Value} = \text{Principal} \times \left(1 + \frac{\text{Interest Rate}}{100}\right)^{\text{Time}} \]

\[ \text{Future Value} = 5000 \times \left(1 + \frac{3}{100}\right)^{10} \]

<Selene Sofiro >:
Principal amount (initial savings): $3000
Interest rate: 10% per year
Time: 10 years

Using the same formula:

\[ \text{Future Value} = \text{Principal} \times \left(1 + \frac{\text{Interest Rate}}{100}\right)^{\text{Time}} \]

\[ \text{Future Value} = 3000 \times \left(1 + \frac{10}{100}\right)^{10} \]

Now, let's calculate these values to find out who has more savings after 10 years.

After calculating the above expressions, it is found that:

\[ \text{Future Value for <Budi Dena >} \approx \$6729.56 \]

\[ \text{Future Value for <Selene Sofiro >} \approx \$7912.85 \]

Therefore, after 10 years, <Selene Sofiro > would have more savings.

In Closing

Redacting text before send it to third parties is one of the way to keep privacy of our data. By using redactd, we have slight control on how we use data on the internet, specifically when using LLMs like ChatGPT

Thank you for reading, and shout out to Pangea Cloud and hashnode for this hackathon!

Try it here: https://redactd.vercel.app/

Github: https://github.com/fandyaditya/redactd

#Pangea #PangeaSecurathon

The Issue

The issue is because it is deployed on vercel free, and 500 pages on my personal blog, vercel 10s timeout limit background worker cannot fetch all the updated data because there to many pages. So, i need to trigger build everytime i post something new. Also with the 10-second timeout limit background process, the tools cannot generate sitemap.xml for the blog because the pages need more than 10-second to generate. Google never knew my blog.

After 1 year keep using it, for this month, I insist on migrating my blog to static-site generator. And my choice is Hugo.

Why Hugo?

Apparently, I don't have any particular reason why I choose Hugo over another alternative like Gatsby.js and etc. Maybe it was just an impulsive decision because currently I'm working with Golang in my workplace (and I love it), so Hugo that builds using Golang become my choice. If you want to get any other good comparison, I think there is already good information out there.

Migration Process

So the process of migration we can separate into these step:

1. Create Hugo project, and import themes

2. Configure the themes

3. Export Notion pages into markdown

4. Change every markdown metadata into Hugo format

5. Copy new formatted markdowns

6. Publish and done!

For step 4, I create a Node.js script to do the job. You can steal the script here:

Basically what the script does is replace all Notion markdown metadata and reform it into Hugo-supported metadata.

And it is done! My new blog has new skin, and it is blazing fast. Also super sleek layout thanks to this theme. Check my blog here:

https://blog.fandyaditya.com/

TIL

After writing this post, I learned that there are already open-source solutions to convert Notion blog to Hugo flawlessly. Here are some of them, you can try it yourself!

• https://github.com/HEIGE-PCloud/Notion-Hugo

• https://github.com/nikvdp/notion-blog-exporter

• https://livingissodear.com/posts/write_in_notion_publish_with_hugo_introducing_n2h/

Types of third-party API providers

There are three types of third-party API providers that we can categorize:

1. Seamless API Providers: These providers specialize in offering APIs for public use. Since this is their primary business, they pay extra attention to this area. They focus on making their APIs seamless and easy to use by providing excellent documentation and support. Users can often self-register and obtain their own authentication keys, such as apiKey or secretKey. Stripe is a good example of this category.

2. Semi-Seamless API Providers: These providers also offer public APIs, but with certain legal agreements in place. Users cannot self-register, and documentation is provided during the agreement process. Although this is not their main focus, they have dedicated teams and good support for their API offerings. Bank integrations are an example of this category.

3. APIs Under Development/Manual Integration: This is the most challenging category to integrate with. These providers are relatively new to offering third-party APIs and lack dedicated teams and support for integration. Agreements are often informal rather than legally binding. In most cases, support from these providers is limited. Businesses that already have established processes and want to expand into third-party API provision fall into this category.

Determining who needs whom

In these collaborative efforts, the fundamental question is: Who needs whom? Ideally, both parties should benefit equally from the collaboration (mutualism). However, when this balance is absent, the party with the greater need is at a disadvantage. If a provider requires us to integrate their APIs rapidly into our system, they are likely to actively provide support and fulfill all requirements to ensure a fast and accurate integration. However, if we rely more on their APIs, the dynamics change. In such cases, we must proactively communicate our needs.

The Issues

Based on my experience, the following issues are likely to arise:

1. Lack of Documentation

   The lack of comprehensive documentation is a common problem. It may include issues such as unclear API requests, insufficient API responses, and unclear error conditions. Additionally, if there is no clear understanding of how an API works or which API needs to be integrated, valuable time is wasted seeking clarification.

2. Errors

   Inadequate documentation makes every bit of support from the provider invaluable, especially during production. If there are any errors caused by their APIs, users may still blame us. Therefore, having a dedicated support team from the provider can save us from significant trouble.

3. Lack of Representative Test Data

   The absence of representative test data makes end-to-end testing difficult in the development environment. Furthermore, when test data is available, discrepancies between the production and development environments often hinder reproduction of specific cases.

4. Breaking Changes

   Unforeseen breaking changes in third-party APIs can occur without prior notification, causing error to our production system.

What Went Well

If in the future I need to develop this kind of apps again, this action that will be keep and repeated:

1. Proactive Approach

   Despite the lack of documentation, our team remains proactive by seeking clarification and initiating discussions to address any issues or problems encountered during the documentation and development process. While the provider's response may not always be immediate, it is essential to make the best of the available information and proceed accordingly.

2. Unified Communication

   We have established dedicated WhatsApp groups for each party involved, including our team, the provider's team, and all stakeholders. This approach ensures transparency and enables us to share any errors, successes, problems, or smooth operations that occur.

3. Stakeholder Understanding

   Our stakeholders comprehend that integrating with third-party APIs presents challenges, including slow development and occasional disruptions to production. Their understanding stems from the transparency we maintain through the WhatsApp groups. Moreover, if the provider's response is delayed, we can escalate our requests through higher channels. Based on my experience, this approach usually yields a higher response rate.

Lessons Learned

If in the future I need to develop this kind of app again, I will do all this action:

1. Extensive Unit Testing

   Although my company follows a standard of achieving 80% code coverage with unit tests, I would emphasize the need for more comprehensive and tailored unit tests. This would enhance our confidence in the integration code and account for a wider range of possibilities.

2. Mock Data

   We have faced delays in releases due to inadequate testing options. Leveraging tools like Postman for mocking responses proved to be beneficial. Although having real example data from the source is preferable, utilizing mocking tools can expedite the testing process.

3. Effective Communication

   Maintaining effective communication channels between teams and stakeholders is crucial for successful collaboration and issue resolution.

4. Meaningful Logging

   Employing well-structured logging mechanisms allows us to track and analyze system behavior effectively. We have done some logging and monitoring on production, but sometime some error still hard to trace. Good logging system will avoid some headaches.

Finally, it may be worth reconsidering the necessity of being an aggregator of third-party APIs. Evaluating if our business can operate without reliance on such APIs might provide valuable insights and potential opportunities for change.

That covers the main points of the article. Hopefully, this article give you some insight on working with third-party APIs.

$ npm install sinon mocha chai --save-dev

--save-dev is used because these modules are only needed during development. We don't need these modules during production.

The Script

The main thing to do when creating a script is to determine which function to test, along with what will be tested in that function.

For example, if we want to do unit testing on the following function:

/*number-lib.js*/

function addInteger(x, y) {
    if (Number.isInteger(x) && Number.isInteger(y)) {
        return x + y   
    }
    else {
        return 'Number is not integer'
    }
    return 
}
module.exports = {
    addInteger
}

The function above is a function to add integers x and y. If they are not integers, it will return a string saying that the input is not an integer.

First, we will determine our test scope as follows:

// number-list-test.spec.js
const numberLib = require('./number-lib.js')

describe('Add integer', () => {
    it('should return 23', () => {
       // Arrange
       ...
       // Act
       ...
       // Assert
       ...
    })
    it('should return "Number is not integer", because input is not integer', () => {
       // Arrange
       ...
       // Act
       ...
       // Assert
    })
})

describe() and it() are functions of Mocha, where we will write our test script. In the describe() function, we determine our test scope. Inside each it() function, we determine the cases that we will perform.

In the case above, our scope is to test the addInteger() function with 2 agreed-upon cases, namely the function must return 23, and the function must return "Number is not integer" because the input we will give is not an integer.

In the it() function, there is an Arrange, Act, and Assert section. Here is an explanation:

1. Arrange

In the Arrange section, it usually contains input initialization and doubles initialization which aims to produce the desired output.

1. Act

In the Act section, we execute the function that we want to test using the input that we have initialized in the Arrange section.

1. Assert

In the Assert section, we check whether the result produced by the function in the Act section matches the expected output that we have initialized in the Arrange section.

Now we will add the code snippet above to make it complete.

// number-list-test.spec.js
const numberLib = require('./number-lib.js')
const expect = require('chai').expect

describe('Add integer', () => {
    it('should return 23', () => {
       // Arrange
       let x = 11
       let y = 12
       // Act
       let result = numberLib.addInteger(x, y)
       // Assert
       expect(result).to.equal(23)
    })
    it('should return "Number is not integer", because input is not integer', () => {
       // Arrange
       let x = 1.1
       let y = 1.2
       // Act
       let result = numberLib.addInteger(x, y)
       // Assert
       expect(result).to.equal('Number is not integer')
    })
})

Doubles

For example, let's update the function we want to test like this:

function addNumber(x, y) {
    if(isInteger(x) && isInteger(y))) {
        return x + y
    }
    else if(isDecimal(x) && isDecimal(y))) {
        return Math.round(x) + Math.round(y)
    }
    else {
        return 'The input is not integer nor decimal value'
    }
}

function isInteger(x) {
    ...
}

function isDecimal(x) {
    ...
}

modules.export = {
    addInteger
}

Correct. There is a function inside a function. The concept of Unit Testing is that we only test the function that we want to test. If the function calls another function, then we ignore the other function by making it a Double.

In our example, we will not care about the isInteger() and isDecimal() functions. Both of these functions will be made into doubles with the help of sinon.

In that case, it was agreed that there are 3 cases to be tested. Here are the tests:

const sinon = require('sinon')
const expect = require('chai').expect
const numberLib = require('./number-lib')

describe('Add Number', () => {
    it('should return 40', () => {
    })
    it('the input is decimal, should return 15', () => {
    })
    it('should return "The input is not integer nor decimal value"', () => {
    })
})

We have determined the cases, now let's add the test script:

const sinon = require('sinon')
const expect = require('chai').expect
const numberLib = require('./number-lib')

describe('Add Number', () => {
    it('should return 40', () => {
        //Create new stub for isInteger that always return true
        let isInteger = sinon.stub(numberLib, 'isInteger').returns(true)
        let x = 20
        let y = 20
        let result = numberLib(x, y)
        expect(result).to.equal(40)
        isInteger.restore()
    })
    it('the input is decimal, should return 15', () => {
        //Create new stub for isInteger that always return false
        //stub for isDecimal always return true
        let isInteger = sinon.stub(numberLib, 'isInteger').return(false)
        let isDecimal = sinon.stub(numberLib, 'isDecimal').returns(true)
        let x = 7.4
        let y = 7.6
        let result = numberLib(x, y)
        expect(result).to.equal(15)
        isInteger.restore()
        isDecimal.restore()
    })
    it('should return "The input is not integer nor decimal value"', () => {
        //Create new stub for isInteger that always return false
        //stub for isDecimal always return false
        let isInteger = sinon.stub(numberLib, 'isInteger').return(false)
        let isDecimal = sinon.stub(numberLib, 'isDecimal').returns(false)
        let x = 'anything'
        let y = 'anything'
        let result = numberLib(x, y)
        expect(result).to.equal('The input is not integer nor decimal value')
        isInteger.restore()
        isDecimal.restore()
    })
})

At the beginning of each case, we always initialize the stub for the isInteger() and isDecimal() functions, by changing its return value according to the needs of the test case. When numberLib() is called, if a stub function we made before is encountered during execution, that function will be changed to a stub.

At the end of each case, we must restore() the stubs we made so that the functions that became stubs can run their own content again. If we don't do a restore(), there is a possibility that the next test will fail because these functions are still stubs.

The above test script can still be improved with the following Mocha functions:

...
const numberLib = require('./number-lib')
describe('Add number', () => {
    let isInteger
    let isDecimal
    beforeEach(() => {
        isInteger = sinon.stub(numberLib, 'isInteger')
        isDecimal = sinon.stub(numberLib, 'isDecimal')
    })
    afterEach(() => {
        isInteger.restore()
        isDecimal.restore()
    })
    it('...', () => {
        isInteger.returns(true)
        ...
    })
    it('...', () => {
        ...
    })
    it('...', () => {
        ...
    })
})

beforeEach() is always called before each test case is executed, while afterEach() is always called after each test case is executed. With the help of beforeEach() and afterEach() we can avoid repetitive initialization code and make our test script more readable.

Anonymous function cases

Sometimes we encounter something like this:

//my-route.js
router.get('/', (req, res) => {
    //function that we want to test
    ...
    ...
})
router.get('/user', (req, res) => {
    //other function
    ...
    ...
})
module.exports = router

The code above is one of the examples to create a router on express.js.

How do we test that function, when that function can't be called from another function or so it is called Anonymous Function? We can test it with Sinon's help.

First, create the test script:

const route = ('./my-route')

describe('Route test', () => {
    it('should ...', () =>{

    })
})

We can see from the function, router call get function that has 2 parameters. First parameter is the endpoint of the route, and the second is the function that we want to test. So we need to create spy on get function.

const route = require(./my-route)
const sinon = require('sinon')

describe('Route test', () => {
    it('should ...', () => {
      //create spy on get
      let get = sinon.spy(route, 'get')
      // Arrange input 
      let req = {}
      let res = {}
      //call the function
      route.router()
      ..
    })
}

When route.router() called, all function (router.get('/') dan router.get('/user')) will be called. Because we only want to test get('/'), the test code become like this:

const route = require(./my-route)
const sinon = require('sinon')

describe('Route test', () => {
    it('should ...', () => {
      let get = sinon.spy(route, 'get')

      let req = {}
      let res = {}

      route.router()
      //Calling the `first` get and then trigger anonymous function on the parameter no 1
      get.firstCall.callArgWith(1, req, res)

      expect(...)
      get.restore()
    })
}

get.firstCall will get only the function that is called first.

get.firstCall.callArgwith(1, req, res) will be triggering the function on second parameter (Zero based). And then, we succesfully call the anonymous function, asserting could be done.

Running

To run the test script, we need some modifications on package.json

// package.json
"scripts": {  
    "test": "mocha test/**/*.spec.js"
}

The goal is to running mocha command on all files in all folders that having .spec.js extension. So we need to name our test function with .spec.js

Run the script:

npm test

We can see the results in terminal:

> mocha test/**/*.spec.js

  Add Integer
    ✓ should return 23
    ✓ should return "Number is not integer", because input is not integer

 2 passing (1s)

Reporting

We need to module for this. Mocha-junit-reporter for report unit-test, and nyc for code coverage

$ npm install mocha-junit-reporter nyc --save-dev

Modify package.json on scripts:

"scripts": {
    "report": "mocha test/**/*.spec.js --reporter mocha-junit-reporter",
    "coverage": "nyc --reporter=lcov --reporter=text-lcov npm test"
},

Run it:

$ npm run report
$ npm run coverage

We will get .xml that we can show on any other application or tool like Jenkins

References

https://martinfowler.com/articles/mocksArentStubs.html

https://www.sitepoint.com/sinon-tutorial-javascript-testing-mocks-spies-stubs/

Luckily, I found this Paul Graham GPT on GitHub. This is basically what I want to build. Tweak some here and there. 60% of the work is already done.

What I need to do now is gather all the data, create the vector, insert it into Pinecone, then queried it.

For better understading, please go to the series: https://dev.fandyaditya.com/series/openai-semantic-search.

Any step that seems missing will be found on that series.

Scrapping the data

For the threads. Because I don't have access to the twitter APIs. I need to find another way.

Rattibha one of Threadsaver-bot, give me access to IM's 400++ threads. Which means, many twitter user use Rattibha to save IM's twitter-thread.

For the essay, I can got it from his blog illimitablemen.com

Maybe because of the structure or I just suck at selecting HTML elements and using any HTML parser, it takes a long time for me to scrape and clean the data. I create a script for it, but I also use Chrome extension web scraper tools to collect some of the data. Manually cleaning it if the script or the tools not cleaning it well. I scrape the text content, link of the source tweet/essay, and the created date of the content and store it into JSON.

Create embedding and store it

After data is gathered, now we create the embeddings. Before we embed, we chunk the data first. I use Langchain to do that.

import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";

const textSplitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000, chunkOverlap: 0 });

const fileJson = //where the data stored

const splittedText = await textSplitter.splitText(fileJSon.text);

RecursiveCharacterTextSplitter is a Langchain module that makes chunking text very easy. I choose 1000 chunk size, which means I want the chunk consists of max 1000 characters each.

Now we need to create embedding and store the data to Pinecone. But before we do that, we need to specify the content of the metadata that we want to query later. For this project, we just need the text content, the title, and the link so we can navigate it to the source essay/tweet.

I use Langchain Pinecone Vector Store module to do this. We also need to convert the chunk into Document object.

import { PineconeStore } from "langchain/vectorstores";
import { OpenAIEmbeddings } from "langchain/embeddings";

let documents = []; 
splittedText.forEach((txt) => {
            const document = new Document({
                pageContent: txt,
                metadata: {
                    title: fileJSon.title,
                    createdAt: fileJSon.createdAt,
                    link: fileJSon.link.url
                }
            });
            documents.push(document);
});

PineconeStore.fromDocuments(documents, new OpenAIEmbeddings(), { pineconeIndex: pinecone.index('indexname'), namespace: 'namespacename' })

With PineconeStore.fromDocuments function, the chunks are automatically turned into embedding and inserted into the pinecone index.

Take note: document.pageContent will be converted into metadata.text in vector in pinecone db. This is important for querying later.

Now the data is stored and ready to use!

Create the Q&A Apps

As i mention before, I built this on top of this repo. Need a little tweak. Because it use supabase, i need to change it into pinecone.

//pinecone.js

export const query = async (embed) => {

    await pinecone.init({
        environment: process.env.PINECONE_ENVIRONMENT,
        apiKey: process.env.PINECONE_API_KEY
    });

    const index = pinecone.Index('indexname');
    const queryRequest = {
        vector: embed,
        topK: 5,
        includeValues: false,
        includeMetadata: true,
        namespace: 'namespacename'
    }
    try {
        const response = await index.query({ queryRequest })
        const mappedResponse = response.matches.map((item) => {
            return {
                metadata: item.metadata
            }
        })
        return { data: mappedResponse }
    }catch(err) {
        return { error: err }
    }   
}

Then change for any code that calls Supabase query into this function.

And done!

Try it here: https://illimitablemen-gpt.vercel.app/.

Special Thanks

Thanks to IM for the awesome writing. Very eye-opening, make me become a better person. I highly recommend to follow him and read his essays.

And also mckaywrigle for Open Sourced Paul Graham GPT.

https://langchainers.hashnode.dev/getting-started-with-langchainjs

My experience with Langchain, it works out of the box. We can cut many manual steps with Langchain. Also as new tools or libraries, there are some learning curves. Mainly because lack of documentation and tutorial (Especially for Node JS Client Library!)

For this project, we will re-create previous project, but use Langchain.

Previous Project's Article:

https://dev.fandyaditya.com/semantic-search-using-open-ai-embedding-pinecone-vector-db-and-node-js

Prerequisite

Database

Same as the previous trial, we will use pinecone. Create an account at pinecone.io first.

And then at the console, create an Index.

Index Name: whateveruwant (i named it"article")
Dimensions: 1536
Metric: cosine
Pod Type: S1 or P1

Library

"@pinecone-database/pinecone"
"dotenv"
"express"
"langchain"
"openai"

npm install @pinecone-database/pinecone dotenv express langchain openai

ENV

As usual, we need OpenAI API key, Pinecone ENV, and Pinecone API key

OPENAI_API_KEY
PINECONE_ENVIRONMENT
PINECONE_API_KEY

Article

Select some article from the internet, and put it in article.txt. I choose this article: https://jamesclear.com/saying-no

Code

Our project structure will look like this:

node_modules
.env
article.txt
embed.js
package.json
server.js
pinecone.js

We will go to pinecone.js first:

//pinecone.js

/**
Read .env
**/
import * as dotenv from 'dotenv';
dotenv.config();


/**
Init Pinecone
**/
import { PineconeClient } from '@pinecone-database/pinecone';
const pinecone = new PineconeClient();

await pinecone.init({
    environment: process.env.PINECONE_ENVIRONMENT,
    apiKey: process.env.PINECONE_API_KEY
});


/**
Export the pinecone index function, named it based on your created index in pinecone. We will need this for Pinecone langchain wrapper.
**/
export const index = pinecone.Index('article')

Now in embed.js

//embed.js

/**
Read .env file
**/
import * as dotenv from 'dotenv';
dotenv.config();


/**
Open AI Embedding wrapper from langchain
**/
import { OpenAIEmbeddings } from 'langchain/embeddings';


/**
Chunk text/text splitter function from langchain
**/
import { RecursiveCharacterTextSplitter } from 'langchain/text_splitter';


/**
Pinecone wrapper from langchain
**/
import { PineconeStore } from 'langchain/vectorstores';


/**
Init fs
**/
import * as fs from 'fs';


/**
Create text splitter with chunksize 1000 character
**/
const textSplitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000, chunkOverlap: 0});


/**
Init OpenAI Embeddings
**/
const embedder = new OpenAIEmbeddings();


/**
Get index from our pinecone.js
**/
import { index } from './pinecone.js';


(async () => {
    //read article
    const article = await fs.readFileSync('article.txt', { encoding: 'utf-8' });
    //split the text
    const splittedText = await textSplitter.createDocuments([article]);

    //store the splitted text to pinecone, in index "article" and namespace "langchain" (namespace is for filter purpose later, can be whatever you want)
    PineconeStore.fromDocuments(splittedText, embedder, { pineconeIndex: index, namespace: 'langchain' });
})()

Now run:

node embed.js

Check pinecone console. If this success, it will create vector data in Pinecone db

Now in server.js,create express app that pass query URL to search it in db using Langchain.

//server.js

/**
Import required langchain and pinecone library like in ./embed.js
**/
import { PineconeStore } from 'langchain/vectorstores';
import { index } from './pinecone.js';
import { OpenAIEmbeddings } from 'langchain/embeddings';
const embedder = new OpenAIEmbeddings();
const pineconeStore = new PineconeStore(embedder, { pineconeIndex: index, namespace: 'langchain' });


/**
Init express app
**/
import express from 'express';
const app = express();
const port = 9000;



app.get('/', async (req, res) => {
    const { q } = req.query;

    try {
        const data = await pineconeStore.similaritySearch(q, 5);
        res.status(200).send([...data])
    }catch(err) {
        res.status(404).send({ message: `${q} doesn't match any search` });
    }
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

We tried to search based on vector similarity from Pinecone DB in index article and namespace langchain based on a passed query with 5 top similar results.

Now run:

node server.js

Tried to hit http://localhost:9000

Good job! You success to do a semantic search using Langchain in NodeJS!

Resources

Github for this project: https://github.com/fandyaditya/semantic-search-langchain

Previous Article:

https://dev.fandyaditya.com/semantic-search-using-openai-embedding-and-postgres-vector-db-in-nodejs

Why Pinecone?

Many people across the internet mention pinecone are good for semantic search, because that is what pinecone is intended to build. After I do quick research (research for the pricing, obviously :D), for today 23 march 2023 pinecone already has NodeJs client, also I think it is affordable and has a generous free tier. So why not?

Prerequisite

For this topic, we will use 90% of code from the previous trial. Here are some additional work we need to do:

Database

As we know, today we will use Pinecone. Create an account at pinecone.io first.

And then at the console, create an Index. Index is like a table in a relational DB.

Index Name: whateveruwant (i named it"article")
Dimensions: 1536
Metric: cosine
Pod Type: S1/P1

We need to specify the Dimensions to 1536 because the model text-embedding-ada-002 we used from OpenAI to create embedding returned 1536 vector dimension.

Library

We need the Pinecone NodeJs library:

"@pinecone-database/pinecone"

npm install @pinecone-database/pinecone

ENV

We need the Pinecone API key and env

PINECONE_API_KEY
PINECONE_ENV

Get them in the Pinecone console, after creating Index.

Code

Our Project structure, with the previous code, will look like this:

node_modules
.env
article.txt
embed.js
openAi.js
package.json
server.js
supabase.js
pinecone.js(new)

Init pinecone, and create upsert function. Based on this documentation

//pinecone.js

//init
const { PineconeClient } = require("@pinecone-database/pinecone");
const uuid = require('uuid').v4; 

const pinecone = new PineconeClient();

pinecone.init({
  environment: process.env.PINECONE_ENV,
  apiKey: process.env.PINECONE_API_KEY,
});

//upsert function
const upsert = async (data) => {
    const index = pinecone.Index('article');
    const { content, content_tokens, embedding } = data;

    const upsertRequest = {
        vectors: [
            {
                id: uuid(),
                values: embedding,
                metadata: {
                    content,
                    content_tokens
                }
            }
        ]
    }
    try {
        const upsertResponse = await index.upsert({ upsertRequest });
        return upsertResponse;
    }catch(err) {
        return err
    }

};

module.exports = {
    upsert,
}

From upsertRequest, use metadata to store value others than the embedding to the database. From this project, we store content and content_tokens

Now, to the embed.js call it.

//embed.js
const pineConeHelper = require('./pinecone');
....
(async () => {
    const article = await fs.readFileSync('article.txt', { encoding: 'utf8' });
    const chunkedArticles = await chunkArticle(article);

    for(let i = 0 ; i < chunkedArticles.length ; i++) {
        const embedding = await openAiHelper.createEmbedding(chunkedArticles[i].content);
        await pineConeHelper.upsert({ 
            content: chunkedArticles[i].content,
            content_tokens: chunkedArticles[i].content_tokens,
            embedding
        });

        //temporary disabling supabase from prev trial

        // const { data,error } = await supabaseHelper
        //     .from('semantic_search_poc')
        //     .insert({
        //         content: chunkedArticles[i].content,
        //         content_tokens: chunkedArticles[i].content_tokens,
        //         embedding
        // })

        setTimeout(() => {}, 500)
    }
})()

Run:

node embed.js

If this success, go to the Pinecone console, and you will see in the Index info, there will be data stored.

Now for the query, create a query function.

//pinecone.js
...

const query = async (embed) => {
    const index = pinecone.Index('article');
    const queryRequest = {
        vector: embed,
        topK: 10,
        includeValues: false,
        includeMetadata: true
    }
    try {
        const response = await index.query({ queryRequest })
        return { data: response }
    }catch(err) {
        return { error: err }
    }

}

module.exports = {
    upsert,
    query
}

The vector field will search the embedding with the top 10 values. We don't want to include the vector values and want the metadata only, so we set includeMetadata=true

Now call it on server.js

const pineconeHelper = require('./pinecone');
...
app.get('/', async (req, res) => {
    const { q } = req.query;

    const embedding = await openAiHelper.createEmbedding(q);

    //temporary disabling supabase from prev trial

    // const { data, error } = await supabaseHelper.rpc('semantic_search', {
    //     query_embedding: embedding,
    //     similiarity_threshold: 0.5,
    //     match_count: 5
    // })

    const { data, error } = await pineconeHelper.query(embedding);

    if(error) {
        res.status(404).send({ message: `${q} doesn't match any search` });
    } else {
        res.status(200).send({...data})
    }
})

Run the server, and try to hit http://localhost:9000/q=ask something

node server.js

It running smoothly!

Resources

GitHub for this project (branch "pinecone"): https://github.com/fandyaditya/semantic-search-poc/tree/pinecone

For the example text, I choose https://jamesclear.com/why-facts-dont-change-minds for the article.

Diagram

Here is the diagram for the API that we want to create. It is split into two functions.

1. Create and store the embedding based on the article text

1. Search the queried text

   

Prerequisite

Database

For the Postgres database, we will be using Supabase. Supabase has a free tier, and we can create and use it instantly.

Let's install the extension for Postgres, open the Supabase SQL editor, and run:

create extension pgvector;

Notes: If this extension needs another extension to install, install that also.

Library

For this project, we use some of the node-js libraries:

"@supabase/supabase-js" // supabase node-js client
"dotenv"// to read .env file
"express" // we use express for rest api framework
"gpt-3-encoder" // we need this for chunk the text
"openai" // openai node-js client

npm install @supabase/supabase-js dotenv express gpt-3-encoder openai

ENV

For the environment variable, we need these 3 for this project:

SUPABASE_PROJECT_URL= 
SUPABASE_SECRET_KEY=
OPENAI_API_KEY=

Get SUPABASE_PROJECT_URL and SUPABASE_SECRET_KEY (service_role secret) from creating a database from Supabase first, and go to API Settings.

The OPENAI_API_KEY you can get it here.

Code

Our Project structure will look like this:

node_modules
.env
article.txt
embed.js
openAi.js
package.json
server.js
supabase.js

article.txt is the content of the article that we want to search for. So make sure to copy the content of the article there first.

Let's create and store the article chunk's embedding first:

1.Get Article Data

We need to get the text from the article.txt

Before that, let's import the necessary library first.

2.Chunk the text

What do we do here?

OpenAI has a token system. 1 token means +- 1 word. What we want now is we chunk the article, and split it into limited CHUNK_LIMIT (200 tokens) each. We chunk it, so we can search for it later.

The function first checks if the length of the encoded article is greater than the CHUNK_LIMIT. If so, it splits the article into individual sentences and then concatenates them into chunks that are less than or equal to the CHUNK_LIMIT. It also ensures that a sentence is not split across different chunks by checking if the last character of a sentence is alphanumeric.

Each chunk is represented as an object with four properties: content (the text content of the chunk), content_length (the length of the text content), content_tokens (the number of tokens in the content), and embedding. embedding will be added later.

If the resulting chunks are smaller than the CHUNK_MINIMAL (100 Tokens ), they are merged with the previous chunk. Finally, the function returns an array of the resulting chunks.

3.Create Chunk's Embedding

const { Configuration, OpenAIApi } = require('openai');
const configuration = new Configuration({
    apiKey: process.env.OPENAI_API_KEY,
});
const openAi = new OpenAIApi(configuration);

const createEmbedding = async (input) => {
    const embeddingRes = await openAi.createEmbedding({
        model: 'text-embedding-ada-002',
        input: input
    });

    const [{embedding}] = embeddingRes.data.data;
    return embedding
}

module.exports = {
    createEmbedding
}

We need OpenAI's API Create Embedding to create the embedding of the chunked text.

4.Store them in Postgres DB

Before we store them, let's create the table first via the Supabase SQL editor:

create table semantic_search_poc (
  id bigserial primary key,
  content text,
  content_tokens bigint,
  embedding vector (1536)
);

You can name it whatever you want. I named it semantic_search_poc so our query will be like this:

Now run the script

node embed.js

If the process is going smoothly, we will able to see the data in the Supabase table view, something like this:

Done! You have success to build the most crucial part of this project!

Now to the next step, queried the data.

5.Get Query from User

We use Express JS to create Rest API Server to get the query from the user. Create simple Express js server code:

6.Create embedding based on query

//server.js
app.get('/', async (req, res) => {
    const { q } = req.query;
    const embedding = await openAiHelper.createEmbedding(q);
})

7.Search it in Postgres

We use the Postgres function for this search function. So let's head out to the Supabase function editor.

create or replace function semantic_search (
  query_embedding vector(1536),
  similiarity_threshold float,
  match_count int
)

returns table (
  id bigint,
  content text,
  content_tokens bigint,
  similiarity float
)
language plpgsql
as $$
begin
  return query
  select
    semantic_search_poc.id,
    semantic_search_poc.content,
    semantic_search_poc.content_tokens,
    1 - (semantic_search_poc.embedding <=> query_embedding) as similiarity
  from semantic_search_poc
  where 1 - (semantic_search_poc.embedding <=> query_embedding) > similiarity_threshold
  order by semantic_search_poc.embedding <=> query_embedding
  limit match_count;
end;
$$;

We create a Postgres script that creates a function called "semantic_search". The function takes in a query embedding, similarity threshold, and match count as inputs and returns a table with columns for ID, content, content tokens, and similarity.

The function uses the "semantic_search_poc" table to find rows where the similarity between the query embedding and the content embedding is greater than the similarity threshold. It then orders the results by similarity and limits the output to the specified match count.

The Postgres function is done, now call it on the code

Run the server:

node server.js

Now try to hit GET / with q params something that you want to ask.

Voila! The API returns the related content ordered by the most similar one.

I hope this article helps to understand how we can create semantic-search apps easily today.

Resources

GitHub for this project: https://github.com/fandyaditya/semantic-search-poc

References: https://github.com/mckaywrigley/paul-graham-gpt

S: Single Responsibility Principle

O: Open/Closed Principle

L: Liskov Substitution Principle

I: Interface Segregation Principle

D: Dependency Inversion Principle

Each character defines a specific rule on how we can achieve a good code. Especially on Object Oriented Programming (OOP).

Today, I found my old notes from a couple of years ago about SOLID, and I think it is still good and relevant now. Here they are:

[S] Single Responsibility Principle

A class should have just one reason to change.

What [S]

When building classes, it's important to consider the context of the class itself. Don't add methods that don't align with the function and responsibility of the class.

Why [S]

Reducing the complexity of a program is crucial. The more changes made to a program, the harder it becomes to maintain. One way to simplify the process is by creating well-organized classes with functions that align with the class name.

How [S]

class User {
    String name;
    String email;

    User(String name, String email) {
        this.name = name;
        this.email = email;
    }

    String getName() {
        return this.name;
    }

    String getEmail() {
        return this.email;
    }

    void makePurchase() {
        .....
    }
}

makePurchase method not relevant to User class. By referring to the Single Responsibility Principle, we can transform it like this.

class User {
    String name;
    String email;

    User(String name, String email) {
        this.name = name;
        this.email = email;
    }

    String getName() {
        return this.name;
    }

    String getEmail() {
        return this.email;
    }
}

class Purchase {
    User user;
    Cart cart;
    Ship ship;

    Purchase(User user, Cart cart, Ship ship) {
        this.user = user;
        this.cart = cart;
        this.ship = ship;
    }
    void checkout() {
        ...
    }
}

class Purchase is added to do the purchasing stuff

[O] Open/Closed Principle

Classes should be open for extension but closed for modification

What [O]

When implementing updates to a program, it's important to create new classes that are related to the updates, rather than directly modifying existing code.

Why [O]

By creating new classes, we can avoid inadvertently breaking existing code that is already functioning as intended.

How [O]

class Item {
    int price;
    String name;
    String type;

    Item(int price, String name, String type) {
        this.price = price;
        this.name = name;
        this.type = type;
    }

    int getTruePrice() {
        if (this.type == “second”) {
            return price * 80%
        }
        else if (this.type == “luxury”) {
            return price * 130%
        }
    }
}

If we want to add a new type to the getTruePrice() method, we would typically need to modify the code within the Item class. However, by applying the Open/Closed principle, we can transform our approach to look like this:

interface IType {
    int getTruePrice(int price);
}

We create interface IType that used for the implementation of the method getTruePrice for every class Type that we want to add.

class Second implement IType {
    int getTruePrice(price) {
        return price * 80%;
    }
}

class Luxury implement IType {
    int getTruePrice(price) {
        return price * 130%;
    }
}

class Donation implement IType {
    int getTruePrice(price) {
        return (price + 10000)*105%;
    }
}

Now we modify the code for class Item

class Item {
    int price;
    String name;
    IType type;

    Item(int price, String name, IType type) {
        this.price = price;
        this.name = name;
        this.type = type;
    }

    int getTruePrice() {
        return this.type.getTruePrice(this.price)
    }
}

We can create instances of class Item with the following way:

Donation donationItem = new Donation()
Item item = new Item(500000, 'Guitar', donationItem )

When we need a new item type, we just need to create a new type class that implements interface IType

[L] Liskov Substitution Principle

When extending a class, remember that you should be able to pass objects of the subclass in place of objects of the parent class without breaking the client code.

What [L]

An approach to creating sub-classes that is always compatible with the parent class" in English

Why [L]

Preventing broken code when a subclass is used in code but in fact not compatible with their parent class

How [L]

//Parent Class
class Robot{
    moving(Destination destination)
    firing(Object object)
}

//Subclass
class HumanRobot extend Robot{
    moving(Destination destination)
    firing(Object object) {
        throw new Exception (“Human Robot is friendly robot, i made to help human job”)
    }
}
class WingsRobot extend Robot{
    moving(Destination destination)
    flying(Destionation destination)
    firing(Object object)
}

When we create a new class by extending class HumanRobot

Class ShopKeeperRobot extend HumanRobot {
    @override
    moving(Destination destination)
    @override
    firing(Object object)
}

ShoperKeeperRobot skRobot = new ShopKeeperRobot()
skRobot.firing(someObject)

skRobot.firing seems like ok to call, but in fact, it makes the program closed. Because method firing() in their parent class HumanRobot throw an exception.

This is not in accordance with the Liskov Substitution principle. We need some change.

class Robot {
    moving()
}

class LandRobot extend Robot {
    moving()
    firing()
}

class WingsRobot extend LandRobot{
    moving(Destination destination)
    firing(Object object)
    flying(Destionation destination)
}

With limiting method on class Robot, it's subclass LandRobot can have more flexibility for adding more methods that it truly needs, without making the parent method obsolete or disabled.

[I] Interface Segregation Principle

Clients should not be forced to depend on methods they do not use

What [I]

Make sure the class implements all methods from the interface they implement. If there is any situation where some method is not needed, separate that method and create a new interface

Why [I]

If in the future we need to change, make it smaller or leaner for example delete some method from the interface, it may potentially become a broken code. Therefore, make sure that methods are truly being used by the implementing class to avoid having to overhaul the interface code.

How [I]

interface Phone {
    void accessCamera()
    void accessNFC()
    void accessBluetooth()
}

class SamsungPhone implement Phone{
    void accessCamera(){...}
    void accessNFC(){...}
    void accessBluetooth(){...}
}

class XiaomiPhone implement Phone{
    void accessCamera(){...}
    void accessNFC(){...}
    void accessBluetooth(){...}
}

class SmartfrenPhone implement Phone {
    void accessCamera(){...}
    void accessNFC(){...}
    void accessBluetooth(){...}
}

All class implementing interface phone. But on SmartfrenPhone because it only creates the old type of smartphone, it does not have an NFC feature. So method accessNFC() on class SmartfrenPhone cannot be used.

We can fix this by following Interface Segregation Principle:

interface Phone {
    void accessCamera()
    void accessBluetooth()
}

interface NewPhone {
    void accessNFC()
}

We create a new interface, and separate it by the feature that smartphones can have

class SamsungPhone implement Phone, NewPhone{
    void accessCamera() {...}
    void accessNFC() {...}
    void accessBluetooth(){...}
}

class XiaomiPhone implement Phone, NewPhone{
    void accessCamera() {...}
    void accessNFC() {...}
    void accessBluetooth(){...}
}

class SmartfrenPhone implement Phone {
    void accessCamera() {...}
    void accessBluetooth(){...}
}

[D] Dependency Inversion Principle

High-level classes shouldn’t depend on low-level classes. Both should depend on abstractions. Abstractions shouldn’t depend on details. Details should depend on abstractions.

What [D]

When we build a module that contains a high-level (business logic application) that uses the low-level module, it should not depend directly on that low-level module. Create an abstraction between them.

Why [D]

Make high-level modules easy to reuse without worrying about the side effects of changes in low-level modules.

How [D]

class Mysql {
    void insert(Object object, String table) {...}
    void update(){...}
    void delete(){...}
}

class Cart {
    Mysql mysql;
    List<item> items;
    Cart() {
        this.mysql = new Mysql()
    }

    void save() {
        this.mysql.insert(items, 'cart')
    }
}

class Cart use Mysl to insert items into the database. But when in the future the team wants to change the database to MongoDB, the code will be broken. By following Dependency Inversion Principle, we can change this:

interface Database {
    void insert(Object object, String table)
    void update(Object object, String table)
    void delete(Object object, String table)
}

class MySql implement Database {
    void insert(Object object, String table) {...}
    void update(Object object, String table) {...}
    void delete(Object object, String table) {...}
}

class Mongo implement Database {
    void insert(Object object, String table) {...}
    void update(Object object, String table) {...}
    void delete(Object object, String table) {...}
}

class Cart {
    Database database;
    List<item> items;
    Cart() {
        this.database = new Mongo()
    }
    void save() {
        this.database.insert(items, ‘cart’)
    }
}

Use interface Database as an abstraction for the database module.

That's all. Thank you for reading!

Everyone is talking about Chat GPT these days, and I'm sure you've already tried it and been amazed by its capability to understand and solve problems, and even perform certain tasks. As of early March, the Open AI Team has released a new API for the Chat GPT model, as well as their speech-to-text model, Whisper. The best part? They're easy to use and super cheap. Just think of all the possibilities and fun things we can create with these!

For today Let's make a simple program to summarize a YouTube video using these two APIs in Node.Js.

Let's start with the diagram:

Diagram

The diagram is pretty straightforward and simple.

Code

Initiate the Node.js project and install this library.

npm install openai
npm install ytdl-core
npm install dotenv

1. openai is Node js client library for OpenAi APIS

2. ytdl-core is a library for downloading the youtube video

3. dotenv for read .env file

Our project structure will be like this

package.json
package-lock.json
index.js
summary.js
.env
node_modules

First, before we jump into coding, get your API key from Open AI API Keys Page

Store it in .env

//.env
OPENAI_API_KEY=your_key

First, create the step that we want to code:

//index js
/*
* 1. Import all necessary library
* 2. Initiate OpenAI Client
* 3. Download the youtube video
* 4. When the download done, transcribe it
* 5. When transcribing done, now summarize it
* 6. Store the summarize text to .txt file
*/

//index js

//1. Import all necessary library
require('dotenv').config();
const { Configuration, OpenAIApi } = require("openai");
const fs = require('fs');
const ytdl = require('ytdl-core');
const summary = require('./summary');

//2. Initiate OpenAI Client
const main = () => {
    const configuration = new Configuration({
      apiKey: process.env.OPENAI_API_KEY,
    });
    const openai = new OpenAIApi(configuration);
    // 3. Download the youtube video
    const videoStream = fs.createWriteStream('video.mp4');
    ytdl(link, { quality: 'lowest'}).pipe(videoStream);
}

/* 4. When the download done, transcribe it
* 5. When transcribing done, now summarize it
* 6. Store the summarize text to .txt file
*/

//index js

//1. Import all necessary library
require('dotenv').config();
const { Configuration, OpenAIApi } = require("openai");
const fs = require('fs');
const ytdl = require('ytdl-core');
const summary = require('./summary');

//2. Initiate OpenAI Client
const main = () => {
    const configuration = new Configuration({
      apiKey: process.env.OPENAI_API_KEY,
    });
    const openai = new OpenAIApi(configuration);
    // 3. Download the youtube video
    const videoStream = fs.createWriteStream('video.mp4');
    ytdl(link, { quality: 'lowest'}).pipe(videoStream);

    //4. When the download done, transcribe it
    videoStream.on('finish', async () => {
            const transcribe = await openai.createTranscription(fs.createReadStream('./video.mp4'), 'whisper-1');
            //5. When transcribing done, summarize it
            summary.main(transcribe.data.text)
    })
}

/* 
* 
* 6. Store the summarize text to .txt file
*/

Now let us code the summary function in summary.js

//summary.js
const { Configuration, OpenAIApi } = require("openai");
require('dotenv').config();
const fs = require('fs');

const main = async (text) => {
    const configuration = new Configuration({
        apiKey: process.env.OPENAI_API_KEY,
    });
    const openai = new OpenAIApi(configuration);
    const summaryStream = fs.createWriteStream('summary.txt');

    const prompt = `Please summary this text: ${text}`

    const completion = await openai.createChatCompletion({
        model: "gpt-3.5-turbo",
        messages: [{role: "user", "content": prompt}]
    });
    //6. Store the summarize text to .txt file
    summaryStream.write(completion.data.choices[0].message.content)
}

module.exports = {
    main
}

And done! Our simple youtube summarizer is ready to use.

For example, I use this video link as a parameter inindex.js

//index.js
require('dotenv').config();

const { Configuration, OpenAIApi } = require("openai");
const fs = require('fs');
const ytdl = require('ytdl-core');
const summary = require('./summary');


const main = (link) => {
 //...all the code above
}

main('https://youtu.be/KO3h_IoqN3g');

Then run in the terminal:

node index.js

Voila! I got the summary of that video in my summary.txt

The text discusses how negative character traits can be turned into positive things if you can figure out how to do it. Using the example of Joan Crawford, who had a difficult childhood and suffered abuse, the author explains how she was able to turn her pain into great acting. Trauma survivors tend to be extremely sensitive to the emotions of others, which can be both positive and negative, but Crawford was able to use this sensitivity to read the minds of her directors and turn her negative experiences into something productive.

Limitation

Due to the beta release of Whisper API, the limit of files that can be transcribed is 25 MB. We can split the video/audio first before we transcribe it.

Thanks for reading! Please comment if there are any questions

I would appreciate any feedback or suggestions that could help enhance the quality and readability of my writing. Thank you!