Weaviate: Stop Building AI the Hard Way

Are you ready to shape what’s next, or will you leave it to others?

Gen AI Launch Pad 2025 puts the tools in your hands.

Weaviate is an open-source vector database designed to simplify the development and scaling of AI applications. It provides a robust platform for storing and querying vector data, enabling developers to build and manage AI-driven solutions efficiently. In this blog, we'll walk through the process of setting up Weaviate, populating it with data, and performing semantic searches and retrieval-augmented generation (RAG) tasks.

1.Introduction to Weaviate

2.Setting Up Weaviate

Installing the Weaviate Client
Setting Up API Keys
Connecting to Weaviate

3.Populating the Database

Defining a Collection
Fetching and Loading Data
Adding Objects to the Collection

4.Performing Semantic Search

5.Retrieval-Augmented Generation (RAG)

6.Conclusion and Resources

1. Introduction to Weaviate

Weaviate is an AI-native vector database optimized for AI applications. It integrates seamlessly with machine learning models and frameworks, offering hybrid search capabilities that support both vector and keyword search. This allows for semantic understanding and precise retrieval of data. Weaviate is scalable, flexible, and designed for real-time data processing, making it an ideal choice for AI-driven applications.

2. Setting Up Weaviate

Installing the Weaviate Client

To get started with Weaviate, you'll need to install the Weaviate client. You can do this using pip:

!pip install -U weaviate-client

Setting Up API Keys

Next, you'll need to set up your API keys. These keys are required to connect to Weaviate and other services like Cohere and OpenAI. If you're using Google Colab, you can store your API keys securely using the userdata module.

from google.colab import userdata

WCD_URL = userdata.get('WCD_URL')
WCD_API_KEY = userdata.get('WCD_API_KEY')
OPENAI_API_KEY = userdata.get('OPENAI_API_KEY')
cohere_api_key = userdata.get('COHERE_API_KEY')

Connecting to Weaviate

Once you have your API keys, you can connect to Weaviate using the weaviate-client library. Here's how you can do it:

import weaviate
from weaviate.classes.init import Auth

wcd_url = WCD_URL
wcd_api_key = WCD_API_KEY

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=wcd_url,
    auth_credentials=Auth.api_key(wcd_api_key),
)

print(client.is_ready())
client.close()

This code connects to the Weaviate cloud instance and checks if the connection is ready. If everything is set up correctly, it should return True.

3. Populating the Database

Defining a Collection

Before you can add data to Weaviate, you need to define a collection. A collection is similar to a table in a relational database. Here's how you can create a collection named "Question" with a Cohere vectorizer:

from weaviate.classes.config import Configure

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=wcd_url,
    auth_credentials=Auth.api_key(wcd_api_key),
)

questions = client.collections.create(
    name="Question",
    vectorizer_config=Configure.Vectorizer.text2vec_cohere(),
    generative_config=Configure.Generative.cohere()
)

client.close()

Fetching and Loading Data

Now that you have a collection, you can populate it with data. For this example, we'll use a sample dataset from a JSON file hosted on GitHub:

import requests
import json

resp = requests.get(
    "https://raw.githubusercontent.com/weaviate-tutorials/quickstart/main/data/jeopardy_tiny.json"
)
data = json.loads(resp.text)

Adding Objects to the Collection

With the data fetched, you can now add it to the "Question" collection. We'll use the batch.dynamic() method to add multiple objects efficiently:

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=wcd_url,
    auth_credentials=Auth.api_key(wcd_api_key),
    headers={"X-Cohere-Api-Key": cohere_api_key},
)

questions = client.collections.get("Question")

with questions.batch.dynamic() as batch:
    for d in data:
        batch.add_object({
            "answer": d["Answer"],
            "question": d["Question"],
            "category": d["Category"],
        })

client.close()

4. Performing Semantic Search

One of the key features of Weaviate is its ability to perform semantic searches. This allows you to search for data based on the meaning of the query rather than just keywords. Here's how you can perform a semantic search for the term "biology":

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=wcd_url,
    auth_credentials=Auth.api_key(wcd_api_key),
    headers={"X-Cohere-Api-Key": cohere_api_key},
)

questions = client.collections.get("Question")

response = questions.query.near_text(
    query="biology",
    limit=2
)

for obj in response.objects:
    print(json.dumps(obj.properties, indent=2))

client.close()

This code will return the top 2 results that are semantically related to "biology".

5. Retrieval-Augmented Generation (RAG)

Retrieval-augmented generation (RAG) is a technique that combines retrieval-based and generative models to produce more accurate and contextually relevant responses. Here's how you can use Weaviate to generate a tweet based on the retrieved data:

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=wcd_url,
    auth_credentials=Auth.api_key(wcd_api_key),
    headers={"X-Cohere-Api-Key": cohere_api_key},
)

questions = client.collections.get("Question")

response = questions.generate.near_text(
    query="biology",
    limit=2,
    grouped_task="Write a tweet with emojis about these facts."
)

print(response.generated)

client.close()

This code will generate a tweet with emojis based on the retrieved facts about biology.

6. Conclusion and Resources

Weaviate is a powerful tool for building AI-driven applications. Its AI-native architecture, hybrid search capabilities, and real-time data processing make it an ideal choice for developers looking to scale their AI solutions. In this blog, we walked through the process of setting up Weaviate, populating it with data, and performing semantic searches and retrieval-augmented generation tasks.

Resources

---------------------------

Stay Updated:- Follow Build Fast with AI pages for all the latest AI updates and resources.

Experts predict 2025 will be the defining year for Gen AI Implementation. Want to be ahead of the curve?

Join Build Fast with AI’s Gen AI Launch Pad 2025 - your accelerated path to mastering AI tools and building revolutionary applications.