How to make LangChain's chatbot answer only from the knowledge base I provided? It shouldn't have access to OpenAI's GPT-3.5 knowledge bas – Langchain

The Problem:

Given a chatbot trained on both an OpenAI knowledge base and a custom knowledge base, how can we construct the chatbot in a way that it only draws information from the custom knowledge base? We want to prevent the chatbot from accessing or incorporating the OpenAI knowledge base in its responses and source document retrievals.

The Solutions:

Solution 1: Use Proper Prompt Roles

To ensure accurate and brief responses from the chatbot, define a proper prompt role that guides the response generation process.
The prompt role should incorporate the following guidelines:

• Instruct the AI assistant to provide answers solely based on the given context.

• Emphasize the importance of accuracy and brevity in responses.

• Define specific rules for handling scenarios when an answer cannot be found within the context:

  • Instruct the chatbot to acknowledge that the information is not available within the given context.
  • Instead of attempting to provide an answer, the chatbot should respond with a statement indicating that it does not have the necessary information.

• Provide clear instructions for handling examples:

  • Specify that the chatbot should only provide examples when explicitly requested.
  • Examples should be relevant to the question and derived from the context.

• Ensure that the chatbot avoids introducing examples or information not found in the context.

• In the absence of context, instruct the chatbot to refrain from providing any response.

• Ensure that the chatbot’s responses are concise, limited to three or four sentences for clarity and conciseness.

By following these guidelines, you can help ensure that the chatbot provides accurate and relevant responses based solely on the knowledge provided in the given context, without accessing external knowledge sources like GPT-3.5.

Solution 2: Use a Similarity Score Threshold

To make the LangChain chatbot answer only from the knowledge base you provided, and not have access to OpenAI’s GPT-3.5 knowledge base, you can use a similarity score threshold when searching for documents in the vector store. Here’s how:

1. Set a Similarity Score Threshold:
   When retrieving documents from the vector store, set a similarity score threshold to filter out documents that have a low similarity score with the query. This ensures that only documents that are highly relevant to the query are retrieved.

   <pre>
   retriever = vectorstore.as_retriever(search_type=&quot;similarity_score_threshold&quot;, search_kwargs={
   &#39;score_threshold&#39;: 0.5})
   </pre>
   

2. Customize the RetrievalQA Chain:
   Modify the RetrievalQA chain to handle cases where no relevant documents are found. You can add a condition to check if any documents were retrieved, and if not, return a message indicating that the chatbot cannot answer the question.

   <pre>
   qa = RetrievalQA.from_chain_type(
       llm=llm,
       chain_type=&quot;stuff&quot;,
       retriever=retriever,
       verbose=True,
       return_source_documents=True
   )
   </pre>
   

3. Update the Agent’s Logic:
   In the agent’s logic, check if any relevant documents were retrieved for a given query. If no documents were retrieved, generate a response indicating that the chatbot cannot answer the question due to lack of relevant information.

   <pre>
   if not result.source_documents:
      result.output = &quot;Sorry, I don&#39;t have information to answer your question.&quot;
   </pre>
   

By implementing these changes, the chatbot will only answer questions that are supported by the provided knowledge base. If the chatbot cannot find any relevant information, it will respond with a message indicating that it lacks the necessary knowledge.

Video Explanation:

The following video, titled "LangChain Crash Course For Beginners | LangChain Tutorial ...", provides additional insights and in-depth exploration related to the topics discussed in this post.

LangChain is an open-source framework that allows you to build applications using LLMs (Large Language Models). In this crash course for ...