Giving SystemMessage/Context to ConversationalRetrievalChain and ConversationBufferMemory in Lang Chain – Python

The Problem:

I’m trying to build a chatbot that can chat about pdfs, and I got it working with memory using ConversationBufferMemory and ConversationalRetrievalChain like in this example. https://python.langchain.com/en/latest/modules/chains/index_examples/chat_vector_db.html

Now I’m trying to give the AI some special instructions to talk like a pirate (just for testing to see if it is receiving the instructions). I think this is meant to be a SystemMessage, or something with a prompt template?

I’ve tried everything I have found, but all the examples in the documentation are for ConversationChain and I end up having problems with. So far the only thing that hasn’t had any errors is this:

template = """Given the following conversation respond to the best of your ability in a pirate voice and end every sentence with Ay Ay Matey
Chat History:
{chat_history}
Follow Up Input: {question}
Standalone question:"""
PROMPT = PromptTemplate(
    input_variables=["chat_history", "question"], template=template
)
memory = ConversationBufferMemory(memory_key='chat_history', return_messages=True, output_key='answer')
qa = ConversationalRetrievalChain.from_llm(OpenAI(temperature=0), vectorstore.as_retriever(), PROMPT, memory=memory, return_source_documents=True)

It still doesn’t have any effect on the results, so I don’t know if it is doing anything at all. I also think it’s the wrong approach, and I should be using SystemMessages (maybe on the memory, not the qa), but nothing I try from the documentation works and I’m not sure what to do.

The Solutions:

Solution 1: Can’t pass PROMPT directly when using ConversationalRetrievalChain

You can’t directly pass the `PROMPT` as a parameter in `ConversationalRetrievalChain.from_llm()`. Instead, use the `combine_docs_chain_kwargs` parameter to pass the customized `PROMPT`.

template = """Given the following conversation respond to the best of your ability in a pirate voice and end every sentence with Ay Ay Matey
Chat History:
{chat_history}
Follow Up Input: {question}
Standalone question:"""

PROMPT = PromptTemplate(
    input_variables=["chat_history", "question"], 
    template=template
)

memory = ConversationBufferMemory(
    memory_key='chat_history', 
    return_messages=True, 
    output_key='answer'
)

qa = ConversationalRetrievalChain.from_llm(
    llm=OpenAI(temperature=0),
    retriever=vectorstore.as_retriever(),
    memory=memory,
    return_source_documents=True,
    combine_docs_chain_kwargs={"prompt": PROMPT}
)

Then, retrieve the result:

result = qa({"question": query})

Solution 2: Wrap SystemMessagePromptTemplate in a ChatPromptTemplate

To pass a system message to a prompt passed to ConversationalRetrievalChain using ChatOpenAI, wrap SystemMessagePromptTemplate in a ChatPromptTemplate. Ensure to include `context` in the template string for recognition.

from langchain.chat_models import ChatOpenAI
from langchain.prompts import SystemMessagePromptTemplate, ChatPromptTemplate, HumanMessagePromptTemplate
from langchain.chains import ConversationalRetrievalChain

messages = [
    SystemMessagePromptTemplate.from_template("talk like a pirate"),
    HumanMessagePromptTemplate.from_template("{question}")
]
prompt = ChatPromptTemplate.from_messages(messages=messages)

qa = ConversationalRetrievalChain.from_llm(
    llm=ChatOpenAI(), 
    retriever=vector_store.as_retriever(), 
    combine_docs_chain_kwargs={"prompt": prompt}
)

result = qa({"question": question, "chat_history": chat_history})

This approach proves useful for informing the bot, for instance, of the current date:

from datetime import datetime
current_date = f'Current date: {datetime.now().strftime("%A, %B %d, %Y")}'

messages = [
    SystemMessagePromptTemplate.from_template(current_date),
    HumanMessagePromptTemplate.from_template("{question}")
]
prompt = ChatPromptTemplate.from_messages(messages=messages)

qa = ConversationalRetrievalChain.from_llm(
    llm=ChatOpenAI(), 
    retriever=vector_store.as_retriever(), 
    combine_docs_chain_kwargs={"prompt": prompt}
)

result = qa({"question": question, "chat_history": chat_history})

Video Explanation: