How can I use the Map_Reduce chain instead of the "stuff" chain in my ConversationalRetrievalChain

Question

I followed this tutorial: https://redis.com/blog/build-ecommerce-chatbot-with-redis/ to create an e-commerce chatbot. It uses ConversationalRetrievalChain that uses two chains, one is a question creating chain and another is question answering chain (code given below)

# use the LLM Chain to create a question creation chain 
question_generator = LLMChain(     
llm=llm,     
prompt=condense_question_prompt )  
# use the streaming LLM to create a question answering chain 
doc_chain = load_qa_chain(    
 llm=streaming_llm,     
chain_type="stuff",     
prompt=qa_prompt )  

My ConversationalRetrievalChain looks like this:

chatbot = ConversationalRetrievalChain(
    retriever=vectordb.as_retriever(),
    combine_docs_chain=doc_chain,
    question_generator=question_generator,
    
)

The problem is, I want to use map_reduce chain instead of stuff because, sometimes, when the relevant data is too large, it gives token limit error. I tried using mapreduce but I cannot use prompt=qa_prompt with it. And when I remove that it shows this error:

ValueError: Cannot stream results with multiple prompts. on these code of lines

result = chatbot(
{"question": question, "chat_history": chat_history}     )    
print("\n")
chat_history.append((result["question"], result["answer"]))
question = input() 

Is there any solution to this? How can I use mapreduce with this chain? Or is there any other way that can help me reduce the tokens being sent at a time?

Secondly, I have noticed that sometimes the model return a very long response. I have set my max_tokens to 150, so sometimes when the response is longer than 150, it stops mid-sentence. Is there any way that I can change this?

Please note that I am relatively new to this so any explanations or step-by-step guidance would be greatly appreciated.

Here is the complete code:

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import Chroma
from langchain.document_loaders import CSVLoader
import os
loader = CSVLoader('adidas_usa.csv')
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
docs = text_splitter.split_documents(documents)



embeddings = OpenAIEmbeddings()
OPENAI_API_KEY = 'sk-X'

 
# set your openAI api key as an environment variable
os.environ['OPENAI_API_KEY'] = "sk-X"
 
embeddings = OpenAIEmbeddings(openai_api_key=OPENAI_API_KEY)

persist_directory = 'adidas'
# we will use OpenAI as our embeddings provider
embedding = OpenAIEmbeddings()
docsearch = Chroma.from_documents(documents=docs, embedding=embedding, persist_directory=persist_directory)

Here is the chatbot creation:

import os
from langchain.vectorstores import Chroma
from langchain.embeddings.openai import OpenAIEmbeddings

from langchain.callbacks.base import BaseCallbackManager as CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain.chains import (
    ConversationalRetrievalChain,
    LLMChain
)
from langchain.chains.question_answering import load_qa_chain
from langchain.llms import OpenAI
from langchain.prompts.prompt import PromptTemplate


OPENAI_API_KEY = 'sk-XXXXX'
os.environ['OPENAI_API_KEY'] = "sk-XXXX"
persist_directory = 'adidas'
# we will use OpenAI as our embeddings provider
embedding = OpenAIEmbeddings()
vectordb = Chroma(persist_directory=persist_directory, embedding_function=embedding)


template = """Given the following chat history and a follow up question, rephrase the follow up input question to be a standalone question.


Chat History:\"""
{chat_history}
\"""

Follow Up Input: \"""
{question}
\"""

Standalone question:"""

condense_question_prompt = PromptTemplate.from_template(template)

template = """You are a friendly Shopping E-commerce Assistant, designed to assist with a variety of tasks related to online shopping. Assistant can answer questions, provide detailed explanations, and engage in natural-sounding conversations about various products and services available for purchase, by using the context given. The Assistant continually learns and improves, utilizing its vast knowledge base to offer accurate and informative responses. Assitant can also generate its own text to discuss, describe, and recommend products to users. Assistant can understand the question well and answer accordingly.
Context:\"""
{context}
\"""
 
Question:\"
\"""

Helpful Answer:"""

qa_prompt= PromptTemplate.from_template(template)


# define two LLM models from OpenAI
llm = OpenAI(temperature=0,model='text-davinci-003')
# llm=OpenAI()

streaming_llm = OpenAI(
    streaming=True,
    model='text-davinci-003',
    callback_manager=CallbackManager([
        StreamingStdOutCallbackHandler()]),
    verbose=True,
    temperature=0.2,
    max_tokens=150
)
# use the LLM Chain to create a question creation chain
question_generator = LLMChain(
    llm=llm,
    prompt=condense_question_prompt
)

# use the streaming LLM to create a question answering chain
doc_chain = load_qa_chain(
    llm=streaming_llm,
    chain_type="stuff",
    prompt=qa_prompt
)


chatbot = ConversationalRetrievalChain(
    retriever=vectordb.as_retriever(),
    combine_docs_chain=doc_chain,
    question_generator=question_generator,
    
)

# create a chat history buffer
chat_history = []

# gather user input for the first question to kick off the bot
question = input("Hi! What are you looking for today?")

# keep the bot running in a loop to simulate a conversation
while True:
    result = chatbot(
        {"question": question, "chat_history": chat_history}
    )
    print("\n")
    chat_history.append((result["question"], result["answer"]))
    question = input()

Answer 1

The "map_reduce" chain type requires a different, slightly more complex type of prompt for the combined_documents_chain component of the ConversationalRetrievalChain compared to the "stuff" chain type:

• (default) "stuff" prompt: https://github.com/hwchase17/langchain/blob/e74733ab9e5e307fd828ea600ea929a1cb24320f/langchain/chains/question_answering/stuff_prompt.py
• (default) "map_reduce" prompt: https://github.com/hwchase17/langchain/blob/e74733ab9e5e307fd828ea600ea929a1cb24320f/langchain/chains/question_answering/map_reduce_prompt.py

This is because the load_qa_chain for the "map_reduce" chain type is more complex (see source code) compared to the load_qa_chain for the "stuff" chain type (source code).

The easiest way to set this up is simply to specify:

    chatbot = ConversationalRetrievalChain.from_llm(
                    llm=<YOUR QA LLM>, 
                    condense_question_llm=<YOUR QUESTION REFORMULATION LLM>,
                    retriever=<YOUR RETRIEVER>,
                    chain_type="map_reduce",
                    memory = <YOUR MOMORY>, 
                    verbose=True,
                    return_generated_question=<BOOL>,
                    get_chat_history=lambda h : h, 
                    return_source_documents=<BOOL>)

You can then see and modify the relevant prompts:

chatbot.combine_docs_chain.llm_chain.prompt
chatbot.combine_docs_chain.combine_document_chain.llm_chain.prompt