ChatGPT Token Limit

Question

I want ChatGPT to remember past conversations and have a consistent (stateful) conversation.

I have seen several code of ChatGPT prompt engineering.

There were two ways to design the prompt shown below (pseudo code):

1. Use a single input (Cheap) <- Better if possible

2. Stack all of previous history (Expensive, Token Limitation)

def openai_chat(prompt):
    completions = openai.Completion.create(
        engine = "text-davinci-003",
        prompt = prompt,
        max_tokens = 1024,
        n = 1,
        temperature = 0.8,
    )
    response = completions.choices[0].text.strip()
    return response

# 1. Use a single input
while True:
    prompt = input("User: ")
    completion = openai_chat(prompt)

# 2. Stack all of previous history (prompt + completion)
prompt = ""
while True:
    cur_prompt = input("User: ")
    prompt += cur_prompt  # pseudo code
    completion = openai_chat(prompt)
    prompt += completion  # pseudo code

Is it possible to choose the first way (the cheap one) to have a consistent conversation?

In other words, does ChatGPT remember past history even if the prompt only has the current input?

Answer 1

A small point: ChatGPT is a very specific version of the GPT model which is used for conversations via ChatGPT online. You are using GPT-3. It is a small point, but an important one.

In terms of remembering past conversation; no, GPT-3 does not do this automatically. You will need to send the data in via the prompt.

There are several workarounds, while none perfect that can be used.

1. Summarize the previous conversation.

   Get GPT-3 to summarize the previous conversations so that it can be provided in the next prompt. You will lose some meaning, but it will reduce your total prompt count.

2. Save previous conversations as a vector embedding, and use vector search to find the most relevant part of the previous conversation and send this via the prompt. This is much more complex and will require an understanding of the GPT-3 embeddings endpoint. But it might solve the problem of losing meaning from the previous prompt.

Answer 2

Tl;Dr: OpenAI API GPT-3 models don't use previously sent input data to generate a new response, but with other models you can use fine-tuning which among other things allows you to send shorter prompts.

---

OpenAI announced today, as I read in their email newsletter, the launch of the official ChatGPT API. In the OpenAI API documentation, it appears as Chat Completions. It has a guide section and a reference section.

It looks that the code you are using is obsolete as it uses "engine" and this API was deprecated (ref. Engines | OpenAI API Reference). Nowadays, instead of engines, OpenAI APIs are using model, i.e., according to https://platform.openai.com/docs/guides/chat ChatGPT (the web app, https://chat.openai.com/chat) is powered by the gpt-3.5-turbo which is a model, not an engine.

The Chat Completions Guide explains that the main input is an array of messages objects. These objects specify the role that might be system, user or assistant of the message object, and mentions that including the history of the conversation might be modified by the developer.

Long story short, on each call to the brand new official ChatGPT API you should send an array of message objects having all the data required for the model to build the response. It does not use information of previous calls.

Regarding the token limit, from https://platform.openai.com/docs/guides/chat/managing-tokens

... as total tokens must be below the model’s maximum limit (4096 tokens for gpt-3.5-turbo-0301)

Both input and output tokens count toward these quantities.

Each model has its own capacity and each of them has its own price by token. OpenAI says (taken from the Chat Completions Guide)

Because gpt-3.5-turbo performs at a similar capability to text-davinci-003 but at 10% the price per token, we recommend gpt-3.5-turbo for most use cases.

Fine-tuning

According to the OpenAI API Fine Tuning guide

Fine-tuning is currently only available for the following base models: davinci, curie, babbage, and ada.

Reference

• Introducing ChatGPT and Whisper APIs