Chain of thought prompt using OpenAI to query an order list providing incorrect answer

Question

I'm new to prompt engineering. I'm trying to create an AI bot to retrieve the order data based on the user's queries using OpenAI and gpt-3.5-turbo model. Below is the Python code that also contains the prompt:

from openai import OpenAI
import os
os.environ["OPENAI_API_KEY"] = "sk-*****************************************"

client = OpenAI()
prompt = """
You are a helpful AI bot who can answer queries on orders that were processed by a logistics company. Use the orders list under the header "Order list:" to answer the question under header "Question:". Please follow the following steps to generate the response:
Loop through each order in the orders list and perform the below steps:
    a. Check thoroughly and carefully twice whether the details in the order satisfy the details requested in the user's query
    b. If the details in the order satisfy the details requested in the user's query, then print the order details and explain how you concludes that the order satisfies the user's query
    c. If the details in the order do not satisfy the details requested in the user's query, do not print anything

Order list:
1. Order number: TM65432223
   Origin location: Dallas
   Destination location: Chicago
   Reference numbers: 23232, 6543, 98765
   Order status: Pending
   Present location: Virginia

2. Order number: TM09876543
   Origin location: Dallas
   Destination location: Los Angeles
   Reference numbers: 65432, 678903, 345678
   Order status: Pending
   Present location: Washington

3. Order number: TM12323232
   Origin location: Chicago
   Destination location: Dallas
   Reference numbers: 23232, 32443534, 321312
   Order status: Pending
   Present location: Los Angeles

4. Order number: TM56789012
   Origin location: Houston
   Destination location: Austin
   Reference numbers: 1235, 7890, 45556
   Order status: Pending
   Present location: New york

Question:
Give me the list of the orders whose destination location is New york.
"""

response = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "user", "content": prompt}],
    temperature=0
)
print(response.choices[0].message.content)

But the response generated for the question in the above prompt is always wrong. Below is the response:

Order number: TM56789012
Origin location: Houston
Destination location: Austin
Reference numbers: 1235, 7890, 45556
Order status: Pending
Present location: New york

Explanation: The order with Order number OX56789012 has a destination location of New York, as mentioned in the user's query. Therefore, this order satisfies the user's query.

Could someone please tell me if I'm missing something? Thanks.

Answer 1

gpt-3.5-turbo struggles with this problem even if you experiment with the prompt, but gpt-4 and gpt-4-turbo are up to the task. With your prompt they give an overly verbose reply, but I found that if you give this slightly simplified prompt they both get it right (although to make it completely silent if there is no match, you will have to experiment with the prompt a little further):

Order list:
1. Order number: TM65432223
   Origin location: Dallas
   Destination location: Chicago
   Reference numbers: 23232, 6543, 98765
   Order status: Pending
   Present location: Virginia

2. Order number: TM09876543
   Origin location: Dallas
   Destination location: Los Angeles
   Reference numbers: 65432, 678903, 345678
   Order status: Pending
   Present location: Washington

3. Order number: TM12323232
   Origin location: Chicago
   Destination location: Dallas
   Reference numbers: 23232, 32443534, 321312
   Order status: Pending
   Present location: Los Angeles

4. Order number: TM56789012
   Origin location: Houston
   Destination location: Austin
   Reference numbers: 1235, 7890, 45556
   Order status: Pending
   Present location: New york

Question:
Give me the list of the orders whose destination location is New york.

If the details in the order satisfy the details requested in the user's query, then print the order details and explain how you concludes that the order satisfies the user's query
If the details in the order do not satisfy the details requested in the user's query, do not print anything

This is the reply from gpt-4-turbo with zero temperature:

None of the orders in the list have New York as their destination location. Therefore, I cannot provide any order details that satisfy the user's query.

And when I edited one of the orders to have a destination of New York, gpt-4-turbo gave this response:

Order number: TM09876543
Origin location: Dallas
Destination location: New York
Reference numbers: 65432, 678903, 345678
Order status: Pending
Present location: Washington

Explanation:
This order satisfies the user's query because the destination location specified in the order details is "New York," which matches the destination location requested in the query.