How do I Stream OpenAI's completion API?

Question

I want to stream the results of a completion via OpenAI's API.

The doc's mention using server-sent events - it seems like this isn't handled out of the box for flask so I was trying to do it client side (I know this exposes API keys). However, because the OpenAI API requires it post it seems like it isn't compatible with the eventSource API. I tried doing it via a fetch (Using readable streams) but when I try to convert to JSON via the example I get the following error: Uncaught (in promise) SyntaxError: Unexpected token 'd', "data: {"id"... is not valid JSON (I know this isn't valid JSON). It seems like it is parsing the entire result not each individual stream.

data: {"id": "cmpl-5l11I1kS2n99uzNiNVpTjHi3kyied", "object": "text_completion", "created": 1661887020, "choices": [{"text": " to", "index": 0, "logprobs": null, "finish_reason": null}], "model": "text-davinci-002"}

data: {"id": "cmpl-5l11I1kS2n99uzNiNVpTjHi3kyied", "object": "text_completion", "created": 1661887020, "choices": [{"text": " AL", "index": 0, "logprobs": null, "finish_reason": null}], "model": "text-davinci-002"}

data: {"id": "cmpl-5l11I1kS2n99uzNiNVpTjHi3kyied", "object": "text_completion", "created": 1661887020, "choices": [{"text": "I", "index": 0, "logprobs": null, "finish_reason": null}], "model": "text-davinci-002"}

Would love some pointers or a simple code example of how to do this because I've been banging my head against it for a while. Thanks!

Answer 1

Finally got this working code:

import { Configuration, OpenAIApi } from "openai";
import dotenv from "dotenv";
dotenv.config({ override: true });

const openai = new OpenAIApi(new Configuration({ apiKey: process.env.OPENAI_KEY }));

const getText = async (prompt, callback) => {
    const completion = await openai.createCompletion(
        {
            model: "text-davinci-003",
            prompt: prompt,
            max_tokens: 1000,
            stream: true,
        },
        { responseType: "stream" }
    );
    return new Promise((resolve) => {
        let result = "";
        completion.data.on("data", (data) => {
            const lines = data
                ?.toString()
                ?.split("\n")
                .filter((line) => line.trim() !== "");
            for (const line of lines) {
                const message = line.replace(/^data: /, "");
                if (message == "[DONE]") {
                    resolve(result);
                } else {
                    let token;
                    try {
                        token = JSON.parse(message)?.choices?.[0]?.text;
                    } catch {
                        console.log("ERROR", json);
                    }
                    result += token;
                    if (token) {
                        callback(token);
                    }
                }
            }
        });
    });
};
    
console.log(await getText("Who was the latest president of USA?", (c) => process.stdout.write(c)));