How to send the output of a long running Python script over a WebSocket?

Question

A user uploads a python file that I need to execute on my server and send back the stdout that's created over a WebSocket. The python file that's executed will run for several minutes and I need to return the stdout over a socket as they are "printed" out in real-time, not at the completion of the script.

I've tried using: Python. Redirect stdout to a socket, but that's not a WebSocket and my React frontend can't connect to it successfully. (if you can solve that, that would also solve my problem)

I've also tried using websocketd but since I can't add sys.stdout.flush() after each of the users' added print statements it doesn't solve my problem.

I've also tried using subprocess's PIPE functionality but that has the same flush issue

async def time(websocket, path):
    while True:
        data = "test"
        await websocket.send(data)
        # Run subprocess to execute python file in here
        # sys.stdout => websocket.send             

start_server = websockets.serve(time, "127.0.0.1", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

This is the python test script I am using:

from time import sleep
for i in range(40):
    print(i)
    sleep(0.1)

Answer 1

This stand-alone example will

1. Read a python script from a web socket
2. Write the script to the file system
3. Run the script with output buffering disabled
4. Read the script output one line at a time
5. Write each line of output to the web socket

import asyncio
import websockets
import subprocess

async def time(websocket, path):
    script_name = 'script.py'
    script = await websocket.recv()
    with open(script_name, 'w') as script_file:
        script_file.write(script)
    with subprocess.Popen(['python3', '-u', script_name],
                          stdout=subprocess.PIPE,
                          bufsize=1,
                          universal_newlines=True) as process:
        for line in process.stdout:
            line = line.rstrip()
            print(f"line = {line}")
            await websocket.send(line)

start_server = websockets.serve(time, "127.0.0.1", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

I used this javascript code to test the server:

const WebSocket = require('ws');
let socket = new WebSocket("ws://127.0.0.1:5678");

socket.onopen = function(e) {
    let script = '\
import time\n\
for x in range(100):\n\
    print(f"x = {x}")\n\
    time.sleep(0.25)\n\
';
    console.log("sending data...");
    socket.send(script);
    console.log("done.");
};

socket.onmessage = function(event) {
    console.log(event.data.toString());
};

socket.onerror = function(event) {
    console.log(event);
};

The use of Popen is based on an answer to this question:

Read streaming input from subprocess.communicate()

The -u option is passed to python to disable output buffering.