php: cannot perform multiple fread() calls on php://input

Question

I'm sending data back and forth to a PHP application using content-encoding: chunked through POSTs. I need my PHP application to read some data, work on it, send back a response, read some more data, and so on. I cannot read all data at once as it won't be available. Imagine a large file upload with a checksum being sent as a response at regular intervals.

The problem is that while I can read a handful of bytes from php://input, subsequent calls to fread do not return the new content.

At the moment I'm using PHP's Docker container. I tried both php:7.0-apache and php:5-apache with the same result.

The PoC client below generates random strings and sends them as chunks to the server at 3-second intervals. The server reads from php://input at 1-second intervals and prints the content. The server output shows only the first three strings are read; also the server seems to 'block' until the first three are read.

Things I've tried, to no avail:

• Using fseek
• Using stream_select does not seem to work with, er, the php://input stream. I have no idea why as this would be ideal for me, but given how poorly PHP is designed and implemented I'm not surprised.
• Closing and re-opening php://input
• Using fgetc

Client output:

    $ python poc.py
Sending:
---
POST /poc.php HTTP/1.1
Host: localhost
accept-encoding: *;q=0
Transfer-Encoding: chunked
Content-Type: application/octet-stream


---

After sending headers, response:
 HTTP/1.1 200 OK
Date: Mon, 29 May 2017 14:25:52 GMT
Server: Apache/2.4.10 (Debian)
X-Powered-By: PHP/5.6.30
transfer-encoding: chunked
Content-Type: application/octet-stream

4
OK


Waiting 3 seconds
Sending string: AuVuvsyGJc

Waiting 3 seconds
Sending string: LfKouYzccV

Waiting 3 seconds
Sending string: WmpPspYqiR

Waiting 3 seconds
Sending string: IApMOjoaIv

Waiting 3 seconds
Sending string: tuGrVklcVy

Waiting 3 seconds
Sending string: btUVIezCow

Waiting 3 seconds
Sending string: XUPOrEidyd

Traceback (most recent call last):
  File "poc.py", line 33, in <module>
    websock.send(to_chunk(rnd))
socket.error: [Errno 32] Broken pipe

Server output:

Connected
Read: AuVuvsyGJc
LfKouYzccV
WmpPspYqiR

Read:
Read:
Read:
Read:
172.17.0.1 - - [29/May/2017:14:25:52 +0000] "POST /poc.php HTTP/1.1" 200 191 "-" "-"

PHP server:

<?php
header("transfer-encoding: chunked");
header("content-type: application/octet-stream");
flush(); 
/**
 * Useful to print debug messages in the Apache logs
 */
function _log($what) {
    file_put_contents("php://stderr", print_r($what, true) . "\n");
}
_log("Connected");

/**
 * To send data as chunks
 */
function _ch($chunk) {
    echo sprintf("%x\r\n", strlen($chunk));
    echo $chunk;
    echo "\r\n";
    flush();
}
// Test chunks
_ch("OK\r\n");

$web_php_input = fopen("php://input", 'r');
$continue = 5;
while ($continue-- > 0) {
    $contents = fread($web_php_input, 1024);
    _log("Read: " . $contents);
    sleep(1);
}
fclose($web_php_input);
?>

Python client:

from __future__ import print_function
import random
import socket
import string
import time

def to_chunk(what):
    return format(len(what), 'X') + "\r\n" + what + "\r\n"

websock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
websock.connect(("localhost", 8080))

# Send the initial chunked POST header
connect_string = ''.join((
    "POST /poc.php HTTP/1.1\r\n",
    "Host: localhost\r\n",
    "accept-encoding: *;q=0\r\n",  # ,gzip;q=0,deflate;q=0\r\n",
    "Transfer-Encoding: chunked\r\n",
    "Content-Type: application/octet-stream\r\n",
    # "Connection: keep-alive\r\n",
    "\r\n",
))
print("Sending:\n---\n{}\n---\n".format(connect_string))
websock.sendall(connect_string)
print("After sending headers, response:\n {}".format(websock.recv(1024)))
c = True
while c:
    print("Waiting 3 seconds")
    time.sleep(3)
    rnd = ''.join(random.choice(string.ascii_letters) for _ in range(10))
    rnd += '\r\n'
    print("Sending string: {}".format(rnd))
    websock.send(to_chunk(rnd))
print("done")

Dockerfile:

FROM php:5-apache
COPY custom.ini /usr/local/etc/php/conf.d

Docker command line:

docker build -t listener .
docker run -i --rm -p 8080:80 -v $(pwd):/var/www/html --name listener listener

custom.ini file to let PHP know that POST body should not be buffered:

enable_post_data_reading=false

Before someone else suggests using another language, or framework, or doing things differently: it has to be PHP; it cannot rely on any third-party library or PECL; and this is precisely what I need.

As a side note, this behaviour is compliant with the HTTP spec; a server does not have to read all inbound data before returning part of a response to a client. See also RFC6202.

Answer 1

In order to understand why this is happening, you need to know how HTTP works which, unfortunately, is not the way you think it does. Chunked transfer-encoding and PHP also don't work the way you think they do. I'll try to explain in a way that is relevant to what I believe you are trying to do.

If I understand correctly, you are trying to send chunks of request and response in an interleaving manner, or sending data back and forth as you describe it. This is a violation of HTTP specification. As such, you won't be able to do so because requests are handled directly by HTTP server, not PHP.

HTTP

HTTP is a request/response protocol (RFC2616 Section 1.4) which has a simple operation:

1. A client sends an HTTP request message to a server.
2. After receiving and interpreting the request message, the server responds with an HTTP response message. (RFC2616 Section 6)

Notice that step 2 says "After", not "While", which means that the server must wait for the request to complete before it can send a response. This is why "the server seems to block".

The life cycles of HTTP Long Polling and HTTP Streaming described in RFC6202 actually work the same way with no violation of HTTP specification. They don't send data back and forth (no interleaving).

Chunked Transfer-Encoding

If the request has Transfer-Encoding: chunked header, the server has to wait for the last chunk. This is decribed in at least two places:

1. In the BNF of Section 3.6.1. Observe that Chunked-Body has to have last-chunk.
2. In the pseudocode of Section 19.4.6. Observe that there is no "send response to client" or anything similar inside the loop (in the entire pseudocode, really).

In short, no interleaving allowed. Chunked transfer-encoding doesn't introduce interleaving and therefore doesn't change the way HTTP works.

PHP

Because the server has to wait for the request, PHP will not be invoked until after the request completes. So when you are sending chunks of data with 3-second delays, your PHP script isn't even running yet.

As for the PHP configuration item enable_post_data_rendering, it doesn't exist. The closest to it is enable_post_data_reading, which simply means that the request body will not be parsed and thus $_FILES and $_POST will be empty. This for effeciency reason: no time spent on parsing the request body and no memory used to hold the values of $_FILES and $_POST. It has nothing to do with POST body buffering.

Let me know if there is something you're still unclear about.

Update

This is the output from my own experiment, with 3-second intervals between events and 15-second socket timeout. The timestamps are useful to determine which events are linked together.

Observe that reading from server always timed-out before the last chunk is sent. Also observe the timestamp 13:43:03 when the last chunk is sent, which is also when PHP is invoked. It shows that the server waited for the last chunk before invoking PHP.

client 13:40:54 opening socket... opened
client 13:40:57 sending request... 130 bytes sent
client 13:41:00 reading from server...
client 13:41:15 timed out
client 13:41:18 sending chunk 0... 14 bytes sent
client 13:41:21 reading from server...
client 13:41:36 timed out
client 13:41:39 sending chunk 1... 14 bytes sent
client 13:41:42 reading from server...
client 13:41:57 timed out
client 13:42:00 sending chunk 2... 14 bytes sent
client 13:42:03 reading from server...
client 13:42:18 timed out
client 13:42:21 sending chunk 3... 14 bytes sent
client 13:42:24 reading from server...
client 13:42:39 timed out
client 13:42:42 sending chunk 4... 14 bytes sent
client 13:42:45 reading from server...
client 13:43:00 timed out
client 13:43:03 sending last chunk... 5 bytes sent
client 13:43:06 reading from server...
client 13:43:06 279 bytes read
client 13:43:06 ---------- start of response
HTTP/1.1 200 OK
Host: localhost
Connection: close
X-Powered-By: PHP/7.0.12
Transfer-Encoding: chunked
Content-Type: application/octet-stream

20
server 2017-06-16 13:43:03 start
2d
13:41:18
13:41:39
13:42:00
13:42:21
13:42:42

1e
server 2017-06-16 13:43:03 end
0

client 13:43:06 ---------- end of response
client 13:43:06 done

This is the server.php:

<?php
while(@ob_end_flush());
header("Transfer-Encoding: chunked");
header("Content-Type: application/octet-stream");

echo chunk("server ".gmdate("Y-m-d H:i:s ")."start");

if($f = fopen("php://input", "r")){
    while($s = fread($f, 1024)){
        echo chunk($s);
    }
    fclose($f);
}

echo chunk("server ".gmdate("Y-m-d H:i:s ")."end");
echo chunk("");

function chunk($s){
    return dechex(strlen($s))."\r\n".$s."\r\n";
}

This is the client.php:

<?php
out("opening socket... ");
if($socket = fsockopen("localhost", 80, $errno, $error)){
    echo "opened\n";
    
    //set socket timeout to 15 seconds
    stream_set_timeout($socket, 15);
    sleep(3);
    
    out("sending request... ");
    $n = fwrite($socket, "POST http://localhost/server.php HTTP/1.1\r\n"
        ."Host: localhost\r\n"
        ."Transfer-Encoding: chunked\r\n"
        ."Content-Type: application/octet-stream\r\n"
        ."\r\n"
    );
    echo "$n bytes sent\n";
    sleep(3);

    readFromServer($socket);
    sleep(3);
    
    for($i=0; $i<5; $i++){
        out("sending chunk {$i}... ");
        $n = fwrite($socket, chunk(gmdate("H:i:s\n")));
        echo "$n bytes sent\n";
        sleep(3);
        readFromServer($socket);
        sleep(3);
    }
    out("sending last chunk... ");
    $n = fwrite($socket, chunk(""));
    echo "$n bytes sent\n";
    sleep(3);

    readFromServer($socket);
    fclose($socket);
}else{
    echo "error\n";
}
out("done\n");

function out($s){
    echo "client ".gmdate("H:i:s ").$s;
}

function chunk($s){
    return dechex(strlen($s))."\r\n".$s."\r\n";
}

function readFromServer($socket){
    out("reading from server... \n");
    $response = fread($socket, 1024);
    $info = stream_get_meta_data($socket);
    if($info['timed_out']){
        out("timed out\n");
    }else{
        out(strlen($response)." bytes read\n");
        if($response){
            out("---------- start of response\n");
            echo $response;
            out("---------- end of response\n");
        }
    }
}