Android: streaming the camera as mjpeg

Question

After several days of searching SO and google I'm beginning to give up, so I thought I might as well post here.

I'm creating an android app which should offer some kind of video chat. As this should be as close as possible to realtime, I read about various protocols and decided to try MJPEG for starters (not concerning with audio for now).

Right now streaming the data is driving me nuts. The connection gets established, the app starts writing the camera preview frames to the stream, but neither VLC nor mplayer start playing video. Monitoring the connection reveals that the data is arriving.

Connecting This code is executed by an async task, a listener is notified on success:

try
    {
        ServerSocket server = new ServerSocket(8080);

        socket = server.accept();

        server.close();

        Log.i(TAG, "New connection to :" + socket.getInetAddress());

        stream = new DataOutputStream(socket.getOutputStream());
        prepared = true;
    }
    catch (IOException e)
    {
        Log.e(TAG, e.getMessage();
    }

On my PC I execute 'mplayer http://tabletIP:8080' and the tablet registers a connection (and thus starts my streamer and the camera preview). This also works with VLC.

Streaming This writes the header to the stream:

if (stream != null)
{
    try
    {
        // send the header
        stream.write(("HTTP/1.0 200 OK\r\n" +
                      "Server: iRecon\r\n" +
                      "Connection: close\r\n" +
                      "Max-Age: 0\r\n" +
                      "Expires: 0\r\n" +
                      "Cache-Control: no-cache, private\r\n" + 
                      "Pragma: no-cache\r\n" + 
                      "Content-Type: multipart/x-mixed-replace; " +
                      "boundary=--" + boundary +
                      "\r\n\r\n").getBytes());

        stream.flush();

        streaming = true;
    }
    catch (IOException e)
    {
        notifyOnEncoderError(this, "Error while writing header: " + e.getMessage());
        stop();
    }
}

Afterwards streaming is triggered through the Camera.onPreviewFrame() Callback:

@Override
public void onPreviewFrame(byte[] data, Camera camera)
{
    frame = data;

    if (streaming)
        mHandler.post(this);
}

@Override
public void run()
{
    // TODO: cache not filling?
    try
    {
                    // buffer is a ByteArrayOutputStream
        buffer.reset();

        switch (imageFormat)
        {
            case ImageFormat.JPEG:
                // nothing to do, leave it that way
                buffer.write(frame);
                break;

            case ImageFormat.NV16:
            case ImageFormat.NV21:
            case ImageFormat.YUY2:
            case ImageFormat.YV12:
                new YuvImage(frame, imageFormat, w, h, null).compressToJpeg(area, 100, buffer);
                break;

            default:
                throw new IOException("Error while encoding: unsupported image format");
        }

        buffer.flush();

        // write the content header
        stream.write(("--" + boundary + "\r\n" + 
                      "Content-type: image/jpg\r\n" + 
                      "Content-Length: " + buffer.size() + 
                      "\r\n\r\n").getBytes());

        // Should omit the array copy
        buffer.writeTo(stream);

        stream.write("\r\n\r\n".getBytes());
        stream.flush();
    }
    catch (IOException e)
    {
        stop();
        notifyOnEncoderError(this, e.getMessage());
    }
}

There is no exception thrown. The mHandler runs in it's own HandlerThread. Just to be sure I tried using an AsyncTask, to no avail (btw, is this better?).

The encoded frames are fine on the android side, I saved them to jpg files and could open them.

My guess is that I have to cluster the data somehow or have to set some options for the socket or something, but....well, I'm stuck.

tl;dr: VLC not playing stream, mplayer says 'cache not filling', problem probably in last code segment, need help~ :)

Thank you kindly!

Answer 1

I got it. Seems, like my http-/content-headers were messed up. The proper headers should be:

stream.write(("HTTP/1.0 200 OK\r\n" +
                          "Server: iRecon\r\n" +
                          "Connection: close\r\n" +
                          "Max-Age: 0\r\n" +
                          "Expires: 0\r\n" +
                          "Cache-Control: no-store, no-cache, must-revalidate, pre-check=0, post-check=0, max-age=0\r\n" +
                          "Pragma: no-cache\r\n" + 
                          "Content-Type: multipart/x-mixed-replace; " +
                          "boundary=" + boundary + "\r\n" +
                          "\r\n" +
                          "--" + boundary + "\r\n").getBytes());

and

stream.write(("Content-type: image/jpeg\r\n" +
                      "Content-Length: " + buffer.size() + "\r\n" +
                      "X-Timestamp:" + timestamp + "\r\n" +
                      "\r\n").getBytes());

buffer.writeTo(stream);
stream.write(("\r\n--" + boundary + "\r\n").getBytes());

Of course, where to put the boundary is your own choice. Also there are probably some fields which are optional (e.g. most in Cache-Control), but this works and till now I was too lazy to strip them down. The important part is to remember the linebreaks (\r\n thingies)...