Mechanism for apps to display their image content in native 4k

Question

As I understand that currently UHD video content is streamed on 4k TV mostly using HEVC codec.

I want to understand how can apps which have UHD image content can display their image content in native 4K?

What I am exactly looking for is rendering 4k(3840*2060) jpeg images. My display supports 4k rendering and the SOC can even output 4k. I am looking for modifications in framework, so that all apps which have 4k images can render them on my device without downscaling.

Actually I am trying to come up with API set which others can use. But my main confusion is : for jpeg image i create a 4k surface, but their are other surfaces as well (buttons etc). They are rendered by surface flinger which renders at 1280*720.

Now what is the best way to compose my 4k surface with these other surfaces? Where should I upscale these surfaces and where to compose all of them?

Answer 1

An important thing to keep clear is what kind of development you have in mind. If you simply wish to put a video stream showing inside your App, you should take the user interface for the main activity that will simply consist of an instance of the VideoView class. The VideoView class has a wide range of methods that may be called in order to manage the playback of video.

Configure the VideoView with the path of the video to be played and then start the playback. Then select the VideoPlayerActivity.java file and modify the OnCreate() method as outlined in the following listing:

package com.example.videoplayer;

import android.os.Bundle;
import android.app.Activity;
import android.view.Menu;
import android.widget.VideoView;

public class VideoPlayerActivity extends Activity {

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_video_player);

        final VideoView videoView = 
               (VideoView) findViewById(R.id.videoView1);

           videoView.setVideoPath(
               "http://www.ebookfrenzy.com/android_book/movie.mp4");

        videoView.start();
    }
.
.
.
}

Essentially what you have is the Android SDK for the App user interface, which means that you can use different choices to actually render the video streaming underneath the UI layer.

Migrating your already existing App from a tablet or mobile to a smart TV is also something that can be achieve quite smoothly. Some few things will have to be adjusted - e.g. touch screen usually for smart TV may not be an option.

Instead you should consider onkeyDown as a more reliable method to input interaction for your App:

@Override
public boolean onKeyDown(int keyCode, KeyEvent event) {

    switch (keyCode) {
        case KeyEvent.KEYCODE_MEDIA_PLAY:{
            if (!mPlaying) {
                startSlideShow();
            }
            mPlaying = true;
            break;
        }
        case KeyEvent.KEYCODE_MEDIA_PAUSE:{
            mPlaying = false;
            showStatusToast(R.string.slideshow_paused);
        }
    }
    return super.onKeyDown(keyCode, event);
}

As part of Android Smart Google TV API, you can also make necessary adjustment for larger screen resolutions:

   // Get the source image's dimensions
   BitmapFactory.Options options = new BitmapFactory.Options();
   options.inJustDecodeBounds = true; // this does not download the actual image, just downloads headers.
   BitmapFactory.decodeFile(IMAGE_FILE_URL, options);

   int srcWidth = options.outWidth;  // actual width of the image.
   int srcHeight = options.outHeight;  // actual height of the image.

   // Only scale if the source is big enough. This code is just trying to fit a image into a certain width.
   if(desiredWidth > srcWidth)
     desiredWidth = srcWidth;

   // Calculate the correct inSampleSize/scale value. This helps reduce memory use. It should be a power of 2.
   int inSampleSize = 1;
   while(srcWidth / 2 > desiredWidth){
     srcWidth /= 2;
     srcHeight /= 2;
     inSampleSize *= 2;
   }

   float desiredScale = (float) desiredWidth / srcWidth;

   // Decode with inSampleSize
   options.inJustDecodeBounds = false;  // now download the actual image.
   options.inDither = false;
   options.inSampleSize = inSampleSize;
   options.inScaled = false;
   options.inPreferredConfig = Bitmap.Config.ARGB_8888;  // ensures the image stays as a 32-bit ARGB_8888 image. 
                                                         // This preserves image quality.
   Bitmap sampledSrcBitmap = BitmapFactory.decodeFile(IMAGE_FILE_URL, options);

   // Resize
   Matrix matrix = new Matrix();
   matrix.postScale(desiredScale, desiredScale);
   Bitmap scaledBitmap = Bitmap.createBitmap(sampledSrcBitmap, 0, 0,
       sampledSrcBitmap.getWidth(), sampledSrcBitmap.getHeight(), matrix, true);
   sampledSrcBitmap = null;

   // Save
   FileOutputStream out = new FileOutputStream(LOCAL_PATH_TO_STORE_IMAGE);
   scaledBitmap.compress(Bitmap.CompressFormat.JPEG, 100, out);
   scaledBitmap = null;

You can also easily convert your LCD TV into a smart TV and start playing and testing how your App for google TV would behave. For that you only need to put your hands in the adaptor kit.

In fact, Lenovo is releasing a 28-inch 4K monitor (the ThinkVision 28) that also runs Android, allowing you to run all of the usual media streaming apps, and Kogan is doing the same.

Hacking a little bit further in playing with gadgets, you can even huck up your mobile with MHL HDMI on a TV or use it as a computer.

So, Android working with MHL 3.0 does 4K video output is a reality that good developers can already make their Apps with the necessary resolution and input adjustment accordingly to the device in use.

If your main concern is to put performance and optimize the video streaming, you may consider following options:

• NDK: You decide to take some library or implement your own program that actually is made in C++ to optimize the video streaming.
• RenderScript: it uses C99 syntax with new APIs that are ultimately compiled to native code. While this syntax is well known, there's a learning curve to using this system because the APIs are not.
• OpenCL: is meant for graphics acceleration and provides many tools for 3D rendering and video stream high performance.

In fact the code in OpenCL resembles C/C++:

    for (int yy=-filterWidth; yy<=filterWidth; ++yy)
    {
      for (int xx=-filterWidth; xx<=filterWidth; ++xx)
       {
          int thisIndex = (y + yy) * width + (x + xx);
          float4 currentPixel = oneover255 *convert_float4(srcBuffer[thisIndex]);
          float domainDistance = fast_distance((float)(xx), (float)(yy));
          float domainWeight = exp(-0.5f * pow((domainDistance/sigmaDomain),2.0f));

          float rangeDistance = fast_distance(currentPixel.xyz, centerPixel.xyz);
          float rangeWeight = exp(-0.5f * pow((rangeDistance/sigmaRange),2.0f));

          float totalWeight = domainWeight * rangeWeight ;
          normalizeCoeff += totalWeight;

          sum4 += totalWeight * currentPixel;
       }
    }

In terms of ARM microprocessors capabilities, it is worth to mention Sigma Designs SMP8756 ARM for Android Set-top-boxes, meant to address full High Efficiency Video Coding (HEVC) capabilities.

• Bilinear Filtering

To resize your image/video, what you need is to apply bilinear filtering. Bilinear Interpolation is the process of using each of the intermediate fields in an interlaced video frame to generate a full size target image. Either all the odd or all the even lines on the field are used. Interpolations are then performed between the lines and between adjoining pixels to generate an entire non-interlaced frame for the progressive scan output.

To implement have your image adjusted properly for the sizes you need, there are plenty of good algorithms that can be used for that purpose, such as some of the OpenCL features for image scaling, and likewise for native C++ other options are also available.