turn video frames to streaming video

Question

The server is sending video frame. I would like to use them in order to do a streaming. I wonder how I could assemble the frames to create a streaming video. So far, I could display the frames as pictures. Below is my angular code

component angular

 getVideo() {
    interval(250).switchMap(() => this.appService.getPictures())
      .subscribe(data => {
        const file = new Blob([data], {type:'image/png'});
        this.url = this.sanitizer.bypassSecurityTrustResourceUrl(URL.createObjectURL(file));
      })
  }

template html

<img div="test" [src]="url" width="400px" height="300px"/>

I am trying to change the picture using the frame rate of the camera. But my picture is not updated and it freezes my browser due to the high number of http requests.

What I want to achieve is to buffer the frame in order to use the video tag instead of the img tag the same way I would connect to a live streaming send by a server using the video tag with src set to the url of the server.

Github link: https://github.com/kedevked/frameProcessing

Answer 1

Instead of triggering http request at a certain interval of times, using websocket is better.

By displaying the images frame by frame, it gives the impression of a live video. The only thing though is that there might be latency in the network which makes the frame rate not to be constant. However, it works well for use cases where buffering the video server side could not be considered such as when we need to send data along each frame.

service using websocket

createStream(url) {
  this.ws = webSocket<any>(url);
  return return this.ws.asObservable()
}

component

constructor (private streamingService: StreamingService) {}
ngOnInit(): void { 
  this.getStream()
}

getStream() {
  this.streamingService.createStream().subscribe(data => {
    const file = new Blob([data], {type:'image/png'});
        this.url = this.sanitizer.bypassSecurityTrustResourceUrl(URL.createObjectURL(file));
 })
}

This solution is well suited when we are sending not only image but also data along the image. In this scenario, the image needs to be sent as base64.

getStream() {
      this.streamingService.createStream().subscribe(data => {
            this.url = data.base64ImageData
     })
    }

Since the base64 encoding has a payload of 133% of the initial image, it might be costly to use it in all settings. In case the image needs to be served along with other data, the whole data can be sent as a binary. This will require encoding the data server-side and decode it client-side. Since this might require a bit of computation, using a web-worker can be considered for not only decoding the data but also for displaying the image.

If the image is sent as a binary, using canvas for rendering will be faster

canvas.getContex('2d').putImageData(imageBinaryData, 0, 0)

Answer 2

I see two problems here, that may improve this solution.

First of all, how do you know that the request will spend less than 250 milliseconds? if not, you know that switchMap will cancel it? Maybe exhaustMap or concatMap could fit better if you don't want to lose the current request.

The second point is that the interval(X) and the HTTP request are inside the angular zone, so they will fire change detection... This could bring you several performance issues. So you should run the interval and the request outside the zone and detect changes manually.

Said that I will purpose a solution like this:

private subscription: Subscription;
cosntructor(public ngZone: NgZone, public cd: ChangeDetectorRef){
   getVideo();
}

getVideo() {
    this.subscription = this.ngZone.runOutsideAngular(() => 
         interval(250).pipe(exhaustMap(() => this.appService.getPictures()))
           .subscribe(data => {
              const file = new Blob([data], {type:'image/png'});
              this.url = this.sanitizer.bypassSecurityTrustResourceUrl(URL.createObjectURL(file));
              // Do change detection in orther to actualize [url] binding
              this.cd.detectChanges();
           })
    );
}
ngOnDestroy() {
    // remember to unsubscribe from interval subscription
    this.subscription.unsubscribe()
}

I don't think that with this code you would have much performance issues... although this type of long polling is never good, I don't think it would freeze your app. Try it and let me know.

Hope this helps.