Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

linux: m3u8: how to manually download and create an mp4 from segments listed in .m3u8

Tags:

linux

bash

ffmpeg

m3u8

aria2

I have an .m3u8 url i.e http://example.com/test.m3u8

I can download it by ffmpeg command:

ffmpeg -y -loglevel verbose -i "http://example.com/test.m3u8" -c copy -f mpegts test.mp4

But i am trying to do this manually:

I tried the following:

1) download the m3u8 file:

wget -O test.m3u8 "http://example.com/test.m3u8"

2) then download each segment into a folder:

aria2c -i test.m3u8

this downloads all the .ts files in the folder.

3) then i combine them

cat *.ts > out.ts

4) then i try to convert to mp4:

$ ffmpeg -i out.ts -c:v libx264 outputfilename.mp4
ffmpeg version 3.4.2 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7.3.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-shared --enable-version3 --enable-omx
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
out.ts: Invalid data found when processing input

I am getting the following the above error finally. Invalid data found when processing input. Also i heard these files are encrypted. So i am not sure about how to do this.

Below is an example .m3u8 file:

#EXTM3U
#EXT-X-VERSION:5
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-TARGETDURATION:5
#UPLYNK-SEGMENT: fadd0d418aea4c5da8b87feec9b0acbc,00000000,segment
#UPLYNK-KEY:https://content-aapm1.uplynk.com/check?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc
#EXT-X-PROGRAM-DATE-TIME:2018-06-01T05:25:50.630000+00:00
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000000
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000000.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000001
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000001.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000002
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000002.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000003
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000003.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000004
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000004.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000005
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000005.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000006
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000006.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000007
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000007.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000008
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000008.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000009
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000009.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000000A
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000000A.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000000B
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000000B.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000000C
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000000C.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000000D
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000000D.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000000E
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000000E.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000000F
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000000F.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000010
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000010.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000011
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000011.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000012
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000012.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000013
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000013.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000014
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000014.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000015
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000015.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000016
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000016.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000017
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000017.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000018
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000018.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000019
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000019.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000001A
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000001A.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000001B
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000001B.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000001C
#EXTINF:3.6405,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000001C.ts?x=0&si=0
#EXT-X-ENDLIST
like image 477
Santhosh Avatar asked Jun 01 '18 10:06

Santhosh

People also ask

Can ffmpeg convert M3U8?

You can use FFmpeg to convert an M3U8 file to MP4 or use Agora's Format Converter script to batch convert multiple TS files into MP4 or audio formats.

2 Answers

You can use youtube-dl for this:

youtube-dl --list-formats M3U8_URL
youtube-dl -f FORMAT_CODE --hls-prefer-native M3U8_URL

The names in capital letters are placeholders. M3U8_URL is the URL of the m3u8 file. Available FORMAT_CODEs are shown in the left column of output of the first command.

like image 137
Falco Preiseni Avatar answered Sep 24 '22 00:09

Falco Preiseni

Regarding the cat issue. Are you accounting for the BOM? All .ts files start with a BOM, so if you cat them, they will break.

According to specification you can find all required encryption details in the media playlist itself.

Example line:

#EXT-X-KEY:METHOD=AES-128,URI="http://example.com/path/to/resource.mp4/crypt.key?id=somekey"

This contains the encryption algorithm and the uri to the key. The encryption can be per list or per track. In your case it is per track.

Side note: I wrote a bit about my experiences with m3u8 / ts. Maybe it can help you or give you some new idea: https://andreasvolkmann.github.io/posts/2018-04-27-m3u8-and-ts-segments/

like image 36
avolkmann Avatar answered Sep 24 '22 00:09

avolkmann
Sign in to Comment

Related questions

Vim on Ubuntu 14.04 uses a funny python path, python can't import _io among other modules
What's the Linux equivalent of Window's GetAsyncKeyState?
Linux Open Suse "pyodbc.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'SQL Server' : file not found (0) (SQLDriverConnect)")"
Raw socket vs TUN device
JavaFX - Caused by: java.lang.UnsupportedOperationException: Unable to open DISPLAY?
Permission denied trying to append to a file in a mounted directory
Electron: electron-builder config
use environment variables of remote server in ssh command
C++ Detecting USB serial device plugged/unplugged
"/bin/sh: XX: command not found" error when trying to install development version of R fst package from github
How to stream file using curl from one server to another(limited server resources)
How to debug an upstart script that intermittently fails?
Difference between bash,ksh,tcsh and zsh
logrotate entire directory containing log files
Strange behaviour with tiny tty linux device driver
Why does importing numpy add 1 GB of virtual memory on Linux?
MySQL 8.0 No data dictionary version number found
How to get uniq strings with different charset
python h5py file read "OSError: Unable to open file (bad superblock version number)"
Library deployment vs unused direct dependencies

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!