How to use AVFoundation to extract (or 'demux') subtitle from mp4 video?

Question

I am trying to create a small procedure that will take an mp4 video and extract the subtitle information from the video using AVFoundation framework. In doing so it will create and return an NSArray of NSDictionary elements in the format startTimeOfSubtitle, endTimeofSubtitle and subtitleString.

This is what I gathered from a release note for AVFoundation. Any code example would be much appreciated.

Thanks in advance.

Selection of audio and subtitle media according to language and other criteria

AVFoundation now offers features for the discovery of options that may be offered by audiovisual media resources to accommodate differing language preferences, accessibility requirements, custom application configurations, and other needs, and for selection of these options for playback. For example, a resource may contain multiple audible options, each with dialog spoken in a different language, to be selected for playback to the exclusion of the others. Similar options in multiple languages can also be provided for legible media, such as subtitles. Both file-based content and HTTP Live Streaming content can offer media options. To obtain information about the groups of options that are offered by an instance of AVAsset: • Load the value of the AVAsset key availableMediaCharacteristicsWithMediaSelectionOptions using AVAsynchronousKeyValueLoading. When loading is complete, -[AVAsset availableMediaCharacteristicsWithMediaSelectionOptions] will provide an NSArray that may contain AVMediaCharacteristicAudible, AVMediaCharacteristicLegible, or AVMediaCharacteristicVisual, or any combination of these, to indicate the availability of groups of mutually exclusive options.

• Each group of mutually exclusive options with a media characteristic of interest can be obtained via -[AVAsset mediaSelectionGroupForMediaCharacteristic:]. To obtain the audible options, pass AVMediaCharacteristicAudible, etc. Each group is represented by an instance of AVMediaSelectionGroup. Each option within a group is represented by an instance of AVMediaSelectionOption. Both of these classes are defined in AVMediaSelectionGroup.h.

To examine available options within a group and to filter them for selection for playback: • AVMediaSelectionGroup offers methods in the category AVMediaSelectionOptionFiltering that perform common filtering operations on arrays of AVMediaSelectionOptions, according to whether the options are playable, match a desired locale, or either have or do not have special media characteristics, such as whether they offer specific features for accessibility. Media characteristics that indicate the presence of accessibility features, which can be used to filter media selection options, have been defined in AVMediaFormat.h.

• AVMediaSelectionOption offers information about options that may be used for display in a user interface that allows users to select among available options or in the implementation of client-defined filtering operations. As an example of client-defined filtering option in an application that makes use of custom media resources, options may be considered eligible for selection only if their associated metadata contains a specific value.

• To select a specific option within a group for playback, use -[AVPlayerItem selectMediaOption:inMediaSelectionGroup:]. To discover the option that's currently selected for playback, use -[AVPlayerItem selectedMediaOptionInMediaSelectionGroup:].

Advice about subtitles

Special care should be taken when displaying options to the user among the available legible options for playback and when making a selection among the available legible options according to user preferences. Some legible content contains "forced" subtitles, meaning that according to the content author's intent the subtitles should be displayed when the user has neither indicated a preference for the display of subtitles nor made an explicit selection of a subtitle option. Forced subtitles are typically used in order to convey the meaning of spoken dialog or visible text in a language that the content provider assumes will not be commonly understood, when comprehension of the dialog or text is nevertheless considered to be essential. Be sure that your app allows them to be displayed appropriately by following the advice below.

An AVMediaSelectionGroup for the characteristic AVMediaCharacteristicLegible can provide two types of legible options: 1) for display of legible content that's considered to be elective along with content that's considered to be essential, and 2) for display of essential legible content only. Legible AVMediaSelectionOptions that include essential content only have the media characteristic AVMediaCharacteristicContainsOnlyForcedSubtitles (defined in AVMediaFormat.h). When offering legible options for display to the end user in a selection interface, or when considering subtitle options for automatic selection according to a user preference for language, legible options with the characteristic AVMediaCharacteristicContainsOnlyForcedSubtitles should be excluded. +[AVMediaSelectionOption mediaSelectionOptionsFromArray:withoutMediaCharacteristics:], specifying AVMediaCharacteristicContainsOnlyForcedSubtitles as a characteristic to exclude, can be used to obtain the legible options that are suitable to offer to the end user in a selection interface or for consideration for selection according to a user preference.

If the user indicates no preference for or makes no selection of legible content, the application should select one of the legible options for playback that has the characteristic AVMediaCharacteristicContainsOnlyForcedSubtitles, if any are present. For most resources containing legible options with forced-only subtitles, an appropriate selection among them can be made in accordance with the current audible selection. Use -[AVMediaSelectionOption associatedMediaSelectionOptionInMediaSelectionGroup:] to obtain the legible option associated with an audible option. If there is no other means available to choose among them, the first legible option with forced-only subtitles in the media selection group is an appropriate default.

Answer 1

Well Iam not sure if you would be able to get start time and end time of subtitles... Iam using following method to get information about subtitle option in HLS stream.

[tmpCurrentAsset loadValuesAsynchronouslyForKeys:[NSArray arrayWithObject:@"tracks"] completionHandler:^{     dispatch_async(queueForMultipleAudioHandling, ^(void) {         AVKeyValueStatus postLoadingStatus = [tmpCurrentAsset statusOfValueForKey:@"tracks" error:NULL];         if (postLoadingStatus == AVKeyValueStatusLoaded)         {             AVMediaSelectionGroup * subtitleGroup = [self.mPlayer.currentItem.asset mediaSelectionGroupForMediaCharacteristic:AVMediaCharacteristicLegible];             if (subtitleGroup) {                  _subtitles = subtitleGroup;                  dispatch_async(dispatch_get_main_queue(), ^{                     [this createSubtitlePopOver];                 });               }          } 

First you need to reload tracks and get information about AVMediaCharacteristicLegible. If you have this, than you can extract information about subtitles like this: _subtitles.options This will gave you array of AVMediaSelectionOption from which you can choose. If you want to play more with AVMediaSelectionOption, you can continue reading here: AVMediaSelectionOption documentation

Hope this was a little bit helpful ;)