How to convert subtitle file to have only one sentence per subtitle?

Question

I am trying to program a method to convert subtitle files, such that there is always just one sentence per subtitle.

My idea is the following:

1. For each subtitle:

1.1 -> I get the subtitle duration

1.2 -> Calculate the characters_per_second

1.3 -> Use this to store (inside dict_times_word_subtitle ) the time it takes to speak the word i

2. I extract the sentences from the entire text

3. For each sentence:

3.1 I store (inside dict_sentences_subtitle ) the time it take to speak the sentence with the specific words (from which I can get the duration to speak them)

4. I create a new srt file (subtitle file) which starts at the same time as the original srt file and the subtitle timings can then be taken from the duration that it takes to speak the sentences.

For now, I have written the following code:

#---------------------------------------------------------
import pysrt
import re
from datetime import datetime, date, time, timedelta
#---------------------------------------------------------

def convert_subtitle_one_sentence(file_name):
    
    sub = pysrt.open(file_name)   

    ### ----------------------------------------------------------------------
    ### Store Each Word and the Average Time it Takes to Say it in a dictionary
    ### ----------------------------------------------------------------------

    dict_times_word_subtitle = {}
    running_variable = 0
    for i in range(len(sub)):

        subtitle_text = sub[i].text
        subtitle_duration = (datetime.combine(date.min, sub[i].duration.to_time()) - datetime.min).total_seconds()

        # Compute characters per second
        characters_per_second = len(subtitle_text)/subtitle_duration

        # Store Each Word and the Average Time (seconds) it Takes to Say in a Dictionary 
        
        for j,word in enumerate(subtitle_text.split()):
            if j == len(subtitle_text.split())-1:
                time = len(word)/characters_per_second
            else:
                time = len(word+" ")/characters_per_second

            dict_times_word_subtitle[str(running_variable)] = [word, time]
            running_variable += 1

            
    ### ----------------------------------------------------------------------
    ### Store Each Sentence and the Average Time to Say it in a Dictionary
    ### ----------------------------------------------------------------------  

    total_number_of_words = len(dict_times_word_subtitle.keys())

    # Get the entire text
    entire_text = ""
    for i in range(total_number_of_words):
        entire_text += dict_times_word_subtitle[str(i)][0] +" "


    # Initialize the dictionary 
    dict_times_sentences_subtitle = {}

    # Loop through all found sentences 
    last_number_of_words = 0
    for i,sentence in enumerate(re.findall(r'([A-Z][^\.!?]*[\.!?])', entire_text)):

        number_of_words = len(sentence.split())

        # Compute the time it takes to speak the sentence
        time_sentence = 0
        for j in range(last_number_of_words, last_number_of_words + number_of_words):
            time_sentence += dict_times_word_subtitle[str(j)][1] 

        # Store the sentence together with the time it takes to say the sentence
        dict_times_sentences_subtitle[str(i)] = [sentence, round(time_sentence,3)]

        ## Update last number_of_words
        last_number_of_words += number_of_words

    # Check if there is a non-sentence remaining at the end
    if j < total_number_of_words:
        remaining_string = ""
        remaining_string_time = 0
        for k in range(j+1, total_number_of_words):
            remaining_string += dict_times_word_subtitle[str(k)][0] + " "
            remaining_string_time += dict_times_word_subtitle[str(k)][1]

        dict_times_sentences_subtitle[str(i+1)] = [remaining_string, remaining_string_time]

    ### ----------------------------------------------------------------------
    ### Create a new Subtitle file with only 1 sentence at a time
    ### ----------------------------------------------------------------------  

    # Initalize new srt file
    new_srt = pysrt.SubRipFile()

    # Loop through all sentence
    # get initial start time (seconds)
    # https://stackoverflow.com/questions/44823073/convert-datetime-time-to-seconds
    start_time = (datetime.combine(date.min, sub[0].start.to_time()) - datetime.min).total_seconds()

    for i in range(len(dict_times_sentences_subtitle.keys())):


        sentence = dict_times_sentences_subtitle[str(i)][0]
        print(sentence)
        time_sentence = dict_times_sentences_subtitle[str(i)][1]
        print(time_sentence)
        item = pysrt.SubRipItem(
                        index=i,
                        start=pysrt.SubRipTime(seconds=start_time),
                        end=pysrt.SubRipTime(seconds=start_time+time_sentence),
                        text=sentence)

        new_srt.append(item)

        ## Update Start Time
        start_time += time_sentence

    new_srt.save(file_name)

The issue:

There are no error messages, but when I apply this to real subtitle files and then watch the video, the subtitles begin correctly, but as the video progresses (error progression) the subtitles get less and less aligned with what is actually said.

Example: The speaker has finished his talk, but the subtitles keep appearing.

Simple example to test

srt = """
1
00:00:13,100 --> 00:00:14,750
Dr. Martin Luther King, Jr.,

2
00:00:14,750 --> 00:00:18,636
in a 1968 speech where he reflects
upon the Civil Rights Movement,

3
00:00:18,636 --> 00:00:21,330
states, "In the end,

4
00:00:21,330 --> 00:00:24,413
we will remember not the words of our enemies

5
00:00:24,413 --> 00:00:27,280
but the silence of our friends."

6
00:00:27,280 --> 00:00:29,800
As a teacher, I've internalized this message.

"""

with open('test.srt', "w") as file:
    file.write(srt)
    
    
convert_subtitle_one_sentence("test.srt")

The output looks like this (yes, there is still some work to do on the sentence recognition par (i.e. Dr. )):

0
00:00:13,100 --> 00:00:13,336
Dr.

1
00:00:13,336 --> 00:00:14,750
Martin Luther King, Jr.

2
00:00:14,750 --> 00:00:23,514
Civil Rights Movement, states, "In the end, we will remember not the words of our enemies but the silence of our friends.

3
00:00:23,514 --> 00:00:26,175
As a teacher, I've internalized this message.

4
00:00:26,175 --> 00:00:29,859
our friends." As a teacher, I've internalized this message.

As you can see the original last time stamp is 00:00:29,800 whereas in the output file it is 00:00:29,859. This might not seem like much in the beginning, but as the video gets longer, the difference increases.

The full sample video can be downloaded here: https://ufile.io/19nuvqb3

The full subtitle file: https://ufile.io/qracb7ai

Attention: The subtitle file will be overridden, so you might want to store a copy with another name to be able to compare.

Method how it could be fixed:

The exact timing for words starting or ending an original subtitle is known. This could be used to cross-check and adjust timing accordingly.

Edit

Here is a code to create a dictionary which stores character, character_duration (average over subtitle), and start or end original time stamb, if it exists for this character.

sub = pysrt.open('video.srt')

running_variable = 0
dict_subtitle = {}

for i in range(len(sub)):

    # Extract Start Time Stamb
    timestamb_start = sub[i].start

    # Extract Text
    text =sub[i].text

    # Extract End Time Stamb
    timestamb_end = sub[i].end

    # Extract Characters per Second 
    characters_per_second = sub[i].characters_per_second
    
    # Fill Dictionary 
    for j,character in enumerate(" ".join(text.split())):
        character_duration = len(character)*characters_per_second
        dict_subtitle[str(running_variable)] = [character,character_duration,False, False]
        if j == 0: dict_subtitle[str(running_variable)] = [character, character_duration, timestamb_start, False]
        if j == len(text)-1 : dict_subtitle[str(running_variable)] = [character, character_duration, False, timestamb_end]
        running_variable += 1

More videos to try

Here you may download more videos and their respective subtitle files: https://filebin.net/kwygjffdlfi62pjs

Edit 3

4
00:00:18,856 --> 00:00:25,904
Je rappelle la définition de ce qu'est un produit scalaire, <i>dot product</i> dans <i>Ⅎ</i>.

5
00:00:24,855 --> 00:00:30,431
Donc je prends deux vecteurs dans <i>Ⅎ</i> et je définis cette opération-là, linéaire, <i>u 

Answer 1

I have re-coded to rely on the pysrt package, as requested, and a smigeon of re.
The idea is to build a dictionary based on start_times.

If the start time exists, data is added to the entry for that time but the end_time is updated at the same time, so the end time advances with the text.

If no start time exists, it is simply a new dictionary entry.

The start time is only advanced once we know that a sentence has been completed.

So in essence, we start to build a sentence with a fixed start time. The sentence continues to be built, by adding more text and updating the end time, until the sentence finishes. Here we advance the start time using the current record, which we know to be a new sentence.

Sub-title entries with multiple sentences are broken up, with start and end times calculated using the pysrt character_per_second entry for the entire sub-title entry, before it was broken up.

Finally, a new sub-title file is written to disk from the entries in the dictionary.

Obviously, with only a single file to play with, I may well be missing some sub-title layout humps in the road, but at least it gives you a working starting point.

The code is commented throughout, so most things should be clear, as to how and why.

Edit: I have refined the checking for existing dictionary start times and altered the method used to decide if a sentence has ended i.e. put the full stops back into the text, after splitting.
The second video you mentioned does have sub-titles that are slightly off, to begin with, notice that there are no milli-second values at all.

The following code does a fair job on the second video and a good job on the first.

Edit 2: Added contiguous full-stops and html <> tag removal

Edit 3: It turns out that pysrt removes the html tags from the calculation for characters per second. I have now done so as well, which means that the <html> formatting can be retained within the sub-titles.

Edit 4: This version copes with full stops in mathematical and chemical formulae, plus ip numbers etc. Basically places where a full stop doesn't mean a full stop. It also allows for sentences which end in ? and !

import pysrt
import re

abbreviations = ['Dr.','Mr.','Mrs.','Ms.','etc.','Jr.','e.g.'] # You get the idea!
abbrev_replace = ['Dr','Mr','Mrs','Ms','etc','Jr','eg']
subs = pysrt.open('new.srt')
subs_dict = {}          # Dictionary to accumulate new sub-titles (start_time:[end_time,sentence])
start_sentence = True   # Toggle this at the start and end of sentences

# regex to remove html tags from the character count
tags = re.compile(r'<.*?>')

# regex to split on ".", "?" or "!" ONLY if it is preceded by something else
# which is not a digit and is not a space. (Not perfect but close enough)
# Note: ? and ! can be an issue in some languages (e.g. french) where both ? and !
# are traditionally preceded by a space ! rather than!
end_of_sentence = re.compile(r'([^\s\0-9][\.\?\!])')

# End of sentence characters
eos_chars = set([".","?","!"])

for sub in subs:
    if start_sentence:
        start_time = sub.start
        start_sentence = False
    text = sub.text

    #Remove multiple full-stops e.g. "and ....."
    text = re.sub('\.+', '.', text)

    # Optional
    for idx, abr in enumerate(abbreviations):
        if abr in text:
            text = text.replace(abr,abbrev_replace[idx])
    # A test could also be made for initials in names i.e. John E. Rotten - showing my age there ;)

    multi = re.split(end_of_sentence,text.strip())
    cps = sub.characters_per_second

    # Test for a sub-title with multiple sentences
    if len(multi) > 1:
        # regex end_of_sentence breaks sentence start and sentence end into 2 parts
        # we need to put them back together again.
        # hence the odd range because the joined end part is then deleted
        for cnt in range(divmod(len(multi),2)[0]): # e.g. len=3 give 0 | 5 gives 0,1  | 7 gives 0,1,2
            multi[cnt] = multi[cnt] + multi[cnt+1]
            del multi[cnt+1]

        for part in multi:
            if len(part): # Avoid blank parts
                pass
            else:
                continue
            # Convert start time to seconds
            h,m,s,milli = re.split(':|,',str(start_time))
            s_time = (3600*int(h))+(60*int(m))+int(s)+(int(milli)/1000)

            # test for existing data
            try:
                existing_data = subs_dict[str(start_time)]
                end_time = str(existing_data[0])
                h,m,s,milli = re.split(':|,',str(existing_data[0]))
                e_time = (3600*int(h))+(60*int(m))+int(s)+(int(milli)/1000)
            except:
                existing_data = []
                e_time = s_time

            # End time is the start time or existing end time + the time taken to say the current words
            # based on the calculated number of characters per second
            # use regex "tags" to remove any html tags from the character count.

            e_time = e_time + len(tags.sub('',part)) / cps

            # Convert start to a timestamp
            s,milli = divmod(s_time,1)
            m,s = divmod(int(s),60)
            h,m = divmod(m,60)
            start_time = "{:02d}:{:02d}:{:02d},{:03d}".format(h,m,s,round(milli*1000))

            # Convert end to a timestamp
            s,milli = divmod(e_time,1)
            m,s = divmod(int(s),60)
            h,m = divmod(m,60)
            end_time = "{:02d}:{:02d}:{:02d},{:03d}".format(h,m,s,round(milli*1000))

            # if text already exists add the current text to the existing text
            # if not use the current text to write/rewrite the dictionary entry
            if existing_data:
                new_text = existing_data[1] + " " + part
            else:
                new_text = part
            subs_dict[str(start_time)] = [end_time,new_text]

            # if sentence ends re-set the current start time to the end time just calculated
            if any(x in eos_chars for x in part):
                start_sentence = True
                start_time = end_time
                print ("Split",start_time,"-->",end_time,)
                print (new_text)
                print('\n')
            else:
                start_sentence = False

    else:   # This is Not a multi-part sub-title

        end_time = str(sub.end)

        # Check for an existing dictionary entry for this start time
        try:
            existing_data = subs_dict[str(start_time)]
        except:
            existing_data = []

        # if it already exists add the current text to the existing text
        # if not use the current text
        if existing_data:
            new_text = existing_data[1] + " " + text
        else:
            new_text = text
        # Create or Update the dictionary entry for this start time
        # with the updated text and the current end time
        subs_dict[str(start_time)] = [end_time,new_text]

        if any(x in eos_chars for x in text):
            start_sentence = True
            print ("Single",start_time,"-->",end_time,)
            print (new_text)
            print('\n')
        else:
            start_sentence = False

# Generate the new sub-title file from the dictionary
idx=0
outfile = open('video_new.srt','w')
for key, text in subs_dict.items():
    idx+=1
    outfile.write(str(idx)+"\n")
    outfile.write(key+" --> "+text[0]+"\n")
    outfile.write(text[1]+"\n\n")
outfile.close()

The output after passing through the above code for your video.srt file is as follows:

1
00:00:13,100 --> 00:00:27,280
Dr Martin Luther King, Jr, in a 1968 speech where he reflects
upon the Civil Rights Movement, states, "In the end, we will remember not the words of our enemies but the silence of our friends."

2
00:00:27,280 --> 00:00:29,800
As a teacher, I've internalized this message.

3
00:00:29,800 --> 00:00:39,701
Every day, all around us, we see the consequences of silence manifest themselves in the form of discrimination, violence, genocide and war.

4
00:00:39,701 --> 00:00:46,178
In the classroom, I challenge my students to explore the silences in their own lives through poetry.

5
00:00:46,178 --> 00:00:54,740
We work together to fill those spaces, to recognize them, to name them, to understand that they don't
have to be sources of shame.

6
00:00:54,740 --> 00:01:14,408
In an effort to create a culture within my classroom where students feel safe sharing the intimacies of their own silences, I have four core principles posted on the board that sits in the front of my class, which every student signs
at the beginning of the year: read critically, write consciously, speak clearly, tell your truth.

7
00:01:14,408 --> 00:01:18,871
And I find myself thinking a lot about that last point, tell your truth.

8
00:01:18,871 --> 00:01:28,848
And I realized that if I was going to ask my students to speak up, I was going to have to tell my truth and be honest with them about the times where I failed to do so.

9
00:01:28,848 --> 00:01:44,479
So I tell them that growing up, as a kid in a Catholic family in New Orleans, during Lent I was always taught that the most meaningful thing one could do was to give something up, sacrifice something you typically indulge in to prove to God you understand his sanctity.

10
00:01:44,479 --> 00:01:50,183
I've given up soda, McDonald's, French fries, French kisses, and everything in between.

11
00:01:50,183 --> 00:01:54,071
But one year, I gave up speaking.

12
00:01:54,071 --> 00:02:03,286
I figured the most valuable thing I could sacrifice was my own voice, but it was like I hadn't realized that I had given that up a long time ago.

13
00:02:03,286 --> 00:02:23,167
I spent so much of my life telling people the things they wanted to hear instead of the things they needed to, told myself I wasn't meant to be anyone's conscience because I still had to figure out being my own, so sometimes I just wouldn't say anything, appeasing ignorance with my silence, unaware that validation doesn't need words to endorse its existence.

14
00:02:23,167 --> 00:02:29,000
When Christian was beat up for being gay, I put my hands in my pocket and walked with my head
down as if I didn't even notice.

15
00:02:29,000 --> 00:02:39,502
I couldn't use my locker for weeks
because the bolt on the lock reminded me of the one I had put on my lips when the homeless man on the corner looked at me with eyes up merely searching for an affirmation that he was worth seeing.

16
00:02:39,502 --> 00:02:43,170
I was more concerned with
touching the screen on my Apple than actually feeding him one.

17
00:02:43,170 --> 00:02:46,049
When the woman at the fundraising gala said "I'm so proud of you.

18
00:02:46,049 --> 00:02:53,699
It must be so hard teaching
those poor, unintelligent kids," I bit my lip, because apparently
we needed her money more than my students needed their dignity.

19
00:02:53,699 --> 00:03:02,878
We spend so much time listening to the things people are saying that we rarely pay attention to the things they don't.

20
00:03:02,878 --> 00:03:06,139
Silence is the residue of fear.

21
00:03:06,139 --> 00:03:09,615
It is feeling your flaws gut-wrench guillotine your tongue.

22
00:03:09,615 --> 00:03:13,429
It is the air retreating from your chest because it doesn't feel safe in your lungs.

23
00:03:13,429 --> 00:03:15,186
Silence is Rwandan genocide.

24
00:03:15,186 --> 00:03:16,423
 Silence is Katrina.

25
00:03:16,553 --> 00:03:19,661
It is what you hear when there
aren't enough body bags left.

26
00:03:19,661 --> 00:03:22,062
It is the sound after the noose is already tied.

27
00:03:22,062 --> 00:03:22,870
It is charring.

28
00:03:22,870 --> 00:03:23,620
 It is chains.

29
00:03:23,620 --> 00:03:24,543
 It is privilege.

30
00:03:24,543 --> 00:03:25,178
 It is pain.

31
00:03:25,409 --> 00:03:28,897
There is no time to pick your battles when your battles have already picked you.

32
00:03:28,897 --> 00:03:31,960
I will not let silence wrap itself around my indecision.

33
00:03:31,960 --> 00:03:36,287
I will tell Christian that he is a lion, a sanctuary of bravery and brilliance.

34
00:03:36,287 --> 00:03:42,340
I will ask that homeless man what his name is and how his day was, because sometimes all people want to be is human.

35
00:03:42,340 --> 00:03:51,665
I will tell that woman that my students can talk about transcendentalism like their last name was Thoreau, and just because you watched
one episode of "The Wire" doesn't mean you know anything about my kids.

36
00:03:51,665 --> 00:04:03,825
So this year, instead of giving something up, I will live every day as if there were a microphone tucked under my tongue, a stage on the underside of my inhibition.

37
00:04:03,825 --> 00:04:10,207
Because who has to have a soapbox when all you've ever needed is your voice?

38
00:04:10,207 --> 00:04:12,712
Thank you.

39
00:04:12,712 --> 00:00:00,000
(Applause)