Sample Times in Fragmented MP4
This document explains how MoofParser reads a fragmented MP4 file and gives
each sample a decode time and a composition range. It also gives the reason
for each step. The examples use small numbers.
Plain and fragmented MP4
A plain MP4 file keeps one sample table for each track in moov/stbl.
MP4Metadata reads that table through mp4parse. The MP4SampleIndex
constructor then copies the samples into MP4SampleIndex::mIndex.
A fragmented MP4 file splits the samples into fragments. Each fragment is a
moof box and an mdat box. The moof box has a traf box for each track.
The traf box has a tfdt box and one or more trun boxes. When the moov
box has no samples, MP4SampleIndex creates a MoofParser. The parser makes
one Moof object for each moof box, and each Moof has its own mIndex.
File type |
Where the samples are |
Index |
|---|---|---|
Plain |
|
|
Fragmented |
|
|
MSE must use fragmented MP4. In the ISO BMFF byte stream format for MSE, a
media segment is a moof box and its mdat box.
Where the parser runs
For file playback, the call chain is:
MP4TrackDemuxer::EnsureUpToDateIndex
MP4SampleIndex::UpdateMoofIndex
MoofParser::RebuildFragmentedIndex
Moof::Moof one call for each moof box
Moof::ParseTraf
Moof::ParseTrun
SampleIterator::GetNext then makes a MediaRawData from each sample:
|
Value from the sample |
|---|---|
|
start of |
|
length of |
|
|
|
|
For MSE, TrackBuffersManager::CreateDemuxerforMIMEType creates an
MP4Demuxer on the appended data. The demuxer uses the same path. Then
TrackBuffersManager::ProcessFrames runs the MSE coded frame processing on
the samples.
Step 1: read each trun entry
Moof::ParseTraf starts the decode time at the tfdt value. If the traf
box has no valid tfdt, it continues from the decode end of the previous
moof.
Moof::ParseTrun then reads one entry for each sample. For each entry, it
does these steps:
It sets the decode time (DTS) of the sample to the current decode time.
It sets the composition range to
[DTS + ctsOffset, DTS + ctsOffset + duration).It adds the entry duration to the current decode time.
The edit list moves both times by the same amount. The examples have no edit list.
The trun duration is a decode delta: the time from this DTS to the next DTS.
It is not the time that the frame stays on the screen.
Step 2: decode order and display order
The parser stores the samples in decode order. B-frames make the display order
different from the decode order. The ctsOffset of each entry moves its
composition time (PTS) away from its DTS.
For example, a 30 fps H.264 stream has two B-frames between its I- and P-frames. Its timescale is 15360, so one frame is 512 ticks. The encoder puts three samples in each moof. The table shows the first two moofs in decode order.
moof |
type |
DTS |
PTS |
|---|---|---|---|
1 |
I |
0 |
512 |
1 |
P |
512 |
2048 |
1 |
B |
1024 |
1024 |
2 |
B |
1536 |
1536 |
2 |
P |
2048 |
3584 |
2 |
B |
2560 |
2560 |
In display order, moof 1 has the frames at 512, 1024 and 2048. The frame at 1536 is in moof 2. So moof 1 has a gap in its composition times, and the next moof fills it.
PTS 512 1024 1536 2048 2560
|-----|-----|-----|-----|
moof 1 [ I ] [ B ] [ P ]
moof 2 [ B ] [ B ] ...
Step 3: close the composition gaps
After ParseTrun, each sample has the composition range from its trun
duration. But the trun duration is a decode delta, so it can be different
from the display spacing. For example, a stream has a timescale of 90000, so
one frame is 3000 ticks. In decode order, its first trun duration is 3000
ticks. Then the durations alternate between 1 and 5999 ticks, and one of them
is 6149. But its composition times are about 3000 ticks apart. With the trun
durations only, the sample at 6000 has a duration of 1 tick, and a gap of one
frame follows it.
So Moof::Moof sorts the samples of the moof by composition start. Then it
sets the end of each sample to a new value.
Why not end at the next start
A simple rule sets each end to the start of the next sample in the moof. This closes every gap in the moof. But it also closes a gap that the next moof fills. In the B-frame example, the frame at 1024 gets the end 2048, and it covers the frame at 1536 in moof 2.
PTS |
|
simple rule end |
bounded rule end |
|---|---|---|---|
512 |
1024 |
1024 |
1024 |
1024 |
1536 |
2048 |
1536 |
2048 |
2560 |
2560 |
2560 |
The video sink shows a frame until the clock passes its end. So the frame at 1024 stays on the screen too long. Then the sink drops the frames that start inside its span.
The bounded rule
Moof::Moof limits each end:
end = min(start of the next sample, start + longest duration in this moof)
The longest trun duration in the moof is the bound. In the B-frame example,
every duration is 512. So the frame at 1024 ends at 1536.
In the example with uneven durations, the longest duration is 6149. The sample at 6000 has a duration of 1 tick, and the next sample starts at 9001. The gap is smaller than the bound, so the sample ends at 9001.
Fallbacks
The rule uses the start of the next sample as the end in two cases:
Case |
Why |
|---|---|
The longest duration is 0 |
A zero bound gives empty samples |
|
The sum is not a valid time |
Known limit
One long sample raises the bound for all the samples in its moof. Suppose that in the B-frame example, the frame at 1024 in moof 1 has a duration of 7680 ticks. The bound becomes 7680, so the gap does not limit the end, and in moof 1 alone the frame at 1024 ends at 2048 again. The next section shows how the next moof ends it at 1536:
PTS |
end |
|---|---|
512 |
1024 |
2048 |
2560 |
1024 |
2048 |
Ends across moofs
ISO BMFF stores no display duration. A sample duration is the step to the next decode time (ISO/IEC 14496-12, 8.6.1.2), and a sample has a composition time (8.6.1.3) but no composition duration; only the composition duration of the last sample is called unclear (8.6.1.4). So a sample is shown until the next composition time in the track, which can be in a later moof. For example, in variable frame rate content, the next frame can come after a long display gap, in the next moof.
Suppose that every sample duration is 512 ticks, and that after the frame at 2048, the next frame comes at 17920, in moof 2. Moof 1 alone gives the frame at 2048 its own duration, so it ends at 2560:
PTS 512 1024 1536 2048 2560 17920 18432
moof 1 [ I ] [ B ] [ P ]
moof 2 [ B ] [ P ]
When moof 2 is added, the next composition time is known, and the frame at 2048 is extended to it:
PTS 512 1024 1536 2048 17920 18432
moof 1 [ I ] [ B ] [ P ......................]
moof 2 [ B ] [ P ]
A sample can also come in between later. For example, with one frame per moof and B-frames, the P-frame comes before the B-frames that are shown before it. Moof 2 extends the I-frame to the P-frame, and then the B-frames of moofs 3 and 4 start inside the I-frame:
PTS 512 1024 1536 2048
moof 1 [ I .............]
moof 2 [ P ]
moof 3 [ B ]
moof 4 [ B ]
So when a new sample starts inside an earlier sample, and the earlier sample ends at the start of the next earlier sample, that end came from the next sample, and the earlier sample is ended at the new sample. The new samples are then extended up to that next earlier sample, so the time stays covered:
PTS 512 1024 1536 2048
moof 1 [ I ]
moof 2 [ P ]
moof 3 [ B ]
moof 4 [ B ]
An earlier sample that ends at no sample start keeps its own duration. Only samples next to the moof boundary in composition order can change. The search covers up to 16 samples before the new moof, the largest decoded picture buffer of H.264 and HEVC. A longer run of non-reference frames can put the next frame further back, and then the search misses it.
Step 4: rewrite the decode times
MSE coded frame processing checks each DTS. TrackBuffersManager::ProcessFrames
finds a discontinuity in two cases:
The DTS is smaller than the last DTS.
The DTS is more than 2 *
mLongestFrameDurationafter the last DTS.
After a discontinuity, the track needs a random access point. So
ProcessFrames drops each frame until the next keyframe.
The DTS from ParseTrun steps by the trun durations. But Step 3 changed the
durations, so the two can be different. So Moof::Moof rewrites the DTS of
each sample in the moof. It keeps the decode span of the moof, from the first
DTS to the decode end of the trun entries. It spreads that span over the
samples in proportion to their durations:
adjust = decode span / sum of the sample durations
DTS(n) = first DTS + (sum of the durations before sample n) * adjust
The last DTS is then never above the decode end of the trun entries. That
end is the start of the next moof only when the next tfdt matches the trun
durations.
Normal case
In moof 1 of the B-frame example, the decode span is 3 * 512 = 1536. The sum
of the durations is also 1536. So adjust is 1.0, and the DTS values stay 0,
512 and 1024. The next moof starts at 1536.
Uneven case
In the example with uneven durations, a moof has a decode span of 69150 ticks.
After Step 3, the sum of the durations is 72150 ticks. So adjust is
69150 / 72150, about 0.958. The first samples get these values:
PTS |
end |
DTS |
|---|---|---|
0 |
3001 |
0 |
6000 |
9001 |
2876 |
3001 |
6000 |
5752 |
Why the sum and not the composition span
The composition span of the moof could also be the denominator. But the span includes the gaps that the next moof fills. In moof 1 of the B-frame example, the span is 2560 - 512 = 2048. The sum of the durations is 1536.
Denominator |
Value |
|
DTS values |
Step into moof 2 |
|---|---|---|---|---|
Composition span |
2048 |
0.75 |
0, 384, 768 |
768 |
Sum of durations |
1536 |
1.0 |
0, 512, 1024 |
512 |
With the span, the DTS steps do not follow the sample durations.
Step 5: the moof time range
Moof::mTimeRange is the composition span of the moof: from the first
composition start to the last composition end. It includes the gaps that the
next moof fills. Moof::FixRounding moves its end to the start of the next
moof when the gap between them is small. The limit is 35 ms plus one tick for
each sample.
These functions use mTimeRange:
Function |
Use |
|---|---|
|
buffered ranges |
|
data that it can remove |
|
time span of the data |