Support I-frame only playback (trick play) in player and buffering logic

[jrocharodrigues]

Hi,

I'm trying to use a trickMode track on a Dash stream to display thumbnails above the seekbar when seeking.
I've already made most development and it's working.
For the thumbnails PlayerView player, i've changed the track selection parameters so that only that trickmode track is enabled.
Now i'm trying to make that player to load as less chunks as possible, and renderer as fast a possible.
To achieve that i've use a DefaultLoadControl, with custom values for Buffer Duration parameters.

val loadControl = DefaultLoadControl.Builder() .setBufferDurationsMs(1000, 5500, 1000, 1000) .createDefaultLoadControl()

With this values it works fine, but for each seek i do, it downloads 3 chunks.

What i'm trying to achieve is to have each seek downloads only one chunk, since i don't need more media (i just need do show a frame of it).
I've tried to change the maxBufferMs to 2000 ms, since my chunks length is 2 seconds, but it doesn't work as expected.
If change the maxBufferMs to a value lower than 5500 ms, on some seek operations (normally the first after player.prepare) the player will never change to STATE_READY, and will not render any media.

I've been going through the issues on github and i've saw some comments saying that lowering the maxBufferMs to extreme low values could break the playback, which i think it's what is happening to me.
I've also have been looking through the exoplayer code to try to understand why this is happening, and if there's a way i could go around it, but so far i didn't find anything very helpful.

Can help me with this?
Do you think there's a way to achieve what i'm trying to do?
Can you give me any directions on where to look to try to understand the problem?

Thanks in advance

[andrewlewis]

@tonihei Please could you take a look?

[tonihei]

Setting the maxBufferMs value to 5500 for a chunk length of 2 seconds means the player needs to load 3 chunks to reach at least 5500ms worth of media. So this is all working as expected.

To load at most one chunk at a time, you need to set maxBufferMs to less than a single chunk duration. This then only downloads the second chunk if the next load position (i.e. 2000ms) minus the current playback position is less than the configured maxBufferMs value.

What i'm trying to achieve is to have each seek downloads only one chunk, since i don't need more media (i just need do show a frame of it).

Just to verify my understanding of your code. Are you pausing the player while you do this? If not, the playback position will advance and start loading the second chunk as soon as it gets closer to its start.

I've tried to change the maxBufferMs to 2000 ms, since my chunks length is 2 seconds, but it doesn't work as expected.

The expectation would be to load 2 chunks if the chunk durations is exactly 2 seconds. Did you see something else in this case?

If change the maxBufferMs to a value lower than 5500 ms, on some seek operations (normally the first after player.prepare) the player will never change to STATE_READY, and will not render any media.

Do you have a reliable way of reproducing that? If so, it would be great if you can provide example media and reproduction steps so that we can investigate.

I'm trying to use a trickMode track on a Dash stream to display thumbnails above the seekbar when seeking.

As a more general remark: We are planning to add better support for this exact use case soon including for the problem of buffering too much data.

[jrocharodrigues]

Hi @tonihei ,
Thanks for the answer.

What i'm trying to achieve is to have each seek downloads only one chunk, since i don't need more media (i just need do show a frame of it).

Just to verify my understanding of your code. Are you pausing the player while you do this? If not, the playback position will advance and start loading the second chunk as soon as it gets closer to its start.

Yes, the player is paused.

I've tried to change the maxBufferMs to 2000 ms, since my chunks length is 2 seconds, but it doesn't work as expected.

The expectation would be to load 2 chunks if the chunk durations is exactly 2 seconds. Did you see something else in this case?

Yes, it downloads 2 chunks.

If change the maxBufferMs to a value lower than 5500 ms, on some seek operations (normally the first after player.prepare) the player will never change to STATE_READY, and will not render any media.

Do you have a reliable way of reproducing that? If so, it would be great if you can provide example media and reproduction steps so that we can investigate.

I will try to setup set the demo app to replicate this behaviour and will share with you, but it has to be shared privately. Where can i send it?

I'm trying to use a trickMode track on a Dash stream to display thumbnails above the seekbar when seeking.

As a more general remark: We are planning to add better support for this exact use case soon including for the problem of buffering too much data.

Cool, is there any code alreaddy on the public branches?

[tonihei]

but it has to be shared privately. Where can i send it?

You can send emails to dev.exoplayer@gmail.com with "Issue #7171" in the subject line.

Cool, is there any code already on the public branches?

No, unfortunately not yet. It's really just a plan so far. We can use this GitHub issue to track this implementation. I'll update the title once we figured out whether the problem with stuck buffering is an actual bug or intended behaviour.

[jrocharodrigues]

Ok, thanks.
I'll send it as soon as possible.

[jrocharodrigues]

Hi,
As agreed i sent to you an email with a patch file with the changes to replicate the issue.The patch file includes the change on media.exolist.json with a sample stream.
I can replicate the issue on my pixel 4 with the latest Android 10 security updated, and on a xiaomi mi box.
To replicate the issue just run the app, an open the "Test Stream 1".
Sometimes the first frame will get rendered, sometimes it will be stuck in buffering from the beginning.
if you click the ff or rw buttons the player will change the state to buffering but will never change to ready.
If you change the maxBufferMs to something bigger like 5500ms, it will work as expected.

Let me know if you need more info.

Thank you

[tonihei]

Thanks for sharing! I can reproduce the issue and understand why it's happening now.

We loaded the first audio and video chunk, and the video chunk has only one single sample (because it's I-frame only). The playback position is still at position 0 and we already rendered the first frame.

• We don't continue loading new chunks because the next load position is 2 seconds. This means our "duration to next load position" is 2 seconds and thus larger than the configured max buffer. This is logically correct as we expect the playback to start first before we want to continue loading.
• We don't start playback though, which would reduce the "duration to next load position" again, because the video renderer is not ready for playback. This is because it already rendered its single first sample and no further samples are available on the input side (because they are not loaded yet), nor on the output side (because we already rendered all available samples). This is logically correct too.

It's of course unfortunate that both states are logically correct in isolation but don't work together. :)
I think the underlying logical problem in our code is that this video frame is "played" for a long period of time (2 seconds) and we usually ignore the inherent duration of a sample. This is possible for other streams given that sample durations are only in the milliseconds range.

If we were considering the sample duration, then the renderer should actually be ready for as long as the last sample is still playing and also, we should take this sample duration into account when calling LoadControl.shouldStartPlayback.

I'll mark this as an enhancement to fix our playback logic for these I-frame only streams. This issue wouldn't occur (even for very small maxBuffer values in LoadControl) if the stream had a normal amount of video frames. This is easily verifiable by playing any other stream with the same LoadControl settings. For the mean time, you can use a slightly larger maxBuffer value to start loading the second chunk even if more inefficient.

Also reassign to @ojw28, because he is currently working on improving I-frame only playback support.

[jrocharodrigues]

Hi @tonihei ,

Thank you for your comprehensive answer.
I'm already using slightly larger maxBuffer as you suggested, and i'll follow the updates to this issue for future updates.

Do you still need to do more tests with the test stream i provided or can i tell the operations team they can disabled it?

[tonihei]

Do you still need to do more tests with the test stream i provided or can i tell the operations team they can disabled it?

As we are currently working on it, it would be great if you could keep the stream for a little while for testing.

[jrocharodrigues]

Sure, no problem, i'll ask them to keep it running as long as possible.
Just let me know when you finish.