HLS support for trick play with EXT-X-I-FRAMES-ONLY and EXT-X-I-FRAME-STREAM-INF

[Arqu]

I can't seem to find any reference in the code to these tags and would like to know if this could be implemented as it would allow more advanced control of the video stream. This would basically lift the ExoPlayer to EXT-X-VERSION 4 of HLS.

[ojw28]

Using I-Frame only variants looks pretty optional to me. Marking as enhancement, but probably wont happen any time soon.

[Arqu]

Yes, the I-Frame only is not really required for stream playback and core functionality, however it allows us to play in fast forward and reverse which can enhance a users experience. It is even referred to as a key feature by Apple.

[Arqu]

I've started to work on this, once there is a substantial amount I'll submit it. Currently I've just added parsing of the media playlist tags, but made sure none of them are processed and executed until this is implemented.

However, I wanted to ask how should I approach implementing playback in this case, i.e. only play those I-FRAMES and play them in some term of I-FRAMES/second instead of the usual FPS? Maybe even expose this playback speed manipulation so that you can pretty much do 2x 4x 8x... fast forward.

[Arqu]

I've found out, that if you simply open up an I-FRAME list, there are two things that happen:

1. It starts playing - thus playback is more or less possible, I just had to play around a little bit with timing and render it immediately to the screen.
2. It gets stuck on the first chunk and keeps reloading it infinitely. Seems like it reads the bytes ok but somewhere along the way the extractor gets stuck and never gets prepared thus throwing the player into an infinite prepare loop.

So any thoughts about case 2? I've spent all day tracking down where it breaks...

[Arqu]

I've managed to get a lot of things going regarding this and most of the code is already up on: https://github.com/Arqu/ExoPlayer/tree/dev however I wouldn't want to merge it yet as it is only basic support for directly playing back IFRAME media playlists (It is not really specified that way in the HLS spec but it helps to get things going). However to get this thing done I need to manage a few more things, one is #531 and the other is how to allow switching to an IFRAME track. I've also added some basic code to indicate if a variant has an IFRAME playlist associated with it, but I still have to get down to the part of managing if we are playing the full stream or the IFRAME playlist. So if somebody could help out a bit, especially with the buffering part, that would be great.

EDIT:
There is something else regarding IFRAME playback, I found out that I had an issue playing Apple's samples of IFRAME playlists and investigated a bit. Found out that in their streams, the IFRAME list doesn't have a single chunk with the PID 0 indicating the required table for demuxing streams therefore it never plays. After further analysis I found that their full video streams have a first chunk with PID 0 and the following are what ever comes from the table, so to implement switching between IFRAME playback and normal playback, we need to actually keep using the same extractor with the correct mappings of PMT and PAT indexes from the original stream and then switch the source of the chunks to the IFRAME list (like when the user clicks on fast forward).

However to get this going I first need to solve the problems above to make IFRAME playback in a sensible manner even possible.

[deepvernekar]

Hi,
Currently i am working on HLS4.0 trick play. i just parsed the i-frame playlist. also i was trying to play those segments. Initially due to byte range offset i was getting few errors and found that offset value will be skips reading PMT/PAT. After that i removed the byterange param and played the segment and found playing perfect. I need to understand more about intraframes as well as byte range. can you please explain me about that. Let me know if you have any good code example for the same.

[n2lovell]

I'm currently planning to implement this in a partial c++ port of exoplayer (ported most of the manifest module layer + fmp4 extractors).

I'm currently thinking about making the I-Frames track(s) as a special type of track that the player explicitly switches to.

General idea: When entering a trick mode, flush the sample queue(s) and then rapidly seek around the special track (number of i-frames seeked to can be determined based upon the current network rate). Switching back is "trivial" at least for segment level seeks (I haven't looked at Exo's support for seeks within a segment, the port's handling of the sample queue is outside of the manifest modules).

1. Expose the special tracks like media playlists, but marked as trick mode tracks.
2. Have the track selector interface get an explicit argument to switch to the trick mode tracks.
   As a result of 1 +2: the chunkSources can then treat the selection mostly as just a track change (conceivably it is just a track change, but I've heard some things that potentially make seeking to the live point problematic -- so its likely that it might have to be able to terminate the trick mode if it can't fulfill the seek)

I'm interested in feedback on the design because I'm expecting that at some point, I can get another team to switch to using mostly mainline Exoplayer for their Android playback. In which case, I'd like to get a similar code change done and attempt to push the change upstream.

[ojw28]

That sounds along the right lines. I'd probably do it as follows, which I think is similar to what you're suggesting:

• Expose the i-frame track as a special track in its own TrackGroup. A good first step would just be getting this working as a track that can be played. This should be fairly straightforward.
• Add a new C.SelectionFlags constant to indicate the track is for trick play, and set it on the Format for the trick play track. This should also be straightforward.
• Add a Player.setTrickPlayEnabled, which triggers a track re-selection internally. TrackSelector.selectTracks would need to be modified to take an additional boolean indicating whether the selection is for trick play mode.

At this point you probably have something that more or less works, although you may want to make some further tweaks to make things better, like:

• If seeks arrive so quickly that each new seek is received before a frame is loaded at the previous position, then I think the player can end up rendering nothing. To fix this, when in trick play mode, it probably makes sense to defer processing seeks if a frame has yet to be rendered after the previous seek. Then when that happens, discard all but the last pending seek, and process that.
• Preventing playback from proceeding whilst in trick play mode, even if playWhenReady is true.
• You may want to have tell the Renderers that they're in trick play mode, in case they want to do any kind of special behavior. I'm not sure what that would be though :).

[n2lovell]

Just to report back, I finally got around to implementing this recently in the port (took ~1 week). Through to the sampleQueues it was pretty simple to implement, after that the HAL layer I used did the rest.

For a clarification as what I meant when I said seek. I think the closest approximation in Exo is changing the nextLoadPosition so that instead of loading the very next segment, instead load N seconds beyond the last load position (this prevents having to download every single i-frame listed in the playlist and instead only loads the ones that will be displayed).

Skipping frames definitely has consequences though, since it requires a flush for every rate change (with the design the port uses) -- in practice I've found that it hasn't mattered though. If you choose not to skip i-frames, then you need to discard frames before sending them to the decoder.

After that it was more or less cheating given that the HAL layer I use already supported trickmodes directly within the decoder. I.e. I just dumped the frames into the decoder, and told it to run at 8fps.

The sequence in the port was roughly:
User enters i-frame based trickmode
-- SelectTracks is triggered with forceReset
-- Causing Flush queues/decoder
-- Reconfigure decoder. Had to propagate a flag to the renderer to say to enter a new mode.
-- ChangeNextLoadPos to skip i-frames based on expected rate

[bkodirov]

@n2lovell Do you have a PR for that?
I'm interested on the implementation.

[stevemayhew]

Oliver's comment seems the best path forward, to treat the track as "special" and use trackselection to activate it.

Other players that implement trickplay with the iFrame playlist treat it basically as a level down, switching to it when the bandwidth required exceeds the matching current spacial resolution's lowest bitrate * the playback rate, for practical purposes this is less then 5-8x.

We will start with TrackGroup / TrackSelector implementation, as that is needed regardless. I do suspect code will need to work with the seek and renderer logic to prioritizing rendering some frame[s] as playback progresses vs achieving a target frame rate.

[stevemayhew]

Oliver (@ojw28), please reference the design document here: https://drive.google.com/file/d/1_efCflJ-OfmRgu03qn1MbqQ7wsNYozj-/view?usp=sharing Let me know if anyone else wants to be involved in the effort please feel free to comment

I am close to having code (ExoPlayer code branched from dev-v2) to share that will publish an Iframe stream that will render with MediaCodecVideoRenderer.

Please have a look at the design and let me know if the path is acceptable, I can include a Markdown version of the document with the pull request.

[ojw28]

Thanks a lot for the doc! As an aside for possible future reference (I wouldn't bother converting what you've done or anything :)), using Google Docs is pretty nice for letting people comment on specific parts of a doc, and/or potentially co-editing.

A few comments:

IFrameVariant

It's probably OK just to use the regular Variant class with the @Nullable fields set to null (except possibly videoGroupId), unless there's a particularly compelling reason for a new class.

except there are no other Renditions possible from it

It's possible to set a VIDEO= attribute on an EXT-X-I-FRAME-STREAM-INF. It seems it's required to do this in the case of alternate video renditions. This is not something we support properly currently, but we'd like to at some point.

is HlsMediaPeriod will expose the IFrameVariant as a TrackGroup

In the case of multiple IFrameVariants, I assume the proposal is to create a single TrackGroup containing all of them?

Some more general feedback:

1. I'm not sure, but thinking about this a bit more, I wonder whether i-frame tracks should just go into the existing video TrackGroup. We don't otherwise split different frame-rates into their own track groups (e.g. 30fps / 60fps), and an i-frame only track is kinda just a track that happens to have a very low frame-rate. Putting them in the existing TrackGroup allows for the possibility of a TrackSelection implementation that adapts between regular and i-frame only tracks, which seems like a nice thing to allow even if that's not what happens by default. Having a separate TrackGroup makes this impossible, which seems like an unnecessary constraint to impose. I think this is definitely a good argument for streams where audio and video are delivered separately (i.e. DASH, SmoothStreaming, and HLS specifically when audio is split out). It becomes a lot more complicated when there can be audio muxed in with the video though :(. Any thoughts? If it's uncertain what's best to do here, we should probably design things to work both ways so that we can easily adjust to one way or another of doing this later on.
2. I think i-frame only tracks should still be TRACK_TYPE_VIDEO. A ROLE_FLAG_TRICK_PLAY or (possibly more accurate) ROLE_FLAG_I_FRAME_ONLY value for Format.RoleFlags seems like an appropriate way to mark tracks that are i-frame only.
3. As you allude to, renderers might want to do something differently when playing a trick play track. It would be preferable if this could be done dynamically by video renderers, by looking at the RoleFlags of the current input format. This would be necessary to support normal track <-> i-frame track adaptation.

[stevemayhew]

On Jul 23, 2019, at 10:31 AM, Oliver Woodman ***@***.***> wrote: Thanks a lot for the doc! As an aside for possible future reference (I wouldn't bother converting what you've done or anything :)), using Google Docs is pretty nice for letting people comment on specific parts of a doc, and/or potentially co-editing.

It’s MarkDown, so I’ll put it in an ExoPlayer feature request, that way people can comment and edit it.

A few comments: IFrameVariant It's probably OK just to use the regular Variant class with the @nullable fields set to null (except possibly videoGroupId), unless there's a particularly compelling reason for a new class. except there are no other Renditions possible from it It's possible to set a VIDEO= attribute on an EXT-X-I-FRAME-STREAM-INF. It seems it's required to do this in the case of alternate video renditions. This is not something we support properly currently, but we'd like to at some point.

Thanks, yes it is possible to have VIDEO renditions on an I-FRAME-STREAM-INF, at least in the spec.. I’ve not seen it in practice, so using Variant will cover us there. Also, some master playlists include only a single i-Frame playlist, others include multiple such that you can ABR between them

is HlsMediaPeriod will expose the IFrameVariant as a TrackGroup In the case of multiple IFrameVariants, I assume the proposal is to create a single TrackGroup containing all of them

Correct, following this pattern and it just works for AdaptiveTrackSelection (beautiful to see ;-)). Apple recommends 1/7 the bandwidth of the same spacial resolution, this allows fast playback.

Some more general feedback: I'm not sure, but thinking about this a bit more, I wonder whether i-frame tracks should just go into the existing video TrackGroup. We don't otherwise split different frame-rates into their own track groups (e.g. 30fps / 60fps), and an i-frame only track is kinda just a track that happens to have a very low frame-rate. Putting them in the existing TrackGroup allows for the possibility of a TrackSelection implementation that adapts between regular and i-frame only tracks, which seems like a nice thing to allow even if that's not what happens by default. Having a separate TrackGroup makes this impossible, which seems like an unnecessary constraint to impose. I think this is definitely a good argument for streams where audio and video are delivered separately (i.e. DASH, SmoothStreaming, and HLS specifically when audio is split out). It becomes a lot more complicated when there can be audio muxed in with the video though :(. Any thoughts? If it's uncertain what's best to do here, we should probably design things to work both ways so that we can easily adjust to one way or another of doing this later on.

Single iFrame track should support different play rates, the trick is how it is rendered ;-).

I think i-frame only tracks should still be TRACK_TYPE_VIDEO. A ROLE_FLAG_TRICK_PLAY or (possibly more accurate) ROLE_FLAG_I_FRAME_ONLY value for Format.RoleFlags seems like an appropriate way to mark tracks that are i-frame only.

Ok cool, I had not seen the RoleFlags, looks perfect for this purpose.

As you allude to, renderers might want to do something differently when playing a trick play track. It would be preferable if this could be done dynamically by video renderers, by looking at the RoleFlags of the current input format. This would be necessary to support normal track <-> i-frame track adaptation.

I think so, still looking at exactly what to do in the renders.

…

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <#474?email_source=notifications&email_token=AADBF6BCPU6HF7A3L5UHR2LQA46APA5CNFSM4BEMAVJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2T3NYY#issuecomment-514307811>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADBF6HAXTQEAUU4A4JCCE3QA46APANCNFSM4BEMAVJA>.

[ojw28]

Single iFrame track should support different play rates, the trick is how it is rendered ;-).

I'm not sure how that addresses my comment about putting regular and i-frame only tracks into the same TrackGroup. Please clarify.

[stevemayhew]

Yes, sorry Oliver, I gave kind of a terse answer for such a detailed question. ’m not sure I understand all of what your asking, but let me try again...

I'm not sure, but thinking about this a bit more, I wonder whether i-frame tracks should just go into the existing video TrackGroup. We don't otherwise split different frame-rates into their own track groups (e.g. 30fps / 60fps), and an i-frame only track is kinda just a track that happens to have a very low frame-rate. Putting them in the existing TrackGroup allows for the possibility of a TrackSelection implementation that adapts between regular and i-frame only tracks, which seems like a nice thing to allow even if that's not what happens by default.

Interesting thought, I need to verse myself more deeply in TrackSelection, is there anything beyond this: https://exoplayer.dev/track-selection.html <https://exoplayer.dev/track-selection.html> One thing to consider, the i-Frame only track not only lowers the frame rate but also drops information (that is all the frames between the IDR are dropped), so I’m not sure it fits as adaptable with the other variants in the TrackGroup. Adapting regular playback to an i-Frame track would be a choice, bandwidth wise, between a very low frame rate at a higher spacial resolution vs lower spacial resolution with all the frames.

… Having a separate TrackGroup makes this impossible, which seems like an unnecessary constraint to impose.

Yes, I see that. It’s an easy change to make (all in same TrackGroup) if use cases make more sense to include i-Frame with the regular variants

I think this is definitely a good argument for streams where audio and video are delivered separately (i.e. DASH, SmoothStreaming, and HLS specifically when audio is split out). It becomes a lot more complicated when there can be audio muxed in with the video though :

The I-Frame track never includes muxed audio… Since the PTS syncs with the demuxed audio and CC tracks it works coincidently to enable all these tracks, however if you fast play (forward or reverse) the I-Frame track the audio will not work (captions maybe, up to a certain playback speed)

…

On Jul 23, 2019, at 12:20 PM, Oliver Woodman ***@***.***> wrote: Single iFrame track should support different play rates, the trick is how it is rendered ;-). I'm not sure how that addresses my comment about putting regular and i-frame only tracks into the same TrackGroup. Please clarify. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#474?email_source=notifications&email_token=AADBF6GJOJSVAJCAURC5ZCTQA5KXNA5CNFSM4BEMAVJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2UE7YI#issuecomment-514346977>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADBF6CAH3OU6AVPV2CK4DDQA5KXNANCNFSM4BEMAVJA>.

[ojw28]

One thing to consider, the i-Frame only track not only lowers the frame rate but also drops information (that is all the frames between the IDR are dropped)

Is that logically any different to dropping every other frame in the case of a 30fps track where 60fps tracks are available (other than being far more extreme)? Putting the i-frame only tracks in the same TrackGroup as regular video tracks doesn't imply that adaptation between them will occur. The TrackSelector (and TrackSelections) determine which tracks from the group are used and how adaptation between them occurs. By default I'd propose i-frame only tracks are not considered. The point is that if they're in the same TrackGroup it will be possible for a TrackSelection to enable adaptation to i-frame tracks, if it wants to. Where-as if they're no, this wont be possible.

Interesting thought, I need to verse myself more deeply in TrackSelection, is there anything beyond this: https://exoplayer.dev/track-selection.html

The JavaDoc for the track selection classes is reasonably good, so you may find that helpful as well.

The I-Frame track never includes muxed audio

Yes, this may be a problem for the HLS case where audio is muxed.

[stevemayhew]

One thing to consider, the i-Frame only track not only lowers the frame rate but also drops information (that is all the frames between the IDR are dropped)

Is that logically any different to dropping every other frame in the case of a 30fps track where 60fps tracks are available (other than being far more extreme)? Putting the i-frame only tracks in the same TrackGroup as regular video tracks doesn't imply that adaptation between them will occur...

Thanks... The more I look at it the more it makes sense. The overhead to run an additional HlsSampleStreamWrapper just for the I-Frame only playlists seems wasted. Interestingly the FRAME-RATE attribute is not allowed on I-Frame only playlist, it can be inferred from the durations in the sub-playlist. So we will want a role or some other Format attribute to mark them.

The point is that if they're in the same TrackGroup it will be possible for a TrackSelection to enable adaptation to i-frame tracks, if it wants to. Where-as if they're no, this wont be possible.

I'm on board now, yes.. I think using adaptation to switch to i-frame tracks based on some criteria is indeed the correct way to go.

The I-Frame track never includes muxed audio

Yes, this may be a problem for the HLS case where audio is muxed.

When you adapt to an I-Frame only track audio and captions are no longer linked. These are not allowed attributes as per the HLS spec (https://tools.ietf.org/html/draft-pantos-hls-rfc8216bis-04#page-35) not sure about DASH. Will look into if adaption changes allow for dropping audio and caption render.

[stevemayhew]

Pull request 6270 will address this issue.

[stevemayhew]

#6270

[lupin4th]

@stevemayhew
How can I playback test with i frame track on the demo app?

[stevemayhew]

I did not want to muddy the pull request with code in the demo app. It’s pretty simple though: 1. Use track selection to turn off audio 2. Set playback speed up considerably (15x or more) ABR will switch to your iframe only track (assuming it is authored correctly, it should have lower bandwidth requirements then any other video track). I’ll put some code snippets here later today when I’m at my computer If you can share your stream with me I’ll check if it’s authored correctly

…

On Thu, Sep 26, 2019 at 1:20 AM lupin4th ***@***.***> wrote: @stevemayhew <https://github.com/stevemayhew> How can I playback test with i frame track on the demo app? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#474?email_source=notifications&email_token=AADBF6BBHKVSIVFLXHVIXITQLRWDZA5CNFSM4BEMAVJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7UXWTI#issuecomment-535395149>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADBF6DF476WVKFDVTI7GY3QLRWDZANCNFSM4BEMAVJA> .

[lupin4th]

@stevemayhew
Thank you for your answer.
It works well.
Is i-frame track playback possible at 4x, 2x?

[stevemayhew]

Sorry, I got busy today. You need to override the trackselector and set it to return the iframe track when playback rate exceeds 3 to 4x and turn of audio With this you should easily get 15x or so depending on your iframe bandwidths and connection speed

…

On Tue, Oct 1, 2019 at 8:54 PM lupin4th ***@***.***> wrote: @stevemayhew <https://github.com/stevemayhew> Thank you for your answer. It works well. Is i-frame track playback possible at 4x, 2x? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#474?email_source=notifications&email_token=AADBF6DGBP6LU6DITSGZZT3QMQLOLA5CNFSM4BEMAVJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEADOR5A#issuecomment-537323764>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADBF6AGJOELT3ZLDX6FLX3QMQLOLANCNFSM4BEMAVJA> .

[lupin4th]

@stevemayhew
I use apple's sample url(https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8).

As you said I turn off Audio

mTrackSelector.setParameters(
				mTrackSelector
						.buildUponParameters()
						.clearSelectionOverrides(1)
						.setRendererDisabled(1, true));

and Set playback speed

PlaybackParameters playbackParameters = new PlaybackParameters(16.0f);
    player.setPlaybackParameters(playbackParameters);

Then I checked selectedPlaylistUrl in getNextChunk method(HlsChunkSource.java).

However, it has not been changed to i-frame url.

[lupin4th]

I changed adapativeTrackSelection canSelectFormat method

protected boolean canSelectFormat(
      Format format, int trackBitrate, float playbackSpeed, long effectiveBitrate) {
    boolean isNonIframeOnly = (format.roleFlags & C.ROLE_FLAG_TRICK_PLAY) == 0;
    boolean canSelect = playbackSpeed < 8;

    return canSelect && isNonIframeOnly;  
  }

As above, the selectedPlaylistUrl changes when the speed is more than 8x.

However, error occurs as shown below.

ava.lang.IllegalArgumentException
        at com.google.android.exoplayer2.util.Assertions.checkArgument(Assertions.java:39)
        at com.google.android.exoplayer2.upstream.DataSpec.<init>(DataSpec.java:248)
        at com.google.android.exoplayer2.upstream.DataSpec.subrange(DataSpec.java:336)
        at com.google.android.exoplayer2.upstream.DataSpec.subrange(DataSpec.java:322)
        at com.google.android.exoplayer2.source.hls.HlsMediaChunk.feedDataToExtractor(HlsMediaChunk.java:346)
        at com.google.android.exoplayer2.source.hls.HlsMediaChunk.loadMedia(HlsMediaChunk.java:325)
        at com.google.android.exoplayer2.source.hls.HlsMediaChunk.load(HlsMediaChunk.java:301)
        at com.google.android.exoplayer2.upstream.Loader$LoadTask.run(Loader.java:381)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:764)
2019-10-10 11:32:00.681 32237-32237/com.google.android.exoplayer2.demo E/EventLogger: internalError [15.42, 60.86, window=0, period=0, loadError]
    java.io.EOFException
        at com.google.android.exoplayer2.extractor.DefaultExtractorInput.readFromDataSource(DefaultExtractorInput.java:265)
        at com.google.android.exoplayer2.extractor.DefaultExtractorInput.readFully(DefaultExtractorInput.java:72)
        at com.google.android.exoplayer2.extractor.DefaultExtractorInput.readFully(DefaultExtractorInput.java:81)
        at com.google.android.exoplayer2.extractor.mp4.FragmentedMp4Extractor.readSample(FragmentedMp4Extractor.java:1293)
        at com.google.android.exoplayer2.extractor.mp4.FragmentedMp4Extractor.read(FragmentedMp4Extractor.java:330)
        at com.google.android.exoplayer2.source.hls.HlsMediaChunk.feedDataToExtractor(HlsMediaChunk.java:357)
        at com.google.android.exoplayer2.source.hls.HlsMediaChunk.loadMedia(HlsMediaChunk.java:325)
        at com.google.android.exoplayer2.source.hls.HlsMediaChunk.load(HlsMediaChunk.java:301)
        at com.google.android.exoplayer2.upstream.Loader$LoadTask.run(Loader.java:381)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:764)

[stevemayhew]

@stevemayhew
Thank you for your answer.
It works well.
Is i-frame track playback possible at 4x, 2x?

Depends on the density of I-Frames in the track, the compression of the I-Frame, network bandwidth, frame rate the codec hardware you have will support.

If you have 2 second GOP then the base rate would be .5fps, so even 30x would be 15fps, very possible.

[stevemayhew]

@2wchuang reverse is possible (in theory), however the current PlaybackParameters restricts this with an assert (wisely so, as it will take rework to support this). For normal streams (mix I/B/P with audio) reverse playback is practically impossible. With an IFRAME only stream (and only rendering video) it is certainly possible, we are investigating a design for this but it will take changes to the Renderer, SampleStream API (needs playback speed/direction) and the sample stream implementations need to be able to deal with:

1. PTS appearing in reverse order.
2. Fetching chunks from a playlist in-reverse order
3. Buffering a back buffer

In short, no small order.

[AquilesCanta]

This is already fully supported.

I think this bug should have been marked as fixed since the last commit (2b1ec5c). Since then, fmp4 i-frame-only playback was broken, for which I filed #7512. #7512 has also been fixed recently.

[stevemayhew]

You issue repeated seeks to play reverse, you can stay paused (play when ready set false). Turn off sound and subtitles then select the iFrame track with an override

…

On Fri, Jan 3, 2020 at 1:23 AM 2wchuang ***@***.***> wrote: Does it support play hls on reverse？ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#474?email_source=notifications&email_token=AADBF6DVHVVY5CTO6N5234DQ337YTA5CNFSM4BEMAVJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIAVYXA#issuecomment-570514524>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADBF6G72YRU43QN2CCKTJ3Q337YTANCNFSM4BEMAVJA> .

[SteveCrichton]

@stevemayhew

First of all, thanks for your work on this issue. I thought I was going to have to generate mp4 files which contain only a single I-Frame and then manually load those into the player at intervals in order to enable scrubbing when fast forwarding and rewinding. Doable but tedious to keep everything synchronized with my actual stream. This is much cleaner.

You issue repeated seeks to play reverse, you can stay paused (play when ready set false). Turn off sound and subtitles then select the iFrame track with an override

Your suggestion for reverse playback makes sense and I have implemented it for enabling this functionality for fmp4 i-frame-only tracks. (I am outputting 1 fps fmp4 variant streams in 6 second chunks which contains only IDR frames from my encoder as it doesn’t support outputting i-frame-only variants, running those through a processor which extracts the 6 samples in each chunk and breaks them into 6 individual fragments each of which contains a single sample, and then dynamically generating media playlists for the i-frame-only streams and adding the appropriate tags to the master playlists).

One thing I’m seeing is that the player is doing a lot of unnecessary buffering when I am “rewinding”. Every time I seek back (while paused) it buffers as if it is going to play the i-frame track from the new position (which will never happen because I will switch back to my “normal” variant before playing again). This all makes sense because the player doesn’t know what I’m doing but it is impacting performance both on the server side and the client side (and the network in between for that matter). My biggest concern is the client side as the player doesn’t return to the ready to play state and display the i-frame I’m seeking to until it has built a buffer (that I know it is never going to use) and this sometimes doesn't complete before I want to seek back again.

I implemented a custom load control which allows me to modify the buffer parameters on the fly (I reduce the thresholds when performing reverse playback) but have had only limited success. If I set the buffer to be too short the player never returns to the ready to play state. In fact, it never even calls the shouldStartPlayback method of my load control. (I think this may have something to do with imperfect alignment between the playlists for my i-frame-only variants my “normal” variants but I haven’t dug into that theory yet).

I’m thinking that the next thing I’ll try is to implement a caching data source so that the player doesn’t have to go back to the server to get the same chunks over and over (if I seek back 1s then it would only need to get one more chunk) but I know this isn’t going to improve things much if I start seeking back several seconds at a time and thought I'd ask if you might be willing to share some advice before I continue.

Any suggestions would be much appreciated.