Summary:
There is an issue with WebRTC for handling of certain H.264 bitstreams where the packets forming the H.264 stream has non-zero packets before the packet containing SPS.
Typically a IDR (key frame) will have SPS/PPS (if present) or the IDR slice in the first packet.
But this is not required in all cases, for example when packetization-mode = 0, you can have each NALU in separate packet. And certain NALUs can exist before SPS, for example SEI, AUD.
The way WebRTC associates width/height to encoded frames is by tracking the dependency of IDR slices to SPS/PPS.
RTP packets containing SPS/PPS have correct width/height stored in them during parsing of SPS in RtpDepacketizerH264::ProcessStapAOrSingleNalu
IDR packets refer to SPS using ppsid, spsid and the width/height fields get transferred from packet containing SPS to IDR packet in H264SpsPpsTracker::CopyAndFixBitstream.
When packets are assembled into a single encoded H264 frame in PacketBuffer::FindFrames, the loop goes through all the packets/nalus in backward scan from last RTP packet of IDR to first one.
Hence the order of NALUs during this scan is : Last parts of IDR Slice -> Mid parts of IDR Slice RTP packet -> first IDR slice Packet (this should have correct width / height) -> RTP packet containing SPS/PPS (this should have correct width/height)
start_index points to the first RTP packet of the frame and its passed into RtpFrameObject's constructor. RtpFrameObject will use the width/height stored in first RTP packet.
This works fine as long as the first RTP packet has width/height, which will be the case if first RTP packet is IDR or SPS.
In H.264 first RTP packet may be AUD, SEI in those cases, RtpFrameObject will create IDR with width/height = 0 and this causes problem for Android hardware decoders.
On Android hardware decoders rely on correct width/height to initialize the hardware decoder.
Verified on real scenario that we have.
Simulated on AppRTCMobile on IOS Simulator
Added unit tests : ninja -C out/Default && ./out/Default/modules_unittests --gtest_filter=*FrameResolution*
Bug: webrtc:11025
Change-Id: Ie2273aae5e81fd62497e1add084876a3aa05af4d
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/156260
Reviewed-by: Philip Eliasson <philipel@webrtc.org>
Reviewed-by: Sergey Silkin <ssilkin@webrtc.org>
Commit-Queue: Shyam Sadhwani <shyamsadhwani@fb.com>
Cr-Commit-Position: refs/heads/master@{#29515}
5b2df17422