HEVC Improves Mobile Video Performance

Our mobile devices wouldn’t be the same without video capabilities. What would life be like for us (or our kids) without YouTube, Tout, Facebook, and now Vine, with all of the advantages of easily posting our own (or our favorite) videos in the cloud to share with others?

Nearly all Americans experience the benefits of video compression every day through satellite and cable TV feeds and from Netflix and Amazon Prime Instant Video streams. New products like Roku, Google Chromecast, and Apple TV are using our Web connections as streaming video spigots. In fact, some pundits nervously note that Netflix and YouTube together already consume about half of North America’s Internet bandwidth!¹

Related Articles

• Compression Requires The Right Tool For The Job
• Big Data Plumbing Problems Hinder Cloud Computing
• New Techniques Help Compression Surpass Its Practical Limits

All of those video streams (to mobile devices, tablets, and traditional TV screens) are compressed. Without the roughly 30:1 compression provided by the Advanced Video Coding (AVC; also known as H.264) standard that was approved in 2003, these video experiences simply wouldn’t be available to huge, growing audiences.

In March 2013, I attended the annual Data Compression Conference and attended the keynote speech by Gary Sullivan, video architect at Microsoft.² Gary described the High Efficiency Video Coding (HEVC; also known as H.265) standard, which had just been approved in January 2013. As co-chair of the HEVC Joint Video Team (JVT), Gary has guided this impressive new video coding standard since 2003. Given the importance of video to billions of consumers viewing hundreds of millions of screens, I’d like to share a brief summary of the HEVC standard, describing both its similarities and differences when compared to its AVC predecessor.

What Stays The Same

HEVC is a significant improvement to AVC, but not a wholesale replacement. Between the two video compression standards, many of the techniques, terminology, and tricks remain the same. Specifically, both video coders remove three kinds of redundancy from video streams: spatial redundancy within each frame, temporal redundancy between frames, and statistical redundancy between coding units.

AVC and HEVC divide video frames into smaller units, but the AVC macroblocks have been augmented into HEVC coding tree units (CTUs), which can include one, four, or 16 macroblocks. Both AVC and HEVC specify the operation of the video decoder and the syntax of the encoded bit stream. As long as a video encoder generates a stream that the decoder can parse and decode, that’s good enough to meet the standard.

The HEVC still uses AVC’s I, P, and B frame types. The I-frames are standalone frames that only take advantage of intra-frame encoding techniques. Motion vectors contained in P (“predicted”) and B (“bi-directional) frames refer to macroblocks in previously transmitted frames to describe macroblocks in the current frame. The transmission order of HEVC coding units can differ from the display order. In other words, encoded frames can be sent out of order and re-ordered in the decoder, which AVC also permitted.

What’s New

What AVC problems or challenges does HEVC solve, or at least ameliorate? First and foremost, HEVC delivers the same video quality at half the corresponding AVC bit rate. Equivalently, a given bit rate delivers visibly better video quality using HEVC than using AVC, as adjudicated by human viewers. HEVC also addresses several implementation details that AVC did not address, or (more likely) details that AVC members didn’t realize would be so challenging to implementers.

Specifically, HEVC was designed for scalability allowing multiple HEVC hardware decode accelerators to operate independently, increasing decoder throughput and decreasing decoder latency. In contrast to AVC’s 2k (1920 by 1080) displays, HEVC supports the faster frame rates and growing pixel counts of today’s 4k and 8k displays. Today’s leading-edge displays require sophisticated video decoders whose components can operate in parallel.

Several video coding parameters have been improved since AVC was ratified (see the table). In the decade since AVC was adopted, the cost of flat-panel displays has declined significantly. Indeed, many of us have mounted such displays on our living room walls. Pixel density has also grown with panel size, and as previously mentioned, today’s larger displays boast four to 16 times more pixels than AVC’s 2k displays. In the last few years, color depth (bits per color component) has also increased from AVC’s 8 bits per pixel. Although experts argue about the ability of human vision to appreciate (or even notice) improved color depth, more bits per pixel has become a recognized marketing feature of flat-panel TVs, computer monitors, and tablets.

HEVC also includes a new “Main 10” profile specifically to support 10-bit video. HEVC CTUs can be stacked hierarchically into tiles and wavefronts, which aid in parallel video decoding. Since each wavefront is independent of other wavefronts, HEVC decoders can decode multiple wavefronts in parallel, using independent HEVC decoding accelerators. By explicitly defining a hierarchy of independently decodable video units, the HEVC standard provides more explicit guidance to hardware manufacturers about how decoding throughput can be improved, as well as how video decoding latency can be reduced.

However, the improved performance of HEVC comes at a significant cost, especially for the HEVC encoder. Although estimates differ, an HEVC decoder is between three and four times more complex (as measured by MIPS or silicon area) than an AVC decoder. A simple metric for determining when compression technologies “hit the wall” is when algorithm complexity increases faster than compression gain.³

Since (compared to AVC) HEVC decoder complexity has tripled while compression ratio has only doubled, it appears that HEVC is close to hitting the wall. Another indicator is the 10 years that passed between the adoption of AVC and HEVC standards, indirectly indicating the diminishing compression return from increasingly complex video coding algorithm improvements.

What Comes After HEVC?

Now that HEVC has been approved and is beginning its commercial deployment, what’s left to do? The Joint Video Team continues to meet, and its new focus areas include 3D video, further scalability, and lower-delay video decoding techniques. Given the significantly increased HEVC complexity, video compression researchers will spend several years developing empirical rules of thumb for HEVC encoding. Developing such rules of thumb will be critical, given the exponential increase in HEVC encoder parameters (settings, levels, and tiers).

We may not see HEVC’s successor for many years. As video standards continue to improve, video researchers may be well advised to moderate video decoder complexity and to reduce the exponential expansion of the video encoder’s parameter space.