Motion estimation (ME) and motion compensation (MC) have been extensively used for video frame interpolation systems since the turn of the century. However, typical solutions are power-hungry and introduce interpolation errors. Pixelworks, a video alchemist skunkworks in San Jose, California, has been quietly developing a new back-end mobile processor that works for both movies and now video games. The result is an incredibly brilliant image, at super high resolutions and high frame rates, but with minimal latency that can actually improve battery life as well—it sounds almost as if they have broken the law of physics. But no, says their CEO, we just know those laws very well and how to work within them.

By using an expert-trained video motion estimation and compensation algorithm in silicon for video frame interpolation, the company has pushed the bounds limiting performance and quality. Through a novel adaptive compensation scheme trained by the company’s picture-quality experts (that the company doesn’t want to talk too much about), Pixelworks has integrated motion processing—things they’ve done for the film and TV industry in software—into a mobile gaming solution that is now possible in a smartphone.

This secret sauce, the expert training by their best engineers and picture-quality people, is differentiable such that the flow and estimation can be optimized jointly in real time, and very fast, with minimal latency. In addition to the smart MEMC, the design can be seamlessly adapted to several other video enhancement tasks such as super-resolution, denoising, and deblocking. Quantitative and qualitative evaluations and demonstrations have shown the X7 post processor outperforms state-of-the-art video frame interpolation and enhancement algorithms on a wide range of datasets. Compared to existing methods, Pixelworks’ magic is computationally efficient and able to generate more visually appealing results.

One segment that appreciates this is gaming, and as most people know, gaming has become a big business in units and dollars worldwide. It is especially big in China, where it is estimated there are 670 million gamers—20 percent of the world’s estimated 3.3 billion gamers. Most of those gamers play on a smartphone. Chinese gamers have adopted Genshin Impact from miHoYo as a benchmark. One of the elements in the game is weather. Weather is an exploration mechanic. While exploring the world, the weather can shift. Depending on the new weather conditions, special properties will be applied to the environment, to the characters, and to opponents.

Want More Tech News? Subscribe to ComputingEdge Newsletter Today!

Games like Genshin Impact and PC games like Cyberpunk 2077 that have been ported from the PC to smartphones challenge the phone’s performance vs. battery and heat operating parameters. Big games demand everything the phone’s processor has, and in the process, runs down the battery while heating up the phone.

Pixelworks saw that as an opportunity to apply its magic and introduced the X7 chip architecture, post processors that exploited some of the techniques the company used in the film and TV industry. The X7 chips created a new distributed computing/rendering chip market segment in mobile—a bit like what Nvidia and AMD have done in the PC and console segments. Chinese phone manufacturers have referred to X7 as stand-alone display solution, a render accelerator, or a dual-chip gaming architecture.

MEMC and resolution scaling (RS) are very demanding on any platform and especially in a mobile device with limited cooling and power supply. The phone must deliver a high frame rate, with high color accuracy, high pixel resolution, no artifact or aliasing, and low latency—and accomplish all that while using very little power.

Pixelworks’ post processor has proven to be just what the doctor ordered: a unique technology for motion compensation and resolution scaling that off-loads the phone’s SoC. For gaming specifically, by taking care of the motion vectors and working with the game engine developers, the company’s X7 processor uses metadata, which guarantees the best gaming experiences.

As a result, three out of four of the largest Chinese phone OEMs have been shipping their top phones with Pixelworks’ chips. The leading game engine providers (e.g., Unity) have been partnering with Pixelworks, the big game studios in China (ByteDance, Tencent, miHoYo, etc.) have engaged with Pixelworks, and mobile SoC processor makers such as MTK and Qualcomm are working with Pixelworks.

The X7 is a tiny chip and draws so little power that the smartphone hardly knows it’s running. Some smartphone suppliers in China have already added it to their phones for the game-crazy customers they have, and if one can satisfy that price-performance market segment, then they must have something special.

This is a classic story of a smallish (~$70 million in revenue) company quietly working in close collaboration with some big-name and colossally demanding customers and perfecting a solution that is scalable, efficient, and economical.

Pixelworks didn’t just stumble into this market. Founded in 1997, the company developed video processors for professional projectors. The video engineers and scientists at the company had a collective range of experience unmatched in companies many times their size. But, they were technogeeks, not salesmen. You almost had to know someone who knew someone to find out about them. Their customers weren’t eager to reveal much about them, either—Pixelworks was their secret sauce and how they differentiated themselves in a cutthroat take-no-prisoners business.

The mythology of the company is the CEO or CTO (the story varies depending on who is telling it) was chatting at a conference with a CTO or president of a phone company who said, “It’s my dream that someday we will have smartphones that you can watch a movie on, and it will be as good as being in the theater.” The Pixelworks guy went home thinking about that, and the next day he told his team, “I’ve got an idea.”

A few years later, when visiting the lab (if you could get in), one could see a big 4K screen showing video and games as if it was driven by an enthusiast-level PC, but it wasn’t—it was driven by a two-year-old smartphone, and not even a top-of-the-line smartphone.

The term skunkworks isn’t used lightly here. Pixelworks has managed to beat out the top SoC builders in Silicon Valley and San Diego in video processing. In one sense, that’s not too surprising—video is all they do. Today’s smartphones are a marvel of technology, but they can’t be the best at everything they do. Product cycles and R&D budgets mitigate how much they can invest in time and dollars. Pixelworks is a specialty company—a pixel polisher. They can also polish pixels coming from the sensor but acknowledge that the big SoC builders had done a really good job with their integrated ISPs—you climb one mountain at a time, the CEO told me.

The tantalizing end of this story is not really an end but the future. Pixelworks has now proven their bespoke approach to video processing—pixel polishing works; in fact, it works almost better than they thought it would. And it’s scalable. It will ride Moore’s law and just get better with each generation. Smartphones have (or will have) never looked so good.