One Solution for Multiple Issues
How did you overcome the problems associated with the wireless system and latency?
Delays occur with conventional wireless and video encoding technologies, so we couldn't use them as they were. Because of that, we had to ask companies who are veterans in their respective fields to take on some new challenges. A lot of companies cooperated with us in this project. (Genyo) Takeda-san10 worked hard in selecting which companies. 10. Genyo Takeda: Senior Managing Director of Nintendo and General Manager of the Integrated Research and Development Division. He is responsible for hardware development at Nintendo. In the past, he has appeared in sessions of Iwata Asks about the Wii hardware, the Wii Remote, Punch-Out!! and the Wii U console.
He brought in various companies, saying "Work hard with us on this in order to achieve something unprecedented!"
Yeah. Also key to that was the IC that Ito-san worked on. Our development partner MegaChips Corporation11 cooperated with us up to the very end, in everything from IC design to the development of the firmware that would run on the IC. 11. MegaChips Corporation: A research and development-oriented fabless maker that utilizes image, sound and communications technology to assist product development in a wide variety of areas such as the digital, mobile and security fields. Headquarters: Osaka.
How did you design the system? You had to carry out a series of actions, compressing the Wii U images, sending them wirelessly as radio waves, receiving and decompressing them on the Wii U GamePad, and then displaying them—and all without any noticeable delay!
Generally, for a video compression/decompression system, compression will take place after a single-frame12 of image data has been put into the IC. Then it is sent wirelessly and decompressed at receiving end. The image is sent to the LCD monitor after decompression is finished.
But since that method would cause latency, this time, we thought of a way to take one image and break it down into pieces of smaller images. We thought that maybe we could reduce the amount of delay in sending one screen if we dealt in those smaller images from output from the Wii U console GPU13 on through compression, wireless transfer, and display on the LCD monitor.
12. Frame: In video, a frame represents a single still image. The greater the number of frames processed in a certain amount of time, the more the volume of data increases. In order to show smooth video, a certain number of frames are necessary. Television and video usually display 30 frames per second. At present, HDTV shows up to 60 frames per second.
13. GPU (graphics processing unit): Also called a graphics chip or video chip, this is a special chip for rendering computer and game console displays.
When you first talked about that with those around you, was there a good feeling from the start?
Yes, there was.
I thought it was good from the start, too. If the amount of data which needs to be buffered is not going to be big, it would minimize latency. You can get by with less memory, and with less power consumption, so it was a good example of a single solution solving multiple issues.
Generally, compression for a single screen can be done per a 16×16 macroblock14. And on Wii U, it rapidly compress the data, and the moment the data has built up to a packet-size15 that can be sent, it sends it to the Wii U GamePad.
14. Macroblock: A macroblock is a small unit used when breaking down a single frame. Generally, it is 16×16 pixels.
15. Packet size: A packet is a unit of measurement for data volume when sending and receiving data. Packet communication refers to sending and receiving data broken down into packets.
That is completely different from the usual way of compressing video. But there is no way to guarantee that data sent via wireless communication will reach its destination, so you had to think about when errors would occur.
Yes. The wireless side was really hard. The data would come in rapidly in small portions, so it was difficult to put it together and deliver it in real time while minimizing errors as much as possible.
And if even a single bit of the data composing a particular screen were missing, there wouldn't be enough data to continue decompressing afterward, so finding how to handle those situations was a problem.
We simulated all kinds of ideas addressing error concealment.16 16. Error concealment: This refers to correction and restoration with regard to errors that occur in digital data.
Usually in image compression and wireless communication, it allow for latency to maintain quality. If we tightened up on latency, even if it ran well under the best conditions, it was difficult to address problems—for example, sending certain images when signals are weak—that arose in specific situations.
It's difficult in wireless communication when conditions aren't constant.
Right. It's extremely tough, wirelessly, to have players hold the Wii U GamePad and move with it because of the Doppler effect17. 17. Doppler effect: This is a phenomenon observed when wave frequencies differ due to relative speeds between an observer and the source of electromagnetic and sound waves.
And games that involve all kinds of movement are going to come out one after the other.
To be honest, I even thought, "No, please don't!" (laughs) Also, it's okay to hold it vertically.
To address why holding it vertically may have been an issue, is because radio waves don't spread well underwater, but the human body is 60 to 70 percent water, so it can interfere with radio wave paths. Since you hold it both vertically and horizontally, it gets more difficult to place the antenna where the radio waves can easily reach it. If the cost wasn't an issue, there would be many ways to resolve it like adding more antennas. Besides, since you make various movements with the Wii U GamePad, the radio waves might get distorted. You had to think about all those problems.
Everyone will probably test to see how far the radio waves will reach in their house. (laughs)
As Nintendo, we say that it will be fine using it within the same living room where the console is in, but a lot of people ask what about through a wall?
Yeah. The other day, someone in a different department asked if he could use it in his bathroom at his house! (laughs)
Differences will arise depending on whether you live in a house made of wood or an apartment of reinforced concrete, and what materials the walls are made out of.
Yes. What we can say for certain is that it will be fine within the same space.
However, if you place the Wii U console in something like a metal TV stand it may deflect the radio waves thus reducing its usable range. Radio waves weakens by the square of the distance, so even within the same space, too much distance could make them weaker, and having obstacles in between would be a disadvantage.
Using the Wii U GamePad with a television is one of the ways that you can use it, but a good thing about the Wii U is that you can also play without the TV*. Because of the way Wii U is used it becomes easier to forget about the console, so it's possible that some people might wonder why it doesn't work when they're away from the TV. By the way, my living room and bathroom are separated by a single wall and I was able to use it. (laughs) (*Editor's note: This functionality is supported in specifically designed games.)
Playing at Yamashita-san's house worked with one wall in-between! (laughs) When asked "Can I use it in my bedroom?" we can say that it would work within the same space when there is nothing in the way, but basically it depends on how your house is constructed, so we have to ask that you test it in your own home.
