Enabling the Fix

Sometimes cliches are true. Sometimes the check is in the mail. And sometimes you can fix it in post. Amazingly, the category of what you can fix might be getting a lot bigger.

At this month’s NAB show, there was the usual parade of new technology, from Sonic Notify’s near-ultrasonic smartphone signaling for extremely local advertising — on the order of two meters or so (palmable transducer shown at left) to Japan’s National Institute of Information and Communications Technology’s TV “white space” transmissions per IEEE 802.22. In shooting, for those who like the large-sensor image characteristics of the ARRI Alexa but need the “systemization” of a typical studio/field camera, there was the Ikegami HDK-97ARRI (right), with the front end of the former and the back end of the latter.

Even where items weren’t entirely new, there was great progress to be seen. Dolby’s autostereoscopic (no glasses) 3D demo (left) has come a long way in one year. So has the European Project FINE, which can create a virtual-camera viewpoint almost anywhere, based on just a few normally positioned cameras. Last year, there was a lot of processing time per frame; this year, the viewpoint repositioning was demonstrated in real-time.

If you’re more interested in displays, consider what’s been going on in direct-view LED video. It started out in outdoor stadium displays, where long viewing distances would hide the visibility of the individual LEDs. At NAB 2013, two companies, Leyard (right) and SiliconCore, showed systems with 1.9-mm pixel pitch, leaving the LED structure virtually invisible even at home viewing distances. Is “virtually” not good enough? SiliconCore also showed their new Magnolia panel, with a pitch of just 1.5 mm!

The Leyard display shown here (and at NAB) was so-called “4K,” with more than twice the number of pixels of so-called “Full HD” across the width of the picture. 4K also typically has 2160 active (picture carrying) lines per frame, twice 1080, so it typically has four times the number of pixels of the highest-resolution for of HD.

4K was unquestionably the major unofficial theme on the NAB show floor, replacing the near-ubiquitous 3D of two years ago. There were 4K lenses, 4K cameras, 4K storage, 4K processing, 4K distribution, and 4K displays. Using a form of the new high-efficiency video codec (HEVC), the Fraunhofer Institute was showing visually perfect 4K pictures with their bit rates reduced to just 5 Mbps; with the approval of the FCC, that means it could be possible to transmit multiple 4K programs simultaneously in a single U.S. broadcast TV channel. But some other things in the same booth seemed to be attracting more attention, including ordinary HD images, shot by INCA, a tiny, 2.5-ounce “intelligent” camera, worn by an eagle in flight. The eagle is shown above left, the camera, with lens, at right. The seemingly giant attached blue rod is a thin USB cable.

Throughout the show floor, wherever manufacturers were highlighting 4K, visitors seemed more interested in other items. The official theme of NAB 2013 was METAMORPHOSIS, with the “ME” intended to stand for media and entertainment, not pure self interest. But most metamorphoses seemed to have happened before the show opened. Digital cinematography cameras aren’t new; neither are second-screen applications. Mobile DTV was introduced years ago. So was LED lighting.

There were some amazing new technologies discussed at NAB 2013 — perhaps worthy of the metamorphosis label.  But they weren’t necessarily on the show floor (at least not publicly exhibited). Attendees at the SMPTE Technology Summit on Cinema (TSC), for example, could watch large-screen bright images that came from a laser projector.

The NAB show was vast, and the associated conferences went on for more than a week. So I’m going to concentrate on just one hour, a panel session called “Advancing Cameras for Cinema,” in one room, the SMPTE TSC, and how it showed the metamorphosis of what might be fixed in post.

Consider the origin of post, the first edit, and it was a doozy! It occurred in 1895 (and technically wasn’t exactly an edit). At a time when movies depicted real scenes, The Execution of Mary, Queen of Scots, in its 27-foot length (perhaps 17 seconds), depicts a living person being led to the chopping block. Then the camera was stopped, a dummy replaced the person, the camera started again, and the head was chopped off. It’s hard to imagine what it must have been like to see it for the first time back then. And, since 1895, much more has been added to the editing tool kit.

It’s now possible to combine different images, generate new ones, “paint” out wires and other undesirable objects, change colors and contrast, and so on. It’s even possible to stabilize jerky images and to change framing at the sacrifice of some resolution. But what if there were no sacrifice involved?

The first panelist of the SMPTE TSC Advancing Cameras session was Takayuki Yamashita of the NHK Science & Technology Research Labs. He described their 8K 120-frame-per-second camera. 8K is to 4K approximately as 4K is to HD, and 120 fps is also four times the 1080i frame rate. This wasn’t a theoretical discussion; cameras were on the show floor. Hitachi showed an 8K camera in a familiar ENG/EFP form (left); Astrodesign showed one dramatically smaller (right).

If pictures are acquired at higher resolutions, they may be reframed in post with no loss of HD resolution. With 8K, four adjacent full-HD-resolution images can be extracted across the width of the 8K frame and four from top to bottom. A shakily captured image that bounces as much as 400% of the desired framing can be stabilized in post with no loss of HD resolution. And the higher spatial sampling rate also increases the contrast ratio of fine detail.

Contrast ratio was just one of the topics in the presentation, “Computational Imaging,” of the second panelist, Peter Centen of Grass Valley. Above is an image he presented at the SMPTE summit. The only light source in the room is the lamp facing the camera lens, but every chip on the reflectance chart is clearly visible and so are the individual coils of the hot tungsten filament. It’s an extraordinarily high dynamic range (HDR); a contrast ratio of about ten million to one — more than 23 stops — was captured.

Yes, that was an image he presented at the SMPTE summit — five years ago in 2008. This year he showed a different version of an HDR image. There’s nothing wrong with the technology, but bringing it to the market is a different matter.

At the 2013 TSC, Centen showed an even older development, one first presented by an MIT-based group at SIGGRAPH in 2007 <http://groups.csail.mit.edu/graphics/CodedAperture>, a so-called “coded aperture.” Consider a point just in front of a camera’s lens. The lens might zoom in or out and might focus on something in the foreground or background. Its aperture might be wide open for shallow depth of field or partially closed for greater depth of field. If it’s a special form of lens (or lenses), it might even deliver stereoscopic 3D. All of those things might happen after the light enters the lens, but all of those possibilities exist in the “lightfield” in front of the lens.

Leg Asea

2013 HPA Tech Retreat Broadcasters Panel: ABC, CBC, CBS, Ericsson, Fox, NAB, NBC, and PBS are shown (not in order); EBU, NHK, Sinclair, Univision, and locals were also present

Joe Zaller, a manager of the very popular (16,000-member) Television Broadcast Technologies group on LinkedIn, tweeted on February 21 from the 2013 HPA Tech Retreat in Indian Wells, California: “Pretty much blown away from how much I learned Wednesday at [the broadcasters] panel… just wish it had been longer.”

Adam Wilt, in his huge, six-part, 14-web-page (each page perhaps 20 screens long) coverage of the five-day event for ProVideoCoalition.com put it this way: “When you get many of the best and brightest in the business together in a conference like this, it’s like drinking from a fire hose. That’s why my notes are only a faint shadow of the on-site experience. Sorry, but you really do have to be there for the full experience”: http://provideocoalition.com/awilt/story/hpa-tech-retreat-wrap-up

In his Display Central coverage, Peter Putman called it “one of the leading cutting-edge technology conferences for those working in movie and TV production”: http://www.display-central.com/free-news/display-daily/4k-in-the-desert-2013-hpa-tech-retreat/. The European Broadcasting Union’s technology newsletter noted of the retreat, held in the Southern California desert, “There were also many European participants at HPA 2013, in particular from universities, research institutes and the supplier industry. It has clearly become an annual milestone conference for technology strategists and experts in the media field”: http://tech.ebu.ch/news/new-media-technology-on-the-agenda-at-te-22feb13

It was all those things and more. HPA is the Hollywood Post Alliance, but the event is older than HPA itself. It is by no means restricted to Hollywood (presenters included the New Zealand team that worked on the high-frame-rate production of The Hobbit and the NHK lab in Japan that shot the London Olympics in 8K), and it’s also not restricted to post. This year’s presentations touched on lighting, lenses, displays, archiving, theatrical sound systems, and even viewer behavior while watching one, two, or even three screens at once.

It is cutting-edge high tech–the lighting discussed included wireless plasmas, the displays brightnesses as high as 20,000 cd/m² (and as low as 0.0027), and the archiving artificial, self-replicating DNA–and yet there was a recognition of a need to deal with legacy technologies as well. Consider that ultra-high-dynamic-range (HDR) display.

The simulator created by Dolby for HDR preference testing is shown at left, minus the curtains that prevented light leakage. About the only way to achieve sufficient brightness today is to have viewers look into a high-output theatrical projector. In tests, viewers preferred levels far beyond those available in today’s home or theatrical displays. But a demonstration at the retreat seemed to come to a different conclusion.

The SMPTE standard for cinema-screen brightness, 196M, calls for 16 foot-lamberts or 55 cd/m² with an open gate (no film in the projector). With film, peak white is about 14 fL or 48 cd/m², a lot lower than 20,000. Whether real-world movie theaters achieve even 48–especially for 3D–is another matter.

During the “More, Bigger, But Better?” super-session at the retreat, a non-depolarizing screen (center at right) was set up, the audience put on 3D glasses, and scenes were projected in 3D at just 4.5 fL (15 cd/m²) and again at 12 fL (41 cd/m²). The audience clearly preferred the latter.

Later, however, RealD chief scientific officer Matt Cowan showed a scene from an older two-dimensional movie at 14, 21, and 28 fL (48, 72, and 96 cd/m²). This time, the audience (but not everyone in the audience) seemed to prefer 21 to 28. Cowan led a breakfast roundtable one morning on the question “Is There a ‘Just Right’ for Cinema Brightness?”

Of course, as Dolby’s brightness-preference numbers showed, a quick demo is not the same as a test, and people might be reacting simply to the difference between what they are used to and what they were shown. The same might be the case with reactions to the high-frame-rate (HFR) 48 frames per second (48 fps) of The Hobbit. When the team from Park Road Post in New Zealand showed examples in their retreat presentation, it certainly looked different from 24-fps material, but whether it was better or worse was a subjective decision that will likely change with time. There were times when the introduction of sound or color were also deemed detrimental to cinematic storytelling.

At least the standardized cinema brightness of 14 fL had a technological basis in arc light sources and film density. A presentation at the retreat revealed the origin of the 24-fps rate and showed that it had nothing to do with visual or aural perception or technological capability; it was just a choice made by Western Electric’s Stanley Watkins (left) after speaking with Warner Bros. chief projectionist Jack Kekaley. And we’ve gotten used to that choice for 88 years.

Today, of course, actual strands of film have nothing to do with the moving-images business–or do they? Technicolor’s Josh Pines noted a newspaper story explaining the recent crop of lengthy movies by saying that digital technology lets directors go longer because they don’t have to worry about the cost of film stock. But Pines analyzed those movies and found they they had actually, for the most part, been shot on film.

Film is also still used for archiving. Major studio blockbusters, even those shot, edited, and projected electronically, are transferred to three strands of black-&-white film (via a color-separation process), even though that degrades the image quality, for “just-in-case” disaster recovery; b&w film is the only moving-image medium to have thus far lasted more than a hundred years.

At one of the 2013 HPA Tech Retreat breakfast roundtables (right) one morning, the head of archiving for a major studio shocked others by revealing they were no longer archiving on film. At the same roundtable, however, studios acknowledged that whenever a new restoration technology is developed, they go to the oldest available source, not a more-recent restoration.

If film brightness, frame rate, and archiving are legacies of the movie business, what about television? There was much discussion at the retreat of beyond-HDTV resolutions and frame rates. Charles Poynton’s seminar on the technology of high[er] frame rates explained why some display technologies don’t have a problem with them while others do.

Other legacies of early television also appeared at the retreat. Do we still need the 0.999000… frame-rate-reduction factor of NTSC color in an age of Ultra-HD? It’s being argued in those beyond-HD standards groups today.

Interlace and its removal appeared in multiple presentations and even in a demo by isovideo in the demo room (a tiny portion of which is shown at left). As with film restoration from the original, the demo recommended archiving interlaced video as such and using the best-available de-interlacer when necessary. And there appeared to be a consensus at the retreat that conversion from progressive to interlace for legacy distribution is not a problem.

There was no such consensus about another legacy of early television, the 4:3 aspect ratio. One of the retreat’s nine quizzes asked who intentionally invented the 16:9 aspect ratio (for what was then called advanced television), what it was called, and why it was created. The answers (all nine quizzes had winners) were: Joseph Nadan of Philips Labs, 5-1/3:3, and because it was considered the minimum change from 4:3 that would be seen as a valuable-enough difference to make consumers want to buy new TV sets. But that was in 1983.

Thirty years later, the retreat officially opened with a “Technology Year in Review,” which called 2013 the “27th (or 78th) Annual ‘This Is the Year of HDTV.’” It noted that, although press feeds often still remain analog NTSC, according to both Nielsen and Leichtman research in 2012 75% of U.S. households had HDTVs. Leichtman added that 3/5 of all U.S. TVs, even in multi-set households, were HDTV. Some of the remainder, even if not HDTV, might have a 16:9 image shape. So why continue to shoot and protect for a 4:3 sub-frame of the 16:9?

On the broadcasters panel, one U.S. network executive explained the decision by pointing to other Nielsen data showing that, as of July 15 of 2012, although roughly 76% of U.S. households had HDTVs (up 14% from the previous year), in May only 29% of English-language broadcast viewing was in HD and only 25% of all cable viewing. Furthermore, much of the legacy equipment feeding the HDTV sets is not HD capable.

A device need not be HD capable, however, to be able to carry a 16:9 image. Every piece of 4:3 equipment ever built can carry a 16:9 image, even if 4:3 image displays will show it squeezed. So the question seems to be whether it’s better for a majority of U.S. TV sets to get the horizontally stretched picture above left or a minority to get the horizontally squeezed picture at right.

What do actual viewers think about legacy technologies? Two sessions scarily provided a glimpse. A panel of students, studying in the moving-image field, offered some comments that included a desire to text during cinema viewing. And Sarah Pearson of Actual Customer Behaviour in the UK showed sequences shot (with permission) in viewer homes on both sides of the Atlantic, analyzed by the 1-3-9 Media Lab (example above left). Viewers’ use of other media while watching TV might shock, but old photos of families gathered around the television often depicted newspapers and books in hand.

It wasn’t only legacy viewing that was challenged at the retreat. Do cameras need lenses? There was a mention of meta-materials-based computational imaging.

Do cameras need to move to change viewpoint, or can that be done in post? Below are two slides from “The Design of a Lightfield Camera,” a presentation by by Siegfried Foessel of Germany’s Fraunhofer Institut (as shot off the screen by Adam Wilt for his ProVideoCoalition.com coverage of the retreat: http://provideocoalition.com/awilt/story/hpa-tech-retreat-day-4). Look at the left of the top of the chair and what’s behind it.

The Habit and “The Hobbit”

Here are a couple of questions to get you started: What is the image at left? And what is the sound of a telephone call?

I’ll offer some more information about the first one. It’s an “intertitle,” the sort of thing inserted into silent movies to help advance their plot.

This one happens to be from a pretty famous movie. Got any idea yet of which one? You’re likely to be familiar with it even if you never saw it. But the answer might be surprising.

Now, how about that telephone call? Bell Labs researcher and audio pioneer Harvey Fletcher wanted its sound to be unidentifiable, i.e., just as good as being there. Today, if you use a certain type of mobile phone, you might be able to identify certain negative artifacts, but, in general, with contemporary technology, Fletcher’s dream has been achieved: a telephone call sounds pretty much like any other reproduction of an electronic audio signal. And that’s a problem.

When the kidnapper calls to demand ransom in a movie or TV thriller, the camera might offer a close-up of the person taking the call, but the kidnapper’s voice shouldn’t sound like it’s coming from the same room. So a voice filter is used, typically restricting the bandwidth of the sound to a range from roughly 300 Hz to 3 kHz as shown at the right in the Cisco white paper “Wideband Audio and IP Telephony” <http://bit.ly/116U1Mn>.

If you’re familiar with sampling theory, you know that, to avoid spurious frequencies known as aliases, sampling must be done at a rate higher than twice the desired highest frequency, and the signal must be filtered to prevent anything higher than that highest desired frequency from entering the sampler. Filters are imperfect, so, if a telephone company wanted to sample 8,000 times per second, it would not be totally unreasonable for the system to pass little more than 3 kHz.

Digital transmission systems don’t care about filtering low frequencies, however, so why the 300 Hz low-frequency cutoff? It dates back to analog transmission systems, wherein different frequencies would be attenuated by different amounts, and an equalizer would restore them. The attenuation might be described as a certain number of decibels per decade. A decade, in this case, is a tenfold increase in frequency, as from 300 Hz to 3 kHz. Going down to 30 Hz from 300 would add another decade, doubling the equalization needed.

Today, in the era of digital transmission, going down to 30 or even 20 Hz would not be a problem, which is why people describe today’s real-world telephone calls in such terms as “sounding like you’re next to me.” But the sound of a telephone-call voice in a movie or on TV still harks back to an earlier era (just as a print ad might tell its viewer to “dial” a certain phone number in an era when it’s hard to find a dial-equipped phone outside a museum).

It’s not easy on a visual web page to provide examples of telephone call sounds, especially since I have no idea what your listening equipment is like. But here is another common example of a motion-image-media indicator that strays from reality: the binoculars mask.

If you use binoculars, you probably know you’re supposed to adjust their eye separation so that there’s one circular image, not the lazy eight shown at left. But, if there’s no binoculars mask effect, how is a viewer supposed to know that the scene is seen through binoculars?

Now, perhaps, we can consider frame rate. Though he wanted telephone calls to sound just like being there in person, Fletcher did the research that identified the 300 Hz-to-3 kHz range for speech intelligibility and identification. Are there physical parameters affecting choice of frame rate? There are more than one.

One is typically called the fusion frequency, the frequency at which a sequence of individual pictures appears to be a motion picture. You can find your own fusion frequency with a common flip book; an 1886 version called a Kineograph is shown at right.

Flip through the pages slowly, and they are individual still pictures. Flip through them quickly, and they are a single motion picture.

Unfortunately, there is no single fusion frequency. It varies from person to person and with illumination, color, angle, and type of presentation.

The type of presentation becomes significant in another frame-rate variable: what’s commonly called the flicker frequency, the rate at which sources of illumination appear to be steady, rather than flickering.

Some of the earliest motion-picture systems took advantage of a fusion frequency generally lower than the flicker frequency. They presented motion pictures, but they flickered, thus an early nickname for movies: flickers or flicks.

One “solution” to the flicker problem was the use of a two-bladed shutter in the projector. A film image would be moved into place, the shutter would turn, the image would appear on screen, the shutter would turn again, the image would disappear, it would turn again, it would reappear, and it would turn again while a new image moved into place. The result was an illumination-repetition rate twice that of the frame rate, perhaps enough to achieve the flicker frequency, depending, again, on a number of viewing factors.

While the two-bladed (or, in some cases, three-bladed) shutter helped ameliorate flicker, it introduced a new artifact into motion presentation. A moving object would appear to move from one frame to another but to stall in mid-motion from one shutter opening to another. Clearly, that was a step away from reality, but, like a limited-bandwidth telephone call and a binoculars mask, it tended to indicate the look of a movie.

What rate is required? When Thomas Edison initially chose 46 frames per second (fps) for his Kinetoscope, he said it was because his research had showed that “the average human retina was capable of taking 45 or 46 photographs in a second and communicating them to the brain.” But the publication Electricity, in its June 6, 1891 issue, contrasted the Kinetoscope’s supposed 46 fps with Wordsworth Donisthorpe’s Kinesigraph’s six-to-eight: “Now, considering that the retina can retain an impression for 1/7 of a second, 8 photographs per second are sufficient for the purpose of reproduction and the remaining 38 are mere waste.”

Is there a “correct” frame rate? This week’s Super Bowl coverage made use of For-A’s FT-One cameras (above), which can shoot 4K images at up to 900 fps. But that was for replay analysis.

At the International Broadcasting Convention (IBC) in Amsterdam in 2008, the British Broadcasting Corporation (BBC) provided a demonstration in the European Broadcasting Union (EBU) “village” that showed how frame rates as high as 300 fps could be beneficial for real-time viewing. At left is a simulation of 50-fps (top) vs. 100-fps (bottom), showing a huge difference in dynamic resolution (detail in moving images).

Note that the stationary tracks and ties are equally sharp in both images. The moving train, however, is not. Other parts of the demonstration showed that high-definition resolution might appear no better than standard-definition for moving objects at common TV frame rates.

A clear case seemed to be made for frame rates higher than those normally used in television. Again, that was in 2008. In 2001, however, Kodak, Laser-Pacific, and Sony each won an engineering Emmy award for making possible 24-fps video–video at a lower frame rate than that normally used.

As the BBC/EBU demo at IBC clearly showed, 24-fps video has worse dynamic resolution than even normal TV frame rates, let alone higher ones. Yet 24-fps video has also been wildly successful. It provides a particular look, just as a binoculars mask does. In this case, the look contributes to a sensation that the sequence was shot on film. But why did movies end up at 24-fps? It’s not Edison’s 46 nor Donisthorpe’s 8.

The figure is based on research but not research into any form of visual perception. Go back to the intertitle at the top of this column. Have you guessed the movie yet? It’s The Jazz Singer, the one that ushered in the age of sound movies, even though, as the intertitle shows, it, itself, was not an all-singing, all-talking movie.

Some say 24-fps was chosen as the minimum frame rate that would provide sufficient sound quality. But The Jazz Singer, like many other sound movies, used a sound-reproduction system, Vitaphone, unrelated to the film rate: phonograph disks. In the 1926 demo photo above, engineer Edward B. Craft holds one of the 16-inch-diameter disks. Their size and rotational speed (33-1/3 rpm, the first time that speed had been used) were carefully chosen for sound quality and capacity, but they could have been synchronized to a projector running at any particular speed.

That was the key. Sound movies did not require 24-fps, but they required a single, standardized speed. The choice of that speed fell to Stanley Watkins, an employee of Western Electric, which developed the Vitaphone process. Watkins diligently undertook research. According to Scott Eyman’s book The Speed of Sound (Simon & Schuster 1997), he explained the process in 1961:

“What happened was that we got together with Warners’ chief projectionist and asked him how fast they ran the film in theaters. He told us it went at 80 to 90 feet per minute in the best first-run houses and in the small ones anything from 100 feet up, according to how many shows they wanted to get in during the day. After a little thought, we settled on 90 feet a minute [24-fps for 35 mm film] as a reasonable compromise.”

That’s it. That’s where 24-fps came from: no visual or acoustic testing, no scientific calculation, just a conversation between one projectionist, one engineer, and, according to Watkins’s daughter Barbara Witemeyer in a 2000 paper (“The Sound of Silents”), Sam Warner (of Warner Bros.) and Walter Rich, president of Vitaphone. After Vitaphone and Warner Bros., Fox adopted the speed, and soon it was ubiquitous.

Fluke or not, 24 fps came to symbolize the look of film, which is why 24-fps video is so popular. We have a habit of associating that rate with movies.

The Hobbit broke that habit. It is available in a 48-fps, so-called “HFR” (high-frame-rate) version. And its look has received some unusual reviews.

Some have complained of nausea. It’s conceivable that there is some artifact of the way The Hobbit has been projected in some theaters (in stereoscopic 3D) that triggers a queasiness response in some viewers, but it seems (to me) more likely that those viewers might be reacting to some overhead, spinning shots in the same way that viewers have reacted to roller-coaster shots in slower-frame-rate movies.

Others have complained of a news-like or video-like look that made it more difficult for them to suspend disbelief and get into the story. That’s certainly possible. If 24-fps contributes to the look of what we are in the habit of thinking of as a movie, then 48-fps is different.

Of course, we no longer watch flickering silent black-&-white movies with intertitles, projected at a rate faster than they were shot, either. Times change.