What’s Next?

These were some of the things that could be seen at Canon Expo at New York’s Javits Convention Center last week: ice skaters pirouetting without ice, people viewing someone dressed as the Statue of Liberty from the moving deck of a fake boat, a machine that can squirt out a printed-and-bound book on demand, and a hand-holdable x-ray system.  Those weren’t directly related to the future of our business.  But what about image sensors with 120 million pixels, others (sensor chips) larger than paperback books, and yet others with more colors than merely red, green, and blue?

[The photo above, by the way, like the others in this post from Canon Expo, was shot by Mark Forman <http://screeningroom.com/> and is used here with his permission (all other rights reserved).]

We can extrapolate from the past to make certain predictions.  It’s extremely likely, for example, that the sun will rise tomorrow (or, for those of a less-poetic bent, that the rotation of the Earth will cause…).  Otherwise, we can’t predict the future, but we’re often put in a position of having to do so:  Will this stock go up?  Will it rain on during an outdoor wedding ceremony?  Will there be a better, less-expensive camera/computer/etc. after a purchase?

That last is usually as assured as a daily sunrise, but how quickly and how great the improvement are hard to know.  For help, there are blogs like this, publications, conferences, and trade shows.

The Internationale Funkausstellung  (IFA) in Berlin is an example of one of the latter.  It’s an international consumer electronics show.

At the latest IFA, among other stereoscopic 3D offerings (including 58-inch, CinemaScope-shaped, 21:9 glasses-based 3D), Philips spinoff Dimenco showed an auto-stereoscopic (no-glasses) 3D display.  Here’s a portion of a photo of it that appeared on TechRadar’s site here: http://www.techradar.com/news/television/hdtv/philips-to-launch-glasses-free-3d-tv-in-2013-713951

This is by no means the first time Philips has ventured into no-glasses 3D, but this one is different.  Autostereoscopic displays usually involve a number of views, and the display resolution gets divided by them.  The more views, the larger the viewing sweet spot and the better the 3D but the lower the resolution.  The new display has five views horizontally and three vertically, but it starts with twice as much resolution as “full 1080-line HD” both horizontally and vertically, so the 3D images end up with a respectable 768 x 720 for each of 15 views.

Perhaps such glasses-free 3D leads to a greater sensation of immersion, but there are other ways to create (or increase) an immersive sensation.  Consider, for example, the CAVE (Cave Automatic Virtual Environment), a room with stereoscopic projections on at least three walls and the floor (sometimes all surfaces).  The photo here is of a CAVE at the University of Illinois in 2001 (it was developed there roughly 10 years earlier).  SGI brought a CAVE to the National Association of Broadcasters convention shortly after it was developed.

Visitors who wore ordinary 3D glasses saw ordinary 3D — boring.  Visitors who got to wear a special pair of 3D glasses that could track their head movements, however, even though they saw exactly the same 3D as the others, were transported into a virtual world responsive to their every movement.  Unfortunately, only one viewer at a time could get the immersive experience.

At Canon Expo, however, there was “mixed reality.”  It’s based on head-mounted displays using two prisms per eye.  One, a special “free-form prism,” delivers images from a small display to the eye.  The other passes “real-world” images from in front of the viewer to both the eye and a video camera that can tell what the viewer is looking at.

The result is definitely mixed reality, a combination of stereoscopic imagery with unprocessed vision, with the 3D virtual images conforming to objects and views in the “real world.”  Virtual images can even be mapped onto real-world surfaces, with the cameras in the headgear telling the processors how to warp the virtual images appropriately.  This photo shows a complex version of the headgear; other mixed-reality viewers at Canon Expo looked little different from some 3D glasses.  Canon’s “interactive mixed reality” brochure showed people wearing the headgear walking around and collaboratively discussing an object that doesn’t exist.

Another form of immersion involves capturing 360-degree images.  At left is the Immersive Media Dodeca® 2360 camera system, combining the images from 11 different cameras and lenses into a seamless panorama.  At Canon Expo, a 360-degree view was achieved with a single lens, a single imaging chip (8984 x 5792, with 3.2 μm pixel pitch) and a mirror shaped like a cross between a donut and a cone that is, in the words of one high-ranking Canon employee, “the single most-precise optical component the company makes.”  The whole package forms a roughly fist-sized bump.

Of course, immersiveness is only one visual sensation.  There are also sharpness and color.

If you work out the math on that  Canon 360-degree image sensor, it comes to about 50 million pixels, which is considerably more than even NHK’s Super Hi-Vision (also known as ultra high-definition television, with four times the detail of 1920 x 1080 HDTV in both the horizontal and vertical directions).  Across the room from Canon’s 360-degree system, however, was their version of ultra-high resolution, with roughly eight times the detail of 1080-line HDTV in both directions.

Four Super Hi-Vision pictures could fit into one from this hyper-resolution sensor.  Canon says its resolution is comparable to the number of human optic nerves.

The full detail of the chip can only (currently) be captured at only about 1.4 frames per second, but while it is shooting hyper-detailed stills, it can (if I interpreted the information provided correctly) simultaneously capture two full-motion full-detail HDTV streams within the image.  The system uses a one-of-a-kind lens, and it’s a work in progress.

The hyper-resolution image sensor had a roughly full-frame 35mm format (comparable to that in the Canon EOS 5D Mark II DSLR), already roughly four-and-a-half times taller than a 2/3–inch format image sensor.  A few feet away was another new sensor that was larger — much larger.  It was made from a semiconductor wafer the size of a dinner plate, and the sensor itself was the size of an old 8-inch-square floppy disk — huge!

What do you get from such a huge sensor?  Extraordinary sensitivity and dynamic range.  One scene (said to have been shot at 60 frames per second with an aperture of f/6.8) showed stars in the sky as seen through a forest canopy — and it was easy to see that the leaves and needles of the trees were green.  In another scene, a woman walks in front of a table lamp, so she is back lit, but every detail and shade of gray in of her front was clearly visible.

Canon Expo demonstrated advances in both immersiveness (aside from the 360-degree and mixed-reality systems, there was also the 9-meter dome projection shown at right) and in spatial sharpness (the hyper-resolution and giant image sensors, the latter because it can deliver more contrast ratio, which affects sharpness).  There are also temporal sharpness (high frame rate) and spatio-temporal sharpness, both of which affect our perceptions of sharpness.  I found no demonstrations of increased temporal or dynamic resolution at Canon Expo, but that doesn’t mean they’re not being developed.

The images at left are portions taken from BBC R&D White Paper number 169 on “High Frame-Rate Television” published in September 2008.  It’s available here: http://www.bbc.co.uk/rd/pubs/whp/whp169.shtml The upper picture shows a toy train shot at the equivalent of 50 frames per second; the lower picture shows the same train at 300-fps.  Note that the stationary tracks and ties are equally sharp in both pictures, but the higher frame rate makes the moving train sharper in the lower picture.

As this post shows, there is immersiveness, and there is sharpness (both spatial and temporal).  Is there anything else that future imaging might bring?  How about advances in color?

Ever since its earliest days, color video has been based on three color primaries.  As this chromaticity diagram shows, however, human vision encompasses a curved space of colors, whereas any three primaries within that space define a triangle, excluding many colors.

At Canon Expo, one portion of the new-technologies section was devoted to hand-held displays that could be tilted back and forth to show the iridescence of butterfly wings and other natural phenomena.  The demonstration wasn’t to highlight the displays but a multi-band camera that captures six color ranges instead of three.

Then there was the Tsuzuri Project exhibit at Canon Expo (http://www.canon.com/tsuzuri/index.html).  It was a gorgeous reproduction of an ancient Japanese screen.  Advanced digital technology was used to capture and reproduce the detail of the original, but then a master gold-leaf artist used his talents to complete the copy.

I look forward to future tools based on what I saw at Canon Expo as well as the BBC’s high frame-rate viewing, Immersive Media’s camera system, and even the Philips autostereoscopic display.  And I’m glad that human artists are still needed to use them.

Tags: 3d, autostereoscopic, BBC R&D, Canon, Canon Expo, Dimenco, Dodeca, immersive media, mixed reality, Philips, Tsuzuri Project, ultra-resolution

Good News: I See 3D

It has been a good 3D month for me.  A local cinema has been running a series of classic 3D movies from the 1950s, including Alfred Hitchcock’s Dial M for Murder, which takes place almost entirely in one room of an apartment (okay, a London flat), an ideal distance range for natural stereoscopic 3D.  The master director was also sure to fill the depth with furniture and other props.

Then I saw Step Up 3D, a younger director’s much more recent masterful use of the medium.  I highly recommend Jon Chu’s feature for 3D viewing.

I’ve been working on a 3D research project; I heard from inventor Jimmie D. Songer, a pioneer of single-lens, single-camera 3D; and then, today, ISee3D.  That last wasn’t a grammatical error and typo.  Perhaps I should have said I saw ISee3D.

They, too, have been working on single-lens, single-camera 3D.  They’ve even done it at high speed for slow motion, and they showed me 3D endoscope footage shot in the interior of a red pepper.  But the best part of the demo was the little camera in a corner of a conference room, shooting live motion full-color 3D (displayed live on an ordinary 3D screen at the opposite corner of the room).

That was one camera with one lens, picking up 3D.  You can read more about it (including a technology white paper and their base patent) here: http://isee3d.com/

No, you can’t rush out and buy one just yet, but, if all goes well, attendees at next April’s NAB convention will see something developed even further.

Tags: 3d, Dial M for Murder, ISee3D, Jimmie Songer, Jon Chu, single-camera 3D, single-lens 3D, Step Up 3D

Can You Fix It in Post?

Can you fix it in post?

The simple answer is: Yes.

Consider Avatar.  Not only was an entire world created and populated in computers, but even a human actor’s legs were atrophied in post.  So, yes, anything can be fixed in post — given enough time and money.  In the worst case, artists would simply “paint” photorealistic images, pixel by pixel and frame by frame.

It wasn’t always so, especially in electronic imagery.  Before “paint” systems, post was extremely limited.  Cuts, dissolves, wipes, and keys were possible, but, in the days of analog recorders, even those often degraded images.  ”We’ll fix it in post” became a laughter-inducing cliché.

Today, not even counting “painting,” there are many real-time processes that can replace production activities.  Rather than having an image specialist controlling camera parameters during shooting, the raw signals from the sensors can be recorded, with a post-production colorist dealing with them.  Instead of optical filters in front of or behind the lens, post-production filters can achieve much the same effects.  Instead of worrying about large, stable camera mounts or optical image stabilizers, producers can turn to post-production image stabilization.

And then there’s 3D.

The images above are taken from the brochure for Sony’s MPE200 stereoscopic image processor.  They show some of the post-shooting corrections the system can accomplish.  At top left there is correction of inter-camera image center as a lens zooms, at top right correction of inter-camera rotation, and, at bottom, from left to right, correction of interaxial spacing, inter-camera elevation, and even inter-camera distance from the scene.

Let’s start with the interaxial-spacing adjustment.  It can move homologous points in the two eye views closer together or farther apart.  Unfortunately, that’s not the only difference between the two camera views.

The image at top above is what a single camera might see when shooting an edge of a cube or building.  Below it are the views of separated eyes.  The left eye (or left camera) sees more of the left side of the object; the right sees more of the right side.  Depending on the exact positioning, shooting distance, and object, one camera might even see things that the other doesn’t.  There’s no way that Sony’s MPE200 — or anyone else’s post-production processor — can know how to put things into the picture that weren’t there in the original.

Now consider some of the other corrections, like that of inter-camera rotation.  An HDTV frame is a 16:9 rectangle.  If one camera’s rectangle is rotated with respect to another’s, as shown above in the blue and red rectangles, the only way to get them to line up is to trim the content of each, as shown in the green rectangle.  That changes the original framing.

It’s not just a 3D problem.  With a stable mount or optical image stabilization, what the shooter sees is (not counting overscan or intentional changes in post) what the viewer sees.  With post-production image stabilization, it can be very different.

Have a look at the second (Mounts — the problem), third (Mounts — fixed in post), and fourth (Mounts — not exactly fixed) files available on this download page: http://schubincafe.com/blog/2010/06/things-you-can-or-can’t-fix-in-post-video-acquisition/.  They were provided by Aseem Agarwala of Adobe Systems, and they demonstrate the tremendous power of post-production image stabilization.

The first clip is an example of a horribly unstable image, shot with a handheld camera.  The second clip shows the post-processed result — so smooth that it appears to have been shot by an experienced crew with a camera mounted on a dolly on track.

The third clip, however, shows the original and the stabilized versions together.  There’s no question that the image has been marvelously stabilized, but the framing is so different that the second story of the building in the background disappears completely in the corrected version.

It’s not just framing.  If there’s any process that can be perfectly duplicated in post, it’s the adjustment of the color parameters of the signal produced by a camera’s image sensors.  As long as all of the information is recorded, it makes no difference from a technical standpoint whether the adjustments are made at the camera or in a colorist’s suite.  Unfortunately, there are standpoints other than technical.

The two pictures shown above are taken from the book Goodman’s Guide to the Panasonic Varicam by Robert Goodman, AMGMedia Publishers, 2004, http://www.goodmansguide.com/theseries.html (and here’s a link to Goodman’s own site: http://www.histories.com/hjemmeside.html).  The upper picture has the master gamma set to .35; in the lower picture, it’s .75.

Neither is necessarily better, and neither is necessarily “right.”  They are simply different.

Above is another pair of images from the same book.  The upper one has dynamic level set to 500; the lower is at 200.  Notice that all of the detail of the collar is easily seen in the 500.  On the other hand, the face seems desaturated.  Again, neither is necessarily good or right.  But there are major differences between these four pictures (there are even more image pairs in the book, demonstrating other parameters).

If the adjustments were made in production, the director might have liked some characteristics of the image (say, the collar detail) but not others (say, the desaturated face) and changed things to compensate (different lighting, makeup, or clothing, for example).  In post, the video parameters can be changed at will, but the lighting, makeup, and clothing remain the same, unless, of course, pixel-by-pixel and frame-by-frame an artist (or, more likely, a team of artists) repaints the images as they might have been captured in the first place.

If you have enough money and time, you can do anything in post.  For the rest of us, it’s a good idea to try to achieve desired looks in production.

Tags: 3d, Adobe, binocular disparity, camera alignment, color, filter, fix it in post, gamma, image stabilization, optical, post, post production

3DTV Today: One Step Back

The Society of Motion-Picture and Television Engineers (SMPTE) just completed an International Conference on Stereoscopic 3D for Media and Entertainment, what it calls “the only scientific gathering focused exclusively on 3D.”  Like many SMPTE conferences, it gazed into the future, with one presentation introducing “the need for additional image processing to get pixel-level geometry matching,” another discussing “Spatial Phase Imaging technology” that “can be made to work with any existing sensor-optics combination, making it amenable to a plethora of applications,” and yet another was titled “What Is Holographic Television, and Will It Ever Be in My Living Room?”

By the accounts of those who attended the event, it was jam-packed with terrific information.  This post, however, is not about the future of 3DTV but about its present.  And it won’t even delve into issues of 3D vision.

The image above is a portion of a television image of the face of Sir James Wilson Vincent Savile, better known as the entertainer Jimmy Savile.  It is clearly a black-&-white image, and it’s from this fascinating web site about experiments to allow color to be recovered from black-&-white video recordings: http://colour-recovery.wikispaces.com/Richard+Russell’s+experiments

The reason the color can be recovered is the same reason the image looks so bad.  When “compatible” color was created, it added a supposedly invisible color subcarrier to the video signal.  But it wasn’t invisible.  People who had been watching pristine images on their black-&-white TVs suddenly got extra dot and line patterns in their pictures.  Those with color TVs got less detail.

When sound was added to movies, something similar happened.  At the left is the four-units-wide by three-units-high (4:3 or 1.33:1 aspect ratio) frame of silent movies.  When a soundtrack was added, it impinged on the frame, giving the picture shape a much squarer 1.16:1 aspect ratio.  The so-called Academy aperture shrunk the picture (losing resolution) to return to a somewhat wider shape (1.375:1).

Now, in this age of digital (no more dot pattern) high-definition (widescreen with no resolution loss) TV, 3DTV sets are being sold.  They can deliver an added sensation of depth (assuming all goes well).  Do they cause anything to be lost as a result?

The picture above is a portion of one eye’s view of a 3D pair of photos shot by Pete Fasciano during a presentation he taught on July 21 about stereoscopic 3D.  I’ve trimmed it to an HDTV-like 16:9 aspect ratio.  Had it been a TV show shot in HDTV, it might have looked like the above.

Someday, we might have a form of 3DTV that will deliver left-eye and right-eye images separately, with full spatial and temporal resolution.  Today, what we have is the HDMI 1.4a standard.  It calls for the above for 1080i HDTV 3D.  The left- and right-eye views are placed side by side and squeezed into the HDTV frame.  Instead of 1080i HDTV’s 1920 pixels across, there are just 960 for each eye’s view.

For today’s 3DTVs that use active-shutter glasses, that’s the only resolution loss.  But some 3DTVs use passive glasses, with different eye views on different scanning lines.  If that’s the case, the side-by-side configuration drops the horizontal resolution from 1920 to 960, and the passive-glasses system drops the vertical from 1080 to 540.

HDMI 1.4a calls for the above for 720p HDTV 3D.  In this case the left-eye image is placed above the right-eye image.  Of course, that drops the vertical resolution from 720 lines to 360.  Now let’s add closed captioning.

Above is a simulated HDTV image with a closed caption.  Were this post about stereoscopic 3D and human vision, I might have pointed out that the caption is in the screen plane, so there will be a visual conflict between it and anything it is occluding that is set to come forward from the screen plane.  But this post is not about that.

Above is the same closed caption, generated, perhaps, by a cable, satellite, or telephone-company set-top box.  The caption appears where the box, not knowing about 3D, thinks it should appear.

Finally, when the captioned 1080i HDTV 3D signal from the set-top box enters the 3DTV, this is the result.  The caption is split in two, is elongated, and appears only half the time.  I’ve illustrated that last by making the caption appear transparent, but it might be a little worse than that.  The portion on the right will appear when the left-eye shutter of active glasses is open; the portion on the left will appear when the right-eye shutter is open.

From a technology standpoint, it’s relatively easy to design a system in which none of these problems will exist.  The presenters and attendees of the SMPTE International Conference on Stereoscopic 3D for Media and Entertainment might be doing that right now.  Unfortunately, 3DTVs are being sold today.

Tags: 3d, 3dtv, Academy aperture, closed caption, color recovery, HDMI 1.4a, Pete Faciano, resolution loss, set-top box, soundtrack

The Elephant in the Room: 3D at NAB 2010

As I roamed the exhibits at the NAB show this month, I kept wondering what other year it seemed most like.  And I was not alone.

There were plenty of important issues covered at the show, from citizen journalism to internet-connected TV.  And then there was the elephant in the room.

It would be a lie to say that 3D technologies could be found at every booth on the show floor.  But it was probably the case that there was 3D in at least every aisle.  There was so much 3D that it tended to diminish all other news.

In acquisition technology, for example, LED lighting was near ubiquitous, with focusable instruments, such as the Litepanels Sola, sometimes painfully bright.  Panasonic and Sony both showed models of future inexpensive video cameras with large-format imagers, and Aaton joined the range of those offering “digital magazines” for film cameras.  In small formats, GoPro’s Hero is a complete HD camcorder weighing just three ounces.

In storage technology, Cache-A, For-A, IBM, and Sony all showed in new offerings that tape is not dead.  Meanwhile, iVDR removable-hard-drive storage could be seen in several new products, and Canon introduced new camcorders based on Compact Flash cards.

Cinedeck looks like a viewfinder but includes built-in storage and editing capability. NextoDI’s NVS 2525 can copy either P2 or SxS cards.

In processing, Dan Carew’s Indie 2.0 blog said of Blackmagic Design’s DaVinci Resolve 7.0, “this best-in-class color correction software was formerly US$250,000 (for software and hardware) and is now available in a Mac software only verions for US$995.” http://indie2zero.com/2010/04/16/what-i-liked-and-saw-at-nab-2010/ Immersive Media’s 11-camera spherical views can now be stitched and streamed live.  NewTek’s TriCaster TCXD850 can deal with 22 inputs and virtual sets.  And, though you might not yet be able to figure out why you’d want this capability, Snell’s Kahuna 360 production switcher can deal with up to 16 shows at once.

In wireless distribution, there was VµbIQ’s 60 GHz uncompressed transmitter on a chip and Streambox’s Avenir for bonding up to four cellular modems to create a 20 Mbps channel.  In wired, there was Pleora’s EtherCast palm-sized bidirectional ASI-IP gateways.  And, in technologies that could be applied to either, there were Fraunhofer’s codec with a latency of just one macroblock line and a Harris-LG/Zenith proposal for expanding ATSC mobile transmission to full-channel use.

In presentation, there was a reference picture monitor from Dolby (seen in almost its final form at the HPA Tech Retreat).  Several booths had OLED monitors, from 7-inch at Sony to 15-inch at TVLogic.  Wohler’s Presto router has an LCD video display on each button.  And Ostendo’s CDM43 is a curved monitor with a 30:9 aspect ratio.

That barely scratches the surface of the non-3D news from NAB.  And then there was 3D.

Even All-Mobile Video’s Epic 3D production truck, parked in Sony’s exhibit, wore 3D glasses.  But it was the glasses on visitors to the truck that proved more instructive.

Sony provided RealD circularly polarized glasses to visitors for looking at everything from relatively small monitors to a giant outdoor-type LED display.  As soon as those visitors entered the control room of AMV’s Epic 3D truck and donned their glasses, however, they saw ghosting — crosstalk between the two eye views.  AMV staff were prepared for the shocked looks.  ”Sit down,” they said.  ”There’s a narrow vertical angle, and you have to be head-on to the monitors.”  Sure enough, that solved the problem — at least for those who could sit.

Another potential 3D problem was mentioned in the two-day 3D Digital Cinema Summit before the show opened.  If 3D is shot for a small screen and blown up to cinema size, it can cause eye divergence.  3ality’s camera rigs indicate when this might happen, but it happened anyway on at least one cinema-sized screen at NAB, leading to some audience queasiness.

Buzz Hays of the Sony 3D Technology Center says making 3D is easy, but making good 3D is hard.  There was a lot of 3D at NAB, including both easy and hard, good and bad.

It was hard to count the number of side-by-side and beam-splitter dual-camera rigs at the show, but, in addition to those, there were integrated (one-piece) 3D cameras and camcorders, in various stages of readiness, from 17 different brands, both on and off the show floor.  It seems that all of them were said to be “the first.”

Much could be learned about 3D at the two-day Digital Cinema Summit before the show opened.  It began with Sony’s Pete Lude showing that an ordinary 2D picture can seem 3D when viewed with just one eye, leading a later speaker (me) to quip that watching with an eye patch, therefore, is an inexpensive way to get 3DTV.

3ality’s Steve Schklair followed Lude with an on-screen, live demonstration-tutorial on the effects of different 3D rig settings: height, rotation, lens interaxial, convergence, etc.  He was followed by directors, stereographers, and trainers of 3D-convergence operators, among others.

Although 3D would seem to require more equipment (two cameras and lenses plus a stereo rig at each location) and more personnel (a convergence operator per camera in addition to a stereographer), there is seemingly one saving grace.  According to Schklair and others, 3D can get away with fewer cameras and less cutting than 2D.

The same thing was said of HD, however, in its early days.  Sure enough, when I worked on one show in 1989, we used just four HD cameras feeding the HD truck and twice as many non-HD cameras feeding the non-HD truck.  In the early days, it was common practice to do separate HD and SD productions.  Today, of course, one HD production feeds all, and it typically uses as many cameras and as rapid cutting as an SD show.

Atop a tower of Fujinon’s NAB booth, Pace showed something that recognizes the current economics of 3D.  With virtually no 3DTV audience, it’s hard to justify separate 3D productions, but, with such major players as ESPN, DirecTV, Discovery, and Sky involved in 3D, the elephant cannot be ignored, either.  So the Pace Shadow system places a 3D rig atop the long lens of a typical 2D sports camera.  Furthermore, it interconnects the controls (in a variety of selectable ways) so that the operator of the 2D camera need not be concerned about shooting 3D: one camera position, one operator, different 2D and 3D outputs.

Screen Subtitling came up with similarly clever solutions to the problem of 3D graphics.  Unless text is closer to the viewer (in 3D depth) than the portion of the image that it is obscuring, it can be uncomfortable to read.

Traditionally, subtitles are at the bottom of a screen, where 3D objects are closest to the viewer.  Raise the graphics to the top, and they might work in the screen plane.

Then there’s the issue of putting the graphics on the screen.  With left- and right-eye views, it might seem that two keying systems are required.  But with much 3D being distributed in a side-by-side format, a single keyer can place 3D graphics directly into the side-by-side feed.

There was much more 3D at the show, in every field of video technology (and, perhaps even audio).  In acquisition, for example, aside from integrated cameras, 3D mounts, and even individual cameras designed specifically for 3D (like Sony’s HDC-P1), there were also 3D lens adaptors, precision-matched lenses, precision lens controls, and even relay optics intended to allow wider cameras to be placed closer together, as in this picture shot by Eric Cheng of WetPixel.com: http://wetpixel.com/i.php/full/2010-nab-show-report-las-vegas/

At the other end of the 3D chain, there were both plasma and LCD autostereoscopic (no-glasses) displays using both lenticular and parallax-barrier technology, small OLED displays with active-shutter glasses and giant LED screens with passive circularly polarized glasses.  There were LCD and plasma screens (up to 152-inch at Panasonic) and DLP rear-projectors using active-shutter glasses, and both LCD and laser projection using passive polarized glasses.

There were dual-panel displays with beam splitters, and displays intended to be viewed through long strips of fixed polarized materials (to accommodate all viewers’ heights).  There were many anaglyph displays in the three-different primary-and-complement color combinations.  There were 3D viewfinders using glasses and others with displays for each eye.

Japan’s Burton showed a laser-plasma display that creates 3D images in mid-air.  Normally, they’ve viewed through laser-protection goggles, as in the image at the right at the top of this post.  But as a safety measure, they showed them instead inside an amber tube at NAB.

In storage, it seems that everyone who had anything that could record images had a version that could do so in 3D.  Even Convergent Design’s tiny Nano was available in a 3D version.  The Abekas Mira is an eight-channel digital production server — or it’s a four-channel 3D digital production server.  Want an uncompressed 3D field recorder?  Keisoku Giken’s UDR-D100 was just one such product at the show.

In processing, just about every form of editing and processing had a 3D version.  Monogram showed a touch-screen 3D “truck-in-a-box” production system.  Belgium’s Imec research lab even showed licensable technology for stereoscopic virtual cameras.

There was a range of equipment and services for converting 2D to 3D either in real time or not, automatically and with human assistance.  And there was a large range of processing equipment designed to fix 3D problems, such as camera rotation and height variation.

Sony’s MPE200 is one such device, with a U.S. list price of $38,000.  The MPES3D01/01 software to run it, however, is another $22,500.  With the least-expensive 3D camera at the show (Minoru 3D) retailing for under $60 at amazon.com, it might be said that 3D is cheap, but good 3D costs.

There was 3D test equipment from many manufacturers.  There was high-speed 3D (Antelope/Vision Research). There was 3D coax (Belden 1694D, complete with anaglyph color coding).  Ryerson University is doing eye-tracking research on what viewers look at in 3D and whether it’s different from HD and 4K.

So why was I wondering what year it was?  At NAB shows there have been many technologies shown that never went anywhere.  We still await voice-recognition production switchers, for example, and also voice-recognition captioning.  But those have generally been shown by only one company or a small number of exhibitors.

Digital video effects were among the fastest technologies to penetrate the industry.  First shown at NAB in 1973, they were commonly seen in homes by the end of the decade.

Then there was HDTV.  Its penetration after NAB introduction took much longer, even if dated only from 1989, when an entire exhibition hall was devoted to the subject (there were many earlier NAB displays).  Estimates vary, but U.S. household penetration of HDTV 21 years later seems to be in the vicinity of half.

At least HDTV did eventually penetrate U.S. households.  Visitors to NAB conventions in the early 1980s could see aisle after aisle of exhibits claiming compatibility with one or both competing standards for teletext.  One standard was being broadcast on CBS and NBC; the other on TBS.  There were professional and consumer equipment manufacturers and services offering support.  Based on the quantity and diversity of promotion at NAB, it was hard to imagine that teletext would not take off in the U.S.

So, will 3DTV emulate digital effects, HDTV, U.S. teletext, or none of the above?  Time will tell.

Tags: 3ality, 3d, 3dtv, Aaton, Abekas, All-Mobile Video, Burton, Buzz Hays, Cache-A, Canon, Cinedeck, Convergent Design, Digital Cinema Summit, For-A, Fujinon, GoPro Hero, immersive media, iVDR, Keisoku Giken, Litepanels Sola, Minoru 3D, Monogram, NAB 2010, NewTek, NextoDI, Ostendo, Pace, Panasonic, Pete Lude, Screen Subtitling, Snell, Sony, Steve Schklair, VµbIQ

How Old Is Your Stereographer?

Speaking at the Sports Video Group Chairman’s Forum in Las Vegas Saturday night, Professor Martin Banks of the Visual Space Perception Laboratory at the University of California – Berkeley raised an interesting issue regarding 3D comfort.  Stereographers (directors of 3D cinematography and videography) are responsible for, among other things, the visual comfort of the audience.  One factor in that comfort is vergence-accommodation conflict, the difference between the focal distance to the screen and the “distance” to which the eyes are pointing.

As people age, unfortunately, they become less able to focus at different distances, a normally occurring condition called “presbyopia.”  And that means that a stereographer with presbyopia can’t properly judge vergence-accommodation conflict.

Tags: 3d, Martin Banks, presbyopia, Sports Video Group, stereographer

3D Brings Science to Showbiz

The Woods Hole Oceanographic Institution is respected worldwide as the largest non-profit ocean research, engineering, and education organization.  Now, through its Advanced Imaging and Visualization Laboratory, it’s also involved in for-hire 3D production and post, offering complete 3D rigs that weigh as little as four pounds as well as systems that will operate from 14,000 feet below sea level to outer space.

There’s more here: http://www.prlog.org/10611247-woods-hole-imaging-systems-group-to-launch-3d-hd-unit-at-2010-nab.html

Tags: 3d, Woods Hole Oceanographic

3D Is Wonderful, Except….

Richard Sandomir of The New York Times was among the press who attended yesterday’s demonstration of what 3D Masters coverage might look like, and he wrote a glowing review of it that appeared in today’s paper: http://www.nytimes.com/2010/04/01/sports/golf/01threedee.html

It begins, “If the test footage shot recently at the Augusta National Golf Club is an authentic gauge, the Masters in 3-D will look terrific.”  And, until the very end of the article, it seemed only to get better still.

At the very end, however, Sandomir acknowledged some “flaws,” one that “blurred” the picture, another that was “visually jarring,” and a third that “was dizzying.”  The review ends with a quote:

“‘We’re figuring out what works and what doesn’t,’ said Mark Francisco, a Comcast Fellow working on 3-D.”

Tags: 3d, 3dtv, Comcast, Masters

The Other Three Dimensions of 3DTV

3DTV suggests that the three Ds are the dimensions of height, width, and depth.  But there are three other linear dimensions that might be worth considering: pupillary distance, viewing distance, and screen size.

Here’s a generic diagram of binocular vision, looking down.  The observer’s eyes are at either end of the short base of the triangle.  The distance between the centers of the pupils of those eyes is the pupillary distance (PD, or interpupillary distance or, sometimes, interocular distance), and the distance from the object being looked at to the eyes is the object distance (OD).  Each eye’s lens focuses (or tries to focus) on the object at the point of the triangle, a process called accommodation.  The two eyes also point to that object, a process called vergence or convergence.  And each eye sees a slightly different view, a process called stereopsis or disparity.

The diagram at the left might be called the plain geometry of 3D.  In typical 3D shooting, two cameras replace the eyes, and the PD is replaced by a distance between lens centers, which might be the same, larger, or smaller, depending on lens magnification and the desires of the stereographer.

If the lenses are separated more than is called for in normal vision, the result is something called hyperstereo, a sensation of viewing the scene through the eyes of a giant.  Distant objects that normally wouldn’t provide much of a sensation of stereopsis do, but everything seems to be closer to the observer.  Hypostereo is the opposite: The lenses are closer than called for in normal vision, objects lose their stereopsis sensation at closer distances, and the overall scene depth seems greater.

That’s shooting.  Now consider viewing.

Start with the PD.  It’s normally considered a fixed number, and, for almost any adult human, it is.  That is to say it is fixed for that one particular adult human.  It varies from person to person based on age, sex, ethnicity, location, and other factors.  That’s why your optician needs to measure your PD when you get a new pair of glasses, to ensure that the optical centers are where they’re supposed to be.

When I looked up “Interpupillary Distance” on Wikipedia recently, I got one range of PD figures and references; when I looked up “Pupillary Distance,” I got a somewhat different set.  A 2004 paper called “Variation and extrema of human interpupillary distance” recommends that the range to be considered for adults be 45 to 80 mm.  For children down to age five, the author recommends reducing the bottom end to 40 mm (and notes a 15-year-old female with a 43-mm PD).  Children younger than age 2 have even smaller PDs: http://www.cl.cam.ac.uk/~nad10/pubs/EI5291A-05.pdf For the record, in September I was measured to have a PD of 68 mm.

The variation in PD poses a problem for a stereographer.  To provide the sort of stereopsis and convergence I get in real life, the only PD to be considered should be 68 mm.  But, if something were to be shot and presented that way and that 15-year-old female with the 43-mm PD were to view it, then, for objects at an infinite distance, for which my eyes should point straight ahead, with no convergence at all, the 15-year-old female’s eyes would diverge, an unnatural condition.

Suppose, then, that the stereographer chooses the low end of the PD range, something good enough for even five-year-olds, 40 mm.  Then, when I view something at an infinite distance, instead of having my eyes point straight ahead, they’ll converge.

The diagram at the right shows that situation (but at an exaggerated scale).  Instead of an isosceles triangle, this time the geometric figure is an isosceles trapezoid (assuming I’m pointed directly at the screen, which is a different 3D issue).  In this case, the complete base is the viewer’s PD, VD is the viewing distance, and SID is the screen infinity distance, how far apart the two eye views are for the stereographer’s selected-observer PD.

At the left, I’ve extended the sides of the trapezoid to the original triangle to show where convergence-muscle feedback puts “infinity.”  Parallel lines are never supposed to meet, but the trapezoid sides aren’t parallel.

If you work out the math for an SID of 40 mm, my PD of 68 mm, and a viewing distance in a movie theater of 50 feet from the screen, the result is that the feedback that my convergence muscles send to my brain when looking at something that’s supposed to be at an infinite distance is that infinity is around 121 feet.

That might seem awfully close for a number you’re not supposed to be able to count to, but, in the grand scheme of vision, it’s not.  Here’s a medical web site dealing with vision issues.  Notice that the top of the page defines “infinity,” in medical visual terms, as simply greater than 20 feet: http://telemedicine.orbis.org/bins/volume_page.asp?cid=1-3-5-62

Different depth cues have different strengths at different distances.  Convergence and accommodation offer strong depth cues up close, but they become relatively insignificant as viewing distances approach that medical definition of infinity, greater than 20 feet.

Viewing distance is another of those under-considered dimensions of 3DTV.  In the example I gave above, I chose a viewing distance of 50 feet, not unusual for a cinema auditorium.  But 3DTV is viewed at closer distances.  If I were to change my viewing distance to the television-viewing Lechner Distance of nine feet, my convergence-based sensation of infinity drops to less than 22 feet — still beyond that medical definition, though I’d now be well within the range where convergence counts, leading to a stimulus conflict between stereopsis and convergence.  But there’s still that third dimension, screen size.

To this point, I haven’t mentioned the screen size, because it hasn’t mattered.  I’ve simply stated that the five-year-olds and I were watching whatever screen size the stereographer intended.  The left- and right-eye views for something at infinite distance are separated on the screen by the 40-mm bottom of the PD range.  But suppose the five-year-olds and I go to see a movie and then later bring home a 3D DVD or Blu-ray disk of the same content.

How large is the largest screen stereographers should consider?  Is it 100 feet?  If there’s a 40-mm SID on the screen, all of the numbers above still hold.  But, if the same material, unmodified, gets put on the consumer playback medium, the ratio between the largest intended screen and the home screen becomes important.

If the largest intended screen is 100 feet and the home screen is 32 inches, then 40 mm on the giant screen becomes just over 1 mm on the home screen.  If I were to sit nine feet away, my convergence-based sensation of something at infinity would put it at about nine feet away, the same as my viewing distance; it would hardly be behind the screen at all.  And it’s not just objects at “infinity” that matter.

At right is yet another diagram.  This time, suppose that there is a negative parallax on the screen matching the viewer’s PD.  Negative parallax indicates that the right-eye view is to the left of the left-eye view.  It’s easy to see from the diagram that the object appears to come out of the screen by half of the viewing distance.  But what is that distance?

Suppose what’s coming out of the screen is an arm, from shoulder to hand.  If the viewer is sitting four feet from a TV screen, the arm is a reasonable two feet long.  If the viewer is sitting 50 feet away from a cinema screen with the same negative parallax, the same arm becomes 25 feet long.

Perhaps other visual scaling factors come into play in such situations.  After all, when we see a close up on a 2D movie screen, we don’t suddenly think the character is a gigantic monster.  But the multiple dimensions of 3DTV seem to complicate matters.

Some have proposed an alternative.  In one paper, it has been called “Just Enough Reality” (http://www.cs.cmu.edu/afs/cs/user/mws/ftp/papers/t3dvt99-final.pdf).  In another, it’s called “Kinder Gentler Stereo” (http://www.ri.cmu.edu/pub_files/pub1/siegel_mel_1999_1/siegel_mel_1999_1.pdf).

The technical term is microstereopsis.  The “bad” news about it is that it doesn’t exactly duplicate the top triangle in this post.  But, thanks to variation in human PD as well as varying viewing distances and screen sizes, it’s unlikely that any real-world 3DTV system will match that triangle.

The good news is that microstereopsis doesn’t necessarily require two lenses per 3D position.  Some of the 3D that Sony showed at the 2009 Consumer Electronics Show was shot with a single-lens microstereopsis camera (the one that drew much interest at the CEATEC show last year).  But an older system (shown in the 1970s by Digital Optical Technology Systems) didn’t require even a camera with dual image sensors.

Microstereopsis might turn out to be a bad idea — or it might be a good one.  Despite the 82 years since the first 3DTV broadcast, it’s still a young field.

Tags: 3d, 3D TV, convergence, divergence, microstereopsi, pupillary distance, screen size, Sony 3D CEATEC, vergence, viewing distance

Panasonic 3D Camcorder: Show Us the Money

In the 3D-in-the-Home “supersession” at next week’s HPA Tech Retreat, one presentation is titled “Are You Nuts?”  I thought of that at today’s Panasonic pre-NAB press conference.

Let me emphasize from the outset that I was not thinking about Panasonic in the “nuts” category.  As best I could tell, no one from the company lied, which is my highest praise at press conferences.  No one avoided questions.  There was legitimate news (such as an inexpensive P2-to-USB adaptor, two tiny HD switchers, and a small HD pan-tilt-zoom camera optimized for IP networks).  I also think Panasonic builds good equipment.

No, the people I thought were nuts were some potential customers for something the company sort of unveiled at the recent Consumer Electronics Show (CES), an integrated (one-piece) 3D camcorder (shown above) to be delivered this fall at a list price of $21,000.  Panasonic said it had received thousands of inquiries about the product, some seeking to buy it sight unseen.

It was those blind-faith customers that I think are nuts.  Here’s why (and also why I said Panasonic only “sort of” unveiled the product at CES):

The camcorder has twin zoom lenses.  What is their widest angle?  Their tightest?  Panasonic representatives at the meeting didn’t avoid the question; they said it hadn’t been determined yet.

The camcorder will be capable of some amount of stereoscopic convergence.  How much?  Again, it has not yet been decided.  Also undecided, for this one-person, compact camcorder, is whether or not there will be any mechanism to tie convergence to focus.

One Panasonic representative did point out that the spacing of the lens centers is smaller than that of an adult human’s pupils and will not be getting bigger.  Based on a rough measurement I made, it appears to be about 57 mm.  That puts an outer limit on the maximum diameter of the lenses, which, coupled with the fact that the system uses 1/4-inch-format image sensors, means it will not be the most sensitive camcorder on the market.

When a journalist at the press conference inquired about using the camcorder for cinema content, a Panasonic representative emphasized that it had those 1/4-inch-format image sensors.  He got high points from me for that answer.

So what is the intended market?  At $21,000, it seems priced too high for most consumers.  At CES DXG showed a pocket-sized $400 3D camcorder (shown here to the left) with a 3D viewfinder (something Panasonic’s AG-3DA1 lacks), albeit non-HD and with much smaller lens-center spacing.

In the professional, HD realm, 3D-One offers four 3D camcorder models, all with nominal adult-vision lens spacing, 3D viewfinders, larger image sensors, and specified lenses and convergence.  Their CP-20 is shown below.  I wrote about them here in September: http://schubincafe.com/blog/2009/09/walkin-in-a-camera-wonderland/

At the press conference, Panasonic indicated receiving inquiries ranging from dental to military applications, including sports.  But a 57-mm lens-center spacing doesn’t lend itself to long-distance 3D shooting in a sports venue.

So, who is really interested in buying what Panasonic says will be a made-to-order product?  At the press conference, the company announced a way to find out.  Starting today, they will accept orders for this device of unknown optical capabilities, but each order is to be accompanied by a non-refundable $1000 deposit.

Panasonic hopes to learn much from this first-generation product.  Maybe we all will.

Tags: 1/4-inch format, 3d, 3d camcorder, 3D-One, AG-3DA1, AW-HS50, BT-3DL2550, convergence, DXG, image sensor, NAB, Panasonic