Advanced Virtual Reality Display Devices

Kenneth E. Harker
kharker@cs.utexas.edu
Department of the Computer Sciences
Taylor Hall TAY 2.124
Austin, TX 78712-1188

Virtual Retina Display

• Projects electronic information on the eye without the use of a intermediary screen of any kind.

• Scans an electronically encoded beam of light through the pupil to the retina, stimulating the receptors on the back of the eye.

• The beams of light are really very low power laser beams.

• Very good at producing reds and greens but blues are not very good yet.

• Can be developed into head mounted or handheld devices, including ``see-through'' devices.

VRD Component Systems

• Drive Electronics
  Processes signal from an image source, i.e. computer, controls the mixture and intensity of color and controls the positions of individual pixels.

• Light Sources
  Uses very low power laser beams to create and convey a single pixel at a time through the pupil to the retina. Red, green, and blue are separate sources that are modulated and then merged before being projected into the eye.

• Scanners
  Horizontal and vertical scanners ``paint'' an image onto an eye by scanning across the eye in a raster pattern.

• Optics
  Refractive and reflective optics are used to focus the beams and to enable ``see-through'' devices to be built.

Virtual Retina Devices

• Resolution
  VRDs can have resolution at least as high as similarly-sized LCD technology. Resolution in principle is limited only by diffraction and optical abberations.

• Contrast Ratio and Luminance
  Both contrast ratio and luminance are easier to control as there is no intermediary screen to work with. In principle, VRDs can be adjusted to work well in any ambient lighting conditions.

• Color Range
  Because the red, green, and blue lasers used emit highly saturated, pure color, VRDs in principle should have a much wider gamut of available color than either LCD or CRT displays. In practice, VRDs seem to look poor reproducing color mixes and blue.

• Applications
  The most obvious application is wearable computer augmented reality systems. In practice, the VRDs of today are much too bulky to use.

VR Responsive Workbench

• First developed by Wolfgang Krueger at the GMD Visualization and Media Systems Design Group.

• Projects video from the computer onto a mirror, which reflects the image onto the back of a large screen forming the workbench. The screen is a 4:3 aspect ratio, identical to the video monitor on the computer.

• The first workbenches oriented the screen horizontally. Subsequent implementations of the idea have tilted the screen as much as thirty degrees.

VR Responsive Workbench

• Use of LCD shutter glasses and a stereo projection mode can create a convincing three-dimensional effect.

• Most workbenches employe the use of tracked glove or wand systems for navigation.

• Some implementations use head tracking to further refine the stereoscopic effects.

Two-User VRRWBs

• In a two-user Virtual Reality Responsive Workbench (VRRWB) system, both users are head-tracked using something like the Polhemus 3Space magnetic field tracking system.

• The system must correctly determine the two users' positions from the tracking hardware and compute four images - one for each eye of both users.

• A hardware solution is employed to correctly correlate the stereo pairs with the correct timing on the shutter glasses. Images are interleaved in an L1 L2 R1 R2 manner.

VR Responsive Workbench

• Some VRRWBs, such as the Mirage system developed for the U.S. Army, use front projection, with a mirror hanging overhead.

• Some systems, such as the VRRWB at Stanford, allow two-handed manipulation of the scene using pinch gloves.

• Stanford's VRRWB has also been used to help develop ideas for 3D scene manipulation toolsets, in the form of 3D button menus.

Teleport

• Teleport is a project of GMD National Research Center for Information Technology (Germany) designed to overcome some of the traditional, perceived ``limitations'' of video conferencing.

• A single wall of a real room is used for the display.

• The video image of the remote participant is ``blended'' into a synthetic 3D scene that is intended to appear as an extension of the real room.

Teleport

• Requires two SGI Onyx Reality Engine 2 machines at each end.

• One machine renders the synthetic room and integrates the video from the remote site, while the other runs a parallel delta-keying algorithm to separate the local participant from the original background and send that video to the remote site.

• The users are tracked so their relative positions in the rooms can be updated at the remote site.

Head Mounted Displays

• Typically provide lower resolution output than CRTs: 720 x 240 for example.

• Field of view approximately 80 degrees horizontal by 60 degrees vertical.

• Fully immersive for one user. The user is unable to view outside world.

• The need to take the HMD on and off makes it unsuitable for many applications.

VisionDome

• Designed by Alternate Realities Corporation.

• A fully immersive, portable, multi-user, single projection environment.

• Resembles a telescope observatory, with a cylindrical section probably 2.5 meters in height topped by a hemisphere 4, 5, or 7 meters in diameter.

• The interior of the room itself is the projection plane, although not all of it will be illuminated at once.

VisionDome Hardware

• Projector
  The projector is a single lens light valve device. It can project a full 360 x 180 degree (an entire hemisphere) area of view. Normally, the projector sits at the center of curvature of the dome, and may be tilted to allow the image to fill the users' front field of view. Of course, being front projected, anything that wanders into the hemisphere will obscure the image.

• Screens
  The screens are designed to be seemless. Because the screens are in a hemisphere, the perspective math is different than for traditional flat displays, and the center-to-edge ``correct view parallax'' is much lower in the VisionDome than in a CAVE.

Light Valves Projectors

• Basically, a light valve is a projection system in which the red, green, and blue light components have been modulated and optically combined into a single light beam that is projected through a single lens.

• A light valve can use a ``fish-eye'' lens to achieve wide angles of projection and yet retain image focus.

• Virtual Retina Displays, in a sense, are light valves.

• Light Valves can change their aspect ratio independently from the software generating the image.

Grating Light Valves

• A technology from Silicon Light Machines, developed at Stanford University.

• Grating Light Valves use special pixels made of small aluminum ribbons and optical filters to produce very high contrast projections. The ribbons act as mirrors or diffractors and can be individually addressed.

• Color shades can be created in an almost continuous spectrum, as the ribbons in the pixels can be switched very fast, and the duty cycle, attack, and decay times can be tailored in an analog manner to get the exact color desired.

Digital Micromirror Devices

• Developed by Texas Instruments.

• The Digital Micromirror Device functions as a light switch. It contains as many as a half-million individual microscopic mirrors. Each mirror corresponds to a pixel and each pixel can be switched on or off at very high rates of speed.

• A ``color wheel'' is spun such that red, green, and blue light are in turn injected into the micromirror array, and in each step, the mirrors are switched according to the amount of that particular light that is needed on each pixel.

• The reflective surface of the DMD forms the image to be projected, which is simply fed through a projective lens.

Personal Monitor

• A product designed by Albatech, Inc.

• A small device fits onto the front of a pair of eyeglasses. It contains an LCD projector, and projects an image onto two small mirrors near the bottom and in front of each eyeglass lens.

• The two mirrors are oriented such that a image appears as if it were a 26'' monitor, located two meters away.

• Obstructs a portion of the view, but keeps the projected image in the field of view at all times.

• Envisioned as an ideal tool for surgeons and dentists.

CAVE

• Developed at the University of Illinois at Chicago.

• A multi-person, room-sized high-resolution, 3D video and audio environment.

• Graphics are rear projected in stereo onto multiple walls, and possibly down- or front-projected onto the floor or ceiling. Users wear stereo glasses to correctly view the stereo images.

• One user can be head-tracked for more accurate updates of the stereo and perspective projections.

• Additional controllers such as tracked wands can be used to allow for navigation through the virtual space.

CAVE Implementation

• Projectors are typically 1024x 768 resolution at 96kHz or better. (Un)fortunately, such projectors are rarely sharp enough to see the individual pixels, which are more than 1/8'' in diameter on the projection screens.

• Head tracking is typically done using devices such as Ascension Flock of Birds or Polhemus 3Space trackers, which employ a ``transmitter'' that generates a magnetic field in the area, and several ``receivers'' that detect the magnetic field.

CAVE Implementation

• Synchronization of the video screens is not difficult if each video ``pipe'' originates on the same SGI Onyx, for example. This is often done in IRIS Performer using ``arenas,'' or shared frame buffer memory.

• Distributed CAVEs can be built so long as the latency of the shared data (tracking, wand, navigation, time) is kept to a minimum. Some implementations have used Systran's Scramnet reflective shared memory to accomplish this.

Other CAVEs

• The original CAVE system consisted of two walls and a floor.

• The Naval Research Lab Advanced Information Technology Division has a MechDyne Surround Screen Virtual Reality three-wall plus floor display called the Graphical Room for Orientation, Training, and Tactical Observation (GROTTO) using essentially the same hardware.

• A research group at Iowa State University is developing a four-wall, floor, and ceiling display they call the C2 inside an engineering building that is currently under construction.

6DOF Tracked Devices

• Ascension Flock of Birds and Polhemus 3Space devices use magnetic fields generated by ``transmitters'' and attach ``receivers'' to the body of the user.

• Logitech created a tracking device that used ultrasonic microphones to pick up on an ultrasonic wave source placed in front of the display device. This is a very directional system, and, incidentally, the basis of the Nintendo PowerGlove system.

• Proposals have been made to develop tracking devices based on Direct Sequence Spread Spectrum (DSSS) radio transmitter/receiver pairs operating in the ISM bands at 925 MHz and 2.3 GHz.

SCRAMNet

• SCRAMNet from Systran stands for Shared Common RAM Network.

• It is a replicated shared-memory communication system designed for use in real-time computer systems. SCRAMNet is optimized for the high-speed transfer of small packets of data when low-latency transmission is required.

• Memory is replicated at each node in the network and data is transfered by very high speed fiber optic links. Data is transfered at almost the same speed as memory-to-CPU transfers.

Software Toolkits

• Most developers still write their own navigation and interface toolkits.

• Unlike 2D window managers, there is very little standardization for 3D iconization or interface tools.

• There is at least one effort to replicate standard 2D tools in floating ``windows.''

CAVE Library

• Designed at the University of Illinois at Chicago for use with the CAVE Automatic Virtual Environment, sold by VRCO.

• Based on IRIS Performer for Silicon Graphics Onyx hardware.

• Can be used for non-CAVE systems. Any virtual environment with multiple display screens can be designed with this library by registering the appropriate transformation with each wall. Up to eight screens are supported.

• Not entirely general: some routine refer specifically to SCRAMNet and are not generalized for other reflective distributed shared memory systems, for example.

NRL Dragon

• Very large system originally designed for the 1997 Hunter Warrior Advanced Warfighting Experiment as a battlefield visualization system.

• All device drivers were written in-house as an extensive library called vrLib.

• Two separate processes, the general entity manager (GEM) and the rendering engine communicate via shared memory through several lists.

• GEM can store typical DIS entities or scientific visualization volume data.

NRL Dragon

• Dragon can drive CRT monitors, head mounted displays, the Virtual Reality Responsive Workbench, and the Graphical Room for Orientation, Training, and Tactical Observation (GROTTO.) Output of all devices can be in mono or stereo mode.

• Interface devices include keyboard, mouse, speech recognition, pinch gloves, tracked ``wands,'' and tracked head positioning. vrLib generalizes all input devices as 6DOF or TEXT virtual devices.

• In practice, gloves proved to be undesirable, and tracked wands were most effective.

NRL Dragon Interfaces

• In all modes, simple text strings at the bottom center of the screen indicate recent changes, such as navigation mode.

• In battlefield visualization mode, a virtual laser pointer is drawn from the eye position to the point of intersection with the terrain.

• Entities can be selected by navigating the virtual laser pointer onto an entity. When selected, a wireframe hemisphere will surround the entity. Entities can be moved, and when released, will gently fall onto the surface of the terrain.

NRL Dragon Interfaces

• Pop-up tables can be displayed that give detailed information on individual entities.

• In ``digital ink'' mode, streams of virtual ``ink balls'' follow the laser pointer movement. Each ink ball's center point lies on the terrain surface.

NRL Dragon Interfaces

• Three modes for navigation.

• In pan/zoom mode, moving the tracked wand left/right or away/towards pans the viewpoint left/right/forwards/backwards, as if looking down at a map on a table. Moving the wand up/down will zoom the viewpoint in/out.

• In viewpoint orientation mode, twisting the wand and changing its orientation in yaw, pitch, and roll will reorient the viewing direction, but leave the viewpoint in place.

• In Digital Ink mode, most behavior is similar to pan/zoom mode, but the user will lay down a trail of virtual ink balls wherever the virtual laser pointer travels.

NPS Bamboo

• Designed as a general interface toolkit for the design of real-time, networked virtual environments.

• Has the ability to dynamically load modules into the executable's memory space. Modules can attach and detach themselves for the executable's event loop.

• Virtualizes devices and separates the idea of the interface from the rest of the modules. As modules are loaded, they can attach or remove features from the GUI.

• Much work has been done on the infrastructure, but there is little in the way of available modules to work with.

MRObjects

• MR stands for Minimal Reality Objects and is a project of the University of Alberta Computer Graphics Research Group.

• An object oriented toolkit for the design of virtual environments. In many respects similar, but different, to IRIS Performer.

• Extensive support for input devices, especially those that behave as joysticks. Supports several tracker devices.

• Supports only mono display devices so far.

• Source code is not available.

Virtual User Interface

• Developed by the Iowa Center for Emerging Manufacturing Technology at Iowa State University for use with their C2.

• Presents standard two-dimensional window manager style windows and buttons in a three-dimensional ``look and feel.''

• May not be available for use on platforms other than the C2.

SANDBOX

• SANDBOX is Scientists Accessing Necessary Data Based On eXperimentation.

• A 1994 collaborative project between researchers at the University of Illinois at Chicago and Wayne State University.

• Uses the CAVE to interface data collected in a scientific database to recreate experiments virtually. Does not address scientific visualization issues.

• Has its own pallettes of interface tools. Uses the floor as the ``work area'' and the walls for 2D data visualization.

PolyShop

• Designed by the Institute for Simulation and Training at the University of Central Florida for the U.S. Army Simulation, Training, and Instrumentation Command (STRICOM.)

• A GUI for a head-mounted display that is aimed for 3D CAD applications. The user sits at a desk, which is replicated in VR, and provides the user with a set of three-dimensional virtual menus and controls on the desktop. Most standard 2D widgets are presented in 3D.

Sense8 World Toolkit

• WorldToolKit is a portable software development system for building high-performance, real-time, integrated 3D applications for scientific and commercial use.

• It is an object-oriented library written in C with more than 1000 high-level functions for configuring, interacting with, and controlling real-time simulations.

• Functions are included for device instancing, display setup, collision detection, loading object geometry from file, dynamic geometry creation, specifying object behavior, and controlling rendering.

Sense8 World Toolkit

• The library does not provide a standard notion of 3D widgets or 3D interaction toolsets.

• The library code supports most major brands of VR input devices, position trackers, and stereo display devices.

Conclusion

• Devices: VRD, VRRWB, VisionDome, CAVE, Personal Monitor.

• Software: CAVE Library, NRL Dragon, NPS Bamboo, MRObjects, VUI, SANDBOX, Polyshop, World Toolkit.