Notice: Wiley Online Library will be unavailable on Saturday 30th July 2016 from 08:00-11:00 BST / 03:00-06:00 EST / 15:00-18:00 SGT for essential maintenance. Apologies for the inconvenience.
 NASA's Mars Exploration Rover (MER) Mission will place a total of 20 cameras (10 per rover) onto the surface of Mars in early 2004. Fourteen of the 20 cameras are designated as engineering cameras and will support the operation of the vehicles on the Martian surface. Images returned from the engineering cameras will also be of significant importance to the scientific community for investigative studies of rock and soil morphology. The Navigation cameras (Navcams, two per rover) are a mast-mounted stereo pair each with a 45° square field of view (FOV) and an angular resolution of 0.82 milliradians per pixel (mrad/pixel). The Hazard Avoidance cameras (Hazcams, four per rover) are a body-mounted, front- and rear-facing set of stereo pairs, each with a 124° square FOV and an angular resolution of 2.1 mrad/pixel. The Descent camera (one per rover), mounted to the lander, has a 45° square FOV and will return images with spatial resolutions of ∼4 m/pixel. All of the engineering cameras utilize broadband visible filters and 1024 × 1024 pixel detectors.
If you can't find a tool you're looking for, please click the link at the top of the page to "Go to old article view". Alternatively, view our Knowledge Base articles for additional help. Your feedback is important to us, so please let us know if you have comments or ideas for improvement.
 The NASA Mars Exploration Rover (MER) mission will land a pair of rovers on the surface of Mars in 2004. The rovers are designed to drive up to 600 m across the Martian surface over the 90-Martian solar day (sol) mission. The MER mission represents a significant advance in our ability to explore Mars. Technological advances in electronics, detectors, and packaging have significantly reduced the mass and power usage of remote sensing instruments relative to previous Mars-landed missions. The MER cameras weigh <300 grams each and use <3 W of power. These characteristics make it possible for the rover to carry an unprecedented nine cameras per rover and one on each lander.
 The operation of a surface rover is an image intensive process. The free-roaming nature of the rovers requires the daily acquisition and downlink of stereo image data in order to operate the vehicle. These images are rich in detail and can be directly applied to investigative studies of surface morphology, rock and soil distribution, and general surface geology. One of the challenges for both the engineering and science teams will be to analyze the new image data quickly (in a few hours), select targets based on scientific merit, assess the traverse options, and command the rover to drive to the designated target. After the rover has completed the traverse, the teams must then use additional image data to verify the post-traverse location of the vehicle relative to the commanded location.
 After every traverse, the views from the rover cameras will be different. These differences will range from slight changes in geometry for small moves to completely new scenes for longer traverses. Although this type of data has been seen in the small traverses of the Sojourner rover on Mars Pathfinder [The Rover Team, 1997b] around the Mars Pathfinder Lander in 1997, the MER rovers will travel significantly farther than Sojourner and will return multiple, 360° panoramic views of the local Martian surface from significantly different vantage points.
 In order to operate the vehicle as described above, the rover is equipped with a set of engineering cameras. This paper describes in brief the characteristics and capabilities of these cameras and their potential use as scientific instruments. Section 1 of the paper discusses the engineering camera functional objectives and requirements. Section 2 describes the camera hardware. Section 3 describes the capabilities of the MER imaging system in general, and section 4 discusses the operation and performance of the cameras. Section 5 describes the MER ground image processing capabilities. This paper complements the papers on the MER Pancam by Bell et al.  and the MER Microscopic Imager by Herkenhoff et al. .
1.1. Instrument Objectives
 The primary objective of the MER engineering cameras is to support the operation of the rover on the Martian surface, beginning just prior to touchdown on Mars. This includes the acquisition of images during various critical events, including images of the surface acquired during the final stages of the descent, images of the deployed lander shortly after roll stop, images of the deployed rover mobility system (and other rover deployables such as the solar panels), and images of the airbags and potential egress paths off of the lander. During the driving phase of the mission, forward and aft images of the local terrain will be used to autonomously detect and avoid hazards. At the end of a drive, panoramic images will be acquired and analyzed back on Earth to characterize the rover's position relative to the surrounding terrain. Finally, the operation of the Instrument Deployment Device (IDD) requires images of the IDD workspace in order to properly plan and execute the placement of the MER in situ instruments on specific targets.
1.2. Instrument Functional Requirements
 The design requirements for the MER engineering cameras are derived from a number of sources, including operational experience gained during the Mars Pathfinder mission, MER project design studies, and a set of MER project requirements for rover traverse distance and navigation accuracy. In addition, the engineering cameras support the following NASA Mars Program requirements to (1) demonstrate long-range traverse capabilities by mobile science platforms to validate long-lived, long-distance rover technologies and (2) demonstrate complex science operations through the simultaneous use of multiple science-focused mobile laboratories. These requirements, along with the high-level objectives described in the previous section, were used to derive the set of instrument functional requirements summarized in Table 1.
Table 1. Summary of Engineering Camera Functional Requirements
Cameras (number per rover)
Descent camera (1)
Acquire 3 images of the Martian surface between altitudes of 2000 m and 1200 m during descent; field of view: 45°, 3-m pixel spatial resolution; broadband, visible filter.
Provide terrain context for traverse planning and Pancam, Mini-TES pointing; 360° field of regard at <1 mrad/pixel angular resolution; stereo ranging out to 100 m (30 cm stereo baseline); broadband, visible filter.
Provide image data for the onboard detection of navigation hazards during a traverse; provide terrain context immediately forward and aft of the rover (in particular the area not viewable by the Navcams) for traverse planning; support Instrument Deployment Device (IDD) operations; support rover fine positioning near IDD targets; wide field of view (120°), 2 mrad/pixel angular resolution; stereo ranging immediately in front of the rover (10 cm stereo baseline) to an accuracy of ±5 mm; broadband, visible filter.
1.3. Science Requirements
 There were no formal requirements placed on the engineering cameras by the MER Athena science team. However, the MER engineering cameras will, as part of the surface navigation process, contribute indirectly to the achievement of the MER and Mars Program scientific objectives by placing the rover at sites of interest and providing contextual support for the pointing and placement of the science instruments. In addition, the images returned by the engineering cameras will directly contribute to the completion of a number of specific science objectives, as mentioned by Crisp et al. . In particular, Navcam and Hazcam images (stereo and monoscopic) will help to (1) investigate the landing sites at cm/pixel resolution, (2) characterize the local terrain and identify potential past water activity based on the morphology of rocks and soils, (3) measure the 360° spatial distribution of rocks and soils around the rover as it traverses across the surface, (4) determine the nature of local surface geologic processes from surface morphology, (5) calibrate and validate orbital remote sensing data, and (6) place the rock and soil types in a geological context.
2. Engineering Camera Descriptions
 In the manufacture of the 20 MER flight model (FM) camera assemblies (10 on each flight vehicle), 20 engineering model (EM) camera assemblies, and 4 Flight Spare (FS) camera assemblies, we chose to combine the design, assembly, and test of all of the MER cameras (both science and engineering) into a single program (see Figure 1). This approach not only provided a cost savings due to economies of scale, but it resulted in scientific quality detector/optical designs for the engineering cameras. In addition to hardware heritage, the engineering and science cameras also share the identical ground and flight command capabilities, data file formats, and ground processing systems.
2.1. Design Heritage
 The engineering cameras were designed and developed in parallel with the Pancam [Bell et al., 2003] and Microscopic Imager [Herkenhoff et al., 2003] science cameras. As a result, they share the same optical design heritage, electronics design, electrical interface, and data formats. The engineering camera performance properties (quantum efficiency, dark current, noise characteristics, etc.) are identical to those of the science cameras. The main difference between the MER engineering cameras and the MER science cameras is that the science cameras have undergone a more rigorous radiometric preflight calibration program. All of the MER cameras and image data are handled identically on board by the rover Flight Software (FSW) and on the Earth by the Ground Data System (GDS) software. This approach greatly simplified the design, implementation, test, calibration, and operation of the cameras. All of the MER cameras were developed at the Jet Propulsion Laboratory (JPL) in collaboration with the Athena science team.
 Each MER camera is composed of two mechanical housings: a detector head and an electronics box. The detector head contains an optical lens assembly and a charge-coupled device (CCD) detector. The electronics box contains the CCD driver electronics, a 12-bit analog-to-digital converter (ADC), and the camera/rover interface electronics. Hardware commands are received and dispatched using an Actel RT 1280 Field Programmable Gate Array (FPGA), which communicates to the rover electronics via a high-speed serial low-voltage differential signal (LVDS) interface. The camera electronics box also contains a heater resistor that warms up the electronics to above the minimum operating temperature of −55°C. Because the detector head is thermally isolated from the electronics box, the camera electronics can be heated without significantly warming the detector head, which helps to keep thermally induced CCD dark current to a minimum. The electronics boxes and detector head are connected through a flex cable, and the cameras are connected to the rover interface electronics using impedance-matched cables. The rover provides supply voltages of +7 V and −10 V to the cameras.
 Each MER camera electronics box is hardwired with a unique eight-bit electronic serial number that identifies the camera (see Table 2). The serial number is returned with every image acquired and has proven to be of value in the organization and reduction of the camera calibration data. The serial numbers will also be used to identify the cameras for ground processing during surface operations. As of this writing, over 135,000 calibration images have been acquired and stored at the JPL Multimission Image Processing Laboratory (MIPL).
Table 2. The Serial Numbers of the Mars Exploration Rover (MER) Cameras
MER 1 (Opportunity) Serial Number
MER 2 (Spirit) Serial Number
Front Left Hazcam
Front Right Hazcam
Rear Left Hazcam
Rear Right Hazcam
2.3. Charged Couple Device Detector and Electronics
 All of the MER cameras use identical 1024 × 2048 pixel Charge Coupled Device (CCD) detectors with 12-micron square pixels and a 100% optical fill factor. Mitel Corporation (now Zarlink Semiconductor) of Ottawa, Canada, manufactured the CCDs. The CCDs operate in frame-transfer mode, which divides the detector into two regions: a 1024 × 1024 pixel photosensitive imaging region where the image is recorded and a 1024 × 1024 shielded storage region in which the recorded image is shifted into and stored during detector readout [see Bell et al., 2003, figures]. The transfer of data from the imaging region to the storage region takes 5.1 ms, and the readout of data from the storage region takes 5.4 s. In addition to the imaging pixels the CCDs also include 32 nonimaging pixels in the serial readout registers. These “reference” pixels allow the monitoring of the CCD electronics offset, detector noise, and readout noise. The Navcam and Hazcam CCD pixels have full well capacities of ∼170,000 electrons (see Table 3 showing camera properties) and are digitized at 12 bits/pixel. The detector systems have gain values of ∼50 e-/digital number (DN), and the RMS read noise is ∼20 electrons at cold temperatures (−55°C), resulting in a system with ∼0.5 DN of root-mean-square (RMS) read noise.
Table 3. MER Camera Properties
CCD full well
CCD readout noise, 55°C
CCD Gain, 55°C
Frame transfer time
CCD readout time, full-frame mode
CCD readout time, 4 × 1 binned mode
12 × 12 microns
12 × 12 microns
12 × 12 microns
12 × 12 microns
12 × 12 microns
>200:1, all λs
0–335.5 s, in steps of 5.12 ms
0–335.5 s, in steps of 5.12 ms
0–335.5 s, in steps of 5.12 ms
0–335.5 s, in steps of 5.12 ms
0–335.5 s, in steps of 5.12 ms
Angular Resolution at the center of the field of view (FOV)
67 × 69 × 34 mm (electronics) 41 × 51 × 15 mm (detector head)
67 × 69 × 34 mm (electronics) 41 × 51 × 15 mm (detector head)
67 × 69 × 34 mm (electronics) 41 × 51 × 15 mm (detector head)
67 × 69 × 34 mm (electronics) 41 × 51 × 15 mm (detector head)
67 × 69 × 34 mm (electronics) 41 × 51 × 15 mm (detector head)
 The absolute CCD quantum efficiency (QE) of the MER CCDs has been measured from 400 to 1000 nm at four different operating temperatures ranging from −55°C to +5°C and is typical of a silicon CCD detector; the sensitivity peaks at 700 nm with a QE value of ∼43%. The QE curve has a relatively flat top and drops off to near zero QE at 400 nm in the blue and 1000 nm in the infrared [see Bell et al., 2003, QE figure]. The variation in QE as a function of temperature is <10% between −55°C and +5°C in the wavelength region between 400 and 900 nm. Given the relatively low readout noise and small dark current rates in the Martian operating environments, the signal-to-noise ratio (SNR) of the detector system is essentially Poisson limited. At 50% full well, and an operating temperature of −55°C, the SNR is >200 to 1.
 The detector has 3 hardware readout modes: full-frame, 4 × 1 binned, and windowed. The most commonly used readout mode for commanding the MER cameras will be the full-frame mode. The 4 × 1 binned and windowed readout modes shorten the readout time of the detector by ∼4 s and are primarily used for time critical activities such as entry, descent, and landing (EDL) and autonomous surface navigation. The three hardware readout modes are described in Table 4.
Table 4. The CCD Hardware Readout Modes of the MER Camerasa
Note that the reference frame for the row and column designations are given in the detector frame. The image frame origin varies from camera to camera and depends on the mounting orientation of the particular camera relative to the image frame.
The entire image region is read out at full resolution. Image size is 1056 columns × 1024 rows)
4 × 1 binned mode
The image region is charge summed in the column direction, in multiples of 4 rows. Image size is 1056 columns × 256 rows.
Windowed mode (row N to row M)
Only a specified number of (contiguous) rows are read out from the detector. Image size is 1056 columns by (M − N + 1) rows.
 The MER CCDs do not have antiblooming circuitry but instead rely on a “clocked antiblooming” readout technique [see Bell et al., 2003]. Although blooming is a not a problem for natural scenes, the effect is often noticeable when imaging regions of the rover hardware with high specular reflectance (e.g., shiny metal objects). The proper choice of autoexposure parameters helps avoid the blooming effect in this case.
2.4. Optical Characteristics
Section 2.4 describes in brief the optical characteristics of the MER engineering cameras. For more detailed information, see the work of Smith et al. .
2.4.1. Descent Camera
 The Descent camera is shown in Figures 2a and 2b. It is mounted on the lander radar bracket and points downward during lander descent (see Figure 2c). The Descent camera was added to the lander payload after much of the overall camera design effort had been completed and as a result shares the identical optical design as the Navcams, an f/12 optical system with a 45° × 45° field of view, a 60.7° diagonal FOV, and an angular resolution of 0.82 mrad/pixel at the image center. The Descent camera uses a broadband filter with a center at ∼750 nm and a Full Width at Half Maximum (FWHM) of ∼200 nm. Figure 3 shows the Descent camera spectral responsivity as a function of wavelength.
 The Navigation Cameras (Navcams, Figures 4a and 4b) are optically identical to the Descent camera: f/12 cameras with a 14.67 mm focal length. Each Navcam camera has a 45° × 45° field of view (60.7° diagonal), which is roughly equivalent to a 40 mm lens on a 35 mm camera. The angular resolution at the center of the field of view is 0.82 mrad/pixel. The depth of field of the Navcam camera ranges from 0.5 m to infinity, with best focus at 1.0 m. The Navcams use a combination of filters (Schott OG590, KG5, and an ND1.3) to create a red band-pass filter centered at ∼650 nm. Figure 5 shows the Navcam spectral responsivity as a function of wavelength. The nominal Navcam exposure time for a noontime image on the surface of Mars (tau = 0.5) is ∼0.25 s. This exposure time is 50 times the frame transfer time of 5.1 ms, which ensures that the image signal is significantly larger than the image smear acquired during the frame transfer.
 The Navcams are attached to a titanium bracket with a left/right stereo baseline of 20 cm (see Figure 6). The Navcam camera boresights are mechanically coaligned on the bracket to better than 0.15° (i.e., <4 Navcam pixels). The bracket, which also holds the Pancams, is mounted to the Pancam Mast Assembly (PMA) elevation assembly, which sits atop the PMA.
2.4.3. Pancam Mast Assembly (PMA)
 The PMA is a pan/tilt assembly capable of 370° of motion in the azimuth (pan) direction and 194° of motion in the elevation (tilt) direction (see Figure 7). This capability enables the precise targeting of the Navcams to points of interest, and also enables the acquisition of 360° Navcam Panoramas. The PMA is capable of pointing the Navcam to an accuracy of better than 0.10° (∼2 Navcam pixels) in both the azimuth and elevation directions. Movement of the PMA actuators (and thus the pointing of the Navcam camera boresights) is controlled directly via image command arguments as azimuth/elevation angles or three-dimensional Cartesian target points in a specified coordinate frame. When the cameras are pointed out toward the horizon, the boresights are 1.54 m above the surface, providing an unobstructed view out to 3.2 km for a featureless sphere of Martian radius. Pointing of the Pancam/Navcams to celestial bodies is done by the onboard Inertial Vector Propagation (IVP) system.
 The Hazard Avoidance Cameras (Hazcams) are shown in Figures 8a and 8b. The Hazcams are an f/15 optical system with a focal length of 5.58 mm. The Hazcam optics are f-theta fish-eye lenses with a 124° × 124° horizontal/vertical field of view and a 180° diagonal FOV. The angular resolution at the center of the image is 2.1 mrad/pixel. The Hazcams use the same combination of spectral filters as the Navcam (Schott OG590, KG5) along with an ND1.1 filter to create the same red band-pass filter as the Navcams (centered at ∼650 nm) and a similar absolute responsivity. See Figure 5 for a plot of the spectral responsivity of the Hazcams as a function of wavelength.
 The Hazcams are mounted to a titanium alignment bracket that provides a 10-cm stereo baseline. The Hazcam stereo boresights are mechanically aligned to better than 0.25°. The front and rear mounting brackets are attached directly to the outside of the rover Warm Electronics Box (WEB) in a fixed (nonpointable) configuration ∼0.5 m above the nominal surface, as shown in Figure 9. The pointing requirement of the Hazcams was specified to provide ∼15° of sky in the top portion of the Hazcam images. This requirement, along with the mechanical constraints in the Hazcam mounting areas on the rover resulted in a Front Hazcam optical boresight pointed at an angle of 45° below the horizon, and a Rear Hazcam optical boresight pointed at an angle of 35° below the horizon. This configuration allows for the viewing of ∼17° of sky in the Front Hazcam images and ∼15° of sky in the Rear Hazcam images (the upper 12° of the Rear Hazcam images are blocked by the rear solar panel on the rover). The top portions of Rear Hazcam images often contain reflections of ground objects in the shiny underside of the rear solar panel.
 All of the MER engineering cameras have been calibrated over the flight range of temperatures. The reduction and analysis of these data are in progress at the time of this writing and will be published in a MER project calibration report. The calibration report will include (for each camera) the measured geometric flat field response, the detector dark current as a function of temperature, the camera absolute responsivity, detector noise performance, detector gain, and a list of bad pixels for each camera (if applicable). The report will also contain a geometric camera model (described in section 3.1.16) for each of the MER cameras.
2.6. Comparison to Other Mars Landed Cameras
Table 5 compares the spatial resolution of the MER cameras with the Mars Pathfinder and Viking cameras. The Navcams have an angular resolution slightly below that of the high-resolution mode of Viking lander cameras and slightly above the Imager for Mars Pathfinder camera. The Hazcams have an angular resolution approximately equal to that of the low-resolution mode of the Viking lander cameras and slightly better than the Mars Pathfinder rover (Sojourner) cameras.
Table 5. Spatial Resolution of the MER Engineering Cameras Compared With Other Mars Landed Cameras
 The MER Imaging Services (IMG) flight software module handles all camera commands, including ground commands from Earth as well as onboard commands for sun finding and autonomous navigation. The IMG module is responsible for controlling the power state of the cameras, setting various hardware parameters, initiating image acquisition, and performing any onboard processing of the image data prior to downlink. There are 30 imaging commands available for the operation of the MER cameras, most of which are related to the management of camera hardware and software parameters. The acquisition of images is done through a single CAPTURE_IMAGE command. The CAPTURE_IMAGE command is a self-contained, 47-argument command that specifies the cameras to be operated, image acquisition parameters, and other onboard image processing options, including image compression. The CAPTURE_IMAGE command structure is conceptually similar to the command structure used for the Imager for Mars Pathfinder [Smith et al., 1997a, 1997b] with additional functionality.
 In addition to ground operators on Earth, there are a number of onboard users of the MER cameras. The autonomous navigation (NAV) software uses the cameras to detect hazards during a traverse, and the Surface Attitude Pointing and Positioning (SAPP) system uses the cameras to locate the sun and calculate the rover orientation. Both of these onboard modules can request an image at any time (typically during or immediately after a traverse) and have access to the same functionality as the ground commands, including the ability to send an image to the downlink system. If desired by the operations team, it is possible to downlink all (or a fraction) of the autonomously collected images through the setting of IMG parameters.
3.1.1. Command Buffering and Sequencing
 Up to two cameras can be powered simultaneously, one left camera and one right camera. When a camera command is received by the IMG module, it automatically powers on the specified camera(s) and acquires the requested image(s). If no new commands are sent to that camera within a user-programmable timeout, then the camera is powered off. Commands are nominally dispatched to the cameras in an “event-driven” fashion; that is, commands are automatically dispatched to the cameras when the previous command has completed. In some cases, multiple image requests (either from the ground or from onboard software) may be sent to the IMG task simultaneously. In these cases the commands are queued internally by IMG and processed in an order determined by a combination of dispatch time, user priority, and resource availability. Commands from the onboard autonomous systems generally receive higher priority than ground commands and are executed in front of any ground commands.
3.1.2. Hardware Parameters
 There are five commandable camera hardware parameters. These parameters (listed in Table 6) are stored in camera hardware memory registers of each camera. Three of the five hardware parameters are sent with every CAPTURE_IMAGE command (exposure time, windowing start row, and number of windowing rows). The remaining parameters are settable via separate commands and are not expected to change often during the mission.
Table 6. Commandable Hardware Parameters for the MER Cameras
Fast flush count
number of erase cycles performed before an image is acquired.
0–335.5 s, in steps of 5.12 ms
length of time in which the imaging area collects photons
Start row for hardware subframing
starting row location to be read out in hardware windowing mode.
Number of rows for hardware subframing
0 (all rows)
number of rows returned by the camera in hardware windowing mode.
video offset value of the ADC. value of 4095 corresponds to full ADC range.
3.1.3. Exposure Types
 There are four types of exposures available for camera operation: “none,” “manual,” “auto,” and “test.” An exposure type of none will result in no image acquisition and is used primarily for prepositioning the PMA prior to Navcam and Pancam imaging. A manual exposure will acquire an image with a user-specified exposure time, and an auto exposure will return an image acquired with an autocalculated exposure time based on scene content. The test exposure will return a fixed-pattern image whose DN values are all equal to the value in the video offset hardware register. The test exposure is not expected to be used during the surface phase of the mission.
 The MER autoexposure algorithm allows the acquisition of image data without a priori knowledge of the illumination intensity of the scene. The algorithm is similar to the autoexposure algorithm used for the Imager for Mars Pathfinder camera [Smith et al., 1997a] and in other planetary missions. The MER autoexposure algorithm is briefly described here. The autoexposure algorithm utilizes the histogram of an acquired image to calculate an exposure time for subsequent images. There are four commandable parameters to the algorithm: a target value for the maximum DN values in an image (the DN threshold), an allowable percentage of pixels that are permitted to exceed the DN threshold (pixel fraction), the number of image acquisition iterations allowed for the autoexposure, and an early termination percentage. The algorithm is illustrated in Figure 10. During the autoexposure iteration process, the exposure time for subsequent images (tnew) is found by multiplying the current exposure time (tmeasured) by the ratio of the desired DN threshold value (DNdesired) and the measured DN threshold value
where DNmeasured is found by counting the number of pixels on the right side of the histogram in Figure 10 until the number of pixels equals the commanded pixel fraction. The early termination percentage specifies the matching threshold between DNdesired and DNmeasured; if the two values are within the termination percentage of each other, the algorithm terminates early, and that image is accepted for downlink.
 The MER autoexposure algorithm makes several assumptions about the scene content, including the assumption that the histogram is approximately linear in the region around the DN threshold area. This approximation is sufficient for our purposes and has been tested in a variety of lighting environments. In cases where the intensity histogram of an image is bimodal (e.g., the image contains significant portions of both terrain and sky), the autoexposure parameters must be chosen correctly to ensure optimal exposure levels in the particular region of interest. The autoexposure algorithm is available for the production of any image data product (e.g., subframes, histograms, and row/column summations).
3.1.5. Image Data Product Options
 A full-frame, uncompressed image from a MER camera is 1024 × 1024 × 12 bits, or 1.5 megabytes in size. Because the onboard flash memory available for data storage is only 256 megabytes, of which ∼200 megabytes is available for instrument data storage, the maximum number of full-frame, uncompressed images that can be stored in memory is ∼130. This is equivalent to ∼2 weeks of downlinked image data at the nominal downlink rates of ∼100 Mbits/day. To work within these constraints, the MER imaging system provides the capability to produce additional, less volume-intensive image data products. Those data products are listed in Table 7.
Table 7. Types of Image Data Products Produced by the MER Imaging System
Data Product Type
raw raster-format, uncompressed image.
ICER or LOCO compression.
Uncompressed thumbnail image
small copy of source image. Raw, raster-format, uncompressed image
Compressed thumbnail image
small copy of source image, ICER or LOCO compressed.
Uncompressed reference pixels
raw raster-format, uncompressed reference pixel data (both “pre” and “post” in the same data product).
Compressed reference pixels
ICER or LOCO compressed reference pixel data
row sum data
column sum data
3.1.6. Exposure Time Tables
 The flight software keeps an onboard table of the most recently used exposure time values for each camera/filter combination and makes these values available for use by subsequent image commands. These exposure time tables are particularly useful when acquiring images of the same general scene in rapid succession (e.g., Hazcam imaging when driving, Navcam/Pancam panorama acquisition, or multispectral Pancam imaging), where the overall lighting level changes from image to image are relatively small. If desired the exposure time table values can be used as seed values in an autoexposure iteration. At the end of the autoexposure iteration the exposure time table is optionally updated with the most recently calculated exposure time for that image.
3.1.7. Exposure Timescale Factors
 The flight software also allows exposure times to be multiplied by a user-supplied floating point scale factor. This feature is particularly useful when the absolute exposure time is not known in advance, but the responsivitiy ratios (i.e., the scale factor) between camera/filter combinations are known. For example, if a Navcam image of the terrain in front of the rover is acquired using autoexposure, a front Hazcam image can be acquired using the previously used Navcam exposure time multiplied by the scale factor representing the ratio of the Navcam/Hazcam camera sensitivities. Similarly, if a multispectral Pancam series begins with an autoexposure using a particular spectral filter, the next image in the series has access (via the exposure time table) to the previously used value and can modify that value by multiplying it by the user-supplied scale factor. The use of the exposure time table and scale factors help to improve image acquisition speed.
3.1.8. Pixel 12 to 8-Bit Scaling
 All of the MER cameras produce 12-bit/pixel image data. If desired, this data can be scaled to 8 bits/pixel by the flight software prior to image compression. There are two methods used to perform the pixel scaling: bit shifting or lookup table (LUT) transformations. The bit-shifting method allows the user to specify which group of 8 bits are sent back in telemetry, as well as an option to perform autoshifting of the data to allow preservation of the highest DN value. The 12-to-8 bit LUT transformations perform scaling using a user-programmable onboard lookup table that maps the 12-bit DN value to a corresponding 8-bit DN value. Up to 5 tables can exist onboard simultaneously. These lookup tables will be optimized for particular types of scene content (e.g., shadowed areas, areas of high rock or soil densities, atmospheric images, etc.).
3.1.9. Pixel Summation
 In some cases, it may be desirable to merely return the row and column summations of an image rather than the image itself. The MER flight software is capable of summing the row and columns of an image and returning the results in a 1-D array of 32-bit integers whose length is equal to the image height (or width). Pixel summations can also be performed on image subframes. Row and column sums are particularly useful when the spatial content of an image has relatively low scene entropy and the emphasis is on radiometry. Row/column summations will be used in the acquisition and downlink of images of the Martian sky (e.g., angular intensity studies, cloud searches, etc.).
 The flight software has the ability to calculate and return the histogram of an image. The resulting data product is a 4096-bin array that contains the number of pixels whose intensity value equals the value of the bin index.
3.1.11. Image Downsampling
 All MER images can be spatially downsampled to a user-specified image size using a user-specified hardware or software downsampling technique. The software techniques include nearest neighbor computation of the mean, computation of the mean with outlier rejection, and median averaging. The hardware techniques include the 4 × 1 binning option described earlier. Image downsampling will be heavily used during the surface mission for both engineering and science purposes due to the significant reduction in downlink volume over a full-size image. The autonomous surface navigation system will rely on the 4 × 1 hardware binning capability to acquire images in rapid fashion.
3.1.12. Shutter Subtraction
 Because the MER cameras use an electronic shuttering technique, all raw camera images contain a shutter-induced transfer smear and a dark current readout ramp superimposed onto the image data. The flight software offers a commandable option to remove these effects from an image. When specifying this option, the software acquires a zero-second exposure image (i.e., an image that contains only the shutter effect) immediately after image acquisition and subtracts it from the image of interest. The resultant image is sent back in telemetry. The shutter subtraction capability also includes an automatic mode in which shutter subtraction is performed if the exposure time falls below a user-settable threshold value.
3.1.13. Bad Pixel Correction
 While the quality of the MER CCD detectors has been very high, a small number of the pixels produce anomalous intensity values. For this reason the flight software includes the capability to replace these bad pixels with values that are interpolated from surrounding pixel neighbors. The locations of these pixels and the cameras to which they apply are stored in an onboard table along with the correction option to be used in the bad pixel replacement process. The correction options are quite extensive and include the ability to replace a pixel with the mean value of various patterns of surrounding pixels (including simple nearest neighbors), varying-sized squares, diagonals, and cross-hatch patterns. The bad pixel replacement function also includes the option to replace corrupted readout columns of image data if necessary. Also available in the replacement process is a provision for replacing pixels on 4 × 1 hardware binned images as well as an option to replace the bad pixel with a constant value.
3.1.14. Flat Field Correction
 The MER flight software includes the capability to remove the low-frequency flat-field falloff that occurs near the edges of the camera images. This intensity falloff is removed by applying an angularly dependent, cubic correction function to the image data. The parameters of this analytic function are derived from flat-field data acquired during the subsystem camera calibration testing. The flat field correction parameters for each of the 10 cameras per rover are stored in an onboard, updateable table.
 When acquiring images, it is often the case where only a specific portion of the image is of interest. For these cases, the flight software allows the option to extract and downlink a subframed region of the image. This subframing option allows full resolution data to be downlinked at a lower data volume than a full image and is expected to be used extensively. The user specifies a choice of hardware and/or software subframing techniques.
3.1.16. Camera Models
 In order to correctly interpret the shapes, distances, and angular extent of objects in a camera image, the camera lens and focal plane must be modeled in a way that relates the 2-D position of a pixel in the image plane to a 3-D direction vector in an external, real-world coordinate system. This model is used to remove geometric distortion artifacts introduced by the optical system. The model is also used to project image pixels out into a 3-D coordinate system for ray tracing and stereo triangulation.
 The geometric models used for all of the MER cameras are descendents of a linear perspective projection model developed in the 1970s by Yakimovsky and Cunningham . The original models are commonly referred to by the acronym CAHV (center, axis, horizontal, vertical), where the letters of the acronym correspond to component names of the model. The CAHV model supports nonsquare pixels, nonorthogonal row, and column axes and allows a projection center anywhere in the image plane (to handle lenses not centered over the active area of the sensor). The model takes the form of four 3-D vectors whose construction leads to very efficient projection from 3-D to 2-D, requiring only three inner (dot) products, two scalar divisions, and one vector subtraction.
 Six of the cameras, the Pancams, Navcams, Microscopic Imager, and Descent camera, use a modification of the model by Gennery . This model adds a radially symmetric transformation prior to the linear model to describe radial distortion, both barrel and pincushion. The vector describing the symmetry axis for the radial distortion is independent of the vectors used in the linear model and accommodates lenses that are not mounted square with the image plane. This type of model is referred to as the CAHVOR (center, axis, horizontal, vertical, optical, radial) model. CAHVOR models have been generated and validated for the MER flight cameras and will be included in the camera calibration report.
 In contrast to the other cameras, the four Hazcams do not have optics that can be modeled adequately by perspective projection models, even with aggressive use of the radial-distortion terms of the earlier models. These cameras have modified fish-eye lenses, with 124° horizontal and vertical fields of view (180° diagonally). It was therefore necessary to create a new model for MER. To this end, Gennery generalized the prior models, creating a third family of models called the CAHVORE (center, axis, horizontal, vector, optical, radial, entrance). This generalized camera model adds a moving entrance pupil to the prior models, as well as another transformation preceding the distortion and linear steps to describe a class of generalized distortion models, two special cases of which are perspective projection and fish-eye. The Hazcams occupy a position in the model space that is neither perspective-projection nor fish-eye, but between the two. With careful surveying of target points, the MER calibration procedure has been able to produce CAHVORE models that produce less than a quarter of a pixel RMS error for a Hazcam 3-D to 2-D projection.
3.2. Image Compression
 To maximize the number of images acquired during the mission, virtually all image data will be compressed by the rover CPU (using either lossy or lossless compression) prior to placement into the telemetry stream. To perform this task the rovers will utilize a software implementation of the JPL-developed ICER wavelet-based image compressor [Kiely and Klimesh, 2003], capable of providing lossy and lossless compression. In cases where lossless compression is desired and speed is particularly important, compression will be performed (in software) by a modified version of the low-complexity (LOCO) lossless image compression algorithm [Klimesh et al., 2001; Weinberger et al., 1996]. The MER mission is utilizing state of the art image compression technology by flying compressors that deliver compression effectiveness comparable to that achieved by the JPEG-2000 image compression standard [Adams, 2001], but with lower computational complexity [Kiely and Klimesh, 2003].
 MER will make use of compression rates ranging from <0.25 bits/pixel (i.e., 48 to 1 compression on a 12-bit/pixel original image), to lossless compression, which typically delivers bit rates of 7–8 bits/pixel on 12-bit originals and 4–5 bits/pixel on 8-bit originals. The lower bit rates (<0.5 bits/pixel) will be used for certain wavelengths of multispectral image sets. The midrange image compression rates (∼1 bit/pixel) will be used for rover navigation and IDD operation, as well as scientific studies. Lossless image compression rates will be used for situations where maximum geometric and radiometric fidelity is required (e.g., radiometric calibration targets).
3.2.1. ICER Compressor
 ICER is a progressive image compressor that can provide lossless and lossy compression, and incorporates a sophisticated error containment scheme to limit the effects of data loss during transmission. Progressive compression means that as more compressed data are received, successively higher quality reconstructed images can be reproduced, as illustrated in Figure 11.
 ICER applies a wavelet transform to decompose the image into a user-controlled number of subbands, each a smaller version of the image but filtered to contain a limited range of spatial frequencies. ICER allows the selection of one of seven integer wavelet transforms. These wavelet transforms are invertible, thus image compression is lossless when all of the compressed subband data are encoded. By using a wavelet transform, ICER avoids the “blocking” artifacts that can occur when the discrete cosine transform (DCT) is used for decorrelation, as in the Joint Photographic Experts Group (JPEG) compressor used on the Mars Pathfinder mission [Smith et al., 1997a]. Figure 12 illustrates such artifacts.
 ICER compresses a simple binary representation of the wavelet-transformed image, achieving progressive compression by successively encoding groups of bits that have the same significance. During this encoding process, ICER maintains a statistical model of the image. This model, which relies on a technique known as context modeling, is used to estimate the probability that the next bit to be encoded is equal to zero. The probability estimates produced by the context modeler are used by an entropy coder to efficiently compress groups of bits.
 Image quality and the amount of compression are primarily controlled by two parameters: a byte quota, which controls the maximum number of compressed bytes produced, and a quality goal parameter that tells ICER to stop producing compressed bytes once a simple image quality criterion is met. ICER attempts to produce a compressed image that meets the quality level using as few compressed bytes as possible. ICER stops producing compressed bytes once the quality goal or byte quota is met, whichever comes first. If during the mission the primary concern becomes the bandwidth available to transmit the compressed image, one can set the quality goal to lossless and allow the byte quota to determine the amount of compression obtained. At the other extreme, when the only important consideration is a minimum acceptable image quality, one can provide a sufficiently large byte quota, and the amount of compression will be determined by the quality goal specified.
 To mitigate the impact of data losses that occur on the deep space channel between Mars and Earth, the MER image compressors incorporate error containment techniques. Without error containment, even a small loss of data can corrupt large portions of a compressed image. To achieve error containment, ICER automatically partitions the wavelet-transformed image data into a user-specified number of segments. Each segment can be decoded independently of the others so that the loss of data from one segment does not affect the ability to reconstruct another segment. These segments approximately correspond to rectangular regions of the original image but are defined in the transform domain. This approach has some advantages over the simpler alternative of partitioning the image directly and applying a wavelet decomposition separately to each segment (i.e., dividing the original image into smaller images that are compressed independently). With lossy compression under this simpler alternative the boundaries between segments would tend to be noticeable in the reconstructed image even when no compressed data are lost, as illustrated in Figure 13a. By segmenting the image in the transform domain we can virtually guarantee that such artifacts will not occur, as illustrated in Figure 13b.
 By applying the wavelet transform to the entire image at once we also achieve better decorrelation and reduce inefficiencies associated with edges of wavelet transforms, thus improving compression effectiveness. It is also easier to maintain a similar image quality in the different segments. A minor side effect is that the effect of data loss in one segment can appear to “bleed” slightly into adjacent segments in the reconstructed image, i.e., a few pixels near the borders of that segment may appear blurred. Because ICER is progressive, some error containment automatically occurs within segments as well: when data loss occurs, any previously received compressed data for the affected segment will still be decompressible and allow a lower fidelity reconstruction of that segment, as illustrated in Figure 14.
 Dividing an image into a large number of segments can confine the effects of data loss to a small area of the image, but it is generally less efficient to compress smaller image segments. By varying the number of segments a user can control this tradeoff between compression effectiveness and robustness to data loss. This functionality is provided to adapt to changes in the data loss rate of the downlink channel and is commanded as part of the CAPTURE_IMAGE command.
 ICER's error containment features are particularly important for MER surface operations, where the daily commanded activities rely heavily on the image data returned from the previous sol. Even with moderate data loss during transmission, images such as those shown in Figure 14 will still prove to be useful for traverse and IDD planning.
 MER will also make use of a modified version of the LOCO lossless image compressor. The LOCO software encodes pixels in raster-scan order by predicting each pixel value based on the values of previously encoded nearby pixels and losslessly encoding the difference. Although ICER can also perform lossless compression, the simple approach used by LOCO is several times faster, with competitive compression effectiveness (i.e., within a few percent in terms of compressed data volume). In lossless compression tests on 8-bit/pixel planetary images, both LOCO and ICER give compressed image sizes that are ∼20% smaller than those produced by the Rice compressor used for lossless image compression on Mars Pathfinder. LOCO uses a simple error containment scheme that, like ICER's, accepts a user-specified number of segments.
3.4. Image Data Product Headers
 The headers of the image data contain useful ancillary information related to the state of the camera and rover at the time of image acquisition. Although this approach increased the volume of nonimage data slightly, the increase in size was small relative to the overall size of an image (the ratio of the size of the image header to the size of a typical compressed image is <1%). In addition to the rover position (in the local site frame), rover orientation and rover Motion Counter (RMC) values (see section 4.6), the image headers also include instrument state data such as CCD and electronics temperatures, camera hardware modes, exposure time, image size, and the entire set of image command arguments. The joint angles of the IDD actuators and the rover mobility system are also included in the header.
4. Camera Operation and Performance
 The primary objectives of the MER engineering cameras are to support the operational phase of the MER mission. Sections 4.1–4.4 describe the use of these cameras for this purpose.
4.1. Descent Camera
 During the vehicle entry into the Martian atmosphere, the MER landing system may accumulate undesired horizontal velocities due to steady state atmospheric winds. Wind models show that the wind could cause the lander horizontal velocity to exceed the air bag design threshold of 24 m/s. To compensate for these undesired horizontal velocities, the vehicle is equipped with a Transverse Impulse Rocket Subsystem (TIRS), which provides a specified amount of thrust in the opposite direction of vehicle motion. One of the inputs to the autonomous rocket-firing algorithm comes from the Descent Image Motion Estimation Subsystem (DIMES), which gathers the results of a real-time image correlation of surface features contained in successive Descent camera images of the Martian surface to compute the horizontal velocity of the descending vehicle. If the images show a significant amount of horizontal vehicle velocity from one frame to the next, DIMES passes the computed horizontal velocity correction to the TIRS. TIRS uses this horizontal velocity measurement along with measurements of the attitude of the backshell to compute a TIRS rocket firing solution that reduces the lander horizontal velocity.
 Three Descent camera images are used for the DIMES correlation. The vehicle altitudes for these images will range from ∼2000 to 1200 m above the surface. To acquire the images rapidly, the images will be commanded with the minimum allowable exposure time of 5.1 ms and the 4 × 1 hardware binning option. The per-pixel resolution of these images will be ∼4 × 1 m. Figures 15a and 15b shows a Descent camera test image acquired during helicopter testing over the Mojave Desert in California during 2002. The DIMES test program acquired 90,000 Descent camera images over three Mars analog sites (Kelso Dunes, Pisgah Crater, and the Ivanpah dry lake bed).
 For each Descent camera image, DIMES also requires elements of the lander state at time of image exposure including the surface relative attitude, the horizontal velocity estimate, and the altitude. Using the lander state information, DIMES reprojects each image to the ground plane (using the geometric camera model described earlier) and then computes horizontal displacements between images by correlating the two scenes. Image correlation is applied to two locations (chosen based on scene entropy) in the first and second image and two locations in the second and third images. This produces four image-based horizontal velocity estimates that are compared for consistency to each other and the acceleration computed from differences of velocities from the Inertial Measurement Unit (IMU). The DIMES system uses these velocity consistency checks along with image correlation metrics to decide if the computed velocity is correct. If the velocity is determined to be correct, a velocity correction is sent to the EDL software module for inclusion in the TIRS firing solution. If the velocity is determined to be incorrect, DIMES reports that a velocity cannot be computed and TIRS proceeds without the input from DIMES. An overview of the DIMES algorithm is given in Figure 16.
 One of the primary objectives of the Navcam camera system is to acquire an “end of day” panorama of the local terrain after a rover traverse. This panorama will provide a rover traverse planning function similar to the Imager for Mars Pathfinder (IMP) End of Day (EOD) images of the Sojourner rover acquired during the Mars Pathfinder surface mission. The major difference for MER, however, is the fact that the origin of the MER image data moves with the rover, and as a result the image scene will change with every rover move. Images of the surrounding terrain (rocks, dunes, etc.) will be used to help calculate the position of the rover relative to its last location, point the Pancam and Mini-TES, and provide general site context. The typical MER EOD panorama will be ICER-compressed at a compression rate of ∼1 bit/pixel (lossy), which puts the total data volume for a 360° end of day panorama at ∼20 Mbits; this includes the image-to-image overlap (usually 10–20%) necessary to ensure full stereo coverage. Figure 17 shows an image from the MER 2 Navcam acquired during thermal testing.
 Panoramas are acquired by executing a series (or sequence) of individual CAPTURE_IMAGE commands. Each individual command in the sequence specifies the desired pointing parameters for each camera. The exposure times from previous images in the panorama can be referenced via an onboard exposure time table (as mentioned earlier). A typical 360° Navcam panorama consists of a sequence of 10 stereo pairs spaced apart by 36°. The image-to-image overlap ensures sufficient overlap in the derived stereo data, which is typically less than the full field of view of the camera.
 In addition to supporting rover fine positioning and IDD placement, images from the Hazcams will be acquired during rover drives. If desired, these traverse images can be downlinked at the original resolution or at a smaller, downsampled resolution. The nominal image size used by the onboard autonomous navigation system is a 128 × 128 pixel (obtained through the combination of 4 × 1 hardware binning and software binning), stereo pair image set. The proximity of the Hazcams to the Martian surface will result in a set of excellent close-up views of the fine-grain texture (see Figure 18).
4.4.1. Onboard Stereo Processing and Rover Navigation
 The Hazcams serve as the terrain-sensing component of the MER onboard predictive obstacle avoidance system. Stereo Hazcam pairs provide the inputs to a real-time geometric model of the terrain conditions in the forward and aft directions of the rover. This geometric model takes the form of a pie-shaped wedge emanating from the rover that is ∼115° wide and extends from 0.30 to 3 m away from the rover. The information in this model is used to avoid dangerous obstacles (mainly rocks higher than ∼0.25 m in height and slopes steeper than ∼20°) during a surface traverse. The NAV software module performs the stereo processing of Hazcam data onboard. Because the autonomous navigation of the rover is a critical operation, a number of validity checks are performed on the stereo data before being ingested into the navigation map. Internal consistency checks on the quality of the correlation, spatial analysis of the correlation maps, and thresholding are all performed on the stereo data before being added to the onboard navigation map. Shadowing has a small effect on the density of the correlation maps, but this effect is relatively unimportant because the majority of the rover driving will be done during the middle of the day (when shadows are minimized).
 The final result of onboard stereo processing is a 2.5-D terrain model of the surface nearby the rover (“2.5” dimensions refers to the fact that the terrain properties are viewed from only one vantage point and thus the full three dimensions of the terrain are not known). A complete 3-D model on board would be ideal but it is not possible to gather this information due to the fact that the cameras cannot image the backs or sides of the objects from a single viewpoint. Additionally, the uncertainty in the rover's onboard position estimate due to wheel slippage and errors in tilt knowledge prevent the accurate onboard merging of detailed 3-D data across multiple views. Onboard range data are typically computed at 128 × 128 pixel resolution, which is sufficient to resolve the locations of obstacles taller than the 0.25-m height of the clearance under the rover body at distances of 3 m.
 In the absence of ground truth, a useful metric for estimating the performance of a self-filtering stereo vision system is the density of data that remains in each XYZ image after discarding uncertain values. The density of an XYZ image is computed by dividing the valid range pixels by the total number of image pixels (less a six-pixel border around the image where stereo correlation is not performed). Table 8 compares the mean and median image densities computed by our stereo vision software for data collected from three rover platforms: the Athena Software Development Model (SDM) rover, the Rocky 7 research rover [Volpe, 1999], and the MER Surface System Testbed (SSTB-Lite) rover. The Athena SDM and Rocky 7 images were collected in JPL's outdoor Marsyard facility, and the SSTB-Lite images were collected in JPL's indoor In Situ Instrument Lab (ISIL) sandbox. The images have been processed using the identical stereo vision software at 128 × 128 pixel image resolution.
Table 8. XYZ Image Densities for the MER Hazcams, Compared to Previous Rover Navigation Systems Developed at JPL
Navigation Camera System
Camera Field of View, deg
Number of Image Pairs
Mean XYZ Image Density
Median XYZ Image Density
 The fish-eye camera models developed for MER result in very dense range images. Although the number of image pairs from the MER SSTB-Lite in Table 8 is limited, and the MER SSTB-Lite images were taken indoors rather than outdoors, the mean and median range density of MER SSTB-Lite imagery is higher than that of the other rovers. Rocky 7 has the lowest range image density because its lenses have a large field of view and were calibrated using a non-fish-eye camera model. The Athena SDM has better results than Rocky 7 because its field of view is smaller and is better approximated by the same non-fish-eye camera model. The MER densities are the highest, due to the more realistic lens modeling of the Hazcam fish-eye lens.
4.4.2. Autonomous Driving With the Hazcams
 The Hazcams are an important component of the autonomous navigation system. Operators specify a goal location from camera images in local site frame coordinates, a radial tolerance around the goal location, and a timeout value. The rover will move toward the goal in a series of short steps (nominally 35 cm), acquiring a Hazcam stereo pair at each step and evaluating several candidate arc turns and straight line motions at different angles. These Hazcam images are generally 128 × 128 pixels in size, and can be downlinked if desired. The onboard navigation algorithm selects the path that moves the rover closer to the goal while ensuring vehicle safety. If no safe path is available the run will be stopped. Paths are actually evaluated out to 3 m, but only small motions will be commanded. This cycle of image acquisition, planning, and move continues until one of several conditions is met: the goal is reached, the timeout is exceeded, an onboard sensor detects an unsafe condition (e.g., a motor stall), or none of the possible paths is considered safe. The key to autonomous driving is the automatic processing of the Hazcam terrain data during the traversability analysis described in section 4.1.1.
 The rover maintains the predicted traversability of the nearby terrain as a grid of cells nominally 20 × 20 cm extending 5 m around the rover (see Figure 19). This is not a complete 3-D world model, but instead each cell is assigned a goodness and certainty estimate that represents the evaluation of vehicle safety at that location. rover-sized planar patches are fit to the measured terrain, and the parameters of these fits are used to predict how well the rover could navigate through them using several filters [Goldberg et al., 2002]. Each filter generates the goodness and certainty values; the overall evaluation will be the minimum of the generated values. The step hazard detector compares the relative min and max elevation measured in each rover-sized patch against the operator-defined maximum traversable obstacle height, which is determined from the clearance under the rover body (i.e., <0.25 m). A roughness hazard detector compares the residual of the planar fit against some fraction of the maximum obstacle height. A border hazard detector marks those cells at the edge of known information, and the tilt hazard detector compares the surface normal of each planar patch against a preset limit. At the time of publication, the entire Hazcam stereo pair processing cycle (acquisition, stereo processing, traversability analysis, and path planning) required ∼65 s.
 Images acquired during an autonomous traverse will be downlinked and used by the surface navigation team on Earth to reconstruct the rover traverse path. Although these images are at an eighth the resolution of the full-size Hazcam images, the 128 × 128 pixel navigation images are expected to provide useful views of the surface directly in front of the rover. Rear Hazcam images of the rover tracks will provide information related to the soil mechanics of the terrain.
4.4.3. IDD Operation With the Front Hazcams
 In addition to serving as the primary obstacle detection sensor during rover driving, the front Hazcam stereo pair is used to provide targeting information to place each of the in situ instruments mounted to the end effector of the IDD. The IDD is a 5°-of-freedom robot arm that is used to control the 3-D position and orientation (azimuth and elevation) of the in situ instruments relative to rock and soil targets. The in situ instruments consist of the Alpha Particle X-ray Spectrometer (APXS), the Mossbauer (MB) spectrometer, the Rock Abrasion Tool (RAT), and the Microscopic Imager (MI). See Squyres et al.  for additional details concerning the IDD and the in situ instruments.
 A primary requirement for the IDD is the placement of the instruments to an accuracy of 10 mm in position and 10° in orientation. This accuracy requirement is driven both by the kinematic positioning capabilities of the IDD itself as well as the accuracy of the front Hazcam stereo ranging system. Preliminary results indicate that the errors associated with the stereo ranging system account for ∼75% of the overall accuracy requirement while the IDD kinematic positioning capabilities account for ∼25% of the overall accuracy requirement. During the System Thermal Test (STT) conducted on both rovers, the front Hazcams were utilized to determine targeting information for the placement of the in situ instruments on calibration and radiation targets located in the IDD workspace. An image of the IDD during placement activities during this test is shown in Figure 20. Further testing of the end-to-end positioning performance of the IDD using the front Hazcams as well as other rover-mounted cameras such as the Navcam will be carried out on the engineering model hardware at JPL.
4.5. Stereo Ranging
 The stereo ranging errors of the MER engineering cameras are an important component of the overall rover navigation error budget and are shown in Figure 21 as a function of distance from the camera. The Hazcams are capable of providing absolute range estimates to accuracies better than 50 cm for objects at a distance of 10 m. The Navcams, with higher angular resolutions and a larger stereo baseline, produce range errors that are five times smaller (10 cm) than the Hazcams at the same 10-m distance. The Navcams also have a higher vantage point than the Hazcams (1.5 m above the surface for the Navcams compared to 0.5 m above the surface for the Hazcams), which makes them useful for far-field target designation. The Pancam range errors are a factor of four smaller (due to the higher resolution and larger stereo baseline) than the Navcams and in certain cases may be used for far-field ranging during surface operations.
 The stereo ranging capabilities of the MER cameras have been tested and validated on all of the flight stereo cameras. During preflight testing on the flight Navcams, the calculated position of an externally surveyed target 22 m from the rover agrees with the survey data to within 0.38 m (1.7 %). The flight Hazcams were used to triangulate a point on a visual target in the IDD workspace (camera object distance of 0.65 m) for IDD instrument placement. The resulting 3-D position of the target agreed with the external surveying data to better than 1 mm. For a more distant target at 22 m from the rover, the calculated position of the target from the Hazcam images agreed with the externally surveyed position to within 1.56 m (7%).
4.6. Surface Coordinate Frames
 MER utilizes three major Cartesian coordinate frames to conduct surface operations: the rover frame, the local level frame, and the site frame. These frames are listed in Table 9 and are descendants of the coordinate frames used on Mars Pathfinder (MPF). MER extends the use of the MPF coordinate frames by introducing the notion of multiple instances of the Surface Fixed frame (MER site frame). Of the three MER surface frames, the site frame is the only coordinate frame whose origin and orientation is fixed to the Martian surface; the other two frames are attached to the rover. Upon landing (and before rover egress), the origin of the first instance of the site frame is coincident with the origin of the rover frame (Figure 22). When the rover egresses from the lander, the site frame stays fixed in space and the local level and rover frame origins move with the rover. The position of the rover in the site frame is equivalent to the positional offset between the site and rover frame.
Table 9. Coordinate Frames Relevant for Image Processing and Rover Operations
Coordinate Frame Name
fixed to the intersection of PMA azimuth and camera elevation bar rotation axes
fixed to azimuth and elevation hardstops
fixed to rover body
fixed to rover body
fixed to rover body
fixed to Mars
fixed to Mars
fixed to Mars
 As the rover traverses across the surface, it updates its position and orientation (relative to the site frame origin) through the monitoring of onboard Inertial Measurement Unit (IMU) data and the rotation and orientation of the rover wheels. Over time the rover accumulates an error in its knowledge of position and orientation. Rover orientation knowledge is automatically updated onboard by measuring the position of the Sun at known times. Rover errors in position knowledge are corrected by acquiring camera images and comparing the locations of objects seen in those images with the locations of the same objects seen in images acquired at previous positions. These position updates are performed on the ground and uplinked to the rover as part of the daily surface operations process.
 To prevent the accumulated rover position knowledge error from propagating into the instrument data over the course of the surface mission, the site frame is systematically reset at strategic locations during the mission by the operations team. When the site frame is reset, the rover position is set to (0, 0, 0) m. Typically a new site frame will be declared just prior to the acquisition of a large image data set (e.g., a panorama). As with all new site frames, the origin initially coincides with the rover frame origin, and the orientation is aligned with the local level frame. This new site frame becomes the operational coordinate system for activities within that site. Figure 23 describes the notion of multiple site frames.
4.7. The Rover Motion Counter
 Each MER rover retains in memory a set of onboard counters that are incremented after specified actuator movements (e.g., wheel rotation, joint movement, etc.). These counters are grouped together and stored in the rover Motion Counter (RMC) onboard variable. The RMC is made up of five components, four of which are tied to specific actuator motions. The first component of the RMC is the site index and is incremented whenever a new site is declared and the site index increment command is sent. At the start of the mission the value of the site index is zero and is expected to increase to a value of >20 by the end of the mission. The four remaining values of the RMC are the drive index (which increments when the rover mobility actuators are in use), the IDD index (which increments during IDD actuator movement), the PMA index (which increments during PMA movement), and the high-gain antenna (HGA) index (which increments when the HGA is in use). When the rover is moving on the Martian surface, the RMC values will increase monotonically.
 The two most significant counters in the RMC are the site and drive index values. If the drive index is incremented, the IDD, PMA, and HGA counters are reset to zero. If the site index is incremented, all of the other RMC counters are set to zero (see Figure 24). The RMC will be placed in the headers of all of the camera and instrument data.
5. Ground Image Data Processing
5.1. Data Processing
 JPL's Multimission Image Processing Laboratory (MIPL) will perform the ground processing of the engineering camera image data during MER surface operations. The image processing software draws significant heritage from the software used on the Mars Pathfinder mission and is described by LaVoie et al. . Immediately upon receipt of the data from the telemetry stream, raw image data are written to Experiment Data Record (EDR) files. A subset of the EDRs are critical for the daily operation of the rover on the surface and will be processed further into Reduced Data Records (RDR) files. The RDR types include linearized images, disparity maps, XYZ images, and range images (see Table 10). Examples of operations critical EDRs include images of the IDD workspace during IDD operation, Hazcam images acquired during a traverse (used for path reconstruction), and Navcam panoramas acquired at the end of a traverse. Both the EDR and RDR files are conformant to the Planetary Data System (PDS) format and will be submitted to the PDS within 6 months of receipt on Earth.
Table 10. Types of Reduced Data Records (RDR) Images Produced by the MER Ground Data System (GDS)
Images that contain the row/column pixel values describing the relationship between the location of an object in the right image and its corresponding location in the left image of a stereo pair, stored as two arrays of floating point numbers. These images, along with the corresponding camera model, are used to triangulate the three-dimensional position of a pixel in Cartesian space.
Image that contains the triangulated 3-dimensional location of an image pixel in Cartesian space, stored as a 3-banded floating point image array.
Image that contains the camera-to-object distance of a pixel in an image, stored as a floating point number.
 Linearized images are generated using the CAHV, CAHVOR, and CAHVORE family of models described earlier. The linearization process removes nonlinear spatial distortions from an image by reprojecting the pixels into a linear image space. This reprojection step is also used to perform epipolar alignment on stereo image pairs, resulting in a pair of images whose rows are aligned with the epipolar lines between the two cameras (the epipolar lines of a stereo camera system are the lines formed by the intersection of the image planes of the two cameras with the plane defined by the optical center of each camera and a 3-D point in the real world). The creation of epipolar-aligned images allows the stereo correlation search to be constrained to a single row in each of the left/right stereo images, resulting in a significant simplification (and corresponding speed increase) of the stereo correlation process. Figure 25 shows a linearized Hazcam image.
5.1.2. Stereo Correlation and Disparity Maps
 To perform stereo triangulation on an object that appears in the left/right images of a stereo pair, the object location must be identified in both images. This matching process is performed using a software correlator that reads in the left and right images of the pair and performs a pixel-by-pixel spatial comparison of the scene as viewed by each camera. When an object is located in both the left and right images, the pixel location of the object in the second image is written to a disparity map image file (the left image is typically used as the base image by convention). In the case of epipolar-aligned images the rows are vertically aligned, and thus the pixel values in the vertical disparity maps contain the line numbers of the rows. For non-epipolar-aligned images, each pixel in the disparity map contains two values: the location of the matched pixel in both the left/right direction and the up/down direction. The majority of the stereo processing performed on MER will be done with linearized, epipolar-aligned RDRs.
5.1.3. XYZ Images and Range Maps
 The final step in the stereo triangulation process involves the calculation of the 3-D position of the pixels recorded in the disparity map. These pixels are projected into 3-D space using the camera models described earlier. The intersection point of these projected vectors is calculated by finding the point midway between the projected vectors at closest approach. The resultant positions are written to a three-band XYZ image file (the X, Y, and Z refer to the Cartesian coordinates of a 3-D point in space). The XYZ files contain the 3-D position of every correlated pixel in the base image and are used to generate higher level products such as terrain maps. Range maps are calculated from the XYZ images by computing the Cartesian distance of the XYZ values at each pixel. The resultant range maps contain the absolute distance between the object and the camera. Figure 26 shows an XYZ image generated from a stereo Hazcam pair, and Figure 27 shows the range map derived from the XYZ image.
 Because a single image from the Navcams or Pancams covers only a small fraction of the viewable terrain as seen from the PMA, it is often useful to assemble multiple images into a single-image mosaic that covers the entire 360° azimuthal field of regard. These mosaics are typically generated from specific sets of panoramic data acquired under similar lighting conditions. The mosaics that will be produced during MER surface operations are listed in Table 11. Of these mosaic types, the point perspective projection is most useful for the display of a small region of interest, the cylindrical projection is useful for a full 360° panorama, and the perspective cylindrical is used when it is necessary to view a full panorama in stereo. Polar and vertical projections are useful for displaying an “aerial” view of the terrain.
Table 11. Types of Mosaic Data to be Produced During MER Surface Operations
Each pixel represents a fixed angle in azimuth and elevation.
Each pixel is projected into a virtual, single “camera” with horizontal epipolar lines
Each column of the mosaic is generated using a separate virtual camera, as if the image was acquired from a line scanning system.
Elevation increases radially from the central nadir and azimuth is measured around the nadir.
Distance from the center of the mosaic is directly proportional to the elevation angle (not orthonormal).
 All of the mosaics are created using a ray-tracing process in which each input image is projected through a camera model onto a surface. The pixels on the surface are optionally projected back into a virtual camera to form the mosaic. Ideally, the measured topography of the actual terrain should be used as the surface on which to project the input images. The resulting mosaic would be free from parallax when viewed from any direction. In reality, the process is significantly simplified through the use of analytical surface models. The most common approximation is a flat plane at the average ground level. The flat plane approximation is sufficient for most mosaics, although if objects are significantly above the surface plane (e.g., large rocks, the rover solar panels, etc.) the parallax effect is noticeable. The resultant mosaic will often contain visual discontinuities (i.e., “seams”) between adjacent images of these “off-plane” objects while the terrain images are seamless.
 The generation of the multi-image mosaics includes an optional step that allows a human (or computer) to apply a correction to the pointing of the camera. This is useful for cases in which the mosaics contain noticeable seams, either due to camera pointing knowledge uncertainty (expected to be only a few pixels for the PMA cameras) or due to the parallax effect described earlier. Using this process, common points are identified in overlapping images and associated together through a process called tie pointing. These tie points are used to produce the new pointing angles that are used as inputs to the ray-tracing process. This process often yields seam-free mosaics for objects on the ground as well as objects significantly above the surface plane. In the future we expect the use of more sophisticated surface models to improve the fidelity of our mosaics. Figure 28 shows a 10-image Navcam polar projection mosaic, Figure 29 shows a vertical projection mosaic, and Figure 30 shows a cylindrical projection mosaic. All mosaics were constructed from the same Navcam source images.
5.3. Planetary Data System (PDS) Archiving
 The MER project is required to submit a copy of the acquired mission data in the PDS within 6 months after receipt on Earth. This delivery will include the entire set of image EDRs from all cameras. It will also include a selected set of image RDRs. Because all of the MER image data used during surface operations are created in PDS format, the format of the archived data will be identical to the original source data. In addition to the operations data, the project will also submit a selected subset of the >180,000 images acquired during camera subsystem calibration and camera system testing prior to launch.
 The MER mission will land 20 cameras onto the surface of Mars in early 2004, and the mobile nature of the MER rovers offers the chance to view the surface at an unprecedented level of detail over a wide swath of Martian terrain. This paper provides an overview of the design, implementation, and capabilities of the 14 engineering cameras, along with a brief description of the overall capabilities of the MER imaging system onboard the rover and the ground processing system on Earth. Images acquired from the MER engineering cameras for the purposes of operating the vehicle on the surface of Mars will also be used to achieve a number of the scientific goals of the MER mission. These goals include the investigation and characterization of the local terrain at cm/pixel resolution, the characterization of the morphology and spatial distribution of rocks and soils, and the validation of orbital remote sensing data. Additional details on the Pancam and Microscopic Imager cameras are described in the complementary publications by Bell et al.  and Herkenhoff et al. .
 The MER camera effort required the concerted effort of skilled and dedicated people. Bringing 20 flight cameras from concept to launch in <3 years is a remarkable accomplishment and could not have been done without the contributions of Dave Thiessen, Darryl Day, Nancy Cowardin, Heather Arneson, Miles Johnson, Jonathan Joseph, Jascha Sohl-Dickstein, Jeff Johnson, Paul Karlmann, Ali Pourangi, Perry Fatehi, Bobbie Woo, Greg Lievense, Greg Smith, Len Wayne, Mary White, Pete Kobzef, Joe Melko, John Callas, Joy Crisp, Pete Theisinger, Richard Cook, Barry Goldstein, Deborah Bass, Charles Budney, Raul Romero, Art Thompson, Leo Bister, Jackie Lyra, Chris Salvo, Pete Darus, Dave Gruel, Jessica Collisson, Jennifer Trosper, Helen Mortensen, Payam Zamani, Hyun Lee, Oleg Pariser, Costin Radulescu, Bruce Bon, Glenn Reeves, Ed Motts, Mark Thompson, and the rest of the MER project.