2MASS Data Access for Science Team
Members
Operational 2MASS data processing commenced on 16 April 1998, starting with data acquired from the northern 2MASS facility on 21 May 1997 UT. Processing is being carried out with version 2.0 of the 2MASS Production Processing System (2MAPPS), including full Quality Assurance review. Data meeting QA minimum standards are being loaded into the working databases, and compressed version of the Atlas Images are being served by the 2MASS Survey Visualizer. The processing rate has been slow for a number of reasons during the first 2 months, including problematic data (e.g. non-optimal dither patterns, elongated images, etc.), exercising the Quality Assurance system for the first time and the inevitable occurrence of minor software bugs that are being encountered with very large amounts of data being processed on a regular basis. As of mid-June, approximately one month of Survey data has completed full processing.
A number of minor changes have been made to 2MAPPS v2.0 in response to bugs encountered in the first month of processing. These changes in general do not alter quantitatively photometry or astrometry, but rather accommodate unusual conditions encountered in certain scans allowing position reconstruction, for example, to be carried out where it previously failed. The software version in use today is more realistically termed v2.0x.
During operational 2MASS processing, one night's data are processed end-to-end, including calibration and automated quality assurance, in approximately one day (nights with large numbers of Galactic Plane scans take somewhat longer). The processed data remain on-line on the production computers (barney - north and rex - south) for 1-3 days during the human Quality Review processed. After Quality Review, source lists, nightly and scan information and Atlas Images are transferred into the database system. We recommend strongly that you plan on accessing data via the DB system.
Delivery of the south-ready 2MAPPS v2.1 is scheduled for the end of July 1998 UT, and southern operational processing will commence shortly thereafter. After the completion of Survey Operations, there will be a complete reprocessing of all Survey data during 2001. This final reprocessing will utilize 2MAPPS v3.0 - a system that will incorporate all of our knowledge accrued during the life of the Survey.
This brief document is designed to provide a guide to finding and accessing processed 2MASS data products for the 2MASS Science Team. Once you have drilled down to real 2MASS data, you will require the Team password to access it.
Okay, that's all well and good. You want to see data now.
Brant Nelson has written a Web-based query tool called FindTile
that allows you to input a RA and DEC, and returns the Tile Database
entry (from the IPAC working copy). This tells you the tile(s) in which
the input coordinates fall, and if the tile has been observed.
Since the beginning of survey operations in the north and south, the Calibration data have been processed for every night during which observations have been made. Operational processing of all Calibration and Survey data has taken place for approximately 30 northern nights to date. To determine the status of data processing for any night during the survey, point your Web browser at the Data Processing Status and History Page . You can also access this site from the IPAC/2MASS Survey Operations link.
The Data Processing Status and History Page lists for every survey night the observation date, and dates of tape receipt and validation and data processing at IPAC. If you examine the processing dates for nights early in the survey, you will see that the date has been updated to reflect the Ops processing, overriding the earlier calibration scan processing. The Processing Status page also provides a link to the 24hr Telescope/Camera Health QA summary page for that night.
If you select on an Observation Date, you will see a list of all scans on that night, and the processing status. "P" indicates that the scan processed successfully, "F" indicates processing failure, "R" indicates the scan is currently running in the pipeline, and "-" indicates that the scan has not been processed. Data that have finished Quality Review and have been loaded into the database will eventually have the quality score and loading indicating in this table, as well, although this is not yet active. Note that you can find the assigned quality for a Tile by reviewing the Tile DB entry using FindTile .
The full-resolution Atlas Images are spooled to tape after processing, so they are available for special requests. It is hoped that the rapidly declining cost of mass storage will ultimately make it possible to put the ~10TB of full-resolution image data on-line.
Some of the older processed full-resolution Images are still available
on the HPSS tape robot at the Caltech CACR. They can be retrieved from
there using the prototype 2MASS
Image Atlas Server. The server allows you to specify an equatorial
position and returns either J, H and Ks sub-images up to 5'x5', or the
entire Atlas Image in which the coordinate is located. The retrieval time
is relatively slow because of the tape robot latency time. We may
use the HPSS as temporary storage for the ops-processed full resolution
images if the band-width to CACR can be improved.
The preferred mode of access to ops-processed point and extended source data, as well as nightly and scan information data, is via the Catalog Scanning web interface (click on Catalogs (CatScan) under the Applications sidebar). Queries can be made on date/scan/magnitude/color/position or just about any other parameter or combination that one can think of.
Science Team members are encouraged to exercise the CatScan web interface
to the 2MASS source databases. Please send any feedback to
Joe Mazzarella at IPAC (mazz@ipac.caltech.edu).