UNDER CONSTRUCTION

I. Introduction

As part of the 2MASS Extended Mission, the point and extended source Working Databases (WDBs) from the Survey and Calibration scans are to be released to the user community. The release of these data sets will enable users to: 1) conduct research in the time domain with 2MASS, studying time variability and proper motion; and 2) utilize near infrared measurements of sources fainter than the SNR limits used to construct the All-Sky Release Catalogs.

II. Contents of the Working Databases

Nightly 2MASS data processing generated lists of positions and photometry for all "sources" extracted from survey and calibration Tile scans that were loaded into point and extended source WDBs. Table 1 contains the number of extractions contained in each of the WDBs, and the number of scans from which they were derived.

Table 1 - Number of Scans and Extractions in the Survey and Calibration Working Databases

WDB	No. Scans	No. Point Srcs	No. Extended Srcs
Survey	70,712	1,314981,867	2,590,500
Calibration	73,230	191,464,020	403,811

The All-Sky Release PSC and XSC were drawn from the much more extensive Survey WDBs with the goal of producing the highly uniform, reliable and complete catalogs that were the main deliverables from the primary 2MASS mission. In addition to the subset of sources that comprised the PSC and XSC, the WDBs contain:

• Multiple detections of sources that were scanned more than once during the survey. For the Survey WDBs, most of these fall in the Tile overlap regions. However, approximately ~14% of all Survey Tiles were scanned two or more times either for validation purposes or to obtain measurements under better atmospheric conditions. In the case of the Calibration WDBs, each of the 35 Tiles was observed ~600 to 3600 times.



• A large number of low signal-to-noise ratio detections that are a mixture of faint, real astrophysical sources and noise excursions. Detections were made at the SNR~3 level during pipeline processing, well below the SNR=7 limit used for the PSC and XSC, to insure the completeness of the catalogs.



• Residual multiple stars and other false resolved sources, in the case of the Extended Source WDB.



• All flagged and unflagged detections of image artifacts produced by bright stars



• Detections of transients such as cosmic rays, residual meteor trails and hot pixel events

The Calibration WDBs contain the same mix of extractions as the Survey WDBs, but will contain a much higher proportion of multiply-detected real sources because of the high degree of redundancy on the observations. Other than the photometry and positions of the standard stars in each of the fields, none of the Calibration WDB data have been released directly.

Links to Analysis Reports of 2MASS WDBs

Links to Analysis Reports of 2MASS Final Products

III. Preparing the WDBs for Release

1. Scope of the Release Products

• All WDB entries
• Flag artifacts
• Reliability "scoring"
• Flag indicating PSC/XSC entries


• Filtered subset of extractions
• Exclude flagged artifacts
• Exclude very low SNR sources
• Exclude other high probability artifacts (e.g. hot pixels, R1 glints, etc.)
• Include or exclude PSC/XSC entries? (i.e. a reject file only)


• Merged Database
• All apparitions of same source grouped, linked

2. Databases Updates

Status of Survey and Calibration WDB Preparation

• Bring into compliance with All-Sky Release Catalogs
• Recalibrated June 1997 nights (DONE)
• Planetary satellite flagging (DONE)
• Photometric uncertainties in scans affected by clouds
• Correct PSC a (optical association) values for associations with separations >5"
• Other?


• Value-added columns that where generated for the All-Sky Release
• PSC: [jhk]_cmsig, [jhk]_msigcom, ph_qual, cc_flg (from DB_MAPCOR), jdate, coadd, coadd_key, designation, use_src, dup_src
• XSC: vc (feye), cc_flg (from DP_MAPCOR) jdate, designation, use_src, dup_src
• Scan Data: [jhk]_msnr10
• Coadd Data: ???


• Other value-added columns
• PSC/XSC flags - indicating of an entry was identically in the PSC/XSC
• Flag extractions with high probability of being artifacts such as hot pixels, R1 glints, edge flares, etc.
• Reliability Score

IV. Merging the WDBs

Merging the source extractions is done by positionally autocorrelating the Working DBs. The matching radius should be as small as possible to minimize false associations, yet large enough to take into account position reconstruction uncertainties on the faintest sources. A 2" matching radius was used for duplicate source resolution when generating the PSC, and 5" was used for the XSC. The proximity distributions for the PSC and XSC show abrupt upturns in the proximity distributions of the catalogs at ~3" and 5", respectively.

It is desirable that the merging results are independent of the order in which sources are encountered in the WDB lists.

2. Limitations

The merging process is limited by several factors:

• Source and Sensitivity Variations - Sources may toggle above and below the detection thresholds either if they vary in brightness or if the achieved sensitivity differs between different observations. This is a band-by-band effect, so depending on color, a source may detected in a different set of bands in different observations.



• Motion - Sources may fail to positionally confirm between scans if they move more than the matching radius in the time between observations. A larger search radius will capture small net motions but will also allow more false associations.



• Artifacts and Noise Detections - Artifacts and/or low signal-to-noise detections of noise excursions may or may not confirm between observations.



• Noise extractions have a nearly random distribution and will thus have a low probability of confirmation between normal survey scans away from the poles. However, for survey scans near the equatorial poles and for all of the calibration scans, the number of overlapping scans is large. Thus the density of all superimposed noise extractions is high leading to a high probability of confirmation of noise extractions as well as false associations between noise extractions with nearby real detections. To avoid this, it may be desirable to apply a modest SNR filter before merging at the expense of confirming the faintest sources.
• Some artifacts, such a dichroic ghosts and latent images may repeat in position leading to confirmation between scans.


• Confusion - Confusion occurs when a source can potentially be matched with more than one candidate source in another scan, or with a second source in their own scan leading to confusion in the merging process. Confusion can be caused by:



• Source pairs with unphysically small separations exist in the database. These are caused by incorrect R1/R2 merges, failed bandmerges and possibly confusion with with unflagged artifacts.
• Differing resolution due to seeing/focus variations. Real sources with separations near the resolution limit of the survey instruments may be resolved in some scans and not others depending on seeing. Thus, an unresolved pair in one scan may have two possible merge candidates in a scan in which the pair was resolved.

3. Quantities to Derive from the Merged Groups

• Average and RMS brightness in 3 bands
• Weighting?
• How to deal with non-detections in bands/scans
• What to do in confused groups
• What to do if merge group members have different artifact flagging?
• What parameters should be derived to provide measures of possible variation? (simple RMS, Stetson variability index, structure function?). Most merge groups will have two apparitions.


• Average and RMS position
• Weighting? (SNR, cross-scan position)
• How to deal with non-detections in bands/scans
• What to do in confused groups
• What parameters should be derived to provide measure of possible motion?


• Confirmation statistics - number of detections (sdet) vs. number of observations (spos) (i.e. N-out-of-M)


• How to compute possible number of observations? Group mean or single seed
• What to do in confused groups


• Group linkages


• WDB cntr values for each linked member of merged group to allow rapid cross-reference access to all group members for light curves, sky plots, etc.


• What parameters should be merged/averaged for extended sources?


• Which magnitudes?
• Which shape parameters?


• Combined artifact or quality flagging

4. How Should Merged DBs be presented?

• Hiearchical data sets
• Top level = merged DB: mean parameters, flags, etc
• Cross-references between merged groups and individual entries in WDBs (group index table)
• Working DBs


• Possible tools to access individual apparitions for given merged group
• For a given merge group (gcntr), list individual WDB entries
• Light curves
• Sky Plots

V. Reliability Scoring

Because a significant fraction of the extractions in the WDBs are likely not real, it is imperative that each extraction be accompanied by some estimate of reliability. This will be especially critical if we choose to release all entries in the WDBs. In addition to all of the reliability-related flags and scores that are derived for each "source", it is proposed that we derive a "reliability score" (0 to 9 with 9 being best and 0 being worst) that relates to the probability that a WDB entry corresponds to a true astrophysical source that was at the reported position at th time of the observation.

This flag is not intended to necessarily encode whether or not the flux and/or position measurement is within some accuracy since there are other flux quality flags and position uncertainties.

The reliability score may be based on, but not be limited to, the following attributes:

• Identification with bright star artifacts (MAPCOR and DB_MAPCOR and extended source visual classification), hot pixels, and other known artifacts (i.e. cc_flg, feye)



• For point sources, frame detection statistics (ndet)



• Number of 2MASS bands detected (i.e. - 3 band most reliable, single band least reliable)



• SNR of detected bands (i.e. derived from repeats of calibration scans)



• For sources in Tile overlaps or repeatedly observed Tiles, confirmation statistics from merging process (ratio of number of times detected and number of times scanned - sdet/spos)



• Results of visual examination (i.e. feye flag for extended sources)



• Associations with external catalogs (e.g.Tycho 2, USNOA/B, UCAC, FIRST, NVSS, SDSS, UGC, etc.)
• How many catalogs to consider? All-sky only?
• How to efficiently cross-identify, deal with confusion?
• Use cross-catalog color-information?


• Etc....

Last Updated: 14 April 2004
R. Cutri - IPAC