I. Final Processing Pipeline (2MAPPS v3.0) Development Strategy

• Processing Schedule



• present - 3/01 - Preliminary processing (through end of observatory ops)
• ~12/00 - Next Incremental Data Release (20-30% of the sky)
• 1/00-3/01 - 2MAPPS v3.0 software development and test
• 3/01-4/01 - Final uniform processing
• 11/01-9/02 - Final product generation
• 10/02 - 2MASS final product release


• 2MAPPS v3.0 development and implementation guidelines (full document)



• Prioritize modifications/improvements according to the following criteria:



1. Fixes to known bugs or deficiencies that compromise Level 1 Requirements
2. Addition of diagnostics to improve quality validation and/or final product generation with existing algorithms
3. Incorporation of a priori knowledge of the system performance and the near-infrared sky into the final processing to yield benefits in the processing, uniformity and data quality
4. Modifications to improve efficiency and runtime of current algorithms
5. Improvements to increase yield of data that meet Level 1 Requirements
6. Modifications to improve data quality beyond Level 1 Requirements


• Modifications/Improvements cannot increase pipeline runtime by more than ~25%



• Final uniform processing will be limited to data taken under conditions known to meet survey quality requirements for atmospheric transparency, seeing, background, etc.



• Wherever possible, incorporate modifications/upgrades into preliminary processing pipeline to validate



• Preliminary pipeline is testbed for final processing

II. Examples of Improvements Planned for Final Processing Pipeline
(go to the links for detailed subsystem development/analysis web pages)

• Meteor trail blanking



• Spurious detections along meteor trails are a major source of unreliability in point and extended source extraction
• Detect trails as single frame events - mask area on frame and remove from Atlas Images
• Currently tuning trail detection algorithms


• Bright star photometry



• No useful photometry derived for R1-saturated stars in preliminary processing (40,000-50,000 full sky)
• Positions and photometry useful scientifically as well as for accurate identification of artifacts produced by these stars
• A simple algorithm has produced 0.2 mag RMS photometry for m<3 stars
• Now testing 1-d radial profile-fitting


• Point source deblending



• Passive source deblending only in preliminary processing
  (i.e. multiple PSFs fit only if >1 source detected within 5")
• Active deblending will allow single detection with large chi-squared to be split into 2 simultaneously-fit components
• Improved photometric accuracy and completeness in confused regions
• Utilize better PSFs and noise model
• Currently testing magnitude and density thresholds for run-time considerations


• Photometric corrections and normalizations



• Preliminary processing determines aperture C.O.G., and PSF-fit and R1 photometry normalizations empirically for each scan
• Use survey-averaged values as a function of FWHM to avoid errors due to effects of confusion and bright star artifacts
• Currently evaluating survey-averaged values and dependencies on seeing and other image quality parameters


• Photometric calibration



• Expanded network of internally-consistent secondary standard stars applied uniformly to all data

Fig II.1 - Nightly photometric zeropoint offset solution from 971015n (1 std/field; RMS~0.02 mag)
Fig II.2 - Nightly photometric zeropoint offset solution from 991015n (>10 std/field; RMS~0.007 mag)
• Utilize seasonally-dependent atmospheric extinction coefficients, if warranted
• Global chi-squared minimization analysis of all calibration scan data will be conducted at completion of preliminary processing


• Astrometry



• Mean astrometric residuals 0.2"-0.3" in preliminary processing, but there are outliers >1.0"
• Use higher density Tycho 2 astrometric reference catalog

Fig II.3 - RA (top) and DEC (bottom) difference histogram for stars in overlap region (black - preliminary processing; red - using Tycho 2)
• Propagate astrometric solution across Tile boundaries using overlap information

Fig II.4 - Radial position difference for stars in overlap plotted as a function of in-scan position (black - preliminary processing; red - using global propagation of solution)
• Process has been demonstrated to reduce mean residuals to 0.1"-0.2" and reduce significantly the number of outliers


• Extended sources



• Expanded and improved training sets for galaxy classification - improved relibility
• Better sensitivity/completeness in the Zone of Avoidance

Fig II.5 - Extended source surface density versus galactic latitude

III. Final Product Generation (FPG)

• Pipeline processing generates Working Source Databases



• "Detections" to low SNR to insure completeness
• Duplicate apparitions for sources in Tile overlap regions
• Artifacts from very bright stars and bright stars off Tiles, meteors and other non-celestial artifacts
• Extended source databases include galaxies and galactic extended sources


• FPG tasks draw high reliability, complete Catalogs from Working Databases



• Select best apparition of Tiles for inclusion in release
• SNR thresholds for reliability and flux accuracy
  
  • Incremental releases use conservative SNR>7 cut
  • Consider lower threshold for final release, and characterize impact
• Point sources - chi-squared and N/M limits to remove radiation hits
• Identify and select correct apparition for duplicate sources
• Identify and remove/flag bright star and other artifacts
• Extended source/galaxy classification
• Astrometric uncertainty updates - will be incporporated into final Tile processing
• Ancillary additions and clean-up:
• Point/Extended source links, contamination flagging
• Asteroid/comet flagging
• Identification/blanking extended sources entries in untracked seeing areas, galactic center
• Miscellaneous artifact clean-up and flag setting
• Extensive analysis to validate Catalog quality
• Prepare release documentation
• Prepare soft and hardcopies of release data
• Prepare IRSA tools/links to Catalogs



• FPG is DB and Human-Intensive Task



• Final release FPG schedule is ambitious - 10 months for full-sky



• Improving DB task performance - goal is 6x, have demonstrated 4x



• Pipeline improvements will reduce load on FPG (e.g.bright star photometry and positions; global astrometric solutions)



• Include new diagnostics in pipeline processing to assist with final product analysis

IV. Contingency Plan if FY01/02 Funding Not Restored by NASA Senior Review

• FY01/02 allocation is $1.78M less than original agreed budget (11% shortfall in FY01, 40% shortfall in FY02)



• Restoration of original budget requested at NASA Senior Review in June 2000.



• Key tasks in FY01/02 are final incremental release, final uniform processingand final release FPG



• Contingency plan:



• Cancel further incremental releases



• Do not carry out final uniform processing



• Limited improvement of existing Working Source Databases, but no new processing (e.g.do not touch bits)



• Limited improvements to point source positions using Tycho 2 and global solutions
• Rederive nightly photometric calibration solutions and apply to existing source photometry


• Generated final release products from improved Working Databases