IPAC/2MASS Operations and Data Management is described by following
2MASS data from observation planning, through acquisition, processing and
archiving.
rodan% monitor
r currently running processes
c completed processes by subsystem
o output for a scan
p problem scans other than 0 return
s disk space
i change update interval
m create or change mask value
q quit monitor
enter letter for desired option:
r - currently running processes:
2MAPPS processing as of Jul 19 15:35:12
EXEC status: Jul 19 15:29:45 970712n 37 cal starting 18 of 60
TYPE PID DATE SCAN PROCESS MON DAY START STEPTOT SCANTOT STAT
cal 1436 970712 035 galworks Jul 19 15:31:15 00:03:57 00:16:33 RUN
cal 4654 970712 036 posfrm Jul 19 15:34:40 00:00:32 00:07:59 RUN
cal 5271 970712 037 pixcal Jul 19 15:29:52 00:05:20 00:05:26 RUN
c - completed processes by subsystem:
2MAPPS scans output as of Jul 19 15:36:34
found 19 scans ...
NUM DATE SCAN TYPE MON DAY START STAT
1 : 970712 006 cal Jul 19 12:34:37 0
2 : 970712 007 cal Jul 19 12:36:37 0
3 : 970712 008 cal Jul 19 12:38:37 0
4 : 970712 009 cal Jul 19 13:33:03 0
5 : 970712 010 cal Jul 19 13:33:23 0
6 : 970712 011 cal Jul 19 13:38:13 0
7 : 970712 012 cal Jul 19 14:30:15 0
8 : 970712 013 cal Jul 19 14:36:59 0
9 : 970712 014 cal Jul 19 14:39:49 0
10 : 970712 015 cal Jul 19 14:46:45 0
11 : 970712 016 cal Jul 19 14:53:46 0
12 : 970712 017 cal Jul 19 14:56:40 0
13 : 970712 032 cal Jul 19 15:02:38 0
14 : 970712 033 cal Jul 19 15:10:59 0
15 : 970712 034 cal Jul 19 15:12:55 0
16 : 970712 035 cal Jul 19 15:18:39 0
17 : 970712 036 cal Jul 19 15:27:13 RUN
18 : 970712 037 cal Jul 19 15:29:46 RUN
19 : 970712 038 cal Jul 19 15:35:17 RUN
error messages currently OFF enter e to display errors
choose number to display, l to list numbers or q to quit:
detailed summary by scan:
error messages currently OFF enter e to display errors
choose number to display, l to list numbers or q to quit: 11
TYPE PID DATE SCAN PROCESS MON DAY START STOP ELAPSE STAT
cal 23937 970712 016 PCP_test Jul 19 14:53:46 15:10:49 00:17:03 0
cal 23937 970712 016 pixcal Jul 19 14:53:52 15:01:08 00:07:16 0
cal 23937 970712 016 posfrm Jul 19 15:01:08 15:02:36 00:01:28 0
cal 23937 970712 016 pixphot Jul 19 15:02:36 15:06:01 00:03:25 0
cal 23937 970712 016 mapcor Jul 19 15:06:01 15:06:16 00:00:15 2
cal 23937 970712 016 bandmerge Jul 19 15:06:16 15:06:20 00:00:04 0
cal 23937 970712 016 pospts Jul 19 15:06:20 15:06:21 00:00:01 0
cal 23937 970712 016 galworks Jul 19 15:06:21 15:10:49 00:04:28 0
error messages currently OFF enter e to display errors
choose number to display, l to list numbers or q to quit:
1. Main processing systems: 2 x Sun Enterprise 3000
4x333MHz CPU
128MB memory/CPU
128GB disk farm
Status: Northern processing system (4x167MHz) in place.
Order Southern processing system (4x250MHz) Fall 1997.
Upgrade both to 4x333MHz first quarter 1998.
2. Tape I/O Systems: 2 x Sun Ultra 30
1x333 MHz CPU
128MB memory
54GB disk farm
Single DLT 4000 tape drive (read Obs. Raw Data Tapes)
Dual DLT 7000 tape drive (write Archive Raw/Processed Data Tapes)
Status: Northern tape I/O system (1x167MHz) in place.
1x333MHz system ordered to replace northern tape I/O.
Order Southern tape I/O system (1x333MHz) Fall 1997.
3. Software and Science Analysis System: Sun Sparc 1000
4x85MHz CPU
128MB memory/CPU
70GB disk farm
Status: In place
4. Database Build System: Sun Sparc 1000
4x85MHz CPU
128MB memory/CPU
70GB disk farm (share DB Server's 600GB)
Status: In place
5. Database Server: Sun Enterprise 3000
4x167MHz CPU
128MB memory/CPU
600GB disk farm
Status: Current northern processing system will become Database Server
in early 1998.
Budget for minimal performance disk array.
Seeking funding for high-performance disk array system.
6. Image Server:
Served by Database Server
Storage:
Full Resolution Atlas Images: Caltech CACR Tape Robot Facility
(5TB Capacity)
20:1 Compressed Images: 2x50 DVD-Rom Jukebox
(1 TB Capacity)
Status: Shipping Full Resolution and Compressed Atlas Images
to CACR Facility. Access limited to 1-2 years?
DVD-Rom Jukebox available now, but mastering stations
won't be until late 1998.
7. All systems linked with 80 Mb/s ATM fiber network (in place)
2MASS 24-Hour Monitor
___________________________________________________________
Observation Date 970711n
=== STATUS RED ===
-------------- pointing summary --------------
PctNPM = 0% (Failed Position Reconstruction)
Pointing:
AvDra = -10.563 arcsec
SigDra = 3.911 arcsec
FAIL AvDdec = -30.844 arcsec
SigDdec= 6.805 arcsec
-------------- scanning summary --------------
Scanning:
dtheta(J,H,Ks) = 0.286 deg
sigthet(J,H,Ks) = 0.017 deg
Offsets:
avyoff(N,S) = 82.626 arcsec -82.633 arcsec
sigyoff(N,S) = 0.222 arcsec 0.246 arcsec
avxoff(N,S) = 0.473 arcsec -0.482 arcsec
sigxoff(N,S) = 0.177 arcsec 0.209 arcsec
Tracking:
maxdra = 4.458 arcsec
rmsdra_first = 1.376 arcsec
------------ seeing/focus summary ------------
---- Avshape(J,H,Ks) = 1.112 1.083 1.087
---- sigshape(J,H,Ks)= 0.073 0.068 0.063
---- AvFWHM(J,H,Ks) = 3.648 arcsec 3.505 arcsec 3.528 arcsec
AvAspct(J,H,Ks) = 0.955 0.984 1.021
FAIL MnAspct(J,H,Ks) = 0.879 0.861 0.760
FAIL MxAspct(J,H,Ks) = 1.153 1.203 1.310
------------ photometric summary ------------
---- AvZP(J,H,Ks) = 0.048 mag 0.093 mag 0.119 mag
FAIL sigZP(J,H,Ks) = 0.097 mag 0.098 mag 0.087 mag
---------- dark/responsivity summary ----------
Read 1:
diffdark(J,H,Ks)= -42.52 DN 1.26 DN 10.72 DN
sigdark(J,H,Ks)= 32.56 DN 70.43 DN 11.87 DN
Ndrkmask(J,H,Ks)= 812 pix 1039 pix 692 pix
diffresp(J,H,Ks)= 0.00008 DN 0.00562 DN 0.00080 DN
sigresp(J,H,Ks)= 0.01343 DN 0.01963 DN 0.00419 DN
Read 2:
diffdark(J,H,Ks)= 48.57 DN -0.29 DN 5.59 DN
sigdark(J,H,Ks)= 13.52 DN 35.01 DN 37.14 DN
Ndrkmask(J,H,Ks)= 765 pix 765 pix 520 pix
diffresp(J,H,Ks)= 0.00008 DN 0.00562 DN 0.00080 DN
sigresp(J,H,Ks)= 0.01343 DN 0.01963 DN 0.00419 DN
2MASS 24-Hour Monitor
Generated on 07/16/1997
at 18:15:17
from q970711n.pfqqas
from q970711n.see
from q970711n.cal
from q970711n.drk
I. Objectives
At the request of Rae Stiening, IPAC has agreed to supply rapid turnaround
reports on 2MASS Camera/Telescope functionality. The objective of these
reports will be to detect hardware or software failures or errors at the
Observatory, and to allow corrective action to be taken by UMASS before too
much time has elapsed.
II. Operational Requirements
The Observatory has agreed to ship Raw Data tapes twice per week. IPAC will
read, validate and report on system Q/A for the tape with the latest date in
each shipment. A report on six basic performance criteria areas will be
returned to UMASS within 24 hours of receipt of the tape shipment.
Note that IPAC must still support validation of other tapes received within
<= 3 days.
III. Report Requirements
Camera and Telescope performance will be monitored through 2MAPPS processing
of the Calibration scans for the latest tape in each tape shipment. A report
in six basic performance areas will be generated from the pipeline output.
In addition to the actual parameters, IPAC will generate a green or red status
for each of the six performance areas by comparing the achieved performance
to canonical values and performance thresholds.
The six performance areas are:
1. Absolute pointing: Did the telescope point to the coordinate specified in
header?)
2. Scanning: Did the telescope scan correctly? Was the dither pattern
optimal?
3. Focus: Was the telescope in focus? Was the image quality within spec?
Note that we currently do not have a proven method to distinguish
poor telescope focus from poor seeing conditions. It may be possible
to determine this from difference between H and Ks image shape, but
this has not yet been confirmed.
4. Photometry/System Gain: Are the counts for 0 magnitude correct for each
band? Note that this will be highly dependent on the photometric
quality of each night. If observations are made through clouds, the
effective system gain is greatly reduced.
5. Darks: Are the dark frames "normal?" Do the dark masks indicate that more
pixels than expected were turned off because of abnormal dark current?
6. Responsivities: Is the responsivity of the system "normal?" Are there
any gross changes in the responsivity maps that would indicate optical
problems such as excessive dust on the camera window? Do the responsivity
masks indicate that more pixels than expected were turned off because of
low responsivity?
The report of the parameters, and the status report will be sent to UMASS
via email within 24 hours of receipt of the tapes.
IV. Schedule
The 24hr turnaround Observatory Q/A reporting will be operational by July 1,
1997 with preliminary thresholds in place. It is expected that the threshold
values will evolve as the system is better characterized.
V. Implementation
1. Scope
Rapid turnaround of system QA requires that the QA processing be carried out
on the Tape I/O computer, otherwise, there will be too much conflict with
normal nightly data processing. Such high throughput on the single cpu
Tape I/O computer requires that the Camera and Telescope performance analysis
will be monitored through 2MAPPS processing of only the calibration scans
obtained on each night in question.
2. Responsible Subsystems
The reported parameters will be evaluated within individual subsystems
and reported concisely in their respective quality output files. The relevant
subsystems are DARKS, POSMAN, SEEMAN/PIXPHOT and CALMON. The QUALITY subsystem
will gather the parameters from each Q/A output file and compile them in
a compact report, and will reference the parameters against threshold values.
A status flag of green or red will be assigned for each performance area/
parameter based on the comparison with the threshold values: parameters
that fall within the threshold values will have status green, those
falling outside the range will be given status red.
3. Performance Parameters:
a. Absolute Pointing:
pctnpm = Percent of scans for which no pattern match could be found
AvDra, AvDdec = Average magnitude of difference between the adjusted* RA
and Dec of commanded telescope start-of-scan position,
as contained in the image header, and the reconstructed
center of the first frame in a scan. (*Note- "adjusted"
means that mean offsets determined from a large sample
of previously processed scans have been applied)
sigDra, sigDdec = RMS of the difference between the right ascension and
declination of the commanded telescope position (adjusted)
and the reconstructed first frame center, for all
calibration scans in the night.
Responsible Subsystem: EXEC, POSMAN
Green Thresholds:
pctnpm < 15%
and
AvDra < 20", AvDdec < 20", sigDra < 15", sigDdec < 15"
b. Scanning Accuracy:
dtheta(J,H,Ks) = Average for the night of the rotation angle of the
three arrays with respect to scan direction.
J and H should be corrected to Ks rotation with
constant offset, and one value quoted.
sigthet(J,H,Ks) = RMS of the rotation angles of the composite of three
arrays for the night.
avyoff, avxoff = Average in-scan and cross-scan step sizes, in arc-seconds,
for all calibration scans during the night.
sigyoff,sigxoff = RMS of the in-scan and cross-scan step sizes, for all
calibration scans during the night.
maxdra = Magnitude of the mean maximum deviation of reconstructed frame-
center RA's from the of-date TA of the first frame in the scan,
for all calibration scans in the night.
sigdra = RMS of the deviation of reconstructed frame-center of-date RA's
from the of-date RA of the first frame in the scan, for all
calibration scans in the night.
pctnpm = Percent of scans for which no pattern match could be found
Responsible Subsystem: POSMAN
Green Thresholds:
0.25 < dtheta(Ks) < 0.35 (J and H corrected to Ks)
sigthet < 0.02
82.4" < avyoff < 82.8"
maxdra < 15"
sigdra < 10"
c. Image Quality
There is as yet no reliable discriminant between poor focus and poor seeing.
The only indicator of a serious instrument problem is if the shapes of the
images in the three bands are very different. So the appropriate conditions
are either the J, H and Ks shapes are all excellent, or if they are not,
then all are at least consistent. This is reflected in the thresholds below.
Avshape(J,H,Ks) = Average image "shape" parameters for all calibration
scans in each of the three bands.
sigshape(J,H,Ks) = RMS of the image "shapes" in each of the three bands.
Responsible Subsystem: SEEMAN/PIXPHOT
Green Thresholds:
if sigZP(J,H,Ks) < 0.05 mags
then
Avshape_j < 1.1 AND avshape_h < 1.1 AND Avshape_k < 1.1
or
1.04 < FWHM_j / FWHM_h < 1.09 AND 0.95 < FWHM_h / FWHM_k < 1.05
where FWHM_j**2 = 4 + 16 * (avshape_j**2 - 0.654)
d. Zero Point/System Gain
Note that non-photometric conditions can make these parameters meaningless
for monitoring system health. As a rule of thumb, the first parameter
to be analyzed should be the dispersion in the photometric zero point,
sigZP. If sigZP indicates very little variation in the zero point for
the night, then it can be assumed that there were not significant clouds
during the night. If the night was clear, then the value of the
zero point offset can be used as a meaningful measure of system gain.
AvZP(J,H,Ks) = Average photometric zero point offset in J, H and Ks for
all calibration scans observed during night.
sigZP(J,H,Ks) = RMS of photometric zero point offsets in J, H and Ks for
all calibration scans observed during night.
Responsible Subsystem: CALMON
Green Thresholds:
if sigZP(J,H,Ks) < 0.05 mags
then
AvZP(J,H,Ks) > -0.3 mags
e. Dark Images
Note that dark and twilight sky flat field images may not be acquired
every night during the survey. Therefore, the 24hr Q/A analysis should
derive the following parameters only if the dark and twilight sky flats
are known to exist. In the absence of darks and/or flats, these parameters
should be assigned the value "NA."
diffdark(J,H,Ks) = Mean offset between average nightly and canonical dark
frames in J, H and Ks.
sigdark(J,H,Ks) = RMS of difference between pixels in nightly and canonical
dark frames.
Ndrkmask(J,H,Ks) = Number of pixels masked in darks.
Responsible Subsytem: DARKS
Green Thresholds: (John Fowler, please supply correct thresholds)
diffdark < 500DN ???
sigdark < 50DN ???
Ndrkmask < normal + 25%
f. Responsivity Images
ratioresp(J,H,Ks) = Median ratio of between nightly and canonical
J, H and Ks responsivity images. (Note, this
may become a difference rather than a ratio. If so, the
threshold conditions will be different.)
sigresp(J,H,Ks) = RMS of differences between nightly and canonical
J, H and Ks responsivity images.
Nrspmask(J,H,Ks) = Number of masked in responsivity.
Responsible Subsytem: DARKS
Green Thresholds: (John Fowler, please supply correct thresholds)
0.5 < ratioresp < 1.5
sigresp < 100DN ???
Nrspmask < normal + 25%
4. Summary Report
For each tested night, a summary report of the performance parameters
will be generated by the QUALITY subsystem. The report will give a
green/red status for each parameter. The format of this report will
be described in quaXX Subsystem Interface Specification Document.
This report will be automatically sent to the 2MASS Operations center
at UMASS via email (ops@m2mass.phast.umass.edu).
VI. Stipulations
1. In order for 24 hour turnaround to be achieved prior to full 7-day staffing
for 2MASS operations at IPAC, UMASS has agreed to not ship tapes so that
they arrive Friday, Saturday or Sunday. Should tapes be received on any of
those days, QA report will not be available before the following Tuesday.
IPAC will notify UMASS if and when 7 day staffing becomes available.