BYYMMDDH % Pass /2massc/del_v3 filename
j970608n 91.8%: datan/jsat.fits
h970608n 80.2%: datan/hsat.fits
h990929n 64.5%: datan/new_hsat.fits
k980106n 95.5%: datan/ksat.fits
j980508s 94.5%: datas/jsat.fits
h980508s 93.9%: datas/hsat.fits
k981120s 97.5%: datas/ksat.fits
Some information on run numbers, fit limits, and RMS accept criteria on above:
Run BYYMMDDH F Fit Lims RMS limits Pass filename
280: h990929n 0 11453554 RMS 120/170 64.5%: n/new_hsat.fits
278: h970608n 0 19471952 RMS 120/ 90 80.2%: n/hsat.fits
277: h980508s 0 00461552 RMS 120/ 90 93.9%: s/hsat.fits
276: j970608n 0 12431946 RMS 180/ 90 91.8%: n/jsat.fits
275: k981120s 0 00431549 RMS 150/ 90 97.5%: s/ksat.fits
274: j980508s 0 00441549 RMS 180/ 90 94.5%: s/jsat.fits
269: k980106n 0 12431550 RMS 200/160 95.5%: n/ksat.fits
Some correlation plots (eg, threshold vs offset, and vs RMS residuals) have been added to the output. Much information previously tabulated by hand added to FITS header:
SIMPLE = T / Written by IDL: Tue Nov 21 21:37:29 2000
BITPIX = -32 /
NAXIS = 2 /
NAXIS1 = 256 /
NAXIS2 = 256 /
BLOCKED = T /
EXTEND = T /
SINGLE = T /T FOR SINGLE NIGHT, F FOR MULTIPLE
RUNDATIM= '2000-11-22T05:30:52' /
RUN_NUM = '0278 ' /
BYYMMDDH= 'H970608N' /
FFLAG = 0 /
YL1 = 18500.0 /
YL2 = 49000.0 /
YQ1 = 18500.0 /
YQ2 = 53000.0 /
MAXTHRSH= 56800.0 /
PQTHRSH = 40000.0 /
MAXRLERR= -0.0100000 /
CT = 10928.6 /
NU1 = 136 /
NU2 = 145 /
NLQOVRLP= 138 /
MDFRMOFF= 11077.3 /
ACMO = 0.0341558 /
BCMO = 119.345 /
A22_MAX = -0.000500000 /
RMS1MAX = 180.000 /
RMS2MAX = 120.000 /
MED_A10 = 8039.66 /
MED_A11 = 26.0288 /
MED_A20 = 6849.06 /
MED_A21 = 28.2462 /
MED_A22 = -0.00120994 /
RMS1_01 = 81.9823 /
RMS1_10 = 93.7347 /
RMS1_50 = 111.359 /
RMS1_90 = 176.729 /
RMS1_99 = 251.243 /
RMS2_01 = 58.6829 /
RMS2_10 = 65.4677 /
RMS2_50 = 76.0571 /
RMS2_90 = 138.667 /
RMS2_99 = 405.357 /
NGOOD_TH= 51209 /
NBADMASK= 1717 /
NBAD_TH = 12615 /
NBADDISC= 9 /
NBADLO = 3 /
NBADHI = 110 /
NBADA22 = 203 /
NBADRMS1= 5790 /
NBADRMS2= 8558 /
FRCGOOD = 0.802347 /
THRSH01 = 47133.2 /
THRSH10 = 48819.8 /
THRSH50 = 50575.0 /
THRSH90 = 52159.2 /
THRSH99 = 53740.3 /
COMMENT IMAGE ARRAY OF PIXEL SATURATION VALUES
END
Items suffixed with %, eg, THRSH01, THRSH10, THRSH50, THRSH90, THRSH99 etc are percentile limits. FRCGOOD is the fraction of passing pixels.
Run: #222 #272
yq2 47.0 52.0
y99 51.2 46.7
(All in 1000's, K)
for upper Q fit limits (=yq2) as shown, for runs 222 and 272. If I assume a linear relationship between yq2 and y99, and I want to set these two equal (ie, 99% of the pixel thresholds are below yq2, the upper Q fit limit) then I get a derivative of about 0.9 K per K, and yq2 = 49K as the right value. This gives me y99 = 49.2K, close enough, though I have to raise RMS1 accept limit to 180 from preferred 120.
Overall:
1. Classify each pixel on pixel-level standards, pass #1 - #5 below.
2. Classify run on run-level standards; pass #1 - #8 below.
3. If run OK by (2), all good pixels by (1) from run classified OK.
4. For cameras seriously affected by offset slope, use F=0, else F=1,
or possibly look at both.
-- result: pixel threshold based on particular night's data.
5. Then given threshold data from several nights, compare and combine.
----------------------------
Run-level criteria:
1. Very small number of threshold failures due to A22>0 or discm < 0; [#TB]
say <100, or preferably < 30.
2. A22 firmly negative, eg, all < -0.00010, say. [#TB]
3. Thresholds all[?] well below saturation limit; eg, all < 47K [#TB]
4. Good fits in RMS plots for both L & Q; say 99% lim < 200 or so. [#TB]
5. Agreement between pixel and array method to within <1K. [#TB]
6. Agreement between F=0 & F=1 method within 1K. [#TB]
7. Histogram of A22 narrow, ie: |med(A22)|/sig(A22) >> 1. [#TB]
8. Weak dependence of threshold histogram on parameter tuning,
especially yl1,yl2 & yq1,yq2, cmo_init.
---------------------------
Pixel-level criteria within a run:
Single run:
1. Pixel not masked.
2. No discriminant failure.
3. Threshold below maxthresh.txt limit.
4. A22 negative, < -0.0005
5. Good L & Q fit RMS: (a) RMS1 for L, < 120; AND (b) RMS2 for Q, < 90.
6. Obtained from a good run, overall.
---------------------------
Multi-run:
1. Reasonable agreement between F=0 & F=1 runs, given offset slope.
2. From reasonable number of separate nights [? TBD]
From the data below and the other data on the runs, one finds that the major questions for the pixel criteria concern the RMS residuals to accept or reject. This complicated because some camera data have big glitches (that can be removed from the data), problems at low light level (can be removed by raising the lower L fit limit) or "frame ripple" which is suppressed by the F=1 analysis if the offset slope is not too large so one does not get a big suprious quadratic term.
For the run-level criteria, problems are mainly in numbers 6, 7, & 8 above, see the data for the individual runs.
Here is a table; it
gives 10%-tile, median, 90%-tile, and 99%-tile values for L & Q fits for F=0 & F=1
runs.
F=0 Linear | F=0 Quadratic | F=1 Linear | F=1 Quadratic
Night 10% 50% 90% 99% 10% 50% 90% 99% 10% 50% 90% 99% 10% 50% 90% 99%
j970608n 110. 132.4 170.5 275. 69.5 116.9 382. -- 89.5 105.8 138. 245.5 44.5 62.9 90.5 298.
j990929n 383. 409.0 437. -- 60. 72.3 89. 292. 83. 100.4 140. ?263. 61. 75.8 98. ?269.
h970608n 122. 142.7 160. -- 91. 99.9 114. -- 52. 59.0 100. 200. 58. 67.5 82. 168.
h980106n 69.5 79.0 113. 361. 63. 67.9 74. -- 61. 69.9 105. 321. 56. 61.8 70. 298.
h990929n 57. 63.8 104. -- 80. 111.1 -- -- 126. 180.7 292. >450. 52. 70.2>500. >500.
k980106n 146. 172.3 206. >292. 113. 123.8 133. 393. 94. 114.7 143. 243. 87. 95.2 107. 387.
k990929n 92. 105.5 119. 448. 61. 71.0 88. >410. 49. 60.6 78. -- 62. 77.1 103. >355.
j980508s 60. 74.8 124. ?230. 50. 56.0 66. 315. 257. 298.5 260. >445. 50. 56.6 70. 347.
j981120s 47. 66.3 118. >220. 50. 59.7 72. 130. 258. 299.3 358. 416. 43. 52.4 63. 147.
j000229s 46. 54.1 92. 127. 50. 61.1 73. 89. 226. 263.4 317. 358. 53. 69.7 92. 114.
j000320s 50. 61.7 108. 148. 51. 61.3 72. 90. 277. 318.4 378. 426. 46. 56.3 70. 90.
h980508s 70. 78.3 90. 330. 52. 56.3 62. 262. 108. 119.0 134. 342. 48. 52.2 58. >238.
h981120s 66. 74.1 84. 141. 41. 48.5 57. 100. 103. 115.7 129. 168. 40. 47.5 57. 91.
h000229s 73. 80.9 90. 110. 53. 61.9 72. 84. 100. 111.9 127. 142. 47. 55.1 64. 78.
k981120s 88. 99.4 116. 176. 43. 48.6 55. 133. 124. 136.3 144. 204. 50. 55.6 62. 136.
k000320s 93. 105.2 122. 170. 44. 50.0 57. 110. 135. 148.1 168. 204. 48. 54.2 62. 108.
10/31/00 Array saturation status
NORTH:
Each camera is different; H may be a problem.
j970608n Have agreement between all 4 modes, but have to tune L & Q limits,
due to offset turn-down at low light level.
j990929n Off-set corrected thresholds agree with j970608n. Further tuning
needed for raw READ2 mode. NB F=1 threshold hist identical to
F=1 for 970608n.
J/N Some further work needed on 990929n, but appears OK with tuning.
______________________________________
h970608n Ripple, major glitches. Offset slope small, but big turndown &
noise. F=1 thresholds appear likely to be OK, though high.
h980106n High ripple and nose, plus high, but consistent thresholds. Offset
has some turn down, very flat but for big glitch.
H990929n New H camera. Very uneven image structure. Ripple, noise, and
rather large slope. High thresholds, but also high hard saturation
level.
H/N Problematic, both for old and new.
______________________________________
k980106n Major ripple and noise, otherwise good offset curve.
k990929n Especially well-behaved offset curve, some low-light roll down.
K/N Looks generally good, both modes, due to very flat offset curve. Some
noise and ripple to deal with.
______________________________________
SOUTH:
All south cameras seem generally similar: substantial offset slope causes
difference between raw READ2 (F=0) and off-set corrected (F=1) modes, but
low-light and noise anomalies permit F=0 mode to be used.
j980508s Low noise, but very large offset slope.
j981120s Low noise, but very large offset slope.
j000229s Small glitch, large offset slope.
j000320s Large offset slope, low noise, very good F=0 fits; some turn-down.
J/S Offset-corrected mode (F=1) grossly disturbed due to large offset slope.
Raw READ2 mode appears OK, and F=1 consistent after slope correction.
______________________________________
h980508s Fairly small offset slope.
h981120s Fairly small offset slope, some ripple.
h000229s Fairly small offset slope, some ripple. Especially good fits.
H/S Moderate offset slope, disturbance of F=1 mode consistent.
______________________________________
k981120s Fairly small offset slope, small threshold difference.
k000320s Fairly small offset slope, small threshold difference. Clear
ripple.
K/S Moderate offset slope, disturbance of F=1 mode consistent.
______________________________________
SUMMARY: Appear to have way clear for generating acceptable thresholds for
all cameras, except for new N/H camera, and possibly old N/H camera. Next
step is to be able to say for each camera how many good thresholds we have,
which should be in 90% to 99% range for all but possibly N/H. Doubtful
pixels thresholds' will be set to safe lower value, eg, 1%-tile, or 10%
-tile lower limit, TBD.
Data {RMS} min max Crossing Low end TH_F0-TH_F1 Comment
j970608n 20.0 180 - 220 (1765,175) (250,180) 42.8 - 41.9 turn down
j990929n 31.0 170 - 220 (1860,205) (280,180) 50.4 - 40.3 turn down
h970608n 19.2 120 - 160 (1850,180) (300,135) 55.9 - 47.0 noise
h980106n 16.8 240 - 240 (1925,240) (720,250) 53.1 - 52.8 little turn down.
h990929n 34.8 0 - 120 (2150,185) (150, 0) 48.5 - 39.9 smooth
k980106n 18.9 150 - 150 (1840,130) (220,140) 45.9 - 46.9 ripple & glitches
k990929n 16.4 150 - 150 (1810,150) (240,150) 45.5 - 46.0 smooth
j980508s 45.7 0 - 205 (2160,280) (200, 25) 44.8 - 37.8
j981120s 47.1 -20 - 180 (2165,275) (120,-20) 44.3 - 37.0
j000229s 47.6 20 - 225 (2260,320) (200, 40) 46.1 - 37.9
j000320s 51.0 -50 - 177 (2020,240) (360, 35) 45.7 - 37.0 slight turn down
h980508s 21.1 0 - 70 (1800, 85) (300, 5) 46.3 - 41.5 curved overall
h981120s 19.8 10 - 80 (1990,130) (100, 10) 47.7 - 41.7
h000229s 21.9 -15 - 80 (2110,130) (130, -5) 49.4 - 41.3
k981120s 23.0 -10 - 70 (1865, 90) (110,-10) 43.3 - 40.7
k000320s 22.8 20 - 110 (2000,140) (100, 20) 42.0 - 39.3
where {RMS} is the median width of the computed offset histogram, min & max are values from the straight part of the offset curve before saturtion, "crossing" is a point on the offset curve estimated by eye, as the intersection of straight line fits to the curve above and below saturation, and "low end" is a similar point on the low end of the offset curve. Thus "crossing" and "low end" can be used together to estimate the slope of the lower straight segment of the curve. Finally, "TH_F0-TH_F1" are the median pixel thresholds from the pixel threshold histogram, for F=0 and F=1.
With these data I can then make a couple of plots, (1) of slope of the offset curve VS RMS:
showing that RMS and slope do indeed correlate as expected, and (2) of [thresholds(F=0) - thresholds(F=1)] vs slope:
where the two marked outliers have been excluded from the fit. Both of these data sets have pathologies mentioned in their respectives notes.
A preliminary attempt to understand the width of the pixel histograms in terms of an intrinsic width convolved with the width of the offsets gave intelligible results only for the south H camera. which has the greatest width. In general, as far as this analysis is concerned, the south cameras have more regular behavior, though they are all afflicted with changing offsets, causing the F=0 and F=1 results to differ.
A recent insight implicit in the above is that the F=1 theshold images and histograms have the detailed effect of the offset removed, and of course it needs to be be restored before the F=1 thresholds could actually be used. In order to make rough comparisons possible, the (READ2-O) data have been adjusted by a constant offset, independent of pixel or frame. The very narrow distribution of A10 (the y-intercept) for F=1 shows that this removal of the offset has been pretty successful. The very smooth images for the F=1 fit co-efficients is further evidence.
As a result of the elimination of this problem, we were able to process data from all six cameras, and including the new H-band array in the north, run with consistent versions of the analysis programs, and the same linear and quadratic fit limits. For the (READ2 - O) form of the analysis, the median values of the threshold by the old (1997) and new (pixel threshold histogram) methods agreed reasonably well. This was reported previously in the 6 September 2000 summary report.
The fit range for the quadratic (Q) and linear (L) fits have been separated so that they can now be set independently. These values are encoded in an 8-digit string labelling several of the standard plots, eg "00431547" gives the limits, (yl1,yl2; yq1,yq2) as 0-43K for L, 15-47K for Q. This turned out to be essential for the pixel method, as the L fit range really needs to be close to the 0- 43K range used in the pipeline processing, whereas the Q range needs to be a good proxy for the data in the soft threshold region.
Histograms of the computed offset O and its RMS variation have been added, along with diagnostic plots of absolute residuals. Plots of frame median offset versus x for each frame have been improved.