Here is a set of procedures to perform when analyzing data from an ops test. This list will expand. Steps 0 and 1 of these procedures apply starting with the June 23&26 ops tests, but the other steps can be applied to past tests post facto, if one wishes to practice. Note that whenever I say "make a note" or something like that, I'm refering to on-line notes that can be emailed, printed, examined by others over the net, etc. Perhaps a good way to do this is for an examiner to edit into a FILE called NOTES in either (or both) of the raw or pipeline-ready directories for each examined segment. I don't know yet exactly how to organize who gets what OBS's. We'll just have to parcel them out and keep in contact with one another. 0) Using a DB query report, determine what observations are expected in the current delivery. Follow up any expected OBS's that do not show up in the examination outlined below. Make a note. I don't know exactly how to do this. We'll have to check with Yi Mei. 1) FOR EVERY OBSID expected, start at the top. Look in the executive directory /proj/wire/soda/da/acts/NAME/exec, where NAME is the ID tag given to this ops test (e.g. 'tvac.1.3'). There will be numerous symbolic links of this form: OOOOOOOO.TYPE where OOOOOOOO = a hex OBSID. TYPE = 'tlm' for CCSDS format telemetry files of image data. Written by WATCH. 'raw' for the raw, unpacked FITS images. Written by INGEST. 'pipe' for the pipeline-ready directory. Written by PIPEPREP. Each symbolic link points to the appropriate data for that TYPE: The 'tlm' link is to the actual telemetry file (not a directory; don't try to 'cd' to it), the 'raw' and 'pipe' links are to the directories containing the data. For 'raw' and 'pipe' you can look at the linked directory with ls OOOOOOOO.TYPE or 'cd' to it with cd OOOOOOOO.TYPE But remember, 'cd ..' will NOT return you to the executive directory. In a directory listing for this directory all TYPEs for the various OBSID's will be sorted together, thus you can check on the processing state for each OBS pretty quickly. If only 'tlm' exists, that OBS hasn't been unpacked yet (INGEST hasn't run). If no 'pipe' link exists, then PIPEPREP hasn't completed. If no 'pipe' entry exists, but there are entries for later (higher number) OBSIDs, IT PROBABLY MEANS TROUBLE! Make a note. Read on ... 2) THE FACT A LINK EXISTS DOES NOT MEAN THE DATA ARE GOOD, only that the process that creates that data has run. So, once you've scoped out the OBS you're interested in in the executive directory, START CHECKING ERROR LOGS! For the 'tlm' data, probably nothing will be wrong, but scope out the watch log file in /proj/wire/soda/da/acts/NAME/logs. For the 'raw' and 'pipe' directories, look at Ingest.log in either the 'raw' or 'pipe' directories and look for the OBSID of interest. It is possible the OBSID won't be printed out with an error message, either due to oversight on my part or because the OBSID is simply not known when the error occured. Thus you should keep a sharp eye for errors that occur anywhere near where the processing for your OBS was happening. If it turns out there was an error that could not be pinned down to a particular OBS, IT IS IMPORTANT TO LET ME KNOW! It is even MORE iMPORTANT to tell me if you discover an error THAT LEFT NO TRACE. THIS IS VERY BAD! NOTE: DO NOT ACCIDENTALLY MODIFY THE LOG FILES!!! They are our trail of breadcrumbs leading us to the source of errors we badly need to find. Without the log files, we're lost. In the future we will find a way to protect them, but for now, you must simply be very careful. Make a note of any errors or omissions you find. Continue with this list; do not give up once you find an error or warning message somewhere. There may be more problems to find. 3) Make an inspection of the contents of the 'raw' and 'pipe' directories and make sure everything looks OK. Make a note of any anomalies. Read the WHAT file and make sure the contents make sense and jive with the directory contents. Look at the BROKEN file, if there is one. You should be able to predict any entries in the BROKEN file since you've already seen the error messages in Ingest.log. Right? 4) Run stattest and make greyscale plots of the raw data. If there's a 'pipe' directory, do it there. If not, use the 'raw' directory. Examine the frames ON THE SCREEN in skyview or another flexible image viewer using multiple stretches. Hardcopy plots can hide subtleties; don't trust them. Run wirelist and examine and save the output. See the Appendix below for brief instructions on running these programs. Remember, you're not simply trying to check that everything went OK, YOU"RE LOOKING FOR TROUBLE. Be aggressive. 5) Run the pipeline if there's a 'pipe' directory. With new data the pipeline should be run using the script called 'RUN' in the 'pipe' directory. If you need to modify the arguments to the pipeline to make it work better on a segment, do this by editing the 'RUN' script. If the segment pipeline ever needs to be rerun, one should expect to do it simply by typing 'RUN'. CAUTION: Later, when running pipemseg on a collection of segments for an area, you should use the '-nop' to suppress segment-by-segment pipelines to preserve the original, by hand processing you did by customizing the 'RUN' script. You should try to run the pipeline even in segment flagged as broken in one way or another. Customize as necessary to get the pipeline to do SOMETHING. There may be more trouble to find which will only become evident by running the pipeline. Before to long the automated pipeline will run and you will be able to skip to step 6. 6) Run stattest on the rectified frames and all other pipeline products (including the median BG, the flat, the coadd, the cleaned coadd, the coadd coverage file, etc.). Examine these images in skyview. LOOK AT THE LOG FILE carefully, looking for error messages and error codes. Make notes and share them. 7) If the pipeline failed and you think you can improve it, rerun the pipeline in private, scratch disk space somewhere until you're happy with the results. Then edit the 'RUN' script to reflect the changes that must be made to process the data. Rerun the pipline for real in the 'pipe' directory using only the 'RUN' script. If necessary, return to step 6. 8) Prepare some sort of final statement regarding the status of each segment you've examined, preparing to write one of four states into the OBS DB: a) Accept, do not retry the OBS. b) Accept, but retry the OBS anyway (????). c) Reject the OBS, do not retry. d) Reject, retry the OBS. 9) Once your recommendation is confirmed by the PI, run the necessary program to update the OBS DB appropriately. 10) Go get more OBS's to look at. Appendix ======== ........ Running stattest: (It is recommended to start a figdisp window in the background: "pgdisp &" .) After "cd-ing" into the subdirectory containing the FITS files for a given segment, run "stattest (FITS files) -d /xdisp -ss". For example, if you are in a "tlm" directory, run "stattest os*.fits -d /xdisp -ss" to look at the 12 micron band images. (Replace "/xdisp" wit "yourfile.ps" to get a PostScript plot file.) You will get a histogram plot for each image file. The raw data often shows two peaks with a longer tail to lower values. Look for unusual patterns in the median and RMS as a function of row and column. Also look for strong and unusual peaks in the the structure function and fourier transform plots. On the screen you will get a tables of values and possible warning messages. ......... Running wirelist: Execute "wirelist -help" for syntax and a brief help. To list a telemetry data set, cd into the "tlm" directory containing the day-number-delivery subdirectories (usually each subdirectory should be a 3-digit number). Go: "wirelist ###" to get a count listing of all the OBS_* directories. For example, for the ops8.12 test, you would go "cd /wire/pit/tlm/ops8.12/ " and then "wirelist 201" to list the day# 201 segments. Look for warning messages. You can make bigger listings using the "-list" and "-biglist" options. You can list a single subdirectory by cd-ing into the segment directory and going: "wirelist 0". To list a pipeout data set, cd into the directory containing the area ID subdirectories. For example, to list area 3008 in ops8.12 you would go: "cd /wire/pit/pipeout/ops8.12/lz/surv" and then "wirelist 3008". Big IPAC Table: To get a big listing IPAC table, use "wirelist # -biglist -nocheck of=(file)". You can use this file and "tblplot" to plot median level, dither amplitude, etc. versus frame number. ......... Running grayim: If you see something funny in stattest or wirelist you can get a quick look at the images using "grayim". Just go "grayim *.fits" in the segment directory. .........