2MASS Tape Operations

2MASS Tape Operations

Roc Cutri - IPAC

Revised 14 August 1996


In an earlier memo projecting the growth rate of the 2MASS raw and processed
data volume, an estimate was made for the total number of tapes required
during the survey.  Based on nominal photometric time percentages of 30% in
the north and 50% in the south, a simple baseline tape operations plan requires
a total of ~4700 DLT 4000 Type IV tapes for the survey.  At the current unit 
cost of DLT Tape IV of ~$90-$100, the total tape budget would approach a half 
million dollars.  The hardware budget for 2MASS data processing will not 
support this amount.

In this memo, I propose a tape operations plan that significantly reduces
the survey tape costs, while maintaining minimum standards for raw data
integrity and archiving.  Additional costs associated with this plan do 
include increased coding time and some hardware to be installed at the 
observatories and IPAC.


The baseline operations plan calls for data to be taken on all possible
nights.  A conservative estimate is that the number of observable nights
will be approximately twice the number of photometric nights.  Two copies of
each night's data are written, one per DLT 4000 Tape IV (20GB native
format), and one is shipped to UMASS and one to IPAC.  At IPAC, nightly 
quality assessment would take place during processing, and the final decision 
to accept or reject a night is based on this assessment.  Over the life 
of the survey, this implies that data tapes would be written and processed 
for approximately 1875 telescope nights (2 x 46 months x 0.3 phot. = 
840 nights north, and 2 x 34 months x 0.5 phot. = 1035 nights south).  Thus,
nearly 3700 DLT tapes will be required to store just the raw data archive.

The baseline plan calls for one night's processed data to be written to one 
archive DLT tape.  Thus, if there are ~940 photometric telescope nights during 
the survey, the total processed data tape archive will require the same number 
of tapes.


1. Observatory Tape Writes

Under my new observatory tape operations plan, data would still be 
acquired on every possible night.  However, a minimal version of the 2MAPPS
pipeline would be installed on mountain computers, and the calibration scans
from each night would be processed on-site and a basic assessment of the
photometric quality of the night made immediately following a nights 
operations.  Only raw data for nights that exhibit at least 4 consecutive 
hours (TBD) of photometric conditions would be written to DLT
tape.  Two uncompressed copies of the raw data tapes will be written;
one copy will be shipped to IPAC and one will remain on the mountain.

This will reduce the estimated number of nights for raw data tape writes
by approximately a factor of two, to ~940.  It also reduces the number of 
potentially bad nights that will have to be processed thereby improving the
capability of the pipeline processing to keep up with the survey pace.

2. Raw Data Archiving at IPAC

Within 2 weeks of receiving a raw data tape from the observatories, the
tape will be read and verified for content and integrity at IPAC, and the 
data written to disk.  Once on disk, the data will be "pre-conditioned" using 
the algorithm devised by Gene Kopan and Nian-Ming Chiu (see Appendix I).  
The pre-conditioning consists of simply evaluating the differences between 
successive frames in each scan.  The differencing reduces the entropy of the 
data, and the resulting frames will realize much larger lossless compression 
ratios using simple LZ algorithms, such as those employed by unix compress, 
gnu gzip, or most tape hardware compression chips.  The first raw frame and 
following 259 differenced frames and all ancillary text files from each scan 
will be compressed, and written to tape using standard tar formats.

As described in Appendix I, this method generally achieves lossless compression 
ratios of 2:1.  Therefore, on average two nights raw data can be archived onto 
each DLT 4000 Tape IV.  Efficient tape "packing" will further reduce the number
of raw data archive tapes.  The general rules for "packing" the raw data 
archive tapes are:

   1) Nights will be written sequentially in date order on each tape.
   2) Nights will not be split between two tapes.
   3) Northern and Southern raw data will not be mixed on archive tapes.

One copy of each raw data archive tape will be sent to UMASS and one will 
be stored on-site at IPAC.

Once an observatory tape has been read and verified, and the two archive copies
have been written and verified, the original observatory tape will be returned 
to the mountain.  Upon receipt of the returned original tape at the 
observatory, both that tape, and the second stored copy will be freed for 
recycling for raw data.  

If we conservatively estimate that the turn-around from data acquisition
to return of tapes to the observatory will be ~4 weeks, then there must
be a rotating stock of approximately 110 tapes per observatory.  Over the life
of the survey, each of these tapes will be cycled 6-8 times.

3. Processed Data Archiving

By far the largest component of the processed data archive will be the
Atlas Images.  Over the life of the survey, the image archive will grow
to ~11.5 TB in size, while the source database will be at most ~0.2TB.
Therefore, efficient packing of the image data will result in the greatest
savings in media costs.  

One night's processed data occupies only ~70% of the volume of the 
raw data, or an average of 12.7GB.  With only a very samll amount of 
compression at least two nights processed data can be fit on each archive 
DLT tape.  The coadd 1995 ProtoCam coadd images currently on-line on
lugosi and karloff are actually compressed versions of the original images
that have been pre-conditioned by binning using the FITS bzero, bscale
capability, and compressed using standard unix "compress."  This is not 
strictly a lossless procedure, but the loss can be limited to a very small, 
low range of bits through judicious selection of bscale.  The coadd ProtoCam  
images on disk now have realized a factor of ~4 compression.

I assume that a factor of 2 compression can be achieved for the survey
Atlas Images after processing, either through the bscale or some other
pre-conditioning scheme.  If that is the case, then on average three nights
processed data can be written to one DLT 4000 tape using the same
packing rules as for the raw data archive.


a. Tape Cost - The current unit cost for DLT 4000 Tape IV has been negotiated
at $87.  Tape vendors have estimated the price to drop by 10-15% over the
next year, according to reports from their suppliers.  Longer term price
profiles are not available.  It is recommended that tapes be purchased in
minimum volume to take full advantage to any future cost reductions unless
a substantial savings can be obtained through large volume purchase.

b. Tape Numbers -

   i. Rotating Telescope Stock:  110 tapes per obs.         220 tapes

   ii. Raw Data Archive :        0.5 DLT/night x 937.5 =    469

   iii. Processed Data Archive:  0.33 DLT/night x 937.5 =   309

   iv. Miscellaneous:                                       100

       TOTAL                                               1098 tapes

c. Cost -

   1098 tapes @ $87/tape (current cost)           $95.5k
	      @ $78/tape (1997 cost)              $86.0k
	      @ $67/tape (avg. for 10% decline
			   per year)              $73.9k


To realize the tape cost savings outline above, there will be compensatory
costs in hardware and software.  Before any decision is may to adopt
all or part of this proposal, we must decide if the potential savings
exceed these new costs.

- The proposed plan requires the installation of a workstation running 
a truncated version of the pipeline at each observatory.  The cost of the 
workstations and installation and maintenance of the pipelines must be 

- A premium will be placed on tape I/O at IPAC.  In order to avoid critical
impact on the processing pipeline, the tape I/O should probably be handled
by separate computers (one for the north, one for the south) that are 
cross-mounted to the production cluster disks.  The role of these computers
would be to read raw mountain DLT tapes, pre-condition and compress the data, 
and write the two raw data archive tapes for each input tape.  These Tape 
Control computers would also be responsible for reading the raw data archive
tapes when processing is to be carried out, and if reprocessing is necessary.
Archiving the processed data to tape may also be handled by these machines.
Depending on architecture and system cost, these machines may provide some 
backup on processing power.  Alternatively, one of these machines, if of
sufficient capacity, could act as the fileserver for the 2MASS sub-net.

- A minimum of 2 DLT drives per production machine will be necessary to handle
the increased I/O function.  A strong case may be made for 3 drives.

- The TAPELOAD subsystem would have to undergo significant revision to handle
input from the raw data archive tapes.

- A new DLTLOAD process would have to be written.  This subsystem would have
the responsibility of doing nothing by reading, writing and verifying tapes, 
and providing some quality measures for feedback to the observatory.

APPENDIX I. -  Gene Kopan and Nian-Ming Chiu's Memo on 2MASS Raw Data 

26 June 1995

	Preliminary results are in on compression, thanks to Nian-Ming
and Dave Gregorich.

	2mass raw data does not compress significantly, however, with
simple data pre-conditioning, compression ratios of 2-2.5 are easily
achieved with the standard unix compress program or the gnu gzip program.
These programs use variants of  Lempel-Ziv coding which has the dual
advantages of speed and single pass operation. In addition, the code
is available for the gnu gzip program and it has been ported to many
architectures including Intel/DOS/Linux. 

	Nian-Ming Chiu has implemented a pre-conditioning/ compression
engine under Solaris which has demonstrated the feasibility. He has 
written a program which pre-conditions the data, taking advantage of
the frame-pair periodicity to reduce the entropy prior to compression.
The method is to simply replace the raw data with the frame pair
differences for all frame pairs after the first. This operation is
quite fast, and allows typical compression ratios of 2-2.5.

	This technique could be used to compress the data on-site and
reduce the time taken to write the data to tape, and the number of
tapes. If the compression can be done while the data is written to
tape, the total time might be reduced. The time to read the data at
IPAC will be reduced by a factor of 2-2.5, and the raw data could be
left in the compressed form until used, reducing io and saving disk
space. The final decision may depend on the practicality at the
observatory. The major gains would be reducing the tape inventory and
io time at IPAC. With a 90Mhz Pentium, it will be tricky saving time
at the observatory.

	The following table summarizes compression results using data pre-
conditioning on a 50Mhz sparc (karloff) and a 90Mhz pentium. The
pentium numbers were provided by Dave Gregorich.

   machine    compression    file size   %    cpu time   throughput time
                 none        68157440   100 
  sparc(50Mhz)	 compress    57335171    84                3:22 / 1:48
  sparc(50Mhz)	 tzip -g     30961812    45                3:36 / 1:31
  sparc(50Mhz)	 tzip -u     27473874    40    156 / 72	   2:39 / 1:43
  sparc(50Mhz)	 tzip -1u    29215162    43                3:02 / 1:55
  sparc(50Mhz) 	 tzip -1g    31715902    46                3:49 / 1:29
  pentium(90)    compress*   27473874    40    156 / 63    4:01 / 2:34
  pentium(90PCI) compress*   27473874    40    155 / 62    3:16 / 1:32

	time required to write 68MB > 8mm tape              ~ 2:20

    Notes:  - tzip is Nian's combined pre-conditioning and compression
	      with several options (see attachment).
	    - the numbers given are for compression / decompression
	    - the pentium tests were preformed on pre-conditioned data
	      and under Linux.
	    - the pentium(90PCI) run used an IDE drive attached to the
	      PCI bus.
	    - the data pre-conditioning takes just a few seconds of cpu

	This process would be more attractive with a 133Mhz or dual processor
Pentium at the site. 


	I attach Nian's info below:


Here is what I have so far.

2mass scan data compression program.
tzip [-[1|2] -[u|g]] input_file output_file
tunzip [-[1|2] -[u|g]] input_file output_file


      tzip 2mass_data 2mass_data.Z
      tunzip 2mass_data.Z 2mass_data

  Options	compress	uncompress
		program		program
	1	sub1		add1
	2	sub		add
	u	compress	uncompress
	g	gzip -1		gunzip

sub (sub1) is a simple filter to preprocess a data file before compression.
It can increase compression for data whose points tend to be close to the last
point.  The output is the difference of successive bytes of the input.
The add filter is used to undo what sub does.

   sub method:
   C1 = B1 - B0
   C2 = B2 - B1
   Cn = Bn - Bn-1

   sub1 method:
   C1 = B1
   C2 = B2 - B1
   C3 = B3 - B1
   Cn = Bn - B1


      sub < 2mass.data | gzip -1 > 2mass.data.gz
      sub1 < 2mass.data | gzip -1 > 2mass.data1.gz

add (add1) is a simple filter to postprocess a data file after decompression.
It can increase compression for data whose points tend to be close to the last
point.  The output is the sum of successive bytes of the input.

   add method:
   B1 = C1 + C0
   B2 = C2 + C1
   Bn = Cn + Cn-1

   add1 method:
   B1 = C1
   B2 = C2 + C1
   B3 = C3 + C1
   Bn = Cn + C1

To expand, use the reverse operation, as in:

      gunzip < 2mass.data.gz | add > 2mass.data
      gunzip < 2mass.data1.gz | add1 > 2mass.data

Test tzip
Machine:		karloff
uncompress data:	/k9/gene/data/sa57ah3
size:			68157440
%:			100.0
compress time:		N/A
uncompress time:	N/A

command:		compress /k9/gene/data/sa57ah3
compress data:		/k9/gene/data/sa57ah3.Z
size:			57335171
%:			84.1
compress time:		3:22
uncompress time:	1:48

command:		tzip -g 
compress data:		/scr/data/sa57ah3.gz
size:			30961812
%:			45.4
compress time:		3:36
uncompress time:	1:31

command:		tzip -u 
compress data:		/scr/data/sa57ah3.z
size:			27473874
%:			40.3
compress time:		2:39
uncompress time:	1:43

command:		tzip -1u
compress data:		/scr/data/sa57ah3.1z
size:			29215162
%:			42.8
compress time:		3:02
uncompress time:	1:55

command:		tzip -1g
compress data:		/scr/data/sa57ah3.1g
size:			31715902
%:			46.5
compress time:		3:49
uncompress time:	1:29

Nian-Ming Chiu			Phone:          595
                                E-Mail:         nmc
				office:		149