THE EFFECT OF UNDERSAMPLING
Since the PSF variance map has an effect on the relative weighting of pixel
values, particularly for the strongest sources, result number (2) suggests
that a magnitude-dependent trend could result from a systematic distortion of
the PSFs. One possible distortion is the presence of small positive and
negative lobes in the form of rings around the estimated PSFs. These
lobes are evident to some extent in all of the estimated PSFs, and are
probably artifacts of the PSF generation process. The most obvious place
that such "ringing" artifacts could arise is in the interpolation step
in PSFMAKE.
The interpolation step in PSFMAKE is necessary because the PSFs are first
estimated on a relatively coarse grid (whose spacing is 0.5 focal-plane pixels)
and then sinc-interpolated onto the fine grid used in PROPHOT. The coarse
grid is necessary because of constraints on computation time, but unfortunately
violates the sampling rule with respect to the Nyquist criterion. Therefore,
the subsequent sinc-interpolation step is, strictly speaking, invalid.
The problem can, however, be alleviated by the use of a finer sampling
interval during the estimation process.
Figure 4 shows profiles of PSFs generated using grids of two different
sampling intervals, corresponding to interpolation factors of 2 and 4 with
respect to a focal-plane pixel. The data were from scan 47 of 990928n.
Figure 4: E-W slices through PSFs estimated from scan 47 of 990928n,
using interpolation intervals of 2 (upper plot) and 4 (lower plot).
___________________________________________________________________________
It is apparent that the anomalous ringing effects resulting from sinc
interpolation are reduced in the case of the more finely sampled PSF.
Based on these considerations, we have compared the photometry
performance of PSFs generated with two different interpolation intervals.
Applying the "in situ" PSFs from scan 47 at J-band to data from the same scan,
we obtain the following results, for an interpolation interval of 2 (Figure 5)
and 4 (Figure 6).
Figure 5: Photometry results for scan 47 of 990928n, based on an "in situ"
PSF estimate using an interpolation factor of 2. The plots show delta-mag
and reduced chi squared versus magnitude.
___________________________________________________________________________
Figure 6: Same as Figure 5, except with an interpolation interval of 4.
___________________________________________________________________________
It can be seen that the use of a finer interpolation interval has greatly
suppressed the delta-mag trend, supporting the hypothesis that the anomalous
trend is due to a PSF distortion resulting from the sinc-interpolation of
an undersampled function.
Encouraged by the above result, we generated a new subset of production PSFs
using a finer interpolation interval, and used these PSFs (after combination
in the same way as earlier) to do photometry on scan 84 of 990928n. The
results are presented in Figure 7.
Figure 7: Plots of delta-mag and reduced chi squared for scan 84 of 990928n,
using a new set of production PSFs, generated using a finer grid than before
(interpolation factor = 4) and combining sets of PSFs in each shape bin.
___________________________________________________________________________
Unfortunately, these results show that although the delta-mag slope has been
reduced, the dispersion problem persists, and the reduced chi squared is
anomalously low for the bright sources.
Repeating the solution using a smeared variance map gives the result in
the bottom plot of the next figure. Also shown, for comparison, are the
previously-obtained results for the old production PSFs and the new production
PSFs generated with a coarser grid (interpolation factor = 2).
Figure 8: Delta-magnitude plots for scan 84 of 990928n, as follows:
(1) Old production PSFs
(2) New production PSFs with interpolation interval of 2
(3) New production PSFs with interpolation interval of 4, and a smoothed
variance map (smoothed with a Gaussian of FWHM = 1 pixel).
___________________________________________________________________________
DISCUSSION
The anomalous trend in delta-mag appears to be due to PSF distortion
associated with the sinc interpolation of an undersampled function.
Such a trend results from the fact that the relative weighting of the
inner and outer parts of the PSF is magnitude-dependent; a centrally-peaked
variance map results in less weight for the central pixels in the case of
a strong source, whereas for a weak source, the pixels are uniformly weighted
since the noise is then dominated by the Poisson noise of the background
rather than PSF error.
We speculate that the reason that the delta-mag trend is not present for
pre-collimation PSFs is that the latter have less high-spatial-frequency
content and are thus less susceptible to undersampling errors.
With regard to the increased dispersion in delta-mag, the bottom
plot of Figure 8 shows that this problem has been alleviated
by the use of a smeared variance map. While this is encouraging, we still
need to understand the fundamental reason for this behavior. We suspect that
it is due to an erroneously high central peak in the variance map which
results in most of the weighting going to pixels in the wings of the PSF,
whose values are particularly sensitive to seeing fluctuations.
WHAT NEXT?
It appears that the remaining problems are related to the variance map,
and we will therefore perform diagnostics based on the residuals from the
profile fits, starting with the particularly bad case of scan 14 discussed
above.
In addition, we will test the hypothesis concerning the difference
in spatial frequency content by looking for a delta-magnitude trend in
the southern data (which apparently have produced relatively sharp PSFs
to begin with), and verifying that the combination of new PSFs with new
PROPHOT produces results which are at least as good as (and hopefully an
improvement on) the old.
UPDATES
Jan 20:
The use of a different approximation in binning the residuals results in
an improvement of the scatter in delta-magnitude for scan 14, as shown in
the J-band plot,
made using an interpolation factor of 2. A comparison with Figure 3 above
shows that the scatter has been reduced.
Repeating the analysis using a PSF made using an interpolation of 4 gives
the result shown here, whereby the
use of a finer grid in the PSF generation process has mysteriously resulted
in more scatter.
Jan 21:
Experiments with varying the size of the bins used for PSF variance estimation
show that the scatter of fitted magnitudes depends quite strongly on this
parameter.
A spectacular improvement for scan 14 at J-band was obtained using bins of
width 1.25 pixels, as shown in
this plot. Note that this is
significantly better than the
corresponding plot for the OLD
production psfs and OLD prophot.
It may be relevant that the above binsize of 1.25 pixels is comparable to
the FWHM of the PSF for this scan (1.17 pixels). Perhaps the optimal
weighting function for binning the residuals is the PSF itself, or more
likely its autocorrelation, since it is playing the role of a covariance
function.
Jan 26:
I've now tried a series of different bin sizes, for scan 14 (low density,
so-so seeing), and scan 47 (moderate density, good seeing), and have made
a plot of the magnitude scatter (specifically, the standard deviation of
delta-magnitude for sources of magnitude 11 and brighter) versus the width
of the psf-residuals bin in units of focal-plane pixels. You can find the
results here. The symbols in this plot are:
diamond = J
+ = H
* = K
Feb 3:
Using a bin size of 1.2 pixels, the psfs appropriate to 990928n/s084 were
generated (26 scans, which get combined into 4 psfs for the relevant
ranges of shape factor). The pixphot results for
j-band can be compared with the
old results (old prophot, old psfmake).
Feb 11:
Repeating the above exercise for bin sizes of 0.8 and 0.5 pixels gave
successively better results. As a metric, we compare the values of
sigmag (defined as the standard deviation of delta-mag for mag<11) for
the 3 bin sizes:
binsize = 0.5 => sigmag = 0.018
binsize = 0.8 => sigmag = 0.019
binsize = 1.2 => sigmag = 0.025
These results show a different behavior than those for in-situ psfs, in that
sigmag is still getting smaller as the bin size is reduced to 0.5 pixels,
rather than having a minimum at a bin size of about 1.2 pixels. This suggests
that much of the behavior of the in-situ psfs is related to the fact that no
allowance was made for seeing variations -- each scan used a one-size-fits-all
psf.
Feb 22:
Production-type PSFs have been generated for 4 different bin sizes (0.25,
0.5, 0.8, and 1.2 pixels). Subsequently, pixphot has been run on 4 separate
scans on the night of 990928n (scans 31, 49, 84, 89), using both single and
combined PSFs for each shape-factor increment. The results are summarized
in this table, in which the metric for photometric
accuracy is the standard deviation of delta-mag for the magnitude range
J < 11, corresponding to the range for which PSF error dominates. Also
listed is the average value of reduced chi squared for that magnitude range.
The results show:
(1) Combined psfs do better than single psfs
(2) The optimal value for the bin size is 0.5 pixels
March 15:
One remaining problem, illustrated by the results for
scan 84, is that the reduced chi squared is
somewhat low for the brightest sources. A set of these plots was generated
for the pixphot runs of all 4 scans (31, 49, 84, 89) using both single PSFs
and combined PSFs. The results showed clearly that the low chi squared problem
occurred only for the combined PSFs, i.e., the problem is connected with the
PSF combination process.
As a diagnostic, pixphot was also run using
single PSFs + combined variance maps and the converse. The results are
summarized in this table, which gives the
mean value of reduced chi squared for the magnitude range J < 9 for each
combination of PSF + variance map.
Some conclusions from this table are:
(1) For both types of variance map (single and combined), combined PSFs
fit the data better than single PSFs. This should be the case, and shows
that the PSF combiner is doing its job.
(2) For single PSFs, the combined variance maps give lower chi squared values
than for the single variance maps. This casts suspicion on the variance-map
stacking process, as discussed below.
As discussed in the PSF combination memo, there are
two steps in the calculation of the combined variance map:
(a) Do a weighted average of the individual variance maps
(b) Apply a correction to account for the fact that the combined PSFs are
more accurate than single PSFs, and therefore the variance should be reduced
somewhat.
Conclusion (2) above implies that step (a) is being done incorrectly.
Otherwise, the reverse would be true, for the following reason: Since the
effect of step (b) is to reduce the weighted-average variance (which would
have been representative of a single scan), the SC values should actually be
larger than the SS values, contrary to the results in the table.
A comparison of the combined variance maps with the sets of individual
variance maps showed no obvious problems. The weighted-average variance maps
look to be quite representative of individual cases, so it is a mystery why
they don't perform as such when used by pixphot.
March 16
Some more tests to try to shed some light on the low-chi-squared problem:
(1) A check was made to see if, for each seeing bin, the variance map
corresponding to the selected "single" PSF was representative of the set
of variance maps in that bin. For most seeing bins, this was found to
be true, with the one exception being 09723. In the latter case, the
variance of the selected PSF was unrepresentatively low. An alternative
interpretation is that there were two variance maps which could be regarded as
being abnormally high. Either way, since this PSF was used by pixphot
in 3 of the 4 scans, there is the possibility that abnormalities in this
one particular combined PSF could lead to a systematic bias.
So a test was made in which the two highest-variance
PSFs were eliminated from the combination which went into 09723. However,
running pixphot on this modified set of PSFs produced essentially the
same results as before.
(2) To test the possiblity that there could be something wrong with the
weighting used in the stacking of the variance maps, an alternative
weighting scheme was tried. In the alternate scheme, each variance map
received the same weight over all of its pixels (as opposed to the "correct"
way of doing it pixel by pixel). However, no appreciable difference resulted
in the pixphot output.
March 27
Last week, Bill Wheaton drew my attention to the fact that in the new
psf production runs, it appears that bright stars have lower chi squared
(sound familiar?). I've now had a look at several of the summary files,
and have plotted the results in the form of delta-mag and chi-squared as
a function of fitted magnitude, in the same way as for prophot. I've
concluded that a chisq v. magnitude trend is a general feature of the
J-band psfmake output,
with 990918n/j061 being a
particularly bad example. The effect does not seem to be present at
H and K; there are also some J-band scans (for example
991009n/j145) in which it is
weak or absent.
THIS INDICATES THAT THE PROBLEM IS IN PSFMAKE AFTER ALL. The reason that
this had eluded us is due to the fact that the prophot output does not
show the trend, probably because it is buried in the greater dispersion
resulting from the varying seeing conditions in the long science scans.
The fact that the chi squared was lower for combined psfs than for
single psfs (see March 15 above) must have been due to random effects in the
"single psf" selection process, which then had a systematic influence on
the results for the 4 scans considered, since each of those 4 scans had
those psfs in common. This is substantiated by a close look at the individual
psfs that went into those tests.
