About the Benchmark
Introduction
In our laboratory we have found it useful to benchmark area detectors by
comparing peak heights from native anomalous difference Patterson maps. These
peaks are faithfully calculated only if the anomalous differences are accurately
measured, and since these measurements typically require the determination of a
small difference between two larger numbers, the test is both challenging and
sensitive. Our standard test is conducted by collecting native data from
crystals of P6 Myoglobin and processing it with Bijvoet pairs kept separate.
After this, peak heights calculated from an anomalous difference Patterson map,
generated using PROTEIN, are used as an indicator of area detector
performance.
Keep in mind that detector results are affected by data collection stategy,
crystal quality, processing software, and collection hardware. As a result, we
encourage the use of this test as a benchmark for any given detector, but we
discourage comparisions between detectors based solely on this one test. It is
our hope that people will find this site useful for comparing the quality of the
data their equipment provides them with data from other labs posessing similar
equipment.
Benchmark Does Not Always Equal Best
It is important to note that the benchmark is intended to meet the requirements
of a benchmark - namely, it's easy to perform and provides useful results. It is
intended to provide a good comparison of detector performance, however it is NOT
necesssarily intended to provide the best method for collection of anomalous
data. For instance, in our lab we use the Enraf-Nonius FAST area detector, and
while we have found that collecting contiguous wedges of true Friedel mates
off-axis provides us the best anomalous data, we have not choosen this method for
the benchmark for two reasons. First, setup time to perform this data collection
is longer and a benchmark should be easy to perform. And second, if the method
of data collection provides data with extremely high redundancy then benchmark
may not be sensitive enough to the kinds of errors we hope to detect. For these
reasons we rountinely collect
120 degrees with c* vertical of
P6 Myoglobin data as a benchmark for our FAST.
It provides good results, is easy to perform, but it does not produce the best
maps we have seen from our system. For comparision, however, we have included
information on both of these benchmarks.
Other Detectors
The benchmark for each detector should be arrived at similarly. We will leave it
up to the user community for each detector to determine the most useful measure
of performance for their individual detectors. We intend this site to be a
repository for these community determined benchmarks.
At this site we will retain benchmark results for each of the various area
detectors, displaying maps having the highest peaks and least noise. We also
keep records of the top five other collections. Additionally, if multiple groups
collect data using the same collection strategy, we will maintain an average for
each of the three peaks (the two doubly weighted peaks and the singly weighted
peak) for all six data sets for each detector.
We hope that with this information owners of area detectors will be able to
gauge their performance relative to others with the same detectors. Please keep
in mind that this is not intended to be a contest to see who can collect the best
data, but rather a tool to help everyone achieve better results. Also bear in
mind that comparisions between detectors is not the intention of this site. In
fact, comparisions between different detectors are mostly useless because the
benchmarking method will differ from detector to detector.
Return to Area Detector Benchmarking Home Page.