James Holton's Benchmarks...

Comparison of x-ray programs on different machines

The following tables are a comparison of standard x-ray data-processing jobs run on various computers I have access to. The machine names have been omitted to protect the innocent. All these jobs were run as the only "major" process on the machine, and always from a local disk. The exact same data directory was tar-ed up and moved from machine to mahcine.

---

Speed of DIALS

The following are the results of processing a 3600-image data set of lysozyme from a Pilatus-sized detector in DIALS. By using simulated images, the download size of the test data is only 10 MB. Everything is downloaded and decompressed automatically using this script dials_bench.com. The timings of each step: import, findspots, index, refine, integrate, export are for all 3600 images. I strongly encourage anyone out there with a new machine running Linux or Mac OSX to give this a try. This list shows only the best result from each machine. The full list of all timings is available in this file. I think all us crystallographers could really use a standard reference for the "speed of DIALS" since it seems to be very difficult (for me at least) to predict how any particular system is going to perform.

mkdir -p /dev/shm/dials_bench/
cd /dev/shm/dials_bench/
wget http://bl831.als.lbl.gov/~jamesh/benchmarks/dials/dials_bench.com
chmod a+x ./dials_bench.com
./dials_bench.com

---

Speed of XDS

The following are the results of processing a 3600-image data set of lysozyme from a Pilatus-sized detector in XDS. By using simulated images, the download size of the test data is only 10 MB. Everything is downloaded and decompressed automatically using this script xds_bench.com. Or, if your machine is off-internet, all the files you need are in this tarball xds_bench.tgz. The BACKGROUND_RANGE is set to 50 images, but all other aspects of the INIT COLSPOT IDXREF INTEGRATE and CORRECT steps are using all 3600 images. Note that the optimal MAXIMUM_NUMBER_OF_JOBS=, MAXIMUM_NUMBER_OF_PROCESSORS=, and NUMBER_OF_IMAGES_IN_CACHE= are different for each stage, so a list of the options used for each stage is shown. I strongly encourage anyone out there with a new machine running Linux or Mac OSX to give this a try. This list shows only the best result from each machine. The full list of all timings (~9000 runs) is available in this file. You may have different ideas about how to evaluate it. I think all us crystallographers could really use a standard reference for the "speed of XDS" on linux systems, since it seems to be very difficult (for me at least) to predict how any particular system is going to perform.

mkdir -p /dev/shm/test/
cd /dev/shm/test/
wget http://bl831.als.lbl.gov/~jamesh/benchmarks/xds/xds_bench.com
chmod a+x ./xds_bench.com
./xds_bench.com

mkdir -p /dev/shm/test/
cd /dev/shm/test/
curl http://bl831.als.lbl.gov/~jamesh/benchmarks/xds/xds_bench.tgz | tar xzvf -
./xds_bench.com

These results are from a 3-macocycle phenix.refine job (12kD protein, 1.8A)

curl http://bl831.als.lbl.gov/~jamesh/benchmarks/phenixrefine_bmark.tgz | tar xzvf -
./test.com

These results are from a 10-cycle refmac job (12kD protein, 1.8A)

curl http://bl831.als.lbl.gov/~jamesh/benchmarks/refmac_bmark.tgz | tar xzvf -
cd refmac_bmark/
./test.com

---

Speed of Mosflm

These are the results from integrating 19 10Mb x-ray images in mosflm.

Speed of scala

scaling 360 batches of data in scala

scaling 70 batches of data in scala

Comments

Overall, performance was quite consistent with multiple runs on all these machines (indicating disk cahce didn't make much difference). In the case of MOSFLM, running two jobs on a two-CPU computer had only a marginal (~3%) impact on the elapsed run time. Suprisingly, compiling with and without optimizations (where compilers were available) made no noticable difference for any of these programs.

---

Speed of O

The following are the results of rotating this 31,262-vector map using the "spin" command in o. I strongly encourage anyone out there with a PC running o (WinNT or Linux) to download my little map, and give it a whirl in o. I think all us crystallographers could really use a standard reference for the "speed of o" on linux systems, since it seems to be very difficult (for me at least) to predict how any particular system is going to perform.

Back to the Elves Homepage.