Multiprocessing

---

The name Elves is plural for a reason. Elves can, in fact, do many things at once. The processing of crystallographic data turns out to be a highly parallelizable problem at several stages. Specifically, diffraction image data are usually collected in multiple contiguous groups, or "wedges", which can each be processed on different computers. In addition, there are space group ambiguities that persist right up until the final map is evaluated. Each of these space groups can be evaluated simultaneously, on different CPUs.

The Elves multiprocessing strategy is incredibly simple, and does not require any third-party process managment programs:

1. Separable tasks are first divided into different directories in the computer file system.
2. An Elves job that is working on one of these tasks will create a file called "busy" in its working directory.
3. Any other Elves job, before beginning work on a directory, will check for this "busy" signal, and, if they find one, they will move on to the next task.
4. This way, an arbitrary number of Elves jobs can "leapfrog" over each other. Until all the tasks are complete.
5. Checks can be made, using the contents of the "busy" file, to verify that the task is actually being worked on, and that the other Elves job was not killed (or something) before it could delete its "busy" signal.
6. The "busy" file also contains information about the "stage" of data processing (integration, scaling, phasing), so "new arrival" Elves jobs know where to begin. This way, additional CPUs can be added or removed to the processing, even if it's already almost done.

To use Elves multiprocessing features, all you have to do is run more than one copy of the Elves main program in the same root data-processing directory:

unix% cd ~/processing/elves
unix% Elves process the data in /data/jamesh/frames
...
meanwhile, on another, 4-processer machine called "crunchbox"
...
crunchbox% cd ~/processing/elves
crunchbox% Elves >&! elves_cpu2.log &
crunchbox% Elves >&! elves_cpu3.log &
crunchbox% Elves >&! elves_cpu4.log &

Each new instance of Elves will check to see if another copy is already running, and, by default, will begin cooperating with the running Elves jobs at whatever stage of data processing they have reached so far. This makes it easy to throw more Elves at a problem, even late in processing, if you decide its taking too long to run.

Back to the Elves Page.