
The Elves' quick guide to their scripts

1) In a rush?  Just type this:

./Elfsheim/Processer hurry >&! process1.log &

That's it.  

If you're in a BIG hurry, and you have several CPUs at your disposal,
you can use all your CPUs in parallel:
./Elfsheim/Processer hurry >&! process1.log &
ssh anothermachine
cd `pwd`
./Elfsheim/Processer hurry >&! process2.log &
./Elfsheim/Processer hurry >&! process3.log &
./Elfsheim/Processer hurry >&! process4.log &
etc..

each "Processer" will take full advantage of one CPU.  As long as
the Processers are all launched in the same directory, no matter
what computer system they are on, they will cooperate with each
other, and, eventually, process all your data.

2) Okay, how does it work?

The Processer Elves will run:
1) Wedger Elves for each of your data wedges
2) Scaler Elves on all the raw intensity data
3) SHELX and/or SOLVE on the datasets output from Scaler
4) Phaser Elves on the resulting heavy-atom positions
5) ARP/wARP on the "best" phased mtz from Phaser (resolution permitting)

in the ./Elfsheim directory...
################################################################################
sendhome

    reads:  x-ray images, as they appear
    
    writes: senthome.log
    
    runs:   ssh, tar, compress, uncompress
    
    output: x-ray images (on a remote filesystem)
    
    example:
	sendhome /data/se1_1_001.img user@remote.college.edu:/bigdisk/user
	    - will send /data/se1_1_001.img (and any other frames in /data
	      that are newer than se1_1_001.img) to /bigdisk/user using
	      "user"'s account on remote.college.edu.
	      
    description:
	sendhome is designed specifically for crystallographers collecting
	x-ray image data that they want to send to another computer (home).
	The file transfer is done via an ssh (version 1 or 2) login session, 
	so broadcasting your password is no longer a concern.  The frames
	are compressed in transit, so sendhome is ~3X faster than FTP, and
	sendhome continuously monitors the disk for newly-collected images, 
	and sends them as soon as they appear, allowing you to concentrate
	on data collection, instead of backups.  Also, once sendhome is
	running, you can <Ctrl-Z> and "bg" the job, and then log out and
	go home yourself.  sendhome will timeout and exit if no new frames
	have appeared after 2 hours.
	If the transfer is ever interrupted, you can usually re-start sendhome
	with just a directory name, it will read the senthome.log file, and
	pick up where it left off.
	WARNING: if you kill sendhome your last file may not be completely
	transferred.  Because of a quirk in unix pipes, the last kilobyte 
	or so of your last frame will not be sent until sendhome exits 
	normally.  If running interactively, sendhome will warn you about this, 
	and offer to transfer the last file explicitly before exiting from 
	a <Ctrl-C>.
    
    note: 
	to send home a processing directory, type something like this:
	cd /local/processing
	tar cBf - . | compress | ssh user@home.college.edu "cd /home/processing ; uncompress | tar xvBf -"

################################################################################
Wedger Elves	- single-wedge x-ray image data processer

    reads:  x-ray images (ADSC Q4, MAR, R-axis II & IV, and others)
	    mosflm scripts and script fragments
	    mosflm orientation matrix files
	    denzo *.x and input files

    writes: mosflm.com	    - a simple script for running ipmosflm on your data
	    merge.com	    - a simple scala/truncate mergeing script
	    SGsearch.com    - routine for running merge.com in alternative space groups
	    autoscala	    - routine for optimizing SDCORR in merge.com
	    autoindex.inp   - script fragment for setting up autoindexing manually
	    patt.com	    - simple difference-Patterson script

    runs:   ipmosflm (merge.com needs CCP4)

    output: raw.mtz	    - raw intensity data in multirecord mtz format
	    merged.mtz	    - scaled and merged reflection
    
    examples:
	Wedger /data/frames/Se1_1_001.img
	    - set-up mosflm.com for wedge beginning with Se1_1_001.img
	Wedger /data/frames/Se1_
	    - set-up mosflm.com for files matching /data/frames/Se1_*
	Wedger ./start -new
	    - read the ./start script fragment, and don't look for other files
	Wedger mosflm.com
	    - pick up processing where mosflm.com left off
	Wedger auto.x
	    - start processing from a denzo indexing solution
    
    description:
	Wedger Elves are capable of everything from autoindexing to scaling and
	mergeing of single-wedge x-ray diffraction image data.  If you have 
	multiple wedges, you should process each of them with Wedger, and then 
	combine the output files with Scaler Elves (below).
	Wedger Elves run mosflm to index and process x-ray images, and are
	designed to teach you how to use mosflm as you are processing your
	data.  
	If you do not have mosflm 6.x, then Wedger Elves can go get it for you.

################################################################################
Spotter Elves	- easy access to important spots

    reads:  x-ray images (ADSC Q4, R-axis II & IV only)
	    scala logs
	    denzo *.x files and scalepack logs

    writes: moviefy	    - movie-maker engine
	    absences.moviekey - textual index of the extracted spot images
	    rejects.moviekey  - textual index of the extracted spot images
    
    runs:   adsc2pgm, osc2pgm (or embedded backup)
	    ImageMagick (convert, mogrify, animate)
	    dmconvert (SGI only)
    
    output: absences.movie  - SGI movie of systematic absence observations
	    absences.gif    - animated GIF equivalent of the above (ImageMagick required)
	    rejects.movie   - SGI movie of rejected observations from scaling
	    rejects.gif	    - animated GIF equivalent of the above (ImageMagick required)
    
    examples:
	Spotter logs/merge.log /data/frames/Se1_1_001.img
	    - display rejected outliers and systematic absence observations
	      from the original images, beginning with Se1_1_001.img
	Spotter scalepack.log
	    - display rejected outliers and systematic absence observations
	      from the original images listed in the *.x files
    
    description:
	The version of Spotter Elves that ships with Elves 0.9 will only work
	on SGIs, and can only use R-axis and ADSC Q4 type images.

################################################################################
Scaler Elves	- multi-wedge, multi-wavelength localscaling

    reads:  */*/raw.mtz (unmerged, multibatch mtz files)
	    scala scripts and script fragments
	    denzo *.x files

    writes: sort_everything.com	- script for combining all input data into a
				  single, multirecord mtz file (regardless of batch numbers)
	    make_reference_set.com - scale and merge a reference data set.
	    import_reference.com   - combine a merged reference with the raw data.
	    rough_scale.com	- batch-wise, initial scaling pass
	    localscale.com	- smooth, continuous scaling procedure
	    merge.com		- merge one of the wavelengths after scaling
	    extract.com		- extract a wavelength without mergeing it
	    scaleit.com		- combine merged wavelengths into one mtz file
	    
	    FreeRer.com		- like "uniqueify", but can "inherit" a Free-R set
				  from another file
	    autoscala		- routine for optimizing SDCORR in a scala script
	    
	    runlist.txt		- scala script fragment, defining "runs" and
				  "wavelengths" (runs merged together)
	    Makefile		- a "make" file for using "make" to rebuild your
				  scaling project with the above scripts
    runs:   scala, truncate, scaleit, cad, mtzutils, etc...
    setup:  solve, shelx, x-plor

    output: mtz/all.mtz		- multi-column mtz file (for mlphare or SHARP)
	    SOLVE/*.fmt		- data formatted for input into solve
	    SHELX/*.hkl		- data formatted for input into shelx
	    XPLOR/*.fobs	- data formatted for input into x-plor/cns

    examples:
	Scaler */*/raw.mtz
	    - set-up mosflm.com for wedge beginning with Se1_1_001.img
	Scaler */*/raw.mtz with 8 Se sites in ~/protein.seq
	    - same thing, but use site and protein sequence data for solve and truncate
	Scaler *.x
	    - use the denzo data in *.x files (york or default format okay)
	Scaler runlist.txt cards.txt
	    - pick up processing after editing the "run" definitions in
	      runlist.txt, and using the scala commands in cards.txt

    description:
	Scaler Elves are an adaptive implementation of the "MAD scaling" 
	procedure outlined in the scala manual.  They easily handle multiple
	input files, and they make sure none of the batch numbers overlap.
	Scaler Elves can carry out the scaling for you (they know how to
	fix the most common problems), or you can take over scaling yourself
	by editing and running the above scripts.
	
	Scaler Elves write a relatively large number of scripts, and so, 
	a Makefile is included to run them all intelligently.  All you have
	to do is type "make" and any changes you made to the scripts will
	be applied.

################################################################################
Phaser Elves	- general heavy-atom phasing/solvent flattening

    reads:  merged, multi-column mtz data
	    solve.status file
	    shelx.res files
	    sites.pdb files (heavy atom sites)
	    random.txt	    (fractional coordinates in "0.000 0.000 0.000" format)

    writes: mlphare.com	- simple mlphare script for refining sites against data
	    dm.com	- basic dm script for solvent-flattening
	    fft.com	- "smart" fft script for calculating/converting maps
	    bestFH.com	- procedure for combining all difference data to estimate FH
	    reindex.com	- general mtz re-indexing procedure (includes P2212, etc.)
	    shelx.com	- combinatorial shelx-running script (try different combinations
			  of space group, resolution, cutoffs)
	    rantan.com	- same thing, but uses rantan instead of shelx
	    pick.com	- sophisticated map-peak-picker (allows "taboo" sites to be specified)
	    
    runs:   mlphare, dm, rantan, shelx, rsps, vecref, fft, peakmax

    output: mlphare.mtz	- result of mlphare run
	    dm_??.mtz	- result of solvent flattening at ??% solvent
	    phased.map	- 
	    
    examples:
	Phaser mtz/all.mtz SOLVE/P2212/solve.status P2212
	    - run mlphare on all.mtz, using the sites found by solve in P2212
	Phaser mtz/all.mtz SHELX/ano.res
	    - refine sites found by shelx in ano.res
	Phaser mtz/all.mtz 
	    - find some heavy-atom sites with shelx or rantan, then start
	      refining them against all.mtz data
	Phaser mtz/all.mtz -new
	    - start over, don't try to restart with an old mlphare.com, etc.
	Phaser scripts/mlphare.com  no flip sg
	    - pick up where we left off in scripts/mlphare.com, but don't
	      flip between alternative space groups. (you can also say:
	      "no flip hand", "no flip occ", or just "no flip")
	Phaser scripts/mlphare.com  no add    40% 45% 50% 
	    - same thing, but don't add difference-Fourier peaks back into
	      mlphare.com.  Also solvent contents of 0.4 0.45 and 0.5 will
	      be tried in dm.
    
    description:
	  Phaser Elves obtain heavy-atom sites either by running shelx, rantan 
	or rsps, or by get them from a solve.status file, pdb file, res file, 
	or just a plain text file with three decimal numbers on each line.
	  Sites are refined against mtz data with mlphare, and the results
	are solvent-flattened with dm.  Multiple solvent contents can be tried
	if you're not sure of your crystal density.
	  Alternative choices of occupancy sign, site hand and space group are
	also explored by Phaser Elves.  
	  Atoms that refine to unrealistic parameter values (occ near 0, B>500)
	are automatically rejected from the mlphare script.  (This behavior can
	be overriden by stating "no reject", but it is not recommended)
	  Once everything is refined, Phaser Elves will calculate a phased 
	difference Fourier (combining all your difference data) and look for
	potential new sites.  By default, these new sites are added to the
	mlphare script, and refinement continues until no novel peaks appear
	in the difference Fouriers.

################################################################################
Processer Elves

    reads:  contents of wedge directories
    writes: "busy" and "done" signals for other Processer jobs
    runs:  Wedger, Scaler, shelx, solve, Phaser, wARP
    
    
    examples:
	Processer >&! processing.log &
	    - launch Processer in the background.  Any existing wedge directories
	      will be processed automatically.  All space-groups that
	      have been set-up in SOLVE or SHELX will also be processed.
	Processer hurry >&! processing.log &
	    - tell Processer (and all the Elves it runs) to "hurry", skiping
	      most optimization steps, and getting to a rough solution as
	      quickly as possible.
	Processer fpp/wedge* lo/wedge* >&! processing.log &
	    - tell Processer to only work on the specified directories (useful
	      if you think one of your wavelengths is badly decayed)
	Processer no solve 
	    - tell Processer to quit after scaling and mergeing the data.
	Processer solve only
	    - tell Processer to jump immediately to the SOLVE and SHELX stage.
	    
    description:
	  Processer Elves will eventuall be incorperated into the Elves main
	program.  But, for now, you should use them to automate everything but 
	the initial set-up and indexing stages of your project.  Once you have
	a directory structure of */wedge*, SOLVE/*/ SHELX/*, etc.  Processer
	Elves are designed to move through these directories, processing data
	as they go.
	  If you launch more than one Processer job in the same, root working
	directory, the jobs will cooperate with each other, each working on a
	different part of your project at the same time, and completing the
	whole data-processing task in proportionately less time.  If you have 
	access to your processing directory tree from another computer, you 
	can run Processers on remote CPUs too.  Processer Elves can still 
	coordinate their efforts from an arbitrary number of different computers.

################################################################################
################################################################################

3) What if something goes wrong?

    The Elves main program is desiged to handle a number of common problems 
encountered in processing x-ray data.  However, they are certainly not capable
of handling EVERY situation that might crop up.

    If you have problems with the Elves main program, there are a series of 
fallback positions you can take:

  A. Problems with integration
    1.	Go through all your data wedge directories yourself.  Use Wedger 
    interactively until you see accurate spot predictions for each wedge.  
    
    2. Then go back to the root directory, and run Processer to tune 
    everything up.
    
  B. Problems with how Processer processes your data
    1.  If you don't like the way Processer does things, you can always
    run the Elves manually: Wedger, Scaler and Phaser are all a lot more
    user-friendly and customizable that Processer Elves.
        Wedger should br run in the wedge directories themselves.  Scaler
    and Phaser Elves can be run from just about anywhere.
    
  C. Problems with how Elves process your data
    2.  The script: ${SCRIPT_dir}master.com is intended to guide you through
    the entire data-processing chain, but by running the scripts Elves wrote 
    directly (no automation).  All of these scripts have been desiged
    to be easily read, edited, and re-run by end-users.  README files have
    also been written to guide you.

