Some of these "tips" are just about basic computer courtesy. I hope that everyone can agree that the beamline's main purpose is to collect data. Remote processing comes second. I would like for people to be able to login and process data here after their collection is done, but you must be careful not to mess things up for people who are trying to collect! Unfortunately, there have been some recent "incidents" of legitimate users logging in remotely and (I assume unintentionally) breaking various things while other people were trying to collect data! About 30% of my time this week was spent cleaning up after a few careless and sloppy people, and this is not fair to the people who are on-site and need my help. So... Some tips: - when processing remotely, use /local on crush13 each "crush" node has a fast, local disk called /local that anyone can put files in. It is much faster than using /home (over the network) and will not interfere with someone who is collecting, and needs to process quickly. Crush13 is set up to share process space with other "crush" computers, so even if it looks "busy" it's not. So, if you are processing remotely, just do this: cd /data/mcfuser/ tar cf - your/stuff | rsh crush13 "cd /local/mcfuser ; tar xvf -" rsh crush13 cd /local/mcfuser/your/stuff Elves . - transfer files home from the file server the computer you log into when you connect to "bl831.als.lbl.gov" (or "server" from the graphics terminals) is the computer with the local /data disk. So this is the best place to launch file transfers. Launching a file transfer from a graphics terminal or a "crush" node requires the data to go back-and-forth three times over the network, and just slows things down - do not run jobs on the file server the computer you log into when you connect to "bl831.als.lbl.gov" (or "server" from the graphics terminals) is also running all the data collection control system programs, and it is also the slowest CPU at the beamline. Running a processing job here may cause the detector to hang in the middle of collection. If I catch anyone doing this, I will kill your job (and, perhaps, you) without hesitation. ;) - /data is for data The /data disk is optimized for high-speed access to large files. Putting your processing files in here will significantly slow down data collection because of the "high-transaction" NFS load that processing puts on a disk, and because the millions (yes, there are millions) of tiny processing files will cause a disk with a large block size to rapidly fill up. /data actually "filled up" at 50% once. I am now in the habit of regularly moving peoples processing files from /data to /home so that /data does not get fragmented. - /home is for processing The /home disk has been optimized for processing. Small block size, and a located in a different computer from /data gives a significant (30-50%) improvement in data processing speed and prevents processing from slowing down or crashing data collection. - /data2 is for emergencies There have been times when I told people to collect here, but that was because /data was full. We are pretty good about keeping up with the backups now, so this shouldn't happen anymore. Use /data2 only if /data fills up in the middle of the night. - only /data is monitored by the DVD robot /home and /data2 are NOT! Every 5 minutes, the DVD robot checks for any new images on /data. Once you have enough to merit a DVD, or when the name of the user changes, the robot will write a DVD. Unfortunately, the robot cannot tell where a new file came from. It expects that all new files are newly-collected data. So, if you are a Credle, and you copy some old data back onto the beamline while somebody else is collecting, the archiver will start rapidly switching between "users" and make a big pile of DVDs with one image file on each. (this could get expensive) - use the "-p" flag in cp and scp. this flag will make sure the date on your files does not change when you copy it. Not only is this nice to the archiving system, but it will help you in the long run figuring out which files are what. - Do not "touch" your files to keep them around Trying to hide your files from the "reaper" that cleans up the /data disk periodically by making them look like recently-collected data is dishonest and also doesn't work. The "reaper" program is now smart enough to recognize files that are out of place and will delete them immediately (without backing them up first!). - feel free to change your filenames, but... Changing your file names or moving data into new directories is something that you do want to be able to do while you are collecting. This is the only reason why I don't make the /data disk read-only like all the other ALS beamlines. However, you do run the risk of confusing the archiver, and your data may end up with it's old name on the DVDs (depending on when the DVD is triggered). Also, programs like "download" that follow the data collection log won't notice moved files. Basically, if you rename stuff, check that it transferred correctly. - Do not put multiple copies of your data on disk If anyone can come up with a good reason why why this is anything but a complete waste of space, I'd like to hear it. - Do not put files in generic directories Please put your files under a directory that contains your name. This makes it easy for the archiver (and me) to find your data in case you need it restored some day. - Do not delete other people's data This hasn't happened yet, and I hope this "rule" doesn't sup rise anyone. - Do not delete your own data This is my job. My big research project at the beamline is studying how to make beamlines more efficient. I need a complete record of the data quality collected at 8.3.1 to make this happen. (this includes snapshots!) - The beamline is not an overflow disk I know that the home labs can be strapped for space sometimes, but please don't try to use the beamline data-collection disk for long-term storage. It's unfair to people who have new data and really do need access to it. Besides, I've got everything on DVD here anyway, and I'd be more than happy to restore some old data that you might need, or to make a DVD of it for you. Sorry to be complaining so much right before the Holidays, but I hope that this will help people work more efficiently at the beamline in the future. And when you work more efficiently, we all have more time to play. -James