rsync + ssh public key

If you are using an UNIX (Linux/Apple/Solaris) system, the best way to transfer your data is via rsync transported with ssh. If you are familiar with the ssh public key scheme, simply send us (webmaster AT genomecenter.ucdavis.edu) your PUBLIC key (please DO NOT send us your PRIVATE key, it stays with you and only you).

If you are not familiar with ssh public key, here are a few simple steps to generate your public/private key pairs. Make sure that you have openssh utility installed (most Linux/Unix/OS X system do).

ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/your_username/.ssh/id_rsa): Return
Enter passphrase (empty for no passphrase):  Enter your own passphrase here
Enter same passphrase again: Enter your own passphrase again
Your identification has been saved in /home/your_username/.ssh/id_rsa.
Your public key has been saved in /home/your_username/.ssh/id_rsa.pub.
The key fingerprint is:
a5:4c:8b:77:00:a4:3b:dd:84:2c:15:66:b5:f4:ff:0d  your_username@yourmachine_name.genomecenter.ucdavis.edu

Send us your public key file (/home/your_username/.ssh/id_rsa.pub) to us and we can enable your access to fetch your files.

If your system have ssh-agent set up automatically for you, you can now authenticate to your ssh-agent by the following command:

ssh-add

give your pass phrase to authenticate. If your ssh-agent is not set up automatically, you need to run it manually,

eval `ssh-agent`
ssh-add

Note that in the expression `ssh-agent`, we use backtick ` here rather than single quote ‘. You can follow the section Archiving Files section below to download your files. You can read more about ssh public key authentication in the following references:

Windows: Cygwin+rsync+openssh

For Windows user, you need to install the freeware package basic cygwin plus the rsync and openssh packages. There are many online instructions on how to install cygwin, rsync, and openssh, here is one:

You can then download files using rsync similar to the other platforms.

Archiving Files

You change to your archive directory on your local machine, for example,

cd /data/archive/myrun
rsync -avL sol@hofmann.genomecenter.ucdavis.edu:Data/random_string/   lane1

The random_string above can be found in your SLIMS (http://slims.bioinformatics.ucdavis.edu) web interface, for the run you want to download

View->View Files

at the “View Files” screen, the address bar contains the random string name similar to the following URL

http://slims.bioinformatics.ucdavis.edu/Data/79856d9a09/

where 79856d9a09 is the random string that you need to download your files.

“sol” is the user name with which you download your file, you are the guest of user “sol” and rsync is the only command available.

You may repeat the above command again to make sure that you have downloaded all your files, rsync will download missing files if the previous session was interrupted, it will report 0 download file if everything on the local machine is in sync with the remote server.

(Note compresion -z in rsync as in “rsync -avz” is not as effient as “rsync -av” for large files in solexa analysis pipeline)

Rsync 3.0 Note

We have patched the server affecting rsync 3.0.x previously and you can download your files using instructions above.

Client FAT32 File System Note

Many external USB hard drives sold on the market are pre-formatted to Microsoft FAT32 file system. Unfortunately, this FAT32 file system has a maximum file size of 4GB, which might prevent you from downloading your files (if there is a single file larger than 4 GB) from our server to your external USB drive. Your need to reformat your file system to NTFS (on Microsoft system), various Linux file systems, Apple OS X HFS+ or whatever other native file systems for your computer, please consult your friendly system administration staff to choose a best file system for you.