NESCC - Boulder

Since NFS to the new MSSes have been turned off, mssGet is provided as a convenience wrapper around scp with an NFS-like appearance.

Command Use and Options

mssGet -- retrieve files from the new MSSes in a more efficient manner

mssGet [ --user <user> ] [ --ssh-opts <opts-str> ] \
       [ --listFile <listFile> [ --listFile ... ] ] \
       [ --verbose ] [ --debug ] [ --help ] \
       [ [ --scp-opts ] <options> ] <file|dir> ... <file|dir>

where:

--user <user>  --  run all scp commands under the auspices of the user <user>. Each path argument may be preceded with '<user>@', which overrides the command-line specification.

--ssh-opts <opts-str>  --  options to pass to the underlying ssh (used by scp and for other communications with the mssGet server). Quote <opts-str> to pass multiple arguments or to protect spaces within arguments.

--listFile <listFile>  --  the path to a file <listFile> that contains source/destination pairs. See below for syntax and usage. There may be several such files placed on the command line.

--verbose  --  print (to the standard output device) the wrapped scp command line(s) prior to executing it (them).

--help  --  print (to the standard output device) this help text.

<options> or --scp-opts <options> --  other options to pass to the underlying scp. Options previously specified with --ssh-opts need not be repeated in --scp-opts (and vice-versa) (when '--ssh-opts' is used).

<file|dir>  --  source or destination file/directory. Since as of now no remote current working directory mechanism is supported, each source path argument must be absolute, starting with '/mss', '/mss2', etc. '/mss' refers to all MSSes (except the earliest, DSRS-based MSS). Each path argument may be preceded with '<host>:', which overrides the derivation from the path's root directory segment.

The --rsh, --rcp, --ssh, --scp, --jet-san and --cp options have been obsoleted. They are ignored if present on the command line.

Caveats

-  file globbing (wildcarding) characters are supported, but must be quoted/escaped to protect them from the shell.

-  all user/group authorization is done remotely by ssh/scp to the MSSes. Please send mail to hpcshelp.fsl@noaa.gov or contact Lee Cohen (x6052) if you run into authorization problems. MSS or other mssGet problems can be directed to Glen Pankow (x7028).

-  the underlying scp used is the one selected from you current environment (probably /usr/bin/scp if you haven't done anything special).

-  the wrapping requires several dozen characters of its own, which will limit command line lengths. There is currently no trapping of command line length violations.

List Files

A list file is a file that contains, one per line, a source/directory pair of entries, handled much like the command-line arguments. For example:

# Retrieve SAO data for Feb 06, 2002.
/mss/ds/2002/02/06/data/sao/netcdf/*o retroSao/feb/
/mss/ds/2002/02/06/data/sao/sao.cdl retroSao/sao.cdl

Like on the command-line, the source files may contain file globbing characters (there's no need to quote them, however); in this example the corresponding destination is a local directory.

If mssGet can easily determine if a destination should be a directory (either the source is a glob which expands to multiple files or the destination is explicitly terminated with the '/' character), mssGet checks that directory for existence, and attempts to create it if necessary (with permissions mode 0777, modified by the user's current unmask setting). If mssGet does not know that the destination must be a directory, it only checks intermediate level subdirectories for existence. In this case, if the destination was meant to be a directory but it didn't exist, mssGet won't know about until the spawned copy commands fail. Thus it's a good idea to add an explicit '/' character on the end of any destination directory.

Like with the command line arguments and the DSRS mcp command, source files/globs may be contain '<user>@' and '<host>:' prefixes, which are passed on to all spawned scp commands. Lastly, a list file may also contain blank lines and comment lines (whose first non-blank character is a '#').

Example

$ mssGet -v '/mss/ds/2001/04/0[1-3]/data/acars/qc/java/*{00,12}00enc.bin' ./
Connected to mssGet server.
$ scp /mss/ds/2001/04/01/data/acars/qc/java/010910000enc.bin /mss/ds/2001/04/01/data/acars/qc/java/010911200enc.bin ./
$ scp /mss/ds/2001/04/02/data/acars/qc/java/010920000enc.bin ./
$ scp /mss/ds/2001/04/02/data/acars/qc/java/010921200enc.bin /mss/ds/2001/04/03/data/acars/qc/java/010930000enc.bin /mss/ds/2001/04/03/data/acars/qc/java/010931200enc.bin ./

Terms

scp  --  secure copy
ssh(1)  --  secure shell (needed to use scp).
sh(1)  --  standard job control shell and command interpreter (for file globbing)
umask(1)  --  set file-creation mode mask
DSRS  --  Data Storage and Retrieval System (the original MSS)
mcp  --  DSRS file retrieval script

Human Resources

hpcshelp.fsl@noaa.gov  --  preferred question/problem access point
Glen Pankow (x7028), Lee Cohen (x6052)  --  MSS usage and internals
Ed Moxley (x6844)  --  MSS system administrator

Author: Glen Pankow, 08/02/02
Last Modified: 4/4/11