MSS Intro | Multiple MSSes | /public Data | Wrapper Scripts | Tips and Traps

MSS Tips and Traps

Tips for Storing Files

  • tar up small files -- MSS retrieval times are affected mostly by the number of files to be retrieved. If you have many small, related files that you'd normally retrieve in groups, tarring them up will greatly speed up the retrieval. It's not as important to compress them as well, butthat may prove useful.
  • limit files per directory -- it's easy to whip up a simple storage script, watch it run correctly for a little while, and then forget about it. An easy trap to fall into is where such scripts write into a single directory or flat directory structure, that over time result in huge directories. Please try to limit directories to contain no more than 1000 files. tarring files can also help mitigate huge directories.

Tips for Retrieving Files

  • use mssGet -- mssGet implements the most efficient retrieval mechanism.
  • retrieve multiple files at a time -- mssGet is no better at speeding up retrievals than any other method if you run it one file at a time.
  • plan ahead -- while an HSM is designed to look like a disk, it is not a disk. Understand how an HSM works -- give yourself plenty of time to retrieve files. A basic rule of thumb is that you should start file retrieval the night before you really need the files.
  • operators are standing by -- if you have large requests, and have plenty of lead time before the request must be satisfied, you can have the ITS operators do the retrieval. To send a request, send an email to help.opers.gsd@noaa.gov.