Recovering files from snapshots

On our Network Appliance filers, there is a feature implemented called snapshots. This feature is available for files that are stored on the NetApp filers filer3 and filer4b. Home directories and most research file structures live on these filers.

What are snapshots?

As the name indicates, a snapshot is a "picture" of each directory and its contents. Everyday at midnight and every four hours starting at 4am (4am, 8am, noon, etc), a snapshot is taken of the files on these NetApp filers. Every midnight, a nightly snapshot is taken instead of an hourly one. There is also a weekly snapshot taken Sunday at midnight (instead of Sunday's nightly snapshot). Midnight snapshots are in nightly.N directories, the 4-hourly snapshots are in hourly.N, and the weekly snapshots are in weekly.N. The snapshots are read-only, so you must copy the file(s) to your home directory or some other location you can write to in order to edit the file(s).

  • Note: Last night's snapshot is in nightly.0, the previous night's snapshot is in nightly.1, and so forth, through nightly.11
  • Note: The nightly snapshots are on a rolling week cycle and the 4-hourly snapshots are over-written daily. Likewise, weekly snapshots are cycled weekly, and the most recent 4 are kept.

About 20% of the available filesystem space is reserved for snapshots. On a filesystem with a lot of activity, there may not be enough space to track all the changes through a full set of snapshots, and the oldest ones will be automatically deleted. Sometimes you might find that only the hourly snapshots and one or two nightly snapshots exist. We cannot recover autodeleted snapshots.

How can I tell if there is a snapshot of the file I just deleted?

The snapshots can be accessed in a .snapshot directory of any directory that normally exists on the NetApp filers - you can change directory into it or list its contents. A file will have a snapshot taken of it if it exists at the time of a snapshot.

This might be the contents of .snapshot:

> ls .snapshot
hourly.0      nightly.1     nightly.3     nightly.5
nightly.0     nightly.2     nightly.4     weekly.0

  • Note: The .snapshot directory name will not be tab-completed
  • Note: A snapshot of your file will not exist unless it was in existence on the filer during a snapshot at midnight or at one of the 4-hourly snapshots.
  • Note: If you can't find the .snapshot directory, if the file you're looking for isn't in there, or if you need further help, send mail to operator@cs.utexas.edu

Look, a snapshot of my file! How do I get it?

As noted above, you can copy the file to any location you are allowed to write to such as your home directory.

Examples

I had a directory called "foo" in my home directory with a file in it called "bar". The file "bar" existed since yesterday before midnight, but I just now deleted it. In a situation like this a couple of cases might happen:

  1. The last time I modified it was last night before midnight, so I should look in the .snapshot/nightly.0 directory. e.g. While in my home directory...

    Looking in the snapshot...

    > ls .snapshot/nightly.0/foo/

    There it is!

    bar

    Copying the file from snapshot...

    > cp .snapshot/nightly.0/foo/bar .

  2. The last time I modified it was today before noon. It is now after noon, but before 4pm, so I should look in the .snapshot/hourly.0 directory. e.g. While in my home directory...

    Looking in the snapshot...

    > ls .snapshot/hourly.0/foo/

    There it is!

    bar

    Copying the file from snapshot..

    > cp .snapshot/hourly.0/foo/bar .

I had a file called "baz" also in the directory "foo" three days ago. I've since deleted it, but it had existed over-night previously. In this case, the file would not be in hourly.0 since that snapshot is over-written every four hours, however, it would be in nightly.2

Additional help

If you need more help restoring your file, if the file you deleted was never on a filer, or you are unsure if was on a filer or not, send mail to operator@cs.utexas.edu