File Management

Introduction

Without good file management and organization, this will be a very long and frustrating semester for you. You are free to adopt what ever file management system works for you, but most students typically err on the side of disorganized and sticking to whatever defaults ArcGIS makes for you. This is a recipe for lots of broken links and red exclamation marks!  in your table of contents (in time you'll see what I mean). This page provides some suggestions for how to manage your files to prevent so many tears.

Most students will probably be working on assignments in a variety of locations on different machines at different times. You may start a lab in the Quinney computer lab, work on it again later at a different machine in the Quinney lab, then work on it from home using your student version of ArcGIS on your personal computer. As such, you will need to keep your map documents and GIS files well organized so you can easily transfer them between different machines. It is virtually guaranteed that at some point this semester, you will run into trouble making a map or doing a simple geoprocessing task, and the reason will point back to a file management problem. 

Storage Choices & Tradeoffs

Storage Location

This table summarizes your primary choices for where to store your GIS data for this class and the pros and cons of each: 

 Storage Location    Pros Cons Tips
Portable USB Drive - You can take it with you to any machine
- You can loose it! Make backups somewhere.
- It can break or become corrupted easily
- If you're drive is small or close to capacity, you can run out of room		- Slower. For 95% of the labs we'll do, this won't be an issue. For large geopocessing tasks it can become cumbersome
- You may need to make a new folder connection every time you start ArcGIS on a new machine
- As your drive may get mapped as a different letter each time, you may need to repair data sources regularly.  		- Follow the instructions provided here on creating map documents that store relative paths

 Cloud Storage Drive  (e.g. DropBox, Box, Egnyte, Google Drive, SkyDrive, etc.)-It is accessible from any computer AS LONG as there is an internet connection
-Backup is secure, and redundant		 -Most are not reliable enough to work on actively in the cloud... thus you need a local copy and then you work on that
- Desktop sync applications don't always work that reliably with big datasets and GIS data		 -Work locally, and use cloud as a back up, which you manually back up to
- If available, use FTP for your backups
 H:\ drive on Quinney Lab Computer-It is accessible from any Quinney lab computer
-It is backed up		 - It is not accessible outside the labs
- We have occasionally noticed that ArcGIS does not like the connection to this network drive
- As it is a network drive, with lots of network traffic it can be slower
- There is a limit to how much storage space is allowed here		 - Keep a synchronized copy on a portable USB Drive in case you need to switch
Folder on local hard drive on your own computer- Fast
- You know where it is		- You'll need to synchronize this to a portable USB drive if you want to use this lab
- It will not be available from any lab or campus machine		- Make sure there are no spaces in any of the folder names or symbolic characters other than dashes (-) or underscores (_)
Folder on Quinney Lab computer's local hard drive  - Fast- Machines are set up to erase such folders upon reboot
- It almost certainly will not be there next time you log in
- It will not be available from any other machine		 - We recommend you use a folder either on your desktop or in the 'My Document's of your user profile as a location for this. Always backup the files/folders you use to your portable hard drive.
Desktop on your own computer - Easy to download to
- Relatively easy to find 		 - It clutters your desktop up and makes it hard to find things (unless you place a folder on your desktop and use that)
- The path is not simple (e.g. C:\Users\User Name\Desktop) and may have spaces (some commands in Arc don't like spaces)
- It will not be available from any lab or campus machine
- To many files on your desktop can slow Windows down.
 - If you do use our desktop, don't store data in the root of your desktop, but instead in a subfolder
Desktop on Quinney Lab- Easy to download to  Same as Desktop cons above plus - Same as Desktop suggestions above
Your ArcGIS Online My Content Folder  - Easy to share and access from multiple locations- Limit of 2GB of space
- Only works if you have an internet connection
- Slower then a local data source		 - Use with caution... we'll cover this in Lab 04

NOTE: It is okay to backup and store your data on the cloud (e.g. Egnyte, SkyDrive, DropBox, Google Drive), but DO NOT actively work off a network drive (recipee for disaster). You can work off of a local drive/folder that is synced to the cloud, but don't work off the cloud live as it depends entirely on the quality of your network connection.

Be Careful with Synchronization & Backups

It is very easy to mess up when synchronizing data between storage locations. The safest bets are typically to:
  • Copy and overwrite everything - this can be very slow, and if you change folder names it does not delete the old folders at the destination folder
  • Delete everything at destination, and copy entire source - this can be very slow
  • Very carefully manually copy and delete files and folders as necessary
  • Use some sort of folder clone (e.g. FolderClone) or folder synchronization (e.g. FolderMatch; e.g. Allway Sync) software (not a backup software)
Do not attempt to synchronize or backup data while ArcGIS software is open that is referencing that data. 

Our Recommendations

  1. Use a portable USB drive if you are working on multiple machines in multiple locations. Make a local copy of your data if you need to do any heavy geoprocessing tasks, and then synchronize back to your portable drive when you are done.
  2. No matter where you work, always keep some sort of backup of your data
  3. We strongly discourage you from creating any MapDocuments that have data layers stored in a mix of locations (i.e. some on a desktop, some on a portable drive, some on a local drive, some on a network drive). You are better to save your Map Document in a folder, in which the data it refers to is the same folder or child folders. The exception to this of course is Arc Basemap layers, data from ArcGIS online, or data referring to some other webGIS server. 

Suggested Folder Organization

Once you decide where you are going store your GIS data, you need to have an organized directory structure that is logical and easy to navigate quickly to the files you need when you need them. We have prepared a folder structure template you might use for this course (click on for larger view):
 
Folder Template: WATS4930.zip 

You can unzip the folder template to any parent folder (WATS4930 in the example above) in your chosen storage locations. The directories are organized as follows:
  • Common - For any common base layers you find yourself using across labs and/or your project
  • Labs - With a subdirectory for each lab.
  • Project - For your project data and maps
Within each Lab0X folder there are four TaskX subfolders for each of the lab tasks associated with each lab (not all labs have that many tasks and you may not use or delete some of these). We recommend that you save a new Map Document (*.mxd) file for each map you make and store these according to task. Feel free to create as many additional subdirectories as you find helpful (just avoid putting spaces or odd characters in the folder names as ArcGIS may complain occasionally). The video below shows you how to unzip this folder to a location you want to use it:


If you are unfamiliar with dealing with zip files, see here. 

Storage Formats

We will cover data types and storage formats in Week 02.

Some other References

Comments