Accessible Storage for Research Computing trying to accelerate research

(This is from a Google document of similar name at http://goo.gl/qHnIC.)

As I understand demand, these are the minimal set of storage options required to address the scholarly and administrative data and management requirements at the U-M:

1. Lustre (parallel storage for HPC)
2. NFSv3 (file system based storage)
3. NFSv4 (file system based storage)
4. CIFS (file system based storage)
5. HDFS (distributed storage supporting “big data”: map-reduce (Hadoop), column-oriented data (NoSQL), etc.)
6. Object Storage (something like Amazon S3, replacing or augmenting file system storage)
7. SQL (Oracle, MS SQL Server, MySQL, etc.)
8. Backups (relatively short time to recovery)
9. Cold storage (relatively long time to recovery)

These categories encode / encapsulate provisioning and configuration decisions regarding:

1. Media (SSD, spinning disk, tape, cloud)
2. On-disk formats and file systems
3. Network protocols and networking / fabric capacity
4. Media management (NAS, SAN, HFS, appliance, DAS, cloud, etc.)

These storage options have to be made available as accessible services at acceptable cost / capability tiers (where storage capability is traditionally expressed in terms of capacity, performance, and availability).

They all require implementation decisions.

These implementation decisions can be optimized along dimensions of interest: the number of platforms, vendors, campus providers, products, feature sets, etc., or otherwise expressed requirements from different communities.

The first of the three components is a high-speed storage service. This service is what is presented to the researcher and is a proxy for the back-up and archive services. The quantity of the storage for which the subscriber pays can be varied over time. The subscriptions can be funded by different sources with the storage presented to the subscriber as either an aggregated amount across funding sources or as separate amounts between funding sources, depending on the needs of the subscriber.

All of the data stored is also backed up and a set of backups are kept for a reasonable period of time and can be restored, deleted, or archived by the owner of each project.

The management of the backups is via a web-based presentation of the data contained in the backups, command-line tools on some systems, and email-based support.

The archive portion of the research working storage service will use a cloud-based data archive solution⁹. The process of depositing and withdrawing data from the archive is researcher-directed and the level of curation is at the discretion of the researcher. The style of archives in this service can be called ``data graveyard’’ or ``data dumping ground’’ in contrast with a curated archival solution that a data management group or library¹⁰ might offer.

The College of Engineering has decided to make an allocation of 100GB an entitlement to all of the researchers in the College. There are many research groups in the College who require more than that to support their work, and they are expected to augment their base allocation to suit their needs with their own funds.

Dr. Smith has a research scientist and six graduate students and usually manages two or three grants, although sometimes there are additional projects.

Her base storage allocation of 100GB provided by the College is held in a project called jmsmith_rs (``Dr. J.M. Smith’s research storage’’) which is accessible by her six graduate students, her research scientist, and herself, each with their own directories and some shared directories for collaboration. She and her graduate students connect to this storage from their laptops and workstations and it is also available on Flux and in computers in the U-M 3-D Lab.

Dr. Smith makes a request for an additional allocation for that project of 400GB for 4 years, bringing the total disk space available in jmsmith_rs to 1/2PB. She does this knowing that she can reduce the amount of space at any of the 6-month billing periods if she needs to, but by making it for 4 years, she avoids the risk of forgetting to renew it.

In addition to her main storage project, she is also working on a project called NexMent with her research scientist and two of her own graduate students and two graduate students from the Physics department in LSA. That research project has its own funding source and storage requirements, so she requests another project to provide storage to herself and the five other people involved and pays for it ith the grant money for that project. She now has another project, called jmsmith1_rs with a similar internal structure to her main project, but a different set of people who are using it.

When the NexMent project ends Dr. Smith initiates the process to archive the data from it and ends the storage allocation. The long-term preservation of the data, although slow to retrieve, fulfills a portion of the required data management plan associated with the grant that funded the project and there are no additional costs assigned to that grant.

As her career follows the typically successful arc of U-M faculty, her resource requirements wax and wane and she can control her costs and adjust her resources to follow that, all while knowing that her data is stored on professionally managed, high-quality infrastructure.

Dr. Jones’ research in the ethnography of music is very data intensive—tape recorders in the field have been replaced with high-fidelity digital recording devices—but the funding for data intensive work in the world of music is sparse at best.

Dr. Jones has two main components to his research: field work collecting samples of music before the limitless arm of iTunes reaches everyone; and analysis, cataloging, and reporting on what he has collected. The funding for these two activities can come together or each one can be funded individually.

Dr. Jones uses the U-M Research Storage Environment for both aspects of his work.

After returning from the field with many gigabytes of audio on his digital recorders, he copies it to the working storage where he knows it is backed up and can be quickly accessed. He is able to search and play and work with his data directly from the working storage without having to hold it all on his laptop (which is good, because the 256GB SSD in his MacBook Air couldn’t hold all of it). When the relatively short-term field-work grant ends, he archives all of the data from the most recent back-up of his working data and works on writing papers and more grants, knowing his data is safely archived and can be re-called when he needs (and can afford) to have it.

When Dr. Jones is funded to analyze a particular sub-genre of music that he has recorded on dozens of field trips over the years, he restores those from the archives to active storage and is able to look at them as a unit. For him, the ability to keep all of his data for a very long time allows for a kind of research that would be impossible if he had to decide what to keep as the end of a grant.

The ability to ``warehouse’’ data at a low cost for long periods of time allows Dr. Jones to use storage as a service instead of risking his data on low-cost, low-performance, low-quality, or low-all-three hardware. The ability to work on data over a high-speed connection allows Dr. Jones to save time transferring data to his laptop and also allows him to work on much bigger data sets than he could on his laptop, while still offering excellent performance for his audio analysis tools.

Dr. Robbins has been at the University of Michigan for twelve years, but has decided to move to Minnesota to be closer to his parents. He has amassed data in support of his research into disease transmission over the years that he will need in his new job. Unfortunately, Minnesota State University does not have a Research Storage Environment like U-M’s, but they will fund the purchase of several USB harddrives for him.

Before Dr. Robbins leaves U-M, he makes his last archive from his the backups of his active storage and prints the web page with the instructions on restoring archived data in a non-U-M environment.

When he arrives at Minnesota State University, he attaches his USB drives, and follows the instructions on restoring data from an archive, which include him paying the archive restoration and transfer fees from his funding in Minesota, so U-M does not incur any cost for this, although U-M will continue to maintain his data in the archive for 10 years after the last piece of data was added to the archive, so when Dr. Robbins’ USB drives fail, he can pay for another restore and transfer.

Scratch storage on Flux is based on the Lustre parallel file system¹⁴. Lustre is tightly integrated with Flux and is not presented to hosts that are not managed by the Flux operators.

This level of integration is important to maintain the performance and security of the file system. In addition, Lustre is only supported on Linux—there are no Mac or Windows clients.

The research working storage service proposed here will provide a location for long-term storage of large inputs or outputs that are best stored on Lustre while the related computational jobs are running or are staged to run.

Because the Lustre implementation on Flux is a very high-speed (40-80Gb/s) and very high-capacity (more than 600TB) filesystem, it has the ability to ingest, store, and output large quantities of data, so a long-term storage location for that data should be as fast as can be afforded, so that researchers don’t spend any longer than necessary moving their data.

The research working storage service proposed here is a good complement to Flux’s Lustre installation.

The NFS service offered by ITS called Value Storage¹⁵ is based on NFSv3 and is available to anyone on campus. It was built as a low-cost, reliable NFS service. It was not built specifically for high speed. Value Storage includes an option to mirror data between two locations and the mirror is updated daily and there are snapshots of data on disk. Backups are not included but are offered via ITS’ TSM service.

As we develop the components of the research working storage service, several may be suitable for integration with Value Storage.

For researchers who don’t require the level of performance provided by the research working storage service proposed here, Value Storage offers good alternative.

Many departments and research laboratories provision local storage and present that to clients via NFSv3 (for Linux or Mac clients) or CIFS (for Windows or Mac clients). Most of these storage services are small in capacity (less than 50TB) and low performance relative to the proposed research working storage service.

The advantage offered by local storage services is that they are usually a one-time cost that can be attributed to a grant as hardware. The disadvantages are that they are often not operated by people with operational experience in storage and that puts the data stored on these systems at some risk; these systems typically provide slow access to data because of their combination of networking (usually 1Gbps) and the number of disks in the system (usually less than 12); and these systems are often not expandable beyond a few tens of terabytes.

We expect that the combination of Value Storage, the proposed research working storage service and its backup and archival components, IT Rationalization with respect to staff, and the increasing requirements for long-term data management will lead to fewer and fewer departments or research laboratories providing local storage.

Much of the data at U-M that would fall under the new umbrella of ``big data’’ or ``unstructured data’’ (as opposed to relational data that is typically stored in relational database management systems like Oracle, MySQL, etc.) is currently stored where it is processed. In some cases this is in a Hadoop cluster, in other cases is it NoSQL systems and in other cases it is flat files.

The research working storage service will have the performance and capacity to ingest, store, and archive data from these systems as the current data is no longer needed but the space on the analysis platform is needed for the next research project.

As a data management support system, the research working storage service is an excellent complement to existing and future big data clusters.

The ITS TSM product¹⁶ offers tape backups of data from many sources, and maintains two copies in separate geographic locations. This service has historically been viewed as expensive, which it is, and a bad value, which, for the right data, it is not. However, there is a class of data on campus for which ITS’ TSM product is too richly featured and thus too expensive. The backups included in this research storage proposal are a very local, highly-integrated part of the service, and will not be offered as a generic backup service separate from the research working storage service. There will also be integration between the backups associated with the research working storage service and the archive, which is likely not appropriate for the TSM service.

In addition, we expect to make archive copies of data from backups, which is not supported in TSM today.

In the College of Engineering researchers have expressed interest to us in a web-based method of sharing data and collaborating with other researchers (especially those from other institutions for whom getting U-M credentials is inconvenient). The characteristics of this web-based data sharing, as we understand them, are around all control of the service being held by the researcher, including hardware and software selection (Windows, Linux, or MacOS; a forum, a wiki, a file upload/download service), maintainence of the access lists, data policies, and presentation.

The research working storage service described here would be suitable as the backing storage for a service like this:

• the performance of the storage would be sufficient to serve web-based requests
• snapshots and backups would offer some insurance against mistakes that would result in data loss were there only one copy
• the ability to archive the data at the end of the project without moving it would be nice
• the ability to have multiple, segregated storage areas (or “projects”) will help with data management

While the option of a Mac Mini, a Drobo and a CrashPlan subscription is likely to be less expensive than a service like this, the features offered by this service may be worthwhile from the perspective of data security and external data management requirements.