Description:

The Earth System Grid Center for Enabling Technologies (ESG-CET) project, a coalition of national laboratories (ANL, LANL, LBNL, LLNL, and ORNL), federal research centers (NCAR, NOAA/PMEL), and universities (USC) has worked to maximize the accessibility of climate simulation data by the international research community. Such access has been critical in supporting the efforts of the Intergovernmental Panel on Climate Change (IPCC), a co-recipient of the 2007 Nobel Peace Prize for its work in "disseminating greater knowledge about man-made climate change". To date, users from more than 3,000 registered analysis projects have downloaded more than half a petabyte of data (1,787,844 files). Their analyses have resulted in more than 500 peer-reviewed publications. Building on the successes of the Earth System Grid (ESG) project, which has enabled thousands of researchers to access hundreds of terabytes of data from a small number of ESG sites, the next-generation ESG-Center for Enabling Technologies (CET) project is working to integrate a far larger number of distributed data partners, provide high-bandwidth wide-area networks, and make available remote computers in a highly collaborative problem-solving environment.

Significance:

A consequence of increased computing power and more comprehensive climate models is a dramatic increase in data output describing the more complex spatial and temporal Earth system model. Over the past decade, model output has increased exponentially—moving from megabytes, to gigabytes, to terabytes, and now petabytes. The worldwide climate community is expected to generate hundreds of petabytes of simulation data within the next five to seven years. Using these data will not be easy. Indeed, merely examining or sorting data will be problematic, partly due to the enormous quantity of data involved, and partly due to the fact that the storage for all the data will be distributed worldwide. Far more than just climate model output data will be involved. For accurate study of the climate, in situ, satellite, biogeochemistry, and ecosystems data will also be required. Because of the high cost to generate the data (in both computer and human time), they must be carefully managed to maximize and preserve their value to the community.

Impact:

For several decades, scientists have been working to understand the Earth’s delicate ecosystem and intricate climate balance. Notably, they have sought an understanding of potential changes in these systems induced by naturally and human-generated greenhouse gases. In one of the largest-ever world-wide collaborative efforts in computational science, climate scientists have participated in the Nobel Prize-winning Intergovernmental Panel on Climate Change report. Their contribution was the completion and analysis of hundreds of massively scaled coupled climate simulations, making the virtual earth into a planet-sized laboratory experiment. The simulations produced an avalanche of data—scores of terabytes in almost two million files. Housing, managing, searching, and disseminating data on this scale was critical to the effort but had never before been attempted; the ESG project devised technology to accomplish just such data facilitation, and has since gone on to create a world-wide repository and access system handling data on a scale that dwarfs even the IPCC effort.

Resources and approaches used:

As part of the SciDAC program, the ESG Center for Enabling Technology has developed metadata technologies (organization, extraction based on netCDF, and Metadata Catalog Services), security technologies (web-based user registration and authentication and community authorization), data transport technologies (for high-performance access, robust multiple file transport and integration with mass storage systems, and support for dataset aggregation and subsetting), web portal technologies (to provide interactive access to the data holdings), as well as popular analysis and visualization tools.