Earth System Grid I (2000-2001)

Participating Instituions

ANL, LANL, LBNL, LLNL, NCAR, ISI 

Sponsor and Program

• Dept. of Energy
  
• Next-Generation Internet (NGI) Program
• Follow-on support from:
  
• OBER
  
• DOE’s Mathematical, Information, and Computational Sciences (MICS) office

Accomplishments

In this prototyping project, we developed Data Grid technologies for managing the movement and replication of large datasets, and applied these technologies in a practical setting (an ESG-enabled data browser based on current climate data analysis tools), achieving cross-country transfer rates of more than 500 Mb/s. Having demonstrated the potential for remotely accessing and analyzing climate data located at sites across the U.S., we won the “Hottest Infrastructure” award in the Network Challenge event at the SC’2000 conference. The ESG began in 2000 with the "Prototyping an Earth System Grid" (ESG I) project, initially supported by the DOE's Next Generation Internet program, with follow-up support from OBER and OASCR. In this prototype, we developed Data Grid technologies for managing the movement and replication of large datasets, and applied these technologies in a practical setting (i.e., an ESG-enabled data browser based on the Climate Data Analysis Tools (CDAT)), achieving cross-country transfer rates of more than 500 Mb/s. Having demonstrated the potential for remotely accessing and analyzing climate data located at sites across the U.S., we won the “Hottest Infrastructure” award in the Network Challenge event at Supercomputing 2000 Conference.

Earth System Grid II (2001-2006)

Participating Instituions

ANL, LANL, LBNL, LLNL, NCAR, ORNL, ISI


Sponsor and Program


• Dept. of Energy
  
• Office of Science
• Advanced Scientific Computing Research (ASCR)
  
• Scientific Discovery through Advanced Computing (SciDAC) program
• Network Collaboratories
  

Accomplishments

While the ESG I prototype project substantiated a proof of concept, the SciDAC Earth System Grid II project (“Turning Climate Datasets into Community Resources”) made this a reality. Our efforts targeted the development of metadata technologies (standard schema, XML metadata extraction based on netCDF, and a Metadata Catalog Service), security technologies (Web-based user registration and authentication, and community authorization), data transport technologies (GridFTP-enabled “OPeNDAP-G” for high-performance access, robust multiple file transport and integration with mass storage systems, and support for dataset aggregation and subsetting), as well as Web portal technologies to provide interactive access to climate data holdings. In 2003, a DOE-appointed external review committee praised these achievements, stating that “ESG is poised to make a substantial impact on the climate modeling community as well as many other data-intensive science disciplines.” Following this statement, the climate modeling community began to see ESG in operation with the cataloging of the CMIP3 (IPCC AR4) archive, in 2004. Since this time it has become in integral part of the way climate scientists expect to carry out their work. At the end of the project, the ESG enterprise had approximately 4,000 registered users and 250 TB of data holdings.  It had delivered 130 TB of data to users.




Earth System Grid Center for Enabling Technologies (2006-2011)

Participating Institutions

ANL, LANL, LBNL, LLNL, NCAR, ORNL, PMEL, ISI (new additions in italics)

Sponsor and Program

• Dept. of Energy
• Office of Science
• Advanced Scientific Computing Research (ASCR) and
  
• Biological and Environmental Research (BER)
• Scientific Discovery through Advanced Computing (SciDAC) program
• Centers for Enabling Technologies
• Computer Science CETs
  

Goals

The Earth System Grid Center for Enabling Technology (ESG-CET, "Scaling the Earth System Grid to Petascale Data") is focused on scaling up the current ESG system to meet the needs of primary customers. Including:


• The Coupled Model Intercomparison, Phase 5 (CMIP5) Intergovernmental Panel on Climate Change (IPCC) 5th Assessment Report (AR5), to be delivered in 2013,
• The SciDAC-2 climate application: A Scalable and Extensible Earth System Model for Climate Change Science,
• The Climate Science Computational End Station (CCES),
• The North American Regional Climate Change Assessment Program (NARCCAP), and
• Other wide-ranging climate model evaluation activities.

Future Accomplishments

The Earth System Grid Center for Enabling Technologies (ESG-CET) is a follow-on to the Earth System Grid (ESG) collaboratory, which was funded as part of the SciDAC-1 program, 2001-2006.  In 2004, the National Center for Atmospheric Research (NCAR), a lead institution for the Community Climate System Model (CCSM) modeling collaboration, began its first publication of climate model data into the ESG system, drawing on simulation data archived at LANL, LBNL, NCAR, and ORNL. Late that same year, the Program for Climate Model Diagnosis and Intercomparison (PCMDI), an internationally recognized climate data center at LLNL, launched a production service providing access to climate model data germane to the Intergovernmental Panel on Climate Change (IPCC) 4th Assessment Report (AR4). Exploding from its initial launch in 2004, ESG is now responsible for hundreds of terabytes of climate data ranging from high-resolution modeling, grand challenge-scale computations on leadership computing systems, regional climate modeling, coupled climate/carbon cycle modeling, land/biosphere modeling, atmospheric chemistry modeling, detection and attribution of climate change, and model intercomparison projects. The next generation ESG Center for Enabling Technologies (ESG-CET) is positioned to support petabyte datasets in a distributed environment through the federation of data centers.