Research and Development

Data Versioning — by Dean Williams — last modified 2009-05-13 08:17

Versioning and change notification of the data holdings for ESG-CET came to the forefront as a critical need. A working group was established consisting of various ESG-CET stakeholders from the international community to define the use cases and requirements. The working group identified the core use cases for versioning data and notifying users of important changes.

Bulk Data Movement — by Dean Williams — last modified 2009-05-13 08:17

Data movement between nodes and data movement from a node to a client (user) site. We expect that each will require a different software tool. However, there are several scenarios that can use the same tool.

Data Replication — by Dean Williams — last modified 2009-05-13 08:17

A node publishes the original version of a dataset. This node has ownership of the dataset (or considered the authoritative source). A second node wishes to replicate the dataset, by copying the physical files from the first node and registering the replicated dataset with ESG and automatically updating from the authoritative source as things change. Access control of the authoritative source must be respected.

Metrics — by Dean Williams — last modified 2009-05-13 08:17

The Datanode requires a "black box" utility that will gather metrics data from all the different subsystems on a datanode. The "black box" will then act as the ESG Gateways contact point for the "Download Metric Data" Use Case. Using this "black box" will keep the ESG Gateway from having to contact all the Datanode subsystems directly. The ESG Gateway will then make requests to the "black box" so that the Datanode does not have to have knowledge of its parent ESG Gateway.