Reference Frameworks for Assessing Maturity of Earth Science Data Products: Part 2

Abstract/Agenda: 

When managing and providing effective stewardship to digital Earth Science data products in the era of Big Data and Open Data, reference frameworks for assessing dataset quality are key components in helping with the major challenges of consistently describing and systematically integrating quality information to diverse audiences of data users.  The natural evolution of user base and data tools over time require continuous monitoring and flexible--but consistent--methods of rating and signaling data quality and maturity.

This workshop consists of two sessions.

The first session ( (http://commons.esipfed.org/node/9036) will bring together a panel of experts in the fields of exploring quantifiably measurements of product, stewardship, and service maturity levels for digital data in order to help data managers, data producers, or institutions to identify the strength and gaps in their procedures or practices applied to individual data products.They will examine the current states of the cutting-edge research, following by Q&A and panel discussion.

The second session will provide updates on use case studies of various maturity assessment models. It follows by training/working time for attendees to get familiar with and apply the assessment models, including the data stewardship maturity model, to their datasets.

Use case study update presentation focus areas
Shelley Stall - AGU Data Management Assessment Program using the Data Management Maturity framework use case study
Sophie Hou and Ruth Duerr - ESIP DSMM use case study
Nancy Ritchey and Ge Peng - NCEI DSMM use case study

AGU’s Data Management Assessment Program Use Case Study

AGU conducted two assessments as part of the introduction and piloting of the Data Management Maturity (DMM) framework.  Each organization assessed has different data management objectives in support of their organization’s strategic goals.  We’ll discuss the flexibility of the DMM Framework, the assessment approach used for each organization, and the results of each assessment. 

Pilot #1: USGS ScienceBase Repository - Data Release Team

Pilot #2: The Biological and Chemical Oceanography Data Management Office (BCO-DMO) Team

The purposes of the ESIP Data Stewardship Maturity Matrix (DSMM) Use Case Study

  • Evaluate and improve DSMM and generalize its across-disciplines application
  • Evaluate and improve the DSMM self-assessment template
  • Develop community-wide dataset-centric stewardship requirements and guidance

The purposes of the NCEI Data Stewardship Maturity Matrix (DSMM) Use Case Study

  • Demonstrate the utility of DSMM, ensure and improve consistency of its application to various NCEI data types
  • Establish baselines for high-utility/impact NCEI core datasets
  • Explore requirements for collecting and capturing content-rich quality metadata
Notes: 
  • The presentations from the panel are attached.
  • Presentation #1: "AGU’s Data Management Assessment Program: Pilot Out-brief" by Shelley Stall of AGU
    • Two AGU DMM assessments were completed:
      • USGS ScienceBase - Data Release team
      • The Biological and Chemical Oceanography Data Management Office (BCO-DMO) Team
    • Shelley shared the key quotes from the actual team members who participated in the assessment process, so that the attendees could get a sense of the assessment experience.
    • Shelley also highlighted the lessons learned from the assessment process to demonstrate the possible outcomes from the assessment process.
  • Presentation #2: "Data Stewardship Maturity Matrix – Use Case Study" by Sophie Hou of the National Center for Atmospheric Research (NCAR)
    • Lessons learned and recommendations are presented based on the experience of using the Data Stewardship Maturity Matrix to assess two different use cases (one from the Research Data Archive at NCAR and one from Santa Barbara Costal of Long Term Ecological Research)
  • Presentation #3: "ESIP Use Case Update: ACADIS Data" by Ruth Duerr of Ronin Institute
    • The key objective of the assessment was to determine if the data stewardship maturity matrix was applicable to long tail data.
    • The value of a data set is often being asked, but it is not always easy to define "value".
      • However, there are several related areas that need to be considered:
        • Preservation readiness, fitness for purpose, and number of potential user communities.
    • The value of a repository is also difficult to define.
    • For the context of Ruth's assessment, the following terms have specific definitions, and these definitions are included in Ruth's presentation:
      • Curation, preservation, archiving, and storage.
    • ACADIS = Advanced Cooperative Arctic Data & Information Service: includes highly inter-disciplinary datasets.
    • Preliminary assessment of ~40 datasets was completed.
    • Currently, the datasets from ACADIS are in the process of being migrated to the NSF Arctic Data Center.
    • The new NSF Arctic Data Center provides bit-preservation services, requires ORCID, and is being backed up by NOAA.
    • However, potentially, the same maturity issues might still exist for the NSF Arctic Data Center. As a result, it could still be beneficial for the NSF Arctic Data Center to apply Data Stewardship Maturity Matrix for its assessment.
  • Presentation #4: "NOAA’s National Centers for Environmental Information (NCEI)" by Nancy Ritchey of NOAA’s National Centers for Environmental Information (NCEI)
    • NCEI pilot use case study is going well
    • NOAA OneStop Project is requiring DSMM assessment to be a part of OneStop-ready
    • Key features of the Data Stewardship Maturity Matrix were presented including the following links to important resources:
    • DSMM Quick Startup Guide: tinyurl.com/DSMMguide
    • DSMM Self-assessment Template: tinyurl.com/DSMMtemplate
    • DSMM Quick Graphics Tool: tinyurl.com/DSMMgraphics
    • DSMM  Introduction: tinyurl.com/DSMMintro
    • DSMM One Page: tinyurl.com/DSMMslide
    • DSMM Paper: tinyurl.com/DSMMpaper

Questions on usefulness and level of effort of utilizing maturity assessment models were asked. It has been indicated by the DSMM use case study presenters that it is useful. According to the presenters and Peng, the level of effor depends largely on familiarity and experience of evaluators with DSMM and availability and readiness of the information needed for assessment.

Citation:
Peng, G.; Duerr, R.; Hou, S.; Robert, D.; Reference Frameworks for Assessing Maturity of Earth Science Data Products: Part 2; 2016 ESIP Summer Meeting. ESIP Commons , February 2016