Airborne Data Management

Abstract/Agenda: 

Airborne in-situ and remote sensing measurements are essential elements of the integrated Earth Observing System (EOS) and provide key insights into Earth processes. Over the past two decades, aircraft data have grown remarkably in both volume and measurement types. While often used in validation and assessment of models and satellite measurements, airborne data are under-utilized largely due to challenges unique to airborne data management. These include data hosting at various locations and in different formats as well as limited spatio-temporal coverage that affect how data centers organize and distribute the data. Recently, data centers, airborne science teams, and user communities have made significant progress to enhance the accessibility and usability of the airborne science data. This session will present and discuss airborne data management challenges; as well as progress made, experiences, ideas and lessons learned in enhancing airborne data systems. The goals are to increase awareness, promote, enhance utilization of data management capabilities and foster collaboration among developers to further enhance these efforts.

Notes: 

July 10 2014 ESIP Summer Meeting

Airborne Data Management session notes

 

Airborne Snow Observatory - Maziyar

  • 2 sensors - Lidar (laser pulse), advanced  light sensor (spectrometer)

  • requirement : process data within 24 hours

  • data volume:  this year 200-300gb / flight (last year 60GB)

  • parallel processing - intermediate file from LIDAR used in CASI processing

  • IRC and “robot” used for communication among team.  Robot reports on processing and software

  • Availability of data products - raw data kept at JPL archive; intermediate products kept by science team; L3 and L4 data/maps published to ASO web site

  • Availability of metadata - need to go back to science team to determine what is collected and how it’s stored

 

NOAA Tropo Chem Airborne Data Archive - Ken Aikin

  • file format:  ICARTT text and Igor binary

    • good comments on ICARTT

  • web site - esrl.noaa.gov/csd/groups/csd7/measurements

    • get individual files or groups in one click; dynamically zipped for download

    • can also get all of a specific parameter for all flights (another page)

    • click through data policy statement to get to data

    • some online documentation generated from ICARTT header

  • listed in GCMD

  • considering netCDF, DOI, online vis (dynamically plot different variables)

  • Recommendation (Ramapriyan) - It would be very useful to include, in a prominent way, information on how to cite the data. See http://commons.esipfed.org/node/308 for ESIP Guidelines for Data Citation.

 

LaRC toolsets for airborne data (TAD) - Aubrey Beach

  • use ICARTT to address inconsistent variable naming  convention issue

  • data tools - temporal merge, average

  • future - allow PIs to upload data to TAD database, vis tools

  • good documentation and metadata/provenance info for on-demand post-processing?

    • list of source files included in output file

    • merge/average processing description not available (never requested)

  • How long does merge process take?  depends on time interval, variables, etc., up to 15 min.

  • Vis tools? plot flight tracks, species over a region, correlation scatter plot

  • Gao notes documentation is uneven across missions and instruments

  • Common airborne data naming convention?  But some instruments have their own heritage naming conventions

  • Common variable names?  currently mapping PI vars to common vocab (manual process)

  • ICARTT issues?

    • enforcement of conventions / recommendations

    • revisions to make more machine friendly

    • tech support

 

Relationship between ESIP cluster and ESDSWG activities

  • reach out to airborne science teams beyond NASA

  • work toward information architecture for airborne data systems

  • not official ESIP cluster or WG at this time

 

 

 

Actions: 

RECOMMENDATIONS FROM ESDIS

  • document state of practice among airborne missions

  • create recommendations for airborne science

    • data formats

    • metadata

    • file naming conventions

    • documentation

    • citation

 

 

  • reach out to airborne science teams beyond NASA

  • work toward information architecture for airborne data systems

Attachments/Presentations: 
AttachmentSize
File ESIP airborne data archive presentation 2014_final.pptx4.7 MB
File ESIP-TADTalk-v5.pptx4.05 MB
Citation:
Law, E.; Chen, G.; Conover, H.; Airborne Data Management; Summer Meeting 2014. ESIP Commons , April 2014
Submitted by elawsesip on 2014-04-01 12:55.