HDF and Services
HDF has long been used to support data visualization and analysis in desktop environments with tools developed by scientists and with open source (Panoply, Octave, ..) and commercial tools (IDL, ENVI, MATLAB, ESRI). As datasets become larger and more complex, it is becoming important to support remote access and analysis using a variety of tools. This approach has been used extensively in the Earth Science community for access to metadata (ncISO), and data (THREDDS, OPeNDAP, ENVI Services Engine). This session will focus on how these services can be used to access HDF data, new services and improvements to existing services and, equally important, how they can be used to augment metadata in legacy HDF files to support access with multiple conventions.
The HDF group part 4 of 4
HDF and Services
Kent Yang (HDF) – HDF OPeNDAP update
· Access HDF via OPeNDAP – hope users access HDF files via OPeNDAP
o HDF Handlers (CF) – CF enhancement
· Handler update
o Add TRMM 7 support & improve performance (include AIRS 6 grid and MOD09_M3)
o Performance – handle MODIS no-CF scale and offset rules
o Reduce # of files open/close calls when using NetCDF modules
o Map HDF4 char array to DAP string
· For MOD09_M3 and AIRS version 6 performance
o Disable StructMetadata as a DAP attribute for CF option
o Use special features in products to build DDS and DAS
· Examples – change from swath to grid (level 1 to level 3)
· HDF5 handler update
o GPM level-1 product – just released 2 days ago – already served by handlers
o GPM = global precipitation measure
· Challenges
o CF convention evolve and tools evolve
o New versions
· Future – performance
o Reduce the DDS and DAS access time in general
o Provide a caching mechanism in the handler level
· Q = do you have opendap urls where these handlers are currently employed in operational use
o There are several – will provide offline
Hyo-Kyung Joe Lee and Ted Habermann (HDF) – Adding new services for HDF in THREDDS Data Server (TDS)
· Would like to open large HDF file X with tool Y on platform Z… mobile platform is not Z, can’t run Y, and can’t open X
· All mobile platforms provide a web browser – even the nook can open data then
· Rn web server on Z, server executes command Y on X send output over the web – looking at THREDDS
· Why THREDDS – portable, 100% java, …
· How to add a new service:
o 1 create a yservicecontroller class (e. H4MapService Controler.java)
o 2 handleRequest() method invokes runtime.exec(command)
o 3 read stdout and build a string outStr
o 4 return ResponseEntity<string>(outStr,…)
o 5 set up catalog for y service for X data
o Run the custom TDS on Z
· Used h4map, h5json, and ncdismember – TREDDS can run the commands and then provide the service
· You can develop your own XML/JSON apps on top of the new HDF web services
· Future
o Supply command line options – haven’t yet determined how to supply command line to the service
o Generalize CLI services through Catalog
o What other HDF services do you want?
· Q = there is only one active compliance checker for CF – can we use this for netcdf files eventual
o Yes for Netcdf4
· Demo – HDF5 Product Designer
· Q = user can choose different variables and delete some – yes and then generate HDF5
· Ted – all HDF4 product reservoir for designing HDF products – ingest them in designer and then output HDF5 that new datasets can be added – reopening conversation about the translation process – Kent talked earlier about augmenting the metadata during translation process
o When translating from HDF 4 to 5 can make things conventional
o Friday bring ideas of how to put things together
·
Thomas Harris (Execlis Vis) & Joe (HDF) – HDF and ENVI Services Engine
· ENVI Services Engine (ESE)
· ENVI LiDAR app program interface (API), using IDL and ENVI
· Scientist can take IDL code, publish it to ENVI service engine (HDF deploys into their own enterprise), then analytic run and returns display
· How serve users better
o Try IDL codes without purchase and installation
o Run everything with web browser – even mobile platform
· Need input
o Purchase ESE and use existing IDL code
o What kind of web services
· Q = is there some way to search the product to see what you provide
o ENVI server engine – leverage existing services
o Ex. Delivery of product – need way to leverage the product
o Plug THREDDS with ENVI services engine
· Q – ncdismember service part of..
o Extracts all your groups and then sends them to compliance checker
o Working on definition of “compliance” with CF – when you have group it is zero – trying to redefine
· HDF is foundation of scientific data formats in many disciplines