NetCDF, HDF and ESRI

Dan goes through introductory slides...

Switch to demos...

Browsing NetCDF data in ArcGIS as raster layer. Version? Not sure.

Q: IPCC data has lon convention 0 to 360, does ArcGIS handle that?

A: if data is CF compliant, yes it is handled.

Q: Advantages of using Arc over Environmental Data Connector extension

A: (Dan) not aware of that

Q: Can you aggregate multipe files for temporal resolution

A: Don't handle that yet

Next demo : as feature layter, can access attribute info

Last: table view of data

On to HDF format, loading MOD16 evapotranspiration product, uses GDAL libraries.

Q: Is NetCDF 3 supported

A: yes

Debate and commentary over internal file structure of the HDF file loaded.

Q: Can you show HDF-5 products?

A: would like to get some good examples to our development team

Back to slides for visualizing mosaic datasets

Back to demos...

Showed use of python scripting interface to get an HDF layer (variable)

Emphasized on-the-fly processing (only applies to pan-and-zoom scene)

Demo applying a function (shaded relief) to sea height data (bathymmetry?)

Demoed an arctic sea-ice concentration animation.

Commentary on NDVI functionality, whether benchmarks are available, Dan commented on the fact that ESRI works closely with ENVI's parent company.

Q: Does this work only on local NetCDF data, or can it do remote?

A: Only local, remote in the works.

Irony: a discussion started on NetCDF 5 and someone had to point out that was actually an April Fools joke started by John Caron.

Demoes sharing NetCDF data - deploying to a server

Time animation tool only works with fixed-interval temporal res.

Attempted to demo some final highlight capabilities with NetCDF and animations, but I didn't see much success or animation here.

 

Kelly's notes

 

NetCDF, HDF, and ESRI

How ESRI support NetCDF and HDF within their system.  Presentation will mainly be demonstrations

·         NetCDF came out of modelers and climate forecast

·         Difference between how access and use each format

·         Talking about working with data only that you need from the system

o   There are ways to pull out only the types or data that you need

o   This is not yet supposed - OPenDAP

·         CF (Climate and Forecast) Conversion

o   Provides geographic grid functions

·         Accessing Data – ArcMap

o   NetCDF can be access as a feature, table, or raster

o   Can access raw point data

o   “multidemension tool box” – available with all licenses

o   Q – IPCC has released NetCDF – long is 0 to 360

§  yes

o   Q – what does NetCDF raster tool require for the file

§  Try best to read the file, but can be some issues

§  Because CF is a convention things are implemented differently

o   Q – what are the advantages Environmental data connector from EPA (sits on ESRI)

§  Built by ESRI – did not know – will look into it (ACTION)

o   Q – can you select multiple NetCDF files

§  Not yet –working with THREDS

o   Can create points and tables of the grid data

o   Q what is the advantage of viewing it as a feature layer

§  Grouping analysis later and other analyses

·         HDF (no tool box)

o   Working in rudimentary (gdal library is installed)

o   Q what veriosn

§  Version 10.0 and above

§  HDF 4 and above

§  As raster development team looks at HDF more – HDFui (sp?)

o   Q for NetCDF use NetCDF library

§  Yes, but need to work with multidimensional tools

§  For HDF use raster tools

o   Q - May I know if this is netCDF 3 data?

§  Yes

o   Did you try to open a netCDF 4 enhanced model data?

§  Data working with is “classic”

o   When adding HDF there may be multiple options, select one to view

o   Q this is a lat/long array

§  This is a grid – not swath data

§  Question as to if the data was gridded (Aleksandar thinks so)

§  Not sure they read swath yet

o   Can we load HDF5

§  But none of them line up projection wise

o   NOAA’s BAG is HDF5 which is supported via Nautical solution

§  It is read through the gdal library

·         How to visualize products

o   SST data

o   Can animate the data because it has time dimension

o   Q course boundaries

§  Because of interpretation of the data

o   Currents – want to display as vectors

§  Not supported directly

§  Q – how with X and Y or magnitude and angle

·         Thinks it will be magnitude and angle

·         Alek suggests both

§  This model process was put together from multidemsion tool book tools

§  Result of the model has a feature class, then angle, and length

§  Q did you calculate the angle

·         From the data – want to be able to calculate it on the fly from the data directly

o   HDF through Mosaic Datasets

§  Also used for on the fly visualization

§  Can use a script to write a table of the sub-datasets (Ex. Evapotransportation

§  Q sub-datasets

·         It is a variable – it is a layer within the file

§  Python will look for environmental dataset and then store this information as a metadata record as a table

§  This can then represent visually

§  Each file is a time set – so this is a spatial aggregation

§  Can run in the background so that you can see what is happening for the presentation

§  Q this was a description operation not a subset

·         It is analogous to creating a THREEDS catalogue (in some ways)

§  Raster types are specific (format + data type) – it can be challenging because of the variability that exists

§  This process created the metadata before to help read the data

§  Gap filling

·         Raster functions – these are functions that only apply to the fnction as you pan and zoom

·         Ex. Add a statistic function to fill gaps (it won’t work… there is one online that works)

§  Q what is the data set on Australia

·         Evapotranspiration that is not stretched (have to zoom in to see features)

§  Created a shaded relief and in memory there is a source data of the function applied

§  Can then apply an animation

o   Animation

§  Sea-ice – animation with shaded relief

§  NetCDF data with different projection and is time enabled

·         If published as a service then time slice can be moved

§  Q what are the element that are being mosaics, time or spatial fragments

·         It is time

·         This is basically a THREED aggregation across a time dimensions with a function applied to it

§  Can create cool animates from it (really)

§  Q since arc9.X the enemy of “envy” calculations – the benchmarks outperformed

·         Not sure if it has been tested

·         ESRI has been focusing on services

·         Envy was an array processor so it was fast

·         Now provide something like a snapshot view

§  Q does this work on local vs. open

·         Only local

§  Q do you have a schedule for the development activity

·         HDF and NetCDF are not scheduled for next release but possibly for the next one

§  ESRI is working on automatic updates – works with dashboard – this facilitates more updates and helps with new feature development

§  Q is there support for other formats

·         GRIB – with gdal

·         Buffer – not heard of anything

·         (these are old and hence not well supported) – partly because of the tables that include the variables

·         Heard of GRIB to NetCDF converter (in Java – NetCDF library)

·         For GRIB2 there is a wGRIB code that maps between GRIB and NetCDF directly

§  Q does this rely on any conventions (CF?)

·         Heavily rely on CF convention

§  Q can it handle 3D data

·         Some of the 3D views will read the 3D aspects of the netCDF data

·         Such as ArcGlobe and ArcScene

§  NetCDF extended model is used more than the classic in the NetCDF4

·         NetCDF4 classic is old

·         Talked about getting groups into NetCDF classic issue

o   Climate data w temperature variation for comparison

§  One file from 1895 – 2100

§  Split into 2 files and then compare

§  Create a 30 band raster

§  Use cell stats to get statistics to compare the two

§  These are out the box tools

o   Python is the fastest way to get information into ESRI system

§  Might want to use the python version of the HDF library

o   How to share NetCDF and HDF data

§  Can deploy as ArcGIS Server

§  Q does the projection matter

·         Depends on where deploying it – but anything can be used

§  Q problem with irregularly spaced time extents

·         Need to have regular time intervals

·         Have a tool called timeliner toll for ArcGIS

o   Desktop add-in

o   More capability for accessing time with your data

o   It is free

o   Can provide link

§  Safesoftware has full support (FM2) convert NetCDF files