Product Designer / HDF View / HDF Compass
In the past, creating HDF files has required downloading libraries and writing code in some language supported by The HDF Group or the HDF community. This approach has served the community well for the development and creation of many products. Recently, as part of the ICESat2 mission, Jeff Lee developed and implemented a different approach that allows science teams to create suites of related products and share information across those products using a web interface. In addition to creating HDF files without programming, his tools generate code in IDL and FORTRAN that can be used to create products in production systems. The HDF Group is working with Jeff to generalize this approach. We need users to help us test this capability and to provide use cases to drive future developments.
The HDF Group is also evaluating future directions for our popular generic tool for viewing HDF files (HDF View). We have been working with a prototype developed by Andrew Collette in python that features a robust plug-in architecture for viewing your own datasets. We are looking for partners with data that can help extending the capabilities of this tool.
The HDF Group Part 2 of 4
Product Designer/ HDF View /HDF Compass - Ted
· HDF view is the most popular download from the site & thinking about the future of tools
· Will talk about each tool
· Original designers are here in the room
John Readey (HDF Group) – HDF Studio
· Background – started at IBM – learned about HDF in relation to Intel Arrya visualizer, then amazon… joined HDF group last month – tools and cloud technology
· HDF Studio
o Server and client software components that enables visualize and share data
o Support large structure datasets
· HDF View – most popular download – what are the limitations as a platform
o Only see data on your desktop (don’t see server data)
o Can great graphs but can’t keep the graphs (can’t save it to share later)
· HDF Studio architecture
o Server in the cloud
o Define a high fidelity REST api of HDF5 for read-write
o HDF5 library would work with data that exist in the cloud – leverage existing tools
§ Python scripts – talking to – H5py – to HDF5 lib – talking to – data in cloud
o Able to open a file & save it to the cloud – anyone can see the change – more collaborative way to share and discuss data
o Web apps mean can create browser based apps to view and modify data
o Support OPeNDAP (it is a read only protocol)
o Community repository – different conventions for viewing data – HDF group will not fully support all options – others can create add-ons/views for their conventions
· Challenges
o Controlling who can modify what … sharing or private data
o Reliability – amazon expects 99.99% availability or “uptime”
· Just getting started… comments, idea, areas of collaboration?
o Will release base components and then release updates
· Q –does the REST api – HDF5 lib to HDF5 server exist
o Not – meaning it would automatically become cloud enabled
o Community repository – way of sharing code related to data
Joel Plutchak (HDF group) – HDF-Java Overview
· HDF Java timeline - Java came out in 1996, JHV & JHI – viewer and other in 1996…
o Summer 2002 – common object model between HDF4 and 5
· HDF-Java Architecture
o Java api – then common object package – then format specific java native interface (JNI)
o HDF View – I/O modules – there are other modules - there are also various GUI modules
· HDF View – includes a tree view (datasets, group, attributes), table view, histogram, image view, and properties
· http://www.hdfgroup.org/products/java/
· Interest in hearing from people who use Java or HDFView
Aleksandar Jelenak (H. Joe Lee, Ted – HDF Group) HDF Product Designer
· HDF Features and then have interoperability issues then 3rd part is mission requirements
· HDF product designer organizes HDF5 file content to data and metadata
o Create and re-use parts of file content, collaborative, HDF5 template in several programming languages
· Process – create and organize content in program – saves continuously
· After designing a file – can export a description in text (open in Excel) – can be imported back into Product Designer (after edits in other programs)
· Can create a template file can be created – or code to generate write code (FORTRAN or IDL) – can be edited to fill in real data
· Benefits – it is user friendly – interoperability is addressed in design phase
· Future work – only started this about 2 months ago (looking at original application) – big question is desktop or website app
o Change from MySQL to SQLite
o Generate portable/free Python and h5py
o Convention compliance checkers (CF, HDF-EOS) and interoperability checkers (netCDF-4)
· Local vs. central application – looking at dual mode – like email software
· Looking for partners – contact [email protected] (new missions, HDF4 to HDF5, or reprocessing data for interoperability, use cases)
· Q – mentioned collaborative design process
o One of the cons of local option is how to be collaborative
· Ted – looking at options of online folders that would be available online and then location folders for individual products
· Q – would there be version control? - yes
Andrew Collette (NASA) – HDF Compass – a new viewer for HDF and related formats
· Why
o It is a highly focused viewer tool
o Native UI - there is little integration and operating system
o Cross-platform
o Easily extensible
· HDF Compass – writing in python - there are 45,000 open source packages on the package index
o Including contour and vector plots – there are existing libraries for lots of the work (HDF5, HDF4)
o Attached to a community that work with these datasets – also tests and supports
o Package that lets you freeze a python – just get a ‘exe’ for windows (for example) – don’t need to install python to use it
o PLUGED IN – OPeNDAP, HDF, other format
o There is a layer between the app and the data layer – (interface is only 4 pages) – able to replace HDF5 with other plugin layer
· Demo of using HDFCompass in mac OS10.x
o View is different than HDFView
o Metadata is displayed on the left of the screen for each file in the list
o In the interface – (uses container class – view makes the nice display) – click to open dataset
o Spreadsheet style viewer – doesn’t load all the data (wxpython) – loads data as you need it
o Button in top right that plots the dataset
§ Opens another window
§ Can save (get save dialogue) – plug in includes formats included (not written for compass – all python plug ins
o Able to easily browse through data that scientist gets
o If right-click on file – can look a list of attributes
· Demo – now in windows
o Things are moved around a bit – windows opens things differently
· Back to macos10 –
o Browse like a file system browser
· Ted – one of the cool thing about HDFCompass is that it looks like a directory system
· Can open a file with a million groups in about 10 seconds
· Q – Mike – common to think of HDF as a group of directories or a multigraph – this approach lends itself to a multigraph
o Meaning multiple links to things – can hard link multiple things together – use tree structures
· Q – Mike – HDF view – breakthrough was bundle JRE together – looks the same with compass
o don’t need to install python on machine
· Q – Openstack experiments – this frozen idea sounds like a perfect fit – what kind of license to create a plug-in for HDFCompass
o Depends on how HDF group license – likely free
o If want to ship a plug-in, if requires binary complied thing – able to add things to python path – able to ship a zip file with all of it
· Q – do we support compound data type
o All numpy + a few addition – YES, compound and structure types and variable length types