Building Catalog-Driven, Web Service-Based Applications


Due to advancements in web services like CSW, OpenSearch, CKAN, WFS, WCS, SOS, WMS, OPeNDAP and Esri REST, the environmental data community can now build applications that start with a catalog search, extract web service endpoints from discovered datasets, access data or map products from the endpoints using standard tools, and create products customized for their users. Although different tools are needed for emergency responders, environmental managers, scientists, engineers and software developers, all can benefit from the advances in the standardized infrastructure. In this session we will share success stories, challenges, and lessons-learned building these catalog-driven, standards-based applications.


Building Catalog-Driven, Web Service-Based Applications


Lightweight Web Applications using Ajax and Standard Services

John Relph, NCI

  • The challenge:

    • general purpose search and discovery interfaces may not be good enough to specific communities

  • geoportal: indexed metadata

    • search terms are based on available info from ISO metadata

    • case study: ocean acidification data portal

      • have some variables that they’re interested in, made an interface to allow user to select variables

      • looks like geoportal search but to support this particular community

      • uses a variety of systems federated to a single system

      • URLs at bottom right reproduce this search precisely as it was entered

  • conclusions

    • uses std NCEI search and access database tool, the “collection” geoportal, no new search database need be configured

    • uses existing services from NCEI and other providers

    • user interface handles community-specific needs

    • working demo of the interface took literally hours to construct

    • little or no security impact to deploy

    • model for NOAA OneStop Data Discovery and Access Framework


Catalog-driven workflows using CSW

Rich Signell, USGS @ Woods Hole

  • case study: Boston Light Swim water temperature

    • created Python notebook for seawater temperature that’s available and can run on the cloud

    • power of CSW:

      • can put into geoportal endpoint and see what comes back

      • find all observations we want to process, plow through those → results

  • you can use these things for your own research group

    • helped set up pycsw for a group

  • workflow (see slides)

  • CSW issues

    • if queried pycsw and geoportal, the bounding box if given just long-lat gave wrong results

      • solution: specify CRS 84 and then you can use geoportal and python

  • can use tool like TerriaJS (infrastructure for building webportals with CSW search ability)

    • different datasets can be just configured

    • or you can say here’s my csw search, here are my parameters, and it will dynamically populate those items

  • benefits of standards-based, catalog-driven, reproducible workflows

    • find the real problems

      • easy problems that can be fixed in minutes to a day

      • harder problems to guide future work

    • fixes for specific workflows benefit everyone

    • build success stories

    • create reproducible workflows that others can learn from, expand on, or transform

    • standardized workflows help develop the 4th network layer for data


Building catalog-driven, web service-based applications with OpenSearch

Doug Newman, NASA Earthdata - Raytheon

  • What is OpenSearch?

    • collection of simple formats for the sharing of search results

  • Why you should use it

    • APIs are simple

      • http request in, atom xml out, client could be as simple as your browser

    • standardized

    • self-describing

      • the osdd describes your search API

      • results describe how to: navigate, drill down, browse, obtain metadata, obtain data, manipulate data (OPeNDAP)

    • sources can be aggregated

    • sources are legion (Doug is tracking 20 Earth data sources at the moment, and that doesn’t include CWIC)

    • OpenSearch is global

  • How you can use it

    • checklist: get your OSDD, construct interface/formulate search, discover the data, navigate through your results, drill down on your results, evaluate your results, download your data

  • Who’s using this now?

    • Earthdata Search

    • CWICSmart

    • FEDEO Earth Observation Portal


CEOS WGISS Integrated Catalog (CWIC)

Lingjun Kang

  • CWIC data partners are all over the world

  • By last November, 1680 datasets in CWIC

  • Question: plan to switch from CSW to OpenSource in the future?
Li, Y.; Signell, R.; Relph, J.; Building Catalog-Driven, Web Service-Based Applications; Winter Meeting 2016. ESIP Commons , October 2015