Challenges and Progress in Search Relevancy

Abstract/Agenda:

As dataset diversity and data volumes continue to increase, providing users with the interfaces, tools and services they need to discover relevant datasets creates new challenges and opportunities for the improvement of search relevancy and search engine result ranking. Diversity of user communities is a challenge as well, given that relevancy depends on specific user types and needs.

In this session we will report on projects and activities to improve search relevancy from the perspective of finding and utilizing Earth science in-situ, satellite and model data. We will explore search relevance on both the dataset and granule levels, dataset relationships/dependencies, semantic relationships, data quality, user characterization and content based ranking.

Specifically we will report on search relevance activities and results ongoing in NASA, NCAR and other organizations with the goal of building synergy among community experiences, and developing strategies to improve search relevance and user experience across the entire spectrum of Earth science data and data users.

PRESENTATIONS:
1) NASA Progress in Search Relevancy
Edward M. Armstrong, Lewis McGibbney, Kim Whitehall
NASA Jet Propulsion Laboratory

Recently the NASA ESDSWG on Search Relevance concluded its first year of activities to address search relevance across the 12 NASA earth science data centers. It was originally proposed to characterize the term search relevancy as it relates to ESDIS, to assess the implementations that address search relevancy, and to determine how existing implementations can be improved and new approaches be enacted. Individually and collectively, the group sought the expertise of persons within ESDIS, industry and academia. Five core subgroups (from an initial collection of ten) were organized on the topics of Spatial Relevance, Temporal Relevance, Dataset Heuristics, Dataset Relationships, and Federated Search:

Spatial relevance; This subgroup aimed to provide direction on substantiated metrics on methods to define spatial overlap in searches with the purpose to improve relevance ranking based on dataset spatial characteristics.
Temporal relevance; This subgroup aimed to provide direction on substantiated metrics on methods to define temporal overlap in searches with the purpose to improve relevance ranking based on dataset temporal characteristics.
Dataset relevance heuristics; This subgroup aimed to identify the top heuristics in Common Metadata Repository (CMR) and other search engines and applicability to EOSDIS and DAACs, with the purpose of taking a first pass at the dataset (collection) search problem.
Dataset relationships; This subgroup aimed to provide a common framework for identifying relatedness across datasets with the purpose of lowering the barrier to obtaining similar datasets for a given user query.
Implementation in Federated Search; This subgroup aimed to provide substantiated metrics and guidance on improving Information Retrieval (IR) practices within a Federated Search (FS) context defined as an IR technology that allows the simultaneous search of multiple searchable resources

In this presentation we will summarize the findings and recommendations of the first year of the group activities as well as discuss our plans and progress for year 2 activities including addressing semantic dataset relationships, granule level relevance. mining user behavior, and optimizing content for commercial search engines.

2) Challenges and Potential Approach in Search Relevance from a Dataset Maturity Perspective
Ge Peng
NOAA CICS-NC/NCEI

3) Connecting diverse data users with diverse data sets at the NCAR Research Data Archive
Grace Peng
NCAR

For more than 40 years, the RDA (rda.ucar.edu) has been collecting and disseminating weather and climate data to the research community. We host a growing collection of over 600 datasets from myriad sources, from ocean observations in 1662 to present day satellite measurements and globally gridded analyses/reanalyses.

From inception, RDA data users have ranged from neophyte graduate student through professors with decades of experience. Increasingly, researchers from outside the weather and climate community (energy, insurance, government sectors) are using our data. This is a sign of our success and maturity. However, diverse user backgrounds means that we can no longer assume a common lingua franca when describing our data.

In order to help researchers sift through our datasets to find what they need, we collect granule-level metadata that powers the RDA search functions. Users may search with free text, or perform faceted searches to successively narrow down possible selections. Once they have identified datasets of interest, they are directed to data set homepages which enable them to examine the parameters available for each file and vertical level. The granule-level metadata also enables us to offer custom subsetting services for most data sets.

Because we are a .edu and teaching is part of our mission, we do not aim to fully automate our data discovery and other services. Each dataset is assigned to a data specialist who serves multiple roles as a data curator engineer software developer subject matter expert and educator. When data users are unsure about some aspect of the data, we want to engage with them to help clear up their confusion. This helps raise the level of sophistication of the data users and our understanding of how to better describe and refactor data to improve future usability.

In this presentation, I will demonstrate some of our data discovery and education capabilities. I will also give an overview of our manual and automated metadata collection processes, which enables our search functions.

4) Earthdata Search: The Relevance of Relevance
Patrick Quinn
NASA

Earthdata Search is a web application which allows users to search, discover, visualize, refine, and access NASA and International/Interagency data about the Earth. As a client to the CMR, its catalog of collections grew 700% in the past year. This massive expansion brought relevancy to the forefront of the client's usability needs. In this talk, we discuss places where the influx of collections helped illuminate existing usability issues and how we have tackled or plan to tackle those challenges by improving relevance and metadata.

Notes:

Challenges and Progress in Search Relevancy

NASA Progress in Search Relevancey
            Edward M. Armstrong, Lewis McGibbney, Kim Whitehall

This is a working group. Major stakeholders are NASA and end users.
            Listing several points of contact and people.

Year 1
Mission: Improve search results relevance for ESODIS data
Several focus areas/objectives and ideas.
Outcomes/deliverables
            Recommendations for Essential Metrics
            Benchmarking study of dataset rankings
            Recommendations for enhancement

Dataset relationships:
Develop a common framework for identifying relationships between datasets.          Similar to the “people also bought” on Amazon.
Better understand User History and Behaviors
Improve relevance rankings based on dataset spatial and temporal characteristics
Provide metrics for improving information retrievals from a federation search context
            Something from this: Normalized Discounted Cumulative Gain
Heuristic implementation for the CMR and other search engines
            Test the intersection of retrieved vs relevant documents
            Log clicking events and more

Summary of key recommendations:
            -Report URL in slides.
            Spatial/temporal overlap
            Accuracy heuristics from search recall and precision should be implemented
            Federated searches should use Normalized Discounted Cumulative Gain
            Scientific literature should be minded for data dataset relationships

Year 2
Goals:
Optimize commercial se
Dataset relevance heuristics
Granule-level relevance
Semantic dataset relevance
User characteristics
Engage ESIP discover and semantic technology groups

Have engaged several subject matter experts.
Have a fall AGU session, check out slides.
Quick plug to participate

Questions:
            How do user characteristics tie into relevance?
                        URS -> User Registration Service (with affiliation) (Code on GitHub)
                        This allows tracking of what products users are most interested in
            How do you fit that information back in?
                        This is all new territory.
            Aren’t all searches taking place off CMR? If so, why are you interested in        Federated searches?
                        There is a federated search, mostly of non-NASA holdings.
            Will contextual relevance come to this? How does this all fit in?
                        Spatial and temporal subgroups are all about context.
                        These sort of things have been started a little bit already.

Challenges and Potential Approach in Search Relevance from a Dataset Maturity Perspective
            Ge Peng

What is Search Relevance Ranking?
            The process of sorting web pages and ranking how relevant they are.
Why do we worry about this?
            It helps solve the information overload that we have now. More than 100,00 datasets between NASA and NOAA.

How is this a challenge, and why?
            If you want Hi-Res Global SST datasets: Google gets 5 million. NOAA gives 3,000, and NASA 64. These are from all over and all from trusted organizations.
            With all of this, how do I get the ONE dataset I actually want/can use?

Two approaches:
            Hardcore
                        Users do not need to understand how they get the results they want
                        This is hard for a search algorithm
            User Understandable
                        Simple navigation approach

How?
            Information downscaling requires a sophisticated capable system.
            Information Filtering: Infinite -> Many -> Limited numbers
                        Example: NIDIS-Data Search system
                                    It works well, but has a very targeted user community
                        GHRRST: Web to Products to Data level
                                    Represents Hi-Res SST community well
                                    Still requires extensive product-specific knowledge

I would like to use the data/product in my _________.
            Has in been evaluated? How?
I want to do ______.
            What dataset should I use?

The only way to answer these as of now is to pick on subject matter expert’s brains.
            What happens when SME leaves/retires/etc?
            Who/which SME should I contact?
            Should/how do/can we get this information?

Perspective here – call for
            Availability of content-rich dataset maturity information
            Standard metadata and documentation
            Readiness for integration into a web search relevance ranking algorithm

Reference Frameworks can help here.
            Need a consistent end to end maturity measure for this.

An example: College Admissions

Application of this:
            NOAA OneStop-Ready

Community consensus and contributions are welcome and needed!

Goals:
            Understandable quality info for human end users
            Actionable information for decision-makers
            Integrable tags for machines

Who is responsible for assigning maturity rankings?
            At NOAA OneStop-Ready, there is a metadata content editor who is working with others do to this.

Connecting diverse data users with diverse data sets at the NCAR RDA
            Grace Peng

Data Archive
            Collect and preserve dtat
Data Center
            Serve data to users
            Not real-time
Data Education Center
            Consulting for data discovery and data flow setup
            Teach how to use data correctly and effectively

They have a very heterogeneous user base:
            NCAR/UCAR internal staff
            Member university researchers
            Other university researchers
            Fledgling researchers
            Government
            Teachers
            Commercial

They work with the data seekers, while educating. Allow for data creators to reach “data immortality”

Stats for 2015
            12,000 unique users
            28 PB processed
            1.6 PB served
            0.5 PB of custom orders

Strategy to assist all 12,000 users
            Collect granule-level metadata
            Use a faceted search
            Standardize data information templates
            Help guides on Youtube videos, blogspot, and more
            Human interaction for the final data selection

Questions
            How does this relate to search relevance?
                        The various ways that the searches are done related very well to                                 relevance.
                        The series of searches leads eventually to a human being.

Earthdata Search: The Relevance of Relevance
            Patrick Quinn

Earthdata Search is a client to NASA’s CMR repository. It lets users search and discover NASA and other agency’s data.

In late 2015, there was a massive growth of matching collections, which lead to relevancy need growth.
This need for better relevance was echoed by users as well.
            Takeaway: If users can’t find the collection they need, none of the other work                                           matters.

What are they finding then?
Metadata isn’t always consistent.
Sometimes metadata is TOO consistent.
Sometimes it’s too complete.
            How do you know what to prioritize?
Facet relevancy matters as well. This is due to the thousands of data collection platforms.

Providing relevant refinement options and distinctions is important as result order. There is a need to give users tools to search.

Easy win solutions:
            Newer versions first
            Collections with granules first
            Remove less useful choices
            Gather metrics for improvements
            Humanized the facets

Design iterations are ongoing throughout this work.
Domain-specific relevancy
            Temporal/spatial overlap
            Understand user experience
            And more!

Questions
            How were the facets improved? Were those decisions on content made by     users or made by the developers?
                        Most of the changes were objective and stylistic.
            How was the user testing done?
                        It was formal, reference 4pm tak.
            Have you been looking at interactive query refinement?
                        It’s a great idea!
            How will you know if the improvements helped?
                        The data/metric collected now will inform us.

Final Discussion
            Is there a need for more basic information retrieval research here?
            If so, are there funding mechanisms to support this work?
                        Funding is complex, but there is need for more research. This is                                   particularly the case in the growing age of massive numbers of                                                 datasets.
            There is a distinction between structured and free text search. This was bad             for users. Free text might be more convenient for users, but structured            search can be more helpful for relevance.
            Can you to touch upon related datasets based on scientific literature?
                        There is some work being done on this, and it’s being discussed.
            How do you know if you’ve addressed the user needs? Is there a validation approach?
                        There is a challenging approach here. But certainly user feedback is                           possible to get and very useful.
            What can we ask the user to do? Users will type in key words rapidly, but slowly look through facets. Free text box then a facet box pops up and it lists all the potential facets. Tight coupling of facets and free text throughout the search process.
            A good metric of this is multiple download by the same user.
            Can you give a few more examples of humanizing facets?
                        Different names for the same thing shouldn’t be listed as unique results (AM1 vs TERRA).

Attachments/Presentations:

Attachment	Size
GePeng_SearchRelevanceRanking_ESIP2016summer_final.pdf	1.05 MB
esipsummer16_NASA_search_relevancy.pdf	7.57 MB
03_EDSCRelevanceESIP2016_PQ.pdf	5.59 MB
GracePengSearchRelevance.pdf	9.16 MB