Challenges and Progress in Search Relevancy
As dataset diversity and data volumes continue to increase, providing users with the interfaces, tools and services they need to discover relevant datasets creates new challenges and opportunities for the improvement of search relevancy and search engine result ranking. Diversity of user communities is a challenge as well, given that relevancy depends on specific user types and needs.
In this session we will report on projects and activities to improve search relevancy from the perspective of finding and utilizing Earth science in-situ, satellite and model data. We will explore search relevance on both the dataset and granule levels, dataset relationships/dependencies, semantic relationships, data quality, user characterization and content based ranking.
Specifically we will report on search relevance activities and results ongoing in NASA, NCAR and other organizations with the goal of building synergy among community experiences, and developing strategies to improve search relevance and user experience across the entire spectrum of Earth science data and data users.
1) NASA Progress in Search Relevancy
Edward M. Armstrong, Lewis McGibbney, Kim Whitehall
NASA Jet Propulsion Laboratory
Recently the NASA ESDSWG on Search Relevance concluded its first year of activities to address search relevance across the 12 NASA earth science data centers. It was originally proposed to characterize the term search relevancy as it relates to ESDIS, to assess the implementations that address search relevancy, and to determine how existing implementations can be improved and new approaches be enacted. Individually and collectively, the group sought the expertise of persons within ESDIS, industry and academia. Five core subgroups (from an initial collection of ten) were organized on the topics of Spatial Relevance, Temporal Relevance, Dataset Heuristics, Dataset Relationships, and Federated Search:
- Spatial relevance; This subgroup aimed to provide direction on substantiated metrics on methods to define spatial overlap in searches with the purpose to improve relevance ranking based on dataset spatial characteristics.
- Temporal relevance; This subgroup aimed to provide direction on substantiated metrics on methods to define temporal overlap in searches with the purpose to improve relevance ranking based on dataset temporal characteristics.
- Dataset relevance heuristics; This subgroup aimed to identify the top heuristics in Common Metadata Repository (CMR) and other search engines and applicability to EOSDIS and DAACs, with the purpose of taking a first pass at the dataset (collection) search problem.
- Dataset relationships; This subgroup aimed to provide a common framework for identifying relatedness across datasets with the purpose of lowering the barrier to obtaining similar datasets for a given user query.
- Implementation in Federated Search; This subgroup aimed to provide substantiated metrics and guidance on improving Information Retrieval (IR) practices within a Federated Search (FS) context defined as an IR technology that allows the simultaneous search of multiple searchable resources
In this presentation we will summarize the findings and recommendations of the first year of the group activities as well as discuss our plans and progress for year 2 activities including addressing semantic dataset relationships, granule level relevance. mining user behavior, and optimizing content for commercial search engines.
2) Challenges and Potential Approach in Search Relevance from a Dataset Maturity Perspective
3) Connecting diverse data users with diverse data sets at the NCAR Research Data Archive
For more than 40 years, the RDA (rda.ucar.edu) has been collecting and disseminating weather and climate data to the research community. We host a growing collection of over 600 datasets from myriad sources, from ocean observations in 1662 to present day satellite measurements and globally gridded analyses/reanalyses.
From inception, RDA data users have ranged from neophyte graduate student through professors with decades of experience. Increasingly, researchers from outside the weather and climate community (energy, insurance, government sectors) are using our data. This is a sign of our success and maturity. However, diverse user backgrounds means that we can no longer assume a common lingua franca when describing our data.
In order to help researchers sift through our datasets to find what they need, we collect granule-level metadata that powers the RDA search functions. Users may search with free text, or perform faceted searches to successively narrow down possible selections. Once they have identified datasets of interest, they are directed to data set homepages which enable them to examine the parameters available for each file and vertical level. The granule-level metadata also enables us to offer custom subsetting services for most data sets.
Because we are a .edu and teaching is part of our mission, we do not aim to fully automate our data discovery and other services. Each dataset is assigned to a data specialist who serves multiple roles as a data curator engineer software developer subject matter expert and educator. When data users are unsure about some aspect of the data, we want to engage with them to help clear up their confusion. This helps raise the level of sophistication of the data users and our understanding of how to better describe and refactor data to improve future usability.
In this presentation, I will demonstrate some of our data discovery and education capabilities. I will also give an overview of our manual and automated metadata collection processes, which enables our search functions.
4) Earthdata Search: The Relevance of Relevance
Earthdata Search is a web application which allows users to search, discover, visualize, refine, and access NASA and International/Interagency data about the Earth. As a client to the CMR, its catalog of collections grew 700% in the past year. This massive expansion brought relevancy to the forefront of the client's usability needs. In this talk, we discuss places where the influx of collections helped illuminate existing usability issues and how we have tackled or plan to tackle those challenges by improving relevance and metadata.
Challenges and Progress in Search Relevancy
NASA Progress in Search Relevancey
Edward M. Armstrong, Lewis McGibbney, Kim Whitehall
This is a working group. Major stakeholders are NASA and end users.
Listing several points of contact and people.
Mission: Improve search results relevance for ESODIS data
Several focus areas/objectives and ideas.
Recommendations for Essential Metrics
Benchmarking study of dataset rankings
Recommendations for enhancement
Develop a common framework for identifying relationships between datasets. Similar to the “people also bought” on Amazon.
Better understand User History and Behaviors
Improve relevance rankings based on dataset spatial and temporal characteristics
Provide metrics for improving information retrievals from a federation search context
Something from this: Normalized Discounted Cumulative Gain
Heuristic implementation for the CMR and other search engines
Test the intersection of retrieved vs relevant documents
Log clicking events and more
Summary of key recommendations:
-Report URL in slides.
Accuracy heuristics from search recall and precision should be implemented
Federated searches should use Normalized Discounted Cumulative Gain
Scientific literature should be minded for data dataset relationships
Optimize commercial se
Dataset relevance heuristics
Semantic dataset relevance
Engage ESIP discover and semantic technology groups
Have engaged several subject matter experts.
Have a fall AGU session, check out slides.
Quick plug to participate
How do user characteristics tie into relevance?
URS -> User Registration Service (with affiliation) (Code on GitHub)
This allows tracking of what products users are most interested in
How do you fit that information back in?
This is all new territory.
Aren’t all searches taking place off CMR? If so, why are you interested in Federated searches?
There is a federated search, mostly of non-NASA holdings.
Will contextual relevance come to this? How does this all fit in?
Spatial and temporal subgroups are all about context.
These sort of things have been started a little bit already.
Challenges and Potential Approach in Search Relevance from a Dataset Maturity Perspective
What is Search Relevance Ranking?
The process of sorting web pages and ranking how relevant they are.
Why do we worry about this?
It helps solve the information overload that we have now. More than 100,00 datasets between NASA and NOAA.
How is this a challenge, and why?
If you want Hi-Res Global SST datasets: Google gets 5 million. NOAA gives 3,000, and NASA 64. These are from all over and all from trusted organizations.
With all of this, how do I get the ONE dataset I actually want/can use?
Users do not need to understand how they get the results they want
This is hard for a search algorithm
Simple navigation approach
Information downscaling requires a sophisticated capable system.
Information Filtering: Infinite -> Many -> Limited numbers
Example: NIDIS-Data Search system
It works well, but has a very targeted user community
GHRRST: Web to Products to Data level
Represents Hi-Res SST community well
Still requires extensive product-specific knowledge
I would like to use the data/product in my _________.
Has in been evaluated? How?
I want to do ______.
What dataset should I use?
The only way to answer these as of now is to pick on subject matter expert’s brains.
What happens when SME leaves/retires/etc?
Who/which SME should I contact?
Should/how do/can we get this information?
Perspective here – call for
Availability of content-rich dataset maturity information
Standard metadata and documentation
Readiness for integration into a web search relevance ranking algorithm
Reference Frameworks can help here.
Need a consistent end to end maturity measure for this.
An example: College Admissions
Application of this:
Community consensus and contributions are welcome and needed!
Understandable quality info for human end users
Actionable information for decision-makers
Integrable tags for machines
Who is responsible for assigning maturity rankings?
At NOAA OneStop-Ready, there is a metadata content editor who is working with others do to this.
Connecting diverse data users with diverse data sets at the NCAR RDA
Collect and preserve dtat
Serve data to users
Data Education Center
Consulting for data discovery and data flow setup
Teach how to use data correctly and effectively
They have a very heterogeneous user base:
NCAR/UCAR internal staff
Member university researchers
Other university researchers
They work with the data seekers, while educating. Allow for data creators to reach “data immortality”
Stats for 2015
12,000 unique users
28 PB processed
1.6 PB served
0.5 PB of custom orders
Strategy to assist all 12,000 users
Collect granule-level metadata
Use a faceted search
Standardize data information templates
Help guides on Youtube videos, blogspot, and more
Human interaction for the final data selection
How does this relate to search relevance?
The various ways that the searches are done related very well to relevance.
The series of searches leads eventually to a human being.
Earthdata Search: The Relevance of Relevance
Earthdata Search is a client to NASA’s CMR repository. It lets users search and discover NASA and other agency’s data.
In late 2015, there was a massive growth of matching collections, which lead to relevancy need growth.
This need for better relevance was echoed by users as well.
Takeaway: If users can’t find the collection they need, none of the other work matters.
What are they finding then?
Metadata isn’t always consistent.
Sometimes metadata is TOO consistent.
Sometimes it’s too complete.
How do you know what to prioritize?
Facet relevancy matters as well. This is due to the thousands of data collection platforms.
Providing relevant refinement options and distinctions is important as result order. There is a need to give users tools to search.
Easy win solutions:
Newer versions first
Collections with granules first
Remove less useful choices
Gather metrics for improvements
Humanized the facets
Design iterations are ongoing throughout this work.
Understand user experience
How were the facets improved? Were those decisions on content made by users or made by the developers?
Most of the changes were objective and stylistic.
How was the user testing done?
It was formal, reference 4pm tak.
Have you been looking at interactive query refinement?
It’s a great idea!
How will you know if the improvements helped?
The data/metric collected now will inform us.
Is there a need for more basic information retrieval research here?
If so, are there funding mechanisms to support this work?
Funding is complex, but there is need for more research. This is particularly the case in the growing age of massive numbers of datasets.
There is a distinction between structured and free text search. This was bad for users. Free text might be more convenient for users, but structured search can be more helpful for relevance.
Can you to touch upon related datasets based on scientific literature?
There is some work being done on this, and it’s being discussed.
How do you know if you’ve addressed the user needs? Is there a validation approach?
There is a challenging approach here. But certainly user feedback is possible to get and very useful.
What can we ask the user to do? Users will type in key words rapidly, but slowly look through facets. Free text box then a facet box pops up and it lists all the potential facets. Tight coupling of facets and free text throughout the search process.
A good metric of this is multiple download by the same user.
Can you give a few more examples of humanizing facets?
Different names for the same thing shouldn’t be listed as unique results (AM1 vs TERRA).