Software Evaluation Guidelines Workshop: Part 1
Come help ESIP develop software assessment guidelines for describing code maturity, project maturity and sustainability for research code and software. Our goal is to provide support and guidance to a broad range of stakeholders creating code products and to define, as a community, what aspects are important to promote for research groups and for the research community.
In this first session, we'll be discussing the guideline revisions to date (available for comment) with a focus on containerization (during development, reusability, and preservation/archiving), interoperability and areas that are missing or lacking consensus. We encourage you to take a look at the draft before the session and to set up an Hypothes.is account for making contributions/comments during the session.
This is a half day working session with the second session described here.
This is our only opportunity for F2F conversations on the software side. The split between the two sessions might change depending on the outcome of the remote codesprint but the need to have the conversations doesn't change.
Meeting Notes:
Software Evaluation Guidelines Workshop: Part 1
Two links for tracking notes and drafts:
http://roomthily.github.io/technology-evaluation-research/ESIPSoftwareGu...
https://titanpad.com/0z9RcazAJy
A few definitions:
Research Code – a code product that leads to some published outcome, but is not a specific deliverable
Research Software – some code product that is intended for reuse
This started as a restructuring question, but these things change.
Education and mentoring is important, and ESIP should have the broad set of guidelines -> for all geoscience. More specific orgs can adjust these guidelines to their specific needs.
Trying to get to the groups that have the least amount of total overall experience, but also needs to meet the needs of developers.
PIs are the priority audience. There is funder-side interest AND developer-side interest. PIs sit in between both these groups.
Keep it broad.
The document is a means of education.
Define some specific scenarios of funding systems and grant activity. What can we actually assess within that? Get some stories rolling.
Specific guideline discussions.
Gap analysis.
Christine White commented that – several of these goals are very useful, including speaking to the education of non-expert developers, and making sure jargon is well defined and google-able.
Which funders are actually interested in this: NASA, NSF, EarthCube, USGS
Is this for software or code?
Both!
Described and discussed some of the titan pad examples.
There is an important need to define practices specific to the project and code. Large projects have different and more in depth demands than a simple 100-line script.
Who is this for? Will this work if funders don’t require it?
There are many reasons to care: personal standards, sharing requirements, just a place to start!
Should there actually be code review in the same way as paper reviews?
Potentially, yes. Large organizations already do code review, what about the smaller research groups? That’s where this comes in to play.
This would be ideal, however is expensive.
How do you give credit?
How do we consider issues with containers and licensing? No strong answer yet, but discussion.
Software Evaluation Guidelines Workshop: Part 2
Started with a quick catch up from part 1
https://titanpad.com/F0TIRgokCq
We need to care about development/funding cycles as well as the lifetime of a project.
Goals:
Think about “happy paths” in code and project maturity
Research situations: sustainability and maturities
For research software:
Proof of Concept
Prototype
Production
Operational Use
What are the research code stages?
The research software stages don’t really fit nicely…
You can’t really define phases for every piece of software
Potentially use categories within all of this.
Categories are important alongside important/clear definitions
“We are making explicit our community expectations and best practices”
SOME practices will be required. What about these???
Questions about how adoption will work.
How can you keep sustainable practices on limited budget and project dependent way?
Is open source the answer?
-Potentially.
-Many questions to ask? Licensing, adoption, and more.
What is the value of these data centers? What would the value of these guidelines and software sharing?
How will this be adopted?
Subsets use by:
-internally by the ESIP community
-EarthCube
NSF is interested in what comes out of this.
USGS could be influenced by this.
NASA/NOAA -> uncertain.
What goals are there?
Assessment of sustainability and maturity.
How to actually develop and write this code?
Evaluating choices?
Necessitates awareness and education.
Should we separate assessment and education?
| Attachment | Size |
|---|---|
 software_bee_nc2016_pt1.pdf | 35.79 MB |