Publishing mechanism

[mishaschwartz]

We would like to provide users with a mechanism that allows them to publish a workflow or data product (product) that they have created on a Marble node.

When a user wants to publish a product, it should be accompanied with some metadata that helps other users search for and use the product for their own research. Another user should also be able to recreate the product using the same steps as the original author.

Publishing requirements (note that this is not an exhaustive list, feel free to add to it):

• author(s)
• input data (description and provenance)
• data description:
  • spatial/temporal extents
  • resolution
  • variables
• steps to recreate data (eg. an accompanying workflow script or similar?)

The entire publishing mechanism should include:

• a UI for users to request that their product be published
• a UI for node administrators to accept, request changes, or reject a request
• a (mostly automated if possible) process to publish the data on the THREDDS/geoserver server with metadata hosted on the STAC catalog.
  • this should be integrated with the data ingestion workflow project if possible

Suggested steps to take for this project:

1. Research the following if you are not already familiar:

   • THREDDS data server (https://www.unidata.ucar.edu/software/tds/)
   • STAC: https://stacspec.org/en
     • Item: https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md
     • Collection: https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md
     • Catalog: https://github.com/radiantearth/stac-spec/blob/master/catalog-spec/catalog-spec.md
     • API: https://github.com/radiantearth/stac-api-spec

2. Research/compile example of the sort of data and workflows that users may want to publish on Marble

   • you may want to do a literature review of climate research papers that have accompanying data products

3. Compile a list of metadata that should accompany a published product

   • describe which metadata is: always required, required depending on the product type, optional
   • describe acceptable values for each metadata type

4. Translate the metadata described above as a STAC extension(s) so that published products can be stored as a STAC entry

5. Design the UI for users to request that their product be published

6. Design the UI for node administrators to accept, request changes, or reject a request

   • requests for changes need to be communicated to the original requestor and there needs to be a UI for them to ammend their request and re-submit it for review

7. Implement the UI for steps 5 and 6 above

8. Write software that takes a data product (and accompanying metadata) once it has been accepted for publishing and:

   • makes it available through the THREDDS server (or similar as appropriate)
     • data can/should be shared through THREDDS or geoserver
     • workflows will probably need a different way of hosting them. Consider that workflows can be defined as jupyter notebooks, weaver jobs, cwl files, etc.
   • adds the metadata as an entry on the STAC API so that the new product is searchable

Deliverables:

• Internal report describing steps 2 and 3 above
• STAC extension(s) required to store published metadata (if needed)
• a UI design (wireframe) for users to request that their product be published
• a UI design (wireframe) for node administrators to accept, request changes, or reject a request
• implementation of the two UI described above
• software to publish the requested data (see step 8)

Participants/Roles:

• Student (TBD): research and software development, UX design/consultation
• Shruti Katkar: UX design and UX research assistance
• Alex Yu: UI development, UI consultation
• Steve Easterbrook: consult on metadata and publishing requirements

[fmigneault]

We are currently working on similar requirements for GeoDataCubes (GDC) in OGC Testbed-20 regarding how to perform a workflow processing on multi-dimensional spatio-temporal data, and how the resulting data products/collections can keep track and report their provenance with the processing pipeline (see also the Integrity, Provenance, and Trust (IPT) track) - ie FAIR principles.

A few relevant documents/issues of ongoing work items for metadata:

• in processing:

  • https://docs.ogc.org/bp/20-089r1.html#toc31
  • Parse additional metadata from schema.org in CWL document to convert into process fields crim-ca/weaver#463
  • Add more metadata mapping details in documentation crim-ca/weaver#613
  • Improve discovery/listing of deployed processes crim-ca/weaver#102
  • Support for STAC metadata crim-ca/weaver#103
  • Provide OGC conformance definitions for "search/filter" query parameters crim-ca/weaver#641
  • Support OGC-API Processes - Part 3: Workflows crim-ca/weaver#412
    ⬆️ these are in our "priority" list of issues for OGC-TB20
    so if you're looking to get involved into this work, or find/share relevant workflow samples, there is a lot of cross-walk work that can be done

• in data collections via STAC / OGC APIs:

  • cube extension: https://github.com/stac-extensions/datacube
  • supported by OGC TB-19 GDC API: http://ogc.pages.ogc.org/T19-GDC/documents/D011/document.pdf
  • very soon integration via GeoServer: Upgrade GeoServer for vulnerabilities bird-house/birdhouse-deploy#465 (comment) (see comment about OGC APIs added - STAC will be too!)

And below, previous searches I did regarding relevant STAC extensions when involving metadata relevant for machine learning or notable filtering of data :

→ Accuacy: combination of STAC extensions

• https://github.com/stac-extensions/accuracy
• Stats within Labels: GitHub - stac-extensions/label: Stats Object

→ Filtering / pre-processing:

• https://github.com/stac-extensions/projection
• https://github.com/stac-extensions/timestamps
• https://github.com/stac-extensions/mgrs
• https://github.com/stac-extensions/hsi

→ Data Sample Elements (samples from DataLoader?):

• https://github.com/stac-extensions/grid
• https://github.com/stac-extensions/xarray-assets
• https://github.com/stac-extensions/item-assets
• https://github.com/stac-extensions/tiled-assets
• https://github.com/stac-extensions/ml-aoi
• Load order critical items according to random seed?
  https://github.com/stac-extensions/order
• Labels to load per sample:
  • https://github.com/stac-extensions/classification
  • https://github.com/stac-extensions/subjects

Note that are many more extensions for different use cases and data types, and the list often expands:
https://stac-extensions.github.io/