Establish a canonical JSON representation for NeXus metadata

[paulmillar]

The NeXus metadata model is abstract. Although it is mostly commonly mapped to HDF5, it can also be mapped to XML. Given the recent (and on-going) work on mapping NeXus definitions to a corresponding ontology, it should also be possible to map NeXus metadata to corresponding statements within that ontology.

JSON is now a well-established format for recording information. There are a number of foundational technologies that work with JSON data, such as schemas, information extraction and manipulation. There are also many storage technologies that provide enhanced support for data in JSON format.

It would be helpful if NIAC established a mapping of NeXus metadata to JSON.

Possible use-cases include:

• storing NeXus metadata separately from the HDF5, in databases optimised for querying
• allowing network-transparent dataset querying using technologies that support JSON-backed information model.
• deliver metadata independently of the data, to allow software to investigate possible datasets of interest without downloading complete HDF5 files.
• support technologies that build summary information, based on JSON information models.

Having NIAC establish a canonical mapping of NeXus to JSON would avoid duplicated effort.

[ggoneiESS]

Hi Paul,

Was this picked up as something that should be actioned? At ESS we are using JSON for our file templates, and can convert between the two.

[paulmillar]

Hi George,

The particular use-case I had in mind is for searching for datasets based on NeXus metadata.

A search API was developed under the PaNOSC and ExPaNDS projects. There is also a search portal based on this API. Although this is functional and supports describing datasets with arbitrary (JSON) metadata, the API definition left the semantics of the metadata terms undefined.

Within the PaNOSC FAIR data management group, we are considering how to define the semantics of this metadata. This would need some common metadata standard, with NeXus being a natural choice.

For this to work, we would need a canonical way to express NeXus metadata in JSON. This would then be adopted as a common standard for searching for metadata; i.e., the search APIs would support searching by expressing JSON constraints, based on this canonical JSON representation of NeXus.

Therefore, it would be helpful if NIAC were to support a canonical JSON representation of NeXus metadata; i.e., define how to map NeXus metadata in some HDF5 file to a corresponding JSON object, ideally without introducing any ambiguity.