NXxas refactoring

[woutdenolf]

WIP: everything will be moved to contributed_definitions once this is up for review.

Contributing: PR to the main branch of https://github.com/XraySpectroscopy/nexus_definitions (merge commits will be removed regularly).

Rendering: https://hdf5.gitlab-pages.esrf.fr/nexus/nxxas/classes/applications/NXxas_new.html

Example files: https://pynxxas.readthedocs.io/

NXDL questions:

- how to re-use an enum
- when enum field X has a specific value, make field Y required OR how to define enum properties
- possibly link to
- how to make a field required in a base class

[lukaspie]

Hi @woutdenolf, I came across this PR by chance and noticed that you are working on a base class for describing an electron level. In the context of photoemission spectroscopy, we in FAIRmat have also been working on something very similar: https://github.com/FAIRmat-NFDI/nexus_definitions/blob/fairmat/contributed_definitions/NXelectron_level.nxdl.xml. In fact, some of the base class we developed was originally adapated from another XAS-related PR: #1293. Would you be open to harmonizing the concept that we were developing with your approach here? I think the two approaches are already very similar right now.

[woutdenolf]

@lukaspie Yes since we are doing the same thing we should try to converge. We are currently discussing this very topic: XraySpectroscopy#2. Feel free to get involved.

The XAS working group will work in https://github.com/XraySpectroscopy/nexus_definitions. It would be helpful to add comments in that repo. This PR will be a draft and is subject to lots of changes until we are finished.

[lukaspie]

@woutdenolf great, thanks for the pointers. I will add some comments there.

[woutdenolf]

another XAS-related PR: #1293

This was the first attempt before we created the XAS working group. It will most likely be closed.

[whs92]

@woutdenolf after asking a question to the nexus mailing list and reading:

#1011
#1293

I arrived here :)

I have had a look at the proposal you made to the NXxas_new

I am particularly interested in the topic raised in #1011 of adding the ability to work with multiple detectors. I see that you've added the entry intensity that is still tied to a single mode (TFY, PFY, TEY etc). How might I add other signals acquired at the same time? How might I add more than one PFY signal?

Do you plan to include the suggestions from @padraic-shafer in https://github.com/padraic-shafer/definitions/blob/1011-multixas/contributed_definitions/NXmultixas.nxdl.xml

When is the next planned meeting of the working group? Can I come along?

[woutdenolf]

Currently we only support a single XAS spectrum. I'm very much open to support multi-detector and/or multi-scan. You are very welcome to join the XAS working group. I'll contact you directly.

[benajamin]

If you want to allow the "electron energy loss" mode, then you should also allow an "electron" probe type.

[benajamin]

What if I want to do a scan that includes multiple absorption edges (e.g. Carbon K-edge and K L-edge)? Can I specify multiple elements?

[benajamin]

Does this include Auger electron yield? Can we include a way to specify the observed electron energy range? Or should that be a detail for NXdetector?

[benajamin]

The i0 should have a corresponding NXmonitor group. There are also many ways of measuring an i0 that is useful to document in the file. For example, is the i0 measured in parallel with the sample absorption (or its proxy), or is it measured in sequence (e.g. before or after the sample), or is it sequential, but interleaved?

There is also question of how "clean" the i0 data is, as in how directly/accurately does the i0 measurement follow the ideal incident flux signal? This issue is particularly significant for C K-edge measurements - one can read my article on the topic to further understand this issue.

[ounsy]

#1352 (comment)
@whs92
@woutdenolf
I also got a request for that from one of our beamlines at SOLEIL

[PeterC-DLS]

With NXedge and NXemission_lines, is it better to have an atomic orbital (in X-ray notation rather than spectroscopic notation) and transition with the latter comprising an initial and final orbital?

[DanPorter]

Hi all, I was asked by @PeterC-DLS to have a look at this as a few beamlines I'm working with at Diamond are using sub-entries with the old NXxas specification. I have a couple of comments:

1. There is a comment from @benajamin about multiple edges - this is a valid point. I think there should be an additional base class NXabsorption_edge that can contain NXelement and NXedge, the name could be an enum like "Co L3". The NXentry could then contain as many NXabsorption_edge fields as required. This way multiple edges/ elements could be specified, the principle or intended edge would be the first used. In my mind this would look something like:

/entry/definition = "NXxas_new"
/entry/mode:NXmode
/entry/Co_L3:NXabsorption_edge
/entry/Co_L3:name = "Co L3"
/entry/Co_L3/element: NXelement = "Co"
/entry/Co_L3/edge_type: NXedge = "L3"
/entry/Co_L3/orbital = "2p3/2"
/entry/Co_L3/energy = 778.1 @units="eV"

As @PeterC-DLS mentioned, this could also be expanded to allow different notations and the specification of specific orbital transitions.
I've written an example enumeration script for NXedge with name type "element edge" here, in case this is useful.

2. The NXdata group should probably contain optional links to the monitor and to fields in NXprocess, for example (I/I0) or log(I/I0). It should be more clear in the application definition that calculated experimental spectra like this should be stored in NXprocess (assuming this is correct).
3. The "calculated" field should probably be renamed "simulated" or, inversely, "measured". 'calculated' could mean calculating a ratio of measurement signals, and I assume this field is to determine if the spectra has originated from simulation software. In which case, should there be some optional fields, analagous to NXinstrument, that capture what software and what inputs were used?

[newville]

@DanPorter
Point 1: It would be helpful for NeXus to have standard classes for Element and either X-ray Level or Electron Level. We in the XAS community spent a lot of time trying to come up with such a proposal and gave up and decided it was not our fight. See @PeterC-DLS' comment for example. We spent a lot of time lost in such discussions. We concluded that "Element" is a string of an atomic symbol, and "Edge" is a string of the principal edge measured. Yes, measurements can be "L3, L2", or "M5, M4, M3" -- those would be allowable.

I would not be in favor of an "element_edge" like "Co L3". We will want to be able to search on Element and also on Edge, and should not expect all downstream software to parse such strings.

Point 2: Yes, the intention is that the process entry should describe how the "rawdata" data gets transformed into the intensity array (and perhaps energy, if that is not directly measured). This should describe what math is done on the raw data arrays (or perhaps other NXxas entries). It will likely be more complex than -log(Column3/Column2), as it might apply deadtime corrections and sum multiple channels from the "rawdata" arrays.
And, since the intensity to be presented (and plotted) is the "lightly processed normalized mu(E) data", the processing required to get to intensity should include some background subtraction, typically the slope/intercept of a linear background in energy, and the normalizing edge_jump used.

Point 3: "Calculated" could be the sum or mean of multiple other spectra, or the result of a fit or some other analysis. But, if the ratio of different signals in the "raw data" within a single measurement, then every spectrum would be a calculation - no one measures one "signal" that is "normalized mu(E)" or even "unnormalized mu(E)". But also, yes, including the details of the calculation or analysis done in the process section would be helpful and/or expected.

[DanPorter]

Thanks for your responses @newville. I certainly don't want to re-open old arguments!

I just thought of another comment whilst whilst working on some XMCD spectra:

4. When looking for magnetic signals we subtract different spectra based on polarisation, so we need to know the incident beam polarisation. Currently we store a string defining 'lh', 'lv', 'pc', 'nc' polarisation states in /entry/instrument/id/polarisation and also put the stokes vector in /entry/sample/beam/incident_polarization. Can at least the NXsample>NXbeam>incident_polarisation field be included as an optional field?

[benajamin]

Should Auger electron yield mode supposed to be recorded as PEY, or can it be added as a new category? In my experience, AEY restricts the energy of the detected electrons to a much smaller window (a single Auger peak) than PEY, which typically operates on a single cut-off energy to reject most of the secondary electrons.

[PeterC-DLS]

Suggested change

	<group type="NXCollection" optional="true">
	<group type="NXcollection" optional="true">

[PeterC-DLS]

This symbol is not used

[newville]

@woutdenolf Thanks for updating this. Any guidance on the delay on merging this? I'm sort of at a loss here.

[woutdenolf]

We need to finish the modes, add it to NXxas and then I can schedule the contribution for the NIAC meeting.

[DanPorter]

Hi all, at a previous meeting I promised some example data and I'd forgotten to share it with you! Here is a repo with some example scans doing XAS and XMCD, where I've tried to implement the new format:

https://github.com/DanPorter/xmcd_analysis_example

You can click on the links in the readme to view the files in myHDF5.

The files also contain processed components described with NXprocess. I found that there are lots of different ways of interpreting how this should be implemented, but the way I've done it provides useful default plotting and what I felt is a full description of the steps taken, with different steps stored as different NXdata groups and different elements in the NXprocess group.

[mretegan]

Hi all,

@DanPorter thanks a lot for providing the formatted files. I don't think we discussed using this definition to describe XMCD, or any other type of dichroic measurements for that matter. I would be more in favor of not adding too much initially so that we have something finished for XAS; this has already taken more time than expected. This week I can update the definition with the information from the modes

Regarding the “calculated” field, it is meant to indicate whether the data is from a theoretical calculation, so its description could be improved.

[woutdenolf]

@phyy-nx @sanbrock Question for the NIAC:

XAS has several variations. The new XAS definition will only cover the variations that can be expressed as "mu-versus-energy" (transmission, total fluorescence, partial fluorescence, HERFD, etc.).

The people who create the HDF5 file need to specify which mode was used so we know what mu represents. Some modes have required metadata fields which others do not have. How can we do this in NeXus?

Some options that were suggested by the participants of the working group:

1. Extend a different application definition from NXxas for each mode: NXxastrans, NXtfy, NXherfd, NXherfd, etc. (there are currently 15 of them). Several of them will not add any new fields so they would solely exist for the purposes of providing meaning the the "mu" field.

2. The NXxas application definitions has a group called "mode" which could have several base classes as type: NXxastrans, NXtfy, NXherfd, NXherfd, etc. Remark: currently NeXus does not support an enumeration of group types in application definitions. Also it means we would create based classes with fields like "emission_lines". Several of them will be empty though and would solely exist for the purposes of providing meaning the the "mu" field which is not even in the base class.

3. Find a way for a field to become required depending on the value of another field (a string enumeration in the case called "mode"). So if for example mode="PFY" the field "emission_lines" would become required.

[woutdenolf]

NeXus has a "choice" mechanism we could perhaps be used, assuming we enforce one single mode per XAS:

definitions/base_classes/NXdetector.nxdl.xml

Line 906 in 5a44a07

<choice name="pixel_shape">

.