[8.3.2] Tomocam Reconstruction

[davramov]

I met with Dinesh about integrating Tomocam with our Prefect workflows. He is working on building a Shifter container for Tomocam that they will maintain, and we can use in our flows.

Some notes from our meeting:

• Tomocam requires 16 x GPU nodes on Perlmutter (64 GPUs). May need to reach out to Bjoern about the number of nodes we can use in the realtime QOS, and if we need to request more.
• Reconstruction takes about 10 minutes.
• Most useful for noisy data.
• This flow could be used selectively (due to the large number of GPU nodes), rather than as an automated flow. In this case, we could trigger the flow directly in Prefect, or develop another web app UI that users can use to trigger the flow.
• Tomocam still uses tomopy for pre-processing steps, but replaces the recon() function as a drop-in replacement.
• There should be good default values in Tomocam already.
• Can save reconstructions as tiffs or hdf5. If we save as tiffs, how do we capture the metadata?

Integration with splash_flows:

• Add another implementation in orchestration/flows/bl832/nersc.py that calls the Tomocam workflow.
• Write a SLURM job script that allocates the appropriate number of nodes.
• Wrap the reconstruction in a Prefect flow called nersc_tomocam_recon_flow and add it to the deployments.

We will meet again in a couple of weeks to discuss.

[davramov]

I have created a new branch tomocam for this issue, which I'll share with Dinesh.