Evaluate and integrate bmdrc pipeline with real data

[christinehc]

Description

Previously, the bmdrc pipeline integration was tested with smaller test files for agile development. The pipeline needs to be adapted to actually process the relevant real data files with the new code.

To-Do

• Update database build workflow to use new bmdrc code
• Benchmark timings for running pipeline with data -- Seems to take ~1 hr per morphology/behavioral dataset pair

[christinehc]

Notes on current progress (12/5):

• Issues were encountered with using bmdrc to process existing zf data files (as defined in https://github.com/PNNL-CompBio/srpAnalytics/blob/main/data/srp_build_files.csv). The dataset generates a large number of empty/NaN dataframe rows in an interrim step, which causes memory explosion as two of these dataframes are merged. The bmdrc package was patched to remove empty/NaN rows prior to joining, which solved the issue.
• Currently, there is an issue with this LPR dataset: 344_zf_LPR_data_phase_1_2_2020JUNE25_updated_plate_id_for_TX_tall_fixed_merged.csv.gz - link to download data not generating a bmd curve. Will work with David to figure out what's going wrong here.

[sgosline]

Thanks for the updates. I fear that every time we update the schema we migth break something, so hopefully that file is just missing a column or something.