Skip to content

Jhrg/hyrax-1963 dmr version bump#320

Merged
jgallagher59701 merged 5 commits intomasterfrom
jhrg/HYRAX-1963-dmr-version-bump
Jan 22, 2026
Merged

Jhrg/hyrax-1963 dmr version bump#320
jgallagher59701 merged 5 commits intomasterfrom
jhrg/HYRAX-1963-dmr-version-bump

Conversation

@jgallagher59701
Copy link
Member

@jgallagher59701 jgallagher59701 commented Dec 17, 2025 •
edited
Loading

I changed the dmrVersion XML attribute from 1.0 to 2.0 (the change in DMR serialization makes the two responses incompatible).

I fixed the test baselines. All the tests pass.
This change will break a number of BES tests. So facr, the only change to the baselines has been to change "1.0" to "2.0".

ndp-opendap
ndp-opendap approved these changes Dec 17, 2025
View reviewed changes
Copy link
Contributor

@ndp-opendap ndp-opendap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My only concern about this is that nothing whatsoever has changed in the DMR XML representation, it's the serialization order that has changed...

hannahilea
hannahilea approved these changes Dec 17, 2025
View reviewed changes
Copy link
Contributor

@hannahilea hannahilea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. let's wait to merge until we have a (passing) bes branch ready to merge immediately after.

@sonarqubecloud
Copy link

sonarqubecloud bot commented Jan 6, 2026

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@jgallagher59701
Copy link
Member Author

jgallagher59701 commented Jan 20, 2026

People, @ndp-opendap, @hannahilea, Now that we have a BES that is dmrVersion immune, I will put the BES and run its tests when linked with this branch. If they all pass - or if the only failures on my dev machine match those when built with the master of libdap, I will merge this branch.

Speak now or forever... ;-)

@hannahilea
Copy link
Contributor

hannahilea commented Jan 21, 2026

Fortune favors the bold! (Or, y'know, something like that.)

@jgallagher59701
Copy link
Member Author

jgallagher59701 commented Jan 21, 2026

Fortune favors the bold! (Or, y'know, something like that.)

I think that there is one test - the GlobalMetadataStoreTest - that fails on the BES with this change. I will double check.

@jgallagher59701
Copy link
Member Author

jgallagher59701 commented Jan 21, 2026

Fortune favors the bold! (Or, y'know, something like that.)

I think that there is one test - the GlobalMetadataStoreTest - that fails on the BES with this change. I will double check.

Here's the problem:

dap/unit-tests % ./GlobalMetadataStoreTest 
........F......................................


!!!FAILURES!!!
Test Results:
Run:  46   Failures: 1   Errors: 0


1) test: bes::GlobalMetadataStoreTest::cache_a_dmr_response (F) line: 433 GlobalMetadataStoreTest.cc
assertion failed
- Expression: stored_response == test_05_dmr_baseline

@jgallagher59701
Copy link
Member Author

jgallagher59701 commented Jan 21, 2026

Also,

netcdf_handler/tests % ./testsuite -j20
## ------------------------------------------------------------------ ##
## bes 3.21.1 test suite: bes/modules/netcdf_handler/tests testsuite. ##
## ------------------------------------------------------------------ ##

  2: nc/bears.nc.2.bescmd                            ok
  6: nc/fnoc1.nc.1.bescmd                            ok
  7: nc/fnoc1.nc.2.bescmd                            ok
  5: nc/fnoc1.nc.0.bescmd                            ok
 11: nc/test.nc.2a.bescmd                            ok
  3: nc/coads_climatology.nc.0.bescmd                ok
  1: nc/bears.nc.1.bescmd                            ok
 13: nc/test.nc.3.bescmd                             ok
 14: nc/test.nc.3a.bescmd                            ok
 12: nc/fnoc1.nc.3.bescmd                            ok
  8: nc/test.nc.0.bescmd                             ok
 15: nc/coads_climatology.nc.dmr.bescmd              FAILED (testsuite.at:26)
  4: nc/coads_climatology.nc.ddx.bescmd              ok
  9: nc/test.nc.1.bescmd                             ok
 10: nc/test.nc.2.bescmd                             ok
 16: nc/coads_d4c_nm_lc.dap.bescmd                   FAILED (testsuite.at:29)
 17: nc/coads_d4c_nm.dap.bescmd                      FAILED (testsuite.at:30)
 18: nc/coads_d4c_m.dap.bescmd                       FAILED (testsuite.at:31)
 20: nc/coads_d4c_m_vc.dap.bescmd                    FAILED (testsuite.at:33)
 19: nc/coads_d4c_m_lc.dap.bescmd                    FAILED (testsuite.at:32)
 32: nc/string_attribute_values.nc.0.bescmd          ok
 30: nc/fnoc1.nc.function.ddx.bescmd                 ok
 31: nc/fnoc1.nc.function.dods.bescmd                ok
 26: nc/zero_length_array.nc.2.bescmd                ok
 35: nc/nc4_nc_classic_no_comp.nc.1.bescmd           ok
 24: nc/zero_length_array.nc.0.bescmd                ok
 33: nc/string_attribute_values.nc.1.bescmd          FAILED (testsuite.at:85)
 28: nc/zero_length_array.nc.4.bescmd                FAILED (testsuite.at:55)
 34: nc/nc4_nc_classic_no_comp.nc.0.bescmd           ok
 25: nc/zero_length_array.nc.1.bescmd                ok
 27: nc/zero_length_array.nc.3.bescmd                ok
 22: nc/fnoc1.nc.dmr.bescmd                          FAILED (testsuite.at:38)
 36: nc/nc4_nc_classic_no_comp.nc.2.bescmd           ok
 29: nc/zero_length_array.nc.5.bescmd                FAILED (testsuite.at:56)
 23: nc/fnoc1.nc.dap.bescmd                          FAILED (testsuite.at:40)
 21: nc/coads_d4c_m_vlc.dap.bescmd                   FAILED (testsuite.at:34)
 38: nc/nc4_nc_classic_compressed.nc.0.bescmd        ok
 39: nc/nc4_nc_classic_compressed.nc.1.bescmd        ok
 37: nc/nc4_nc_classic_no_comp.nc.3.bescmd           ok
 40: nc/nc4_nc_classic_compressed.nc.2.bescmd        ok
 44: nc/nc4_unsigned_types.nc.2.bescmd               ok
 50: nc/nc4_strings.nc.0.bescmd                      ok
 46: nc/nc4_unsigned_types_comp.nc.0.bescmd          ok
 43: nc/nc4_unsigned_types.nc.1.bescmd               ok
 45: nc/nc4_unsigned_types.nc.3.bescmd               ok
 51: nc/nc4_strings.nc.1.bescmd                      ok
 49: nc/nc4_unsigned_types_comp.nc.3.bescmd          ok
 55: nc/nc4_strings_comp.nc.1.bescmd                 ok
 48: nc/nc4_unsigned_types_comp.nc.2.bescmd          ok
 47: nc/nc4_unsigned_types_comp.nc.1.bescmd          ok
 42: nc/nc4_unsigned_types.nc.0.bescmd               ok
 52: nc/nc4_strings.nc.2.bescmd                      ok
 53: nc/nc4_strings.nc.3.bescmd                      ok
 54: nc/nc4_strings_comp.nc.0.bescmd                 ok
 41: nc/nc4_nc_classic_compressed.nc.3.bescmd        ok
 56: nc/nc4_strings_comp.nc.2.bescmd                 ok
 58: nc/ref_tst_compound.nc.0.bescmd                 ok
 59: nc/ref_tst_compound.nc.1.bescmd                 ok
 57: nc/nc4_strings_comp.nc.3.bescmd                 ok
 60: nc/ref_tst_compound.nc.2.bescmd                 ok
 63: nc/ref_tst_compound2.nc.1.bescmd                ok
 68: nc/ref_tst_compound3.nc.2.bescmd                ok
 66: nc/ref_tst_compound3.nc.0.bescmd                ok
 64: nc/ref_tst_compound2.nc.2.bescmd                ok
 61: nc/ref_tst_compound.nc.3.bescmd                 ok
 65: nc/ref_tst_compound2.nc.3.bescmd                ok
 62: nc/ref_tst_compound2.nc.0.bescmd                ok
 67: nc/ref_tst_compound3.nc.1.bescmd                ok
 70: nc/ref_tst_compound4.nc.1.bescmd                ok
 69: nc/ref_tst_compound3.nc.3.bescmd                ok
 71: nc/ref_tst_compound4.nc.2.bescmd                ok
 73: nc/ref_tst_compound5.nc.1.bescmd                ok
 72: nc/ref_tst_compound4.nc.3.bescmd                ok
 74: nc/ref_tst_compound5.nc.2.bescmd                ok
 75: nc/ref_tst_compound5.nc.3.bescmd                ok
 76: nc/ref_tst_opaque_data.nc.0.bescmd              ok
 78: nc/ref_tst_opaque_data.nc.2.bescmd              ok
 79: nc/ref_tst_opaque_data.nc.3.bescmd              ok
 77: nc/ref_tst_opaque_data.nc.1.bescmd              ok
 80: nc/ref_tst_enum_data.nc.0.bescmd                ok
 81: nc/ref_tst_enum_data.nc.1.bescmd                ok
 84: nc/ref_tst_enum_data_1.nc.3.bescmd              ok
 85: nc/ref_tst_enum_data_2.nc.3.bescmd              ok
 82: nc/ref_tst_enum_data.nc.2.bescmd                ok
 83: nc/ref_tst_enum_data.nc.3.bescmd                ok
 88: nc/nc4_strings.nc.dmr.bescmd                    FAILED (testsuite.at:205)
 87: nc/nc4_nc_classic_no_comp.nc.dmr.bescmd         FAILED (testsuite.at:204)
 86: nc/nc4_nc_classic_compressed.nc.dmr.bescmd      FAILED (testsuite.at:203)
 90: nc/nc4_unsigned_types.nc.dmr.bescmd             FAILED (testsuite.at:207)
 89: nc/nc4_strings_comp.nc.dmr.bescmd               FAILED (testsuite.at:206)
 91: nc/nc4_unsigned_types_comp.nc.dmr.bescmd        FAILED (testsuite.at:208)
 92: nc/ref_tst_compound.nc.dmr.bescmd               FAILED (testsuite.at:209)
 93: nc/ref_tst_compound2.nc.dmr.bescmd              FAILED (testsuite.at:210)
 94: nc/ref_tst_compound3.nc.dmr.bescmd              FAILED (testsuite.at:211)
 95: nc/ref_tst_compound4.nc.dmr.bescmd              FAILED (testsuite.at:212)
 96: nc/ref_tst_compound5.nc.dmr.bescmd              FAILED (testsuite.at:213)
 97: nc/ref_tst_enum_data.nc.dmr.bescmd              FAILED (testsuite.at:214)
100: nc/ref_tst_opaque_data.nc.dmr.bescmd            FAILED (testsuite.at:217)
 98: nc/ref_tst_enum_data_1.nc.dmr.bescmd            FAILED (testsuite.at:215)
 99: nc/ref_tst_enum_data_2.nc.dmr.bescmd            FAILED (testsuite.at:216)
101: nc/stringtest.nc.dmr.bescmd                     FAILED (testsuite.at:218)
109: nc/ref_tst_compound2.nc.dap.bescmd              FAILED (testsuite.at:228)
104: nc/nc4_strings.nc.dap.bescmd                    FAILED (testsuite.at:223)
108: nc/ref_tst_compound.nc.dap.bescmd               FAILED (testsuite.at:227)
110: nc/ref_tst_compound3.nc.dap.bescmd              FAILED (testsuite.at:229)
107: nc/nc4_unsigned_types_comp.nc.dap.bescmd        FAILED (testsuite.at:226)
106: nc/nc4_unsigned_types.nc.dap.bescmd             FAILED (testsuite.at:225)
105: nc/nc4_strings_comp.nc.dap.bescmd               FAILED (testsuite.at:224)
102: nc/nc4_nc_classic_compressed.nc.dap.bescmd      FAILED (testsuite.at:221)
103: nc/nc4_nc_classic_no_comp.nc.dap.bescmd         FAILED (testsuite.at:222)
112: nc/ref_tst_compound5.nc.dap.bescmd              FAILED (testsuite.at:231)
111: nc/ref_tst_compound4.nc.dap.bescmd              FAILED (testsuite.at:230)
113: nc/ref_tst_enum_data.nc.dap.bescmd              FAILED (testsuite.at:232)
114: nc/ref_tst_enum_data_1.nc.dap.bescmd            FAILED (testsuite.at:233)
117: nc/zero_length_array.nc.6.bescmd                ok
118: nc/zero_length_array.nc.7.bescmd                ok
115: nc/ref_tst_opaque_data.nc.dap.bescmd            FAILED (testsuite.at:235)
116: nc/stringtest.nc.dap.bescmd                     FAILED (testsuite.at:236)
120: nc/zero_length_array.nc.9.bescmd                FAILED (testsuite.at:243)
119: nc/zero_length_array.nc.8.bescmd                ok
122: nc/string_attribute_values.nc.2.bescmd          ok
123: nc/string_attribute_values.nc.3.bescmd          FAILED (testsuite.at:247)
121: nc/zero_length_array.nc.10.bescmd               FAILED (testsuite.at:244)

## ------------- ##
## Test results. ##
## ------------- ##

ERROR: All 123 tests were run,
46 failed unexpectedly.
## -------------------------- ##
## testsuite.log was created. ##
## -------------------------- ##

Please send 'modules/netcdf_handler/tests/testsuite.log' and all information you think might help:

   To: <support@opendap.org>
   Subject: [bes 3.21.1] testsuite: 15 16 17 18 19 20 21 22 23 28 29 33 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 120 121 123 failed

Where we look at one failed test and see:

netcdf_handler/tests % ./testsuite -v 15
## ------------------------------------------------------------------ ##
## bes 3.21.1 test suite: bes/modules/netcdf_handler/tests testsuite. ##
## ------------------------------------------------------------------ ##
15. testsuite.at:26: testing nc/coads_climatology.nc.dmr.bescmd ...
COMMAND: besstandalone  -c bes.conf -i nc/coads_climatology.nc.dmr.bescmd
./testsuite.at:26: besstandalone $repeat -c $abs_builddir/$bes_conf -i $input
stdout:
<?xml version="1.0" encoding="ISO-8859-1"?>
<Dataset xmlns="http://xml.opendap.org/ns/DAP/4.0#" dapVersion="4.0" dmrVersion="2.0" name="coads_climatology.nc">
    <Dimension name="TIME" size="12"/>
    <Dimension name="COADSY" size="90"/>
    <Dimension name="COADSX" size="180"/>
    <Float64 name="TIME">
        <Dim name="/TIME"/>
        <Attribute name="units" type="String">
            <Value>hour since 0000-01-01 00:00:00</Value>
        </Attribute>
...
            <Value>FERRET V4.30 (debug/no GUI) 15-Aug-96</Value>
        </Attribute>
    </Attribute>
    <Attribute name="DODS_EXTRA" type="Container">
        <Attribute name="Unlimited_Dimension" type="String">
            <Value>TIME</Value>
        </Attribute>
    </Attribute>
</Dataset>
./testsuite.at:26: diff -b -B $baseline stdout
--- /dev/null	2026-01-21 09:50:51
+++ /Users/jimg/src/opendap/hyrax/bes/modules/netcdf_handler/tests/testsuite.dir/at-groups/15/stdou2026-01-21 09:50:52
@@ -0,0 +1,4 @@
+2c2
+< <Dataset xmlns="http://xml.opendap.org/ns/DAP/4.0#" dapVersion="4.0" dmrVersion="removed" name="coads_climatology.nc">
+---
+> <Dataset xmlns="http://xml.opendap.org/ns/DAP/4.0#" dapVersion="4.0" dmrVersion="2.0" name="coads_climatology.nc">
./testsuite.at:26: exit code was 1, expected 0
15. testsuite.at:26:  FAILED (testsuite.at:26)

@hannahilea
Copy link
Contributor

hannahilea commented Jan 21, 2026

Interesting; I don't understand how that wasn't caught in Travis when we stripped out dmr version?! The nc ones, also.

I wonder if there's something unique about how the tests are running locally for you vs locally for me vs in Travis....

I'm taking a look now.

@hannahilea
Copy link
Contributor

hannahilea commented Jan 21, 2026

When I check out this branch locally (m4, Sequoia 15.4.1) and run the bes tests, I do see the GlobalMetadataStoreTest failure (working on a fix now! tldr; it doesn't go through our existing "strip versions" mechanisms yet) but do not see the netcdf failures. Very weird. Can you confirm that you're otherwise up to date with bes master, and there are no additional changes in bes on your machine?

@hannahilea
Copy link
Contributor

hannahilea commented Jan 21, 2026

Okay, I added the fix to bes for the GlobalMetadataStoreTest in OPENDAP/bes#1232.

I was not able to reproduce the failing netcdf_handler tests, even after manually removing ./testsuite (as discussed in slack). @jgallagher59701, can an you rerun your local tests on this new bes branch and report back if both types of tests are still failing?

@jgallagher59701
Copy link
Member Author

jgallagher59701 commented Jan 21, 2026

Okay, I added the fix to bes for the GlobalMetadataStoreTest in OPENDAP/bes#1232.

I was not able to reproduce the failing netcdf_handler tests, even after manually removing ./testsuite (as discussed in slack). @jgallagher59701, can an you rerun your local tests on this new bes branch and report back if both types of tests are still failing?

When you removed testsuite you 'fixed' the problem. In fact, and I probably was not clear before, the problem only exists with an old testsuite script.

@hannahilea
Copy link
Contributor

hannahilea commented Jan 22, 2026

When you removed testsuite you 'fixed' the problem. In fact, and I probably was not clear before, the problem only exists with an old testsuite script.

Okay, I think I understand: it was failing on your machine because you hadn't yet removed the stale testsuite?

If that's correct, then I think we should be good to go here, now that #BES1232 is merged

@jgallagher59701
Copy link
Member Author

jgallagher59701 commented Jan 22, 2026

When you removed testsuite you 'fixed' the problem. In fact, and I probably was not clear before, the problem only exists with an old testsuite script.

Okay, I think I understand: it was failing on your machine because you hadn't yet removed the stale testsuite?

If that's correct, then I think we should be good to go here, now that #BES1232 is merged

OK, I'm going to merge this...

@jgallagher59701 jgallagher59701 merged commit e3bf0b3 into master Jan 22, 2026
5 checks passed
@Mikejmnez
Copy link
Contributor

Mikejmnez commented Jan 22, 2026

@jgallagher59701 Out of curiousity - how long until the test.opendap server bumps this version too?

@jgallagher59701
Copy link
Member Author

jgallagher59701 commented Jan 22, 2026

@jgallagher59701 Out of curiousity - how long until the test.opendap server bumps this version too?

Once we have a new container, we could do that today. Or maybe tomorrow. I may need to leave early today because I need to run an errand at a place that closes at 4pm. I will work at home after that and I guess I could update test.o.o from there...

So, today or tomorrow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hannahilea hannahilea hannahilea approved these changes

@ndp-opendap ndp-opendap ndp-opendap approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jgallagher59701 @hannahilea @Mikejmnez @ndp-opendap