HDDS-8511. Enforce strict S3-compliant name for object store buckets #9462

Russole · 2025-12-08T12:36:24Z

What changes were proposed in this pull request?

This patch limits the non-strict naming behavior only to FSO buckets, which do not rely on S3 naming semantics.
Legacy/ObjectStore buckets—which represent the S3 bucket layout in Ozone—continue to enforce strict, S3-compliant naming rules, even when strict mode is disabled.

In summary, this PR ensures that only FSO buckets may use non-strict bucket names when
ozone.om.namespace.s3.strict=false; S3 buckets (Legacy/ObjectStore) must remain strict.

Appoach

Updated OMBucketCreateRequest to apply non-strict bucket name validation only when strict mode is disabled and the bucket layout is FSO.
Ensured that S3 buckets (Legacy/ObjectStore layout) always enforce strict, S3-compliant naming rules, regardless of the strict-mode setting.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-8511

How was this patch tested?

I relied on existing unit and integration tests, and verified that the full CI passed on my fork.
Please let me know if additional tests are needed.

…me including non-S3 compliant charaters when the om server side config 'ozone.om.namespace.s3.strict' is set false

sreejasahithi · 2025-12-08T14:53:07Z

Thanks @Russole for working on this,
According to me volume based S3 bucket detection does not look completely right because what I understand is that in multi-tenancy S3 requests route to respective tenant volumes (not always s3v) as each tenant has its own designated volume. I think its better to do by bucket layout and not by volume name.

Tejaskriya

Thanks @Russole for picking this up.
There are tests: TestOMBucketCreateRequestWithFSO and TestOMBucketCreateRequest.
Could you please add a test for this case in the FSO file?

Also, there is a method of creating linked buckets through the ozone sh bucket link (LinkBucketHandler code) command.
This also might need a check. You can choose to handle this in this PR, or create a different jira too.

adoroszlai

Thanks @Russole for working on this.

Please also update:

test cases in TestOMBucketCreateRequest (search for isStrictS3)
description of config property in hadoop-hdds/common/src/main/resources/ozone-default.xml

adoroszlai · 2025-12-14T07:20:29Z

...e-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java

+
+    // Convert BucketLayoutProto -> BucketLayout enum.
+    BucketLayout bucketLayout =
+        BucketLayout.valueOf(bucketInfo.getBucketLayout().name());
+
+    // Determine if this bucket belongs to the S3 namespace.
+    String s3VolumeName = ozoneManager.getConfiguration().get(
+        OzoneConfigKeys.OZONE_S3_VOLUME_NAME,
+        OzoneConfigKeys.OZONE_S3_VOLUME_NAME_DEFAULT);
+    boolean isS3Bucket =
+        s3VolumeName != null && s3VolumeName.equals(bucketInfo.getVolumeName());
+
+    // Compute effectiveStrict based on global strict flag, S3 namespace,
+    // and bucket layout.
+    boolean globalStrict = ozoneManager.isStrictS3();
+    boolean effectiveStrict = globalStrict;
+
+
+    // When strict=false, only S3 buckets with FSO layout should allow
+    // non-S3-compliant characters (e.g. underscore).
+    // All other S3 bucket layouts must still enforce strict naming rules.
+    if (!globalStrict && isS3Bucket) {
+      if (bucketLayout == BucketLayout.FILE_SYSTEM_OPTIMIZED) {
+        // S3 + FSO + strict=false → bypass strict S3 validation (allow '_' and other non-S3 characters)
+        effectiveStrict = false;
+      } else {
+        // S3 + non-FSO + strict=false → strict S3 validation still applies
+        // '_' and other non-S3 characters are not permitted
+        effectiveStrict = true;
+      }
+    }
+
    // Verify resource name
    OmUtils.validateBucketName(bucketInfo.getBucketName(),
-        ozoneManager.isStrictS3());
+        effectiveStrict);


Considering bucket links, multitenancy, and that S3 volume name can be updated in the config, I don't think this should be restricted to buckets in the S3 volume.

Check eligibility using bucketLayout#isObjectStore to handle LEGACY, too.

Use existing conversion logic BucketLayout#fromProto.

Suggested change

// Convert BucketLayoutProto -> BucketLayout enum.

BucketLayout bucketLayout =

BucketLayout.valueOf(bucketInfo.getBucketLayout().name());

// Determine if this bucket belongs to the S3 namespace.

String s3VolumeName = ozoneManager.getConfiguration().get(

OzoneConfigKeys.OZONE_S3_VOLUME_NAME,

OzoneConfigKeys.OZONE_S3_VOLUME_NAME_DEFAULT);

boolean isS3Bucket =

s3VolumeName != null && s3VolumeName.equals(bucketInfo.getVolumeName());

// Compute effectiveStrict based on global strict flag, S3 namespace,

// and bucket layout.

boolean globalStrict = ozoneManager.isStrictS3();

boolean effectiveStrict = globalStrict;

// When strict=false, only S3 buckets with FSO layout should allow

// non-S3-compliant characters (e.g. underscore).

// All other S3 bucket layouts must still enforce strict naming rules.

if (!globalStrict && isS3Bucket) {

if (bucketLayout == BucketLayout.FILE_SYSTEM_OPTIMIZED) {

// S3 + FSO + strict=false → bypass strict S3 validation (allow '_' and other non-S3 characters)

effectiveStrict = false;

} else {

// S3 + non-FSO + strict=false → strict S3 validation still applies

// '_' and other non-S3 characters are not permitted

effectiveStrict = true;

}

}

// Verify resource name

OmUtils.validateBucketName(bucketInfo.getBucketName(),

ozoneManager.isStrictS3());

effectiveStrict);

BucketLayout bucketLayout = BucketLayout.fromProto(bucketInfo.getBucketLayout());

boolean strict = ozoneManager.isStrictS3()

|| bucketLayout.isObjectStore(ozoneManager.getConfig().isFileSystemPathEnabled());

OmUtils.validateBucketName(bucketInfo.getBucketName(), strict);

Russole · 2025-12-16T00:17:25Z

Thanks @sreejasahithi, @adoroszlai, and @Tejaskriya for the detailed reviews.
I’ll start updating the patch based on your comments.

adoroszlai

Thanks @Russole for updating the patch.

adoroszlai · 2025-12-20T07:04:38Z

...e-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java

+    OmConfig omConfig = ozoneManager.getConfig();
+    boolean fsPathEnabled = false;
+    if (omConfig != null) {
+      fsPathEnabled = omConfig.isFileSystemPathEnabled();
+    }


Instead of adding this logic, please mock ozoneManager.getConfig() in TestOMClientRequestWithUserInfo:

ozone/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestOMClientRequestWithUserInfo.java

Line 80 in 66cb1e7

when(ozoneManager.getConfiguration()).thenReturn(ozoneConfiguration);

like:

ozone/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/file/TestOMDirectoryCreateRequest.java

Lines 101 to 102 in 66cb1e7

when(ozoneManager.getConfiguration()).thenReturn(ozoneConfiguration);

when(ozoneManager.getConfig()).thenReturn(ozoneConfiguration.getObject(OmConfig.class));

adoroszlai · 2025-12-20T07:04:53Z

...nager/src/test/java/org/apache/hadoop/ozone/om/request/bucket/TestOMBucketCreateRequest.java

+  protected OMBucketCreateRequest doPreExecute(String volumeName,
+                                               String bucketName,
+                                               OzoneManagerProtocolProtos.BucketLayoutProto layout) throws Exception {


nit: Please do not format method signature like this. Whenever visibility / return type / method name / other modifiers are changed, we would have to reindent all parameters.

adoroszlai · 2025-12-20T07:14:49Z

...java/org/apache/hadoop/ozone/om/request/bucket/TestOMBucketCreateRequestWithObjectStore.java

+    // Explicitly set bucket layout to OBJECT_STORE so the test doesn't depend on
+    // defaults or mocked OM behavior.
+    OzoneManagerProtocolProtos.BucketInfo.Builder bucketInfo =
+        newBucketInfoBuilder(bucketName, volumeName)
+            .setBucketLayout(
+                OzoneManagerProtocolProtos.BucketLayoutProto.OBJECT_STORE);


Then we don't need this new test class, test case can be added in TestOMBucketCreateRequest.

Russole · 2025-12-22T21:48:32Z

I re-ran the job on my fork and it passed.
I don’t have permission to rerun the upstream CI,
so I’d appreciate it if someone could help rerun the failed job.

ivandika3

Thanks for the patch. Left some comments.

ivandika3 · 2025-12-30T01:36:54Z

...rc/test/java/org/apache/hadoop/ozone/om/request/bucket/TestOMBucketCreateRequestWithFSO.java

+    // Arrange
+    ozoneManager.getConfiguration().setBoolean(
+        OMConfigKeys.OZONE_OM_NAMESPACE_STRICT_S3, false); // 如果常數名稱不同，改成實際的 key


Nit: Please remove the zh comment (since some community members may not understand it) and the "Arrange" comment also can be removed.

ivandika3 · 2025-12-30T01:43:28Z

...e-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java

+    boolean strict = ozoneManager.isStrictS3()
+        || bucketLayout.isObjectStore(ozoneManager.getConfig().isFileSystemPathEnabled());


Let's only check for OBS bucket (i.e. not use BucketLayout#isObjectStore which also takes into account LEGACY buckets) since it provides a stronger guarantee than LEGACY bucket which can become "object store" if we change the configuration ("ozone.om.enable.filesystem.paths").

Also let's add a small comment that OBS bucket needs to have strictly S3 name bucket unlikes FSO and LEGACY bucket.

Yes I think let's keep this s3 strict compliant name only for OBS buckets.

ivandika3 · 2025-12-30T01:48:00Z

...nager/src/test/java/org/apache/hadoop/ozone/om/request/bucket/TestOMBucketCreateRequest.java

+  protected OMBucketCreateRequest doPreExecute(String volumeName,
+      String bucketName,
+      OzoneManagerProtocolProtos.BucketLayoutProto layout) throws Exception {
+    OzoneManagerProtocolProtos.BucketInfo.Builder bucketInfo =
+        newBucketInfoBuilder(bucketName, volumeName)
+            .setBucketLayout(layout);
+    if (layout == OzoneManagerProtocolProtos.BucketLayoutProto.FILE_SYSTEM_OPTIMIZED) {
+      bucketInfo.addMetadata(OMRequestTestUtils.fsoMetadata());
+    }
+    return doPreExecute(bucketInfo);
+  }


I think this extra doPreExecute is not necessary. How about we simply use the current newBucketInfoBuilder#setBucketLayout(FILE_SYSTEM_OPTIMIZED)? Also the addMetadata is also not necessary.

Russole added 4 commits December 6, 2025 15:52

HDDS-8511. Allow only FSO type of bucket to be created with bucket na…

df9d920

…me including non-S3 compliant charaters when the om server side config 'ozone.om.namespace.s3.strict' is set false

Merge branch 'master' into HDDS-8511

ac35a4e

Add comments to clarify logic

28ccd86

Fix checkStyle

a7349c6

jojochuang added the s3 S3 Gateway label Dec 8, 2025

Tejaskriya reviewed Dec 9, 2025

View reviewed changes

adoroszlai changed the title ~~Hdds 8511. Allow only FSO type of bucket to be created with bucket name including non-S3 compliant charaters when the om server side config 'ozone.om.namespace.s3.strict' is set false~~ Hdds 8511. Enforce strict S3-compliant name for OBJECT_STORE buckets Dec 10, 2025

adoroszlai changed the title ~~Hdds 8511. Enforce strict S3-compliant name for OBJECT_STORE buckets~~ HDDS-8511. Enforce strict S3-compliant name for OBJECT_STORE buckets Dec 10, 2025

adoroszlai reviewed Dec 14, 2025

View reviewed changes

adoroszlai changed the title ~~HDDS-8511. Enforce strict S3-compliant name for OBJECT_STORE buckets~~ HDDS-8511. Enforce strict S3-compliant name for object store buckets Dec 14, 2025

Russole added 2 commits December 17, 2025 23:42

Add Unit test

d3fcfc0

Add tests for strict bucket name validation by layout

c3aad96

Russole requested review from Tejaskriya and adoroszlai December 20, 2025 01:22

Prevent NPE Exception for TestOMClientRequestWithUserInfo

cad0e4a

adoroszlai reviewed Dec 20, 2025

View reviewed changes

Address review feedback

3a62c7d

Russole requested a review from adoroszlai December 21, 2025 00:31

adoroszlai requested a review from ashishkumar50 December 22, 2025 17:54

adoroszlai requested a review from ivandika3 December 23, 2025 09:03

ivandika3 reviewed Dec 30, 2025

View reviewed changes

Refine strict bucket name validation and simplify tests

5609201

	when(ozoneManager.getConfiguration()).thenReturn(ozoneConfiguration);
	when(ozoneManager.getConfig()).thenReturn(ozoneConfiguration.getObject(OmConfig.class));

		boolean strict = ozoneManager.isStrictS3()
		\|\| bucketLayout.isObjectStore(ozoneManager.getConfig().isFileSystemPathEnabled());

HDDS-8511. Enforce strict S3-compliant name for object store buckets #9462

Are you sure you want to change the base?

HDDS-8511. Enforce strict S3-compliant name for object store buckets #9462

Conversation

Russole commented Dec 8, 2025

What changes were proposed in this pull request?

Appoach

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

sreejasahithi commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tejaskriya left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai Dec 14, 2025

Choose a reason for hiding this comment

Uh oh!

Russole commented Dec 16, 2025

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai Dec 20, 2025

Choose a reason for hiding this comment

Uh oh!

adoroszlai Dec 20, 2025

Choose a reason for hiding this comment

Uh oh!

adoroszlai Dec 20, 2025

Choose a reason for hiding this comment

Uh oh!

Russole commented Dec 22, 2025

Uh oh!

ivandika3 left a comment

Choose a reason for hiding this comment

Uh oh!

ivandika3 Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivandika3 Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

ashishkumar50 Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

ivandika3 Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sreejasahithi commented Dec 8, 2025 •

edited

Loading

Tejaskriya left a comment •

edited

Loading

ivandika3 Dec 30, 2025 •

edited

Loading