OCPEDGE-2326: add storageclass configuration options to lvms#1965
OCPEDGE-2326: add storageclass configuration options to lvms#1965Neilhamza wants to merge 16 commits intoopenshift:mainfrom
Conversation
|
@Neilhamza: This pull request references OCPEDGE-2326 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Neilhamza The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@Neilhamza: This pull request references OCPEDGE-2326 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1965 +/- ##
==========================================
- Coverage 48.11% 46.83% -1.28%
==========================================
Files 52 52
Lines 3735 3978 +243
==========================================
+ Hits 1797 1863 +66
- Misses 1782 1949 +167
- Partials 156 166 +10
🚀 New features to boost your workflow:
|
8a06966 to
bd2bf5a
Compare
|
@Neilhamza: This pull request references OCPEDGE-2326 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
fc7f998 to
d8a2f74
Compare
|
/сс |
| @@ -117,6 +118,57 @@ func DeleteResource(ctx context.Context, obj client.Object) { | |||
| }, timeout, interval).WithContext(ctx).Should(Satisfy(k8serrors.IsNotFound)) | |||
| } | |||
|
|
|||
There was a problem hiding this comment.
waitForStorageCleanup prevents race where LVM metadata flush is async: vgremove/lvremove
return before disk flush completes, causing next test to find stale metadata (VG exists,
thin-pool missing) and mark cluster Degraded. Observed in HyperShift CI job.
|
/retest |
|
/retest list |
|
/test e2e-aws-hypershift |
1 similar comment
|
/test e2e-aws-hypershift |
Signed-off-by: nhamza <nhamza@redhat.com>
Signed-off-by: nhamza <nhamza@redhat.com>
Signed-off-by: nhamza <nhamza@redhat.com>
Signed-off-by: nhamza <nhamza@redhat.com>
Signed-off-by: nhamza <nhamza@redhat.com>
Signed-off-by: nhamza <nhamza@redhat.com>
d1dd8d6 to
b8ef8f6
Compare
Signed-off-by: nhamza <nhamza@redhat.com>
b8ef8f6 to
7c139c0
Compare
api/v1alpha1/lvmcluster_webhook.go
Outdated
| } | ||
|
|
||
| scName := constants.StorageClassPrefix + newDC.Name | ||
| sc := &storagev1.StorageClass{} |
There was a problem hiding this comment.
Where do you use it? I can't find anywhere
There was a problem hiding this comment.
The sc variable is required for the Get() call right after, which is used to check whether the StorageClass already exists.
This existence check gates the immutability validation:
– If the StorageClass does not exist yet (NotFound), immutability checks are skipped
– If the StorageClass exists, updates to storageClassOptions are rejected
There was a problem hiding this comment.
But it can be created during this webhook call. Basically we need to check old and new CR configuration, not with existing objects. The cluster will be reconciled later accordingly
There was a problem hiding this comment.
completely rewrote the immutability check in api/v1alpha1/lvmcluster_webhook.go:
removed:
- The StorageClass GET call (the r.Get(ctx, types.NamespacedName{Name: scName}, sc)
- Any dependency on checking the actual StorageClass object in the cluster
added:
- immutableSCFields struct (lines 623-629) - holds only the immutable fields we care about
- effectiveImmutable() helper (lines 631-661) - extracts immutable fields from a DeviceClass and applies defaults (so nil -> WaitForFirstConsumer, nil -> Delete, etc.)
- verifyImmutableStorageClassFields() rewrite (lines 663-693) - now compares old CR spec vs new CR spec only, not cluster objects:
internal/controllers/lvmcluster/resource/topolvm_storageclass.go
Outdated
Show resolved
Hide resolved
internal/controllers/lvmcluster/resource/topolvm_storageclass.go
Outdated
Show resolved
Hide resolved
internal/controllers/lvmcluster/resource/topolvm_storageclass.go
Outdated
Show resolved
Hide resolved
internal/controllers/lvmcluster/resource/topolvm_storageclass.go
Outdated
Show resolved
Hide resolved
| return sc | ||
| } | ||
|
|
||
| func (s topolvmStorageClass) applyAdditionalLabels( |
There was a problem hiding this comment.
Why do we need this? Why do we need to store additional labels in annotation?
There was a problem hiding this comment.
he annotation tracks which labels were added by LVMS via additionalLabels so they
can be safely pruned when removed from the spec, without touching user's own labels.
Example scenario:
- User adds label "env=prod" via additionalLabels then annotation stores "env"
- User removes "env=prod" from spec then we check annotation, see "env" was LVMS-managed
- We safely delete "env" label from StorageClass
- Any labels NOT in the annotation (user's own labels) are left untouched
This prevents orphaned labels when users modify their additionalLabels configuration.
See applyAdditionalLabels() lines 258-263 for pruning logic.
There was a problem hiding this comment.
if annotation LVMS managed - user is unable to modify it. It's our object, we manage it, and we allow user to modify only certain fields
There was a problem hiding this comment.
Right now (current implemntation) LVMS uses a partial ownership model for StorageClass labels: we only manage the keys coming from additionalLabels, and the annotation tracks which keys LVMS previously applied so we can safely prune them when removed from the spec without touching unrelated labels added by admins or external tooling.
The alternative (to implement your suggestion) would be strict ownership, where LVMS overwrites the entire labels map from the CR on every reconcile (no annotation tracking needed), but that would drop any non-LVMS labels set directly on the StorageClass.
The first approach keeps day-2 reconciliation safe while avoiding unintended deletion of user/policy-applied labels.
can i have your suggestion on how to proceed?
Signed-off-by: nhamza <nhamza@redhat.com>
Signed-off-by: nhamza <nhamza@redhat.com>
|
/retest |
Signed-off-by: nhamza <nhamza@redhat.com>
|
/retest Red Hat Konflux / lvm-operator-integration-tests-4-21 / lvm-operator-4-21 |
|
/test Red Hat Konflux / lvm-operator-integration-tests-4-21 / lvm-operator-4-21 |
Signed-off-by: nhamza <nhamza@redhat.com>
|
/retest |
1 similar comment
|
/retest |
Signed-off-by: nhamza <nhamza@redhat.com>
| } | ||
|
|
||
| for _, deviceClass := range l.Spec.Storage.DeviceClasses { | ||
| if deviceClass.StorageClassOptions == nil { |
There was a problem hiding this comment.
what if they were exist and now deleted? Will values be defaulted and operator might try to change immutable parameters and fail?
There was a problem hiding this comment.
Added effectiveImmutable() helper that applies defaults to both old and new specs before comparison, catching nil -> defaults immutability violations.
api/v1alpha1/lvmcluster_webhook.go
Outdated
| } | ||
|
|
||
| scName := constants.StorageClassPrefix + newDC.Name | ||
| sc := &storagev1.StorageClass{} |
There was a problem hiding this comment.
But it can be created during this webhook call. Basically we need to check old and new CR configuration, not with existing objects. The cluster will be reconciled later accordingly
internal/controllers/lvmcluster/resource/topolvm_storageclass.go
Outdated
Show resolved
Hide resolved
| continue | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
you need to check if deletion timestamp is not nil, otherwise delete will return error
There was a problem hiding this comment.
Added if !pv.DeletionTimestamp.IsZero() { continue } check before attempting delete in cleanup function.
internal/controllers/lvmcluster/resource/topolvm_storageclass.go
Outdated
Show resolved
Hide resolved
| } | ||
|
|
||
| // Delete LogicalVolumes | ||
| lvList := &topolvmv1.LogicalVolumeList{} |
There was a problem hiding this comment.
topolvm logical volumes managed by topolvm controllers and will be deleted automatically when lvmcluster is being deleted
There was a problem hiding this comment.
this is the most tricky part, this function is only called when we are in retain policy. if we dont manually delete LogicalVolumes the user cannot delete lvm cluster because it will simply hang - since there will be LV's leftover (which is retain..)
i can either go with the option to leave the code as is, or block the user from deleting lvm-cluster and send a message that he must manually delete retained volumes (and finalizes)
There was a problem hiding this comment.
i updated code by adding multiple gates that checks if we are in/touching retain related
Kept LV deletion scoped only to Retain deviceClasses with bounded wait. Without this, LV finalizers block deletion after TopoLVM controller is removed.
internal/controllers/lvmcluster/resource/topolvm_storageclass.go
Outdated
Show resolved
Hide resolved
|
|
||
| // Only set immutable spec fields on creation (ResourceVersion is empty for new objects) | ||
| if sc.ResourceVersion == "" { | ||
| sc.Provisioner = desired.Provisioner |
There was a problem hiding this comment.
btw, do we need this? We do query actual StorageClass in s.getTopolvmStorageClasses. Can't we check for immutable fields there so we won't override them using actual StorageClass? This would also reduce this function size and avoid duplication in construction of StorageClass
There was a problem hiding this comment.
We still need this guard because getTopolvmStorageClasses() builds the desired spec, but CreateOrUpdate will re-run the mutate function on every reconcile, without the ResourceVersion == "" check we could accidentally try to update immutable SC fields. This makes sure immutable fields are only set on create, so we don’t accidentally try to change them on later reconciles
| return sc | ||
| } | ||
|
|
||
| func (s topolvmStorageClass) applyAdditionalLabels( |
There was a problem hiding this comment.
if annotation LVMS managed - user is unable to modify it. It's our object, we manage it, and we allow user to modify only certain fields
Signed-off-by: nhamza <nhamza@redhat.com>
|
/retest |
|
@Neilhamza: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
this PR completes jiras:
https://issues.redhat.com/browse/OCPEDGE-2326
https://issues.redhat.com/browse/OCPEDGE-2327
https://issues.redhat.com/browse/OCPEDGE-2328
Adds configurable StorageClassOptions at the DeviceClass level in LVMCluster, allowing users to control how LVMS-generated StorageClasses are created while respecting Kubernetes immutability guarantees.
Unit tests and e2e coverage will be added in a follow-up PR.
Changes
API
Added StorageClassOptions to DeviceClass with optional:
reclaimPolicy
volumeBindingMode
additionalParameters
additionalLabels
Webhook
Validates StorageClassOptions on create/update
Warns on conflicts with LVMS-owned parameters and labels
Validates label keys/values per Kubernetes rules
Rejects updates to immutable StorageClass fields after creation
Controller
Uses CreateOrUpdate for StorageClass reconciliation
Sets immutable fields at creation time only
Supports day-2 updates for metadata-only fields (additionalLabels)
Ownership-aware label pruning ensures spec is the source of truth
No delete/recreate behavior for StorageClasses
Immutability Model
StorageClass behavior fields are immutable in Kubernetes and are configurable only at creation time.
Metadata-only fields (labels) are safely reconciled day-2 without impacting existing PVCs.
Example
apiVersion: lvm.topolvm.io/v1alpha1
kind: LVMCluster
metadata:
name: lvmcluster
spec:
storage:
deviceClasses:
- name: vg1
storageClassOptions:
reclaimPolicy: Retain
volumeBindingMode: Immediate
additionalParameters:
provisioner-secret-name: my-secret
additionalLabels:
environment: production