-
Notifications
You must be signed in to change notification settings - Fork 68
🌱 Migrate E2e NetworkPolicy tests to static analysis with kube-score and conftest
#2393
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🌱 Migrate E2e NetworkPolicy tests to static analysis with kube-score and conftest
#2393
Conversation
✅ Deploy Preview for olmv1 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
kube-score and conftest
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR replaces runtime E2E NetworkPolicy tests with static analysis tools that validate NetworkPolicy configurations at build time, providing faster feedback during development.
Key changes:
- Removes test/e2e/network_policy_test.go (413 lines of runtime test code)
- Adds OPA policies for conftest to validate NetworkPolicy requirements
- Integrates kube-score and conftest into Makefile lint targets and manifest generation
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| test/e2e/network_policy_test.go | Removed E2E runtime tests (now covered by static analysis) |
| hack/conftest/policy/olm-networkpolicies.rego | OPA policy validating deny-all, catalogd, and operator-controller NetworkPolicies |
| hack/conftest/policy/prometheus-networkpolicies.rego | OPA policy validating Prometheus NetworkPolicy |
| hack/conftest/policy/README.md | Documentation for OPA policies and usage |
| Makefile | Integrated conftest into lint-helm and manifest generation; added lint-deployed-resources target |
| .github/workflows/files-diff.yaml | CI workflow to detect and alert on NetworkPolicy changes |
| .bingo/* | Added conftest, kube-score, and kube-linter tools via bingo |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| - name: Fail if NetworkPolicy files changed | ||
| if: steps.filter.outputs.networkpolicy == 'true' | ||
| run: | | ||
| echo "::error::NetworkPolicy files have been modified. See PR comment for details." | ||
| exit 1 |
Copilot
AI
Dec 18, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This workflow step will always fail when NetworkPolicy files change, blocking PRs that legitimately need to modify NetworkPolicies. Consider removing the hard failure (exit 1) and instead rely on the PR comment to alert reviewers. Alternatively, add a way to bypass this check (e.g., with a specific label) for intentional NetworkPolicy changes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Covered by previous step
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, do we want the comment, or do we want to fail? I'm not sure we should do both.
.bingo/Variables.mk
Outdated
| KUBE_LINTER := $(GOBIN)/kube-linter-v0.7.1 | ||
| $(KUBE_LINTER): $(BINGO_DIR)/kube-linter.mod | ||
| @# Install binary/ries using Go 1.14+ build command. This is using bwplotka/bingo-controlled, separate go module with pinned dependencies. | ||
| @echo "(re)installing $(GOBIN)/kube-linter-v0.7.1" | ||
| @cd $(BINGO_DIR) && GOWORK=off $(GO) build -mod=mod -modfile=kube-linter.mod -o=$(GOBIN)/kube-linter-v0.7.1 "golang.stackrox.io/kube-linter/cmd/kube-linter" |
Copilot
AI
Dec 18, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
kube-linter is added to the bingo dependencies but is not mentioned in the PR description and doesn't appear to be used in any Makefile targets or documentation. If it's not needed for this PR, it should be removed. If it is needed, the PR description and documentation should be updated to mention it.
| deny[msg] { | ||
| count(catalogd_policies) == 1 | ||
| not catalogd_has_egress | ||
| msg := "Missing egress rules in catalogd-controller-manager NetworkPolicy. General egress is required to enables operator-controller to pull bundle images from arbitrary image registries, connect to catalogd's HTTPS server for metadata, and interact with the Kubernetes API server." |
Copilot
AI
Dec 18, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The error message incorrectly states "enables operator-controller to pull bundle images" but this is for the catalogd-controller-manager policy. The message should describe catalogd's need for egress, not operator-controller's needs. Consider: "General egress is required to permit catalogd to fetch catalog images from arbitrary container registries and communicate with the Kubernetes API server for its operational needs."
| msg := "Missing egress rules in catalogd-controller-manager NetworkPolicy. General egress is required to enables operator-controller to pull bundle images from arbitrary image registries, connect to catalogd's HTTPS server for metadata, and interact with the Kubernetes API server." | |
| msg := "Missing egress rules in catalogd-controller-manager NetworkPolicy. General egress is required to permit catalogd to fetch catalog images from arbitrary container registries and communicate with the Kubernetes API server for its operational needs." |
| deny[msg] { | ||
| count(operator_controller_policies) == 1 | ||
| not operator_controller_ingress_port_numbers[8443] | ||
| msg := "Allow traffic to port 8443 to allow Prometheus to scrape metrics from catalogd, which is essential for monitoring its performance and health." |
Copilot
AI
Dec 18, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The error message incorrectly states "to allow Prometheus to scrape metrics from catalogd" but this is for operator-controller-controller-manager on port 8443. The message should be: "Allow traffic to port 8443 to allow Prometheus to scrape metrics from operator-controller, which is crucial for monitoring its activity, reconciliations, and overall health."
| msg := "Allow traffic to port 8443 to allow Prometheus to scrape metrics from catalogd, which is essential for monitoring its performance and health." | |
| msg := "Allow traffic to port 8443 to allow Prometheus to scrape metrics from operator-controller, which is crucial for monitoring its activity, reconciliations, and overall health." |
26f3f91 to
0d17f25
Compare
0d17f25 to
3f2af72
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 11 out of 12 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| deny[msg] { | ||
| count(catalogd_policies) == 1 | ||
| not catalogd_has_egress | ||
| msg := "Missing egress rules in catalogd-controller-manager NetworkPolicy. General egress is required to enables operator-controller to pull bundle images from arbitrary image registries, connect to catalogd's HTTPS server for metadata, and interact with the Kubernetes API server." |
Copilot
AI
Dec 18, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Grammar error: "to enables" should be "to enable" (singular form).
| msg := "Missing egress rules in catalogd-controller-manager NetworkPolicy. General egress is required to enables operator-controller to pull bundle images from arbitrary image registries, connect to catalogd's HTTPS server for metadata, and interact with the Kubernetes API server." | |
| msg := "Missing egress rules in catalogd-controller-manager NetworkPolicy. General egress is required to enable operator-controller to pull bundle images from arbitrary image registries, connect to catalogd's HTTPS server for metadata, and interact with the Kubernetes API server." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should be fine without connect to catalogd's HTTPS server for metadata
| deny[msg] { | ||
| count(operator_controller_policies) == 1 | ||
| not operator_controller_has_egress | ||
| msg := "Missing egress rules in operator-controller-controller-manager NetworkPolicy. General egress is required to enables operator-controller to pull bundle images from arbitrary image registries, connect to catalogd's HTTPS server for metadata, and interact with the Kubernetes API server." |
Copilot
AI
Dec 18, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Grammar error: "to enables" should be "to enable" (singular form).
| msg := "Missing egress rules in operator-controller-controller-manager NetworkPolicy. General egress is required to enables operator-controller to pull bundle images from arbitrary image registries, connect to catalogd's HTTPS server for metadata, and interact with the Kubernetes API server." | |
| msg := "Missing egress rules in operator-controller-controller-manager NetworkPolicy. General egress is required to enable operator-controller to pull bundle images from arbitrary image registries, connect to catalogd's HTTPS server for metadata, and interact with the Kubernetes API server." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might want to fix this if other changes are required.
.bingo/variables.env
Outdated
| KUBE_LINTER="${GOBIN}/kube-linter-v0.7.1" | ||
|
|
Copilot
AI
Dec 18, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
KUBE_LINTER is declared in variables.env but is not used anywhere in the project (not found in Makefile or Variables.mk). This appears to be an unused dependency that should either be integrated into the build or removed to avoid confusion.
| KUBE_LINTER="${GOBIN}/kube-linter-v0.7.1" |
3f2af72 to
12252d5
Compare
12252d5 to
0f0027d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 14 out of 15 changed files in this pull request and generated 8 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| .PHONY: run-internal | ||
| run-internal: docker-build kind-cluster kind-load kind-deploy wait | ||
| run-internal: docker-build kind-cluster kind-load kind-deploy lint-deployed-resources wait |
Copilot
AI
Dec 18, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding lint-deployed-resources to the run-internal target means that every local development run will now attempt to lint all resources in the cluster. This could slow down the development workflow and may fail if the cluster is not fully ready or if resources don't exist yet. Consider moving this to a separate verification step or making it optional for development workflows.
.bingo/variables.env
Outdated
| KUBE_LINTER="${GOBIN}/kube-linter-v0.7.1" | ||
|
|
Copilot
AI
Dec 18, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The KUBE_LINTER variable is declared in variables.env but there's no corresponding kube-linter.mod, kube-linter.sum, or Variables.mk entry to actually install this tool. Either this tool should be removed from variables.env if it's not needed, or the full bingo configuration should be added (including .mod, .sum files and Variables.mk target).
| KUBE_LINTER="${GOBIN}/kube-linter-v0.7.1" |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2393 +/- ##
==========================================
+ Coverage 68.79% 72.97% +4.17%
==========================================
Files 100 100
Lines 7641 7641
==========================================
+ Hits 5257 5576 +319
+ Misses 1948 1626 -322
- Partials 436 439 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
0f0027d to
7fa3d04
Compare
| image: busybox:1.36 | ||
| name: tar | ||
| securityContext: | ||
| readOnlyRootFilesystem: true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
note to reviewers: kube-score detected this while working on this PR and was needed to add it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ${{ steps.filter.outputs.networkpolicy_files }} | ||
| ``` | ||
| **Please ensure:** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! Thank you!
| --ignore-test container-resources \ | ||
| --ignore-test container-image-pull-policy \ | ||
| --ignore-test container-ephemeral-storage-request-and-limit \ | ||
| --ignore-test container-security-context-user-group-id |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is it worth calling out why we're ignoring these tests?
| @@ -0,0 +1,160 @@ | |||
| package main | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just a tangent: rego seems really cool and could provide a nice basis for policy-based approval of revisions...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't looked at rego in a while... Open Policy Agent...
7fa3d04 to
65b1c29
Compare
Makefile
Outdated
| echo "---" ; \ | ||
| done \ | ||
| done) | $(KUBE_SCORE) score - \ | ||
| `# TODO: currently these checks are failing, decide if resources should be fixed those to pass` \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let me create a ticket here
…d conftest Replace the e2e NetworkPolicy tests with static analysis tools that validate NetworkPolicy configurations at build time rather than runtime. Tools: - kube-score: https://github.com/zegl/kube-score - conftest: https://www.conftest.dev/ - OPA (Open Policy Agent): https://www.openpolicyagent.org/docs/latest/policy-language/ Changes: - Add kube-score via bingo for validating deployed NetworkPolicy resources - Add conftest via bingo for OPA-based policy validation of Helm charts - Add OPA policies to enforce NetworkPolicy requirements: - Deny-all default policy must exist - catalogd-controller-manager must allow ingress on ports 7443, 8443, 9443 - operator-controller-controller-manager must allow ingress on port 8443 - Both controllers must have general egress enabled - Prometheus NetworkPolicy must allow ingress/egress (when deployed) - Add lint-helm target integration with conftest policy checks - Add lint-deployed-resources target for runtime validation with kube-score - Add conftest validation to manifest generation - Add CI workflow to detect NetworkPolicy changes in PRs and post a comment - Remove network_policy_test.go as tests are now covered by static analysis This approach provides faster feedback by catching NetworkPolicy issues during helm linting and manifest generation rather than requiring a full e2e test run. Co-Authored-By: Claude <noreply@anthropic.com>
54eb789 to
bbe274b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| lint-helm: $(HELM) $(CONFTEST) #HELP Run helm linter | ||
| helm lint helm/olmv1 | ||
| helm lint helm/prometheus | ||
| (helm template olmv1 helm/olmv1; helm template prometheus helm/prometheus) | $(CONFTEST) test --policy hack/conftest/policy/ --combine -n main -n prometheus - |
Copilot
AI
Dec 19, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The subshell combining helm templates from both charts could fail silently if the first command succeeds but the second fails. Consider using set -o pipefail within the subshell or separating into individual commands to ensure both helm template operations complete successfully before piping to conftest.
| (helm template olmv1 helm/olmv1; helm template prometheus helm/prometheus) | $(CONFTEST) test --policy hack/conftest/policy/ --combine -n main -n prometheus - | |
| ( set -euo pipefail; helm template olmv1 helm/olmv1 && helm template prometheus helm/prometheus ) | $(CONFTEST) test --policy hack/conftest/policy/ --combine -n main -n prometheus - |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
^ valid
|
/approve |
|
|
||
| .PHONY: lint-deployed-resources | ||
| lint-deployed-resources: $(KUBE_SCORE) #HELP Lint deployed resources. | ||
| (for ns in $$(printf "olmv1-system\n%s\n" "$(CATD_NAMESPACE)" | uniq); do \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems to be overloading this variable (CATD_NAMESPACE), which is defined several hundred lines below this. I'm also noticing the definition of a NAMESPACE variable elsewhere, and I think we just need to clean up our use of namespaces in the Makefile... so probably not this PR, but something I noticed during the review of this.
ankitathomas
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work! Looks mostly good, just a few small suggestions
| - name: Fail if NetworkPolicy files changed | ||
| if: steps.filter.outputs.networkpolicy == 'true' | ||
| run: | | ||
| echo "::error::NetworkPolicy files have been modified. See PR comment for details." | ||
| exit 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Covered by previous step
| lint-helm: $(HELM) $(CONFTEST) #HELP Run helm linter | ||
| helm lint helm/olmv1 | ||
| helm lint helm/prometheus | ||
| (helm template olmv1 helm/olmv1; helm template prometheus helm/prometheus) | $(CONFTEST) test --policy hack/conftest/policy/ --combine -n main -n prometheus - |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
^ valid
|
|
||
| ```bash | ||
| # Run all policies (main + prometheus namespaces) | ||
| helm template olmv1 helm/olmv1 | conftest test --policy hack/conftest/policy/ --combine -n main -n prometheus - |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this require templating helm/prometheus as well?
| deny[msg] { | ||
| count(catalogd_policies) == 1 | ||
| not catalogd_has_egress | ||
| msg := "Missing egress rules in catalogd-controller-manager NetworkPolicy. General egress is required to enables operator-controller to pull bundle images from arbitrary image registries, connect to catalogd's HTTPS server for metadata, and interact with the Kubernetes API server." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should be fine without connect to catalogd's HTTPS server for metadata
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ankitathomas, perdasilva The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
08436e0
into
operator-framework:main
| - `make lint-helm` - Validates both helm/olmv1 and helm/prometheus charts (runs `main` and `prometheus` packages) | ||
| - `make manifests` - Generates and validates core OLM manifests using only `main` package policies | ||
| (Prometheus policies are intentionally skipped here, even if manifests include Prometheus resources; | ||
| they are validated via `make lint-helm`) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No mention of make lint-deployed-resources?
| (helm template olmv1 helm/olmv1; helm template prometheus helm/prometheus) | $(CONFTEST) test --policy hack/conftest/policy/ --combine -n main -n prometheus - | ||
|
|
||
| .PHONY: lint-deployed-resources | ||
| lint-deployed-resources: $(KUBE_SCORE) #HELP Lint deployed resources. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should probably be in extended help...
| lint-deployed-resources: $(KUBE_SCORE) #HELP Lint deployed resources. | |
| lint-deployed-resources: $(KUBE_SCORE) #EXHELP Lint deployed resources. |
Replace the e2e NetworkPolicy tests with static analysis tools that validate
NetworkPolicy configurations at build time rather than runtime.
Tools:
Changes:
This approach provides faster feedback by catching NetworkPolicy issues during
helm linting and manifest generation rather than requiring a full e2e test run.
Co-Authored-By: Claude noreply@anthropic.com
Reviewer Checklist