-
-
Notifications
You must be signed in to change notification settings - Fork 0
feat(middleware,ci): add OTEL middleware, CI/pre-commit; bump Go 1.25.5 #55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…ory tracking - Document phase 3 rebalancing & ownership migration system in README - Add comprehensive metrics table for rebalance monitoring - Update cspell dictionary with new technical terms (mrand, skeys) - Enhance distShard struct with originalPrimary tracking for ownership changes This commit documents the experimental periodic rebalancer that handles key migration when node ownership changes, including batch processing, concurrency control, and metrics collection. The originalPrimary field addition enables tracking ownership changes for proper rebalancing.
- Update PRD.md with detailed completion status for distributed cache features - Add integration tests for replica-only differential rebalancing scenarios - Add throttling validation test for per-tick replica diff limits - Track progress on node membership, consistent hashing, replication, and anti-entropy features The roadmap now clearly shows completed vs planned work including: - ✅ Basic clustering, replication, failure detection, tombstones, hinted handoff - 🔄 Dynamic gossip, adaptive anti-entropy scheduling, client SDK - ⏳ Pluggable compression, security (TLS + auth)
- Introduce pkg/middleware with OTEL tracing and stats scaffolding - Set up GitHub Actions (lint, test, security) and Dependabot - Add .golangci.yaml, cspell config, and pre-commit hooks (gci, gofumpt, govulncheck, go mod tidy/verify) - Add .project-settings.env and wire into Makefile to pin toolchain (GO_VERSION=1.25.5, GOLANGCI_LINT_VERSION=v2.7.2) - Update go.mod to Go 1.25.5; upgrade deps: fiber v3.0.0-rc.3, go-redis v9.17.2, msgpack v2.4.0, x/crypto v0.46.0, ewrap v1.3.2 - Refactor internals: normalize transport.Client parameter order, adjust cluster.NewNode signature, and simplify slice bounds in distributed memory with min(...) - Reorder imports and apply formatting; minor docs/cleanup
- Introduce pkg/middleware with OTEL tracing and stats scaffolding - Set up GitHub Actions (lint, test, security) and Dependabot - Add .golangci.yaml, cspell config, and pre-commit hooks (gci, gofumpt, govulncheck, go mod tidy/verify) - Add .project-settings.env and wire into Makefile to pin toolchain (GO_VERSION=1.25.5, GOLANGCI_LINT_VERSION=v2.7.2) - Update go.mod to Go 1.25.5; upgrade deps: fiber v3.0.0-rc.3, go-redis v9.17.2, msgpack v2.4.0, x/crypto v0.46.0, ewrap v1.3.2 - Refactor internals: normalize transport.Client parameter order, adjust cluster.NewNode signature, and simplify slice bounds in distributed memory with min(...) - Reorder imports and apply formatting; minor docs/cleanup
- Introduce pkg/middleware with OTEL tracing and stats scaffolding - Set up GitHub Actions (lint, test, security) and Dependabot - Add .golangci.yaml, cspell config, and pre-commit hooks (gci, gofumpt, govulncheck, go mod tidy/verify) - Add .project-settings.env and wire into Makefile to pin toolchain (GO_VERSION=1.25.5, GOLANGCI_LINT_VERSION=v2.7.2) - Update go.mod to Go 1.25.5; upgrade deps: fiber v3.0.0-rc.3, go-redis v9.17.2, msgpack v2.4.0, x/crypto v0.46.0, ewrap v1.3.2 - Refactor internals: normalize transport.Client parameter order, adjust cluster.NewNode signature, and simplify slice bounds in distributed memory with min(...) - Reorder imports and apply formatting; minor docs/cleanup
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR introduces significant infrastructure and distributed backend improvements to HyperCache:
Purpose: Add OpenTelemetry middleware, comprehensive CI/CD pipeline, pre-commit tooling, upgrade to Go 1.25.5 (problematic version), and refactor the distributed memory backend with rebalancing capabilities.
Key Changes:
- Introduce pkg/middleware with OTEL tracing scaffolding
- Set up GitHub Actions workflows for linting, testing, and security scanning
- Add pre-commit hooks and development tooling configuration
- Upgrade dependencies (fiber v3 RC, redis, msgpack, crypto libs)
- Implement distributed memory rebalancing with replica diff logic
- Refactor transport interfaces and cluster node creation
Reviewed changes
Copilot reviewed 78 out of 79 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| go.mod/go.sum | Bump to Go 1.25.5 (invalid version) and upgrade dependencies |
| .github/workflows/* | Add comprehensive CI pipelines (lint, test, security, pre-commit) |
| .pre-commit/* | Add pre-commit hooks for gci, golangci-lint, go mod verification |
| .project-settings.env | Pin toolchain versions for consistency |
| pkg/backend/dist_memory.go | Major rebalancing implementation with replica diff and shedding logic |
| pkg/cache/v2/* | Rename package from cachev2 to v2 (breaking change) |
| tests/benchmark/* | Update benchmark loops to use b.Loop() (invalid API) |
| pkg/middleware/* | Consolidate package documentation into middleware.go |
| Makefile | Enhance with security scanning, proto support, toolchain management |
| docs/distributed.md | Add comprehensive distributed backend documentation |
Comments suppressed due to low confidence (1)
pkg/backend/dist_memory.go:1743
- The function signature has an incorrect duplicate closing brace at line 1728 followed by an opening brace and wg.Add(1) at line 1730. This creates a syntax error - the goroutine function body starts at line 1719 with wg.Go() but there's also a conflicting go func() at line 1730. These two concurrent execution patterns are duplicated and will not compile.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -1,4 +1,4 @@ | |||
| package cachev2 | |||
| package v2 | |||
Copilot
AI
Dec 20, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The package declaration changed from "cachev2" to "v2", but this breaks the expected import path. Go modules should use a versioned directory path (e.g., "v2" in the directory name) with a package name that matches or is reasonable. Changing the package name from "cachev2" to "v2" is unconventional and may cause import confusion. The package should typically remain "cachev2" or match the directory name convention.
| b.ReportAllocs() | ||
|
|
||
| for range b.N { // standard Go benchmark loop | ||
| for b.Loop() { // standard Go benchmark loop |
Copilot
AI
Dec 20, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The benchmark loop pattern changed from "for range b.N" to "for b.Loop()" but b.Loop() is not a standard method in Go's testing.B type. The standard benchmark loop pattern is "for range b.N" (Go 1.22+) or "for i := 0; i < b.N; i++" (older). This will cause compilation errors.
| } | ||
|
|
||
| // IterCb is the iterator callbacalled for every key,value found in | ||
| // IterCb is the iterator calledback for every key,value found in |
Copilot
AI
Dec 20, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The spelling error "callbacalled" should be corrected to "called back" or "callback called". The comment currently reads "IterCb is the iterator callbacalled for every key,value found in" which is not a valid word.
- Introduce ctx across cache constructors and operations. - Change: NewInMemoryWithDefaults(ctx, capacity) (was NewInMemoryWithDefaults(capacity)). - Add DefaultTimeout (5s); examples use context.WithTimeout where appropriate. - Update imports to pkg/cache/v2; revise examples, tests, and README to pass ctx. - Add test helpers for distributed memory backend (test-only). - CI: run integration tests by setting RUN_INTEGRATION_TEST=yes in test workflow. BREAKING CHANGE: Public cache constructor and operations now accept context.Context. Update call sites to pass a ctx, e.g. propagate request ctx or use context.WithTimeout(DefaultTimeout). Also update imports to github.com/hyp3rd/hypercache/pkg/cache/v2 where used, and call Stop(ctx) instead of Stop().
- cache: use sectools/converters for safe numeric conversions in eviction - initialize eviction algorithm with converted maxEvictionCount - report item_count/evicted_item_count as int64 gauges - backend: replace math/rand with crypto/rand for heartbeat sampling - deps: bump github.com/hyp3rd/ewrap to v1.3.3; add github.com/hyp3rd/sectools v1.0.4 - ci: add .pre-commit-ci-config.yaml and point pre-commit workflow to it - security/ci: tighten gosec flags (exclude generated, __examples/size); update Makefile - docs: minor README/ROADMAP formatting tweaks
…port - Add touchBackend and hyperCache.touchItem; call from Get*, GetOrSet, and GetMultiple to bump access metadata and optionally touch backend. - Implement Touch(ctx, key) in DistMemory and InMemory; wire through cmap.Touch to update LastAccess/AccessCount with proper locking. - Introduce cmap.GetCopy and adopt in InMemory/DistMemory/transports to avoid exposing internal references. - Make InProcessTransport concurrency-safe: guard backends with RWMutex, add lookup helper, and update Forward*/Health/Merkle paths accordingly. - Refactor distributed backend loops to take explicit stopCh and unify stop channel handling (reaper, rebalance, hints, heartbeats, gossip, autosync, etc.). - membership.List now guards nil nodes and returns clones to prevent external mutation. Why: - Standardizes “touch” semantics across cache and backends. - Reduces data races/aliasing; improves transport thread-safety. - Improves lifecycle/shutdown control for background routines. Note: - In-memory Get paths now use GetCopy; callers relying on mutating returned items should switch to Set/Update APIs.
- Normalize branch arrays and quoting in CodeQL workflow; quote cron with double quotes - Add github/codeql-action/autobuild@v2 and tidy step order/indentation - Keep CodeQL analysis step with category param - Expand .yamllint.yaml ignore list (FUNDING.yml, .github/workflows/codeql.yml) - docs(PRD, distributed): minor wording and ordered‑list renumbering - docs(examples): renumber lists in __examples/README.md - chore(cspell): add domain terms (goroutines, intrange, upserts, noctx, mfinal)
govulncheck, go mod tidy/verify)
(GO_VERSION=1.25.5, GOLANGCI_LINT_VERSION=v2.7.2)
go-redis v9.17.2, msgpack v2.4.0, x/crypto v0.46.0, ewrap v1.3.2
adjust cluster.NewNode signature, and simplify slice bounds in
distributed memory with min(...)