Skip to content

Conversation

@hyp3rd
Copy link
Owner

@hyp3rd hyp3rd commented Dec 20, 2025

  • Introduce pkg/middleware with OTEL tracing and stats scaffolding
  • Set up GitHub Actions (lint, test, security) and Dependabot
  • Add .golangci.yaml, cspell config, and pre-commit hooks (gci, gofumpt,
    govulncheck, go mod tidy/verify)
  • Add .project-settings.env and wire into Makefile to pin toolchain
    (GO_VERSION=1.25.5, GOLANGCI_LINT_VERSION=v2.7.2)
  • Update go.mod to Go 1.25.5; upgrade deps: fiber v3.0.0-rc.3,
    go-redis v9.17.2, msgpack v2.4.0, x/crypto v0.46.0, ewrap v1.3.2
  • Refactor internals: normalize transport.Client parameter order,
    adjust cluster.NewNode signature, and simplify slice bounds in
    distributed memory with min(...)
  • Reorder imports and apply formatting; minor docs/cleanup

hyp3rd and others added 5 commits August 25, 2025 01:28
…ory tracking

- Document phase 3 rebalancing & ownership migration system in README
- Add comprehensive metrics table for rebalance monitoring
- Update cspell dictionary with new technical terms (mrand, skeys)
- Enhance distShard struct with originalPrimary tracking for ownership changes

This commit documents the experimental periodic rebalancer that handles
key migration when node ownership changes, including batch processing,
concurrency control, and metrics collection. The originalPrimary field
addition enables tracking ownership changes for proper rebalancing.
- Update PRD.md with detailed completion status for distributed cache features
- Add integration tests for replica-only differential rebalancing scenarios
- Add throttling validation test for per-tick replica diff limits
- Track progress on node membership, consistent hashing, replication, and anti-entropy features

The roadmap now clearly shows completed vs planned work including:
- ✅ Basic clustering, replication, failure detection, tombstones, hinted handoff
- 🔄 Dynamic gossip, adaptive anti-entropy scheduling, client SDK
- ⏳ Pluggable compression, security (TLS + auth)
- Introduce pkg/middleware with OTEL tracing and stats scaffolding
- Set up GitHub Actions (lint, test, security) and Dependabot
- Add .golangci.yaml, cspell config, and pre-commit hooks (gci, gofumpt,
  govulncheck, go mod tidy/verify)
- Add .project-settings.env and wire into Makefile to pin toolchain
  (GO_VERSION=1.25.5, GOLANGCI_LINT_VERSION=v2.7.2)
- Update go.mod to Go 1.25.5; upgrade deps: fiber v3.0.0-rc.3,
  go-redis v9.17.2, msgpack v2.4.0, x/crypto v0.46.0, ewrap v1.3.2
- Refactor internals: normalize transport.Client parameter order,
  adjust cluster.NewNode signature, and simplify slice bounds in
  distributed memory with min(...)
- Reorder imports and apply formatting; minor docs/cleanup
- Introduce pkg/middleware with OTEL tracing and stats scaffolding
- Set up GitHub Actions (lint, test, security) and Dependabot
- Add .golangci.yaml, cspell config, and pre-commit hooks (gci, gofumpt,
  govulncheck, go mod tidy/verify)
- Add .project-settings.env and wire into Makefile to pin toolchain
  (GO_VERSION=1.25.5, GOLANGCI_LINT_VERSION=v2.7.2)
- Update go.mod to Go 1.25.5; upgrade deps: fiber v3.0.0-rc.3,
  go-redis v9.17.2, msgpack v2.4.0, x/crypto v0.46.0, ewrap v1.3.2
- Refactor internals: normalize transport.Client parameter order,
  adjust cluster.NewNode signature, and simplify slice bounds in
  distributed memory with min(...)
- Reorder imports and apply formatting; minor docs/cleanup
- Introduce pkg/middleware with OTEL tracing and stats scaffolding
- Set up GitHub Actions (lint, test, security) and Dependabot
- Add .golangci.yaml, cspell config, and pre-commit hooks (gci, gofumpt,
  govulncheck, go mod tidy/verify)
- Add .project-settings.env and wire into Makefile to pin toolchain
  (GO_VERSION=1.25.5, GOLANGCI_LINT_VERSION=v2.7.2)
- Update go.mod to Go 1.25.5; upgrade deps: fiber v3.0.0-rc.3,
  go-redis v9.17.2, msgpack v2.4.0, x/crypto v0.46.0, ewrap v1.3.2
- Refactor internals: normalize transport.Client parameter order,
  adjust cluster.NewNode signature, and simplify slice bounds in
  distributed memory with min(...)
- Reorder imports and apply formatting; minor docs/cleanup
Copilot AI review requested due to automatic review settings December 20, 2025 13:19
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces significant infrastructure and distributed backend improvements to HyperCache:

Purpose: Add OpenTelemetry middleware, comprehensive CI/CD pipeline, pre-commit tooling, upgrade to Go 1.25.5 (problematic version), and refactor the distributed memory backend with rebalancing capabilities.

Key Changes:

  • Introduce pkg/middleware with OTEL tracing scaffolding
  • Set up GitHub Actions workflows for linting, testing, and security scanning
  • Add pre-commit hooks and development tooling configuration
  • Upgrade dependencies (fiber v3 RC, redis, msgpack, crypto libs)
  • Implement distributed memory rebalancing with replica diff logic
  • Refactor transport interfaces and cluster node creation

Reviewed changes

Copilot reviewed 78 out of 79 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
go.mod/go.sum Bump to Go 1.25.5 (invalid version) and upgrade dependencies
.github/workflows/* Add comprehensive CI pipelines (lint, test, security, pre-commit)
.pre-commit/* Add pre-commit hooks for gci, golangci-lint, go mod verification
.project-settings.env Pin toolchain versions for consistency
pkg/backend/dist_memory.go Major rebalancing implementation with replica diff and shedding logic
pkg/cache/v2/* Rename package from cachev2 to v2 (breaking change)
tests/benchmark/* Update benchmark loops to use b.Loop() (invalid API)
pkg/middleware/* Consolidate package documentation into middleware.go
Makefile Enhance with security scanning, proto support, toolchain management
docs/distributed.md Add comprehensive distributed backend documentation
Comments suppressed due to low confidence (1)

pkg/backend/dist_memory.go:1743

  • The function signature has an incorrect duplicate closing brace at line 1728 followed by an opening brace and wg.Add(1) at line 1730. This creates a syntax error - the goroutine function body starts at line 1719 with wg.Go() but there's also a conflicting go func() at line 1730. These two concurrent execution patterns are duplicated and will not compile.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@@ -1,4 +1,4 @@
package cachev2
package v2
Copy link

Copilot AI Dec 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The package declaration changed from "cachev2" to "v2", but this breaks the expected import path. Go modules should use a versioned directory path (e.g., "v2" in the directory name) with a package name that matches or is reasonable. Changing the package name from "cachev2" to "v2" is unconventional and may cause import confusion. The package should typically remain "cachev2" or match the directory name convention.

Copilot uses AI. Check for mistakes.
b.ReportAllocs()

for range b.N { // standard Go benchmark loop
for b.Loop() { // standard Go benchmark loop
Copy link

Copilot AI Dec 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The benchmark loop pattern changed from "for range b.N" to "for b.Loop()" but b.Loop() is not a standard method in Go's testing.B type. The standard benchmark loop pattern is "for range b.N" (Go 1.22+) or "for i := 0; i < b.N; i++" (older). This will cause compilation errors.

Copilot uses AI. Check for mistakes.
}

// IterCb is the iterator callbacalled for every key,value found in
// IterCb is the iterator calledback for every key,value found in
Copy link

Copilot AI Dec 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spelling error "callbacalled" should be corrected to "called back" or "callback called". The comment currently reads "IterCb is the iterator callbacalled for every key,value found in" which is not a valid word.

Copilot uses AI. Check for mistakes.
- Introduce ctx across cache constructors and operations.
- Change: NewInMemoryWithDefaults(ctx, capacity) (was NewInMemoryWithDefaults(capacity)).
- Add DefaultTimeout (5s); examples use context.WithTimeout where appropriate.
- Update imports to pkg/cache/v2; revise examples, tests, and README to pass ctx.
- Add test helpers for distributed memory backend (test-only).
- CI: run integration tests by setting RUN_INTEGRATION_TEST=yes in test workflow.

BREAKING CHANGE: Public cache constructor and operations now accept context.Context.
Update call sites to pass a ctx, e.g. propagate request ctx or use context.WithTimeout(DefaultTimeout).
Also update imports to github.com/hyp3rd/hypercache/pkg/cache/v2 where used, and call Stop(ctx) instead of Stop().
- cache: use sectools/converters for safe numeric conversions in eviction
  - initialize eviction algorithm with converted maxEvictionCount
  - report item_count/evicted_item_count as int64 gauges
- backend: replace math/rand with crypto/rand for heartbeat sampling
- deps: bump github.com/hyp3rd/ewrap to v1.3.3; add github.com/hyp3rd/sectools v1.0.4
- ci: add .pre-commit-ci-config.yaml and point pre-commit workflow to it
- security/ci: tighten gosec flags (exclude generated, __examples/size); update Makefile
- docs: minor README/ROADMAP formatting tweaks
…port

- Add touchBackend and hyperCache.touchItem; call from Get*, GetOrSet,
  and GetMultiple to bump access metadata and optionally touch backend.
- Implement Touch(ctx, key) in DistMemory and InMemory; wire through
  cmap.Touch to update LastAccess/AccessCount with proper locking.
- Introduce cmap.GetCopy and adopt in InMemory/DistMemory/transports to
  avoid exposing internal references.
- Make InProcessTransport concurrency-safe: guard backends with RWMutex,
  add lookup helper, and update Forward*/Health/Merkle paths accordingly.
- Refactor distributed backend loops to take explicit stopCh and unify
  stop channel handling (reaper, rebalance, hints, heartbeats, gossip,
  autosync, etc.).
- membership.List now guards nil nodes and returns clones to prevent
  external mutation.

Why:
- Standardizes “touch” semantics across cache and backends.
- Reduces data races/aliasing; improves transport thread-safety.
- Improves lifecycle/shutdown control for background routines.

Note:
- In-memory Get paths now use GetCopy; callers relying on mutating
  returned items should switch to Set/Update APIs.
- Normalize branch arrays and quoting in CodeQL workflow; quote cron with double quotes
- Add github/codeql-action/autobuild@v2 and tidy step order/indentation
- Keep CodeQL analysis step with category param
- Expand .yamllint.yaml ignore list (FUNDING.yml, .github/workflows/codeql.yml)
- docs(PRD, distributed): minor wording and ordered‑list renumbering
- docs(examples): renumber lists in __examples/README.md
- chore(cspell): add domain terms (goroutines, intrange, upserts, noctx, mfinal)
@hyp3rd hyp3rd merged commit b8ad374 into main Dec 23, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants