Skip to content

Conversation

@ANAMASGARD
Copy link

What this PR does

Adds a type label to the cortex_discarded_samples_total metric to distinguish between float and native histogram samples.

  • type="float" for regular float samples
  • type="histogram" for native histogram samples

This enhancement helps operators identify whether discarded samples are floats or native histograms, improving observability when debugging data ingestion issues.

Which issue(s) this PR fixes

Fixes #6221

Changes

pkg/util/validation/validate.go

  • Added SampleTypeFloat = "float" and SampleTypeHistogram = "histogram" constants
  • Updated discardedSamples metric definition to include "type" label
  • Updated all DiscardedSamples.WithLabelValues calls in ValidateSampleTimestamp, ValidateLabels, and ValidateNativeHistogram functions

pkg/ingester/ingester.go

  • Updated all DiscardedSamples.WithLabelValues calls with appropriate sample types

pkg/distributor/distributor.go

  • Updated all DiscardedSamples.WithLabelValues calls
  • Split combined float+histogram counts into separate calls for accurate categorization

pkg/util/validation/validate_test.go

  • Updated all test assertions to expect the new type label

CHANGELOG.md

  • Added enhancement entry

Checklist

  • Tests updated
  • CHANGELOG.md updated
  • Documentation: N/A (no new flags/configs)

Breaking Change Notice

Warning: This is a breaking change for dashboards and alerting rules that query cortex_discarded_samples_total. The metric now has an additional type label.

Migration Example

# Before
cortex_discarded_samples_total{reason="sample_out_of_bounds"}

# After (to get same behavior)
sum by (reason, user) (cortex_discarded_samples_total{reason="sample_out_of_bounds"})

# Or query specific type
cortex_discarded_samples_total{type="float"}
cortex_discarded_samples_total{type="histogram"}

Add a 'type' label to the cortex_discarded_samples_total metric to
distinguish between float and native histogram samples.

- type="float" for regular float samples
- type="histogram" for native histogram samples

This helps operators identify whether discarded samples are floats or
native histograms, improving observability for debugging ingestion issues.

Fixes cortexproject#6221

Signed-off-by: Gauarv Chaudhary <chaudharygaurav2004@gmail.com>
@dosubot dosubot bot added type/breaking type/observability To help know what is going on inside Cortex labels Dec 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/L type/breaking type/observability To help know what is going on inside Cortex

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add type label to discarded samples to distinguish between float and native histogram samples

1 participant