Open
Conversation
closes #3957 removes non primary bots from aggregations, scoring, and ranks
Contributor
📝 WalkthroughWalkthroughChanges introduce differential handling for primary versus non-primary bots across scoring and aggregation logic. Non-primary bots are now unconditionally excluded from rankings and aggregations, while primary bots are preserved when applicable. Affects evaluation question scoring, ranking assignments, and forecast aggregations. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Poem
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
🚀 Preview EnvironmentYour preview environment is ready!
Details
ℹ️ Preview Environment InfoIsolation:
Limitations:
Cleanup:
|
…disqualify-non-primary-bots
…disqualify-non-primary-bots
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In `@scoring/score_math.py`:
- Around line 365-369: The inline comment on the if not only_include_user_ids
branch is incorrect: the code excludes non-primary bots
(base_forecasts.exclude(author__is_bot=True, author__is_primary_bot=False)) when
user IDs are NOT specified, so update the comment to accurately describe that
behavior (e.g., "exclude forecasts by non-primary bots when user ids are not
explicitly specified") so it matches the conditional around
only_include_user_ids and the base_forecasts exclusion.
In `@utils/the_math/aggregations.py`:
- Around line 752-756: The comment is misleading: change the comment above the
forecasts.filter(...) call to state that non-primary bot forecasts are excluded
unless user IDs are explicitly specified (i.e., "exclude non-primary bot
forecasts unless user IDs are explicitly specified"); keep the filter logic
using forecasts.filter(Q(author__is_bot=False) | Q(author__is_primary_bot=True))
intact and ensure the corrected comment sits directly above that line so it
accurately documents the behavior.
- Around line 994-998: The comment above the filter is misleading: the code on
forecasts.filter(Q(author__is_bot=False) | Q(author__is_primary_bot=True))
actually includes non-bot authors OR primary bots (i.e., it excludes non-primary
bots), so update the comment to reflect that behavior (or if the intended
behavior was different, change the filter to match the original intent). Refer
to the forecasts variable and the filter call in this function (same pattern as
in get_aggregations_at_time) and either reword the comment to state "include
only non-bot authors or primary bots when user ids are not specified" or adjust
the Q(...) expressions to implement the original "include forecasts by
non-primary bots only when user ids explicitly specified" semantics.
Comment on lines
+365
to
+369
| if not only_include_user_ids: | ||
| # only include forecasts by non-primary bots if user ids explicitly specified | ||
| base_forecasts = base_forecasts.exclude( | ||
| author__is_bot=True, author__is_primary_bot=False | ||
| ) |
Contributor
There was a problem hiding this comment.
Same misleading comment as in aggregations.py.
Line 366 says "only include forecasts by non-primary bots if user ids explicitly specified" — the code excludes non-primary bots in this branch.
Proposed fix
if not only_include_user_ids:
- # only include forecasts by non-primary bots if user ids explicitly specified
+ # exclude non-primary bot forecasts unless user ids are explicitly specified
base_forecasts = base_forecasts.exclude(
author__is_bot=True, author__is_primary_bot=False
)🤖 Prompt for AI Agents
In `@scoring/score_math.py` around lines 365 - 369, The inline comment on the if
not only_include_user_ids branch is incorrect: the code excludes non-primary
bots (base_forecasts.exclude(author__is_bot=True, author__is_primary_bot=False))
when user IDs are NOT specified, so update the comment to accurately describe
that behavior (e.g., "exclude forecasts by non-primary bots when user ids are
not explicitly specified") so it matches the conditional around
only_include_user_ids and the base_forecasts exclusion.
Comment on lines
+752
to
+756
| else: | ||
| # only include forecasts by non-primary bots if user ids explicitly specified | ||
| forecasts = forecasts.filter( | ||
| Q(author__is_bot=False) | Q(author__is_primary_bot=True), | ||
| ) |
Contributor
There was a problem hiding this comment.
Misleading comment — says the opposite of what the code does.
The comment on line 753 reads "only include forecasts by non-primary bots if user ids explicitly specified," but the code here excludes non-primary bot forecasts. The intent is: "exclude non-primary bot forecasts unless user IDs are explicitly specified."
Proposed fix
else:
- # only include forecasts by non-primary bots if user ids explicitly specified
+ # exclude non-primary bot forecasts unless user ids are explicitly specified
forecasts = forecasts.filter(
Q(author__is_bot=False) | Q(author__is_primary_bot=True),
)🤖 Prompt for AI Agents
In `@utils/the_math/aggregations.py` around lines 752 - 756, The comment is
misleading: change the comment above the forecasts.filter(...) call to state
that non-primary bot forecasts are excluded unless user IDs are explicitly
specified (i.e., "exclude non-primary bot forecasts unless user IDs are
explicitly specified"); keep the filter logic using
forecasts.filter(Q(author__is_bot=False) | Q(author__is_primary_bot=True))
intact and ensure the corrected comment sits directly above that line so it
accurately documents the behavior.
Comment on lines
+994
to
+998
| else: | ||
| # only include forecasts by non-primary bots if user ids explicitly specified | ||
| forecasts = forecasts.filter( | ||
| Q(author__is_bot=False) | Q(author__is_primary_bot=True), | ||
| ) |
Contributor
There was a problem hiding this comment.
Same misleading comment as in get_aggregations_at_time.
Same issue — the comment says "only include forecasts by non-primary bots if user ids explicitly specified" but the code excludes them.
Proposed fix
else:
- # only include forecasts by non-primary bots if user ids explicitly specified
+ # exclude non-primary bot forecasts unless user ids are explicitly specified
forecasts = forecasts.filter(
Q(author__is_bot=False) | Q(author__is_primary_bot=True),
)🤖 Prompt for AI Agents
In `@utils/the_math/aggregations.py` around lines 994 - 998, The comment above the
filter is misleading: the code on forecasts.filter(Q(author__is_bot=False) |
Q(author__is_primary_bot=True)) actually includes non-bot authors OR primary
bots (i.e., it excludes non-primary bots), so update the comment to reflect that
behavior (or if the intended behavior was different, change the filter to match
the original intent). Refer to the forecasts variable and the filter call in
this function (same pattern as in get_aggregations_at_time) and either reword
the comment to state "include only non-bot authors or primary bots when user ids
are not specified" or adjust the Q(...) expressions to implement the original
"include forecasts by non-primary bots only when user ids explicitly specified"
semantics.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #3957
removes non primary bots from aggregations, scoring, and ranks
Summary by CodeRabbit
Improvements
Tests