Click v8.2+ Compatibility - Breaking Changes with `mix_stderr` Parameter

[ninjinkun]

Issue: Click v8.2+ Compatibility - Breaking Changes with mix_stderr Parameter

Summary

The CLI test suite fails with Python 3.10 + Click 8.2+ due to breaking changes in Click's CliRunner class, specifically the removal of the mix_stderr parameter. This parameter was used extensively in our test suite to control how stdout and stderr are handled in test outputs.

Background

In our test suite, we use CliRunner(mix_stderr=False) to separate stdout and stderr outputs in test results. Many tests rely on this behavior when verifying output patterns and error messages. With Click 8.2+, this parameter has been completely removed, causing test failures.

Error Details

When running tests with Python 3.10 (which uses Click 8.2+), we get the following error:

TypeError: CliRunner.__init__() got an unexpected keyword argument 'mix_stderr'
======================================================================
ERROR [0.002s]: test_record_test (tests.test_runners.test_robot.RobotTest.test_record_test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/.local/share/virtualenvs/cli-uK8bV6hy/lib/python3.11/site-packages/responses/__init__.py", line 232, in wrapper

Changes in Click 8.2+

According to the Click GitHub Issues and Pull Request:

1. Before Click 8.2:

   • CliRunner had a mix_stderr parameter
   • mix_stderr=True (default): stdout and stderr mixed in result.stdout and result.output
   • mix_stderr=False: stdout in result.stdout, stderr in result.stderr

2. After Click 8.2:

   • The mix_stderr parameter has been completely removed
   • Results always provide separate stdout and stderr access:
     • result.stdout: Contains only stdout
     • result.stderr: Contains only stderr
     • result.output: Combined output (similar to what a user would see in a terminal)

Impact on Our Tests

Several test cases fail because they expect error messages to be in result.stdout when mix_stderr=False is specified, but with Click 8.2+ these messages are now in result.stderr.

Affected test files include:

• tests/commands/test_subset.py
• tests/test_runners/test_raw.py
• And potentially others that rely on the mix_stderr behavior

Dependency Configuration

Our current dependency configuration in setup.cfg specifies:

install_requires =
    click>=8.0,<8.1;python_version=='3.6'
    click>=8.1,<8.2;python_version>'3.6'

This means we're trying to use Click < 8.2, but when running with Python 3.10 in CI environments, Click 8.2+ is being installed, causing these compatibility issues.

Potential Solutions

1. Modify test cases to check both stdout and stderr for expected messages
2. Create a compatibility wrapper for CliRunner that emulates the old behavior
3. Conditionally set parameters based on the Click version
4. Update dependency constraints to explicitly limit Click version to < 8.2 for all Python versions

Next Steps

We need to decide on the best approach to maintain compatibility with both older Click versions and Click 8.2+, while minimizing code changes and maintaining test clarity.

Related Information

• Click Repository Issue: #2522
• Click Pull Request: #2523
• Python Version: 3.10
• Click Version: 8.2.1