Skip to content

fix(testing): Fix LayoutXLM tokenization test and LightOnOCR SDPA flash test failures on main CI#43988

Open
harshaljanjani wants to merge 1 commit intohuggingface:mainfrom
harshaljanjani:fix/layoutxlm-lighton-ocr-test-failures
Open

fix(testing): Fix LayoutXLM tokenization test and LightOnOCR SDPA flash test failures on main CI#43988
harshaljanjani wants to merge 1 commit intohuggingface:mainfrom
harshaljanjani:fix/layoutxlm-lighton-ocr-test-failures

Conversation

@harshaljanjani
Copy link
Contributor

What does this PR do?

The following failing tests were identified and fixed in this PR:

LayoutXLM: This PR (rm slow tokenizers) changed models/auto/tokenization_auto.py to map configs to single tokenizer classes instead of (slow, fast) tuples. It did touch tests/test_tokenization_common.py; having said that, it didn't update the merge_model_tokenizer_mappings fn so the [0] and [1] subscripts fail now.
LightOnOCR: This commit (Add LightOnOCR model implementation) added the LightOnOCR model which internally uses Pixtral's vision encoder. The skip list for test_sdpa_can_dispatch_on_flash already included "pixtral" (added on Jun 27, 2025), but LightOnOCR was not added to the same list despite having identical behavior (internally generates a block attention mask incompatible with Flash Attention), this should fix that :)

cc: @Rocketknight1

After: Both tests correctly skip gracefully (ref screenshots). merge_model_tokenizer_mappings should handle V5's single-class tokenizer mapping, and lighton_ocr should be skipped in test_sdpa_can_dispatch_on_flash alongside pixtral!

CI Failures:

Screenshot 2026-02-13 214801

Before the fix (feel free to cross-check; these errors are reproducible):

before

After the fix (feel free to cross-check):

after

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you fix any necessary existing tests?

@harshaljanjani harshaljanjani marked this pull request as ready for review February 13, 2026 20:13
@github-actions github-actions bot requested a review from ydshieh February 13, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant