Latin-1.3 chars not taken into consideration while claculation the right border position

[Marek-Cichowski]

Hi,

we love your "tabulate" library which we use for more structured log displays :-)

• At this place I'd like to report a small flaw.
• If using Latin-1.3 chars (European special chars for names) the boundary calculation is not quite working see below

• Issue is resulting from the fact the Latin-1.3 chars can be 1, 2 or even 3 bytes long, which is causing the right margin being not aligned.
• For sure it is just a "cosmetic" issue, but you have introduced 'tabulate' for its "logging beauty", so I do hope for your consideration.

Best regard

Marek

[DimitriPapadopoulos]

What is "Latin-1.3" and "European special chars for names"?

How to reproduce the above output?

>>> from tabulate import tabulate
>>> 
>>> s = "ÉåMt̕Z̈ȱȯm̐p̣wRǞT̕ǹ"
>>> 
>>> data = [
...     ["Attribute(s)",   "Value(s)"],
...     ["Length [bytes]", len(s.encode('utf-8'))],
...     ["Length [chars]", len(s)],
...     ["String",         s],
... ]
>>> 
>>> print(tabulate(data,headers='firstrow',tablefmt="simple_grid"))
┌────────────────┬────────────────┐
│ Attribute(s)   │ Value(s)       │
├────────────────┼────────────────┤
│ Length [bytes] │ 30             │
├────────────────┼────────────────┤
│ Length [chars] │ 19             │
├────────────────┼────────────────┤
│ String         │ ÉåMt̕Z̈ȱȯm̐p̣wRǞT̕ǹ │
└────────────────┴────────────────┘
>>> 

Note that the length of ÉåMt̕Z̈ȱȯm̐p̣wRǞT̕ǹ in chars is 19, not 14. The difference is 5, which could well be the size of the misalignment. Actual code would help understand the underlying issue here.

[Marek-Cichowski]

@DimitriPapadopoulos,

thank you for your fast reply.
The length in chars actually refers to "visual characters" so called grapheme clusters and those are in the fact 14 (we do count it during the generation process of this random string s).
19 mentioned by you are the Unicode code points and those are 19.
Regardless what we will display for length in chars - grapheme clusters or code points - the "trouble" is during display of the row with String s.

we do not have Attribute/Values as separate 1. row but using it as fix headers, besides we use simple_outline as being more "beautiful" - see below:

print(tabulate(data,headers=["Attribute(s)", "Value(s)"],tablefmt="simple_outline", ))

So in summary, what are we doing wrong ? Why do we have a shift of right border for s == "tÉåMt̕Z̈ȱȯm̐p̣wRǞT̕ǹ" ?

My assumption is that tabulate calculates the length of string "s" in code points which is 19 but the visual chars displayed are in the fact 14 so the border for the string line is misaligned by 5. Can we mitigate it somehow giving the grapheme clusters = 14 to tabulate?

One option could be that grapheme_length = grapheme.length(s) is used by tabulate to calculate the max length for the table and not code points.

I am providing some displays from our adapted code:

• random string (showing length in [graphemes])

• random phone number

[DimitriPapadopoulos]

Instead of copying/pasting images, please post an actual piece of code that reproduces the problem you're experiencing, because I cannot reproduce it.

>>> from tabulate import tabulate
>>> 
>>> s = "tÉåMt̕Z̈ȱȯm̐p̣wRǞT̕ǹ"
>>> 
>>> data = [
... ["Attribute(s)", "Value(s)"],
... ["String",       s],
... ]
>>> 
>>> print(tabulate(data,headers='firstrow',tablefmt="simple_outline"))
┌────────────────┬─────────────────┐
│ Attribute(s)   │ Value(s)        │
├────────────────┼─────────────────┤
│ String         │ tÉåMt̕Z̈ȱȯm̐p̣wRǞT̕ǹ │
└────────────────┴─────────────────┘
>>> 

I suspect you're using an old version of python-tabulate.

[Marek-Cichowski]

`"""
Created on Tue Sep 16 08:03:44 2025

@author: marekcichowski --> example from Dmitri
"""

from tabulate import tabulate

s = "tÉåMt̕Z̈ȱȯm̐p̣wRǞT̕ǹ"
data = [["String", s]]
print(tabulate(data,headers=["Attribute(s)", "Value(s)"],tablefmt="simple_outline"))`

With the above example I get exactly the results as described before in Spyder (Python 3.10.14 ) AND in Pycharm under Robot framework (Python 3.9) - on both systems tabulate version is Version: 0.9.0

But if tabulate calculates the length of line for string based on code points and not grapheme images it can't work.

• Are you please using grapheme.length(s) to calculate the length of the string?
• Any hidden parameter that must be used/activated?

[DimitriPapadopoulos]

I cannot reproduce this, even with python-tabulate 0.9.0. Again, see my example:

$ pip install tabulate --user --upgrade
Requirement already satisfied: tabulate in /usr/lib/python3/dist-packages (0.8.10)
Collecting tabulate
  Using cached tabulate-0.9.0-py3-none-any.whl.metadata (34 kB)
Using cached tabulate-0.9.0-py3-none-any.whl (35 kB)
Installing collected packages: tabulate
Successfully installed tabulate-0.9.0
$ 

>>> from tabulate import tabulate
>>> 
>>> s = "tÉåMt̕Z̈ȱȯm̐p̣wRǞT̕ǹ"
>>> 
>>> data = [
... ["Attribute(s)", "Value(s)"],
... ["String",       s],
... ]
>>> 
>>> print(tabulate(data,headers='firstrow',tablefmt="simple_outline"))
┌────────────────┬─────────────────┐
│ Attribute(s)   │ Value(s)        │
├────────────────┼─────────────────┤
│ String         │ tÉåMt̕Z̈ȱȯm̐p̣wRǞT̕ǹ │
└────────────────┴─────────────────┘
>>> 

Please get rid of Spyder, PyCharm and the Robot framework, try directly in the Python shell instead. Does it work any better?

Also I'm on Linux. What's your OS?

[Marek-Cichowski]

We have the same behaviour in our Linux VM and Windows VM, where we work directly on Python. So definitely IDE is not the issue. You have not answered my question: Are you please using grapheme.length(s) to calculate the length of the string?

BTW in the python shell - I DID TRY - the string is misinterpreted - see below

This string: "ċỌL̥Ễg̀ŹỠẮǏữṣ̄fȨr̥H̄" is displayed as below:

which are 20 visual characters - but in the reality ċỌL̥Ễg̀ŹỠẮǏữṣ̄fȨr̥H̄ are 15 grapheme visuals in Latin-1.3 - string is displayed correctly in our system. Python shell is not able to display Latin-1.3 correctly.

Are we coming closer to mutual understanding ?

[DimitriPapadopoulos]

Please post a Python script that reproduces the problem.

[DimitriPapadopoulos]

Something's obviously wrong in the screen capture you've sent, and it doesn't seem related to python-tabulate:

Instead of:

tÉåMt̕Z̈ȱȯm̐p̣wRǞT̕ǹ

I see:

tÉåM☒ȯ☒RǞ☒

where ☒ stands for overwritten characters in the output, which account for the the 5 character difference.

[DimitriPapadopoulos]

I guess I'm using single Unicode codepoints in my example, not combining diacritical marks:

>>> s = "tÉåMt̕Z̈ȱȯm̐p̣wRǞT̕ǹ"

That's why it is important you actually spend time preparing and posting an example that reproduces the problem.

[DimitriPapadopoulos]

Python shell is not able to display Latin-1.3 correctly.

So it's a Python issue, ins't it? Still not sure what Latin-1.3 is.

[DimitriPapadopoulos]

This is my current script:
test.py

from tabulate import tabulate
from unicodedata import normalize
import grapheme

s = "ċỌL̥Ễg̀ŹỠẮǏữṣ̄fȨr̥H̄"

data = [
    ["Attribute(s)", "Value(s)"],
    ["Length", len(s)],
    ["Length [graphems]", grapheme.length(s)],
    ["String", s],
]
print(tabulate(data,headers='firstrow',tablefmt="simple_outline"))

The output is:

┌───────────────────┬─────────────────┐
│ Attribute(s)      │ Value(s)        │
├───────────────────┼─────────────────┤
│ Length            │ 20              │
│ Length [graphems] │ 15              │
│ String            │ ċỌL̥Ễg̀ŹỠẮǏữṣ̄fȨr̥H̄ │
└───────────────────┴─────────────────┘

See screen capture with native Python 3.12.3 on Ubuntu 24.04.2:

Can you modify it to reproduce the issue?

[Marek-Cichowski]

from tabulate import tabulate
from unicodedata import normalize
import grapheme

s = "ċỌL̥Ễg̀ŹỠẮǏữṣ̄fȨr̥H̄ "

data = [
["Attribute(s)", "Value(s)"],
["Length", len(s)],
["Length [graphems]", grapheme.length(s)],
["String", s],
]
print(tabulate(data,headers='firstrow',tablefmt="simple_outline"))

delivers in my system 3.10.14

[Marek-Cichowski]

Python shell is not able to display Latin-1.3 correctly.

So it's a Python issue, inst'it? Still not sure what Latin-1.3 is.

Characters incluing diacritical marks.

[DimitriPapadopoulos]

So far I have tried with:

• Python 3.12.3 / Ubuntu 24.04.2
• Python 3.8.10 / Ubuntu 20.04.6

It works for me in both cases. The locale on these machines is fr_FR.UTF-8 for what it's worth.

It would be best to upload the example Python script as a file (drop the file in the text area), like this:

test.py

Contents of the above file

#!/usr/bin/python3

from tabulate import tabulate
import grapheme

s = "ċỌL̥Ễg̀ŹỠẮǏữṣ̄fȨr̥H̄"

data = [
    ["Attribute(s)", "Value(s)"],
    ["Length [bytes]", len(s)],
    ["Length [graphemes]", grapheme.length(s)],
    ["String", s],
]
print(tabulate(data,headers='firstrow',tablefmt="simple_outline"))

I fear that combining diacritical marks might be lost when copying/pasting through GitHub's web interface.