double-width characters in tables

double-width characters in tables

[Eric Abrahamsen]

This problem has been flagged up before:

http://orgmode.org/worg/org-issues.html#mid-87pqt04qg1-2Efsf-40gmail-2Ecom

It's causing me headaches at the moment, so I'm trying to see if I can
find a solution.

The proximate cause of issues with double-width characters in table
fields (they also mess with justification) is that the contents of the
fields are propertized with a length based on `length', and not
`string-width'. So you get this:

#("《蛙》" 0 3 (fontified t line-prefix #("    " 0 4 (face org-indent))
 wrap-prefix #("    " 0 4 (face org-indent)) face org-table))

3 is the number of characters in the string, but it takes up 6 columns
of screen width. That messes up justification.

I can't for the life of me figure out where that number is getting added
to the string! I tried changing `length' to `string-width' in all manner
of org functions, such as `org-add-props', but could never get that
number changed during the fontification of the org tables.

I wouldn't be surprised if this is something outside of org, or if
"fixing" it would lead to more troubles, but does anyone have any ideas
about this? Fixable, or not?

Thanks!
Eric

Re: double-width characters in tables

[Jambunathan K]

File a bug report - M-x report-emacs-bug RET - against Orgmode.  I am
sure something useful will come out of it, eventually.  It is good to
get eyes and mouths of emacs-devel-ers involved, for whatever it is
worth.  Believe me it is good, for the discussion and suggestions it
will open up.

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> This problem has been flagged up before:
>
> http://orgmode.org/worg/org-issues.html#mid-87pqt04qg1-2Efsf-40gmail-2Ecom
>
> It's causing me headaches at the moment, so I'm trying to see if I can
> find a solution.
>
> The proximate cause of issues with double-width characters in table
> fields (they also mess with justification) is that the contents of the
> fields are propertized with a length based on `length', and not
> `string-width'. So you get this:
>
> #("《蛙》" 0 3 (fontified t line-prefix #("    " 0 4 (face org-indent))
>  wrap-prefix #("    " 0 4 (face org-indent)) face org-table))
>
> 3 is the number of characters in the string, but it takes up 6 columns
> of screen width. That messes up justification.
>
> I can't for the life of me figure out where that number is getting added
> to the string! I tried changing `length' to `string-width' in all manner
> of org functions, such as `org-add-props', but could never get that
> number changed during the fontification of the org tables.
>
> I wouldn't be surprised if this is something outside of org, or if
> "fixing" it would lead to more troubles, but does anyone have any ideas
> about this? Fixable, or not?
>
> Thanks!
> Eric
>
>
>

-- 

Re: double-width characters in tables

[Bastien]

Hi Eric,

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> This problem has been flagged up before:
>
> http://orgmode.org/worg/org-issues.html#mid-87pqt04qg1-2Efsf-40gmail-2Ecom
>
> It's causing me headaches at the moment, so I'm trying to see if I can
> find a solution.

The problem for me here is that even if (string-width "《蛙》")
returns 6, it is not visually equivalent to 6 characters.  

Is it for you?  

If not, let's report this to Emacs...

PS: Btw (org-string-width "《蛙》") returns the correct value.

-- 
 Bastien

Re: double-width characters in tables

[Bastien]

Jambunathan K <kjambunathan@gmail.com> writes:

> File a bug report - M-x report-emacs-bug RET - against Orgmode.

If users believe this is a bug in Org-mode, they should send
the bug report to the Orgmode list.

If they believe this is a bug in Emacs, they should discuss it
to emacs-devel.

Please help me in letting people know about this policy.

-- 
 Bastien

Re: double-width characters in tables

[Eric Abrahamsen]

Bastien <bzg@altern.org> writes:

> Hi Eric,
>
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> This problem has been flagged up before:
>>
>> http://orgmode.org/worg/org-issues.html#mid-87pqt04qg1-2Efsf-40gmail-2Ecom
>>
>> It's causing me headaches at the moment, so I'm trying to see if I can
>> find a solution.
>
> The problem for me here is that even if (string-width "《蛙》")
> returns 6, it is not visually equivalent to 6 characters.  
>
> Is it for you?  
>
> If not, let's report this to Emacs...
>
> PS: Btw (org-string-width "《蛙》") returns the correct value.

Yes, org-string-width eventually calls string-width, so that behaves
"correctly" as far as it goes, but unfortunately that's not where the
value in the text properties comes from...

《蛙》
123456

Doesn't that line up for you? Those bracket characters come with their
own "whitespace", maybe this is clearer:

正能量
123456

One Chinese character should definitely take up two screen columns.

I'd love to file a bug report but I honestly don't know where this comes
from. If it's not org-add-props or relatives, what emacs function is
responsible?

Thanks for the swift responses!

Eric

Re: double-width characters in tables

[Nick Dokos]

Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> 《蛙》
> 123456
> 
> Doesn't that line up for you? Those bracket characters come with their
> own "whitespace", maybe this is clearer:
> 
> 正能量
> 123456
> 

I use a fixed-width font in my emacs and those two do not line up for me
(the Chinese chars reach to about the middle of the "5").

> One Chinese character should definitely take up two screen columns.
> 

It does not seem to, in my case.

Nick

PS. In case it matters, I have this in my .emacs:

(set-default-font "-misc-fixed-medium-r-normal--15-140-75-75-c-90-iso8859-1")

Re: double-width characters in tables

[Bastien]

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

Hi Eric,

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> 《蛙》
> 123456
>
> Doesn't that line up for you? Those bracket characters come with their
> own "whitespace", maybe this is clearer:
>
> 正能量
> 123456

The chinese line and the 123456 one are not of the same
visual/geometrical width for me, even if they both count as 6
characters.

See the screenshots, where you can see it quickly.

> One Chinese character should definitely take up two screen columns.

Not for me... perhaps that's because I don't have a monospace font
for chinese characters?  Do you?

> I'd love to file a bug report but I honestly don't know where this comes
> from. If it's not org-add-props or relatives, what emacs function is
> responsible?

Unless you don't see the difference I observe here on my screen, 
this may be a bug in Emacs internal display engine.


[-- Attachment #2: emacs-string-width1.jpg --]
[-- Type: image/jpeg, Size: 70327 bytes --]

[-- Attachment #3: emacs-string-width2.jpg --]
[-- Type: image/jpeg, Size: 98622 bytes --]

[-- Attachment #4: Type: text/plain, Size: 14 bytes --]


-- 
 Bastien

Re: double-width characters in tables

[Takaaki ISHIKAWA]

[-- Attachment #1: Type: text/plain, Size: 1730 bytes --]

Dear Eric,  

I've checked the width of characters you mentioned.
Please find attached images.

In this case, the ascii font is Monaco:12pt.
And Japanese font is "Migu 2M":12pt with 1.2 scaling factor.
The "Migu 2M" font is monospace.

I've got unexpected results, the length of "蛙" or "《" is 2, but
"○" is 1. I don't know the reason. Is the "○" handled as a symbol?

Best regards,
Takaaki Ishikawa

--
Takaaki ISHIKAWA <takaxp@ieee.org>
  GITI, Waseda University
    ( ' -')b http://about.me/takaxp


On 2013年2月13日Wednesday at 0:55, Bastien wrote:

> Hi Eric,
>  
> Eric Abrahamsen <eric@ericabrahamsen.net (mailto:eric@ericabrahamsen.net)> writes:
>  
> > 《蛙》
> > 123456
> >  
> > Doesn't that line up for you? Those bracket characters come with their
> > own "whitespace", maybe this is clearer:
> >  
> > 正能量
> > 123456
>  
>  
>  
> The chinese line and the 123456 one are not of the same
> visual/geometrical width for me, even if they both count as 6
> characters.
>  
> See the screenshots, where you can see it quickly.
>  
> > One Chinese character should definitely take up two screen columns.
>  
> Not for me... perhaps that's because I don't have a monospace font
> for chinese characters? Do you?
>  
> > I'd love to file a bug report but I honestly don't know where this comes
> > from. If it's not org-add-props or relatives, what emacs function is
> > responsible?
>  
>  
>  
> Unless you don't see the difference I observe here on my screen,  
> this may be a bug in Emacs internal display engine.
>  
>  
> --  
> Bastien
>  
>  
> Attachments:  
> - emacs-string-width1.jpg
>  
> - emacs-string-width2.jpg
>  



[-- Attachment #2: ishot-1.png --]
[-- Type: image/png, Size: 83127 bytes --]

[-- Attachment #3: ishot-2.png --]
[-- Type: image/png, Size: 86974 bytes --]

Re: double-width characters in tables

[Achim Gratz]

Eric Abrahamsen writes:
> Yes, org-string-width eventually calls string-width, so that behaves
> "correctly" as far as it goes, but unfortunately that's not where the
> value in the text properties comes from...
>
> 《蛙》
> 123456
>
> Doesn't that line up for you? Those bracket characters come with their
> own "whitespace", maybe this is clearer:
>
> 正能量
> 123456
>
> One Chinese character should definitely take up two screen columns.

That's a function of the font that is ultimately used and their
run-length.  Since you are unlikely to use a font that has _all_ those
glyphs, eventually the ones that are missing in your specified font will
be replaced with the glyph corresponding to the same codepoint in a
different font if it exists or possibly a composition of multiple glyphs
(if not you'll get a placeholder).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: double-width characters in tables

[Eric Abrahamsen]

Achim Gratz <Stromeko@nexgo.de> writes:

> Eric Abrahamsen writes:
>> Yes, org-string-width eventually calls string-width, so that behaves
>> "correctly" as far as it goes, but unfortunately that's not where the
>> value in the text properties comes from...
>>
>> 《蛙》
>> 123456
>>
>> Doesn't that line up for you? Those bracket characters come with their
>> own "whitespace", maybe this is clearer:
>>
>> 正能量
>> 123456
>>
>> One Chinese character should definitely take up two screen columns.
>
> That's a function of the font that is ultimately used and their
> run-length.  Since you are unlikely to use a font that has _all_ those
> glyphs, eventually the ones that are missing in your specified font will
> be replaced with the glyph corresponding to the same codepoint in a
> different font if it exists or possibly a composition of multiple glyphs
> (if not you'll get a placeholder).

Thanks for all the responses here! Obviously the question is more
complicated than I thought. Chinese fonts are fixed-width by default, so
I assumed things would line up with a fixed-width latin font, though
clearly I wasn't thinking hard enough about typefaces and their
different widths/sizes. It's obvious from the screenshots that
everyone's got a little something different going on.

Takaaki's unicode WHITE CIRCLE is weird -- it's classified as CJK, but
Chinese characters are usually named CJK IDEOGRAPH BLAH, and this isn't.
It's displayed using a Chinese font on my machine, making it
double-width, but sure enough `string-width' returns 1. I actually have
the same problem with EM DASH here -- Inconsolata doesn't have that
glyph, so a Chinese font is substituted, and I get a double-width
character that reports its width as 1.

Anyhoo... I still think the original problem is valid: strings in table
fields should be measured with `string-width', not `length'. It might
not be perfect, but surely it would be better than the present
situation?

On second thought I don't think it's a problem with text properties.
(add-text-properties 0 6 '() "正能量") gives an Args out of range error,
and it probably should, since all it cares about is the number of
characters in the string.

Solving the problem then might mean just using `org-string-width'
directly in the code, in the places where string width is currently
calculated from text property values. I'll poke at it, and see how badly
I break it.

Thanks again,
Eric

[PATCH] Re: double-width characters in tables

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 1250 bytes --]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Achim Gratz <Stromeko@nexgo.de> writes:
>
>> Eric Abrahamsen writes:
>>> Yes, org-string-width eventually calls string-width, so that behaves
>>> "correctly" as far as it goes, but unfortunately that's not where the
>>> value in the text properties comes from...
>>>
>>> 《蛙》
>>> 123456
>>>
>>> Doesn't that line up for you? Those bracket characters come with their
>>> own "whitespace", maybe this is clearer:
>>>
>>> 正能量
>>> 123456
>>>
>>> One Chinese character should definitely take up two screen columns.

[...]

> On second thought I don't think it's a problem with text properties.
> (add-text-properties 0 6 '() "正能量") gives an Args out of range error,
> and it probably should, since all it cares about is the number of
> characters in the string.

Here's one provisional attempt to fix one instance of weirdness, inside
`org-table-justify-field-maybe'. This handles re-justification of table
fields when you hit TAB or S-TAB. It turns out this spot doesn't use
text properties, but match-end/beginning locations. This patch Works For
Me, though it's a little ugly and I have no idea if it may cause other
repercussions. Could someone just glance over it?

Thanks,
Eric


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Make-table-field-justification-respect-string-width.patch --]
[-- Type: text/x-patch, Size: 1281 bytes --]

From 46a126e57494479db6b64035dfc43698963cffb5 Mon Sep 17 00:00:00 2001
From: Eric Abrahamsen <eric@ericabrahamsen.net>
Date: Wed, 13 Feb 2013 10:11:37 +0800
Subject: [PATCH] Make table field justification respect string width

---
 lisp/org-table.el | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/lisp/org-table.el b/lisp/org-table.el
index 204787f..6bbe732 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -977,14 +977,16 @@ Optional argument NEW may specify text to replace the current field content."
 	    (progn
 	      (setq s (match-string 1)
 		    o (match-string 0)
-		    l (max 1 (- (match-end 0) (match-beginning 0) 3))
+		    l (max 1
+			   (- (org-string-width
+			       (buffer-substring
+				(match-end 0) (match-beginning 0))) 3))
 		    e (not (= (match-beginning 2) (match-end 2))))
 	      (setq f (format (if num " %%%ds %s" " %%-%ds %s")
 			      l (if e "|" (setq org-table-may-need-update t) ""))
 		    n (format f s))
 	      (if new
-		  (if (<= (length new) l)      ;; FIXME: length -> str-width?
+		  (if (<= (org-string-width new) l)
 		      (setq n (format f new))
 		    (setq n (concat new "|") org-table-may-need-update t)))
 	      (if (equal (string-to-char n) ?-) (setq n (concat " " n)))
-- 
1.8.1.3

[PATCH] Re: double-width characters in tables

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Achim Gratz <Stromeko@nexgo.de> writes:
>
>> Eric Abrahamsen writes:
>>> Yes, org-string-width eventually calls string-width, so that behaves
>>> "correctly" as far as it goes
[...]

And more, this time to prevent errors when using column-narrowing
cookies on columns containing peculiar-width strings. It's still going
to look weird, often as not, but this will at least prevent blowups.

I'm providing two possible patches. One is a one-liner that does the
trick but produces ugly results. The other goes a wee bit overboard, but
looks a little better. I offer either, or neither, for consideration.

Eric


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Make-table-column-narrowing-play-nice-with-variable-.patch --]
[-- Type: text/x-patch, Size: 1867 bytes --]

From ccf294f9dad0b81240b8f1a0051cecfddd47d1c4 Mon Sep 17 00:00:00 2001
From: Eric Abrahamsen <eric@ericabrahamsen.net>
Date: Wed, 13 Feb 2013 16:46:33 +0800
Subject: [PATCH] Make table column narrowing play nice with variable-width
 strings

---
 lisp/org-table.el | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/lisp/org-table.el b/lisp/org-table.el
index 3f276f8..d79b650 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -765,7 +765,7 @@ When nil, simply write \"#ERROR\" in corrupted fields.")
 	 (hfmt1 (concat
 		 (make-string sp2 ?-) "%s" (make-string sp1 ?-) "+"))
 	 emptystrings links dates emph raise narrow
-	 falign falign1 fmax f1 len c e space)
+	 falign falign1 fmax f1 f2 len c e space)
     (untabify beg end)
     (remove-text-properties beg end '(org-cwidth t org-dwidth t display t))
     ;; Check if we have links or dates
@@ -850,10 +850,19 @@ When nil, simply write \"#ERROR\" in corrupted fields.")
 		  (unless (> f1 1)
 		    (error "Cannot narrow field starting with wide link \"%s\""
 			   (match-string 0 xx)))
-		  (add-text-properties f1 (length xx) (list 'org-cwidth t) xx)
-		  (add-text-properties (- f1 2) f1
-				       (list 'display org-narrow-column-arrow)
-				       xx)))))
+		  (setq f2 (length xx))
+		  (if (= (org-string-width xx)
+			 f2)
+		      (setq f2 f1)
+		    (setq f2 1)
+		    (while (< (org-string-width (substring xx 0 f2))
+			      f1)
+		      (setq f2 (1+ f2))))
+		  (add-text-properties f2 (length xx) (list 'org-cwidth t) xx)
+		  (add-text-properties (if (>= (string-width (substring xx (1- f2) f2)) 2)
+					   (1- f2) (- f2 2)) f2
+					   (list 'display org-narrow-column-arrow)
+					   xx)))))
       ;; Get the maximum width for each column
       (push (apply 'max (or fmax 1) 1 (mapcar 'org-string-width column))
 	    lengths)
-- 
1.8.1.3


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0001-Prevent-errors-with-table-column-narrowing-and-varia.patch --]
[-- Type: text/x-patch, Size: 877 bytes --]

From 319931ba76fc3ce5b8157f131d7c71c18f4c5d60 Mon Sep 17 00:00:00 2001
From: Eric Abrahamsen <eric@ericabrahamsen.net>
Date: Wed, 13 Feb 2013 15:07:59 +0800
Subject: [PATCH] Prevent errors with table column narrowing and variable-width
 strings

---
 lisp/org-table.el | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lisp/org-table.el b/lisp/org-table.el
index 3f276f8..6fbff7a 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -850,6 +850,7 @@ When nil, simply write \"#ERROR\" in corrupted fields.")
 		  (unless (> f1 1)
 		    (error "Cannot narrow field starting with wide link \"%s\""
 			   (match-string 0 xx)))
+		  (setq f1 (round (* (/ (float (length xx)) (org-string-width xx)) f1)))
 		  (add-text-properties f1 (length xx) (list 'org-cwidth t) xx)
 		  (add-text-properties (- f1 2) f1
 				       (list 'display org-narrow-column-arrow)
-- 
1.8.1.3

Re: double-width characters in tables

[Bastien]

Hi Eric,

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Solving the problem then might mean just using `org-string-width'
> directly in the code, in the places where string width is currently
> calculated from text property values. I'll poke at it, and see how badly
> I break it.

Thanks for digging further -- if you can find a bug that doesn't
involve mixing languages (e.g. if users are inserting special but
common characters?), that'd even better!

-- 
 Bastien

Re: [PATCH] Re: double-width characters in tables

[Bastien]

Hi Eric,

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> -		    l (max 1 (- (match-end 0) (match-beginning 0) 3))
> +		    l (max 1
> +			   (- (org-string-width
> +			       (buffer-substring
> +				(match-end 0) (match-beginning 0))) 3))

(Why not just (org-string-width (match-string 0))?)

Let me know when you're done with extensive testing for this!

Thanks,

-- 
 Bastien

Re: [PATCH] Re: double-width characters in tables

[Eric Abrahamsen]

Bastien <bzg@altern.org> writes:

> Hi Eric,
>
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> -		    l (max 1 (- (match-end 0) (match-beginning 0) 3))
>> +		    l (max 1
>> +			   (- (org-string-width
>> +			       (buffer-substring
>> +				(match-end 0) (match-beginning 0))) 3))
>
> (Why not just (org-string-width (match-string 0))?)
>
> Let me know when you're done with extensive testing for this!

I think the reason I went with buffer-substring is that it will *fix*
messed up justification. (match-string 0) will not mess up the
justification if it's already correct, but it won't fix it if it isn't.

Actually, shortly after sending that patch I moved to using
buffer-substring-no-properties, which has the added benefit that fields
with links or other types of hidden text will justify correctly. Unless
I'm mistaken, putting links in table fields will currently result in
too-short justification as handled by org-table-justify-field-maybe (as
always, a full re-align with org-table-align fixes things). Unless the
buffer-substring call is particularly expensive, I think it's worth it.

Re testing: apart from mixing scripts and languages in one field, what
else should be tested? You mentioned cleaning up other problems while
we're at it -- does anyone have anything outstanding?

Inevitably (I think), none of this works when the wrong typeface is
being used for particular characters. I have no specifications for
Cyrillic, for example, which means my system renders it in with a
Chinese typeface, resulting in double-wide glyphs for characters that
should be single-width, and bad table justification. I think this has to
be addressed by the user.

E