bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Dov Grobgeld]

Do you think that you can provide some lisp code that builds the
buffer contents and the display properties for the three options that
ou refered to as an illustration?

I tried inserting tabs into the buffer before the vertical bars, but
after reordering it still didn't come out right.

Regards,
Dov

On Thu, Jun 14, 2012 at 6:10 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 13 Jun 2012 22:26:35 +0300
>> From: Dov Grobgeld <dov.grobgeld@gmail.com>
>>
>> Imagine you have a buffer with the following logical contents (using the
>> convention that capitals are RTL characters).
>>
>> | abcdef | abc |
>> | ABCDEF | ABC |
>>
>> I would like this to be displayed as:
>>
>> | abcdef | abc |
>> | FEDCBA | CBA |
>>
>> The problem is that I want to each column of the table to be isolated
>> (with regards to bidi influence) from other columns in the table. (Of
>> course we also want to choose the table direction, but that is a
>> different and solvable issue.) If there is no such separation, which
>> is the behaviour currently get in emacs HEAD, then the resulting
>> rendered buffer is:
>>
>> | abcdef | abc |
>> | CBA | FEDCBA |
>>
>> Is this even solvable in the current emacs bidi model?
>
> Yes, it is.  The solution involves putting segment separators between
> the table columns.  These could be TAB characters or a display
> property whose value is (space . :width N) or (space . :align-to COL).
>
> Org maintainers, please ask if you need help in fixing this.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> Date: Thu, 14 Jun 2012 21:10:46 +0300
> From: Dov Grobgeld <dov.grobgeld@gmail.com>
> Cc: 11700@debbugs.gnu.org
> 
> Do you think that you can provide some lisp code that builds the
> buffer contents and the display properties for the three options that
> ou refered to as an illustration?
> 
> I tried inserting tabs into the buffer before the vertical bars, but
> after reordering it still didn't come out right.

??? What exactly did you try?  You need to have a TAB before and after
each '|' that's between the cells (the outer ones do not need a TAB).
Like this (you should see this correctly in Emacs 24.1; move cursor
with C-f to see the logical order):

‎|      אבגד 	|	הוזחטי     |

Isn't this what you wanted?

As for the display properties, replace the TABs in the example above
with something like this:

  (insert (propertize " " '(display (space :align-to 10))))

The number 10 should be the column where you want the "|" separator to
be aligned.  For the replacement of the second TAB, increment the
value to get the desired alignment of the text of the second cell.

bug#11700: bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Dov Grobgeld]

On Thu, Jun 14, 2012 at 10:37 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Thu, 14 Jun 2012 21:10:46 +0300
>> From: Dov Grobgeld <dov.grobgeld@gmail.com>
>> Cc: 11700@debbugs.gnu.org
>> [stuff deleted]
>> I tried inserting tabs into the buffer before the vertical bars, but
>> after reordering it still didn't come out right.
>
> ??? What exactly did you try?  You need to have a TAB before and after
> each '|' that's between the cells (the outer ones do not need a TAB).
> Like this (you should see this correctly in Emacs 24.1; move cursor
> with C-f to see the logical order):
>
> ‎|      אבגד    |       הוזחטי     |
>
> Isn't this what you wanted?

Yes. Great! This is indeed what I wanted. My mistake was that I tried
it with a tab character before OR after the vertical bar. This
solution should be really simple to implement in org-mode as it means
that instead of joining the table columns with "<space><vbar><space>"
as is currently done, you just need to use "<tab><vbar><tab>" as well
as setting the tab width to 1.

But I just wonder, is there any other character (preferably white
space character) with the same end-of-segment-boundary properties as
tab, in case tab is used for something else in org-mode? Or is it
possible to take an arbitrary character, e.g. U+E0020, and bless it
with end-of-segment boundary properties and then use that in the
org-mode buffer?

As a side note, I really like the idea of end of segment separator,
and I wasn't aware of it when I wrote fribidi a long time ago. I
wonder if the semantics of the emacs segment separator follows the
Unicode Bidi Algorithm?

bug#11700: bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> Date: Fri, 15 Jun 2012 09:39:35 +0300
> From: Dov Grobgeld <dov.grobgeld@gmail.com>
> Cc: 11700@debbugs.gnu.org
> 
> Yes. Great! This is indeed what I wanted. My mistake was that I tried
> it with a tab character before OR after the vertical bar. This
> solution should be really simple to implement in org-mode as it means
> that instead of joining the table columns with "<space><vbar><space>"
> as is currently done, you just need to use "<tab><vbar><tab>" as well
> as setting the tab width to 1.

Yep.

> But I just wonder, is there any other character (preferably white
> space character) with the same end-of-segment-boundary properties as
> tab, in case tab is used for something else in org-mode?

That's the (space . :align-to COLUMN) display property I was talking
about.  With it, you can arrange for a single blank, say, to produce a
stretch of whitespace of arbitrary size, in character cell units,
aligned to a specified column.  If you use :width instead of
:align-to, you can have the size tuned in pixels.  Emacs treats text
covered by such display properties as segment separators, so they
produce the same effect as TAB does.  That's because conceptually,
such display properties are used to separate text into columnar
display, exactly like TAB is used in plain-text tables.

You can find examples of using these display properties in
minibuffer.el, where they are used to produce the display in the
*Completions* buffer.  Their documentation is in the ELisp manual.

> Or is it possible to take an arbitrary character, e.g. U+E0020, and
> bless it with end-of-segment boundary properties and then use that
> in the org-mode buffer?

Try using u+2029, it should do the trick, I think.

> As a side note, I really like the idea of end of segment separator,
> and I wasn't aware of it when I wrote fribidi a long time ago. I
> wonder if the semantics of the emacs segment separator follows the
> Unicode Bidi Algorithm?

Of course, it does; Emacs implements the UBA to the letter, taking
only the "high-level protocols" fire-escape to guide the reordering using
Emacs-specific context.  (But even the high-level protocols feature is
part of the UBA, see section 4.3 there.)

The Segment Separator feature is not an Emacs invention, it exists in
the UBA.  The key to understanding it is this part of the UBA:

  L1. On each line, reset the embedding level of the following
      characters to the paragraph embedding level:

  1. Segment separators,
  2. Paragraph separators,
  3. Any sequence of whitespace characters preceding a segment
     separator or paragraph separator, and 
  4. Any sequence of white space characters at the end of the line.

And the TAB character has "S", i.e. "segment separator", as its bidi
property.  Since its embedding level is reset to the base embedding
level of the paragraph, the result is that text on both sides of a TAB
is reordered separately and independently.

Moreover, the high-level protocols feature give us one more
possibility:

  HL4. Apply the Bidirectional Algorithm to segments.

    . The Bidirectional Algorithm can be applied independently to one
      or more segments of structured text. For example, when
      displaying a document consisting of textual data and visible
      markup in an editor, a higher-level process can handle syntactic
      elements in the markup separately from the textual data.

Emacs uses this to treat the `space' display properties as segment
separators.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Nicolas Goaziou]

Hello,

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 13 Jun 2012 22:26:35 +0300
>> From: Dov Grobgeld <dov.grobgeld@gmail.com>
>> 
>> Imagine you have a buffer with the following logical contents (using the
>> convention that capitals are RTL characters).
>> 
>> | abcdef | abc |
>> | ABCDEF | ABC |
>> 
>> I would like this to be displayed as:
>> 
>> | abcdef | abc |
>> | FEDCBA | CBA |
>> 
>> The problem is that I want to each column of the table to be isolated
>> (with regards to bidi influence) from other columns in the table. (Of
>> course we also want to choose the table direction, but that is a
>> different and solvable issue.) If there is no such separation, which
>> is the behaviour currently get in emacs HEAD, then the resulting
>> rendered buffer is:
>> 
>> | abcdef | abc |
>> | CBA | FEDCBA |
>> 
>> Is this even solvable in the current emacs bidi model?
>
> Yes, it is.  The solution involves putting segment separators between
> the table columns.  These could be TAB characters or a display
> property whose value is (space . :width N) or (space . :align-to COL).
>
> Org maintainers, please ask if you need help in fixing this.

*raises a hand*

I'd rather preserve structure of Org documents outside of Emacs. So,
`:align-to' is not an option. 

IIUC, I need to replace the closest space from vertical bars with 

  #(" " 0 1 (space :width 1))

This doesn't sound too difficult.

However, could someone provide tests cases so we get it right once and
for all?

Thank you.

Regards,

-- 
Nicolas Goaziou                                                0x80A93738

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Dov Grobgeld]

[-- Attachment #1: Type: text/plain, Size: 2059 bytes --]

The correct Unicode≥6.3 way to do this would be with the unicode isolation
characters. I.e. you would wrap each of the columns with <FSI>column
contents<BDI>. Does emacs honor these? Should be easy to test.

Regards,
Dov

On Mon, Dec 4, 2017 at 9:27 PM, Nicolas Goaziou <mail@nicolasgoaziou.fr>
wrote:

> Hello,
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> >> Date: Wed, 13 Jun 2012 22:26:35 +0300
> >> From: Dov Grobgeld <dov.grobgeld@gmail.com>
> >>
> >> Imagine you have a buffer with the following logical contents (using the
> >> convention that capitals are RTL characters).
> >>
> >> | abcdef | abc |
> >> | ABCDEF | ABC |
> >>
> >> I would like this to be displayed as:
> >>
> >> | abcdef | abc |
> >> | FEDCBA | CBA |
> >>
> >> The problem is that I want to each column of the table to be isolated
> >> (with regards to bidi influence) from other columns in the table. (Of
> >> course we also want to choose the table direction, but that is a
> >> different and solvable issue.) If there is no such separation, which
> >> is the behaviour currently get in emacs HEAD, then the resulting
> >> rendered buffer is:
> >>
> >> | abcdef | abc |
> >> | CBA | FEDCBA |
> >>
> >> Is this even solvable in the current emacs bidi model?
> >
> > Yes, it is.  The solution involves putting segment separators between
> > the table columns.  These could be TAB characters or a display
> > property whose value is (space . :width N) or (space . :align-to COL).
> >
> > Org maintainers, please ask if you need help in fixing this.
>
> *raises a hand*
>
> I'd rather preserve structure of Org documents outside of Emacs. So,
> `:align-to' is not an option.
>
> IIUC, I need to replace the closest space from vertical bars with
>
>   #(" " 0 1 (space :width 1))
>
> This doesn't sound too difficult.
>
> However, could someone provide tests cases so we get it right once and
> for all?
>
> Thank you.
>
> Regards,
>
> --
> Nicolas Goaziou                                                0x80A93738
>

[-- Attachment #2: Type: text/html, Size: 3200 bytes --]

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> From: Dov Grobgeld <dov.grobgeld@gmail.com>
> Date: Mon, 4 Dec 2017 21:35:40 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, 11700@debbugs.gnu.org
> 
> The correct Unicode≥6.3 way to do this would be with the unicode isolation characters. I.e. you would wrap
> each of the columns with <FSI>column contents<BDI>. Does emacs honor these?

Yes, Emacs implements Unicode 9.0, including the UBA with isolates.

However, I suspect that the results of wrapping with isolates will be
different from using the original proposal of using segment separators.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> Date: Mon, 04 Dec 2017 22:43:12 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: mail@nicolasgoaziou.fr, 11700@debbugs.gnu.org
> 
> Yes, Emacs implements Unicode 9.0, including the UBA with isolates.

Actually, the current development sources and the upcoming Emacs 26.1
already support Unicode 10.0.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
> Cc: Dov Grobgeld <dov.grobgeld@gmail.com>,  11700@debbugs.gnu.org
> Date: Mon, 04 Dec 2017 21:27:53 +0100
> 
> I'd rather preserve structure of Org documents outside of Emacs. So,
> `:align-to' is not an option. 
> 
> IIUC, I need to replace the closest space from vertical bars with 
> 
>   #(" " 0 1 (space :width 1))
> 
> This doesn't sound too difficult.
> 
> However, could someone provide tests cases so we get it right once and
> for all?

Such tests can only be run interactively, because bidi reordering is a
display-time feature in Emacs.  Is that OK with you?

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Nicolas Goaziou]

Eli Zaretskii <eliz@gnu.org> writes:

> Such tests can only be run interactively, because bidi reordering is a
> display-time feature in Emacs.  Is that OK with you?

That's better than no test at all in my book, so I'm fine with it, yes.

I can use isolation characters instead (if anyone cares to point me to
what those are), if you think that's better.

Regards,

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
> Cc: dov.grobgeld@gmail.com,  11700@debbugs.gnu.org
> Date: Mon, 04 Dec 2017 22:02:00 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Such tests can only be run interactively, because bidi reordering is a
> > display-time feature in Emacs.  Is that OK with you?
> 
> That's better than no test at all in my book, so I'm fine with it, yes.

OK.

Can one of you please provide a short Lisp snippet that generates a
2x2 Org table and inserts it in a buffer, which I could use as the
basis for the test?  That would get me off the ground quicker, since
I'm a very infrequent user of Org tables.

> I can use isolation characters instead (if anyone cares to point me to
> what those are), if you think that's better.

I will experiment with both and see which one is better.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Nicolas Goaziou]

Hello,

Eli Zaretskii <eliz@gnu.org> writes:

> Can one of you please provide a short Lisp snippet that generates a
> 2x2 Org table and inserts it in a buffer, which I could use as the
> basis for the test?  That would get me off the ground quicker, since
> I'm a very infrequent user of Org tables.

For tests, we use `org-test-with-temp-text' macro, e.g.,

  (org-test-with-temp-text "| a | b |\n| c | d |"
    ... do something in that buffer ...)

> I will experiment with both and see which one is better.

OK. Thank you.

Regards,

-- 
Nicolas Goaziou

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
> Date: Fri, 08 Dec 2017 18:08:57 +0100
> Cc: dov.grobgeld@gmail.com, 11700@debbugs.gnu.org
> 
> For tests, we use `org-test-with-temp-text' macro, e.g.,
> 
>   (org-test-with-temp-text "| a | b |\n| c | d |"
>     ... do something in that buffer ...)

You didn't say that this macro is only available in the Org's Git
repository...

Anyway, since it's a very simple macro, I just used its guts below.

I found both methods doing well, so I'm going to show both, and let
you decide which one is better.  The below provides a demonstration of
each method by displaying a buffer with a table whose columns include
both L2R and R2L text, such that the table columns are still laid out
left to right, unlike when one just types the characters in the cells.

Method 1: wrap each string with (invisible) bidi formatting control
characters which isolate each string from the surrounding text.

(defun bidi-isolate-string (str)
  (concat (propertize (string ?\x2068) 'invisible t)
	  str
	  (propertize (string ?\x2069) 'invisible t)))

(with-current-buffer (get-buffer-create "bidi-org-table1")
  (org-mode)
  (insert (concat "| "
		(bidi-isolate-string "abcd")
		" | "
		(bidi-isolate-string "efgh")
		" |\n| "
		(bidi-isolate-string "אבגד")
		" | "
		(bidi-isolate-string "הוזח")
		" |"))
  (pop-to-buffer "bidi-org-table"))

This has a minor issue: it fails to conceal the bidi control
characters on display, although I used the 'invisible' property for
that purpose.  I'm guessing that Org takes control of the 'invisible'
properties, in which case perhaps this method should use some other
property, if possible.  If it is not practical to conceal the bidi
controls on display, the following method is preferable.

Method 2: give the spaces around the cell text the display property
which makes the spaces serve as segment separators for the purposes of
the bidirectional reordering.

(defun bidi-separator-space ()
  (propertize " " 'display '(space :width 1)))

(with-current-buffer (get-buffer-create "bidi-org-table2")
  (org-mode)
  (insert (concat "|"
		  (bidi-separator-space)
		  "abcd"
		  (bidi-separator-space)
		  "|"
		  (bidi-separator-space)
		  "efgh"
		  (bidi-separator-space)
		  "|\n|"
		  (bidi-separator-space)
		  "אבגד"
		  (bidi-separator-space)
		  "|"
		  (bidi-separator-space)
		  "הוזח"
		  (bidi-separator-space)
		  "|"))
  (pop-to-buffer "bidi-org-table2"))

Let me know if I can help you further, or if you have additional
questions.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> Date: Sat, 23 Dec 2017 15:38:11 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: dov.grobgeld@gmail.com, 11700@debbugs.gnu.org
> 
> I found both methods doing well, so I'm going to show both, and let
> you decide which one is better.

On second thought, I think Method 2 is better, because it does exactly
what segment separators were invented for: to separate cells in
tables.  By contrast, the bidi formatting control characters are for
controlling the display order in general text, not necessarily in
tables.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Nicolas Goaziou]

Hello,

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
>> Date: Fri, 08 Dec 2017 18:08:57 +0100
>> Cc: dov.grobgeld@gmail.com, 11700@debbugs.gnu.org
>> 
>> For tests, we use `org-test-with-temp-text' macro, e.g.,
>> 
>>   (org-test-with-temp-text "| a | b |\n| c | d |"
>>     ... do something in that buffer ...)
>
> You didn't say that this macro is only available in the Org's Git
> repository...

I believed our test suite had been merged with Emacs. Sorry about that.

> Method 2: give the spaces around the cell text the display property
> which makes the spaces serve as segment separators for the purposes of
> the bidirectional reordering.

Sounds good. I implemented it in Org. Hopefully, Org is now friendlier
to BiDi.

Please let me know if something is missing.

Thank you.

Regards,

-- 
Nicolas Goaziou                                                0x80A93738