Recent regression in Org master with parsing of verbatim code with spaces

Recent regression in Org master with parsing of verbatim code with spaces

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

Hello,

I just noticed a regression in master when compared to 9.1.2.

MWE:

=====
=title = "foo"​=

~title = "foo"​~
=====

On master:

C-c C-e t A outputs (ignoring, author, title, toc, etc):

=====
=title = "foo"​=

~title = "foo"​~
=====

Also the above text does NOT get fontified as verbatim in the Org buffer.

On 9.1.2 (that's on emacs-26 branch):

C-c C-e t A outputs (ignoring, author, title, toc, etc):

=====
`title = "foo"​'

`title = "foo"​'
=====

Also the above text DOES get fontified as verbatim in the Org buffer.

I scanned through recent commits, but haven't yet figured out which commit
affected this. Before I end up with git bisect, does anyone know a master
branch commit that could have caused this?
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 1488 bytes --]

Re: Recent regression in Org master with parsing of verbatim code with spaces

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]

On Tue, Oct 10, 2017 at 1:08 PM Kaushal Modi <kaushal.modi@gmail.com> wrote:

> I scanned through recent commits, but haven't yet figured out which commit
> affected this. Before I end up with git bisect, does anyone know a master
> branch commit that could have caused this?
>

Hello,

I found the commit causing this change:

http://orgmode.org/cgit.cgi/org-mode.git/commit/?id=8383f7e08b964e129b175ef1c0297b1f4a8ed235
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 998 bytes --]

Re: Recent regression in Org master with parsing of verbatim code with spaces

[Eric S Fraga]

[-- Attachment #1: Type: text/plain, Size: 319 bytes --]

Anecdotally, I am seeing some strange fontifications in some of my org
files where my headline *s are being interpreted as starting a bold
text.  I've not been able to create a small example yet but thought I
would mention it in case it helps.

-- 
: Eric S Fraga via Emacs 27.0.50, Org release_9.1.2-98-g0b8316

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: Recent regression in Org master with parsing of verbatim code with spaces

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

On Tue, Oct 10, 2017 at 2:31 PM Eric S Fraga <esflists@gmail.com> wrote:

> Anecdotally, I am seeing some strange fontifications in some of my org
> files where my headline *s are being interpreted as starting a bold
> text.
>

I have seen that too, though I thought that was because I was setting the
NEWLINE component in the org-regexp-emphasis component to a value greater
than the default of 1. It's on my list to get back to debugging that.
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 846 bytes --]

Re: Recent regression in Org master with parsing of verbatim code with spaces

[Reuben Thomas]

[-- Attachment #1: Type: text/plain, Size: 898 bytes --]

On 10 October 2017 at 18:29, Kaushal Modi <kaushal.modi@gmail.com> wrote:

> On Tue, Oct 10, 2017 at 1:08 PM Kaushal Modi <kaushal.modi@gmail.com>
> wrote:
>
>> I scanned through recent commits, but haven't yet figured out which
>> commit affected this. Before I end up with git bisect, does anyone know a
>> master branch commit that could have caused this?
>>
>
> Hello,
>
> I found the commit causing this change:
>
> http://orgmode.org/cgit.cgi/org-mode.git/commit/?id=
> 8383f7e08b964e129b175ef1c0297b1f4a8ed235
>

​I haven't seen the rest of this thread, but at a guess is the problem to
do with [:space:] matching vertical space such as newline and vertical tab?
Fixing that is a bit painful, since as far as I know there's no regex for
"horizontal space"; I guess it would be a matter of enumerating a suitable
range of code points.​

-- 
https://rrt.sc3d.org

[-- Attachment #2: Type: text/html, Size: 1904 bytes --]

Re: Recent regression in Org master with parsing of verbatim code with spaces

[Robert Klein]

Hi,

On Wed, 11 Oct 2017 00:47:04 +0100
Reuben Thomas <rrt@sc3d.org> wrote:

> On 10 October 2017 at 18:29, Kaushal Modi <kaushal.modi@gmail.com>
> wrote:
> 
> > On Tue, Oct 10, 2017 at 1:08 PM Kaushal Modi
> > <kaushal.modi@gmail.com> wrote:
> >  
> >> I scanned through recent commits, but haven't yet figured out which
> >> commit affected this. Before I end up with git bisect, does anyone
> >> know a master branch commit that could have caused this?
> >>  
> >
> > Hello,
> >
> > I found the commit causing this change:
> >
> > http://orgmode.org/cgit.cgi/org-mode.git/commit/?id=
> > 8383f7e08b964e129b175ef1c0297b1f4a8ed235
> >  
> 
> ​I haven't seen the rest of this thread, but at a guess is the
> problem to do with [:space:] matching vertical space such as newline
> and vertical tab? Fixing that is a bit painful, since as far as I
> know there's no regex for "horizontal space"; I guess it would be a
> matter of enumerating a suitable range of code points.​

I'm fiddling with org-emphasis-regexp-components in my .emacs.  I
put all horizontal whitespace characters (Wikipedia: "whitespace
character" with Unicode character property WSpace="y") in a string and
concat org-emphasis-regexp-components.  Note, I include some extra
characters for pre, post, and borders (typographical double quotes).

(setq horizontal-whitespace
      (string
       (cdr (assoc-string "CHARACTER TABULATION" (ucs-names)))
       (cdr (assoc-string "SPACE" (ucs-names)))
       (cdr (assoc-string "NO-BREAK SPACE" (ucs-names)))
       (cdr (assoc-string "OGHAM SPACE MARK" (ucs-names)))
       (cdr (assoc-string "EN QUAD" (ucs-names)))
       (cdr (assoc-string "EM QUAD" (ucs-names)))
       (cdr (assoc-string "EN SPACE" (ucs-names)))
       (cdr (assoc-string "EM SPACE" (ucs-names)))
       (cdr (assoc-string "THREE-PER-EM SPACE" (ucs-names)))
       (cdr (assoc-string "FOUR-PER-EM SPACE" (ucs-names)))
       (cdr (assoc-string "SIX-PER-EM SPACE" (ucs-names)))
       (cdr (assoc-string "FIGURE SPACE" (ucs-names)))
       (cdr (assoc-string "PUNCTUATION SPACE" (ucs-names)))
       (cdr (assoc-string "THIN SPACE" (ucs-names)))
       (cdr (assoc-string "HAIR SPACE" (ucs-names)))
       (cdr (assoc-string "NARROW NO-BREAK SPACE" (ucs-names)))
       (cdr (assoc-string "MEDIUM MATHEMATICAL SPACE" (ucs-names)))
       (cdr (assoc-string "IDEOGRAPHIC SPACE" (ucs-names)))))
(setq org-emphasis-regexp-components
      `(,(concat horizontal-whitespace
                 "('\"{„“")
        ,(concat "-"
                 horizontal-whitespace
                 ".,:!?;'\")}\\[“”")
        ,(concat horizontal-whitespace
                 "[:space:],\"'„“”")
        "."
        10))


Best regards
Robert

Re: Recent regression in Org master with parsing of verbatim code with spaces

[Nicolas Goaziou]

Hello,

Kaushal Modi <kaushal.modi@gmail.com> writes:

> Hello,
>
> I just noticed a regression in master when compared to 9.1.2.
>
> MWE:
>
> =====
> =title = "foo"​=
>
> ~title = "foo"​~
> =====
>
> On master:
>
> C-c C-e t A outputs (ignoring, author, title, toc, etc):
>
> =====
> =title = "foo"​=
>
> ~title = "foo"​~
> =====
>
> Also the above text does NOT get fontified as verbatim in the Org
> buffer.

Why do you consider it to be a regression? IIRC, the change allowed to
use zero-width space as a way to escape markup.

Regards,

-- 
Nicolas Goaziou

Re: Recent regression in Org master with parsing of verbatim code with spaces

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 436 bytes --]

On Thu, Oct 12, 2017 at 11:01 AM Nicolas Goaziou <mail@nicolasgoaziou.fr>
wrote:

> Why do you consider it to be a regression? IIRC, the change allowed to
> use zero-width space as a way to escape markup.
>

Earlier,

=title = "foo"​=

just worked.

Now that does not work.

Also below does not work.

=title <ZWS>= "foo"​=

(where <ZWS> is the unicode 0x200b inserted for ZERO WIDTH SPACE)
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 1012 bytes --]

Re: Recent regression in Org master with parsing of verbatim code with spaces

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1993 bytes --]

On Thu, Oct 12, 2017 at 11:20 AM Kaushal Modi <kaushal.modi@gmail.com>
wrote:

> On Thu, Oct 12, 2017 at 11:01 AM Nicolas Goaziou <mail@nicolasgoaziou.fr>
> wrote:
>
>> Why do you consider it to be a regression? IIRC, the change allowed to
>> use zero-width space as a way to escape markup.
>
>
Also I spent few hours trying to understand why

(setq org-emphasis-regexp-components
      '("-[:space:]('\"{"             ;prematch allowed
        "-[:space:].,:!?;'\")}\\["     ;postmatch allowed
        " \t\r\n"                     ;border *not* allowed
        "."                           ;body
        20))                           ;newlines

seems to resolve this issue.. I just changed the "border not allowed"
regexp from "[:space:]" to " \t\r\n".

I spent few hours, using edebug, message based debugging but couldn't
figure out.

If I try this sandbox method to debug:

(dolist (str '("=title = \"foo\"=

~title = \"foo\"​~"))
  (message "String: %s" str)
  (let* ((working-verbatim-re
"\\([-[:space:]('\"{]\\|^\\)\\(\\(?3:[=~]\\)\\([^
\n]\\|[^

\n].*?\\(?:\n.*?\\)\\{0,1\\}[^

\n]\\)\\3\\)\\([-[:space:].,:!?;'\")}\\[]\\|$\\)")
         (bad-verbatim-re
"\\([-[:space:]('\"{]\\|^\\)\\(\\([=~]\\)\\([^[:space:]]\\|[^[:space:]].*?\\(?:\n.*?\\)\\{0,1\\}[^[:space:]]\\)\\3\\)\\([-[:space:].,:!?;'\")}\\[]\\|$\\)")
         (using-working-verbatim-re (string-match-p working-verbatim-re
str))
         (using-bad-verbatim-re (string-match-p bad-verbatim-re str))
         (match? (equal using-working-verbatim-re using-bad-verbatim-re)))
    (message ".. working match %S" using-working-verbatim-re)
    (message ".. bad match %S" using-bad-verbatim-re)
    (message ".. match? %S" match?)))

Both good and "bad" regexp are matching that test string.

So don't know how exactly this commit is breaking this.. but I know for
sure that reverting org-emphasis-regexp-components to the earlier value
fixes things.
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 3338 bytes --]

Re: Recent regression in Org master with parsing of verbatim code with spaces

[Nicolas Goaziou]

Kaushal Modi <kaushal.modi@gmail.com> writes:

> On Thu, Oct 12, 2017 at 11:01 AM Nicolas Goaziou <mail@nicolasgoaziou.fr>
> wrote:
>
>> Why do you consider it to be a regression? IIRC, the change allowed to
>> use zero-width space as a way to escape markup.
>>
>
> Earlier,
>
> =title = "foo"​=
>
> just worked.
>
> Now that does not work.

Again, that's the whole point of the change. So, to move discussion
forward, what do you need the zero width space for, in this particular
case?

It may be that your use case and the one implemented by the change are
not compatible. It doesn't mean it's a regression, though.

> Also below does not work.
>
> =title <ZWS>= "foo"​=
>
> (where <ZWS> is the unicode 0x200b inserted for ZERO WIDTH SPACE)

What "does not work"? What do you want to achieve?

Regards,

Re: Recent regression in Org master with parsing of verbatim code with spaces

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]

On Thu, Oct 12, 2017 at 11:35 AM Nicolas Goaziou <mail@nicolasgoaziou.fr>
wrote:

> Again, that's the whole point of the change. So, to move discussion
> forward, what do you need the zero width space for, in this particular
> case?
>

I do not need zero width space.. I mentioned it and tried it because you
said:

> IIRC, the change allowed to use zero-width space as a way to escape
markup.

I believe we have some confusion here.

For clarity: I do not plan to use zero-width space for escaping markup.


> > Also below does not work.
> >
> > =title <ZWS>= "foo"​=
> >
> > (where <ZWS> is the unicode 0x200b inserted for ZERO WIDTH SPACE)
>
> What "does not work"? What do you want to achieve?
>

I expect  "=title = "foo"​=" to export to `title = "foo"​' as before (using
ASCII backend as example as in my first post).

Before this change:
- =title = "foo"​= got fontified with org-verbatim face
- =title = "foo"​= exported to `title = "foo"​ (C-c C-e t A)

After this change:
- =title = "foo"​= does not get fontified with org-verbatim face (no Org
face is applied)
- =title = "foo"​= exports to =title = "foo"​= as it is​ (C-c C-e t A)
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 2400 bytes --]

Re: Recent regression in Org master with parsing of verbatim code with spaces

[Nicolas Goaziou]

Kaushal Modi <kaushal.modi@gmail.com> writes:

> I believe we have some confusion here.
>
> For clarity: I do not plan to use zero-width space for escaping
> markup.

There is a zero-width space right before the last "=" character. I don't
understand why you do use it if that's not your plan. Maybe you haven't
noticed there is one?

>> > Also below does not work.
>> >
>> > =title <ZWS>= "foo"​=
>> >
>> > (where <ZWS> is the unicode 0x200b inserted for ZERO WIDTH SPACE)
>>
>> What "does not work"? What do you want to achieve?
>>
>
> I expect  "=title = "foo"​=" to export to `title = "foo"​' as before (using
> ASCII backend as example as in my first post).

Then remove the last zero-width space.

Regards,

Re: Recent regression in Org master with parsing of verbatim code with spaces

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]

On Thu, Oct 12, 2017 at 11:53 AM Nicolas Goaziou <mail@nicolasgoaziou.fr>
wrote:

> There is a zero-width space right before the last "=" character. I don't
> understand why you do use it if that's not your plan. Maybe you haven't
> noticed there is one?
>

(Blush) that is embarrassing.. sorry for the trouble. That must have
sneaked in as I was trying that fontification to work through multiple
iterations..

Then remove the last zero-width space.
>

I just did.. (I need to now fontify zero-width space to prevent such
occurrences.)
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 1111 bytes --]