Show presence of zero width spaces using overlay

Show presence of zero width spaces using overlay

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 731 bytes --]

Hi all,

The suggestion to use zero width spaces to sort of "escape" stuff in org
mode has come up many times. I have started using it a lot and suggest that
to other people now.

Here is one such recent QnA on emacs.SE:
http://emacs.stackexchange.com/a/16702/115

The OP of that question made a valid point that while using zero width
spaces is convenient, it is not apparent where it has been used in the
document.

To resolve that, I am requesting a feature.

Can the zero width spaces be optionally represented with something like a
single space overlay with a colored background (like purple)?

Making this officially part of org-mode will probably make more people make
use of it.

.. Or should people use zero width spaces?

[-- Attachment #2: Type: text/html, Size: 993 bytes --]

Re: Show presence of zero width spaces using overlay

[Nicolas Goaziou]

Hello,

Kaushal Modi <kaushal.modi@gmail.com> writes:

> The suggestion to use zero width spaces to sort of "escape" stuff in org
> mode has come up many times. I have started using it a lot and suggest that
> to other people now.
>
> Here is one such recent QnA on emacs.SE:
> http://emacs.stackexchange.com/a/16702/115
>
> The OP of that question made a valid point that while using zero width
> spaces is convenient, it is not apparent where it has been used in the
> document.
>
> To resolve that, I am requesting a feature.
>
> Can the zero width spaces be optionally represented with something like a
> single space overlay with a colored background (like purple)?
>
> Making this officially part of org-mode will probably make more people make
> use of it.
>
> .. Or should people use zero width spaces?

Entities can be used to escape most Org syntax, e.g. 

  \ast{}shrug\ast

I see no compelling reason to treat zero width spaces specially.


Regards,

-- 
Nicolas Goaziou

Re: Show presence of zero width spaces using overlay

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1589 bytes --]

My most common uses are escaping double quotes (") and equals (=) within
org verbatim blocks (=VERBATIM=)

Examples:

1. =var=[ZWS]val=
2. =[ZWS]"something"[ZWS]=

Here [ZWS] is the 0x200b zero width space unicode char.

I found [ZWS] useful as a generic escape char for org mode. There are few
other cases where this has been useful, but I can't recall right now.

In any case, what would be the recommended way to escape " and = in the
above 2 examples?

On Fri, Sep 18, 2015 at 5:09 PM Nicolas Goaziou <mail@nicolasgoaziou.fr>
wrote:

> Hello,
>
> Kaushal Modi <kaushal.modi@gmail.com> writes:
>
> > The suggestion to use zero width spaces to sort of "escape" stuff in org
> > mode has come up many times. I have started using it a lot and suggest
> that
> > to other people now.
> >
> > Here is one such recent QnA on emacs.SE:
> > http://emacs.stackexchange.com/a/16702/115
> >
> > The OP of that question made a valid point that while using zero width
> > spaces is convenient, it is not apparent where it has been used in the
> > document.
> >
> > To resolve that, I am requesting a feature.
> >
> > Can the zero width spaces be optionally represented with something like a
> > single space overlay with a colored background (like purple)?
> >
> > Making this officially part of org-mode will probably make more people
> make
> > use of it.
> >
> > .. Or should people use zero width spaces?
>
> Entities can be used to escape most Org syntax, e.g.
>
>   \ast{}shrug\ast
>
> I see no compelling reason to treat zero width spaces specially.
>
>
> Regards,
>
> --
> Nicolas Goaziou
>

[-- Attachment #2: Type: text/html, Size: 2340 bytes --]

Re: Show presence of zero width spaces using overlay

[Eric Abrahamsen]

Kaushal Modi <kaushal.modi@gmail.com> writes:

> My most common uses are escaping double quotes (") and equals (=)
> within org verbatim blocks (=VERBATIM=)
>
> Examples:
>
> 1. =var=[ZWS]val=
> 2. =[ZWS]"something"[ZWS]=
>
> Here [ZWS] is the 0x200b zero width space unicode char.
>
> I found [ZWS] useful as a generic escape char for org mode. There are
> few other cases where this has been useful, but I can't recall right
> now.
>
> In any case, what would be the recommended way to escape " and = in
> the above 2 examples?

Check out the variable `org-entities' for all the replaceable escape
codes. It's got quotes and equal!

E

Re: Show presence of zero width spaces using overlay

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1655 bytes --]

Thanks for letting me know about org-entities. That is awesome. I now know
how to escape various characters in general, but unfortunately this does
not work within verbatim formatting (which makes sense).

Here's a minimum working example:

=====

* Escaping =equal= sign in verbatim formatting.
=a\equal{}b+c=
** Here's the same but using zero width spaces instead of /org entities/.
=a=b+c=

* Here I am trying to have double quotes in verbatim formatting.
=\quot{}This is inbetween double quotes\quot{}=.
** Here's the same but using zero width spaces instead of /org entities/.
=​"This is inbetween double quotes"​=.

* And here's to escapes asterisks
This is *bold*. But this is \ast{}not bold\ast{}.
This works!

=====

Here's what it looks like when exported:
https://dl.dropboxusercontent.com/u/10985/escape-chars.pdf

On Fri, Sep 18, 2015 at 9:48 PM Eric Abrahamsen <eric@ericabrahamsen.net>
wrote:

> Kaushal Modi <kaushal.modi@gmail.com> writes:
>
> > My most common uses are escaping double quotes (") and equals (=)
> > within org verbatim blocks (=VERBATIM=)
> >
> > Examples:
> >
> > 1. =var=[ZWS]val=
> > 2. =[ZWS]"something"[ZWS]=
> >
> > Here [ZWS] is the 0x200b zero width space unicode char.
> >
> > I found [ZWS] useful as a generic escape char for org mode. There are
> > few other cases where this has been useful, but I can't recall right
> > now.
> >
> > In any case, what would be the recommended way to escape " and = in
> > the above 2 examples?
>
> Check out the variable `org-entities' for all the replaceable escape
> codes. It's got quotes and equal!
>
> E
>
>
>

[-- Attachment #2: Type: text/html, Size: 2431 bytes --]

Re: Show presence of zero width spaces using overlay

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 2450 bytes --]

Here's the MWE once again with proper indication ([ZWS]) of where you need
to insert the zero width space char.

=====

* Escaping =equal= sign in verbatim formatting.
=a\equal{}b+c=
** Here's the same but using zero width spaces instead of /org entities/.
=a=[ZWS]b+c=

* Here I am trying to have double quotes in verbatim formatting.
=\quot{}This is inbetween double quotes\quot{}=.
** Here's the same but using zero width spaces instead of /org entities/.
=[ZWS]"This is inbetween double quotes"[ZWS]=.

* And here's to escapes asterisks
This is *bold*. But this is \ast{}not bold\ast{}.
This works!

=====



On Sat, Sep 19, 2015 at 12:07 AM Kaushal Modi <kaushal.modi@gmail.com>
wrote:

> Thanks for letting me know about org-entities. That is awesome. I now know
> how to escape various characters in general, but unfortunately this does
> not work within verbatim formatting (which makes sense).
>
> Here's a minimum working example:
>
> =====
>
> * Escaping =equal= sign in verbatim formatting.
> =a\equal{}b+c=
> ** Here's the same but using zero width spaces instead of /org entities/.
> =a=b+c=
>
> * Here I am trying to have double quotes in verbatim formatting.
> =\quot{}This is inbetween double quotes\quot{}=.
> ** Here's the same but using zero width spaces instead of /org entities/.
> =​"This is inbetween double quotes"​=.
>
> * And here's to escapes asterisks
> This is *bold*. But this is \ast{}not bold\ast{}.
> This works!
>
> =====
>
> Here's what it looks like when exported:
> https://dl.dropboxusercontent.com/u/10985/escape-chars.pdf
>
> On Fri, Sep 18, 2015 at 9:48 PM Eric Abrahamsen <eric@ericabrahamsen.net>
> wrote:
>
>> Kaushal Modi <kaushal.modi@gmail.com> writes:
>>
>> > My most common uses are escaping double quotes (") and equals (=)
>> > within org verbatim blocks (=VERBATIM=)
>> >
>> > Examples:
>> >
>> > 1. =var=[ZWS]val=
>> > 2. =[ZWS]"something"[ZWS]=
>> >
>> > Here [ZWS] is the 0x200b zero width space unicode char.
>> >
>> > I found [ZWS] useful as a generic escape char for org mode. There are
>> > few other cases where this has been useful, but I can't recall right
>> > now.
>> >
>> > In any case, what would be the recommended way to escape " and = in
>> > the above 2 examples?
>>
>> Check out the variable `org-entities' for all the replaceable escape
>> codes. It's got quotes and equal!
>>
>> E
>>
>>
>>

[-- Attachment #2: Type: text/html, Size: 3821 bytes --]

Re: Show presence of zero width spaces using overlay

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 4975 bytes --]

I got really interested in org-entities (to deal with the case I mentioned
in the first email in this thread like \ast{}shrug\ast{}) and came up with
this:

=====

(defun modi/org-entity-get-name (char)
  "Return the entity name for CHAR. For example, return \"ast\" for *."
  (let ((ll (append org-entities-user
                    org-entities))
        e name utf8)
    (catch 'break
      (while ll
        (setq e (pop ll))
        (when (not (stringp e))
          (setq utf8 (nth 6 e))
          (when (string= char utf8)
            (setq name (car e))
            (throw 'break name)))))))

(defun modi/org-insert-org-entity-maybe (orig-fun &rest args)
  "When the universal prefix C-u is used before entering any character,
insert the character's `org-entity' name if available."
  (let ((pressed-key (this-command-keys))
        entity-name)
    (when (and (listp args) (eq 4 (car args)))
      (setq entity-name (modi/org-entity-get-name pressed-key))
      (when entity-name
        (setq entity-name (concat "\\" entity-name "{}"))
        (insert entity-name)
        (message (concat "Inserted `org-entity' "
                         (propertize entity-name
                                     'face 'font-lock-function-name-face)
                         " for the symbol "
                         (propertize pressed-key
                                     'face 'font-lock-function-name-face)
                         "."))))
    (when (null entity-name)
      (apply orig-fun args))))

(advice-add 'org-self-insert-command :around
#'modi/org-insert-org-entity-maybe)

=====

After evaluating the above, whenever you do C-u *, C-u /, C-u =, etc, that
symbol's org-entity will be inserted (if available in either
org-entities-user or org-entities).
If an org-entity match is not found, that symbol will be inserted 4 times,
as C-u would usually do.

The message tells the user if the org-entity got inserted so that there is
not confusion.

The advised portion gets executed only if the below 2 conditions match:
- User used C-u prefix (not M-4 or C-u C-u or anything else)
- The entered character has a match in the org entities lists.

Question to the list is: Does this advise mask any useful functionality of
org-self-insert-command?


--
Kaushal Modi

On Sat, Sep 19, 2015 at 12:11 AM, Kaushal Modi <kaushal.modi@gmail.com>
wrote:

> Here's the MWE once again with proper indication ([ZWS]) of where you need
> to insert the zero width space char.
>
> =====
>
> * Escaping =equal= sign in verbatim formatting.
> =a\equal{}b+c=
> ** Here's the same but using zero width spaces instead of /org entities/.
> =a=[ZWS]b+c=
>
> * Here I am trying to have double quotes in verbatim formatting.
> =\quot{}This is inbetween double quotes\quot{}=.
> ** Here's the same but using zero width spaces instead of /org entities/.
> =[ZWS]"This is inbetween double quotes"[ZWS]=.
>
> * And here's to escapes asterisks
> This is *bold*. But this is \ast{}not bold\ast{}.
> This works!
>
> =====
>
>
>
> On Sat, Sep 19, 2015 at 12:07 AM Kaushal Modi <kaushal.modi@gmail.com>
> wrote:
>
>> Thanks for letting me know about org-entities. That is awesome. I now
>> know how to escape various characters in general, but unfortunately this
>> does not work within verbatim formatting (which makes sense).
>>
>> Here's a minimum working example:
>>
>> =====
>>
>> * Escaping =equal= sign in verbatim formatting.
>> =a\equal{}b+c=
>> ** Here's the same but using zero width spaces instead of /org entities/.
>> =a=b+c=
>>
>> * Here I am trying to have double quotes in verbatim formatting.
>> =\quot{}This is inbetween double quotes\quot{}=.
>> ** Here's the same but using zero width spaces instead of /org entities/.
>> =​"This is inbetween double quotes"​=.
>>
>> * And here's to escapes asterisks
>> This is *bold*. But this is \ast{}not bold\ast{}.
>> This works!
>>
>> =====
>>
>> Here's what it looks like when exported:
>> https://dl.dropboxusercontent.com/u/10985/escape-chars.pdf
>>
>> On Fri, Sep 18, 2015 at 9:48 PM Eric Abrahamsen <eric@ericabrahamsen.net>
>> wrote:
>>
>>> Kaushal Modi <kaushal.modi@gmail.com> writes:
>>>
>>> > My most common uses are escaping double quotes (") and equals (=)
>>> > within org verbatim blocks (=VERBATIM=)
>>> >
>>> > Examples:
>>> >
>>> > 1. =var=[ZWS]val=
>>> > 2. =[ZWS]"something"[ZWS]=
>>> >
>>> > Here [ZWS] is the 0x200b zero width space unicode char.
>>> >
>>> > I found [ZWS] useful as a generic escape char for org mode. There are
>>> > few other cases where this has been useful, but I can't recall right
>>> > now.
>>> >
>>> > In any case, what would be the recommended way to escape " and = in
>>> > the above 2 examples?
>>>
>>> Check out the variable `org-entities' for all the replaceable escape
>>> codes. It's got quotes and equal!
>>>
>>> E
>>>
>>>
>>>

[-- Attachment #2: Type: text/html, Size: 11568 bytes --]

Re: Show presence of zero width spaces using overlay

[Eric Abrahamsen]

Kaushal Modi <kaushal.modi@gmail.com> writes:

> I got really interested in org-entities (to deal with the case I
> mentioned in the first email in this thread like \ast{}shrug\ast{})
> and came up with this:

[...]

> Question to the list is: Does this advise mask any useful
> functionality of org-self-insert-command?

This is very cool, in principle! My first reaction would be: people will
probably just want a single command that inserts an entity, either by
name or by the character itself. Tapping into basic keystrokes like this
is very clever, but people only like clever when it's clever exactly the
way they want it, and the more clever a thing is the more opinions
diverge, if that makes sense. Why not just do something like, for
example, `sgml-name-char'?

Re: Show presence of zero width spaces using overlay

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1438 bytes --]

That's correct. That code might be an overkill for people who never needed
org entities.

I thought that I have never used `C-u` before plain single characters, so
why not make use of that in org mode. With that code in place, inserting an
org entity is now C-u away :)

If I happen to need org entities for nonascii chars, I can associate them
to plain letters like `a`, `b`, `c`... in org-entities-user. For example,
`C-u c` can be used to enter copyright symbol org entity.

I haven't yet used `sgml-name-char`, thanks for the tip.

--
Kaushal Modi
On Sep 19, 2015 3:26 AM, "Eric Abrahamsen" <eric@ericabrahamsen.net> wrote:

> Kaushal Modi <kaushal.modi@gmail.com> writes:
>
> > I got really interested in org-entities (to deal with the case I
> > mentioned in the first email in this thread like \ast{}shrug\ast{})
> > and came up with this:
>
> [...]
>
> > Question to the list is: Does this advise mask any useful
> > functionality of org-self-insert-command?
>
> This is very cool, in principle! My first reaction would be: people will
> probably just want a single command that inserts an entity, either by
> name or by the character itself. Tapping into basic keystrokes like this
> is very clever, but people only like clever when it's clever exactly the
> way they want it, and the more clever a thing is the more opinions
> diverge, if that makes sense. Why not just do something like, for
> example, `sgml-name-char'?
>
>
>

[-- Attachment #2: Type: text/html, Size: 1907 bytes --]

Re: Show presence of zero width spaces using overlay

[Nicolas Goaziou]

Kaushal Modi <kaushal.modi@gmail.com> writes:

> My most common uses are escaping double quotes (") and equals (=) within
> org verbatim blocks (=VERBATIM=)
>
> Examples:
>
> 1. =var=[ZWS]val=
> 2. =[ZWS]"something"[ZWS]=
>
> Here [ZWS] is the 0x200b zero width space unicode char.
>
> I found [ZWS] useful as a generic escape char for org mode. There are few
> other cases where this has been useful, but I can't recall right now.
>
> In any case, what would be the recommended way to escape " and = in the
> above 2 examples?

As I said, entities can only escape most Org syntax. Verbatim and code
objects are an exception. Here I think we should introduce an escape
mechanism, much like the one used in macro arguments in order to escape
commas.

For example, in a verbatim object, one only needs to escape "=", using
common "\". Obviously, "\=" constructs can be escaped with "\\=" and so
on. Likewise, in code objects, only "~" needs to be escaped.

In any case, I don't think Org syntax should rely on non-ASCII
characters.

Regards,

Re: Show presence of zero width spaces using overlay

[timor]

Hi,

(this is my first post on this list, to a message that I don't have in
my inbox, so I don't know if it will thread correctly)

Kaushal Modi <kaushal.m...@gmail.com> writes:

> The suggestion to use zero width spaces to sort of "escape" stuff in org
> mode has come up many times. I have started using it a lot and suggest that
> to other people now.
>
> Here is one such recent QnA on emacs.SE:
> http://emacs.stackexchange.com/a/16702/115
>
> The OP of that question made a valid point that while using zero width
> spaces is convenient, it is not apparent where it has been used in the
> document.
>
> To resolve that, I am requesting a feature.
>
> Can the zero width spaces be optionally represented with something like a
> single space overlay with a colored background (like purple)?
>
> Making this officially part of org-mode will probably make more people make
> use of it.
>
> .. Or should people use zero width spaces?

I am the OP of that QnA

I am wondering why there is no simple backslash-escaping of org-mode syntax?

regards,
timor

Re: Show presence of zero width spaces using overlay

[Nicolas Goaziou]

Hello,

timor <timor.dd@gmail.com> writes:

> I am wondering why there is no simple backslash-escaping of org-mode
> syntax?

I think a backslash escaping is needed. However, I don't think it is
going to be that simple. Do you have some specifications in mind?


Regards,

-- 
Nicolas Goaziou

Re: Show presence of zero width spaces using overlay

[timor]

Hello Nicolas,

Sorry for not replying earlier. If my comments don't matter anymore,
please ignore.

Specification-wise, I cannot think of anything other than

"Use <foo-mechanism> to prevent any special characters from formatting text."

I guess one would have to define the list of characters which can be escaped.
I also don't know if backslash is the correct way to go.

Another thing I think is important is that <foo-mechanism> is
semi-obvious to non-org-mode people. That's why I think using
org-entities is not a valid workaround for my original problem.

regards,
timor

2015-09-27 11:41 GMT+02:00 Nicolas Goaziou <mail@nicolasgoaziou.fr>:
> Hello,
>
> timor <timor.dd@gmail.com> writes:
>
>> I am wondering why there is no simple backslash-escaping of org-mode
>> syntax?
>
> I think a backslash escaping is needed. However, I don't think it is
> going to be that simple. Do you have some specifications in mind?
>
>
> Regards,
>
> --
> Nicolas Goaziou

Re: Show presence of zero width spaces using overlay

[Nicolas Goaziou]

Hello,

timor <timor.dd@gmail.com> writes:

> Specification-wise, I cannot think of anything other than
>
> "Use <foo-mechanism> to prevent any special characters from formatting text."
>
> I guess one would have to define the list of characters which can be escaped.
> I also don't know if backslash is the correct way to go.

Actually, this is not really a specification of the problem, which is
quite tricky. The list of characters is but one part of it. Another part
is where escaping should be allowed. Note that allowing to escape
everywhere would be terrible, if only performance-wise. The last part is
about handling it at the parser level (there's also the fontification,
which doesn't use the parser ATM).

> Another thing I think is important is that <foo-mechanism> is
> semi-obvious to non-org-mode people. That's why I think using
> org-entities is not a valid workaround for my original problem.

That's all we have at the moment.


Regards,

-- 
Nicolas Goaziou

Re: Show presence of zero width spaces using overlay

[timor]

Hello Nicolas,

2016-04-19 12:16 GMT+02:00 Nicolas Goaziou <mail@nicolasgoaziou.fr>:
> Actually, this is not really a specification of the problem, which is
> quite tricky. The list of characters is but one part of it. Another part
> is where escaping should be allowed. Note that allowing to escape
> everywhere would be terrible, if only performance-wise. The last part is
> about handling it at the parser level (there's also the fontification,
> which doesn't use the parser ATM).

Good points.  I don't understand enough org mode internals to judge
how much effort is needed on which level for things like this.

Maybe it is viable to start defining and implementing some obvious
cases, which can later be generalized into a rule, so that code has
not to be rewritten on every level at once.

In light of this, I can think of defining the case that bugged me the
most like this:

In plain-text parts of org documents (i.e. that would be exported as
the content of <p> attributes in html, or to plain text paragraphs in
Markdown, etc.), there shall be a way to treat strings in the same way
as the surrounding plain text, which would otherwise be handled by
org-mode in a special way.

Does that help?

regards,
timor

Re: Show presence of zero width spaces using overlay

[Nicolas Goaziou]

timor <timor.dd@gmail.com> writes:

> Maybe it is viable to start defining and implementing some obvious
> cases, which can later be generalized into a rule, so that code has
> not to be rewritten on every level at once.

I'd rather define some specifications right from the beginning, i.e.,
what, where and, ultimately, how.

> In plain-text parts of org documents (i.e. that would be exported as
> the content of <p> attributes in html, or to plain text paragraphs in
> Markdown, etc.), there shall be a way to treat strings in the same way
> as the surrounding plain text, which would otherwise be handled by
> org-mode in a special way.

I don't understand.

Regards,

Re: Show presence of zero width spaces using overlay

[timor]

Hello Nicolas

>> In plain-text parts of org documents (i.e. that would be exported as
>> the content of <p> attributes in html, or to plain text paragraphs in
>> Markdown, etc.), there shall be a way to treat strings in the same way
>> as the surrounding plain text, which would otherwise be handled by
>> org-mode in a special way.
>
> I don't understand.

I don't really know how to express it better, so I will just give an example:

"This ==is really verbatim== stuff."
should be parsed and treated exactly like
"This is really verbatim stuff."

Right now, it is treated in the same way as ~code fragments~.  In
practice other delimiters than "==" would probably be chosen. It would
be an element in org-emphasis-alist, that in fact prevents any
emphasis at all.

In effect, it would allow to really escape parts of paragraphs or link
descriptions, or wherever these kinds of empahsis characters are
valid.

This is only an example of what I want to be able to do, no
specification, sorry.

regards
timor