Bug: Table export to [tc]sv doesn't convert \vert [7.8.11]

Bug: Table export to [tc]sv doesn't convert \vert [7.8.11]

[Nathan Trapuzzano]

Converting \vert and \vert{} when exporting an Org table to csv or tsv does not
properly convert them to the vertical bar. Instead, they are retained literally.

Emacs  : GNU Emacs 24.2.1 (x86_64-unknown-linux-gnu) of 2012-11-20 on Nathan-GNU
Package: Org-mode version 7.8.11

Re: Bug: Table export to [tc]sv doesn't convert \vert [7.8.11]

[Bastien]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

Hi Nathan,

Nathan Trapuzzano <nbtrap@nbtrap.com> writes:

> Converting \vert and \vert{} when exporting an Org table to csv or tsv does not
> properly convert them to the vertical bar. Instead, they are retained literally.

A hackish patch like this one fixes this, but it's not general enough.
Also, converting entities in tables is prone to false-positives, so
let's live with this limitation until we generalize the way entities
are handled.

Best,


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: org-convert-csv-entities.patch --]
[-- Type: text/x-patch, Size: 740 bytes --]

diff --git a/lisp/org.el b/lisp/org.el
index 04bb2e9..a366f95 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -20030,8 +20030,12 @@ With prefix arg UNCOMPILED, load the uncompiled versions."
 (defun org-quote-csv-field (s)
   "Quote field for inclusion in CSV material."
   (if (string-match "[\",]" s)
-      (concat "\"" (mapconcat 'identity (split-string s "\"") "\"\"") "\"")
-    s))
+      (setq s (concat "\"" (mapconcat 'identity (split-string s "\"") "\"\"") "\"")))
+  (if (string-match "\\\\\\([a-zA-Z]+[0-9]*\\)\\({}\\)?" s)
+      (setq s (replace-match
+	       (org-entity-get-representation
+		(match-string 1 s) 'ascii) t t s 0)))
+  s)
 
 (defun org-force-self-insert (N)
   "Needed to enforce self-insert under remapping."

[-- Attachment #3: Type: text/plain, Size: 14 bytes --]


-- 
 Bastien

Re: Bug: Table export to [tc]sv doesn't convert \vert [7.8.11]

[Nathan Trapuzzano]

Thanks Bastien. I agree that we need a more general fix for \vert escapes for
exports to all formats. Unfortunately, the best way of going about it (in my
opinion) would not be backwards compatible.

I'd offer the patch myself if I knew it would be accepted on the master branch,
so I will live with it the way it is until a solution is proposed.

Nathan

On Fri, 28 Dec 2012 18:26:26 +0100
Bastien <bzg@altern.org> wrote:

> Hi Nathan,
> 
> Nathan Trapuzzano <nbtrap@nbtrap.com> writes:
> 
> > Converting \vert and \vert{} when exporting an Org table to csv or tsv does
> > not properly convert them to the vertical bar. Instead, they are retained
> > literally.
> 
> A hackish patch like this one fixes this, but it's not general enough.
> Also, converting entities in tables is prone to false-positives, so
> let's live with this limitation until we generalize the way entities
> are handled.
> 
> Best,
> 

Re: Bug: Table export to [tc]sv doesn't convert \vert [7.8.11]

[Bastien]

Hi Nathan,

Nathan Trapuzzano <nbtrap@nbtrap.com> writes:

> Thanks Bastien. I agree that we need a more general fix for \vert escapes for
> exports to all formats. Unfortunately, the best way of going about it (in my
> opinion) would not be backwards compatible.

Hopefully we can fix this while working on Org 8.0, which greatest
feature will be the new exporter.  I expect everyone to expect this
exporter can do miracles, even for such details.

> I'd offer the patch myself if I knew it would be accepted on the master branch,
> so I will live with it the way it is until a solution is proposed.

We cannot decide whether the patch will be accepted until we see it :)

Up to you, thanks in advance if you give this a shot,

-- 
 Bastien

Re: Bug: Table export to [tc]sv doesn't convert \vert [7.8.11]

[Nathan Trapuzzano]

Of course the particular implementation will have to be seen before it
can be accepted, but I'd like to get the spec accepted (provisionally)
before setting to work on it.

Here's what I propose:

1. Do away with \vert{} entirely, leaving just \vert as an escape
sequence standing for |, no matter where it appears. \vert{}
unnecesarily complicates things, in my opinion.

2. The escape sequence must itself be escapable, wherefore I propose
to give the backslash special meaning in front of the string
"vert". Specifically:

  a. An even number of consecutive backslashes followed by "vert"
  stands for that number of backslashes divided by two followed by
  "vert".

  b. An odd number of consecutive backslashes followed by "vert"
  stands for that many backslashes integer-divided by two, followed by
  "|".

For example, "\vert" exports to "|", "\\vert" to "\vert", "\\\vert" to
"\|", "\\\\vert" to "\\vert", and so on. Obviously, upon importing,
the reverse of the above will be carried out.

Nathan

On Sun, 30 Dec 2012 10:00:09 +0100
Bastien <bzg@altern.org> wrote:

> Hi Nathan,
> 
> Nathan Trapuzzano <nbtrap@nbtrap.com> writes:
> 
> > Thanks Bastien. I agree that we need a more general fix for \vert escapes
> > for exports to all formats. Unfortunately, the best way of going about it
> > (in my opinion) would not be backwards compatible.
> 
> Hopefully we can fix this while working on Org 8.0, which greatest
> feature will be the new exporter.  I expect everyone to expect this
> exporter can do miracles, even for such details.
> 
> > I'd offer the patch myself if I knew it would be accepted on the master
> > branch, so I will live with it the way it is until a solution is proposed.
> 
> We cannot decide whether the patch will be accepted until we see it :)
> 

Re: Bug: Table export to [tc]sv doesn't convert \vert [7.8.11]

[Bastien]

Hi Nathan,

Nathan Trapuzzano <nbtrap@nbtrap.com> writes:

> Of course the particular implementation will have to be seen before it
> can be accepted, but I'd like to get the spec accepted (provisionally)
> before setting to work on it.

As a start, you can look at the way org-e-*.el backend export tables,
(use `org-e-ascii-table-cell' as an entry point).  What we need is a
callback function to convert entities in cells, in org-export.el.

Since we already have org-entities, I would use it together with
`org-entity-get-representation' to convert entities from the \vert{}
representation to the "|" character (in the ASCII backend.)

> Here's what I propose:
>
> 1. Do away with \vert{} entirely, leaving just \vert as an escape
> sequence standing for |, no matter where it appears. \vert{}
> unnecesarily complicates things, in my opinion.

Better to rely on org-entities and the way entities are treated so far.

> 2. The escape sequence must itself be escapable, wherefore I propose
> to give the backslash special meaning in front of the string
> "vert". Specifically:
>
>   a. An even number of consecutive backslashes followed by "vert"
>   stands for that number of backslashes divided by two followed by
>   "vert".
>
>   b. An odd number of consecutive backslashes followed by "vert"
>   stands for that many backslashes integer-divided by two, followed by
>   "|".
>
> For example, "\vert" exports to "|", "\\vert" to "\vert", "\\\vert" to
> "\|", "\\\\vert" to "\\vert", and so on. Obviously, upon importing,
> the reverse of the above will be carried out.

I would not take that route -- from experience, escaping espace
sequences can drive you mad, and all this is not intuitive for users.

I would simply convert entities by default and use a special table 
cookie in lines where you do not want the conversion to happen.

For example:

| Header 1 | Header 2 |
|----------+----------|
| \vert{}  | ABC      |

=> convert to "|" 

But:

|   | Header 1 | Header 2 |
|---+----------+----------|
| \ | \vert{}  | ABC      |

Don't convert.  The "\" char is free and a good choice here.

-- 
 Bastien

Re: Bug: Table export to [tc]sv doesn't convert \vert [7.8.11]

[Nathan Trapuzzano]

Bastien <bzg@altern.org> writes:

> Hi Nathan,
>
> Nathan Trapuzzano <nbtrap@nbtrap.com> writes:
>
>> Of course the particular implementation will have to be seen before it
>> can be accepted, but I'd like to get the spec accepted (provisionally)
>> before setting to work on it.
>
> As a start, you can look at the way org-e-*.el backend export tables,
> (use `org-e-ascii-table-cell' as an entry point).  What we need is a
> callback function to convert entities in cells, in org-export.el.
>
> Since we already have org-entities, I would use it together with
> `org-entity-get-representation' to convert entities from the \vert{}
> representation to the "|" character (in the ASCII backend.)
>
>> Here's what I propose:
>>
>> 1. Do away with \vert{} entirely, leaving just \vert as an escape
>> sequence standing for |, no matter where it appears. \vert{}
>> unnecesarily complicates things, in my opinion.
>
> Better to rely on org-entities and the way entities are treated so far.

I only recently started using org-mode, let alone looking at the
source. I'll see what you mean if I get the time.

>> 2. The escape sequence must itself be escapable, wherefore I propose
>> to give the backslash special meaning in front of the string
>> "vert". Specifically:
>>
>>   a. An even number of consecutive backslashes followed by "vert"
>>   stands for that number of backslashes divided by two followed by
>>   "vert".
>>
>>   b. An odd number of consecutive backslashes followed by "vert"
>>   stands for that many backslashes integer-divided by two, followed by
>>   "|".
>>
>> For example, "\vert" exports to "|", "\\vert" to "\vert", "\\\vert" to
>> "\|", "\\\\vert" to "\\vert", and so on. Obviously, upon importing,
>> the reverse of the above will be carried out.
>
> I would not take that route -- from experience, escaping espace
> sequences can drive you mad, and all this is not intuitive for users.

The problem with not escaping escape sequences is that there always ends
up being some string of characters that cannot be represented. As for
driving one mad, I think that "\\\\+vert" is much simpler to understand
than "\\(\\b\\\\vert\\b\\|\\\\vert{}\\)", especially if the latter only
applies when the row in which the sequence falls is not escaped as you
describe below.

> I would simply convert entities by default and use a special table 
> cookie in lines where you do not want the conversion to happen.
>
> For example:
>
> | Header 1 | Header 2 |
> |----------+----------|
> | \vert{}  | ABC      |
>
> => convert to "|" 
>
> But:
>
> |   | Header 1 | Header 2 |
> |---+----------+----------|
> | \ | \vert{}  | ABC      |
>
> Don't convert.  The "\" char is free and a good choice here.

This would apply the escape to the entire row. I supposed it's better
than the way things are now, but it's still not a very good solution in
my mind.

Re: Bug: Table export to [tc]sv doesn't convert \vert [7.8.11]

[Bastien]

Hi Nathan,

Nathan Trapuzzano <nbtrap@nbtrap.com> writes:

> This would apply the escape to the entire row. I supposed it's better
> than the way things are now, but it's still not a very good solution in
> my mind.

Please do feel free to have a go and let us know what works best.

Thanks!

-- 
 Bastien