Syntax Proposal: Multi-line Table Cells/Text Wrapping

Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Atlas Cove]

Hello!
Sorry to barge in with such a big request as someone entirely new, but I have a feature proposal that I'd to discuss.

I'd like to propose an addition to the table syntax that would allow for text wrapping in tables, I personally have myself managing
very large org tables, and having to scroll through them is often cumbersome. As a result, I often yearn for greater control over how I format
my tables.

Allow me to give an example of the updated syntax I propose.

```
| Name         | Description            | Price |
|--------------+------------------------+-------|
| Orange Juice | Very Citrusy! Very \\  |  5.00 |
|              | nice indeed!           |       |
| Grape Juice  | It's like wine, but \\ |  6.00 |
|              | you can have it all \\ |       |
|              | day!                   |       |
```

As you can see, this is a lot more space-efficient than the current, vanilla solution, especially if there's a large block of text.

```
| Name         | Description                                  | Price |
|--------------+----------------------------------------------+-------|
| Orange Juice | Very Citrusy! Very nice indeed!              |  5.00 |
| Grape Juice  | It's like wine, but you can have it all day! |  6.00 |
```

I *am* aware of table-mode's existence, however, I would prefer to work with native org syntax, which often has a cleaner, less bloated HTML output.

Code-wise, the use of the '\\' symbol is only tentative, as I understand that '\' has special meaning elsewhere in org's syntax, (I was even going to submit a
patch to add two special symbols somewhere down the line). '\\' (or whatever symbol is decided on) would cause the org parser to concatenate the next line
into the outputted previous cell, rendering the output of the first example identical to the second example.

Please tell me what you think of this proposal.

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Daniele Nicolodi]

On 17/03/2021 21:29, Atlas Cove wrote:

> Allow me to give an example of the updated syntax I propose.
> 
> ```
> | Name         | Description            | Price |
> |--------------+------------------------+-------|
> | Orange Juice | Very Citrusy! Very \\  |  5.00 |
> |              | nice indeed!           |       |
> | Grape Juice  | It's like wine, but \\ |  6.00 |
> |              | you can have it all \\ |       |
> |              | day!                   |       |
> ```
> 
> As you can see, this is a lot more space-efficient than the current, vanilla solution, especially if there's a large block of text.
> 
> ```
> | Name         | Description                                  | Price |
> |--------------+----------------------------------------------+-------|
> | Orange Juice | Very Citrusy! Very nice indeed!              |  5.00 |
> | Grape Juice  | It's like wine, but you can have it all day! |  6.00 |
> ```

Nitpicking: technically the second table is more space efficient than
the first: in the first table the information is represented in 350
characters, but in just 288 characters in the second table.

Cheers,
Dan

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Juan Manuel Macías]

Hi,

Atlas Cove <Atlas48@gmx.com> writes:

> Code-wise, the use of the '\\' symbol is only tentative, as I
> understand that '\' has special meaning elsewhere in org's syntax, (I
> was even going to submit a patch to add two special symbols somewhere
> down the line). '\\' (or whatever symbol is decided on) would cause
> the org parser to concatenate the next line into the outputted
> previous cell, rendering the output of the first example identical to
> the second example.

I agree that what you are commenting on is the great problem of Org Mode
tables. Just a question. If I have understood it correctly, with this
new syntax you propose, should Org understand all the 'concatenated
text'

Very Citrusy! Very \\
nice indeed!

as a single row (and not as multiple rows)? If so that would be
great, since the best way to deal with this kind of tables with a lot of
text in LaTeX is with the tabularx package. For example, I would export
your table like this:

#+begin_src org
  ,#+LaTeX_Header:\usepackage{tabularx,array}
  ,#+ATTR_LaTeX: :environment tabularx :align l>{\raggedright\arraybackslash}Xl :width .6\textwidth :center t
  | Name         | Description                                           | Price |
  |--------------+-------------------------------------------------------+-------|
  | Orange Juice | Very Citrusy! Very nice indeed! ... and a lot of text |  5.00 |
  | Grape Juice  | It's like wine, but you can have it all day!          |  6.00 |
#+end_src

Best regards,

Juan Manuel 

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Atlas Cove]

[-- Attachment #1: Type: text/plain, Size: 727 bytes --]

> technically the second table is more space efficient than the first

I was referring to screen space.

> Should Org understand all the 'concatenated text' as a single row (and not as multiple rows)?

In effect, yes. I'm proposing it as a syntax addition to make it easier to read, export, and manage, larger tables.
I'm unsure if this would fit within the scope of org, but [[https://github.com/RedBug312/markdown-it-multimd-table][other projects]],
like [[https://fletcher.github.io/MultiMarkdown-6/MMD_Users_Guide.html#tables][MultiMarkdown]] have exactly what I want in terms of table use.
Since I've already proposed a single, solitary addition to org, we won't discuss all of those nice features /just/ yet!


[-- Attachment #2: Type: text/html, Size: 1094 bytes --]

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Daniele Nicolodi]

On 18/03/2021 14:38, Atlas Cove wrote:
>> technically the second table is more space efficient than the first
> 
> I was referring to screen space.

Assuming that you are looking at this with a monospaced font (and I
don't see how you could use a variable width font to look at an Org
table effectively), screen space is proportional to the number of
characters.

Cheers,
Dan

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Maxim Nikulin]

On 18/03/2021 03:29, Atlas Cove wrote:
> 
> I'd like to propose an addition to the table syntax that would allow for 
> text wrapping in tables, I personally have myself managing
> very large org tables, and having to scroll through them is often 
> cumbersome. As a result, I often yearn for greater control over how I 
> format my tables.

https://orgmode.org/worg/org-faq.html#table-multiline-fields
Org-mode Frequently Asked Questions

> Will there ever be support for multiple lines in a table field?
> 
> No.
> 
> You can embed tables created with the table.el package in org-mode
> buffers, with mixed success when it comes to export and publishing.
Notice however

mid:87czz4ielc.fsf@gnu.org
https://orgmode.org/list/87czz4ielc.fsf@gnu.org/
On 21/12/2020 00:15, Bastien wrote:
> Would it be so bad if org-mode decides to stop supporting table.el tables?
> 
> I don't see the benefit of supporting both Org tables and tables.el tables,
> and it calls for confusion.

There is org-table-wrap-region function that could help to type text
occupying several cell, but the result will not become single cell 
during export.

I have seen several threads related to multiline cells, e.g.

https://orgmode.org/list/f2e9aaacd51ee95dcc696ed9485741dde1067340.camel@sindominio.net/T/#u
org-cell in org-table with a list or a new paragraph

or proposal of another extension:

https://orgmode.org/list/87k0v361x9.fsf@gmail.com
Tables: missing multi-col/row syntax

As to MultiMarkdown, the link that you posted says

> Cell content must be on one line only

Personally I do not mind to have paragraph-like cells, but I am afraid 
that it requires a lot of work to support export, feeding src blocks, 
alignment.

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Timothy]

Hi!

Interesting suggestion you have here.
On a related note, I wonder if you might have seen this thread I raised
a while ago: https://orgmode.org/list/87k0v361x9.fsf@gmail.com/
The discussion has died down (unfortunately), but the idea is still on
my mind.

--
Timothy

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Atlas Cove]

On 18/03/2021 14:26, Timothy wrote:
>Interesting suggestion you have here.
>On a related note, I wonder if you might have seen this thread I raised
>a while ago: https://orgmode.org/list/87k0v361x9.fsf@gmail.com/
>The discussion has died down (unfortunately), but the idea is still on
>my mind.
>
>--
>Timothy

I'd like to propose to fold my discussion into yours.

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Juan Manuel Macías]

Atlas Cove <Atlas48@gmx.com> writes:

> In effect, yes. I'm proposing it as a syntax addition to make it easier to read, export, and manage, larger tables.
> I'm unsure if this would fit within the scope of org, but [[https://github.com/RedBug312/markdown-it-multimd-table][other projects]],
> like [[https://fletcher.github.io/MultiMarkdown-6/MMD_Users_Guide.html#tables][MultiMarkdown]] have exactly what I want in terms of table use.
> Since I've already proposed a single, solitary addition to org, we won't discuss all of those nice features /just/ yet!

Thanks for your clarification. I strongly support your proposal.

Best regards,

Juan Manuel 

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Tim Cross]

Atlas Cove <Atlas48@gmx.com> writes:

> On 18/03/2021 14:26, Timothy wrote:
>>Interesting suggestion you have here.
>>On a related note, I wonder if you might have seen this thread I raised
>>a while ago: https://orgmode.org/list/87k0v361x9.fsf@gmail.com/
>>The discussion has died down (unfortunately), but the idea is still on
>>my mind.
>>
>>--
>>Timothy
>
> I'd like to propose to fold my discussion into yours.

In principal, it wold be great to be able to support multi-row columns.
However, I'm not sure how easily this can actually be implemented in a
consistent and maintainable manner.

From watching these discussions in the past, I think the big stumbling
block is how easily multi-row columns can be added and maintained in the
various export formats. Some are easy, like HTML, but others are less
so. In particular, I know from my many years working with Latex,
multi-row columns are not straight-forward. There are lots of edge cases
to deal with and it is hard to get a consistent result programatically. 

Proposals like this one can seem simple and straight-forward on the
surface. However, implementation is another matter. All of the exporters
will need to be updated to handle this new syntax and it will probably
take a fair bit of work to handle it correctly in just plain org files
(formatting, highlighting etc).

If this was something people were prepared to put the time into
implementing, I think it must be done in a completely separate feature
branch and would need to be fully tested (including all back end
exporters) before it can be merged into the mainline branch. It would
also be important to profile and ensure it does not have significant
impact on performance.

I am a little concerned about the expansion and addition of features in
org mode when there seems to already be a challenge with respect to
maintenance and bug fixing. Personally, I would prefer an org mode which
is consistent and reliable over one with large numbers of features that
is less stable and slower.

-- 
Tim Cross

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Juan Manuel Macías]

Tim Cross <theophilusx@gmail.com> writes:

> From watching these discussions in the past, I think the big stumbling
> block is how easily multi-row columns can be added and maintained in the
> various export formats. Some are easy, like HTML, but others are less
> so. In particular, I know from my many years working with Latex,
> multi-row columns are not straight-forward. There are lots of edge cases
> to deal with and it is hard to get a consistent result programatically. 
>
> Proposals like this one can seem simple and straight-forward on the
> surface. However, implementation is another matter. All of the exporters
> will need to be updated to handle this new syntax and it will probably
> take a fair bit of work to handle it correctly in just plain org files
> (formatting, highlighting etc).

I don't know if anyone has come up with this other possibility: if the
problem is (usually) the inconvenience of editing cells with a lot of
text within an Org document (since when exporting to LaTeX for example,
it makes no difference whether the cell content is on a single line, if
the tabularx package is used properly), then the possibility of editing
a cell in a dedicated buffer would be very practical here. A kind of
`org-edit-special' for cells... That could be combined with
`org-table-toggle-column-width'.

Best regards,

Juan Manuel 

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Timothy]

Tim Cross <theophilusx@gmail.com> writes:

> In principal, it wold be great to be able to support multi-row columns.
> However, I'm not sure how easily this can actually be implemented in a
> consistent and maintainable manner.

Mmmm, this of feels like something where you'll quickly learn how hard
it is/isn't when you try to implement it.

> From watching these discussions in the past, I think the big stumbling
> block is how easily multi-row columns can be added and maintained in the
> various export formats. Some are easy, like HTML, but others are less
> so. In particular, I know from my many years working with Latex,
> multi-row columns are not straight-forward. There are lots of edge cases
> to deal with and it is hard to get a consistent result programatically.
>
> Proposals like this one can seem simple and straight-forward on the
> surface. However, implementation is another matter. All of the exporters
> will need to be updated to handle this new syntax and it will probably
> take a fair bit of work to handle it correctly in just plain org files
> (formatting, highlighting etc).

Currently if you were to try this content with the proposed syntax,
content is just put in the top left cell of the group. This seems like a
reasonable fallback to me. Then for HTML we have colspan/rowspan, and
for LaTeX there's \multicolumn and with the multirow package \multirow.

FWIW just formatting would need to be updated for Org files.
Highlighting is fine as is.

> If this was something people were prepared to put the time into
> implementing, I think it must be done in a completely separate feature
> branch and would need to be fully tested (including all back end
> exporters) before it can be merged into the mainline branch. It would
> also be important to profile and ensure it does not have significant
> impact on performance.
>
> I am a little concerned about the expansion and addition of features in
> org mode when there seems to already be a challenge with respect to
> maintenance and bug fixing. Personally, I would prefer an org mode which
> is consistent and reliable over one with large numbers of features that
> is less stable and slower.

I appreciate this concern, but I do think that the ability to have
multi-col/row cells is invaluable in many large tables, and so would be
a very good addition to Org.

--
Timothy

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Tim Cross]

Timothy <tecosaur@gmail.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> In principal, it wold be great to be able to support multi-row columns.
>> However, I'm not sure how easily this can actually be implemented in a
>> consistent and maintainable manner.
>
> Mmmm, this of feels like something where you'll quickly learn how hard
> it is/isn't when you try to implement it.
>
>> From watching these discussions in the past, I think the big stumbling
>> block is how easily multi-row columns can be added and maintained in the
>> various export formats. Some are easy, like HTML, but others are less
>> so. In particular, I know from my many years working with Latex,
>> multi-row columns are not straight-forward. There are lots of edge cases
>> to deal with and it is hard to get a consistent result programatically.
>>
>> Proposals like this one can seem simple and straight-forward on the
>> surface. However, implementation is another matter. All of the exporters
>> will need to be updated to handle this new syntax and it will probably
>> take a fair bit of work to handle it correctly in just plain org files
>> (formatting, highlighting etc).
>
> Currently if you were to try this content with the proposed syntax,
> content is just put in the top left cell of the group. This seems like a
> reasonable fallback to me. Then for HTML we have colspan/rowspan, and
> for LaTeX there's \multicolumn and with the multirow package \multirow.
>
> FWIW just formatting would need to be updated for Org files.
> Highlighting is fine as is.
>
>> If this was something people were prepared to put the time into
>> implementing, I think it must be done in a completely separate feature
>> branch and would need to be fully tested (including all back end
>> exporters) before it can be merged into the mainline branch. It would
>> also be important to profile and ensure it does not have significant
>> impact on performance.
>>
>> I am a little concerned about the expansion and addition of features in
>> org mode when there seems to already be a challenge with respect to
>> maintenance and bug fixing. Personally, I would prefer an org mode which
>> is consistent and reliable over one with large numbers of features that
>> is less stable and slower.
>
> I appreciate this concern, but I do think that the ability to have
> multi-col/row cells is invaluable in many large tables, and so would be
> a very good addition to Org.

We can debate how easy or hard this is to implement indefinitely. What
is needed is for someone to implement a working version which is
consistent, reliable and provides expected results across all export
environments.

The devil is in the details and I suspect that once you start trying to
implement the feature is when many of those challenges become clear.

My view is 'go for it'. Just create a new feature branch and implment
the functionality in that branch. We can then try using it and see where
it works and where it doesn't. Once this is done, a call can be made as
to whether it should be implemented in the main code base
-- 
Tim Cross

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Timothy]

Tim Cross <theophilusx@gmail.com> writes:

> My view is 'go for it'. Just create a new feature branch and implment
> the functionality in that branch. We can then try using it and see where
> it works and where it doesn't. Once this is done, a call can be made as
> to whether it should be implemented in the main code base

I'm all for this approach, it will just be quite a while till I have the
time free to do this. In addition to the 2-3 currently unresolved
patches from me on the mailing list I already have another 11 patches
lined up (see https://tecosaur.com/public/todo.html#patches-from-my)
among other things 😅.

--
Timothy

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1322 bytes --]

On Thu, Mar 18, 2021 at 11:41:11PM +0100, Juan Manuel Macías wrote:
> Tim Cross <theophilusx@gmail.com> writes:
> 
> > From watching these discussions in the past, I think the big stumbling
> > block is how easily multi-row columns can be added and maintained in the
> > various export formats [...]

> I don't know if anyone has come up with this other possibility: if the
> problem is (usually) the inconvenience of editing cells with a lot of
> text within an Org document (since when exporting to LaTeX [...]

I think this point is very important: "multi-row" is ambiguous.

Does it mean the rendering in Org?  In someother output format? Are the
row breaks chosen by the user (and thus they have /some/ "meaning") or
are they chosen by some rendering mechanism (word wrap, perhaps
hyphenation)?

> the possibility of editing a cell in a dedicated buffer would be very
> practical here.

If the aim is "just rendering in Org" and "breaks have no special
meaning" (so each render is allowed to re-flow), then your approach
makes sense (I think Org table has something like that: you can
set the column width and shrink/expand the column appropriately.
Personally, I've found that somewhat awkward, so I don't use it,
but OTOH I'm not a heavy table user: perhaps that's why).

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Juan Manuel Macías]

Hi Tomas,

<tomas@tuxteam.de> writes:

> If the aim is "just rendering in Org" and "breaks have no special
> meaning" (so each render is allowed to re-flow), then your approach
> makes sense (I think Org table has something like that: you can
> set the column width and shrink/expand the column appropriately.
> Personally, I've found that somewhat awkward, so I don't use it,
> but OTOH I'm not a heavy table user: perhaps that's why).

I was writing yesterday some rudimentary code (just a proof of concept)
and I've made this screencast:

https://lunotipia.juanmanuelmacias.com/edit-cell-sample-2021-03-19_09.29.17.mp4

Best regards,

Juan Manuel

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[tomas]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

On Fri, Mar 19, 2021 at 09:44:48AM +0100, Juan Manuel Macías wrote:
> Hi Tomas,

[...]

> I was writing yesterday some rudimentary code (just a proof of concept)
> and I've made this screencast:
> 
> https://lunotipia.juanmanuelmacias.com/edit-cell-sample-2021-03-19_09.29.17.mp4

Looks nice. And for a heavy table user, it might be useful.

It seems you are in the "table row is a big unstructured text" camp.

For my case, as I said, I'm not a heavy table user anyway.
Opening another window is (to me, at least) so disruptive
that I'd be motivated to find alternatives (I very rarely
use C-c ', org-edit-special for source blocks for the same
reason).

But I can envision people needing that badly.

My post was rather a warning that "multi-line" will mean
different things to different people.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Juan Manuel Macías]

tomas@tuxteam.de writes:

> My post was rather a warning that "multi-line" will mean
> different things to different people.

Yes I agree. It is not the same as how a table is represented in Org and
how it should be rendered when exported to some typographic format like
(let's say) LaTeX. I think also 'multi-line' is a very imprecise concept
for Org tables. If we are talking about a cell whose content is a
paragraph, then for LaTeX it is no problem, with the help of the
tabularx package (in these contexts it is the best solution).

The problem here (IMHO) is a problem of discomfort of edit within Org,
when the cell has a lot of text and when we are under the 'tyranny' of
the single line. Luckily it is not a very extended issue, and the tables
in Org are much easier to view and edit than in LaTeX code, when it
comes to simple cells. What happens is that the LaTeX's `tabular(x)'
environment encompasses many more (typographic) contexts than the
concept of `table' in Org. It is a translation problem between Org and
LaTeX and vice versa...

Best regards,

Juan Manuel 

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[tomas]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On Fri, Mar 19, 2021 at 10:22:20AM +0100, Juan Manuel Macías wrote:
> tomas@tuxteam.de writes:
> 
> > My post was rather a warning that "multi-line" will mean
> > different things to different people.
> 
> Yes I agree. It is not the same as how a table is represented in Org and
> how it should be rendered when exported to some typographic format like
> (let's say) LaTeX [...]

Exactly :-)

> The problem here (IMHO) is a problem of discomfort of edit within Org,
> when the cell has a lot of text and when we are under the 'tyranny' of
> the single line.

I understood that this is your focus. And it's IMHO a good idea to
separate things and solve one at a time.

For export formats, there could be (yet another) "signal" to convey
"new paragraph here" which then is the exporter's job. Perhaps an
empty line (as is traditional in TeX) or something else.

But that's another building site :)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Juan Manuel Macías]

tomas@tuxteam.de writes:

> For export formats, there could be (yet another) "signal" to convey
> "new paragraph here" which then is the exporter's job. Perhaps an
> empty line (as is traditional in TeX) or something else.

I think that signal would be a good solution within a single line. For
example, for an emergency a simple macro would suffice, something as:

#+MACRO: par @@latex:\par{}@@

Anyway, I suspect that Org tables are not originally intended for such a
'literary' content :-) ... LaTeX tabular(x) environment and Org tables
have in common that they are plain text, but that’s where the
similarities end. Org tables are visual (close to the WYSIWYG concept),
which is why they are wonderful when it comes to mere data tables (and n
files x n rows). The LaTeX tabular environment is crude, it is not
'visual' (except for the facilities provided by the editor), and when
the table becomes large and complex it can be a torture working in
there... So perhaps (IMHO) the solution to edit a cell in a dedicated
buffer could be the least traumatic in complex cells, in case one wants
to avoid going crazy by editing this kind of huge and literary tables in
LaTeX ;-)

Best regards,

Juan Manuel 

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Juan Manuel Macías]

* sorry for the typo. I meant "...n columns x n rows..." 

Juan Manuel Macías <maciaschain@posteo.net> writes:

> Anyway, I suspect that Org tables are not originally intended for such a
> 'literary' content :-) ... LaTeX tabular(x) environment and Org tables
> have in common that they are plain text, but that’s where the
> similarities end. Org tables are visual (close to the WYSIWYG concept),
> which is why they are wonderful when it comes to mere data tables (and n
> files x n rows). The LaTeX tabular environment is crude, it is not
> 'visual' (except for the facilities provided by the editor), and when
> the table becomes large and complex it can be a torture working in
> there... So perhaps (IMHO) the solution to edit a cell in a dedicated
> buffer could be the least traumatic in complex cells, in case one wants
> to avoid going crazy by editing this kind of huge and literary tables in
> LaTeX ;-)

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Eric S Fraga]

On Friday, 19 Mar 2021 at 08:58, Tim Cross wrote:
> I am a little concerned about the expansion and addition of features in
> org mode when there seems to already be a challenge with respect to
> maintenance and bug fixing. Personally, I would prefer an org mode which
> is consistent and reliable over one with large numbers of features that
> is less stable and slower.

+1 on this.

With respect to the topic at hand, I believe it's the result of the same
tendency that Excel users have of using spreadsheets (aka tables) for
everything, something I hate when I'm given some Excel sheet that I need
to modify and where entries are huge paragraphs.  The UI in Excel is
horrible for these types of tables.  I would hate to see org go in the
same direction.

In many cases, I believe that a simple text based database format would
be more appropriate.  I wonder if the time would be better spent
providing native support for GNU recutils [1] in org instead.  It could
even have a table like export...

Just my 2¢.  Ignore as you see fit!

eric

Footnotes:
[1]  https://www.gnu.org/software/recutils/

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.4-254-g37749c

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1126 bytes --]

On Fri, Mar 19, 2021 at 11:53:14AM +0100, Juan Manuel Macías wrote:
> tomas@tuxteam.de writes:
> 
> > For export formats, there could be (yet another) "signal" to convey
> > "new paragraph here" which then is the exporter's job. Perhaps an
> > empty line (as is traditional in TeX) or something else.
> 
> I think that signal would be a good solution within a single line. For
> example, for an emergency a simple macro would suffice, something as:
> 
> #+MACRO: par @@latex:\par{}@@

Yes, something like that. My point was that it probably makes sense to
separate concerns here.

> Anyway, I suspect that Org tables are not originally intended for such a
> 'literary' content :-) ...

That's right. OTOH, people will try to stretch it in every conceivable
direction. That's in a way Org's biggest strength (and at the same time
its biggest weakness :)

> LaTeX tabular(x) environment and Org tables
> have in common that they are plain text, but that’s where the
> similarities end [...]

Rigth. (La)TeX is a typesetting language. Cool things can be done,
but they have a cost.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Juan Manuel Macías]

tomas@tuxteam.de writes:

> [...] My point was that it probably makes sense to separate concerns
> here.

Yes, that separation is essential. I agree.

> That's right. OTOH, people will try to stretch it in every conceivable
> direction. That's in a way Org's biggest strength (and at the same time
> its biggest weakness :)

Great truth! I always try to take things to the extreme :-D, and I
continue thinking that Org is (among many other things) an excellent
'interface' for LaTeX (maybe the best). But I have to admit that a table
with a lot of complexity is a special case when maybe you have to write
LaTeX code, and get lost between all those & and backslashes, at least
if the goal is the typographic refinement. The greatest power of Org
resides in his lightness and in its chameleon versatility, but his worst
temptation may be to become in a (poor) LaTeX 'translation'.

Best regards,

Juan Manuel 

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Tim Cross]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Friday, 19 Mar 2021 at 08:58, Tim Cross wrote:
>> I am a little concerned about the expansion and addition of features in
>> org mode when there seems to already be a challenge with respect to
>> maintenance and bug fixing. Personally, I would prefer an org mode which
>> is consistent and reliable over one with large numbers of features that
>> is less stable and slower.
>
> +1 on this.
>
> With respect to the topic at hand, I believe it's the result of the same
> tendency that Excel users have of using spreadsheets (aka tables) for
> everything, something I hate when I'm given some Excel sheet that I need
> to modify and where entries are huge paragraphs.  The UI in Excel is
> horrible for these types of tables.  I would hate to see org go in the
> same direction.
>

Those excel 'workbooks' are the absolute worst. I sometimes wonder if
project managers would be more popular if they hadn't decided to use
that as their primary tool. The UI to navigate through those
spreadsheets is horrible.  

The other problem is that tables with large cells of 'paragraph' like
data are not terribly nice from an accessibility perspective. They can
be difficult to navagate and things like screen readers can find it
difficult to 'render' the content in a sensible manner i.e. where the
spoken representation 'makes sense'.

It has also been stated that the Latex exporter won't be a problem as
tabularx (and other Latex packages) will just handle this. Sadly, I don't think it is
that simple. I have used that package a lot over the years and there
have been times I've had to render tables with multiple columns and
rows. While the various latex table like packages do provide a much
better outcome for tables with more complex cell structure, it is rare
that you don't need to do a fair amount of tweaking to actually get good
looking complex tables. Handling a cell with a couple of lines may be
fine, but it is very easy to hit the limits of what the package can
handle without some tweaking. This is where I think things begin to fall
down. My suspicion is that the amount of work needed to make the Latex
exporter handle the majority of common use cases for this new syntax is
much larger and more complex than it seems and getting this to work
reliably and consistently will be extremely difficult. 

We can be pretty certain that if we introduce some additional syntax to
allow for multi-line cells and perhaps multi-row+multi-column cells, if
the exporters do not render good looking output for every use case, bugs
will be reported. I also have no idea how this will impact some of the
other exporters, such as odt, texinfo etc. 

I do think we can probably add additional functionality to make working
with more complex tables in org files somewhat easier from an editing
perspective. I do have some concern over the potential performance
impact, but in general, feel this is probably the easier part of the
challenge. 

> In many cases, I believe that a simple text based database format would
> be more appropriate.  I wonder if the time would be better spent
> providing native support for GNU recutils [1] in org instead.  It could
> even have a table like export...
>

That would be an interesting exercise. However, it does add another
dependency and in some ways breaks the 'everything as text' philosophy
(though I guess the last 'rendering' in the org file is still all text). 

To some extent, discussions like this remind me of the early days of the
web where tables were used as the primary formatting 'tool'. Remember
those horrible web pages which consisted of deeply nested tables? We
learnt pretty quickly what a bad idea that turned out to be.

I wonder if part of the issue here is with the different characteristics
of displays and formatted or printed output. Displays have the property
of being 'infinite' - a row can be as long as you like and a column can
be as high as you like. With the growth in large screens many users now
have windows far wider than the old 80 characters that were standard in
old terminals. This tends to make tables a more appealing 'layout'
facility as you can have a number of columns with multiple rows fitting
on the screen. However, now you have a new problem - how do you map this
very wide structure to a fixed width, relatively narrow, sheet of
paper? 

The advantage of 80 characters was that generally, you could fit that on
a standard sheet of paper. It is also a good width for reading, allowing
you to take in multiple 'blocks' of text at a time. Really wide text can
look ok, but many find such wide text more difficult to read as you have
to scan long lines and cannot easily take in a 'block' at a time.

I think there is probably some very good reasons multi-row cells were
not supported in org mode initially. I suspect the reasons are not
necessarily obvious until you try to implement such support. I could
also be completely wrong and the issues I've seen in dealing with such
tables in Latex are due solely to me not using the available facilities
correctly. For this reason and because this question does seem to come
up fairly regularly, my recommendation is to create a new feature branch
and let those who are interested in supporting this proposal work on it.
I would encourage them to keep good notes (and include them as a file in
the feature branch) so that we have a record of the process. Once it
becomes mature enough and works with all the bundled exporters, the rest
of us can try it out and provide additional testing, verification of
backwards compatibility and assessment of any performance implications.
If it all goes well, then we could consider merging it into the main
branch. If it fails to reach maturity or creates problems or for some
other reason cannot be brought into the main branch, we would at least
have this as a record and perhaps a starting point for the next time
someone raises the question of multi-line table cells. 

-- 
Tim Cross

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Eric S Fraga]

On Saturday, 20 Mar 2021 at 08:33, Tim Cross wrote:
> That would be an interesting exercise. However, it does add another
> dependency and in some ways breaks the 'everything as text' philosophy
> (though I guess the last 'rendering' in the org file is still all text). 

Yes, very true.  So let me rephrase: I wonder if there is another
formatting approach, not table based, that would suit what is
essentially categories and sub-categories of information, one that would
export nicely/easily to a number of targets?  Probably not.

In practice, I find that proper use of subheadings and description lists
end up being enough for most of my uses.  When something more
challenging arises, I have been known to resort to drawers with
processing via one of the hooks and special blocks (minipages, for
instance), but only for LaTeX export.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.4-254-g37749c

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Juan Manuel Macías]

Tim Cross <theophilusx@gmail.com> writes:

> It has also been stated that the Latex exporter won't be a problem as
> tabularx (and other Latex packages) will just handle this. Sadly, I
> don't think it is that simple. I have used that package a lot over the
> years and there have been times I've had to render tables with
> multiple columns and rows. While the various latex table like packages
> do provide a much better outcome for tables with more complex cell
> structure, it is rare that you don't need to do a fair amount of
> tweaking to actually get good looking complex tables. Handling a cell
> with a couple of lines may be fine, but it is very easy to hit the
> limits of what the package can handle without some tweaking. This is
> where I think things begin to fall down. My suspicion is that the
> amount of work needed to make the Latex exporter handle the majority
> of common use cases for this new syntax is much larger and more
> complex than it seems and getting this to work reliably and
> consistently will be extremely difficult.

tabularx does a fairly acceptable job out of the box on tables whose
cells have complex or 'multi-line' content, such as paragraphs. But,
naturally, it all depends on the context and scenarios: table, page
dimensions, book type and many more factors. The typography, when
demanding, is the art of exceptions. And I think this is valid even in
such an 'automated' tool like LaTeX (fortunately, LaTeX and TeX are also
very ductile, if you know the code well).

In any case, I agree that it would not be a good idea to translate the
LaTeX tabular environment, with all its nuances, to Org tables. In
addition to the fact that Org would be burdened with a unnecessary
complexity, is the fact that the syntax of both types of tables differ
essentially. Org tables have a more visual character; the tabular
environment is a pure "what you see is what you mean".

In a tabular environment I see no problem in writing something like
this row (ugly, but acceptable) in 3 lines:

#+begin_src latex
  lorem & Lorem ipsum dolor sit amet, consectetuer adipiscing elit.
  Donec hendrerit tempor tellus. Donec pretium posuere tellus. Proin
  quam nisl, tincidunt et, mattis eget, convallis nec, purus & ipsum \\
#+end_src

But in an Org table I don't see how it could be done without messing up
the spirit of its syntax ... that's why I think (IMHO) the least
traumatic option, in cases where cells have 'a lot of content', is the
possibility of edit cells in a dedicated buffer, like this proof of
concept that I did:

https://lunotipia.juanmanuelmacias.com/edit-cell-sample-2021-03-19_09.29.17.mp4

And if the table is very complex then I don't see any other option than
writing directly the LaTeX code (in case the intended target is only
LaTeX/PDF).

Best regards,

Juan Manuel 

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Andreas Eder]

On Fr 19 Mär 2021 at 13:33, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:

> With respect to the topic at hand, I believe it's the result of the same
> tendency that Excel users have of using spreadsheets (aka tables) for
> everything, something I hate when I'm given some Excel sheet that I need
> to modify and where entries are huge paragraphs.  The UI in Excel is
> horrible for these types of tables.  I would hate to see org go in the
> same direction.

+1 for that
Unfortunately I have to deal with this type of people at work.

> In many cases, I believe that a simple text based database format would
> be more appropriate.  I wonder if the time would be better spent
> providing native support for GNU recutils [1] in org instead.

recutils has an emacs mode an I think it even has some sort of org-mode
integration.

'Andreas

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Tim Cross]

Andreas Eder <a_eder_muc@web.de> writes:

> On Fr 19 Mär 2021 at 13:33, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
>
>> With respect to the topic at hand, I believe it's the result of the same
>> tendency that Excel users have of using spreadsheets (aka tables) for
>> everything, something I hate when I'm given some Excel sheet that I need
>> to modify and where entries are huge paragraphs.  The UI in Excel is
>> horrible for these types of tables.  I would hate to see org go in the
>> same direction.
>
> +1 for that
> Unfortunately I have to deal with this type of people at work.
>

In my world, I suspect project managers would be liked a lot more if
they had never discovered excel!

Funny thing, survived for years without the scourge of excel, but
noticed it creeping in more around 2007 or so, then by 2015 it was
everywhere - project plans, requirement specifications, software testing
and even software/API design documentation.

I didn't mind when the core of the document was still about linked cells
with calculated values which would change based on changes in various
cell entries - that is what spreadsheets are for. However, often now,
I'm presented with a spreadsheet (or workbook) with not a single
calculation or cell formula - it is just about formatting. In many ways,
workbooks are being treated like a poor persons slow inefficient
database with a really poor interface.

More than once, I've taken these spreadsheets and extracted the data
into an org file with headings, sub-headings, TODOs and lists, possibly
with a couple of clock tables and been far better off. In one project,
we were really lucky because there was multiple Emacs users and we were
able to get org used as the primary data format for managing the project
(we even used taskjuggler to generate the diagrams needed for steering
committee). Unfortunately, that is a very rare situation and most of the
time, we are forced to use excel simply because it is a format available
to everyone on every platform.

This I think is the key problem - MS Office and excel in particular have
become ubiquitous and now people think the way information is presented
in spreadsheets is the 'default' or acceptable way to format data (i.e.
tables with large columns cntaining paragraphs of data). IMO tables are
really only a good presentation format when you can see the whole table
in a page or screen. Once you need to navigate around large cells,
columns, rows, it is really just about navigation around the data and I
find is often less efficient or useful and navigation around a well
structured org file. Where excel has us beat is in sharing of
information and enabling others to add/update the data. Everyone has
access to excel or excel compatible software, hardly anyone uses org (as
a proportion of total users, doesn't mean org doesn't have a lot of users). 

-- 
Tim Cross

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Samuel Wales]

i like that.  such an org edit special type feature could efven in
principle work for a column or a row all at once.


On 3/18/21, Juan Manuel Macías <maciaschain@posteo.net> wrote:
> Tim Cross <theophilusx@gmail.com> writes:
>
>> From watching these discussions in the past, I think the big stumbling
>> block is how easily multi-row columns can be added and maintained in the
>> various export formats. Some are easy, like HTML, but others are less
>> so. In particular, I know from my many years working with Latex,
>> multi-row columns are not straight-forward. There are lots of edge cases
>> to deal with and it is hard to get a consistent result programatically.
>>
>> Proposals like this one can seem simple and straight-forward on the
>> surface. However, implementation is another matter. All of the exporters
>> will need to be updated to handle this new syntax and it will probably
>> take a fair bit of work to handle it correctly in just plain org files
>> (formatting, highlighting etc).
>
> I don't know if anyone has come up with this other possibility: if the
> problem is (usually) the inconvenience of editing cells with a lot of
> text within an Org document (since when exporting to LaTeX for example,
> it makes no difference whether the cell content is on a single line, if
> the tabularx package is used properly), then the possibility of editing
> a cell in a dedicated buffer would be very practical here. A kind of
> `org-edit-special' for cells... That could be combined with
> `org-table-toggle-column-width'.
>
> Best regards,
>
> Juan Manuel
>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Juan Manuel Macías]

Hi Samuel,

Samuel Wales <samologist@gmail.com> writes:

> i like that.  such an org edit special type feature could efven in
> principle work for a column or a row all at once.

I like the idea of being able to edit rows or columns as well...

As I mentioned in a previous message, I made this rudimentary proof of
concept (only with cells):
https://lunotipia.juanmanuelmacias.com/edit-cell-sample-2021-03-19_09.29.17.mp4

The idea is that the content of the cell would be 'filled' while it is in
the edit buffer and, back to the cell, it would be a single line again.

Certain marks (in my sample it is a simple macro (nl = latex \newline)
would cause a line break in the edit buffer, to facilitate editing.

Best regards,

Juan Manuel 

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Eric S Fraga]

On Sunday, 21 Mar 2021 at 09:06, Tim Cross wrote:
> However, often now, I'm presented with a spreadsheet (or workbook)
> with not a single calculation or cell formula -it is just about
> formatting. 

Exactly.  And this is why I am not happy about org being modified to
enable this.  Make the path of least resistance the most attractive:

> In many ways, workbooks are being treated like a poor persons slow
> inefficient database with a really poor interface.

And also why I suggested that it would be good to have some type of
database functionality in org.  Well, we do already in the form of
properties.  I use these, with column view, all the time, but maybe we
need to think about how such are exported and encourage its use?

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.4-254-g37749c

Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

[Tom Gillespie]

Tim,
   Your concerns are well founded. Essentially any cosmetic extension
to the org table syntax will be an unmaintainable, bug ridden
nightmare and would be an eternal burden on any attempts to formalize
correct behavior. I have a draft of a grammar for a significant
portion of Org syntax (forthcoming), and getting tables right was
quite hard, despite their seeming simplicity. Any surface syntax level
additions would require updating an unknown number of parts of the
codebase to accommodate the change, since every part of org implements
its own parser (e.g. there is already a bug in the interaction between
tables and #+macro: commands containing the pipe character that
org-export implements incorrectly). Relatively speaking, the potential
approaches discussed in Timothy's previous proposal are much more
likely to be tractable. Best,
Tom