how to document/comment a complex org table formula

how to document/comment a complex org table formula

[Uwe Brauer]

Hi 

It turns out that I have to deal with a bit more complex table formula
of the form 

#+begin_src elisp
#+TBLFM: $6=if($2>10,($2-10)*@2$7,0)+ min(10,$2)*@2$2+ min(10,$3)*@2$3 + if($4>10,($4-10)*@2$7,$4)+min(10,$4)*@2$4 +@2$5*$5;f1 
#+end_src

Is there any way to document or comment such a formula, since I am sure
that within days I don't remember why I chose that formula?

Regards

Uwe Brauer  

Re: how to document/comment a complex org table formula

[Eric S Fraga]

Sorry; I cannot help you directly.

My rule of thumb is that if formulas are getting too complex to
understand/recall clearly, it's time to use a proper programming
language instead.  The nice thing about org is you can have tables as
inputs to and outputs of src blocks...

My papers often have awk source blocks that process tables to generate
statistics for some (numerical) experiments.

Spreadsheets, org tables being an example of such, are brilliant tools
for simple calculations but are pretty much "write only programming
languages".

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-577-gf76d4d
: Latest paper written in org: https://arxiv.org/abs/2106.05096

Re: how to document/comment a complex org table formula

[Uwe Brauer]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

>>> "ESF" == Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Sorry; I cannot help you directly.
> My rule of thumb is that if formulas are getting too complex to
> understand/recall clearly, it's time to use a proper programming
> language instead.  The nice thing about org is you can have tables as
> inputs to and outputs of src blocks...

> My papers often have awk source blocks that process tables to generate
> statistics for some (numerical) experiments.

> Spreadsheets, org tables being an example of such, are brilliant tools
> for simple calculations but are pretty much "write only programming
> languages".

I agree, the problem is I have to collaborate with colleagues and they
use excel. I still struggle to understand their formula, so I thought it
would be a good idea to comment the terms I understand but even if I
call

org-edit-special, I cannot add comments, well I can but they are
destroyed when I turn back to the main file. Other tools such as
virtual-comment do not work neither in such a buffer.


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5673 bytes --]

Re: how to document/comment a complex org table formula

[Tim Visher]

[-- Attachment #1: Type: text/plain, Size: 1828 bytes --]

On Mon, Jul 5, 2021 at 10:14 AM Eric S Fraga <e.fraga@ucl.ac.uk> wrote:

> Sorry; I cannot help you directly.
>
> My rule of thumb is that if formulas are getting too complex to
> understand/recall clearly, it's time to use a proper programming
> language instead.  The nice thing about org is you can have tables as
> inputs to and outputs of src blocks...
>
> My papers often have awk source blocks that process tables to generate
> statistics for some (numerical) experiments.
>
> Spreadsheets, org tables being an example of such, are brilliant tools
> for simple calculations but are pretty much "write only programming
> languages".
>

I had never heard of using an org table as input into a source block.
That's really interesting.

The OP also poses an interesting idea that I have occasionally wanted. I
have generally accomplished this by simply including the documentation
above or below the table. In other words just having human text around it
that says 'This =formula= bit is because of X'.

But the idea of a multiline TBLFM syntax seems to be already be almost
supported. I was mildly shocked to find that this mostly works

```
|---+---|
| a | 1 |
| b | 3 |
|---+---|
|   | 4 |
#+TBLFM: @2$2=@-1*3
#+TBLFM: @3$2=vsum(@I..II)
```

Note that I don't need the usual `::` separation between formulas. Where it
breaks down is that I can't seem to reevaluate the whole table's formula by
whacking `C-u C-c C-c` anymore and, obviously, there's no syntax for adding
comments. Also I doubt that I'd be able to use any of org's keys for
editing formulas. I wonder how hard it would be to extend org's
understanding of the TBLFM to allow for this kind of syntax.

Once you had it then adding a comment character to it should be very simple.

--

In Christ,

Timmy V.

https://blog.twonegatives.com
http://five.sentenc.es

[-- Attachment #2: Type: text/html, Size: 2645 bytes --]

Re: how to document/comment a complex org table formula

[Jude DaShiell]

Why not use functional narrative form to document complex formulas?
formula name:
formula function:
formula uses:
formula returns:
usually one line of text for each of those.  I know, this comes from
structured programming where no block of actual code is supposed to go
above 50 lines and that for many people cramps their styles.  It makes
maintenance a little easier though.


On Mon, 5 Jul 2021, Tim Visher wrote:

> On Mon, Jul 5, 2021 at 10:14 AM Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
>
> > Sorry; I cannot help you directly.
> >
> > My rule of thumb is that if formulas are getting too complex to
> > understand/recall clearly, it's time to use a proper programming
> > language instead.  The nice thing about org is you can have tables as
> > inputs to and outputs of src blocks...
> >
> > My papers often have awk source blocks that process tables to generate
> > statistics for some (numerical) experiments.
> >
> > Spreadsheets, org tables being an example of such, are brilliant tools
> > for simple calculations but are pretty much "write only programming
> > languages".
> >
>
> I had never heard of using an org table as input into a source block.
> That's really interesting.
>
> The OP also poses an interesting idea that I have occasionally wanted. I
> have generally accomplished this by simply including the documentation
> above or below the table. In other words just having human text around it
> that says 'This =formula= bit is because of X'.
>
> But the idea of a multiline TBLFM syntax seems to be already be almost
> supported. I was mildly shocked to find that this mostly works
>
> ```
> |---+---|
> | a | 1 |
> | b | 3 |
> |---+---|
> |   | 4 |
> #+TBLFM: @2$2=@-1*3
> #+TBLFM: @3$2=vsum(@I..II)
> ```
>
> Note that I don't need the usual `::` separation between formulas. Where it
> breaks down is that I can't seem to reevaluate the whole table's formula by
> whacking `C-u C-c C-c` anymore and, obviously, there's no syntax for adding
> comments. Also I doubt that I'd be able to use any of org's keys for
> editing formulas. I wonder how hard it would be to extend org's
> understanding of the TBLFM to allow for this kind of syntax.
>
> Once you had it then adding a comment character to it should be very simple.
>
> --
>
> In Christ,
>
> Timmy V.
>
> https://blog.twonegatives.com
> http://five.sentenc.es
>

Re: how to document/comment a complex org table formula

[Uwe Brauer]

[-- Attachment #1: Type: text/plain, Size: 680 bytes --]

>>> "JD" == Jude DaShiell <jdashiel@panix.com> writes:

> Why not use functional narrative form to document complex formulas?
> formula name:
> formula function:
> formula uses:
> formula returns:


That would be also nice, however the problem I am facing to have a long
and complicated expression with a lot of ifs, maybe best practise would
be to split it and distribute it over various column and add comments as
in 

#+begin_src elisp
|----+----+----+---+---+----|
|  1 |  4 |  5 | 3 | 3 |  6 |
|  3 | 12 | 15 | 9 | 2 | 11 |
| 12 |  2 | 14 | 7 | 3 | 10 |
#+TBLFM: $3=vsum($1..$2);f1:: #one comment
#+TBLFM: $6=vsum($4..$5);f1:: #second comment
#+end_src

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5673 bytes --]

Re: how to document/comment a complex org table formula

[Jude DaShiell]

So long as each piece of documentation has a continuation pointer to the
next piece of documentation in the chain I think that would work.


On Mon, 5 Jul 2021, Uwe Brauer wrote:

> >>> "JD" == Jude DaShiell <jdashiel@panix.com> writes:
>
> > Why not use functional narrative form to document complex formulas?
> > formula name:
> > formula function:
> > formula uses:
> > formula returns:
>
>
> That would be also nice, however the problem I am facing to have a long
> and complicated expression with a lot of ifs, maybe best practise would
> be to split it and distribute it over various column and add comments as
> in
>
> #+begin_src elisp
> |----+----+----+---+---+----|
> |  1 |  4 |  5 | 3 | 3 |  6 |
> |  3 | 12 | 15 | 9 | 2 | 11 |
> | 12 |  2 | 14 | 7 | 3 | 10 |
> #+TBLFM: $3=vsum($1..$2);f1:: #one comment
> #+TBLFM: $6=vsum($4..$5);f1:: #second comment
> #+end_src
>

Re: how to document/comment a complex org table formula

[Eric S Fraga]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

On Monday,  5 Jul 2021 at 10:55, Tim Visher wrote:
> I had never heard of using an org table as input into a source block.
> That's really interesting.

Attached, for illustration, is a simple example, extracted from a paper
I'm currently writing.  The "input" to the awk script is the table at
the top of the document.  The output, two columns of numbers, get
converted to an org table, at the bottom of the document, automatically.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-577-gf76d4d
: Latest paper written in org: https://arxiv.org/abs/2106.05096

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: example.org --]
[-- Type: text/x-org, Size: 1082 bytes --]


#+name: values
|-----------------------|
|  0.006780639787578286 |
|  0.006780639787578433 |
|  0.008195683980621288 |
|  0.006780639787578433 |
|  0.006780639787578286 |
|  0.006780639787578286 |
|  0.008195683980621288 |
|  0.006780639787578433 |
|  0.006780639787578286 |
| 0.0071466305079598775 |
|   0.01154503021335327 |
|  0.006780639787578433 |
|  0.010129986020310415 |
|   0.00819568398062114 |
|  0.006780639787578433 |
| 0.0071466305079598775 |
| 0.0071466305079598775 |
|-----------------------|

The table of values above include cases where rounding errors lead to slightly different results for what are essentially the same result.  The following awk script looks at only the significant digits to identify similar results:

#+name: frequency
#+begin_src awk :stdin values
  {
      if ($1 != "hline") {
          s = substr($1,2,9)
          n[s] += 1;
      }
  }
  END {
      for (val in n) {
          printf "%7.5f %d\n", val, n[val]
      }
  }
#+end_src

#+results: frequency
|  0.0082 | 3 |
| 0.00678 | 9 |
| 0.01013 | 1 |
| 0.00715 | 3 |
| 0.01154 | 1 |

Re: how to document/comment a complex org table formula

[Uwe Brauer]

[-- Attachment #1: Type: text/plain, Size: 2439 bytes --]

>>> "TV" == Tim Visher <tim.visher@gmail.com> writes:

> On Mon, Jul 5, 2021 at 10:14 AM Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
>> Sorry; I cannot help you directly.
>> 
>> My rule of thumb is that if formulas are getting too complex to
>> understand/recall clearly, it's time to use a proper programming
>> language instead.  The nice thing about org is you can have tables as
>> inputs to and outputs of src blocks...
>> 
>> My papers often have awk source blocks that process tables to generate
>> statistics for some (numerical) experiments.
>> 
>> Spreadsheets, org tables being an example of such, are brilliant tools
>> for simple calculations but are pretty much "write only programming
>> languages".
>> 

> I had never heard of using an org table as input into a source block.
> That's really interesting.

> The OP also poses an interesting idea that I have occasionally wanted. I
> have generally accomplished this by simply including the documentation
> above or below the table. In other words just having human text around it
> that says 'This =formula= bit is because of X'.

> But the idea of a multiline TBLFM syntax seems to be already be almost
> supported. I was mildly shocked to find that this mostly works

> ```
> |---+---|
> | a | 1 |
> | b | 3 |
> |---+---|
> |   | 4 |

> #+TBLFM: @2$2=@-1*3
> #+TBLFM: @3$2=vsum(@I..II)

> ```

> Note that I don't need the usual `::` separation between formulas. Where it
> breaks down is that I can't seem to reevaluate the whole table's formula by
> whacking `C-u C-c C-c` anymore 

You have to put the cursor on the beginning of each #+TBLFM
and hit C-u C-c C-c

> and, obviously, there's no syntax for adding
> comments. Also I doubt that I'd be able to use any of org's keys for
> editing formulas. I wonder how hard it would be to extend org's
> understanding of the TBLFM to allow for this kind of syntax.

Well the following  works!

#+begin_src elisp
|----+----+----+---+---+----|
|  1 |  4 |  5 | 3 | 3 |  6 |
|  3 | 12 | 15 | 9 | 2 | 11 |
| 12 |  2 | 14 | 7 | 3 | 10 |
#+TBLFM: $3=vsum($1..$2);f1:: #one comment
#+TBLFM: $6=vsum($4..$5);f1:: #second comment
#+end_src


> Once you had it then adding a comment character to it should be very simple.

I hoped to add comments to the formula editor, but that most likely
requires text-properties or overlays or something of that sort.

Regards


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5673 bytes --]

Re: how to document/comment a complex org table formula

[Karl Voit]

Hi Uwe,

* Uwe Brauer <oub@mat.ucm.es> wrote:
>
> Is there any way to document or comment such a formula, since I am sure
> that within days I don't remember why I chose that formula?

This is not a solution for really complex formulas. However, please
do note that standard org has:

Named references: https://orgmode.org/manual/References.html
Name assignment:
https://orgmode.org/manual/Advanced-features.html#Advanced-features

For some cases, this may help.

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: how to document/comment a complex org table formula

[Uwe Brauer]

[-- Attachment #1: Type: text/plain, Size: 600 bytes --]

>>> "KV" == Karl Voit <devnull@Karl-Voit.at> writes:

Hi Karl,


> Hi Uwe,
> * Uwe Brauer <oub@mat.ucm.es> wrote:
>> 
>> Is there any way to document or comment such a formula, since I am sure
>> that within days I don't remember why I chose that formula?

> This is not a solution for really complex formulas. However, please
> do note that standard org has:

> Named references: https://orgmode.org/manual/References.html
> Name assignment:
> https://orgmode.org/manual/Advanced-features.html#Advanced-features

Thanks, right. That is indeed helpful but not what I really need. 

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5673 bytes --]

how to document/comment a complex org table formula

[CHARPENTIER Emmanuel via General discussions about Org-mode.]

Limbic answer : 
  - create an emacs-lisp function doing what you want
  - comment /ad libitum/
  - call this function in your spreadsheet.

HTH,

Re: how to document/comment a complex org table formula

[Uwe Brauer]

[-- Attachment #1: Type: text/plain, Size: 371 bytes --]

>>> "GdaO" == General discussions about Org-mode <CHARPENTIER> writes:

> Limbic answer : 
>   - create an emacs-lisp function doing what you want
>   - comment /ad libitum/
>   - call this function in your spreadsheet.

I hoped such a function already existed and somebody would point it out
to me (I searched the package-list-package list but nothing came up)

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5673 bytes --]

Re: how to document/comment a complex org table formula

[Nick Dokos]

Uwe Brauer <oub@mat.ucm.es> writes:

>>>> "GdaO" == General discussions about Org-mode <CHARPENTIER> writes:
>
>> Limbic answer : 
>>   - create an emacs-lisp function doing what you want
>>   - comment /ad libitum/
>>   - call this function in your spreadsheet.
>
> I hoped such a function already existed and somebody would point it out
> to me (I searched the package-list-package list but nothing came up)
>

IIUC, the function would replace the formula, so it's really the
formula in a different guise, similar to what Eric F. was talking
about: implementing the formula as a source block. Both of them put
the code in some other place, where there is space to add comments.
IOW, it's not a general-purpose function. But I may be misconstruing
what Emmanuel was suggesting.

-- 
Nick

"There are only two hard problems in computer science: cache
invalidation, naming things, and off-by-one errors." -Martin Fowler