[PATCH] doc/org-manual.org: Extend table formulas Lisp form documentation

[PATCH] doc/org-manual.org: Extend table formulas Lisp form documentation

[Daniele Nicolodi]

[-- Attachment #1: Type: text/plain, Size: 283 bytes --]

Hello,

I always found the description of Lisp forms in Org table formulas not
extremely clear, especially in regard to the use of mode flags. The
attached patch tries to clarify the manual a bit.

Would it be worth to mention org-sbe in the same section of the manual?

Cheers,
Dan

[-- Attachment #2: 0001-doc-org-manual.org-Extend-table-formulas-Lisp-form-d.patch --]
[-- Type: text/plain, Size: 4108 bytes --]

From aad5ba2217c30ff219069afbfdd8c65851f899d2 Mon Sep 17 00:00:00 2001
From: Daniele Nicolodi <daniele@grinta.net>
Date: Sat, 14 Nov 2020 18:33:41 +0100
Subject: [PATCH] doc/org-manual.org: Extend table formulas Lisp form
 documentation

Be more explicit about how fields are interpolated into the Lisp
forms, clarify the use of mode flags, and add a cuple more examples.
---
 doc/org-manual.org | 60 +++++++++++++++++++++++++++-------------------
 1 file changed, 36 insertions(+), 24 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 040fccc21..208e53023 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -2165,38 +2165,50 @@ It is also possible to write a formula in Emacs Lisp.  This can be
 useful for string manipulation and control structures, if Calc's
 functionality is not enough.
 
-If a formula starts with a single-quote followed by an opening
-parenthesis, then it is evaluated as a Lisp form.  The evaluation
-should return either a string or a number.  Just as with Calc
-formulas, you can specify modes and a ~printf~ format after
-a semicolon.
+A formula is evaluated as a Lisp form when it starts with a
+single-quote followed by an opening parenthesis.  Cell table
+references are interpolated into the Lisp form before execution.  The
+evaluation should return either a string or a number.  Evaluation
+modes and a ~printf~ format used to render the returned values can be
+specified after a semicolon.
 
-With Emacs Lisp forms, you need to be conscious about the way field
-references are interpolated into the form.  By default, a reference is
-interpolated as a Lisp string (in double-quotes) containing the field.
-If you provide the =N= mode switch, all referenced elements are
-numbers---non-number fields will be zero---and interpolated as Lisp
-numbers, without quotes.  If you provide the =L= flag, all fields are
-interpolated literally, without quotes.  For example, if you want a
-reference to be interpreted as a string by the Lisp form, enclose the
-reference operator itself in double-quotes, like ="$3"=.  Ranges are
-inserted as space-separated fields, so you can embed them in list or
-vector syntax.
+By default, references are interpolated as literal Lisp strings: the
+field content is replaced in the Lisp form stripped of leading and
+trailing white space and surrounded in double-quotes.  For example:
 
-Here are a few examples---note how the =N= mode is used when we do
-computations in Lisp:
+: '(concat $1 $2)
 
-- ='(concat (substring $1 1 2) (substring $1 0 1) (substring $1 2))= ::
+concatenates the content of columns 1 and column 2.
 
-  Swap the first two characters of the content of column 1.
+When the =N= flag is used, all referenced elements are parsed as
+numbers and interpolated as Lisp numbers, without quotes. Fields that
+cannot be parsed as numbers are interpolated as zeros.  For example:
 
-- ='(+ $1 $2);N= ::
+: '(+ $1 $2);N
 
-  Add columns 1 and 2, equivalent to Calc's =$1+$2=.
+adds columns 1 and 2, equivalent to Calc's =$1+$2=.  Ranges are
+inserted as space-separated fields, so they can be embedded in list or
+vector syntax. For example:
 
-- ='(apply '+ '($1..$4));N= ::
+: '(apply '+ '($1..$4));N
 
-  Compute the sum of columns 1 to 4, like Calc's =vsum($1..$4)=.
+computes the sum of columns 1 to 4, like Calc's =vsum($1..$4)=.
+
+When the =L= flag is used, all fields are interpolated literally: the
+cell content is replaced in the Lisp form stripped of leading and
+trailing white space and without quotes.  If a reference is intended
+to be interpreted as a string by the Lisp form, the reference operator
+itself should be enclosed in double-quotes, like ="$3"=.  The =L= flag
+is useful when strings and numbers are used in the same Lisp form.  For
+example:
+
+: '(substring "$1" $2 $3);L
+
+extracts the part of the string in column 1 between the character
+positions specified in the integers in column 2 and 3 and it is easier
+to read than the equivalent:
+
+: '(substring $1 (string-to-number $2) (string-to-number $3))
 
 *** Durations and time values
 :PROPERTIES:
-- 
2.29.2

Re: [PATCH] doc/org-manual.org: Extend table formulas Lisp form documentation

[Eric S Fraga]

Daniele,

this looks good.  One minor pedantic point: I think you mean
"interpreted" when you say "interpolated" (several times in the
text).  Otherwise, this is a very useful addition to the manual.

Thank you,
eric

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-107-ga0aaca.dirty

Re: [PATCH] doc/org-manual.org: Extend table formulas Lisp form documentation

[Daniele Nicolodi]

On 16/11/2020 11:25, Eric S Fraga wrote:
> Daniele,
> 
> this looks good.  One minor pedantic point: I think you mean
> "interpreted" when you say "interpolated" (several times in the
> text).  Otherwise, this is a very useful addition to the manual.

Thank you for reading and for the comment.

"interpolated" looks strange to me in this context too, but it is the
word that is currently used in the manual. I decided to stick to this
term for consistency, however, I haven't check if it is used with the
same meaning elsewhere.

I don't think it is wrong to use "interpolated", but if you thing it
should be changed I can change it and check the manual for consistency.
However, I don't think "interpreted" is the right word either. Probably
"replaced" or "substituted" are better choices in this context.

Cheers,
Dan

Re: [PATCH] doc/org-manual.org: Extend table formulas Lisp form documentation

[Tim Cross]

Daniele Nicolodi <daniele@grinta.net> writes:

> On 16/11/2020 11:25, Eric S Fraga wrote:
>> Daniele,
>>
>> this looks good.  One minor pedantic point: I think you mean
>> "interpreted" when you say "interpolated" (several times in the
>> text).  Otherwise, this is a very useful addition to the manual.
>
> Thank you for reading and for the comment.
>
> "interpolated" looks strange to me in this context too, but it is the
> word that is currently used in the manual. I decided to stick to this
> term for consistency, however, I haven't check if it is used with the
> same meaning elsewhere.
>
> I don't think it is wrong to use "interpolated", but if you thing it
> should be changed I can change it and check the manual for consistency.
> However, I don't think "interpreted" is the right word either. Probably
> "replaced" or "substituted" are better choices in this context.
>

I agree. Interpolated is consistent with manuals for other programming
languages which have similar functionality. However, org is also used by
a more diverse community than typical programming languages, so perhaps
'replaced' or 'substituted' would be a better choice?

--
Tim Cross

Re: [PATCH] doc/org-manual.org: Extend table formulas Lisp form documentation

[TEC]

I have 2c on the use of "interpolated".

1. I tend to think of "interpolated" in terms of it's mathematical
   meaning
2. The other denotations relate to insertion and renewing, which 
simply
   doesn't fit.

I appreciate that other people may have used this too, but as I 
see it
that just means that other people have engaged in strange word 
choices.

Suggested alternatives: Substituted, transpiled, or translated.

Timothy.

-----

For context, here's the definition, etymology, and symonyms.

Definition
  Intransitive Verb
​​​​   1. To renew; to carry on with intermission. [Obs.] 
​​​​   2. To alter or corrupt by the insertion of new or foreign 
      matter; especially, to change, as a book or text, by the
      insertion of matter that is new, or foreign to the purpose
      of the author.
​​​​   3. (Mathematics) To fill up intermediate terms of, as of a 
series, 
      according to the law of the series; to introduce, as a
      number or quantity, in a partial series, according to the
      law of that part of the series.
  Adjective
​​​​   1. Inserted in, or added to, the original; introduced; 
      foisted in; changed by the insertion of new or spurious
      matter.

​​​​   2. (Math.) 
​​​​​      (a) Provided with necessary interpolations; as, an 
          interpolated table.
​​​​​      (b) Introduced or determined by interpolation; as, 
          interpolated quantities or numbers.

​​Etymology
 
​​​interpolate verb 

1610s, "to alter or enlarge (a writing) by inserting new 
material," from Latin 
interpolatus, past participle of interpolare "alter, freshen up, 
polish;" of 
writing, "falsify," from inter "among, between" (see inter-) + 
polare, which is 
related to polire "to smoothe, polish," from PIE root *pel- ( 5) 
"to thrust, 
strike, drive," the connecting notion being "to full cloth" 
[Watkins].

Sense evolved in Latin from "refurbish," to "alter appearance of," 
to "falsify 
(especially by adding new material)." Middle English had 
interpolen (early 15c.) 
in a similar sense. Related: Interpolated; interpolating.

​​Synonyms
 
​​​verb adjective 
​​​​1. Insert (wrongfully),  foist in.
​​​​2. (Math .) Introduce, intercalate (terms to complete a series).


Tim Cross <theophilusx@gmail.com> writes:

> Daniele Nicolodi <daniele@grinta.net> writes:
>
>> On 16/11/2020 11:25, Eric S Fraga wrote:
>>> Daniele,
>>>
>>> this looks good.  One minor pedantic point: I think you mean
>>> "interpreted" when you say "interpolated" (several times in 
>>> the
>>> text).  Otherwise, this is a very useful addition to the 
>>> manual.
>>
>> Thank you for reading and for the comment.
>>
>> "interpolated" looks strange to me in this context too, but it 
>> is the
>> word that is currently used in the manual. I decided to stick 
>> to this
>> term for consistency, however, I haven't check if it is used 
>> with the
>> same meaning elsewhere.
>>
>> I don't think it is wrong to use "interpolated", but if you 
>> thing it
>> should be changed I can change it and check the manual for 
>> consistency.
>> However, I don't think "interpreted" is the right word either. 
>> Probably
>> "replaced" or "substituted" are better choices in this context.
>>
>
> I agree. Interpolated is consistent with manuals for other 
> programming
> languages which have similar functionality. However, org is also 
> used by
> a more diverse community than typical programming languages, so 
> perhaps
> 'replaced' or 'substituted' would be a better choice?

Re: [PATCH] doc/org-manual.org: Extend table formulas Lisp form documentation

[Charles Millar]

On 11/18/20 2:42 PM, TEC wrote:
> 
> I have 2c on the use of "interpolated".
> 
> 1. I tend to think of "interpolated" in terms of it's mathematical
>    meaning
> 2. The other denotations relate to insertion and renewing, which simply
>    doesn't fit.
> 
> I appreciate that other people may have used this too, but as I see it
> that just means that other people have engaged in strange word choices.
> 
> Suggested alternatives: Substituted, transpiled, or translated.
> 
> Timothy.
> 
> -----
> 
> For context, here's the definition, etymology, and symonyms.
> 
> Definition
>   Intransitive Verb
> ​​​​   1. To renew; to carry on with intermission. [Obs.] 
> ​​​​   2. To alter or corrupt by the insertion of new or foreign 
>       matter; especially, to change, as a book or text, by the
>       insertion of matter that is new, or foreign to the purpose
>       of the author.
> ​​​​   3. (Mathematics) To fill up intermediate terms of, as of a series, 
>       according to the law of the series; to introduce, as a
>       number or quantity, in a partial series, according to the
>       law of that part of the series.
>   Adjective
> ​​​​   1. Inserted in, or added to, the original; introduced; 
>       foisted in; changed by the insertion of new or spurious
>       matter.
> 
> ​​​​   2. (Math.) 
> ​​​​​      (a) Provided with necessary interpolations; as, an 
>           interpolated table.
> ​​​​​      (b) Introduced or determined by interpolation; as, 
>           interpolated quantities or numbers.
> 
> ​​Etymology
>  
> ​​​interpolate verb 
> 
> 1610s, "to alter or enlarge (a writing) by inserting new material," from 
> Latin interpolatus, past participle of interpolare "alter, freshen up, 
> polish;" of writing, "falsify," from inter "among, between" (see inter-) 
> + polare, which is related to polire "to smoothe, polish," from PIE root 
> *pel- ( 5) "to thrust, strike, drive," the connecting notion being "to 
> full cloth" [Watkins].
> 
> Sense evolved in Latin from "refurbish," to "alter appearance of," to 
> "falsify (especially by adding new material)." Middle English had 
> interpolen (early 15c.) in a similar sense. Related: Interpolated; 
> interpolating.
> 
> ​​Synonyms
>  
> ​​​verb adjective 
> ​​​​1. Insert (wrongfully),  foist in.
> ​​​​2. (Math .) Introduce, intercalate (terms to complete a series).
> 
> 
> Tim Cross <theophilusx@gmail.com> writes:
> 
>> Daniele Nicolodi <daniele@grinta.net> writes:
>>
>>> On 16/11/2020 11:25, Eric S Fraga wrote:
>>>> Daniele,
>>>>
>>>> this looks good.  One minor pedantic point: I think you mean
>>>> "interpreted" when you say "interpolated" (several times in the
>>>> text).  Otherwise, this is a very useful addition to the manual.
>>>
>>> Thank you for reading and for the comment.
>>>
>>> "interpolated" looks strange to me in this context too, but it is the
>>> word that is currently used in the manual. I decided to stick to this
>>> term for consistency, however, I haven't check if it is used with the
>>> same meaning elsewhere.
>>>
>>> I don't think it is wrong to use "interpolated", but if you thing it
>>> should be changed I can change it and check the manual for consistency.
>>> However, I don't think "interpreted" is the right word either. Probably
>>> "replaced" or "substituted" are better choices in this context.
>>>
>>
>> I agree. Interpolated is consistent with manuals for other programming
>> languages which have similar functionality. However, org is also used by
>> a more diverse community than typical programming languages, so perhaps
>> 'replaced' or 'substituted' would be a better choice?
> 
> 
> 
Just my 2 cents, not being a programmer so I see one way that 
interpolate may be correct

Are the cell references actually inserted into the lisp form when the 
formula is evaluated Please note that the word "the" is used not an 
indefinite 'a", i.e.

Cell table +references are interpolated into the Lisp form before execution

as opposed to

Cell table
+references are interpolated into a lisp form before execution

Charlie Millar

Re: [PATCH] doc/org-manual.org: Extend table formulas Lisp form documentation

[Kyle Meyer]

Daniele Nicolodi writes:

> Hello,
>
> I always found the description of Lisp forms in Org table formulas not
> extremely clear, especially in regard to the use of mode flags. The
> attached patch tries to clarify the manual a bit.

Thanks.

> Would it be worth to mention org-sbe in the same section of the manual?

Yeah, it looks like there's no mention of org-sbe in the manual, so I
think so (as a separate patch).

> Subject: [PATCH] doc/org-manual.org: Extend table formulas Lisp form
>  documentation
>
> Be more explicit about how fields are interpolated into the Lisp
> forms, clarify the use of mode flags, and add a cuple more examples.

s/cuple/couple/

Typically a manual change will get a "* doc/org-manual.org (<section>):"
entry in the commit message.

> ---
>  doc/org-manual.org | 60 +++++++++++++++++++++++++++-------------------
>  1 file changed, 36 insertions(+), 24 deletions(-)
[...]
> +By default, references are interpolated as literal Lisp strings: the
> +field content is replaced in the Lisp form stripped of leading and
> +trailing white space and surrounded in double-quotes.  For example:
>  
> -Here are a few examples---note how the =N= mode is used when we do
> -computations in Lisp:
> +: '(concat $1 $2)
>  
> -- ='(concat (substring $1 1 2) (substring $1 0 1) (substring $1 2))= ::
> +concatenates the content of columns 1 and column 2.

This and similar spots in this patch produce incorrect indentation in
the info output:

  trailing white space and surrounded in double-quotes.  For example:
  
       '(concat $1 $2)
  
     concatenates the content of columns 1 and column 2.

Adding "#+texinfo: @noindent" above the line would prevent that.  To
check the result, you can generate the info output with `make info' and
then visit it with `C-u C-h i doc/org'.

Aside from that, the changes here look like an improvement to me.  As
far as "interpolated" goes, this patch adds one more instance to a
section that carries 4 of the 5 occurrences in the code base, so I'd say
it's fine to leave as is.  I think the "replaced" or "substituted"
suggestions by Tim Cross are good ones, though, if anyone cares to send
a follow-up patch.

Re: [PATCH] doc/org-manual.org: Extend table formulas Lisp form documentation

[Daniele Nicolodi]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

On 25/11/2020 05:37, Kyle Meyer wrote:
> Daniele Nicolodi writes:
> 
>> Hello,
>>
>> I always found the description of Lisp forms in Org table formulas not
>> extremely clear, especially in regard to the use of mode flags. The
>> attached patch tries to clarify the manual a bit.
> 
> Thanks.

Thank you for the review, Kyle. An updated patch is attached.

>> Would it be worth to mention org-sbe in the same section of the manual?
> 
> Yeah, it looks like there's no mention of org-sbe in the manual, so I
> think so (as a separate patch).

After playing a bit with org-sbe, I came to the conclusion that it is
broken beyond repair, at least without breaking it for the people that
managed to make it work for them.

I think that adding mention of it in the manual and explain all the
quirks of the macro is much more work than replace it with something
better. I tried to write a better macro, please have a look here:

https://orgmode.org/list/d429d29b-42fa-7d7b-6f3a-9fe692fd6dc7@grinta.net/

and the parent message for an explanation of what I think is broken in
org-sbe. Would you support adding org-sbx (for a lack of a better name)
to ob-table.el and mention it in the manual instead of org-sbe? I would
not go as far as deprecating org-sbe, just yet, but maybe soon...

Cheers,
Dan

[-- Attachment #2: 0001-doc-org-manual.org-Extend-table-formulas-Lisp-form-d.patch --]
[-- Type: text/plain, Size: 4248 bytes --]

From d39ec4465605f56d5f53a36faf4e419ae1b862f0 Mon Sep 17 00:00:00 2001
From: Daniele Nicolodi <daniele@grinta.net>
Date: Sat, 14 Nov 2020 18:33:41 +0100
Subject: [PATCH] doc/org-manual.org: Extend table formulas Lisp form
 documentation

doc/org-manual.org (Emacs Lisp forms as formulas): Be more
explicit about how fields are interpolated into the Lisp forms,
clarify the use of mode flags, and add a couple more examples.
---
 doc/org-manual.org | 64 +++++++++++++++++++++++++++++-----------------
 1 file changed, 40 insertions(+), 24 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 2f7f5f847..97018d075 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -2178,38 +2178,54 @@ It is also possible to write a formula in Emacs Lisp.  This can be
 useful for string manipulation and control structures, if Calc's
 functionality is not enough.
 
-If a formula starts with a single-quote followed by an opening
-parenthesis, then it is evaluated as a Lisp form.  The evaluation
-should return either a string or a number.  Just as with Calc
-formulas, you can specify modes and a ~printf~ format after
-a semicolon.
+A formula is evaluated as a Lisp form when it starts with a
+single-quote followed by an opening parenthesis.  Cell table
+references are interpolated into the Lisp form before execution.  The
+evaluation should return either a string or a number.  Evaluation
+modes and a ~printf~ format used to render the returned values can be
+specified after a semicolon.
 
-With Emacs Lisp forms, you need to be conscious about the way field
-references are interpolated into the form.  By default, a reference is
-interpolated as a Lisp string (in double-quotes) containing the field.
-If you provide the =N= mode switch, all referenced elements are
-numbers---non-number fields will be zero---and interpolated as Lisp
-numbers, without quotes.  If you provide the =L= flag, all fields are
-interpolated literally, without quotes.  For example, if you want a
-reference to be interpreted as a string by the Lisp form, enclose the
-reference operator itself in double-quotes, like ="$3"=.  Ranges are
-inserted as space-separated fields, so you can embed them in list or
-vector syntax.
+By default, references are interpolated as literal Lisp strings: the
+field content is replaced in the Lisp form stripped of leading and
+trailing white space and surrounded in double-quotes.  For example:
 
-Here are a few examples---note how the =N= mode is used when we do
-computations in Lisp:
+: '(concat $1 $2)
 
-- ='(concat (substring $1 1 2) (substring $1 0 1) (substring $1 2))= ::
+#+texinfo: @noindent
+concatenates the content of columns 1 and column 2.
+
+When the =N= flag is used, all referenced elements are parsed as
+numbers and interpolated as Lisp numbers, without quotes. Fields that
+cannot be parsed as numbers are interpolated as zeros.  For example:
+
+: '(+ $1 $2);N
+
+#+texinfo: @noindent
+adds columns 1 and 2, equivalent to Calc's =$1+$2=.  Ranges are
+inserted as space-separated fields, so they can be embedded in list or
+vector syntax. For example:
 
-  Swap the first two characters of the content of column 1.
+: '(apply '+ '($1..$4));N
 
-- ='(+ $1 $2);N= ::
+#+texinfo: @noindent
+computes the sum of columns 1 to 4, like Calc's =vsum($1..$4)=.
+
+When the =L= flag is used, all fields are interpolated literally: the
+cell content is replaced in the Lisp form stripped of leading and
+trailing white space and without quotes.  If a reference is intended
+to be interpreted as a string by the Lisp form, the reference operator
+itself should be enclosed in double-quotes, like ="$3"=.  The =L= flag
+is useful when strings and numbers are used in the same Lisp form.  For
+example:
 
-  Add columns 1 and 2, equivalent to Calc's =$1+$2=.
+: '(substring "$1" $2 $3);L
 
-- ='(apply '+ '($1..$4));N= ::
+#+texinfo: @noindent
+extracts the part of the string in column 1 between the character
+positions specified in the integers in column 2 and 3 and it is easier
+to read than the equivalent:
 
-  Compute the sum of columns 1 to 4, like Calc's =vsum($1..$4)=.
+: '(substring $1 (string-to-number $2) (string-to-number $3))
 
 *** Durations and time values
 :PROPERTIES:
-- 
2.29.2

Re: [PATCH] doc/org-manual.org: Extend table formulas Lisp form documentation

[Kyle Meyer]

Daniele Nicolodi writes:

> On 25/11/2020 05:37, Kyle Meyer wrote:
>> Daniele Nicolodi writes:
[...]
>>> Would it be worth to mention org-sbe in the same section of the manual?
>> 
>> Yeah, it looks like there's no mention of org-sbe in the manual, so I
>> think so (as a separate patch).
>
> After playing a bit with org-sbe, I came to the conclusion that it is
> broken beyond repair, at least without breaking it for the people that
> managed to make it work for them.
>
> I think that adding mention of it in the manual and explain all the
> quirks of the macro is much more work than replace it with something
> better.

Okay.  org-sbe isn't something I've ever used, and I only have a
superficial understanding of it.

> I tried to write a better macro, please have a look here:
>
> https://orgmode.org/list/d429d29b-42fa-7d7b-6f3a-9fe692fd6dc7@grinta.net/
>
> and the parent message for an explanation of what I think is broken in
> org-sbe. Would you support adding org-sbx (for a lack of a better name)
> to ob-table.el and mention it in the manual instead of org-sbe? I would
> not go as far as deprecating org-sbe, just yet, but maybe soon...

Sorry, I haven't got around to looking at that thread yet (though
ideally someone that knows more about org-sbe will chime in).

> Subject: [PATCH] doc/org-manual.org: Extend table formulas Lisp form
>  documentation
>
> doc/org-manual.org (Emacs Lisp forms as formulas): Be more
> explicit about how fields are interpolated into the Lisp forms,
> clarify the use of mode flags, and add a couple more examples.

Thanks for the update.  Applied (b2d38a822), adding an asterisk before
"doc/org-manual.org".