Adding new table rows/cols in a formula update

Adding new table rows/cols in a formula update

[Dima Kogan]

Hi.

Suppose I have this .org file:

 |   |
 #+TBLFM: @1$2=5

It's a 1x1 table with a formula. The formula sets a cell that's out of
bounds in the table, so evaluating this formula results in an error. How
set-in-stone is this behavior? I haven't dug too deeply into the code,
but are there fundamental assumptions here? Would a patch that extends
the table before applying such a formula be too naive in some way?

Thanks!

dima

Re: Adding new table rows/cols in a formula update

[Dima Kogan]

[-- Attachment #1: Type: text/plain, Size: 563 bytes --]

Dima Kogan <lists@dima.secretsauce.net> writes:

> Suppose I have this .org file:
>
>  |   |
>  #+TBLFM: @1$2=5
>
> It's a 1x1 table with a formula. The formula sets a cell that's out of
> bounds in the table, so evaluating this formula results in an error. How
> set-in-stone is this behavior? I haven't dug too deeply into the code,
> but are there fundamental assumptions here? Would a patch that extends
> the table before applying such a formula be too naive in some way?

Here's a tiny patch that adds the columns (not rows) as needed. Is this
reasonable?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-table-field-formulas-can-now-create-new-columns-.patch --]
[-- Type: text/x-diff, Size: 830 bytes --]

From 93e9927dd49d100036853963e899c8b6af5325de Mon Sep 17 00:00:00 2001
From: Dima Kogan <dima@secretsauce.net>
Date: Tue, 30 Sep 2014 12:27:26 -0700
Subject: [PATCH] org-table: field formulas can now create new columns as
 needed

---
 lisp/org-table.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/org-table.el b/lisp/org-table.el
index 7607ead..31365ad 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -3125,7 +3125,7 @@ known that the table will be realigned a little later anyway."
       (while (setq eq (pop eqlname1))
 	(message "Re-applying formula to field: %s" (car eq))
 	(org-goto-line (nth 1 eq))
-	(org-table-goto-column (nth 2 eq))
+	(org-table-goto-column (nth 2 eq) nil 'force)
 	(org-table-eval-formula nil (nth 3 eq) 'noalign 'nocst
 				'nostore 'noanalysis))
 
-- 
2.0.0

Re: Adding new table rows/cols in a formula update

[Subhan Michael Tindall]

I would include a customization variable to control this behavior, defaulting to whatever the current behavior is:
IE:
(setq org-calc-extend-file nil) default system behavior
(setq org-calc-extend-file t) always silently extend rows
(setq org-calc-extend-file "warn") issue warning in message buffer that line was extended
(setq org-calc-extend-file "prompt") prompt user y/n on whether or not to extend column



> -----Original Message-----
> From: emacs-orgmode-bounces+subhant=familycareinc.org@gnu.org
> [mailto:emacs-orgmode-bounces+subhant=familycareinc.org@gnu.org] On
> Behalf Of Dima Kogan
> Sent: Tuesday, September 30, 2014 12:28 PM
> To: emacs-orgmode@gnu.org
> Subject: Re: [O] Adding new table rows/cols in a formula update
> 
> Dima Kogan <lists@dima.secretsauce.net> writes:
> 
> > Suppose I have this .org file:
> >
> >  |   |
> >  #+TBLFM: @1$2=5
> >
> > It's a 1x1 table with a formula. The formula sets a cell that's out of
> > bounds in the table, so evaluating this formula results in an error.
> > How set-in-stone is this behavior? I haven't dug too deeply into the
> > code, but are there fundamental assumptions here? Would a patch that
> > extends the table before applying such a formula be too naive in some
> way?
> 
> Here's a tiny patch that adds the columns (not rows) as needed. Is this
> reasonable?


This message is intended for the sole use of the individual and entity to which it is addressed and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the intended addressee, nor authorized to receive for the intended addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email and delete the message.  Thank you.

Re: Adding new table rows/cols in a formula update

[Dima Kogan]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

>> Dima Kogan <lists@dima.secretsauce.net> writes:
>> 
>> > Suppose I have this .org file:
>> >
>> >  |   |
>> >  #+TBLFM: @1$2=5
>> >
>> > It's a 1x1 table with a formula. The formula sets a cell that's out of
>> > bounds in the table, so evaluating this formula results in an error.
>> > How set-in-stone is this behavior? I haven't dug too deeply into the
>> > code, but are there fundamental assumptions here? Would a patch that
>> > extends the table before applying such a formula be too naive in some
>> way?
>
> Subhan Michael Tindall <SubhanT@familycareinc.org> writes:
>
> I would include a customization variable to control this behavior, defaulting to whatever the current behavior is:
> IE:
> (setq org-calc-extend-file nil) default system behavior
> (setq org-calc-extend-file t) always silently extend rows
> (setq org-calc-extend-file "warn") issue warning in message buffer that line was extended
> (setq org-calc-extend-file "prompt") prompt user y/n on whether or not to extend column

OK. Patch attached.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Field-formulas-can-now-add-columns-as-needed.patch --]
[-- Type: text/x-diff, Size: 2313 bytes --]

From 3b6581c647cb87f0d3e8cee94ce2fb1fb122d3fd Mon Sep 17 00:00:00 2001
From: Dima Kogan <dima@secretsauce.net>
Date: Tue, 30 Sep 2014 22:36:21 -0700
Subject: [PATCH] Field formulas can now add columns as needed

The org-table-formula-make-new-cols customization controls whether and how this
is done
---
 lisp/org-table.el | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/lisp/org-table.el b/lisp/org-table.el
index 7607ead..f2933ed 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -354,6 +354,16 @@ portability of tables."
 	  (const :tag "Stick to hline" nil)
 	  (const :tag "Error on attempt to cross" error)))
 
+(defcustom org-table-formula-make-new-cols nil
+  "Non-nil means that evaluation of a field formula can add new
+columns if an out-of-bounds field is being set."
+  :group 'org-table-calculation
+  :type '(choice
+	  (const :tag "Setting an out-of-bounds field generates an error (default)" nil)
+	  (const :tag "Setting an out-of-bounds field silently adds columns as needed" t)
+	  (const :tag "Setting an out-of-bounds field adds columns as needed, but issues a warning message" warn)
+	  (const :tag "When setting an out-of-bounds field, the user is prompted" prompt)))
+
 (defgroup org-table-import-export nil
   "Options concerning table import and export in Org-mode."
   :tag "Org Table Import Export"
@@ -3125,7 +3135,22 @@ known that the table will be realigned a little later anyway."
       (while (setq eq (pop eqlname1))
 	(message "Re-applying formula to field: %s" (car eq))
 	(org-goto-line (nth 1 eq))
-	(org-table-goto-column (nth 2 eq))
+	(let* ((column-target (nth 2 eq))
+	       (column-count (progn (end-of-line)
+				    (1- (org-table-current-column))))
+	       (create-new-column
+		(and (> column-target column-count)
+		     (or (eq org-table-formula-make-new-cols t)
+			 (and
+			  (eq org-table-formula-make-new-cols 'warn)
+			  (progn
+			    (org-display-warning "Out-of-bounds formula added columns")
+			    t))
+			 (and
+			  (eq org-table-formula-make-new-cols 'prompt)
+			  (yes-or-no-p "Out-of-bounds formula. Add columns?"))))))
+	  (org-table-goto-column column-target nil create-new-column))
+
 	(org-table-eval-formula nil (nth 3 eq) 'noalign 'nocst
 				'nostore 'noanalysis))
 
-- 
2.0.0

Re: Adding new table rows/cols in a formula update

[Nicolas Goaziou]

Hello,

Dima Kogan <dima@secretsauce.net> writes:

> OK. Patch attached.

Thanks for your patch. Some comments follow.

> From 3b6581c647cb87f0d3e8cee94ce2fb1fb122d3fd Mon Sep 17 00:00:00 2001
> From: Dima Kogan <dima@secretsauce.net>
> Date: Tue, 30 Sep 2014 22:36:21 -0700
> Subject: [PATCH] Field formulas can now add columns as needed
>
> The org-table-formula-make-new-cols customization controls whether and how this
> is done

Your commit message is missing information and "TINYCHANGE" at its end.
See http://orgmode.org/worg/org-contribute.html#sec-5

> +(defcustom org-table-formula-make-new-cols nil

What about `org-table-formula-create-columns'?

> +  "Non-nil means that evaluation of a field formula can add new
> +columns if an out-of-bounds field is being set."

First line needs to be complete. E.g.,

  "Non-nil means a field formula can create a new column."

> +  :group 'org-table-calculation
> +  :type '(choice
> +	  (const :tag "Setting an out-of-bounds field generates an error (default)" nil)
> +	  (const :tag "Setting an out-of-bounds field silently adds columns as needed" t)
> +	  (const :tag "Setting an out-of-bounds field adds columns as needed, but issues a warning message" warn)
> +	  (const :tag "When setting an out-of-bounds field, the user is
> prompted" prompt)))

You need to add :version and :package-version keywords.

>  (defgroup org-table-import-export nil
>    "Options concerning table import and export in Org-mode."
>    :tag "Org Table Import Export"
> @@ -3125,7 +3135,22 @@ known that the table will be realigned a little later anyway."
>        (while (setq eq (pop eqlname1))
>  	(message "Re-applying formula to field: %s" (car eq))
>  	(org-goto-line (nth 1 eq))
> -	(org-table-goto-column (nth 2 eq))
> +	(let* ((column-target (nth 2 eq))
> +	       (column-count (progn (end-of-line)
> +				    (1- (org-table-current-column))))
> +	       (create-new-column
> +		(and (> column-target column-count)

Is this check really necessary? Doesn't `org-table-goto-column' already
figures it out before creating a new column?


Regards,

-- 
Nicolas Goaziou

Re: Adding new table rows/cols in a formula update

[Dima Kogan]

[-- Attachment #1: Type: text/plain, Size: 2823 bytes --]

Hi. Thanks for replying.


Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Thanks for your patch. Some comments follow.
>
>> From 3b6581c647cb87f0d3e8cee94ce2fb1fb122d3fd Mon Sep 17 00:00:00 2001
>> From: Dima Kogan <dima@secretsauce.net>
>> Date: Tue, 30 Sep 2014 22:36:21 -0700
>> Subject: [PATCH] Field formulas can now add columns as needed
>>
>> The org-table-formula-make-new-cols customization controls whether and how this
>> is done
>
> Your commit message is missing information and "TINYCHANGE" at its end.
> See http://orgmode.org/worg/org-contribute.html#sec-5

OK. Added. By the way, I signed the copyright-assignment paperwork for
Emacs itself. Would I need to do this again for org-mode?



>> +(defcustom org-table-formula-make-new-cols nil
>
> What about `org-table-formula-create-columns'?

Sure, why not.



>> +  "Non-nil means that evaluation of a field formula can add new
>> +columns if an out-of-bounds field is being set."
>
> First line needs to be complete. E.g.,
>
>   "Non-nil means a field formula can create a new column."
>
>> +  :group 'org-table-calculation
>> +  :type '(choice
>> +	  (const :tag "Setting an out-of-bounds field generates an error (default)" nil)
>> +	  (const :tag "Setting an out-of-bounds field silently adds columns as needed" t)
>> +	  (const :tag "Setting an out-of-bounds field adds columns as needed, but issues a warning message" warn)
>> +	  (const :tag "When setting an out-of-bounds field, the user is
>> prompted" prompt)))
>
> You need to add :version and :package-version keywords.

OK. I'm not sure what those values should be, so please double-check.


>>  (defgroup org-table-import-export nil
>>    "Options concerning table import and export in Org-mode."
>>    :tag "Org Table Import Export"
>> @@ -3125,7 +3135,22 @@ known that the table will be realigned a little later anyway."
>>        (while (setq eq (pop eqlname1))
>>  	(message "Re-applying formula to field: %s" (car eq))
>>  	(org-goto-line (nth 1 eq))
>> -	(org-table-goto-column (nth 2 eq))
>> +	(let* ((column-target (nth 2 eq))
>> +	       (column-count (progn (end-of-line)
>> +				    (1- (org-table-current-column))))
>> +	       (create-new-column
>> +		(and (> column-target column-count)
>
> Is this check really necessary? Doesn't `org-table-goto-column' already
> figures it out before creating a new column?

This is necessary if we want to be able to warn the user or to prompt
them ONLY if a new column has to be made. I.e. if we're not looking past
the bounds of the table the user should never be pestered. If the
customization variable is only t/nil then this can be dramatically
simplified, as you have observed.

By the way, is (org-display-warning) the preferred way to produce a
warning? It does into a new buffer, not into *Messages*.

Tweaked patch attached.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-table.el-field-formulas-can-now-create-columns-a.patch --]
[-- Type: text/x-diff, Size: 2479 bytes --]

From 7b3ab8eeffb2047b966c624707766ec29a416583 Mon Sep 17 00:00:00 2001
From: Dima Kogan <dima@secretsauce.net>
Date: Tue, 30 Sep 2014 22:36:21 -0700
Subject: [PATCH] org-table.el: field formulas can now create columns as needed

(org-table-recalculate): use the new org-table-formula-make-new-cols
customization to control whether org creates new columns when a
formula explicitly targets them

TINYCHANGE
---
 lisp/org-table.el | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lisp/org-table.el b/lisp/org-table.el
index 7607ead..090bb75 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -354,6 +354,18 @@ portability of tables."
 	  (const :tag "Stick to hline" nil)
 	  (const :tag "Error on attempt to cross" error)))
 
+(defcustom org-table-formula-create-columns nil
+  "Non-nil means that evaluation of a field formula can add new
+columns if an out-of-bounds field is being set."
+  :group 'org-table-calculation
+  :version "24.5"
+  :package-version '(Org . "8.3")
+  :type '(choice
+	  (const :tag "Setting an out-of-bounds field generates an error (default)" nil)
+	  (const :tag "Setting an out-of-bounds field silently adds columns as needed" t)
+	  (const :tag "Setting an out-of-bounds field adds columns as needed, but issues a warning message" warn)
+	  (const :tag "When setting an out-of-bounds field, the user is prompted" prompt)))
+
 (defgroup org-table-import-export nil
   "Options concerning table import and export in Org-mode."
   :tag "Org Table Import Export"
@@ -3125,7 +3137,22 @@ known that the table will be realigned a little later anyway."
       (while (setq eq (pop eqlname1))
 	(message "Re-applying formula to field: %s" (car eq))
 	(org-goto-line (nth 1 eq))
-	(org-table-goto-column (nth 2 eq))
+	(let* ((column-target (nth 2 eq))
+	       (column-count (progn (end-of-line)
+				    (1- (org-table-current-column))))
+	       (create-new-column
+		(and (> column-target column-count)
+		     (or (eq org-table-formula-create-columns t)
+			 (and
+			  (eq org-table-formula-create-columns 'warn)
+			  (progn
+			    (org-display-warning "Out-of-bounds formula added columns")
+			    t))
+			 (and
+			  (eq org-table-formula-create-columns 'prompt)
+			  (yes-or-no-p "Out-of-bounds formula. Add columns?"))))))
+	  (org-table-goto-column column-target nil create-new-column))
+
 	(org-table-eval-formula nil (nth 3 eq) 'noalign 'nocst
 				'nostore 'noanalysis))
 
-- 
2.0.0

Re: Adding new table rows/cols in a formula update

[Nick Dokos]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Dima Kogan <dima@secretsauce.net> writes:
>
>> OK. Patch attached.
>
> Thanks for your patch. Some comments follow.
>
>> From 3b6581c647cb87f0d3e8cee94ce2fb1fb122d3fd Mon Sep 17 00:00:00 2001
>> From: Dima Kogan <dima@secretsauce.net>
>> Date: Tue, 30 Sep 2014 22:36:21 -0700
>> Subject: [PATCH] Field formulas can now add columns as needed
>>
>> The org-table-formula-make-new-cols customization controls whether and how this
>> is done
>
> Your commit message is missing information and "TINYCHANGE" at its end.
> See http://orgmode.org/worg/org-contribute.html#sec-5
>
>> +(defcustom org-table-formula-make-new-cols nil
>
> What about `org-table-formula-create-columns'?
>
>> +  "Non-nil means that evaluation of a field formula can add new
>> +columns if an out-of-bounds field is being set."
>
> First line needs to be complete. E.g.,
>
>   "Non-nil means a field formula can create a new column."
>
>> +  :group 'org-table-calculation
>> +  :type '(choice
>> +	  (const :tag "Setting an out-of-bounds field generates an error (default)" nil)
>> +	  (const :tag "Setting an out-of-bounds field silently adds columns as needed" t)
>> +	  (const :tag "Setting an out-of-bounds field adds columns as needed, but issues a warning message" warn)
>> +	  (const :tag "When setting an out-of-bounds field, the user is
>> prompted" prompt)))
>
> You need to add :version and :package-version keywords.
>
>>  (defgroup org-table-import-export nil
>>    "Options concerning table import and export in Org-mode."
>>    :tag "Org Table Import Export"
>> @@ -3125,7 +3135,22 @@ known that the table will be realigned a little later anyway."
>>        (while (setq eq (pop eqlname1))
>>  	(message "Re-applying formula to field: %s" (car eq))
>>  	(org-goto-line (nth 1 eq))
>> -	(org-table-goto-column (nth 2 eq))
>> +	(let* ((column-target (nth 2 eq))
>> +	       (column-count (progn (end-of-line)
>> +				    (1- (org-table-current-column))))
>> +	       (create-new-column
>> +		(and (> column-target column-count)
>
> Is this check really necessary? Doesn't `org-table-goto-column' already
> figures it out before creating a new column?
>
>
> Regards,

Does it need sanity checking? A typo in a formula can create "billions
and billions"[fn:1] of columns inadvertently.

Footnotes:

[fn:1] with apologies to Carl Sagan (RIP)...

Re: Adding new table rows/cols in a formula update

[Dima Kogan]

[-- Attachment #1: Type: text/plain, Size: 245 bytes --]

Nick Dokos <ndokos@gmail.com> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
> Does it need sanity checking? A typo in a formula can create "billions
> and billions"[fn:1] of columns inadvertently.

Sure. Updated patch attached.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-table.el-field-formulas-can-now-create-columns-a.patch --]
[-- Type: text/x-diff, Size: 2755 bytes --]

From 06f09e58a0f8b62429dda2faa913e297218b67b2 Mon Sep 17 00:00:00 2001
From: Dima Kogan <dima@secretsauce.net>
Date: Tue, 30 Sep 2014 22:36:21 -0700
Subject: [PATCH] org-table.el: field formulas can now create columns as needed

(org-table-recalculate): use the new org-table-formula-make-new-cols
customization to control whether org creates new columns when a
formula explicitly targets them

TINYCHANGE
---
 lisp/org-table.el | 35 ++++++++++++++++++++++++++++++++---
 1 file changed, 32 insertions(+), 3 deletions(-)

diff --git a/lisp/org-table.el b/lisp/org-table.el
index 7607ead..14c68d6 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -354,6 +354,18 @@ portability of tables."
 	  (const :tag "Stick to hline" nil)
 	  (const :tag "Error on attempt to cross" error)))
 
+(defcustom org-table-formula-create-columns nil
+  "Non-nil means that evaluation of a field formula can add new
+columns if an out-of-bounds field is being set."
+  :group 'org-table-calculation
+  :version "24.5"
+  :package-version '(Org . "8.3")
+  :type '(choice
+	  (const :tag "Setting an out-of-bounds field generates an error (default)" nil)
+	  (const :tag "Setting an out-of-bounds field silently adds columns as needed" t)
+	  (const :tag "Setting an out-of-bounds field adds columns as needed, but issues a warning message" warn)
+	  (const :tag "When setting an out-of-bounds field, the user is prompted" prompt)))
+
 (defgroup org-table-import-export nil
   "Options concerning table import and export in Org-mode."
   :tag "Org Table Import Export"
@@ -3125,9 +3137,26 @@ known that the table will be realigned a little later anyway."
       (while (setq eq (pop eqlname1))
 	(message "Re-applying formula to field: %s" (car eq))
 	(org-goto-line (nth 1 eq))
-	(org-table-goto-column (nth 2 eq))
-	(org-table-eval-formula nil (nth 3 eq) 'noalign 'nocst
-				'nostore 'noanalysis))
+	(let ((column-target (nth 2 eq)))
+	  (when (> column-target 1000)
+	    (user-error "Formula column target too large"))
+	  (let* ((column-count (progn (end-of-line)
+				      (1- (org-table-current-column))))
+		 (create-new-column
+		  (and (> column-target column-count)
+		       (or (eq org-table-formula-create-columns t)
+			   (and
+			    (eq org-table-formula-create-columns 'warn)
+			    (progn
+			      (org-display-warning "Out-of-bounds formula added columns")
+			      t))
+			   (and
+			    (eq org-table-formula-create-columns 'prompt)
+			    (yes-or-no-p "Out-of-bounds formula. Add columns?"))))))
+	    (org-table-goto-column column-target nil create-new-column))
+
+	  (org-table-eval-formula nil (nth 3 eq) 'noalign 'nocst
+				  'nostore 'noanalysis)))
 
       (org-goto-line thisline)
       (org-table-goto-column thiscol)
-- 
2.0.0

Re: Adding new table rows/cols in a formula update

[Nicolas Goaziou]

Hello,

Dima Kogan <dima@secretsauce.net> writes:

> Sure. Updated patch attached.

Thanks. It looks good.

Would you mind providing a test for it in org-test-table.el and an entry
in ORG-NEWS?

> Subject: [PATCH] org-table.el: field formulas can now create columns as needed
>
> (org-table-recalculate): use the new org-table-formula-make-new-cols
> customization to control whether org creates new columns when a
> formula explicitly targets them
>
> TINYCHANGE

The commit message should be something like the following

    org-table: Field formulas can now create columns as needed
    
    * org-table.el (org-table-formula-create-columns): New variable.
    (org-table-recalculate): Use the new org-table-formula-make-new-cols
    customization to control whether org creates new columns when
    a formula explicitly targets them.


Regards,

-- 
Nicolas Goaziou

Re: Adding new table rows/cols in a formula update

[Dima Kogan]

[-- Attachment #1: Type: text/plain, Size: 231 bytes --]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Would you mind providing a test for it in org-test-table.el and an entry
> in ORG-NEWS?
>
> The commit message should be something like the following...

OK. New patch attached.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-table-Field-formulas-can-now-create-columns-as-n.patch --]
[-- Type: text/x-diff, Size: 5150 bytes --]

From 273e642c937f0e12a2f71cf6499415406d708629 Mon Sep 17 00:00:00 2001
From: Dima Kogan <dima@secretsauce.net>
Date: Tue, 30 Sep 2014 22:36:21 -0700
Subject: [PATCH] org-table: Field formulas can now create columns as needed

* org-table.el (org-table-formula-create-columns): New variable.
(org-table-recalculate): Use the new org-table-formula-make-new-cols
customization to control whether org creates new columns when
a formula explicitly targets them.

TINYCHANGE
---
 etc/ORG-NEWS                   |  6 +++++
 lisp/org-table.el              | 35 +++++++++++++++++++++++---
 testing/lisp/test-org-table.el | 56 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 94 insertions(+), 3 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 1af54ad..0a5af68 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -90,6 +90,12 @@ docstrings (including ~orgtbl-to-generic~) for details.
 *** Non-floating minted listings in Latex export
 It is not possible to specify =#+attr_latex: :float nil= in conjunction with
 source blocks exported by the minted package.
+*** Field formulas can now create columns as needed
+Previously, evaluating formulas that referenced out-of-bounds columns
+would throw an error. A new variable
+~org-table-formula-create-columns~ was added to adjust this
+behavior. It is now possible to silently add new columns, to do so
+with a warning or to explicitly ask the user each time.
 ** Miscellaneous
 *** File names in links accept are now compatible with URI syntax
 Absolute file names can now start with =///= in addition to =/=. E.g.,
diff --git a/lisp/org-table.el b/lisp/org-table.el
index 7607ead..14c68d6 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -354,6 +354,18 @@ portability of tables."
 	  (const :tag "Stick to hline" nil)
 	  (const :tag "Error on attempt to cross" error)))
 
+(defcustom org-table-formula-create-columns nil
+  "Non-nil means that evaluation of a field formula can add new
+columns if an out-of-bounds field is being set."
+  :group 'org-table-calculation
+  :version "24.5"
+  :package-version '(Org . "8.3")
+  :type '(choice
+	  (const :tag "Setting an out-of-bounds field generates an error (default)" nil)
+	  (const :tag "Setting an out-of-bounds field silently adds columns as needed" t)
+	  (const :tag "Setting an out-of-bounds field adds columns as needed, but issues a warning message" warn)
+	  (const :tag "When setting an out-of-bounds field, the user is prompted" prompt)))
+
 (defgroup org-table-import-export nil
   "Options concerning table import and export in Org-mode."
   :tag "Org Table Import Export"
@@ -3125,9 +3137,26 @@ known that the table will be realigned a little later anyway."
       (while (setq eq (pop eqlname1))
 	(message "Re-applying formula to field: %s" (car eq))
 	(org-goto-line (nth 1 eq))
-	(org-table-goto-column (nth 2 eq))
-	(org-table-eval-formula nil (nth 3 eq) 'noalign 'nocst
-				'nostore 'noanalysis))
+	(let ((column-target (nth 2 eq)))
+	  (when (> column-target 1000)
+	    (user-error "Formula column target too large"))
+	  (let* ((column-count (progn (end-of-line)
+				      (1- (org-table-current-column))))
+		 (create-new-column
+		  (and (> column-target column-count)
+		       (or (eq org-table-formula-create-columns t)
+			   (and
+			    (eq org-table-formula-create-columns 'warn)
+			    (progn
+			      (org-display-warning "Out-of-bounds formula added columns")
+			      t))
+			   (and
+			    (eq org-table-formula-create-columns 'prompt)
+			    (yes-or-no-p "Out-of-bounds formula. Add columns?"))))))
+	    (org-table-goto-column column-target nil create-new-column))
+
+	  (org-table-eval-formula nil (nth 3 eq) 'noalign 'nocst
+				  'nostore 'noanalysis)))
 
       (org-goto-line thisline)
       (org-table-goto-column thiscol)
diff --git a/testing/lisp/test-org-table.el b/testing/lisp/test-org-table.el
index e083683..d99db27 100644
--- a/testing/lisp/test-org-table.el
+++ b/testing/lisp/test-org-table.el
@@ -1550,6 +1550,62 @@ See also `test-org-table/copy-field'."
 	     (progn (search-forward "# END RECEIVE ORGTBL table")
 		    (match-beginning 0)))))))
 
+(ert-deftest test-org-table/field-formula-outside-table ()
+  "If `org-table-formula-create-columns' is nil, then a formula
+that references an out-of-bounds column should do nothing. If it
+is t, then new columns should be added as needed"
+
+  (let ((org-table-formula-create-columns nil))
+
+    ;; need condition-case to trap the out-of-bounds user-error
+    (condition-case
+	nil
+	(org-test-table-target-expect
+	 "
+| 2 |
+| 4 |
+| 8 |
+"
+	 "
+| 2 |
+| 4 |
+| 8 |
+"
+	 1
+	 "#+TBLFM: @1$2=5")
+      ('user-error t)))
+
+  (let ((org-table-formula-create-columns t))
+
+    ;; make sure field formulas work
+    (org-test-table-target-expect
+     "
+| 2 |
+| 4 |
+| 8 |
+"
+     "
+| 2 | 5 |
+| 4 |   |
+| 8 |   |
+"
+     1
+     "#+TBLFM: @1$2=5")
+
+    ;; and make sure column formulas work too
+    (org-test-table-target-expect
+     "
+| 2 |
+| 4 |
+| 8 |
+"
+     "
+| 2 |   | 15 |
+| 4 |   | 15 |
+| 8 |   | 15 |
+"
+     1
+     "#+TBLFM: $3=15")))
 
 (provide 'test-org-table)
 
-- 
2.0.0

Re: Adding new table rows/cols in a formula update

[Bastien]

Hi Dima,

Dima Kogan <dima@secretsauce.net> writes:

> OK. New patch attached.

Thanks for your work on this, the patch looks good but it cannot be
accepted as a "tiny change".  Would you be willing to sign the FSF
copyright assignment papers?

If yes, please fill in the form from here:
http://orgmode.org/cgit.cgi/org-mode.git/plain/request-assign-future.txt

Best,

-- 
 Bastien

Re: Adding new table rows/cols in a formula update

[Nicolas Goaziou]

Hello,

Bastien <bzg@gnu.org> writes:

> Thanks for your work on this, the patch looks good but it cannot be
> accepted as a "tiny change".  Would you be willing to sign the FSF
> copyright assignment papers?

Actually, it's not a tiny change. He signed copyright assignment
already. See http://permalink.gmane.org/gmane.emacs.orgmode/91352


Regards,

-- 
Nicolas Goaziou

Re: Adding new table rows/cols in a formula update

[Dima Kogan]

[-- Attachment #1: Type: text/plain, Size: 434 bytes --]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> Thanks for your work on this, the patch looks good but it cannot be
>> accepted as a "tiny change".  Would you be willing to sign the FSF
>> copyright assignment papers?
>
> Actually, it's not a tiny change. He signed copyright assignment
> already. See http://permalink.gmane.org/gmane.emacs.orgmode/91352

Here's the same patch without the TINYCHANGE marker, if that's helpful.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-table-Field-formulas-can-now-create-columns-as-n.patch --]
[-- Type: text/x-diff, Size: 5138 bytes --]

From 6a361837f1f7b71a02ab5b918509def84c9fa43e Mon Sep 17 00:00:00 2001
From: Dima Kogan <dima@secretsauce.net>
Date: Tue, 30 Sep 2014 22:36:21 -0700
Subject: [PATCH] org-table: Field formulas can now create columns as needed

* org-table.el (org-table-formula-create-columns): New variable.
(org-table-recalculate): Use the new org-table-formula-make-new-cols
customization to control whether org creates new columns when
a formula explicitly targets them.
---
 etc/ORG-NEWS                   |  6 +++++
 lisp/org-table.el              | 35 +++++++++++++++++++++++---
 testing/lisp/test-org-table.el | 56 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 94 insertions(+), 3 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 1af54ad..0a5af68 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -90,6 +90,12 @@ docstrings (including ~orgtbl-to-generic~) for details.
 *** Non-floating minted listings in Latex export
 It is not possible to specify =#+attr_latex: :float nil= in conjunction with
 source blocks exported by the minted package.
+*** Field formulas can now create columns as needed
+Previously, evaluating formulas that referenced out-of-bounds columns
+would throw an error. A new variable
+~org-table-formula-create-columns~ was added to adjust this
+behavior. It is now possible to silently add new columns, to do so
+with a warning or to explicitly ask the user each time.
 ** Miscellaneous
 *** File names in links accept are now compatible with URI syntax
 Absolute file names can now start with =///= in addition to =/=. E.g.,
diff --git a/lisp/org-table.el b/lisp/org-table.el
index 7607ead..14c68d6 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -354,6 +354,18 @@ portability of tables."
 	  (const :tag "Stick to hline" nil)
 	  (const :tag "Error on attempt to cross" error)))
 
+(defcustom org-table-formula-create-columns nil
+  "Non-nil means that evaluation of a field formula can add new
+columns if an out-of-bounds field is being set."
+  :group 'org-table-calculation
+  :version "24.5"
+  :package-version '(Org . "8.3")
+  :type '(choice
+	  (const :tag "Setting an out-of-bounds field generates an error (default)" nil)
+	  (const :tag "Setting an out-of-bounds field silently adds columns as needed" t)
+	  (const :tag "Setting an out-of-bounds field adds columns as needed, but issues a warning message" warn)
+	  (const :tag "When setting an out-of-bounds field, the user is prompted" prompt)))
+
 (defgroup org-table-import-export nil
   "Options concerning table import and export in Org-mode."
   :tag "Org Table Import Export"
@@ -3125,9 +3137,26 @@ known that the table will be realigned a little later anyway."
       (while (setq eq (pop eqlname1))
 	(message "Re-applying formula to field: %s" (car eq))
 	(org-goto-line (nth 1 eq))
-	(org-table-goto-column (nth 2 eq))
-	(org-table-eval-formula nil (nth 3 eq) 'noalign 'nocst
-				'nostore 'noanalysis))
+	(let ((column-target (nth 2 eq)))
+	  (when (> column-target 1000)
+	    (user-error "Formula column target too large"))
+	  (let* ((column-count (progn (end-of-line)
+				      (1- (org-table-current-column))))
+		 (create-new-column
+		  (and (> column-target column-count)
+		       (or (eq org-table-formula-create-columns t)
+			   (and
+			    (eq org-table-formula-create-columns 'warn)
+			    (progn
+			      (org-display-warning "Out-of-bounds formula added columns")
+			      t))
+			   (and
+			    (eq org-table-formula-create-columns 'prompt)
+			    (yes-or-no-p "Out-of-bounds formula. Add columns?"))))))
+	    (org-table-goto-column column-target nil create-new-column))
+
+	  (org-table-eval-formula nil (nth 3 eq) 'noalign 'nocst
+				  'nostore 'noanalysis)))
 
       (org-goto-line thisline)
       (org-table-goto-column thiscol)
diff --git a/testing/lisp/test-org-table.el b/testing/lisp/test-org-table.el
index e083683..d99db27 100644
--- a/testing/lisp/test-org-table.el
+++ b/testing/lisp/test-org-table.el
@@ -1550,6 +1550,62 @@ See also `test-org-table/copy-field'."
 	     (progn (search-forward "# END RECEIVE ORGTBL table")
 		    (match-beginning 0)))))))
 
+(ert-deftest test-org-table/field-formula-outside-table ()
+  "If `org-table-formula-create-columns' is nil, then a formula
+that references an out-of-bounds column should do nothing. If it
+is t, then new columns should be added as needed"
+
+  (let ((org-table-formula-create-columns nil))
+
+    ;; need condition-case to trap the out-of-bounds user-error
+    (condition-case
+	nil
+	(org-test-table-target-expect
+	 "
+| 2 |
+| 4 |
+| 8 |
+"
+	 "
+| 2 |
+| 4 |
+| 8 |
+"
+	 1
+	 "#+TBLFM: @1$2=5")
+      ('user-error t)))
+
+  (let ((org-table-formula-create-columns t))
+
+    ;; make sure field formulas work
+    (org-test-table-target-expect
+     "
+| 2 |
+| 4 |
+| 8 |
+"
+     "
+| 2 | 5 |
+| 4 |   |
+| 8 |   |
+"
+     1
+     "#+TBLFM: @1$2=5")
+
+    ;; and make sure column formulas work too
+    (org-test-table-target-expect
+     "
+| 2 |
+| 4 |
+| 8 |
+"
+     "
+| 2 |   | 15 |
+| 4 |   | 15 |
+| 8 |   | 15 |
+"
+     1
+     "#+TBLFM: $3=15")))
 
 (provide 'test-org-table)
 
-- 
2.0.0

Re: Adding new table rows/cols in a formula update

[Nicolas Goaziou]

Hello,

Dima Kogan <dima@secretsauce.net> writes:

> Here's the same patch without the TINYCHANGE marker, if that's
> helpful.

Applied. Thank you for this work.


Regards,

-- 
Nicolas Goaziou

Re: Adding new table rows/cols in a formula update

[Bastien]

Hi,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> Here's the same patch without the TINYCHANGE marker, if that's
>> helpful.
>
> Applied. Thank you for this work.

Indeed, thanks.

I double-checkedd and found the copyright assignment for Dima.

Can one of you add Dima to
http://orgmode.org/worg/org-contribute.html#contributors_with_fsf_papers

?

Thanks!

-- 
 Bastien

Re: Adding new table rows/cols in a formula update

[Nicolas Goaziou]

Hello,

Bastien <bzg@altern.org> writes:

> Can one of you add Dima to
> http://orgmode.org/worg/org-contribute.html#contributors_with_fsf_papers
>
> ?

Done.


Regards,

-- 
Nicolas Goaziou                                                0x80A93738

Re: Adding new table rows/cols in a formula update

[Bastien]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Bastien <bzg@altern.org> writes:
>
>> Can one of you add Dima to
>> http://orgmode.org/worg/org-contribute.html#contributors_with_fsf_papers
>>
>> ?
>
> Done.

Thanks!

-- 
 Bastien

Re: Adding new table rows/cols in a formula update

[Achim Gratz]

Dima Kogan writes:
> --- a/testing/lisp/test-org-table.el
[…]
> +    (condition-case
[…]
> +	 "#+TBLFM: @1$2=5")
> +      ('user-error t)))

That part of the test, specifically the attempt to catch the error is
not working for me on Windows, most likely because I use an older
version of Emacs there (I don't recall which one, either 23.4 or 24.1 I
think).  This probably comes about because ert-deftest is a macro, but I
don't have time right now to dig into it.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: Adding new table rows/cols in a formula update

[Achim Gratz]

Achim Gratz writes:
> Dima Kogan writes:
>> --- a/testing/lisp/test-org-table.el
> […]
>> +    (condition-case
> […]
>> +	 "#+TBLFM: @1$2=5")
>> +      ('user-error t)))
>
> That part of the test, specifically the attempt to catch the error is
> not working for me on Windows, most likely because I use an older
> version of Emacs there (I don't recall which one, either 23.4 or 24.1 I
> think).  This probably comes about because ert-deftest is a macro, but I
> don't have time right now to dig into it.

It turns out that older Emacsen throw 'error rather than user-error, so
you need to catch both.  Fixed in 82599d3 on master.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada