extra space at the end of lines in source

extra space at the end of lines in source

[Greg Minshall]

hi.  i can't date it exactly, but in the last week or so, editing a
source buffer (with =C-c '=) adds spaces (to the "tab location") of
previously blank lines.

i.e., the second line in each of the following source blocks is empty,
but if i =C-c '= then =C-c '=, each will end up with a few spaces on it.

#+BEGIN_SRC R
  cat("hello, world\n")

  23
#+END_SRC


#+BEGIN_SRC bash
  echo hello, world

  echo 23
#+END_SRC

i'm running
: Org mode version 9.4.6 (release_9.4.6-559-g5e4815 @ /home/minshall/.emacs.d/straight/build/org/)
via
: emacs -Q --eval "(add-to-list 'load-path \"~/.emacs.d/straight/build/org\")" foo.org

any thoughts?

cheers, Greg

Re: extra space at the end of lines in source

[Sébastien Miquel]

Hi Greg,

Greg Minshall writes:
> hi.  i can't date it exactly, but in the last week or so, editing a
> source buffer (with =C-c '=) adds spaces (to the "tab location") of
> previously blank lines.

See https://lists.gnu.org/archive/html/emacs-orgmode/2021-05/msg00867.html

The goal of the change was to fix some (electric) indentation issues when
editing a src block directly.

As I write in the linked thread, setting `org-src--preserve-indentation' 
should
revert this behaviour.

Regards,

-- 
Sébastien Miquel

Re: extra space at the end of lines in source

[Greg Minshall]

hi, Sebastien,

thanks for the reply.  i remember that thread, but obviously i wasn't
paying enough attention.

> The downside is that, unless ~org-src--preserve-indentation~ is `t`,
> when editing a src block, every empty line will be indented with
> spaces (according to ~org-edit-src-content-indentation~ + the
> indentation of the #+begin_src line). I think this is reasonable, but
> perhaps some might disagree.

indeed.  i disagree.  it's long-term emacs behavior to eliminate spaces
at the end of lines, at least in programming modes.  i don't know what
would be involved to keep the fix for the original problem (which i was
also seeing), without adding these extra spaces.  but, i suspect it
would be worth it, if feasible.

cheers, Greg

Re: extra space at the end of lines in source

[Sébastien Miquel]

For the record, the original issue is that with 
`org-src-tab-acts-natively' set
to t (which is the new default) you couldn't use tab to indent an empty 
line, and
electric-indent-mode wouldn't work properly.

Greg Minshall writes:
> i don't know what
> would be involved to keep the fix for the original problem (which i was
> also seeing), without adding these extra spaces.  but, i suspect it
> would be worth it, if feasible.

I think the only way would be to change 
`org-src--contents-for-write-back' to
keep track of where the point was in the edit buffer, and clean up spaces in
other lines. Doesn't seem difficult ; maybe you could replace the 
`buffer-string`
call by two calls, to get the part before point and the one after, then work
with that.

> it's long-term emacs behavior to eliminate spaces
> at the end of lines, at least in programming modes.
As for the `org-src--content-indentation' spaces, they are quite 
peculiar. Note
that they are removed when you call =C-c '= and when you tangle (they 
should,
but I haven't tested it), so strictly speaking, they are removed in 
programming
modes. What if your org block itself is indented, do you also expect 
blank lines
to be entirely empty ?

As for additional indentation in a blank line, it will indeed never get 
cleaned
up by org, which isn't ideal. But then, spaces at the end of non blank lines
don't get cleaned up either (I think) and never were. It's up to the user to
remove them, in the appropriate language mode.

Despite these arguments, I have no opinion on the matter.

Regards,

-- 
Sébastien Miquel

Re: extra space at the end of lines in source

[Greg Minshall]

Sébastien Miquel,

thanks for the reply.

> > it's long-term emacs behavior to eliminate spaces
> > at the end of lines, at least in programming modes.

(Tim -- i guess i should have said, "it's my long-term experience with
emacs...".)

> As for the `org-src--content-indentation' spaces, they are quite
> peculiar. Note that they are removed when you call =C-c '= and when
> you tangle (they should, but I haven't tested it), so strictly
> speaking, they are removed in programming modes. What if your org
> block itself is indented, do you also expect blank lines to be
> entirely empty ?

there are two "practical" issues, plus one aesthetic issue, for me:

- because of the changes to source blocks i've edited, but not, in fact,
  tried to change, i'm getting loads of diffs when checking in a new
  version of the .org file

- the next time i open the Org Src buffer, whatever lint-like process is
  running for that language may complain about extra spaces at the end
  of a line.  (does that mean my experience is different from yours, or
  at least from your expectation?)

- "aesthetically" (or, "conditioned", by years of some expected
  behavior), i find the thought that there may be lines like these in
  the file somewhat distressing.

i'm resigned to the extra spaces when an org block is indented, so
wouldn't find the existence of spaces there any more worrisome.
(though, i would expect the spaces to be up to the indentation of the
org block, not all the way up to "the programming level".)

i do think the default should be to *not* add these spaces.  if
possible.

cheers, Greg

Re: extra space at the end of lines in source

[Tim Cross]

Greg Minshall <minshall@umich.edu> writes:

> Sébastien Miquel,
>
> i do think the default should be to *not* add these spaces.  if
> possible.
>

+1 Org (and Emacs more generally) should not add additional spaces
unless they are required (for example, to make code syntactically
correct) or the user has requested them (by setting a variable/option).

I also think it is reasonable to assume that if you edit a source block
directly in the org file (i.e. not using C-c '), you don't get the
language specific formatting you would expect when editing the source in
either a dedicated source buffer or with C-c '. Editing source in a
source block without using C-c ' is ok for simple edits, but for
anything more complex or requiring language specific formatting etc, C-c
' should be used. 

-- 
Tim Cross

Re: [PATCH] extra space at the end of lines in source

[Sébastien Miquel]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

Greg Minshall writes:
> - the next time i open the Org Src buffer, whatever lint-like process is
>    running for that language may complain about extra spaces at the end
>    of a line.  (does that mean my experience is different from yours, or
>    at least from your expectation?)
If I try your original examples with `emacs -q' I do not get extra 
whitespace in
the org src buffer. Those two spaces in the original org buffer -- that 
are due
to `org-edit-src-content-indentation' -- are removed in the org src 
buffer. If
you do not find it to be the case, then I'm missing something.

Anyway, here's a patch that cleans up blank lines, except the current 
one. It
preserves the fix for the original issue.

Can you try it out ?

Regards,

-- 
Sébastien Miquel


[-- Attachment #2: 0001-org-src.el-org-src-contents-for-write-back-Do-not-in.patch --]
[-- Type: text/x-patch, Size: 2600 bytes --]

From 405c2be7487c564e72a9f01a940f96dc19ff16ad Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?S=C3=A9bastien=20Miquel?= <sebastien.miquel@posteo.eu>
Date: Wed, 23 Jun 2021 15:25:58 +0200
Subject: [PATCH] org-src.el (org-src--contents-for-write-back): Do not indent
 blank lines

* lisp/org-src.el (org-src--contents-for-write-back): Do not indent
blank lines, except for the current line.

This was the original behaviour for all blank lines, before `857ae366b3`.
---
 lisp/org-src.el | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)

diff --git a/lisp/org-src.el b/lisp/org-src.el
index 79f002e56..faacb53e3 100644
--- a/lisp/org-src.el
+++ b/lisp/org-src.el
@@ -443,14 +443,20 @@ Assume point is in the corresponding edit buffer."
 		0))))
 	(use-tabs? (and (> org-src--tab-width 0) t))
 	(source-tab-width org-src--tab-width)
-	(contents (org-with-wide-buffer (buffer-string)))
-	(write-back org-src--allow-write-back))
+	(contents (org-with-wide-buffer
+                   (let ((eol (progn (end-of-line) (point))))
+                     (list (buffer-substring (point-min) eol)
+                           (buffer-substring eol (point-max))))))
+	(write-back org-src--allow-write-back)
+        marker)
     (with-current-buffer write-back-buf
       ;; Reproduce indentation parameters from source buffer.
       (setq indent-tabs-mode use-tabs?)
       (when (> source-tab-width 0) (setq tab-width source-tab-width))
       ;; Apply WRITE-BACK function on edit buffer contents.
-      (insert (org-no-properties contents))
+      (insert (org-no-properties (car contents)))
+      (setq marker (point-marker))
+      (insert (org-no-properties (car (cdr contents))))
       (goto-char (point-min))
       (when (functionp write-back) (save-excursion (funcall write-back)))
       ;; Add INDENTATION-OFFSET to every line in buffer,
@@ -458,10 +464,13 @@ Assume point is in the corresponding edit buffer."
       (when (> indentation-offset 0)
 	(while (not (eobp))
 	  (skip-chars-forward " \t")
-	  (let ((i (current-column)))
-	    (delete-region (line-beginning-position) (point))
-	    (indent-to (+ i indentation-offset)))
-	  (forward-line))))))
+          (when (or (not (eolp)) ; ignore blank lines
+                    (eq (point) (marker-position marker)))
+	    (let ((i (current-column)))
+	      (delete-region (line-beginning-position) (point))
+	      (indent-to (+ i indentation-offset))))
+	  (forward-line)))
+      (set-marker marker nil))))
 
 (defun org-src--edit-element
     (datum name &optional initialize write-back contents remote)
-- 
2.32.0

Re: [PATCH] extra space at the end of lines in source

[Greg Minshall]

Sébastien Miquel,


> If I try your original examples with `emacs -q' I do not get extra
> whitespace in the org src buffer. Those two spaces in the original org
> buffer -- that are due to `org-edit-src-content-indentation' -- are
> removed in the org src buffer. If you do not find it to be the case,
> then I'm missing something.

i apologize, i think i described the problem badly.  the problem i saw
is not extra spaces in the Org Src buffer; it is the appearance of the
extra spaces in the original .org file after closing (C-c ') the Org Src
buffer that i see as problematic.

> Anyway, here's a patch that cleans up blank lines, except the current
> one. It preserves the fix for the original issue.

thanks.  my trivial test shows this works *except* in the particular
case where, when closing the Org Src buffer, `point` is on an empty
line.  in this case, that one empty line is given extra spaces.

if i try my test with
: Org mode version 9.4.4 (release_9.4.4 @ /usr/share/emacs/27.2/lisp/org/)
i do not see the extra spaces in the .org file.

the test src block is
----
#+BEGIN_SRC bash
  echo hello, world

  echo 23


#+END_SRC
----
(modulo any e-mail mangling).

again, thanks.  Greg

Re: [PATCH] extra space at the end of lines in source

[Sébastien Miquel]

[-- Attachment #1: Type: text/plain, Size: 507 bytes --]

Greg Minshall writes:
> thanks.  my trivial test shows this works*except*  in the particular
> case where, when closing the Org Src buffer, `point` is on an empty
> line.  in this case, that one empty line is given extra spaces.
Yes, I was aware of this, but didn't think we could do better in this 
case. I've
thought about it some more, and came up with a solution.

Here's a new patch that's smarter about indenting the current line, and 
resolves
this remaining issue.

Regards,

-- 
Sébastien Miquel


[-- Attachment #2: 0001-org-src.el-Do-not-indent-blank-lines-except-current-.patch --]
[-- Type: text/x-patch, Size: 6066 bytes --]

From 8c31e6b732a45c0ddbbf0a0db7a2cb4ef3af0414 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?S=C3=A9bastien=20Miquel?= <sebastien.miquel@posteo.eu>
Date: Wed, 23 Jun 2021 15:25:58 +0200
Subject: [PATCH] org-src.el: Do not indent blank lines, except current one

* lisp/org-src.el (org-src--contents-for-write-back): Do not indent blank lines, except for the
current line maybe.
(org-src--preserve-blank-line): New variable, whether to preserve
indentation of the current blank line.
(org-src--edit-element): Set `org-src--preserve-blank-line'.
* lisp/org.el (org-indent-line): When tab acts natively, do some
preindentation, which signals `org-src--edit-element' to
preserve the indentation of current blank line.

Removing all the whitespace was the original behaviour for all blank lines, before `857ae366b3`.
---
 lisp/org-src.el | 34 +++++++++++++++++++++++++++-------
 lisp/org.el     | 15 +++++++++++++++
 2 files changed, 42 insertions(+), 7 deletions(-)

diff --git a/lisp/org-src.el b/lisp/org-src.el
index 79f002e56..c6311adbb 100644
--- a/lisp/org-src.el
+++ b/lisp/org-src.el
@@ -299,6 +299,9 @@ is 0.")
   "File name associated to Org source buffer, or nil.")
 (put 'org-src-source-file-name 'permanent-local t)
 
+(defvar-local org-src--preserve-blank-line nil)
+(put 'org-src--preserve-blank-line 'permanent-local t)
+
 (defun org-src--construct-edit-buffer-name (org-buffer-name lang)
   "Construct the buffer name for a source editing buffer."
   (concat "*Org Src " org-buffer-name "[ " lang " ]*"))
@@ -443,14 +446,21 @@ Assume point is in the corresponding edit buffer."
 		0))))
 	(use-tabs? (and (> org-src--tab-width 0) t))
 	(source-tab-width org-src--tab-width)
-	(contents (org-with-wide-buffer (buffer-string)))
-	(write-back org-src--allow-write-back))
+	(contents (org-with-wide-buffer
+                   (let ((eol (line-end-position)))
+                     (list (buffer-substring (point-min) eol)
+                           (buffer-substring eol (point-max))))))
+	(write-back org-src--allow-write-back)
+        (preserve-blank-line org-src--preserve-blank-line)
+        marker)
     (with-current-buffer write-back-buf
       ;; Reproduce indentation parameters from source buffer.
       (setq indent-tabs-mode use-tabs?)
       (when (> source-tab-width 0) (setq tab-width source-tab-width))
       ;; Apply WRITE-BACK function on edit buffer contents.
-      (insert (org-no-properties contents))
+      (insert (org-no-properties (car contents)))
+      (setq marker (point-marker))
+      (insert (org-no-properties (car (cdr contents))))
       (goto-char (point-min))
       (when (functionp write-back) (save-excursion (funcall write-back)))
       ;; Add INDENTATION-OFFSET to every line in buffer,
@@ -458,10 +468,14 @@ Assume point is in the corresponding edit buffer."
       (when (> indentation-offset 0)
 	(while (not (eobp))
 	  (skip-chars-forward " \t")
-	  (let ((i (current-column)))
-	    (delete-region (line-beginning-position) (point))
-	    (indent-to (+ i indentation-offset)))
-	  (forward-line))))))
+          (when (or (not (eolp))                               ; not a blank line
+                    (and (eq (point) (marker-position marker)) ; current line
+                         preserve-blank-line))
+	    (let ((i (current-column)))
+	      (delete-region (line-beginning-position) (point))
+	      (indent-to (+ i indentation-offset))))
+	  (forward-line)))
+      (set-marker marker nil))))
 
 (defun org-src--edit-element
     (datum name &optional initialize write-back contents remote)
@@ -506,6 +520,11 @@ Leave point in edit buffer."
 	     (block-ind (org-with-point-at (org-element-property :begin datum)
 			  (current-indentation)))
 	     (content-ind org-edit-src-content-indentation)
+             (blank-line (save-excursion (beginning-of-line)
+                                         (looking-at-p "^[[:space:]]*$")))
+             (empty-line (and blank-line (looking-at-p "^$")))
+             (preserve-blank-line (or (and blank-line (not empty-line))
+                                      (and empty-line (= (+ block-ind content-ind) 0))))
 	     (preserve-ind
 	      (and (memq type '(example-block src-block))
 		   (or (org-element-property :preserve-indent datum)
@@ -554,6 +573,7 @@ Leave point in edit buffer."
 	(setq org-src--overlay overlay)
 	(setq org-src--allow-write-back write-back)
 	(setq org-src-source-file-name source-file-name)
+        (setq org-src--preserve-blank-line preserve-blank-line)
 	;; Start minor mode.
 	(org-src-mode)
 	;; Clear undo information so we cannot undo back to the
diff --git a/lisp/org.el b/lisp/org.el
index 1bd9e02eb..3bc4d0383 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -19164,6 +19164,21 @@ Also align node properties according to `org-property-format'."
 		     (org-with-point-at (org-element-property :end element)
 		       (skip-chars-backward " \t\n")
 		       (line-beginning-position))))
+             ;; At the beginning of a blank line, do some preindentation.  This
+             ;; signals org-src--edit-element to preserve the indentation on exit
+             (when (and (looking-at-p "^[[:space:]]*$")
+                        (not org-src-preserve-indentation))
+               (let ((element (org-element-at-point))
+                      block-content-ind some-ind)
+                 (org-with-point-at (org-element-property :begin element)
+                   (setq block-content-ind (+ (current-indentation)
+                                              org-edit-src-content-indentation))
+                   (forward-line)
+		   (save-match-data (re-search-forward "^[ \t]*\\S-" nil t))
+                   (backward-char)
+                   (setq some-ind (if (looking-at-p "#\\+end_src")
+                                      block-content-ind (current-indentation))))
+                 (indent-line-to (min block-content-ind some-ind))))
 	     (org-babel-do-key-sequence-in-edit-buffer (kbd "TAB")))
 	    (t
 	     (let ((column (org--get-expected-indentation element nil)))
-- 
2.32.0

Re: [PATCH] extra space at the end of lines in source

[Greg Minshall]

Sébastien Miquel,

> > thanks.  my trivial test shows this works*except*  in the particular
> > case where, when closing the Org Src buffer, `point` is on an empty
> > line.  in this case, that one empty line is given extra spaces.
> Yes, I was aware of this, but didn't think we could do better in this
> case. I've
> thought about it some more, and came up with a solution.
> 
> Here's a new patch that's smarter about indenting the current line,
> and resolves this remaining issue.

indeed, this seems to work as advertised.  thank you very much for
spending time on this issue.

Greg

Re: [PATCH] extra space at the end of lines in source

[Greg Minshall]

hi.  i don't know who might merge Sébastien Miquel's fix into the source
code, but i'd be a promoter of this happening whenever it's convenient.

cheers, Greg

Sébastien Miquel <sebastien.miquel@posteo.eu> wrote:

> Greg Minshall writes:
> > thanks.  my trivial test shows this works*except*  in the particular
> > case where, when closing the Org Src buffer, `point` is on an empty
> > line.  in this case, that one empty line is given extra spaces.
> Yes, I was aware of this, but didn't think we could do better in this
> case. I've
> thought about it some more, and came up with a solution.
> 
> Here's a new patch that's smarter about indenting the current line,
> and resolves
> this remaining issue.
> 
> Regards,
> 
> -- 
> Sébastien Miquel
> 

Re: [PATCH] extra space at the end of lines in source

[Bastien]

Hi Sébastien,

Sébastien Miquel <sebastien.miquel@posteo.eu> writes:

> Here's a new patch that's smarter about indenting the current line,
> and resolves this remaining issue.

Applied, thanks!

Re: [PATCH] extra space at the end of lines in source

[Greg Minshall]

Bastien,

> Applied, thanks!

thank *you*!

Re: extra space at the end of lines in source

[Tim Cross]

Greg Minshall <minshall@umich.edu> writes:

> hi, Sebastien,
>
> thanks for the reply.  i remember that thread, but obviously i wasn't
> paying enough attention.
>
>> The downside is that, unless ~org-src--preserve-indentation~ is `t`,
>> when editing a src block, every empty line will be indented with
>> spaces (according to ~org-edit-src-content-indentation~ + the
>> indentation of the #+begin_src line). I think this is reasonable, but
>> perhaps some might disagree.
>
> indeed.  i disagree.  it's long-term emacs behavior to eliminate spaces
> at the end of lines, at least in programming modes.  i don't know what
> would be involved to keep the fix for the original problem (which i was
> also seeing), without adding these extra spaces.  but, i suspect it
> would be worth it, if feasible.
>

I'm not sure that is correct. Elimination of spaces at the end of lines
has always been something you could enable (either directly as a setting
in the mode, which might be enabled by default, or via a package like
whitespace.el. I think it would be going against Emacs philosophy to
have a mode automatically remove end of line spaces - this should be
something the user controls.

Of course, the converse is also true. It would be a mistake to have an
Emacs mode add additional whitespace at the end of lines which is not
requested by the user as well. This could also cause issues/conflicts
with packages like whitespace.el or diffs and VCS etc. It may also make
source blocks 'ugly' if you have your Emacs configured to show
additional whitespace at eol. 

-- 
Tim Cross