did behaviour of RET change again?

did behaviour of RET change again?

[Eric S Fraga]

Just a quick heads-up:

I have just installed org from git (a few hours ago) and now it seems
that RET no longer indents.  Is this intentional?

I know that there has been some discussion on the mailing list but I
seem to have lost track of the decision, if any, taken.  RET and C-j now
behave the same (although bound to different functions).  I have
electric-indent-mode enabled and expected RET to indent accordingly.

Have I missed something?

thank you,
eric

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.3-150-g6b83c6

Re: did behaviour of RET change again?

[Pankaj Jangid]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:


> I have just installed org from git (a few hours ago) and now it seems
> that RET no longer indents.  Is this intentional?
>
> I know that there has been some discussion on the mailing list but I
> seem to have lost track of the decision, if any, taken.  RET and C-j now
> behave the same (although bound to different functions).  I have
> electric-indent-mode enabled and expected RET to indent accordingly.

I am using 9.4.2 from emacs-master. It is working fine there.

Re: did behaviour of RET change again?

[Eric S Fraga]

On Friday, 18 Dec 2020 at 20:12, Pankaj Jangid wrote:
> I am using 9.4.2 from emacs-master. It is working fine there.

I'm on 9.4.3...

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.3-150-g6b83c6

Re: did behaviour of RET change again?

[Berry, Charles via General discussions about Org-mode.]

> On Dec 18, 2020, at 5:06 AM, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
> 
> Just a quick heads-up:
> 
> I have just installed org from git (a few hours ago) and now it seems
> that RET no longer indents.  Is this intentional?
> 
> I know that there has been some discussion on the mailing list but I
> seem to have lost track of the decision, if any, taken.  


That was Subject: How to preserve empty headings

X-Universally-Unique-Identifier: 3B7C5634-13EF-4F54-B9A6-C3D8C4342A8D


AFAICS, the behavior discussed there is unchanged as of this moment (9.4.3 from git master), viz RET and C-j behave differently and honor electric-indent-mode in doing so.

Also, I don't see any git log entries that are suggestive since that time.
HTH,

Chuck


[snip]

Re: did behaviour of RET change again?

[Pankaj Jangid]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Friday, 18 Dec 2020 at 20:12, Pankaj Jangid wrote:
>> I am using 9.4.2 from emacs-master. It is working fine there.
>
> I'm on 9.4.3...

Today, I have 9.4.3 from emacs-master. And the feature is working
perfectly fine in this as well C-j and RET, both working accordingly.

Re: did behaviour of RET change again?

[Bastien]

Hi Eric,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Just a quick heads-up:
>
> I have just installed org from git (a few hours ago) and now it seems
> that RET no longer indents.  Is this intentional?

I've not closely followed this, sorry.

I see something wrong right now: RET after a headline should only try
to indent below the beginning of the headline with org-adapt-indentation 
is t, but not nil or headline-data.

For now, when org-adapt-indentation is headline-data, RET still indents.

I will see how to fix this.

Also, I'm thinking of using headline-data as the new default for the
org-adapt-indentation option.  WDYT?

I know Kevin as a good overview of the whole topic, maybe he can also
advise about what should be done here.

Best,

-- 
 Bastien

Re: did behaviour of RET change again?

[Gustavo Barros]

Hi All,

On Sun, 20 Dec 2020 at 18:25, Bastien <bzg@gnu.org> wrote:

>
> Also, I'm thinking of using headline-data as the new default for the
> org-adapt-indentation option.  WDYT?
>
> I know Kevin as a good overview of the whole topic, maybe he can also
> advise about what should be done here.

I cannot but raise a friendly flag here.

I've reported the following behavior I've found for `headline-data'
some time ago: https://orgmode.org/list/87r1qukjjs.fsf@gmail.com/

Nicholas Savage did try and could not reproduce.  Last time I've tried
it, not long ago, I still found the behavior.  I would thus suggest that
at least some more people try it and check their end before going that
route.  I'll be happy if it's just me.

That said, I do think `headline-data' is a great default value for
`org-adapt-indentation', despite the fuss that is bound to ensue for
*any* change there.

Best regards,
Gustavo.

Re: did behaviour of RET change again?

[Eric S Fraga]

On Sunday, 20 Dec 2020 at 18:25, Bastien wrote:
> I see something wrong right now: RET after a headline should only try
> to indent below the beginning of the headline with org-adapt-indentation 
> is t, but not nil or headline-data.

Interesting.  I have org-adapt-indentation set to nil as I use
org-indent-mode and don't physically indent any normal lines.

The problem, for me, arises within lists where I do expect
indentation.  RET used to indent; now it doesn't.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.3-150-g6b83c6

Re: did behaviour of RET change again?

[Eric S Fraga]

Just to say that RET seems to be working again.  No idea what happened
or changed, mind you...  Sorry for the noise.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.3-150-g6b83c6

Re: did behaviour of RET change again?

[Kévin Le Gouguec]

[-- Attachment #1: Type: text/plain, Size: 1598 bytes --]

Bastien <bzg@gnu.org> writes:

> I see something wrong right now: RET after a headline should only try
> to indent below the beginning of the headline with org-adapt-indentation 
> is t, but not nil or headline-data.
>
> For now, when org-adapt-indentation is headline-data, RET still indents.
>
> I will see how to fix this.
>
> Also, I'm thinking of using headline-data as the new default for the
> org-adapt-indentation option.  WDYT?

I personally agree that headline-data makes more sense as a default
given the feedback we received a few weeks ago; as you noticed though
there might be a few loose ends to tie up before making the switch:

- As you said, RET after a headline indents, but the common case for
  hitting RET after a headline is probably to write text, since IME
  headline drawers are always inserted with dedicated commands; thus RET
  should not indent after a header.

- RET after a headline drawer's :END: also indents.

- RET in a list item does not indent; it's not obvious that it should,
  but FWIW (1) RET indents when org-adapt-indentation is t (2) that
  would be my preference.

Also, Greg Minshall (CC'ed) helpfully laid out how org-adapt-indentation
and electric-indent-mode play together in one neat table:

https://orgmode.org/list/2020-11-13T18-23-43@devnull.Karl-Voit.at/t/#mec37faab85f3de59e25a7c1640e5f50be5494192

I didn't take the time to properly review his findings, but there might
be more inconsistencies lurking in there.

Finally, not a big problem if headline-data becomes the default, but:
the :safe predicate is still booleanp.  Patch attached:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-lisp-org.el-org-adapt-indentation-Mark-headline-data.patch --]
[-- Type: text/x-patch, Size: 771 bytes --]

From fd8dab0c42d7104566437b51526b25979f1056fe Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?K=C3=A9vin=20Le=20Gouguec?= <kevin.legouguec@gmail.com>
Date: Mon, 21 Dec 2020 12:09:56 +0100
Subject: [PATCH] * lisp/org.el (org-adapt-indentation): Mark 'headline-data as
 safe

---
 lisp/org.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/org.el b/lisp/org.el
index 1f7e434ce..f75745aba 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -1610,7 +1610,7 @@ time in Emacs."
 	  (const :tag "Adapt indentation for headline data lines"
 		 'headline-data)
 	  (const :tag "Do not adapt indentation at all" nil))
-  :safe #'booleanp)
+  :safe (lambda (x) (memq x '(t nil headline-data))))
 
 (defvaralias 'org-special-ctrl-a 'org-special-ctrl-a/e)
 
-- 
2.29.2


[-- Attachment #3: Type: text/plain, Size: 466 bytes --]



Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Just to say that RET seems to be working again.  No idea what happened
> or changed, mind you...  Sorry for the noise.

Glad things fell back in place somehow.  The only explanation I can
conjure for weird/transient/irreproducible behaviour is directory-local
settings, e.g. org-mode.git's .dir-locals.el sets org-adapt-indentation
to nil… but there might have been something else at work in your case 🤷

Re: did behaviour of RET change again?

[Kyle Meyer]

Bastien writes:

> Also, I'm thinking of using headline-data as the new default for the
> org-adapt-indentation option.  WDYT?

FWIW if we're going to change the default, I'd vote for the simpler and
longer-existing nil value.

Re: did behaviour of RET change again?

[Samuel Wales]

there are just a few defaults i think are better for new users,
despite discoverability.

no indentation is one such.

1 changes org files less [better for e.g. merging]

2 requires less filling maintenance [for the body text; bastien's
change works here]

3 requires less adjustment when plain-text changes are made

4 is parseable by third party code that has whitespace line prefixes
in derived formats


another default i'd change is sub-superscript, which has littered
variable_[name as a subscript] all over the web -- even when highly
experienced org users on this mailing list export to html.  :)

Re: did behaviour of RET change again?

[Samuel Wales]

in case not obvious, i am suggesting a nil value for org adapt indentation.

thus no physical indentation of all lines including planning lines.

i'd even suggest no physical indentation as default for example and
source blocks, but that is a can of worms.

On 12/22/20, Samuel Wales <samologist@gmail.com> wrote:
> there are just a few defaults i think are better for new users,
> despite discoverability.
>
> no indentation is one such.
>
> 1 changes org files less [better for e.g. merging]
>
> 2 requires less filling maintenance [for the body text; bastien's
> change works here]
>
> 3 requires less adjustment when plain-text changes are made
>
> 4 is parseable by third party code that has whitespace line prefixes
> in derived formats
>
>
> another default i'd change is sub-superscript, which has littered
> variable_[name as a subscript] all over the web -- even when highly
> experienced org users on this mailing list export to html.  :)
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: did behaviour of RET change again?

[Tom Gillespie]

> in case not obvious, i am suggesting a nil value for org adapt indentation.
> thus no physical indentation of all lines including planning lines.
> i'd even suggest no physical indentation as default for example and
> source blocks, but that is a can of worms.

I know that this is a can of worms, but I agree. Given that the effects of
org-adapt-indentation can be mimicked in other ways without having the
literal spaces present in the file it may not be as big a deal as we think.

The other reason I think this is a good idea is because I have been working
on a formal grammar for the org syntax, and everything would be SO much
simpler about the implementation after the first pass parse if the canonical
representation of an Org file did not allow significant whitespace (with an
exception for plain lists).

Just avoiding having to deal with any number of nasty edge cases for correctly
aligning org babel blocks would be worth it. Not to mention the fact that it
means that you have to do a triple pass over each incoming line in order to be
sure that what you are passing to an org babel block has had the leading
whitespace removed (once for a normal parse, second time to adjust whitespace
and a third time to actually parse the babel block). No significant
leading whitespace
would remove the need for an entire pass in the parser.

I will have more on the subject when I finally get around to sharing the
grammar, but suffice to say, that having org-adapt-indentation set to true
and putting the leading spaces in the file (instead of doing whatever it is
that doom does by default) induces significant complexity into the
implementation. I would love to see it gone, as I'm sure anyone wanting
to parse org files in future will too.

Best!
Tom

Re: did behaviour of RET change again?

[Greg Minshall]

Tom,

> The other reason I think this is a good idea is because I have been
> working on a formal grammar for the org syntax, and everything would
> be SO much simpler about the implementation after the first pass parse
> if the canonical representation of an Org file did not allow
> significant whitespace (with an exception for plain lists).

possibly i'm misunderstanding, but my sense is that the value of org
adapt indentation doesn't change what you might actually find ("in a
.org file in the wild").  so, whatever its value, your grammar would
have to deal with all cases?

or, and maybe this would make sense, you'd define an "interoperability"
form of .org, that all "wild" .org files could be (programmatically)
converted into, without losing any of their semantics?

(anyway, good luck with that, even with any significant subset of that!)

cheers, Greg

Re: did behaviour of RET change again?

[Tom Gillespie]

> possibly i'm misunderstanding, but my sense is that the value of org
> adapt indentation doesn't change what you might actually find ("in a
> .org file in the wild").  so, whatever its value, your grammar would
> have to deal with all cases?

Yep, we can't magically change all the files out in the wild.
Until I wrote this email out I agreed about the grammar too, but
as it turns out I think there might be a compromise, which is that
a grammar could be allowed to interpret certain types of lines
with leading whitespace as if they were paragraph lines instead of
as say, the start of an org-babel block. That way an interoperable
org spec could be made simpler without preventing e.g. the elisp
implementation from going above and beyond the spect to provide
support for leading whitespace.

My interest in the org-adapt-indentation setting is to try to align what
most users are doing out in the wild with a representation for their
org files that is less ambiguous and more performant (among other
things).

If I had to guess I would say that most new org files have leading
whitespace precisely because org-adapt-indentation is set to t by
default. Getting users to transition their files requires an incentive,
and some users may find that they use org in such a way that they
don't need to.

While writing this email I thought of a reasonable incentive, which is
that only files without leading significant whitespace (i.e. that look like
those that are authored with org-adapt-indentation nil) have specified
behavior for things like org-babel blocks. This would allow best effort
by the elisp org-mode implementation to allow users who don't care
about interoperability to continue as they have been doing, while also
providing clear guidance for what users must do if they want
consistent behavior on other tools that consume org formatted files.
This is critical for keeping the org spec from becoming overly complex.

The first step would thus be to reduce the rate at which new org files
are created with leading whitespace by changing org-adapt-indentation
to be nil by default.

The second step would be to give users a clear way forward to safely
normalizing their files. Org has a habit of reindenting subsets of files
from time to time, but in general doing a complete switch to have no
significant whitespace can be risky.

The third step would be to let the incentives and needs of users
play out over time, but users would generally probably be happier
because by default they would be in the "my files are portable and
generally interpretable without me having to do any additional work"
state instead of the "why did no one tell me the defaults weren't
portable" state.

> or, and maybe this would make sense, you'd define an "interoperability"
> form of .org, that all "wild" .org files could be (programmatically)
> converted into, without losing any of their semantics?

Yep exactly. For many use cases the cost of stripping the whitespace
that corresponds to heading level is not unreasonable, e.g. since it
would only have to be done once. However, if you are writing another
org implementation, then every single time you parse a heading and
its accompanying section you have to strip the whitespace, and that
will happen every single time a user modifies the file, which starts to
get expensive. Alternately you could implement it so that everything
is stripped once and then you keep a version in memory that the user
edits which doesn't have leading whitespace, but then every time you
save you have to splice it all back in to preserve the roundtrip.

This also doesn't even begin to deal with the fact that users can create
malformed org files that can have lines that are less than the expected
significant indentation. This becomes a complete nightmare when trying
to come up with some rational way of dealing with org babel blocks for
languages like python where there is significant whitespace.

Consider the horror if trying to specify the correct behavior in a situation
like this. Better just to declare it a malformed babel block and move on.
Unfortunately you can't always detect that such things are malformed
during the first pass of parsing because you have to count the number of
spaces.

#+begin_src org
# -*- org-adapt-indentation: t -*-
,*** Oh No
    ,#+begin_src python
  class What:
      have = 'you'
      done = 'now'
,#+end_src
#+end_src

In order to ensure that org files can be reliably interpreted this either
means that the specification for handling cases like this becomes
extremely complex, or the spec can say "you can have leading
whitespace, but nasal demons may come for you, there is only
specified behavior if you do not have leading whitespace."

Having unspecified behavior in cases like that would mean that an
implementation could be fully compliant if it were to treat any
#+begin_src lang line that started with whitespace as if it were
a paragraph instead of a babel block. I suspect that this will be
the best way forward.

> (anyway, good luck with that, even with any significant subset of that!)

Thanks, and thanks for the inspiration!
Tom

Re: did behaviour of RET change again?

[Devin Prater]

For me, I don't like my paragraphs indented. That just adds more speech 
from Emacspeak. I'm glad y'all talked about which setting does this so I 
can turn it off.. I mean, I don't know of any reason why paragraphs 
should be indented, but that's just my opinion. Maybe visual appeal, 
looking more like a word processor?

On 12/24/20 12:35 AM, Tom Gillespie wrote:
>> possibly i'm misunderstanding, but my sense is that the value of org
>> adapt indentation doesn't change what you might actually find ("in a
>> .org file in the wild").  so, whatever its value, your grammar would
>> have to deal with all cases?
> Yep, we can't magically change all the files out in the wild.
> Until I wrote this email out I agreed about the grammar too, but
> as it turns out I think there might be a compromise, which is that
> a grammar could be allowed to interpret certain types of lines
> with leading whitespace as if they were paragraph lines instead of
> as say, the start of an org-babel block. That way an interoperable
> org spec could be made simpler without preventing e.g. the elisp
> implementation from going above and beyond the spect to provide
> support for leading whitespace.
>
> My interest in the org-adapt-indentation setting is to try to align what
> most users are doing out in the wild with a representation for their
> org files that is less ambiguous and more performant (among other
> things).
>
> If I had to guess I would say that most new org files have leading
> whitespace precisely because org-adapt-indentation is set to t by
> default. Getting users to transition their files requires an incentive,
> and some users may find that they use org in such a way that they
> don't need to.
>
> While writing this email I thought of a reasonable incentive, which is
> that only files without leading significant whitespace (i.e. that look like
> those that are authored with org-adapt-indentation nil) have specified
> behavior for things like org-babel blocks. This would allow best effort
> by the elisp org-mode implementation to allow users who don't care
> about interoperability to continue as they have been doing, while also
> providing clear guidance for what users must do if they want
> consistent behavior on other tools that consume org formatted files.
> This is critical for keeping the org spec from becoming overly complex.
>
> The first step would thus be to reduce the rate at which new org files
> are created with leading whitespace by changing org-adapt-indentation
> to be nil by default.
>
> The second step would be to give users a clear way forward to safely
> normalizing their files. Org has a habit of reindenting subsets of files
> from time to time, but in general doing a complete switch to have no
> significant whitespace can be risky.
>
> The third step would be to let the incentives and needs of users
> play out over time, but users would generally probably be happier
> because by default they would be in the "my files are portable and
> generally interpretable without me having to do any additional work"
> state instead of the "why did no one tell me the defaults weren't
> portable" state.
>
>> or, and maybe this would make sense, you'd define an "interoperability"
>> form of .org, that all "wild" .org files could be (programmatically)
>> converted into, without losing any of their semantics?
> Yep exactly. For many use cases the cost of stripping the whitespace
> that corresponds to heading level is not unreasonable, e.g. since it
> would only have to be done once. However, if you are writing another
> org implementation, then every single time you parse a heading and
> its accompanying section you have to strip the whitespace, and that
> will happen every single time a user modifies the file, which starts to
> get expensive. Alternately you could implement it so that everything
> is stripped once and then you keep a version in memory that the user
> edits which doesn't have leading whitespace, but then every time you
> save you have to splice it all back in to preserve the roundtrip.
>
> This also doesn't even begin to deal with the fact that users can create
> malformed org files that can have lines that are less than the expected
> significant indentation. This becomes a complete nightmare when trying
> to come up with some rational way of dealing with org babel blocks for
> languages like python where there is significant whitespace.
>
> Consider the horror if trying to specify the correct behavior in a situation
> like this. Better just to declare it a malformed babel block and move on.
> Unfortunately you can't always detect that such things are malformed
> during the first pass of parsing because you have to count the number of
> spaces.
>
> #+begin_src org
> # -*- org-adapt-indentation: t -*-
> ,*** Oh No
>      ,#+begin_src python
>    class What:
>        have = 'you'
>        done = 'now'
> ,#+end_src
> #+end_src
>
> In order to ensure that org files can be reliably interpreted this either
> means that the specification for handling cases like this becomes
> extremely complex, or the spec can say "you can have leading
> whitespace, but nasal demons may come for you, there is only
> specified behavior if you do not have leading whitespace."
>
> Having unspecified behavior in cases like that would mean that an
> implementation could be fully compliant if it were to treat any
> #+begin_src lang line that started with whitespace as if it were
> a paragraph instead of a babel block. I suspect that this will be
> the best way forward.
>
>> (anyway, good luck with that, even with any significant subset of that!)
> Thanks, and thanks for the inspiration!
> Tom
>