[BUG] Adding note to heading without newline at the end

[BUG] Adding note to heading without newline at the end

[Tor Kringeland]

Consider the following buffer

#+begin_example
* test<POINT>
#+end_example

without a newline at the end.  Calling `org-add-note' will result in an
error and the note being placed /before/ the heading.

Re: [BUG] Adding note to heading without newline at the end

[Ihor Radchenko]

Tor Kringeland <tor.kringeland@ntnu.no> writes:

> Consider the following buffer
>
> #+begin_example
> * test<POINT>
> #+end_example
>
> without a newline at the end.  Calling `org-add-note' will result in an
> error and the note being placed /before/ the heading.

I am unable to reproduce.
Can you reproduce the problem starting from emacs -Q?
See https://orgmode.org/manual/Feedback.html

Best,
Ihor

Re: [BUG] Adding note to heading without newline at the end

[Tor Kringeland]

Ihor Radchenko <yantar92@gmail.com> writes:

> Can you reproduce the problem starting from emacs -Q?

I did.  The bug only occurs if there is no newline at the end.  So for
example if you open a new buffer and don't save it, the bug should
occur.

Similarly, in my own config (where I have enabled org-log-into-drawer),
the note gets inserted below as is correct but the last letter of the
heading gets inserted at the end, like:

#+begin_example
* tes
  :LOGBOOK:
  - Note taken on [2022-06-20 Mon 17:07]
  :END:
t
#+end_example

(it doesn't seem to matter /where/ in the heading you are when you
insert the note --- it's always the last letter falling down).

I haven't been able to reproduce this with emacs -Q yet, though, but I
suspect the source of the error is the same.  Anyway, running Emacs as
before with -Q and inserting a note, the result is:

#+begin_example
- Note taken on [2022-06-20 Mon 17:07]
* test
#+end_example

Again, if there is a newline at the end (as would happen if you first
save the file) the bug doesn't occur.

I'm using a recent build of Emacs 29.

Re: [BUG] Adding note to heading without newline at the end

[Samuel Banya]

Unable to reproduce this bug with 'emacs -Q -L ./lisp -l org' on the
latest version of Org Mode

Re: [BUG] Adding note to heading without newline at the end

[Tor Kringeland]

Samuel Banya <sbanya@fastmail.com> writes:

> Unable to reproduce this bug with 'emacs -Q -L ./lisp -l org' on the
> latest version of Org Mode

Hmm ... this occurs to me both in Org 9.5 and in the latest Org version
on Emacs 29.  The error message I get says

  Before first headline at position 1 in buffer test.org

However if there is a heading above the one I try to insert the note for
(and the current heading is at the last line in the buffer, with no
newline at the end), I do not get an error message, but the same bug
occurs.

FWIW I get the same error message if I try to insert a note in an Org
buffer without headings.  Maybe the heading isn't properly registered
when it is at the last line in the buffer (again without a newline at
the end)?

[PATCH] Re: [BUG] Adding note to heading without newline at the end

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

Tor Kringeland <tor.kringeland@ntnu.no> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Can you reproduce the problem starting from emacs -Q?
>
> I did.  The bug only occurs if there is no newline at the end.  So for
> example if you open a new buffer and don't save it, the bug should
> occur.

Aha. Not saving is an important piece of information.
(said the person with compulsive saving syndrome)

Can you test the attached patch?

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-log-beginning-Fix-for-headline-at-eob-with-no-tr.patch --]
[-- Type: text/x-patch, Size: 1732 bytes --]

From 7c40a003166c9ab8f5f034b8bb6d506ed2b8b62f Mon Sep 17 00:00:00 2001
Message-Id: <7c40a003166c9ab8f5f034b8bb6d506ed2b8b62f.1655778085.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Tue, 21 Jun 2022 10:18:58 +0800
Subject: [PATCH] org-log-beginning: Fix for headline at eob with no trailing
 newline

* lisp/org.el (org-log-beginning): Fix edge case when there is a
headline at the end of buffer and that headline does not have a
trailing newline.

Fixes https://orgmode.org/list/m24k0ffjyd.fsf@ntnu.no
---
 lisp/org.el | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index 220210992..78f51f9ad 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -10121,12 +10121,17 @@ (defun org-log-beginning (&optional create)
 	   (end-of-line -1))))
       (t
        (org-end-of-meta-data org-log-state-notes-insert-after-drawers)
-       (skip-chars-forward " \t\n")
-       (beginning-of-line)
-       (unless org-log-states-order-reversed
-	 (org-skip-over-state-notes)
-	 (skip-chars-backward " \t\n")
-	 (beginning-of-line 2)))))
+       (let ((endpos (point)))
+         (skip-chars-forward " \t\n")
+         (beginning-of-line)
+         (unless org-log-states-order-reversed
+	   (org-skip-over-state-notes)
+	   (skip-chars-backward " \t\n")
+	   (beginning-of-line 2))
+         ;; When current headline is at the end of buffer and does not
+         ;; end with trailing newline the above can move to the
+         ;; beginning of the headline.
+         (when (< (point) endpos)) (goto-char endpos)))))
    (if (bolp) (point) (line-beginning-position 2))))
 
 (defun org-add-log-setup (&optional purpose state prev-state how extra)
-- 
2.35.1

Re: [PATCH] Re: [BUG] Adding note to heading without newline at the end

[Tor Kringeland]

Ihor Radchenko <yantar92@gmail.com> writes:

> Aha. Not saving is an important piece of information.
> (said the person with compulsive saving syndrome)

Thanks!  This fixes the bug (which was present in both Org 9.5 and 9.6)
for me.  However, my original bug, which is only present in Org 9.6, is
still there.  Do the same thing but set org-log-into-drawer to t.  Then

#+begin_example
* test<POINT>
#+end_example

becomes

#+begin_example
* tes
:LOGBOOK:
- Note taken on [2022-06-21 Tue 12:55]
:END:
t<POINT>
#+end_example

This bug also only happens if there is no trailing newline in the
buffer.

Re: [PATCH] Re: [BUG] Adding note to heading without newline at the end

[Tim Cross]

Tor Kringeland <tor.kringeland@ntnu.no> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Aha. Not saving is an important piece of information.
>> (said the person with compulsive saving syndrome)
>
> Thanks!  This fixes the bug (which was present in both Org 9.5 and 9.6)
> for me.  However, my original bug, which is only present in Org 9.6, is
> still there.  Do the same thing but set org-log-into-drawer to t.  Then
>
> #+begin_example
> * test<POINT>
> #+end_example
>
>
> becomes
>
> #+begin_example
> * tes
> :LOGBOOK:
> - Note taken on [2022-06-21 Tue 12:55]
> :END:
> t<POINT>
> #+end_example
>
> This bug also only happens if there is no trailing newline in the
> buffer.

Not sure if this makes any difference wrt your bug, but wanted to point
out that your test example block is non-conforming. If you have a line
starting with an '*', you have ot proceed it with a comma to escape the
meaning of the start. If you use org-edit-special to edit the block
contents, the escaping is done for you automatically. Running org-lint
is also useful when you think you might have a bug as it will ensure you
have a relatively clean and correct input file. 

Re: [PATCH] Re: [BUG] Adding note to heading without newline at the end

[Ihor Radchenko]

Tor Kringeland <tor.kringeland@ntnu.no> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Aha. Not saving is an important piece of information.
>> (said the person with compulsive saving syndrome)
>
> Thanks!  This fixes the bug (which was present in both Org 9.5 and 9.6)
> for me.  However, my original bug, which is only present in Org 9.6, is
> still there.  Do the same thing but set org-log-into-drawer to t.  Then
> ...

Confirmed.

Samuel, do you want to try fixing this?
It should be fairly easy to debug.

Best,
Ihor

Re: [PATCH] Re: [BUG] Adding note to heading without newline at the end

[Tor Kringeland]

Ihor Radchenko <yantar92@gmail.com> writes:

> Confirmed.
>
> Samuel, do you want to try fixing this?
> It should be fairly easy to debug.

Nice.  But these are distinct bugs, it seems like.  The one you sent a
patch for earlier fixes the former bug (which is also present in Org
9.5).  While my original problem is only present in Org 9.6.

Re: [PATCH] Re: [BUG] Adding note to heading without newline at the end

[Samuel Banya]

[-- Attachment #1: Type: text/plain, Size: 591 bytes --]

Hey Ihor,

I can check it out on the weekend, sounds like fun.

Will step through the debugger to see what's up and get back to you on this, thanks for tagging me on this.

On Wed, Jun 22, 2022, at 10:28 AM, Tor Kringeland wrote:
> Ihor Radchenko <yantar92@gmail.com> writes:
> 
> > Confirmed.
> >
> > Samuel, do you want to try fixing this?
> > It should be fairly easy to debug.
> 
> Nice.  But these are distinct bugs, it seems like.  The one you sent a
> patch for earlier fixes the former bug (which is also present in Org
> 9.5).  While my original problem is only present in Org 9.6.

[-- Attachment #2: Type: text/html, Size: 1087 bytes --]

Re: [PATCH] Re: [BUG] Adding note to heading without newline at the end

[Samuel Banya]

[-- Attachment #1.1: Type: text/plain, Size: 1187 bytes --]

Hey everyone,

I'm using Emacs 28.1 on Manjaro, and pulled the latest version of Org Mode from the git repo.

I'm still unable to reproduce this as I basically just opened up a new .org file and did Tom's exact steps. I was able to add a blank note without receiving an error at the exact cursor point that was mentioned.

Ihor, can you take a look at the .mkv video I attached of my test and let me know what you did to be able to reproduce this because I'm not observing any errors on my machine.

Thanks,

Sam

On Thu, Jun 23, 2022, at 11:59 AM, Samuel Banya wrote:
> Hey Ihor,
> 
> I can check it out on the weekend, sounds like fun.
> 
> Will step through the debugger to see what's up and get back to you on this, thanks for tagging me on this.
> 
> On Wed, Jun 22, 2022, at 10:28 AM, Tor Kringeland wrote:
>> Ihor Radchenko <yantar92@gmail.com> writes:
>> 
>> > Confirmed.
>> >
>> > Samuel, do you want to try fixing this?
>> > It should be fairly easy to debug.
>> 
>> Nice.  But these are distinct bugs, it seems like.  The one you sent a
>> patch for earlier fixes the former bug (which is also present in Org
>> 9.5).  While my original problem is only present in Org 9.6.
> 

[-- Attachment #1.2: Type: text/html, Size: 1912 bytes --]

[-- Attachment #2: orgAddNoteBugVideo-2022-06-26_19.25.54.mkv --]
[-- Type: video/x-matroska, Size: 5378592 bytes --]

Re: [PATCH] Re: [BUG] Adding note to heading without newline at the end

[Ihor Radchenko]

"Samuel Banya" <sbanya@fastmail.com> writes:

> Ihor, can you take a look at the .mkv video I attached of my test and let me know what you did to be able to reproduce this because I'm not observing any errors on my machine.

You misunderstood the email formatting.

#+begin_example
* test<POINT>
#+end_example

actually meant the following:

1. Create a new org file
2. Type (literally) the following:
* test
3. You will get a file containing "* test" exactly, no newline at the
   end
4. Do _not_ save the file
5. Run C-c C-z and add some note

Best,
Ihor

Re: [PATCH] Re: [BUG] Adding note to heading without newline at the end

[Samuel Banya]

[-- Attachment #1: Type: text/plain, Size: 2709 bytes --]

Gotcha, after this great advice, I am also able to get the following result:
```
- Note taken on [2022-06-26 Sun 22:31] \\
  Adding test note
* test
```

However, even after toggling 'M-x toggle-debug-on-error', it didn't enter the debugger since I guess this would be considered a 'user error' according to the '*Messages*' buffer present:
```
user-error: Before first headline at position 43 in buffer orgModeNoteBug.org
```
I checked 'org.el' which has the related 'org-add-note' function present.

This appears to call the 'org-add-log-setup' function, and passes the 'note' argument to it.

I then took a look at the 'org-add-log-setup' function.

From the bug's nature itself, it most likely is moving the cursor incorrectly to the previous line above it, when it should be really doing it on the line AFTER it.

This to me is probably the 'moving' section that should be modified:
```
  (move-marker org-log-note-marker (point))
```

It's probably the position that needs to be recalculated or adjusted accordingly. 

I used 'C-h f' and found that 'move-marker' is just an alias for 'set-marker' which is in the Emacs codebase itself.

I think the issue is HOW we are calling it here.

Can you give me a few pointers in terms of how we can maybe force it to go to the next line instead of adjusting it based on the point present?

Thanks,

Sam


On Sun, Jun 26, 2022, at 7:32 PM, Samuel Banya wrote:
> Hey everyone,
> 
> I'm using Emacs 28.1 on Manjaro, and pulled the latest version of Org Mode from the git repo.
> 
> I'm still unable to reproduce this as I basically just opened up a new .org file and did Tom's exact steps. I was able to add a blank note without receiving an error at the exact cursor point that was mentioned.
> 
> Ihor, can you take a look at the .mkv video I attached of my test and let me know what you did to be able to reproduce this because I'm not observing any errors on my machine.
> 
> Thanks,
> 
> Sam
> 
> On Thu, Jun 23, 2022, at 11:59 AM, Samuel Banya wrote:
>> Hey Ihor,
>> 
>> I can check it out on the weekend, sounds like fun.
>> 
>> Will step through the debugger to see what's up and get back to you on this, thanks for tagging me on this.
>> 
>> On Wed, Jun 22, 2022, at 10:28 AM, Tor Kringeland wrote:
>>> Ihor Radchenko <yantar92@gmail.com> writes:
>>> 
>>> > Confirmed.
>>> >
>>> > Samuel, do you want to try fixing this?
>>> > It should be fairly easy to debug.
>>> 
>>> Nice.  But these are distinct bugs, it seems like.  The one you sent a
>>> patch for earlier fixes the former bug (which is also present in Org
>>> 9.5).  While my original problem is only present in Org 9.6.
>> 
> 
> 
> *Attachments:*
>  * orgAddNoteBugVideo-2022-06-26_19.25.54.mkv

[-- Attachment #2: Type: text/html, Size: 4045 bytes --]

Re: [PATCH] Re: [BUG] Adding note to heading without newline at the end

[Ihor Radchenko]

"Samuel Banya" <sbanya@fastmail.com> writes:
> I checked 'org.el' which has the related 'org-add-note' function present.
>
> This appears to call the 'org-add-log-setup' function, and passes the 'note' argument to it.
>
> I then took a look at the 'org-add-log-setup' function.
>
> From the bug's nature itself, it most likely is moving the cursor incorrectly to the previous line above it, when it should be really doing it on the line AFTER it.
>
> This to me is probably the 'moving' section that should be modified:
> ```
>   (move-marker org-log-note-marker (point))
> ```
>
> It's probably the position that needs to be recalculated or adjusted accordingly. 

The offending function is different. This piece of code is admittedly
slightly tricky - org-add-log-setup does not determine where the note
will be inserted. It defers note taking to org-add-log-note, which, in
turn defers saving the note text to the time user presses C-c C-c in the
note buffer.

The function doing the actual insertion is org-store-log-note and the
function determining where to insert the note is org-log-beginning.

Note that the patch I attached in my earlier message contains the fix.
What is left is studying a similar edge case reported in the reply to my
patch:

Tor Kringeland <tor.kringeland@ntnu.no>:
>> Thanks!  This fixes the bug (which was present in both Org 9.5 and 9.6)
>> for me.  However, my original bug, which is only present in Org 9.6, is
>> still there.  Do the same thing but set org-log-into-drawer to t.  Then

That is, you need to follow the same steps, but set org-log-into-drawer
to t before creating a note.

Best,
Ihor

Re: [PATCH] Re: [BUG] Adding note to heading without newline at the end

[Samuel Banya]

[-- Attachment #1: Type: text/plain, Size: 2645 bytes --]

So I went through the same steps as before:
1. Pulled the latest Emacs Org Mode

2. Ran the following command in a terminal to get a clean Emacs without a config:
emacs -Q -L ./lisp -l org

3. Create a new org file

4.. Type (literally) the following:
* test

5. You will get a file containing "* test" exactly, no newline at the
   end

6. Do __not__ save the file

7. I then set the 'org-log-into-drawer' variable to true by evaluating the following environment variable with 'M-:':
'M-:' (setq org-log-into-drawer t)

This returned 't' in the minibuffer.

8.  I then used C-c C-c, and received the following example output:
```
* tes
:LOGBOOK:
- Note taken on [2022-06-27 Mon 22:38] \\
  Adding note with org-log-into-drawer enabled
:END:
t
```

Let me know if this is the desired output in that respect, or if this fulfills the weird edge case present.

Thanks,

Sam


On Mon, Jun 27, 2022, at 6:18 AM, Ihor Radchenko wrote:
> "Samuel Banya" <sbanya@fastmail.com> writes:
> > I checked 'org.el' which has the related 'org-add-note' function present.
> >
> > This appears to call the 'org-add-log-setup' function, and passes the 'note' argument to it.
> >
> > I then took a look at the 'org-add-log-setup' function.
> >
> > From the bug's nature itself, it most likely is moving the cursor incorrectly to the previous line above it, when it should be really doing it on the line AFTER it.
> >
> > This to me is probably the 'moving' section that should be modified:
> > ```
> >   (move-marker org-log-note-marker (point))
> > ```
> >
> > It's probably the position that needs to be recalculated or adjusted accordingly. 
> 
> The offending function is different. This piece of code is admittedly
> slightly tricky - org-add-log-setup does not determine where the note
> will be inserted. It defers note taking to org-add-log-note, which, in
> turn defers saving the note text to the time user presses C-c C-c in the
> note buffer.
> 
> The function doing the actual insertion is org-store-log-note and the
> function determining where to insert the note is org-log-beginning.
> 
> Note that the patch I attached in my earlier message contains the fix.
> What is left is studying a similar edge case reported in the reply to my
> patch:
> 
> Tor Kringeland <tor.kringeland@ntnu.no>:
> >> Thanks!  This fixes the bug (which was present in both Org 9.5 and 9.6)
> >> for me.  However, my original bug, which is only present in Org 9.6, is
> >> still there.  Do the same thing but set org-log-into-drawer to t.  Then
> 
> That is, you need to follow the same steps, but set org-log-into-drawer
> to t before creating a note.
> 
> Best,
> Ihor
> 

[-- Attachment #2: Type: text/html, Size: 4094 bytes --]

Re: [PATCH] Re: [BUG] Adding note to heading without newline at the end

[Ihor Radchenko]

"Samuel Banya" <sbanya@fastmail.com> writes:

> 8.  I then used C-c C-c, and received the following example output:
> ```
> * tes
> :LOGBOOK:
> - Note taken on [2022-06-27 Mon 22:38] \\
>   Adding note with org-log-into-drawer enabled
> :END:
> t
> ```
>
> Let me know if this is the desired output in that respect, or if this fulfills the weird edge case present.

Note that the headline "* test" is split into "* tes...t". It is
certainly not desired.

I see the same issue on my side. I also suspect that the offending
functions are the same (but you may need to check).

Best,
Ihor

Re: [PATCH] Re: [BUG] Adding note to heading without newline at the end

[Ihor Radchenko]

Tor Kringeland <tor.kringeland@ntnu.no> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Aha. Not saving is an important piece of information.
>> (said the person with compulsive saving syndrome)
>
> Thanks!  This fixes the bug (which was present in both Org 9.5 and 9.6)
> for me.  However, my original bug, which is only present in Org 9.6, is
> still there.  Do the same thing but set org-log-into-drawer to t.  Then

Fixed on main now.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=8ec328e827edf67a09b6612ae32aba79ceb98e9f
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=f8d740f707ceab1e865fd1745ffaaa531b9fdd0e

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92