Bug: Unexpected behavior marking recurring tasks as DONE

Bug: Unexpected behavior marking recurring tasks as DONE

[Alan Ristow]

I recently updated org from 9.3 (release_9.3) to 9.4.6 (9.4.6-gfdb98a) 
and observed several changes in behavior when marking recurring tasks as 
DONE. I have not found reports of anything similar via Google or the 
mailing list archives, so rather than a bug it might be a package 
conflict or a configuration issue; in any case, I am having a really 
tough time figuring it out and I hope somebody here might be able to 
give me some pointers.

I use straight.el for package management and as part of this update I 
had to switch from using org-plus-contrib to using org-contrib. 
Normally, straight.el would make it easy for me to go back to my old 
configuration -- I have the lockfile with all the package versions -- 
but because of the movement in org repos and the change from 
org-plus-contrib to org-contrib, I am running into difficulty doing 
that, which probably contributes to my difficulties in troubleshooting.


EXPECTED (FORMER) BEHAVIOR

Suppose I have a task that looks like this:

** TODO Daily review
    SCHEDULED: <2021-07-07 Wed .+1d>
    :PROPERTIES:
    :LAST_REPEAT: [2021-07-06 Tue 10:03]
    :END:
    :LOGBOOK:
    - State "DONE"       from "TODO" [2021-07-06 Tue 10:03]
    :END:

I then do the following:

1. Move the cursor to the TODO line and mark it DONE (t-x in agenda
    view, C-t x in the file buffer).
2. The scheduled date is updated to the next recurrence, and the
    current time is recorded in the LAST_MODIFIED property drawer and in
    the LOGBOOK.

After, the task looks like this:

** TODO Daily review
    SCHEDULED: <2021-07-08 Thu .+1d>
    :PROPERTIES:
    :LAST_REPEAT: [2021-07-07 Wed 11:18]
    :END:
    :LOGBOOK:
    - State "DONE"       from "TODO" [2021-07-07 Wed 11:18]
    - State "DONE"       from "TODO" [2021-07-06 Tue 10:03]
    :END:

I get the same result when the task is marked as a habit and also when I 
bulk-process tasks from the agenda view using org-agenda-bulk-action.


ACTUAL (CURRENT) BEHAVIOR

I have observed two changes in behavior since updating to org 9.4.6 and 
org-contrib.

First, if I do exactly the same as above, the time of completion is 
logged twice:

** TODO Daily review
    SCHEDULED: <2021-07-08 Thu .+1d>
    :PROPERTIES:
    :LAST_REPEAT: [2021-07-07 Wed 11:18]
    :END:
    :LOGBOOK:
    - State "DONE"       from "TODO" [2021-07-07 Wed 11:18]
    - State "DONE"       from "TODO" [2021-07-07 Wed 11:18]
    - State "DONE"       from "TODO" [2021-07-06 Tue 10:03]
    :END:

Second, if I bulk-process a habit via org-agenda-bulk-action, the task 
is simply marked DONE. Bot the recurrence and the LAST_REPEAT field are 
ignored, but the time stamp is only entered into the LOGBOOK once:

** DONE Walk
    CLOSED: [2021-07-07 Wed 11:26] SCHEDULED: <2021-07-07 Wed .+1d>
    :PROPERTIES:
    :STYLE:  habit
    :LAST_REPEAT: [2021-07-06 Tue 15:33]
    :END:
    :LOGBOOK:
    - State "DONE"       from "TODO" [2021-07-07 Wed 11:26]
    - State "DONE"       from "TODO" [2021-07-06 Tue 15:33]
    :END:


MY CONFIGURATION

The extensions to org that I load are: org-super-agenda, org-superstar, 
org-capture, org-tempo, org-checklist, org-habit, helm-org-rifle, 
org-drill, ox-pandoc, org-make-toc, org-ql, org-roam, org-journal, 
org-ref, org-ref-helm-bibtex, and org-roam-bibtex. All are the latest 
versions accessible to straight.el.

My init.el includes the following:

   (setq org-log-done 'time
         org-log-redeadline 'time
         org-log-reschedule 'time
         org-log-into-drawer t
org-log-state-notes-insert-after-drawers nil)
   (setq org-todo-keywords
         '((sequence "TODO(t)" "NEXT(n)" "STARTED(s)" "WAITING(w@/!)" 
"|" "DONE(x!)" "DELEGATED(d@)")
           (sequence "DEFERRED(f@/!)" "INACTIVE(i@/!)" "|" "CANCELED(c@)")))


Best regards,
Alan

Re: Bug: Unexpected behavior marking recurring tasks as DONE

[Alan Ristow]

I think I have solved this, though I am not really sure why the solution 
works. As noted previously, my org-todo-keywords are defined as follows:

   (setq org-todo-keywords
         '((sequence "TODO(t)" "NEXT(n)" "STARTED(s)" "WAITING(w@/!)" 
"|" "DONE(x!)" "DELEGATED(d@)")
           (sequence "DEFERRED(f@/!)" "INACTIVE(i@/!)" "|" "CANCELED(c@)")))

By trial-and-error with a minimal init.el file, I discovered that 
removing the exclamation point from the DONE fast access key solved the 
problem, at least in the testing I have done so far (with both the 
minimal init.el and my full configuration). That is to say, the new 
setting is as follows:

   (setq org-todo-keywords
         '((sequence "TODO(t)" "NEXT(n)" "STARTED(s)" "WAITING(w@/!)" 
"|" "DONE(x)" "DELEGATED(d@)")
           (sequence "DEFERRED(f@/!)" "INACTIVE(i@/!)" "|" "CANCELED(c@)")))

I have not touched this variable in many years, and as far as I can tell 
the documentation on the use of "!" is the same as it has been since the 
day I started using org. Even without the "!", the time I completed the 
task still gets logged in the LOGBOOK, but now only once instead of 
being duplicated.

I also have this in my init.el:

   (setq org-log-done 'time)

Could this be conflicting with "DONE(x!)" in org-todo-keywords?

Best regards,

Alan


On 7/7/21 11:50 AM, Alan Ristow wrote:
> I recently updated org from 9.3 (release_9.3) to 9.4.6 (9.4.6-gfdb98a) 
> and observed several changes in behavior when marking recurring tasks 
> as DONE. I have not found reports of anything similar via Google or 
> the mailing list archives, so rather than a bug it might be a package 
> conflict or a configuration issue; in any case, I am having a really 
> tough time figuring it out and I hope somebody here might be able to 
> give me some pointers.
>
> I use straight.el for package management and as part of this update I 
> had to switch from using org-plus-contrib to using org-contrib. 
> Normally, straight.el would make it easy for me to go back to my old 
> configuration -- I have the lockfile with all the package versions -- 
> but because of the movement in org repos and the change from 
> org-plus-contrib to org-contrib, I am running into difficulty doing 
> that, which probably contributes to my difficulties in troubleshooting.
>
>
> EXPECTED (FORMER) BEHAVIOR
>
> Suppose I have a task that looks like this:
>
> ** TODO Daily review
>    SCHEDULED: <2021-07-07 Wed .+1d>
>    :PROPERTIES:
>    :LAST_REPEAT: [2021-07-06 Tue 10:03]
>    :END:
>    :LOGBOOK:
>    - State "DONE"       from "TODO" [2021-07-06 Tue 10:03]
>    :END:
>
> I then do the following:
>
> 1. Move the cursor to the TODO line and mark it DONE (t-x in agenda
>    view, C-t x in the file buffer).
> 2. The scheduled date is updated to the next recurrence, and the
>    current time is recorded in the LAST_MODIFIED property drawer and in
>    the LOGBOOK.
>
> After, the task looks like this:
>
> ** TODO Daily review
>    SCHEDULED: <2021-07-08 Thu .+1d>
>    :PROPERTIES:
>    :LAST_REPEAT: [2021-07-07 Wed 11:18]
>    :END:
>    :LOGBOOK:
>    - State "DONE"       from "TODO" [2021-07-07 Wed 11:18]
>    - State "DONE"       from "TODO" [2021-07-06 Tue 10:03]
>    :END:
>
> I get the same result when the task is marked as a habit and also when 
> I bulk-process tasks from the agenda view using org-agenda-bulk-action.
>
>
> ACTUAL (CURRENT) BEHAVIOR
>
> I have observed two changes in behavior since updating to org 9.4.6 
> and org-contrib.
>
> First, if I do exactly the same as above, the time of completion is 
> logged twice:
>
> ** TODO Daily review
>    SCHEDULED: <2021-07-08 Thu .+1d>
>    :PROPERTIES:
>    :LAST_REPEAT: [2021-07-07 Wed 11:18]
>    :END:
>    :LOGBOOK:
>    - State "DONE"       from "TODO" [2021-07-07 Wed 11:18]
>    - State "DONE"       from "TODO" [2021-07-07 Wed 11:18]
>    - State "DONE"       from "TODO" [2021-07-06 Tue 10:03]
>    :END:
>
> Second, if I bulk-process a habit via org-agenda-bulk-action, the task 
> is simply marked DONE. Bot the recurrence and the LAST_REPEAT field 
> are ignored, but the time stamp is only entered into the LOGBOOK once:
>
> ** DONE Walk
>    CLOSED: [2021-07-07 Wed 11:26] SCHEDULED: <2021-07-07 Wed .+1d>
>    :PROPERTIES:
>    :STYLE:  habit
>    :LAST_REPEAT: [2021-07-06 Tue 15:33]
>    :END:
>    :LOGBOOK:
>    - State "DONE"       from "TODO" [2021-07-07 Wed 11:26]
>    - State "DONE"       from "TODO" [2021-07-06 Tue 15:33]
>    :END:
>
>
> MY CONFIGURATION
>
> The extensions to org that I load are: org-super-agenda, 
> org-superstar, org-capture, org-tempo, org-checklist, org-habit, 
> helm-org-rifle, org-drill, ox-pandoc, org-make-toc, org-ql, org-roam, 
> org-journal, org-ref, org-ref-helm-bibtex, and org-roam-bibtex. All 
> are the latest versions accessible to straight.el.
>
> My init.el includes the following:
>
>   (setq org-log-done 'time
>         org-log-redeadline 'time
>         org-log-reschedule 'time
>         org-log-into-drawer t
> org-log-state-notes-insert-after-drawers nil)
>   (setq org-todo-keywords
>         '((sequence "TODO(t)" "NEXT(n)" "STARTED(s)" "WAITING(w@/!)" 
> "|" "DONE(x!)" "DELEGATED(d@)")
>           (sequence "DEFERRED(f@/!)" "INACTIVE(i@/!)" "|" 
> "CANCELED(c@)")))
>
>
> Best regards,
> Alan
>
>

Re: Bug: Unexpected behavior marking recurring tasks as DONE

[Bhavin Gandhi]

Hello Alan,

Thank you for sharing a detailed description.

On Wed, 7 Jul 2021 at 23:52, Alan Ristow wrote:
> […]
> >
> > First, if I do exactly the same as above, the time of completion is
> > logged twice:
> >
> > ** TODO Daily review
> >    SCHEDULED: <2021-07-08 Thu .+1d>
> >    :PROPERTIES:
> >    :LAST_REPEAT: [2021-07-07 Wed 11:18]
> >    :END:
> >    :LOGBOOK:
> >    - State "DONE"       from "TODO" [2021-07-07 Wed 11:18]
> >    - State "DONE"       from "TODO" [2021-07-07 Wed 11:18]
> >    - State "DONE"       from "TODO" [2021-07-06 Tue 10:03]
> >    :END:
> >

I think this is same issue as reported in this bug report:
https://orgmode.org/list/87o8c8xp9b.fsf@gmail.com/

> > Second, if I bulk-process a habit via org-agenda-bulk-action, the task
> > is simply marked DONE. Bot the recurrence and the LAST_REPEAT field
> > are ignored, but the time stamp is only entered into the LOGBOOK once:
> >
> > ** DONE Walk
> >    CLOSED: [2021-07-07 Wed 11:26] SCHEDULED: <2021-07-07 Wed .+1d>
> >    :PROPERTIES:
> >    :STYLE:  habit
> >    :LAST_REPEAT: [2021-07-06 Tue 15:33]
> >    :END:
> >    :LOGBOOK:
> >    - State "DONE"       from "TODO" [2021-07-07 Wed 11:26]
> >    - State "DONE"       from "TODO" [2021-07-06 Tue 15:33]
> >    :END:

I was not able to reproduce this correctly, I will try to reproduce it
again later.

Re: Bug: Unexpected behavior marking recurring tasks as DONE

[Alan Ristow]

[-- Attachment #1: Type: text/plain, Size: 2212 bytes --]

Hi Bhavin,

On 7/8/21 8:19 PM, Bhavin Gandhi wrote:
> Hello Alan,
>
> Thank you for sharing a detailed description.
> [...]
> I think this is same issue as reported in this bug report:
> https://orgmode.org/list/87o8c8xp9b.fsf@gmail.com/

Thank you for the pointer to this -- clearly I missed it in my initial 
search of the mailing list archive, but I agree it appears to be the 
same issue. I'm glad to know that somebody has reproduced it.

>>> Second, if I bulk-process a habit via org-agenda-bulk-action, the task
>>> is simply marked DONE. Bot the recurrence and the LAST_REPEAT field
>>> are ignored, but the time stamp is only entered into the LOGBOOK once:
>>>
>>> ** DONE Walk
>>>     CLOSED: [2021-07-07 Wed 11:26] SCHEDULED: <2021-07-07 Wed .+1d>
>>>     :PROPERTIES:
>>>     :STYLE:  habit
>>>     :LAST_REPEAT: [2021-07-06 Tue 15:33]
>>>     :END:
>>>     :LOGBOOK:
>>>     - State "DONE"       from "TODO" [2021-07-07 Wed 11:26]
>>>     - State "DONE"       from "TODO" [2021-07-06 Tue 15:33]
>>>     :END:
> I was not able to reproduce this correctly, I will try to reproduce it
> again later.

I had a difficult time reproducing this one reliably. If I remember 
correctly, I could only reproduce it when I set a key binding to 
org-store-link. It is a binding I never use, I only set it because it 
was in seemingly every org config on the planet when I first started 
with orgmode, and I certainly don't explicitly invoke it when processing 
in bulk.

Looking back at my last emails, I realize I have submitted a pretty 
lousy bug report -- sorry for that! The init.el I used for debugging is 
attached, though it relies on straight and use-package for reasons of 
time and convenience. The org settings as-written have reproduced both 
behaviors reliably for me so far, and removing the "!" from 
org-todo-keywords has fixed them both. I am a bit pressed for time at 
the moment, but later today or over the weekend I will put together a 
more vanilla init.el that hopefully reproduces both problems (or reveals 
a package conflict...). I have also attached my test file, though it is 
not substantively different from the one you used yourself in the 
previous thread.

Best regards,

Alan


[-- Attachment #2: init.el --]
[-- Type: text/x-emacs-lisp, Size: 1198 bytes --]

;;; -*- lexical-binding: t -*-

(defvar bootstrap-version)
(let ((bootstrap-file
       (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
      (bootstrap-version 5))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
        (url-retrieve-synchronously
         "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
         'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp)))
  (load bootstrap-file nil 'nomessage))

(straight-use-package 'use-package)
(setq straight-use-package-by-default t)

(eval-when-compile
  (require 'use-package))
(require 'bind-key)

(setq use-package-always-defer t)

(setq use-package-verbose nil)

(use-package org
  :defer nil
  :bind (("C-c l" . org-store-link)
	 ("C-c a" . org-agenda))
  :mode ("\\.\\(org\\|org_archive\\)$" . org-mode)
  :init
  (setq org-directory "~/org-dev"
        org-agenda-files (list org-directory))
  :custom
  (org-log-done 'time)
  (org-log-redeadline 'time)
  (org-log-reschedule 'time)
  (org-log-into-drawer t)
  (org-log-state-notes-insert-after-drawers nil)
  (org-todo-keywords
   '((sequence "TODO(t)" "|" "DONE(x!)"))))

[-- Attachment #3: test.org --]
[-- Type: application/vnd.lotus-organizer, Size: 54 bytes --]

Re: Bug: Unexpected behavior marking recurring tasks as DONE

[Allen Li]

On Fri, Jul 9, 2021 at 7:49 AM Alan Ristow <alan@ristow.info> wrote:
> On 7/8/21 8:19 PM, Bhavin Gandhi wrote:
> >>> Second, if I bulk-process a habit via org-agenda-bulk-action, the task
> >>> is simply marked DONE. Bot the recurrence and the LAST_REPEAT field
> >>> are ignored, but the time stamp is only entered into the LOGBOOK once:
> >>>
> >>> ** DONE Walk
> >>>     CLOSED: [2021-07-07 Wed 11:26] SCHEDULED: <2021-07-07 Wed .+1d>
> >>>     :PROPERTIES:
> >>>     :STYLE:  habit
> >>>     :LAST_REPEAT: [2021-07-06 Tue 15:33]
> >>>     :END:
> >>>     :LOGBOOK:
> >>>     - State "DONE"       from "TODO" [2021-07-07 Wed 11:26]
> >>>     - State "DONE"       from "TODO" [2021-07-06 Tue 15:33]
> >>>     :END:
> > I was not able to reproduce this correctly, I will try to reproduce it
> > again later.
>
> I had a difficult time reproducing this one reliably. If I remember
> correctly, I could only reproduce it when I set a key binding to
> org-store-link. It is a binding I never use, I only set it because it
> was in seemingly every org config on the planet when I first started
> with orgmode, and I certainly don't explicitly invoke it when processing
> in bulk.

I have also experienced repeating tasks getting marked as DONE
(not because I explicitly stopped the repetition).  I haven't able to
reproduce it reliably either, as it's seemingly random.