[BUG] org-agenda generation missing some entries

[BUG] org-agenda generation missing some entries

[Christopher M. Miles]

[-- Attachment #1.1: Type: text/plain, Size: 1167 bytes --]



After git bisect, I found the bad commit:

#+begin_example
0ef88e2d9 * | bad @ org-agenda.el/org-agenda-get-scheduled: Use cache
#+end_example

after org-agenda generation, lot of entries not displayed. Then I found lot of errors in =*Messages*= buffer.

#+begin_example
Wrong type argument: stringp, (timestamp (:type active :raw-value "<2021-02-28 Sun>" :year-start 2021 :month-start 2 :day-start 28 :hour-start nil :minute-start nil :year-end 2021 :month-end 2 :day-end 28 :hour-end nil :minute-end nil :begin 7679 :end 7696 :post-blank 1)); Skipping entry
Wrong type argument: stringp, (timestamp (:type active :raw-value "<2021-02-28 Sun>" :year-start 2021 :month-start 2 :day-start 28 :hour-start nil :minute-start nil :year-end 2021 :month-end 2 :day-end 28 :hour-end nil :minute-end nil :begin 11212 :end 11229 :post-blank 1)); Skipping entry
Wrong type argument: stringp, (timestamp (:type active :raw-value "<2020-06-30 Tue>" :year-start 2020 :month-start 6 :day-start 30 :hour-start nil :minute-start nil :year-end 2020 :month-end 6 :day-end 30 :hour-end nil :minute-end nil :begin 13781 :end 13798 :post-blank 1)); Skipping entry

......
#+end_example


[-- Attachment #1.2: Type: text/html, Size: 1453 bytes --]

[-- Attachment #2: Type: text/plain, Size: 282 bytes --]


<#secure method=pgpmime mode=sign>
-- 
[ stardiviner ]
       I try to make every word tell the meaning that I want to express.

       Blog: https://stardiviner.github.io/
       IRC(freenode): stardiviner, Matrix: stardiviner
       GPG: F09F650D7D674819892591401B5DF1C95AE89AC3

Re: [BUG] org-agenda generation missing some entries

[Ihor Radchenko]

"Christopher M. Miles" <numbchild@gmail.com> writes:

> After git bisect, I found the bad commit:
>
> #+begin_example
> 0ef88e2d9 * | bad @ org-agenda.el/org-agenda-get-scheduled: Use cache
> #+end_example
>
> after org-agenda generation, lot of entries not displayed. Then I found lot of errors in =*Messages*= buffer.

Can you provide a backtrace with debug-on-error set to t?

Best,
Ihor

Re: [BUG] org-agenda generation missing some entries

[Christopher M. Miles]

[-- Attachment #1.1: Type: text/plain, Size: 977 bytes --]



Ihor Radchenko <yantar92@gmail.com> writes:

> "Christopher M. Miles" <numbchild@gmail.com> writes:
>
>> After git bisect, I found the bad commit:
>>
>> #+begin_example
>> 0ef88e2d9 * | bad @ org-agenda.el/org-agenda-get-scheduled: Use cache
>> #+end_example
>>
>> after org-agenda generation, lot of entries not displayed. Then I found lot of errors in =*Messages*= buffer.
>
> Can you provide a backtrace with debug-on-error set to t?
>
> Best,
> Ihor

It's same, there is no backtrace even I toggled debug-on-error to t.

And I also tried you org-element-cache options:

#+begin_src emacs-lisp
(setq org-element--cache-self-verify 'backtrace)
(setq org-element--cache-self-verify-frequency 1)
(setq org-element--cache-diagnostics t)
#+end_src

This does not generate backtrace too.

Now I tried your latest push. The errors gone. But org-agenda entries still missing. I checked
*Messages* buffer, no error messages, including other error special buffers. no information.


[-- Attachment #1.2: Type: text/html, Size: 2084 bytes --]

[-- Attachment #2: Type: text/plain, Size: 282 bytes --]


<#secure method=pgpmime mode=sign>
-- 
[ stardiviner ]
       I try to make every word tell the meaning that I want to express.

       Blog: https://stardiviner.github.io/
       IRC(freenode): stardiviner, Matrix: stardiviner
       GPG: F09F650D7D674819892591401B5DF1C95AE89AC3

Re: [BUG] org-agenda generation missing some entries

[Ihor Radchenko]

"Christopher M. Miles" <numbchild@gmail.com> writes:
> Now I tried your latest push. The errors gone. But org-agenda entries still missing. I checked
> *Messages* buffer, no error messages, including other error special buffers. no information.

Agenda now uses a new algorithm to search entries. There may be a bug
there. Can you provide a minimal org file with a heading that should be
listed in agenda, but did not show up?

Best,
Ihor

Re: [BUG] org-agenda generation missing some entries

[Christopher M. Miles]

[-- Attachment #1.1: Type: text/plain, Size: 1177 bytes --]



Ihor Radchenko <yantar92@gmail.com> writes:

> "Christopher M. Miles" <numbchild@gmail.com> writes:
>> Now I tried your latest push. The errors gone. But org-agenda entries still missing. I checked
>> *Messages* buffer, no error messages, including other error special buffers. no information.
>
> Agenda now uses a new algorithm to search entries. There may be a bug
> there. Can you provide a minimal org file with a heading that should be
> listed in agenda, but did not show up?
>
> Best,
> Ihor

I'm git bisect on the latest git-pulled org-mode. Still located bad commit on this bad commit. Seems
commit also introduced other problem.

#+begin_example
0ef88e2d9 * | bad @ org-agenda.el/org-agenda-get-scheduled: Use cache
#+end_example

I compared good *Org Agenda* buffer and bad *Org Agenda* buffer, bellowing are some of missing
org-agenda entries:

#+begin_src org
,* PROJECT 🅰 task A
  SCHEDULED: <2019-04-16 Tue>
  :LOGBOOK:
  - State "PROJECT"    from              [2019-04-16 Tue 13:37]
  :END:
#+end_src

I checked most of them, almost all missing org-agenda entries are "SCHEDULED" tasks with logbook or
clocking log.



[-- Attachment #1.2: Type: text/html, Size: 2703 bytes --]

[-- Attachment #2: Type: text/plain, Size: 282 bytes --]


<#secure method=pgpmime mode=sign>
-- 
[ stardiviner ]
       I try to make every word tell the meaning that I want to express.

       Blog: https://stardiviner.github.io/
       IRC(freenode): stardiviner, Matrix: stardiviner
       GPG: F09F650D7D674819892591401B5DF1C95AE89AC3

Re: [BUG] org-agenda generation missing some entries

[Ihor Radchenko]

"Christopher M. Miles" <numbchild@gmail.com> writes:

> I'm git bisect on the latest git-pulled org-mode. Still located bad commit on this bad commit. Seems
> commit also introduced other problem.

> #+begin_example
> 0ef88e2d9 * | bad @ org-agenda.el/org-agenda-get-scheduled: Use cache
> #+end_example

That's understandable, but not sufficient to pinpoint the bug.

> I compared good *Org Agenda* buffer and bad *Org Agenda* buffer, bellowing are some of missing
> org-agenda entries:
>
> #+begin_src org
> ,* PROJECT 🅰 task A
>   SCHEDULED: <2019-04-16 Tue>
>   :LOGBOOK:
>   - State "PROJECT"    from              [2019-04-16 Tue 13:37]
>   :END:
> #+end_src
>
> I checked most of them, almost all missing org-agenda entries are "SCHEDULED" tasks with logbook or
> clocking log.

Is it still missing if you put this heading into separate buffer and
call agenda with buffer restriction? I just tried to build agenda after
changing the keyword from PROJECT to TODO and I can see the heading in
my agenda view.

Are you using any non-standard agenda settings in addition to
org-agenda-skip-scheduled-delay-if-deadline set to t?

Can you try to rebuild agenda after running M-x org-element-cache-reset?

Best,
Ihor

Re: [BUG] org-agenda generation missing some entries

[Christopher M. Miles]

[-- Attachment #1.1: Type: text/plain, Size: 1771 bytes --]



Ihor Radchenko <yantar92@gmail.com> writes:

> "Christopher M. Miles" <numbchild@gmail.com> writes:
>
>> I'm git bisect on the latest git-pulled org-mode. Still located bad commit on this bad commit. Seems
>> commit also introduced other problem.
>
>> #+begin_example
>> 0ef88e2d9 * | bad @ org-agenda.el/org-agenda-get-scheduled: Use cache
>> #+end_example
>
> That's understandable, but not sufficient to pinpoint the bug.
>
>> I compared good *Org Agenda* buffer and bad *Org Agenda* buffer, bellowing are some of missing
>> org-agenda entries:
>>
>> #+begin_src org
>> ,* PROJECT 🅰 task A
>>   SCHEDULED: <2019-04-16 Tue>
>>   :LOGBOOK:
>>   - State "PROJECT"    from              [2019-04-16 Tue 13:37]
>>   :END:
>> #+end_src
>>
>> I checked most of them, almost all missing org-agenda entries are "SCHEDULED" tasks with logbook or
>> clocking log.
>
> Is it still missing if you put this heading into separate buffer and
> call agenda with buffer restriction? I just tried to build agenda after
> changing the keyword from PROJECT to TODO and I can see the heading in
> my agenda view.
>
> Are you using any non-standard agenda settings in addition to
> org-agenda-skip-scheduled-delay-if-deadline set to t?
>
> Can you try to rebuild agenda after running M-x org-element-cache-reset?
>
> Best,
> Ihor

I put this scheduled task into a separated org file and clean org-agenda-files variable, and set it
to only this new separated org file. Not displayed.

I changed "PROJECT" to "TODO", Not displayed.

I have variable org-agenda-skip-scheduled-delay-if-deadline setted to 'post-deadline. Then I tried
setting it to t or nil. Bot not displayed.

I tried [M-x org-element-cache-reset], Still not displayed.


[-- Attachment #1.2: Type: text/html, Size: 2902 bytes --]

[-- Attachment #2: Type: text/plain, Size: 282 bytes --]


<#secure method=pgpmime mode=sign>
-- 
[ stardiviner ]
       I try to make every word tell the meaning that I want to express.

       Blog: https://stardiviner.github.io/
       IRC(freenode): stardiviner, Matrix: stardiviner
       GPG: F09F650D7D674819892591401B5DF1C95AE89AC3

Re: [BUG] org-agenda generation missing some entries

[Ihor Radchenko]

"Christopher M. Miles" <numbchild@gmail.com> writes:

> I put this scheduled task into a separated org file and clean org-agenda-files variable, and set it
> to only this new separated org file. Not displayed.

I finally managed reproduce the problem. It only exists in Emacs <28.
byte-compiled version of agenda matcher is somehow different from raw
lambda. I just pushed a workaround to main. Let me know if it fixed the
issue on your side.

Best,
Ihor

Re: [BUG] org-agenda generation missing some entries

[Christopher M. Miles]

[-- Attachment #1.1: Type: text/plain, Size: 749 bytes --]



Ihor Radchenko <yantar92@gmail.com> writes:

> "Christopher M. Miles" <numbchild@gmail.com> writes:
>
>> I put this scheduled task into a separated org file and clean org-agenda-files variable, and set it
>> to only this new separated org file. Not displayed.
>
> I finally managed reproduce the problem. It only exists in Emacs <28.
> byte-compiled version of agenda matcher is somehow different from raw
> lambda. I just pushed a workaround to main. Let me know if it fixed the
> issue on your side.
>
> Best,
> Ihor

My Emacs version (on M1 macOS)

GNU Emacs 28.0.60 (build 1, aarch64-apple-darwin20.6.0, NS appkit-2022.60 Version 11.6 (Build 20G165)) of 2021-10-20

Thanks for your continuous working. I pulled latest fix, still not working.


[-- Attachment #1.2: Type: text/html, Size: 1133 bytes --]

[-- Attachment #2: Type: text/plain, Size: 282 bytes --]


<#secure method=pgpmime mode=sign>
-- 
[ stardiviner ]
       I try to make every word tell the meaning that I want to express.

       Blog: https://stardiviner.github.io/
       IRC(freenode): stardiviner, Matrix: stardiviner
       GPG: F09F650D7D674819892591401B5DF1C95AE89AC3

Re: [BUG] org-agenda generation missing some entries

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

"Christopher M. Miles" <numbchild@gmail.com> writes:

> My Emacs version (on M1 macOS)
>
> GNU Emacs 28.0.60 (build 1, aarch64-apple-darwin20.6.0, NS appkit-2022.60 Version 11.6 (Build 20G165)) of 2021-10-20
>
> Thanks for your continuous working. I pulled latest fix, still not working.

I also tested using Emacs 28.0.60, but on Linux. Can you try exactly
same steps I used for testing?

1. Save the attached file to /tmp/problem.org

2.

mkdir /tmp/orgmode-test
cd /tmp/orgomode-test
git clone git://git.sv.gnu.org/emacs/org-mode.git
cd org-mode/
emacs -Q -L ./lisp -l org /tmp/problem.org

3. M-x org-agenda <RET> < a

If you do not see the heading listed in agenda view after doing above
steps, can you also try to go to org-element.el and change
"(version< emacs-version "28")" in org-element-cache-map to
"(version< emacs-version "29")". Then, try the above steps again.

Best,
Ihor


[-- Attachment #2: df.org --]
[-- Type: application/vnd.lotus-organizer, Size: 132 bytes --]

* PROJECT 🅰 task A
 SCHEDULED: <2019-04-16 Tue>
 :LOGBOOK:
 - State "PROJECT"    from              [2019-04-16 Tue 13:37]
 :END:

Re: [BUG] org-agenda generation missing some entries

[Christopher M. Miles]

[-- Attachment #1.1: Type: text/plain, Size: 1203 bytes --]



Ihor Radchenko <yantar92@gmail.com> writes:

> "Christopher M. Miles" <numbchild@gmail.com> writes:
>
>> My Emacs version (on M1 macOS)
>>
>> GNU Emacs 28.0.60 (build 1, aarch64-apple-darwin20.6.0, NS appkit-2022.60 Version 11.6 (Build 20G165)) of 2021-10-20
>>
>> Thanks for your continuous working. I pulled latest fix, still not working.
>
> I also tested using Emacs 28.0.60, but on Linux. Can you try exactly
> same steps I used for testing?
>
> 1. Save the attached file to /tmp/problem.org
>
> 2.
>
> mkdir /tmp/orgmode-test
> cd /tmp/orgomode-test
> git clone git://git.sv.gnu.org/emacs/org-mode.git
> cd org-mode/
> emacs -Q -L ./lisp -l org /tmp/problem.org
>
> 3. M-x org-agenda <RET> < a
>
> If you do not see the heading listed in agenda view after doing above
> steps, can you also try to go to org-element.el and change
> "(version< emacs-version "28")" in org-element-cache-map to
> "(version< emacs-version "29")". Then, try the above steps again.
>
> Best,
> Ihor
>
> [2. application/vnd.lotus-organizer; df.org]...

Confirmed after change from "28" to "29" solved the problem in your steps.

I also tested this change on my current Emacs config. Also confirmed solved the problem.


[-- Attachment #1.2: Type: text/html, Size: 1846 bytes --]

[-- Attachment #2: Type: text/plain, Size: 282 bytes --]


<#secure method=pgpmime mode=sign>
-- 
[ stardiviner ]
       I try to make every word tell the meaning that I want to express.

       Blog: https://stardiviner.github.io/
       IRC(freenode): stardiviner, Matrix: stardiviner
       GPG: F09F650D7D674819892591401B5DF1C95AE89AC3

Re: [BUG] org-agenda generation missing some entries

[Ihor Radchenko]

"Christopher M. Miles" <numbchild@gmail.com> writes:

> Confirmed after change from "28" to "29" solved the problem in your steps.
>
> I also tested this change on my current Emacs config. Also confirmed solved the problem.

I pushed this change to main in e2fa3c4c4.

Best,
Ihor

[SOLVED] Re: [BUG] org-agenda generation missing some entries

[Christopher M. Miles]

[-- Attachment #1.1: Type: text/plain, Size: 383 bytes --]



Ihor Radchenko <yantar92@gmail.com> writes:

> "Christopher M. Miles" <numbchild@gmail.com> writes:
>
>> Confirmed after change from "28" to "29" solved the problem in your steps.
>>
>> I also tested this change on my current Emacs config. Also confirmed solved the problem.
>
> I pushed this change to main in e2fa3c4c4.
>
> Best,
> Ihor

Thanks for quick response. mark solved.


[-- Attachment #1.2: Type: text/html, Size: 780 bytes --]

[-- Attachment #2: Type: text/plain, Size: 282 bytes --]


<#secure method=pgpmime mode=sign>
-- 
[ stardiviner ]
       I try to make every word tell the meaning that I want to express.

       Blog: https://stardiviner.github.io/
       IRC(freenode): stardiviner, Matrix: stardiviner
       GPG: F09F650D7D674819892591401B5DF1C95AE89AC3

Re: [BUG] org-agenda generation missing some entries

[Christopher M. Miles]

[-- Attachment #1.1: Type: text/plain, Size: 1233 bytes --]



Ihor Radchenko <yantar92@gmail.com> writes:

> "Christopher M. Miles" <numbchild@gmail.com> writes:
>> Now I tried your latest push. The errors gone. But org-agenda entries still missing. I checked
>> *Messages* buffer, no error messages, including other error special buffers. no information.
>
> Agenda now uses a new algorithm to search entries. There may be a bug
> there. Can you provide a minimal org file with a heading that should be
> listed in agenda, but did not show up?
>
> Best,
> Ihor

Also include some scheduled repeated tasks like this:

#+begin_src org
,** HABIT Write Blog -- =[C-c o c b]= / =[C-x t l n]=              :repeat:
   SCHEDULED: <2018-10-21 Sun 19:15 .+1d/2d>
   :PROPERTIES:
   :STYLE:    habit
   :END:

,** HABIT monthly check accounts                                    :repeat:
   SCHEDULED: <2021-03-07 Sun .+1m>
   :PROPERTIES:
   :LAST_REPEAT: [2021-02-07 Sun 17:22]
   :STYLE:    habit
   :END:
   :LOGBOOK:
   - State "DONE"       from "HABIT"      [2021-02-07 Sun 17:22]
   - State "DONE"       from "HABIT"      [2020-12-02 Wed 17:11]
   - State "DONE"       from "HABIT"      [2020-10-25 Sun 15:43]
   - State "DONE"       from              [2020-08-04 Tue 20:36]
   :END:
#+end_src



[-- Attachment #1.2: Type: text/html, Size: 5469 bytes --]

[-- Attachment #2: Type: text/plain, Size: 282 bytes --]


<#secure method=pgpmime mode=sign>
-- 
[ stardiviner ]
       I try to make every word tell the meaning that I want to express.

       Blog: https://stardiviner.github.io/
       IRC(freenode): stardiviner, Matrix: stardiviner
       GPG: F09F650D7D674819892591401B5DF1C95AE89AC3

Re: [BUG] org-agenda generation missing some entries

[Ihor Radchenko]

I pushed tentative fix to main. Let me know if you keep seeing the
error.

Best,
Ihor