Hi all, BACKGROUND: I'm using org-depend since ages[1]. As a matter of fact, it was the initial reason why I started with Org mode in the first place. One of my many workflows[2] exports the agenda for the upcoming 60 days into an ICS file via a cron job during night: : emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))' my-export-agenda[6] basically consists of: : (org-agenda-list nil nil 60) : (org-agenda-write (concat my-user-emacs-directory "var/export/agenda-export.ics")) This ICS file is then postprocessed and uploaded to my calendar server so that my mobile phone calendar is able to display events and appointments. This workflow stopped working end of December. ERRORS and INVESTIGATION: Error message when this batch run of Emacs exits after 5 hours(!): : Lisp nesting exceeds ‘max-lisp-eval-depth’ I updated my Org version[3] recently to get v9.3 from git maint. This was Wed Dec 4 10:37:19 2019 +0100. So the issue with the ICS export did not appear at the same time of the update. I also get: : [...] : Preparing diary...done : Preparing diary...done : Position saved to mark ring, go back with ‘C-c &’. [1038 times] : org-before-first-heading-p: Variable binding depth exceeds max-specpdl-size I changed the value for the variables without changing the result: : (setq max-specpdl-size 10000) : (setq max-lisp-eval-depth 50000) I even upgraded my Emacs version from 25 to 27[3]. No change. I reduced my set of org-agenda-files and did some test runs: most single org files resulted in an ICS export. Some single org files did result in the error. Reducing the export period from 60 to 7 days did not change the error. I also get following as execution cancellation error: : Before first headline at position 586 in buffer *temp*-99684 When running the job, I see *lots* lines like[4]: : Garbage collecting... : Garbage collecting...done To me, it's a data-related issue with my Org mode files so that others most likely are not able to reproduce my issue at all. Disabling org-depend results in a working ICS export that took much less time. I'm not sure if org-depend is directly related or just reduces "the complexity" such that the job is able to be finalized. QUESTIONS: I'm looking for ideas what I may check in order to be able to use org-depend again. Should I do a profiler report? How do I have to do this in order to get good results? Further more: is here somebody able to tell if org-edna[5] does have this issues as well? After all, org-edna is somewhat more complex than org-depend. A migration from org-depend to org-edna would be doable to me but with a decent effort. (I'm thinking of migrating only tasks that are not DONE or CANCELLED as I do think closed tasks do not irritate org-edna with old org-depend syntax.) [1] https://karl-voit.at/2016/12/18/org-depend/ Almost everything I use is :BLOCKER: properties and very rarely :TRIGGER: events. [2] just a selection is described on https://karl-voit.at/2019/09/25/using-orgmode/ [3] Org mode version 9.3 (release_9.3-3-g46319b.dirty @ mixed installation! GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.22.11) of 2019-12-15, unofficial emacs-snapshot build [4] I setup up "gcmh" package because of a comment on: https://www.reddit.com/r/orgmode/comments/e9p84n/scaling_org_better_to_use_more_medsize_files_or/fcm5bsc/ [5] https://elpa.gnu.org/packages/org-edna.html [6] https://github.com/novoid/dot-emacs/blob/master/config.org -- get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode: > get Memacs from https://github.com/novoid/Memacs < Personal Information Management > http://Karl-Voit.at/tags/pim/ Emacs-related > http://Karl-Voit.at/tags/emacs/
Hi Karl, did you manage to fix this one? I read "Property drawers allowed before first headline" in etc/ORG-NEWS from the master branch: Property drawers are now allowed before the first headline. Org mode is moving more towards making things before the first headline behave just as if it was at outline level 0. Inheritance for properties will work also for this level. In other words: defining things in a property drawer before the first headline will make them "inheritable" for all headlines. I guess your problem might be related to this new behavior. 2 cents, -- Bastien
* Bastien <bzg@gnu.org> wrote: > Hi Karl, Hi Bastien, > did you manage to fix this one? Well ... > I read "Property drawers allowed before first headline" in > etc/ORG-NEWS from the master branch: > > Property drawers are now allowed before the first headline. > > Org mode is moving more towards making things before the first > headline behave just as if it was at outline level 0. Inheritance > for properties will work also for this level. In other words: > defining things in a property drawer before the first headline will > make them "inheritable" for all headlines. > > I guess your problem might be related to this new behavior. Since I'm not using drawers before the first headline, I don't understand how this might relate to my issue. However, I just re-enabled org-depend which I had disabled for batch mode only in order to get the export done. Then I ran the batch process for the export and it finished without an error within 1min53sec. :-) So the issue seems to be gone (for now) but I don't know why it appeared in the first place nor why it disappeared in the meantime. I'm using Org mode from maint. Currently on my side: d20e45b18 (3 days ago). The version that had the issue was from November or December. I could provide the exact ID hash if anybody is interested. Thanks anyway and thanks for the pointer with the new feature. That might come handy when there is a property which does not have a file-local version. -- get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode: > get Memacs from https://github.com/novoid/Memacs < Personal Information Management > http://Karl-Voit.at/tags/pim/ Emacs-related > http://Karl-Voit.at/tags/emacs/
Hi Karl,
Karl Voit <devnull@Karl-Voit.at> writes:
> So the issue seems to be gone (for now) but I don't know why it
> appeared in the first place nor why it disappeared in the meantime.
/me looks for a good xkcd quote. :)
--
Bastien
* Bastien <bzg@gnu.org> wrote: > Hi Karl, > > Karl Voit <devnull@Karl-Voit.at> writes: > >> So the issue seems to be gone (for now) but I don't know why it >> appeared in the first place nor why it disappeared in the meantime. > > /me looks for a good xkcd quote. :) Alternatively: https://en.wikipedia.org/wiki/Haiku_in_English ;-) -- get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode: > get Memacs from https://github.com/novoid/Memacs < Personal Information Management > http://Karl-Voit.at/tags/pim/ Emacs-related > http://Karl-Voit.at/tags/emacs/
Hi Bastien, * Bastien <bzg@gnu.org> wrote: > Hi Karl, > > did you manage to fix this one? > > I read "Property drawers allowed before first headline" in > etc/ORG-NEWS from the master branch: > > Property drawers are now allowed before the first headline. > > Org mode is moving more towards making things before the first > headline behave just as if it was at outline level 0. Inheritance > for properties will work also for this level. In other words: > defining things in a property drawer before the first headline will > make them "inheritable" for all headlines. > > I guess your problem might be related to this new behavior. A couple days ago, the issue re-appeared (with org-depend being active again). To me, this is a clear indicator, that the issue is content-related and not config/binary-related: Emacs 26.3: : emacs --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))' : [...] : Garbage collecting... : Garbage collecting...done : Garbage collecting... : Garbage collecting...done : Garbage collecting... : Garbage collecting...done : Position saved to mark ring, go back with ‘C-c &’. : Position saved to mark ring, go back with ‘C-c &’. : Position saved to mark ring, go back with ‘C-c &’. : Position saved to mark ring, go back with ‘C-c &’. : Position saved to mark ring, go back with ‘C-c &’. : Position saved to mark ring, go back with ‘C-c &’. : Garbage collecting... : Garbage collecting...done : Variable binding depth exceeds max-specpdl-size : emacs --batch --load /home/vk/.emacs.d/init.el --eval 8029,80s user 4,30s system 99% cpu 2:14:02,16 total Yes, that's over two hours. With Emacs 27.0.50, the same job is currently running for over five hours. I guess this will end with an error as well some day. -- get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode: > get Memacs from https://github.com/novoid/Memacs < Personal Information Management > http://Karl-Voit.at/tags/pim/ Emacs-related > http://Karl-Voit.at/tags/emacs/
Hi Karl,
Karl Voit <devnull@Karl-Voit.at> writes:
> A couple days ago, the issue re-appeared (with org-depend being
> active again). To me, this is a clear indicator, that the issue is
> content-related and not config/binary-related:
Can you bisect and deduce what part of your contents is producing
this? When you have a clue, could you also bisect and see what
change in Org introduced this problem for the problematic contents?
Such issues are difficult to reproduce and track outside your files.
Thanks!
--
Bastien
Hi, * Bastien <bzg@gnu.org> wrote: > > Karl Voit <devnull@Karl-Voit.at> writes: > >> A couple days ago, the issue re-appeared (with org-depend being >> active again). To me, this is a clear indicator, that the issue is >> content-related and not config/binary-related: > > Can you bisect and deduce what part of your contents is producing > this? This is not trivial in my case since I'd have to merge all my Org mode content into one single large file and make sure that I remove content during bi-secting that has no link destination from the remaining content. Personally, I'm not convinced that this method will lead to a minimum example in any case. I can imagine that "number of Org mode headings/lines" might be involved here which would not trigger the error when bi-secting via removing. Currently, I don't see that this is possible without half a day work or (IMHO more likely) longer which I can't afford right now. :-( > When you have a clue, could you also bisect and see what > change in Org introduced this problem for the problematic contents? This would be trivial when I could locate the issue in a minimal example. > Such issues are difficult to reproduce and track outside your files. Totally agree. I thought somebody else might have faced the same issue. As long as I am the only person affected, I don't assume that it can be solved in a proper way. :-( I ran another test with Emacs 27.0.50 and disabled org-depend. The task finished after 4½ minutes without any error. Therefore, following dependencies has to be the culprit here. Moving from org-depend to org-edna is not trivial for me as well, considering the amount of meta-data to be modified. On the other hand, this would be interesting to see if org-edna can handle this use-case in a better way. In case somebody has experience with migration process and/or dependency lookup performance, I'm very interested. -- get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode: > get Memacs from https://github.com/novoid/Memacs < Personal Information Management > http://Karl-Voit.at/tags/pim/ Emacs-related > http://Karl-Voit.at/tags/emacs/
Hi Karl, another suggestion: just send us a minimal extract of your files so that we see how org-depend is called, together with what you do and what goes wrong when you do it. Perhaps this will lead to something. I've never used org-edna, I'm curious to know if there are many users. Thanks, -- Bastien
* Bastien <bzg@gnu.org> wrote: > Hi Karl, Hi, > another suggestion: just send us a minimal extract of your files so > that we see how org-depend is called, together with what you do and > what goes wrong when you do it. Perhaps this will lead to something. Sure. https://github.com/novoid/dot-emacs/blob/master/config.org contains: | (setq my-user-emacs-directory "~/.emacs.d/") | (use-package org-depend | :load-path (lambda () (expand-file-name (concat my-user-emacs-directory "contrib/org-mode/contrib/lisp/"))) | ) ... which loads org-depend from my maint Org mode (currently: 9.3 (release_9.3-102-gd20e45.dirty @ mixed installation![...]). I only use org-depend for: * TODO Example 1 :PROPERTIES: :ID: 2020-02-20-foo :END: * TODO Example 2 :PROPERTIES: :BLOCKER: 2020-02-20-foo :END: Very rarely, I am using: * TODO Example 3 :PROPERTIES: :TRIGGER: 2020-02-20-foo :END: Most of the time, source-destination are within the same file. However, in some cases of using BLOCKER, the two headings might be in two different files. That's basically it: mostly BLOCKER and a few TRIGGER. I'm unsure if there is currently a TRIGGER that is still waiting to be activated. (By the way: is there a built-in feature to look for IDs that are linked somewhere (BLOCKER properties or ID links) that are broken?) Ad "what goes wrong when you do it": in my daily work, org-depend is working fine. The only thing that does not work is exporting my agenda with: | emacs --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))' and: | (defun my-export-agenda() | "Exports Org-mode agenda to ics file" | (interactive) | (save-some-buffers) | | ;; I don't want the error messages in my exported agenda: | (setq org-agenda-files (remove "~/org/errors.org" org-agenda-files)) | | (setq max-specpdl-size 10000) ;; does not solve issue | (setq max-lisp-eval-depth 50000) ;; does not solve issue | | (org-agenda-list nil nil 60) | (org-agenda-write (concat my-user-emacs-directory "var/export/agenda-export.ics")) | ) With disabled org-depend, it is working and finishes in ~5 minutes. With enabled org-depend it (sometimes) end up in an error as described. This can take hours. > I've never used org-edna, I'm curious to know if there are many users. Me too ;-) It's a bit more complicated in syntax but has advanced features org-depend does not offer. The main reason why I'm thinking of switching is that org-edna supports something like TRIGGER that also adds relative SCHEDULED date-stamps. This allows for "if $THAT task gets done, mark $NEXT task as a TODO item and add a SCHEDULED with three days in the future" which is very handy in some cases. > Thanks, Oh, it's me, who is very thankful, considering the tricky situation to investigate this issue. -- get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode: > get Memacs from https://github.com/novoid/Memacs < Personal Information Management > http://Karl-Voit.at/tags/pim/ Emacs-related > http://Karl-Voit.at/tags/emacs/
Hi Karl,
Karl Voit <devnull@Karl-Voit.at> writes:
> With disabled org-depend, it is working and finishes in ~5 minutes. With
> enabled org-depend it (sometimes) end up in an error as described.
Do you need org-depend when exporting?
If not, I simply suggest to turn org-depend off... or to let-bind
`org-trigger-hook' and `org-blocker-hook' to nil in your function.
Probably org-depend does a lot of moving back and forth in files to
check for IDs, and that may explain the time spend by this command
when you export many files.
--
Bastien
Bonjour Bastien, * Bastien <bzg@gnu.org> wrote: > > Karl Voit <devnull@Karl-Voit.at> writes: > >> With disabled org-depend, it is working and finishes in ~5 minutes. With >> enabled org-depend it (sometimes) end up in an error as described. > > Do you need org-depend when exporting? Well: in all cases where I defined a dependency of task B to a task A, with disabled org-depend I end up seeing task B on the agenda although I can not do it at the moment. So as long as I don't use dependencies, it does not matter. However, I'm organizing a lecture in the summer term using a very large and complicated yankpad/yasnippet template for the whole term that is using many dependencies for approx. 70 todos. With disabled org-depend, I end up seeing all 70 todos although I may be able to work on two or three of them at any moment in time. > If not, I simply suggest to turn org-depend off... or to let-bind > `org-trigger-hook' and `org-blocker-hook' to nil in your function. > > Probably org-depend does a lot of moving back and forth in files to > check for IDs, and that may explain the time spend by this command > when you export many files. This sounds reasonable. I'm fine with - let's say - +50% 1-day agenda duration and I'm fine with even an hour of 60d agenda export to ics with enabled org-depend. The show-stopper is, when org-depend does run in an error after hours of exporting without a result. This way, I'm losing my Org->Android/Radicale workflow or my dependencies. The longer this thread gets, the more I have to accept that I might be the only one using dependencies in Org mode. I'm puzzled here because I do think that dependencies are such a natural thing to want and use. (I also have several ideas to improve usability of them to minimize manual effort when defining dependencies.) So maybe I have to invest a weekend or so to bisect the issue. My gut feeling tells me that I won't like the output somehow. -- get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode: > get Memacs from https://github.com/novoid/Memacs < Personal Information Management > http://Karl-Voit.at/tags/pim/ Emacs-related > http://Karl-Voit.at/tags/emacs/
Hi Karl,
Karl Voit <devnull@Karl-Voit.at> writes:
> So maybe I have to invest a weekend or so to bisect the issue. My
> gut feeling tells me that I won't like the output somehow.
In the meantime, can you share the values of your org-id-* options?
Thanks,
--
Bastien
* Bastien <bzg@gnu.org> wrote: > Hi Karl, > > Karl Voit <devnull@Karl-Voit.at> writes: > >> So maybe I have to invest a weekend or so to bisect the issue. My >> gut feeling tells me that I won't like the output somehow. > > In the meantime, can you share the values of your org-id-* options? Easy: I could not find any change of the defaults in my configuration[1]. So all of them have to be the default values. [1] https://github.com/novoid/dot-emacs/blob/master/config.org -- get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode: > get Memacs from https://github.com/novoid/Memacs < Personal Information Management > http://Karl-Voit.at/tags/pim/ Emacs-related > http://Karl-Voit.at/tags/emacs/
Hi Karl,
Karl Voit <devnull@Karl-Voit.at> writes:
> Easy: I could not find any change of the defaults in my
> configuration[1]. So all of them have to be the default values.
>
> [1] https://github.com/novoid/dot-emacs/blob/master/config.org
I would try setting `org-agenda-before-write-hook' to nil (it is set
to `org-agenda-add-entry-text' by default.
Also, I see `org-icalendar-include-body' is set to 1000 in your setup.
That's a lot and this will generate a lot of strings replacements,
I suggest trying a lower value for your export-and-write function.
HTH,
--
Bastien
Hi Bastien, * Bastien <bzg@gnu.org> wrote: > > Karl Voit <devnull@Karl-Voit.at> writes: > >> Easy: I could not find any change of the defaults in my >> configuration[1]. So all of them have to be the default values. >> >> [1] https://github.com/novoid/dot-emacs/blob/master/config.org > > I would try setting `org-agenda-before-write-hook' to nil (it is set > to `org-agenda-add-entry-text' by default. > > Also, I see `org-icalendar-include-body' is set to 1000 in your setup. > > That's a lot and this will generate a lot of strings replacements, > I suggest trying a lower value for your export-and-write function. Thanks for the advice. I ran three test runs in order to find out how output is affected: Testrun with disabled org-depend: emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda)) 261.95s user 1.57s system 99% cpu 4:24.01 total → I got a result without error Testrun with enabled org-depend: emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))' 117.58s user 1.46s system 99% cpu 1:59.60 total → much faster result without error :-) (probably caching?) revoking the two suggested changes and do another test run with org-depend, expecting error: emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))' 114.38s user 1.53s system 99% cpu 1:56.33 total → much faster result without error (contrary to expection) → the two setting changes do not have an effect for now regarding error/no error due to the fact that I can not reproduce the error. Retrying when error re-appears. -- get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode: > get Memacs from https://github.com/novoid/Memacs < Personal Information Management > http://Karl-Voit.at/tags/pim/ Emacs-related > http://Karl-Voit.at/tags/emacs/
Hi Karl,
Karl Voit <devnull@Karl-Voit.at> writes:
> Retrying when error re-appears.
OK, thanks for the feedback!
--
Bastien
This week, the error re-appeared. * Karl Voit <devnull@Karl-Voit.at> wrote: > > Thanks for the advice. I ran three test runs in order to find out how output is > affected: > > > Testrun with disabled org-depend: > emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda)) > 261.95s user 1.57s system 99% cpu 4:24.01 total > > → I got a result without error > > > Testrun with enabled org-depend: > emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))' > 117.58s user 1.46s system 99% cpu 1:59.60 total > > → much faster result without error :-) (probably caching?) > > > revoking the two suggested changes and do another test run with org-depend, expecting error: > emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))' > 114.38s user 1.53s system 99% cpu 1:56.33 total > > → much faster result without error (contrary to expection) > > > → the two setting changes do not have an effect for now regarding > error/no error due to the fact that I can not reproduce the > error. > > Retrying when error re-appears. - setting `org-agenda-before-write-hook' to nil - `org-icalendar-include-body': 42 - enabled org-depend : emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))' → did not finish after running for 4 hours (aborted) - setting `org-agenda-before-write-hook' to nil - `org-icalendar-include-body': 42 - *disabled org-depend* → produced output within 4min39s - setting `org-agenda-before-write-hook' to default - `org-icalendar-include-body': to default - disabled org-depend → produced output within 2min07s Any other test you want me to run? -- get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode: > get Memacs from https://github.com/novoid/Memacs < Personal Information Management > http://Karl-Voit.at/tags/pim/ Emacs-related > http://Karl-Voit.at/tags/emacs/