Re: Asynchronous org-agenda-redo

[Ihor Radchenko <yantar92@gmail.com>]

> Be sure to read the Emacs Lisp manual regarding threads.  They are
> cooperative, so functions called as threads must yield back to the main
> thread for Emacs to do anything else before the function returns.

I tried to read the manual, but I clearly misunderstand something.
The manual says:

>   Currently, thread switching will occur upon explicit request via
> ‘thread-yield’, when waiting for keyboard input... 

So, except directly calling thread-yield, it should be possible to
trigger switching the current thread when keyboard input is expected.
I tried the following demo code:

(defun test ()
  (let ((a 0))
    (dotimes (_ 5)
      (setq a (1+ a))
      (sleep-for 2)
      (message "%s" a))))

(progn ;This should return to command loop quickly
  (make-thread #'test)
  (message "Executed...")); `eval-last-sexp' here

I can move around the buffer while the progn is running.
However, it is not the case with `org-agenda-redo' for a reason I do not
fully understand. 

For the async.el, I agree that loading packages may take time, but I
believe that the configuration might be transferred more easily (though
it may depend on the org-agenda and user-defined
org-agenda-skip-functions implementation).

> 1.  The process would have to load the same Org buffers, which takes
>     time, especially in large buffers.  Depending on configuration, it
>     can take some time, indeed.

> 3.  Ensuring that configuration and state between the main Emacs process
>     and the separate, agenda-generating process is not necessarily
>     simple.  Consider as well that if a buffer had unsaved changes,
>     those would not be readable by the other process, which would lead
>     to invalid results.  One could force the buffers to be saved first,
>     but that may not always be desirable, as saving buffers can have
>     side effects.

Why cannot org-buffer simply be copied into the subordinate process? If
all be buffer-locals, text properties, and overlays are copied directly
from the main emacs process, there may be no need to even initialise
org-mode (the idea is to do something similar to clone-buffer). The
question though is whether buffer-locals + overlays + propertized .org
files text + org-agenda-buffer copy can be sufficient to make the
org-agenda-redo run properly. Are there any other buffers, variables, or
other environment settings used by org-agenda-redo?

> If your agenda buffers are taking too long to refresh, you might
> consider org-ql's views/saved-searches as an alternative. ...

I know org-ql and I am pretty sure that it will improve performance.
Actually, if one can make built-in org-agenda asynchronous, org-ql can
probably use similar approach and become even faster :)
I am trying on default org-agenda now mostly because my current config
is heavily geared towards default agenda and I am not sure if
refactoring everything to use org-ql will worth it at the end in terms
of performance. I use too many slow custom skip-functions.

> ... The built-in
> caching in org-ql significantly improves performance, especially when
> refreshing views.

Yeah. I wish org-entry-get and other org-get* functions support caching
as well... Or, at least, org-agenda functions might also support simple
caching based on file modifications.

Best,
Ihor

Adam Porter <adam@alphapapa.net> writes:

> Be sure to read the Emacs Lisp manual regarding threads.  They are
> cooperative, so functions called as threads must yield back to the main
> thread for Emacs to do anything else before the function returns.
>
> If you're feeling adventurous, you could experiment with adding yields
> in relevant agenda functions.  But that wouldn't be suitable for merging
> into Org, because that yielding also decreases performance generally.
>
> As long as Elisp threads are cooperative, they are of very limited use.
>
> Generating agendas with async.el in a separate Emacs process is an
> interesting idea, but probably generally impractical for a few reasons:
>
> 1.  The process would have to load the same Org buffers, which takes
>     time, especially in large buffers.  Depending on configuration, it
>     can take some time, indeed.
> 2.  The process would also have to load the same packages (or, at least,
>     all the necessary ones, which depends on configuration), which takes
>     time.
> 3.  Ensuring that configuration and state between the main Emacs process
>     and the separate, agenda-generating process is not necessarily
>     simple.  Consider as well that if a buffer had unsaved changes,
>     those would not be readable by the other process, which would lead
>     to invalid results.  One could force the buffers to be saved first,
>     but that may not always be desirable, as saving buffers can have
>     side effects.
>
> If your agenda buffers are taking too long to refresh, you might
> consider org-ql's views/saved-searches as an alternative.  The built-in
> caching in org-ql significantly improves performance, especially when
> refreshing views.
>
>

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg