[BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

This is a followup for another bug report
https://orgmode.org/list/87plxg95u4.fsf@gmail.com

Using the MWE from that report I am seeing a different bug and it is not
very clear for me what is causing it.

1. When I create /tmp/bug.org with the following contents:

    #+begin_src python :dir otherdir :async yes :session pysession :return figname :results file value :mkdirp yes
    import matplotlib.pyplot as plt
    plt.figure(figsize=(1, 1))
    plt.plot([1, 2])
    figname = 'fig.svg'
    plt.savefig(figname)
    #+end_src

2. Go to Org git folder
3. make repro REPRO_ARGS="-l ob-python /tmp/bug.org"
4. Evaluate the code block
5. I get

    #+RESULTS:
    [[file:babel-GJy87v/python-xr8wq9]]

    rather than expected [[file:otherdir/fig.svg]]

I use matplotlib 3.8.2-r1.
The contents of Python REPL is suspicious:

Python 3.11.7 (main, Jan  8 2024, 18:11:18) [GCC 13.2.1 20231216] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
ob_comint_async_python_file_/tmp/babel-s8nwvB/python-NmMFRv
>>> 

This is most likely something about my current system setup - I can
reproduce with other Org mode and Emacs versions. But I have no clue
what is the cause.

Emacs  : GNU Emacs 30.0.50 (build 4, x86_64-pc-linux-gnu, GTK+ Version 3.24.39, cairo version 1.18.0)
 of 2024-01-30
Package: Org mode version 9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)
-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

>
> This is most likely something about my current system setup - I can
> reproduce with other Org mode and Emacs versions. But I have no clue
> what is the cause.

I'm getting the same as you with your MWE.
   
The tag, used by ob-comint async, is:

   "/tmp/babel-zqh04P/python-GL5N5d"
   
but, in "/tmp/bug.org" it becomes:

   "babel-zqh04P/python-GL5N5d"

(`org-babel-result-to-file' transformed it into a simpler relative
path).

The filter `org-babel-comint-async-filter' cannot spot it, because
it's searching for the exact string "/tmp/babel-zqh04P/python-tXsdFw".

Here is how to reproduce:
   #+begin_src elisp :results table
     (let* ((tag "/tmp/babel-zqh04P/python-tXsdFw")
            (repro 
             (lambda (fn)
               (let ((lnk 
                      (with-temp-buffer
                        (org-mode)
                        (let* ((default-directory "/tmp")
                               (buffer-file-name  fn)
                               (cbuf (clone-indirect-buffer "tmp" nil)))
                          (with-current-buffer cbuf
                            (org-babel-result-to-file tag))))))
                 (list fn
                       (not (eq nil (string-match-p (regexp-quote tag) lnk)))
                       lnk)))))
       (cons (list "Filename" "string-match-p" "Org link")
             (cons 'hline
                   (mapcar repro (list "/tmp/bug.org" 
                                       "/somewhere/else/bug.org")))))
   #+end_src

   #+RESULTS:
   | Filename                | match-p | Org link                                      |
   |-------------------------+---------+-----------------------------------------------|
   | /tmp/bug.org            | nil     | [[file:babel-zqh04P/python-tXsdFw]]           |
   | /somewhere/else/bug.org | t       | [[file:../../tmp/babel-zqh04P/python-tXsdFw]] |


I don't know what a proper fix would be though.
   
Hoping this help,

Bruno


>
> Emacs  : GNU Emacs 30.0.50 (build 4, x86_64-pc-linux-gnu, GTK+ Version 3.24.39, cairo version 1.18.0)
>  of 2024-01-30
> Package: Org mode version 9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Jack Kamm]

Bruno Barbier <brubar.cs@gmail.com> writes:

> Hi Ihor,
>
> Ihor Radchenko <yantar92@posteo.net> writes:
>
>>
>> This is most likely something about my current system setup - I can
>> reproduce with other Org mode and Emacs versions. But I have no clue
>> what is the cause.
>
> I'm getting the same as you with your MWE.
>    
> The tag, used by ob-comint async, is:
>
>    "/tmp/babel-zqh04P/python-GL5N5d"
>    
> but, in "/tmp/bug.org" it becomes:
>
>    "babel-zqh04P/python-GL5N5d"
>
> (`org-babel-result-to-file' transformed it into a simpler relative
> path).
>
> The filter `org-babel-comint-async-filter' cannot spot it, because
> it's searching for the exact string "/tmp/babel-zqh04P/python-tXsdFw".

Thanks, I believe your diagnosis is correct, and can confirm the bug
occurs when the Org file is in /tmp. The problem goes away when
`org-link-file-path-type' is set to absolute.

I think the correct solution would be for `org-babel-insert-result' to
not insert file results (or any other special results) for async session
blocks.  Perhaps in this case, `org-babel-insert-result' could return a
new result type, named "async", "future", or similar.

Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Jack Kamm <jackkamm@gmail.com> writes:

> I think the correct solution would be for `org-babel-insert-result' to
> not insert file results (or any other special results) for async session
> blocks.  Perhaps in this case, `org-babel-insert-result' could return a
> new result type, named "async", "future", or similar.

That will not work.
`org-babel-comint-async-filter' expects a unique result to be inserted
into Org buffer, so that it can be located, and replaced by the async
evaluation output.

So, we have to insert some kind of indicator for async result.

Of course, the existing scheme of coordination between
`org-babel-insert-result' and `org-babel-comint-async-filter' is
erroneous:

1. We have the problem with :results file value discussed here
2. We have a worse problem with :results file :file foo when the result
   may not be unique
3. We have :results append/prepend completely broken because
   `org-babel-comint-async-filter' simply calls
   `org-babel-insert-result' implicitly assuming that the existing
   indicator is replaced.

The whole thing should be re-designed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Jack Kamm]

Ihor Radchenko <yantar92@posteo.net> writes:

> Jack Kamm <jackkamm@gmail.com> writes:
>
>> I think the correct solution would be for `org-babel-insert-result' to
>> not insert file results (or any other special results) for async session
>> blocks.  Perhaps in this case, `org-babel-insert-result' could return a
>> new result type, named "async", "future", or similar.
>
> That will not work.
> `org-babel-comint-async-filter' expects a unique result to be inserted
> into Org buffer, so that it can be located, and replaced by the async
> evaluation output.
>
> So, we have to insert some kind of indicator for async result.

I meant that we could return something like "async:uuid-abcd-1234" or
"async:/path/to/tmpfile", so that `org-babel-comint-async-filter' could
still find the result.

> Of course, the existing scheme of coordination between
> `org-babel-insert-result' and `org-babel-comint-async-filter' is
> erroneous:
>
> 1. We have the problem with :results file value discussed here
> 2. We have a worse problem with :results file :file foo when the result
>    may not be unique
> 3. We have :results append/prepend completely broken because
>    `org-babel-comint-async-filter' simply calls
>    `org-babel-insert-result' implicitly assuming that the existing
>    indicator is replaced.
>
> The whole thing should be re-designed.

I agree that it would be good to redesign it, but am not sure where to
start.

A bit of a tangent, but if you are thinking about re-designing this,
then it may be worth considering ob-jupyter's implementation of async
sessions [1]. In particular, I believe it leaves a marker [2] where it
needs to insert the future result. I don't remember the details,
e.g. how it keeps track of which marker is for which result. But it
seems neat, and might work better for some cases such as
appending/prepending results.

[1] https://github.com/emacs-jupyter/jupyter
[2] https://www.gnu.org/software/emacs/manual/html_node/elisp/Markers.html

Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Jack Kamm <jackkamm@gmail.com> writes:

>> So, we have to insert some kind of indicator for async result.
>
> I meant that we could return something like "async:uuid-abcd-1234" or
> "async:/path/to/tmpfile", so that `org-babel-comint-async-filter' could
> still find the result.

Of course, we could. But that would not solve all the possible problems.
In particular, when header arguments tell Org babel to write result to a
file, returning UUID will still not work.

>> Of course, the existing scheme of coordination between
>> `org-babel-insert-result' and `org-babel-comint-async-filter' is
>> erroneous:
>>
>> 1. We have the problem with :results file value discussed here
>> 2. We have a worse problem with :results file :file foo when the result
>>    may not be unique
>> 3. We have :results append/prepend completely broken because
>>    `org-babel-comint-async-filter' simply calls
>>    `org-babel-insert-result' implicitly assuming that the existing
>>    indicator is replaced.
>>
>> The whole thing should be re-designed.
> ...
> A bit of a tangent, but if you are thinking about re-designing this,
> then it may be worth considering ob-jupyter's implementation of async
> sessions [1]. In particular, I believe it leaves a marker [2] where it
> needs to insert the future result. I don't remember the details,
> e.g. how it keeps track of which marker is for which result. But it
> seems neat, and might work better for some cases such as
> appending/prepending results.

Markers are not as reliable as you think. If text around marker gets
deleted, the marker will still exist potentially causing the async
result to be inserted in the middle of unexpected place.

Having an actual text indicator is more reliable - if user removes it
before the async evaluation is completed, we will not write anything
unexpected.

Also,
https://github.com/emacs-jupyter/jupyter/blob/master/ob-jupyter.el#L540

        ;; KLUDGE: Remove the file result-parameter so that
        ;; `org-babel-insert-result' doesn't attempt to handle it while
        ;; async results are pending.  Do the same in the synchronous
        ;; case, but not if link or graphics are also result-parameters,
        ;; only in Org >= 9.2, since those in combination with file mean
        ;; to interpret the result as a file link, a useful meaning that
        ;; doesn't interfere with Jupyter style result insertion.

They also had to work around the same problem.

> I agree that it would be good to redesign it, but am not sure where to
> start.

For example,

1. Change `org-babel-comint-async-register' to return UUID and to store
   PARAMS as passed by the backend (current approach with PARAMS being
   derived from src blocks prevents backends to transform src block
   PARAMS dynamically).
2. Change `org-babel-insert-result' to handle :async t specially,
   inserting something reliable, like #+async: <UUID> in place of result
   without performing extra transformations.
3. Change `org-babel-insert-result' to accept an internal parameter
   that will make it replace #+async: <UUID> keyword rather than perform
   normal result insertion.
4. Change `org-babel-comint-async-filter' to use the previously passed
   PARAMS, remove :async t from them, and arrange the call to
   `org-babel-insert-result' to replace the #+async: <UUID> keyword.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Bruno Barbier]

Hi Ihor, Jack,

Ihor Radchenko <yantar92@posteo.net> writes:

> Jack Kamm <jackkamm@gmail.com> writes:
>

>> I agree that it would be good to redesign it, but am not sure where to
>> start.
>
> For example,
>
> 1. Change `org-babel-comint-async-register' to return UUID and to store
>    PARAMS as passed by the backend (current approach with PARAMS being
>    derived from src blocks prevents backends to transform src block
>    PARAMS dynamically).
> 2. Change `org-babel-insert-result' to handle :async t specially,
>    inserting something reliable, like #+async: <UUID> in place of result
>    without performing extra transformations.
> 3. Change `org-babel-insert-result' to accept an internal parameter
>    that will make it replace #+async: <UUID> keyword rather than perform
>    normal result insertion.
> 4. Change `org-babel-comint-async-filter' to use the previously passed
>    PARAMS, remove :async t from them, and arrange the call to
>    `org-babel-insert-result' to replace the #+async: <UUID> keyword.
>

FWIW, I've been trying to use asynchronous blocks for everything, not
only the source blocks that are based on the comint mode.  I think it
would be good if ob-core itself could provide an asynchronous API.  I've
modified my Org so that it does have such an API.  This is work in
progress; let me describe it.

I've modified ob-core itself to allow asynchronicity.  In the
asynchrosous case, instead of calling:

  (org-babel-execute:LANG body params)

I'm calling:

  (org-babel-schedule:LANG body params handle-result)

where `org-babel-schedule:LANG' is in charge of calling `handle-result'
with the result (or the error) when it is known; `handle-result' takes
care to call `org-babel-insert-result' at the correct place (and
`org-babel-insert-result' is only called with a real result).

While the execution is pending, I'm using the same technique that Org is
using when a source block is being edited: the result is left untouched,
but below an overlay.  The overlay is used to know where to insert the
result and to display the status/progress of the execution.  If the file
is closed and the execution fails, nothing is lost, the old result is
still available.

If that technique looks safe enough and interesting, I can prepare a set
of patches so that we can discuss it further and, maybe, add it in Org.

Let me know,
Thanks,

Bruno

Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

> FWIW, I've been trying to use asynchronous blocks for everything, not
> only the source blocks that are based on the comint mode.  I think it
> would be good if ob-core itself could provide an asynchronous API.  I've
> modified my Org so that it does have such an API.  This is work in
> progress; let me describe it.
>
> I've modified ob-core itself to allow asynchronicity.  In the
> asynchrosous case, instead of calling:
>
>   (org-babel-execute:LANG body params)
>
> I'm calling:
>
>   (org-babel-schedule:LANG body params handle-result)
>
> where `org-babel-schedule:LANG' is in charge of calling `handle-result'
> with the result (or the error) when it is known; `handle-result' takes
> care to call `org-babel-insert-result' at the correct place (and
> `org-babel-insert-result' is only called with a real result).

LGTM.

> While the execution is pending, I'm using the same technique that Org is
> using when a source block is being edited: the result is left untouched,
> but below an overlay.  The overlay is used to know where to insert the
> result and to display the status/progress of the execution.  If the file
> is closed and the execution fails, nothing is lost, the old result is
> still available.
>
> If that technique looks safe enough and interesting, I can prepare a set
> of patches so that we can discuss it further and, maybe, add it in Org.

Using overlay also sounds good.

I am wondering what you do when there is no result yet or when user
edits whatever is under the overlay, but that's just a technical detail.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Bruno Barbier]

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> While the execution is pending, I'm using the same technique that Org is
>> using when a source block is being edited: the result is left untouched,
>> but below an overlay.  The overlay is used to know where to insert the
>> result and to display the status/progress of the execution.  If the file
>> is closed and the execution fails, nothing is lost, the old result is
>> still available.
>>
>> If that technique looks safe enough and interesting, I can prepare a set
>> of patches so that we can discuss it further and, maybe, add it in Org.
>
> Using overlay also sounds good.
>
> I am wondering what you do when there is no result yet or when user
> edits whatever is under the overlay,
> but that's just a technical detail.

I don't think it's really robust yet. We'll see how to improve/change
it.

I'll prepare the set of patchs.


Bruno


>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Jack Kamm]

Bruno Barbier <brubar.cs@gmail.com> writes:

> FWIW, I've been trying to use asynchronous blocks for everything, not
> only the source blocks that are based on the comint mode.  I think it
> would be good if ob-core itself could provide an asynchronous API.  I've
> modified my Org so that it does have such an API.  This is work in
> progress; let me describe it.
>
> I've modified ob-core itself to allow asynchronicity.  In the
> asynchrosous case, instead of calling:
>
>   (org-babel-execute:LANG body params)
>
> I'm calling:
>
>   (org-babel-schedule:LANG body params handle-result)
>
> where `org-babel-schedule:LANG' is in charge of calling `handle-result'
> with the result (or the error) when it is known; `handle-result' takes
> care to call `org-babel-insert-result' at the correct place (and
> `org-babel-insert-result' is only called with a real result).

Sounds interesting, a couple questions:

1. Which languages have you implemented/tested this on?

2. Does it apply for sessions, nonsessions, or both?

> While the execution is pending, I'm using the same technique that Org is
> using when a source block is being edited: the result is left untouched,
> but below an overlay.  The overlay is used to know where to insert the
> result and to display the status/progress of the execution.  If the file
> is closed and the execution fails, nothing is lost, the old result is
> still available.

Also interesting, I think it's worth exploring/testing this overlay idea
out. Does that mean that output is asynchronously printing into the Org
buffer? It sounds cool but I wonder if it might cause problems while
trying to edit another part of the buffer.

Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Matt]

If I followed correctly, the topic switched to discussing async generally within Babel.  I've started a new thread accordingly.

 ---- On Tue, 06 Feb 2024 20:24:13 +0100 Bruno Barbier

 > FWIW, I've been trying to use asynchronous blocks for everything, not only the source blocks that are based on the comint mode.

I've been trying to figure out how to make everything async by default.  I'm super interested to see what you've come up with.

 > I think it would be good if ob-core itself could provide an asynchronous API.

Fortunately or unfortunately, depending on how you look at it, Babel already does.

The challenge is that the Org Babel API has grown piecemeal over 14 years.  It's been written by several authors with limited time and knowledge.  The result, while powerful and useful, is a bit of a hodgepodge.  A prime example is that the concepts of "persistence" and "synchronicity" are conflated.  "Session" is often used to mean "asynchronous" even though the two ideas are orthogonal.  Emacs provides primitives that could make non-session blocks asynchronous.  It's historical accident that blocks aren't async by default.

For me, the issue is that the Babel API needs some high level perspective in order to make it consistent.  I see the following terms as guides.  If we can separate these concepts within the API, then Babel to *feel* like an API:

- "Session" means a shell environment is "persistent."  Each call is executed in the same environment.  State exists between calls.

- "Non-session" means a shell environment is "temporary."  Each call is executed in an independent environment.  State does not exist between calls.

- "Synchronous" means that execution prevents the user from editing the document while results are obtained.

- "Asynchronous" means that execution does not prevent the user from editing the document while results are obtained.

 > I've modified my Org so that it does have such an API.  This is work in progress; let me describe it.
 >
 > I've modified ob-core itself to allow asynchronicity.  In the asynchrosous case, instead of calling:
 >
 >   (org-babel-execute:LANG body params)
 >
 > I'm calling:
 >
 >   (org-babel-schedule:LANG body params handle-result)
 >
 > where `org-babel-schedule:LANG' is in charge of calling `handle-result' with the result (or the error) when it is known; `handle-result' takes care to call `org-babel-insert-result' at the correct place (and `org-babel-insert-result' is only called with a real result).
 >
 > While the execution is pending, I'm using the same technique that Org is using when a source block is being edited: the result is left untouched, but below an overlay.  The overlay is used to know where to insert the result and to display the status/progress of the execution.  If the file is closed and the execution fails, nothing is lost, the old result is still available.

The use of the overlay is a really cool idea!

I hesitate to say that's a good way to convey success or failure.  If a process failed, I want to see the output which tells me why so that I can correct it.  Or, I might actually want the failure output.  Maybe I want to literally demonstrate what a code failure looks like.  Maybe I want to use that output in another block.  For example, shell blocks have multiple output types.  A shell process may return standard output/error or a failure code.  The result of the failure may trigger something else.

However, using an overlay to communicate "the process is still running" could be good.  We'd need to be careful about accessibility, though, and make sure the overlay is apparent, visually and otherwise.

 > If that technique looks safe enough and interesting, I can prepare a set of patches so that we can discuss it further and, maybe, add it in Org.

Please do!  I'm super interested.  I've put a lot of thought into how we might make Babel async by default.  I'm excited to see your interest in the topic and look forward to seeing what you've come up with.

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Bruno Barbier]

[-- Attachment #1: Type: text/plain, Size: 8614 bytes --]


Hi Matt, Jack, Ihor,

Sorry for the late reply.  Cleaning the code took me longer than
expected.


Jack Kamm <jackkamm@gmail.com> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> FWIW, I've been trying to use asynchronous blocks for everything, not
>> only the source blocks that are based on the comint mode.
>> ...
>
> Sounds interesting, a couple questions:
>
> 1. Which languages have you implemented/tested this on?

I'm not using it with official org backends (yet).  I'm using it with
several custom backends that I'm working on.  One of the backend
delegate the block executions to emacs subprocesses: so I have a kind of
asynchronous executions for free for any language, including elisp
itself.


> 2. Does it apply for sessions, nonsessions, or both?
>

The new API itself is more about how to wait for and display one block
result.  So, it's not really aware of sessions.

But, I usually try to think as "no session" as a "one shot" session
(like Matt wrote in his email).  So, in that sense, it works for both
anyway ;-)


> Also interesting, I think it's worth exploring/testing this overlay idea
> out. Does that mean that output is asynchronously printing into the Org
> buffer? It sounds cool but I wonder if it might cause problems while
> trying to edit another part of the buffer.

Currently, I've limited the progress feedback to fit on one line to
avoid anoying vertical display jumps.  When the computation is
successful, the result is inserted as usual, and, that may be annoying;
when it updates a previous result of the same height, it's ok.  Else, we
could fold it to stay on one line too (using the overlay), until the
user explicitly request to see it.

Jack Kamm <jackkamm@gmail.com> writes:
>
> That all sounds reasonable...if you work on this, let me know if you
> want any help with testing.

Thanks.  I'll definitely appreciate your help, to test the current
code with the Python backend or any other backend if you prefer.


Matt <matt@excalamus.com> writes:

> If I followed correctly, the topic switched to discussing async
> generally within Babel.  I've started a new thread accordingly.

Indeed. Thank you!


>  > FWIW, I've been trying to use asynchronous blocks for everything, not only the source blocks that are based on the comint mode.
>
> I've been trying to figure out how to make everything async by default.  I'm super interested to see what you've come up with.

Good to know.  Let's we make this happen! 


>  > I think it would be good if ob-core itself could provide an asynchronous API.
>
> Fortunately or unfortunately, depending on how you look at it, Babel already does.
>

> The challenge is that the Org Babel API has grown piecemeal over 14 years.  It's been written by several authors with limited time and knowledge.  The result, while powerful and useful, is a bit of a hodgepodge.  A prime example is that the concepts of "persistence" and "synchronicity" are conflated.  "Session" is often used to mean "asynchronous" even though the two ideas are orthogonal.  Emacs provides primitives that could make non-session blocks asynchronous.  It's historical accident that blocks aren't async by default.

> For me, the issue is that the Babel API needs some high level perspective in order to make it consistent.

> I see the following terms as guides.  If we can separate these concepts within the API, then Babel to *feel* like an API:
>
> - "Session" means a shell environment is "persistent."  Each call is executed in the same environment.  State exists between calls.
>
> - "Non-session" means a shell environment is "temporary."  Each call is executed in an independent environment.  State does not exist between calls.
>
> - "Synchronous" means that execution prevents the user from editing the document while results are obtained.
>
> - "Asynchronous" means that execution does not prevent the user from editing the document while results are obtained.

I mostly think the same.  Sessions (including the "none" session)
definitely need some generic API and some generic tests that all
backends could just reuse.

To execute Python blocks, using the proposed async API:

   - I've (re)implemented the "asynchronous with session" case (copying/pasting
     the relevant part from ob-python).
   
   - The "synchronous case" is just artificially blocking the user until
     the asynchronous result is known (which looks incredibly tricky to
     implement if even possible...).
   
   - The "no session" case is just about creating a new unique session
     and throwing it away immediately.

But, some users may rely on some particular behavior, of the current
implementations, that may be hard to implement in such a generic way.


> The use of the overlay is a really cool idea!
>
> I hesitate to say that's a good way to convey success or failure.  If a process failed, I want to see the output which tells me why so that I can correct it.  Or, I might actually want the failure output.  Maybe I want to literally demonstrate what a code failure looks like.  Maybe I want to use that output in another block.  For example, shell blocks have multiple output types.  A shell process may return standard output/error or a failure code.  The result of the failure may trigger something else.

I'm not sure I fully understand what you mean. The API just assumes the
backend returns the outcome: either success or failure, where failure
means "no result" (the previous result, if it exists, is even preserved
in the document).  The backend is free to transform a failure into a
success to make that result available though.


> However, using an overlay to communicate "the process is still running" could be good.  We'd need to be careful about accessibility, though, and make sure the overlay is apparent, visually and otherwise.

You should be able to test my current implementation, see below.  It's
almost too visual in my opinion.  But, that's probably something that we
should make easy to configure.

>  > If that technique looks safe enough and interesting, I can prepare a set of patches so that we can discuss it further and, maybe, add it in Org.
>
> Please do!  I'm super interested.  I've put a lot of thought into how we might make Babel async by default.  I'm excited to see your interest in the topic and look forward to seeing what you've come up with.

Good to know!


So, here we go.  You'll find attach a set of patchs.  It works for me with
Emacsc 30.50 and 9.7-pre (from today).

I didn't check yet that the code and the commits follow all the
guidelines.  This is just a preliminary version for feedbacks.
Corrections/critiques are welcome, but don't check the details until I
check them myself.


The 5 first patchs provide an API to handle asynchronous execution in
ob-core, i.e. an API to report progress and to insert results in the
asynchronous case.  The 5th one isn't really about asynchronicity; but
it adds a new keyword `:execute-with' that allows to delegate block
executions to some other package; it's useful for testing or plugging
other execution engines.

That's a cleaned-up version of what I been using myself for a while,
with 4 different values for `:execute-with'.

The remaining patchs are new code that I've just written to show how to
use this new API.

I tried first to use it for ob-python, as an example.  I just needed to
figure out where to place the callbacks ... well ... "just" ... :)


So, I decided to rewrite the whole thing, taking code from the
synchronous case (following `org-babel-python-evaluate-session').  I
also created a package that contains all the functions that should be
reusable for any language.  The patch [1] adds some generic functions to
help dealing with asynchronicity (process, comint, etc.). The patch [2]
shows a new possible way to execute python code blocks, both
synchronously and asynchronously, with or without sessions.  You should
just need to open [3] and follow what's written there, and execute the
existing bash and python code blocks.

Note that this will create a folder `scratch/bba-ob-core-async' in your
repository to place the temporary files there.  If you know a better way
to do this, let me know, thanks.

I think the first 5 patchs could be included almost as-is in org.  About
the emaining ones, I'm not sure exactly how we should proceed.

Let me know if you need help to test it.

Feedbacks, corrections, critiques, etc are most welcome!

Thanks!

Bruno




[1] lisp/org-elib-async.el: New package about async helpers
[2] scratch/bba-ob-core-async: Some temporary test files
[3] scratch/bba-ob-core-async/my-async-tests.org



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-core-async-Add-faces-1-5.patch --]
[-- Type: text/x-patch, Size: 1367 bytes --]

From f67829454ac0d3cd142da1bd0006efa37acce588 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:31:36 +0100
Subject: [PATCH 1/8] ob-core async: Add faces [1/5]

lisp/org-faces.el (org-async-scheduled, org-async-pending,
org-async-failure): new faces
---
 lisp/org-faces.el | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/lisp/org-faces.el b/lisp/org-faces.el
index 0e20de51a..5a8a8fd51 100644
--- a/lisp/org-faces.el
+++ b/lisp/org-faces.el
@@ -736,6 +736,24 @@ (defface org-mode-line-clock-overrun
   "Face used for clock display for overrun tasks in mode line."
   :group 'org-faces)
 
+(defface org-async-scheduled '((t :inherit org-tag :background "gray"))
+  "Face for babel results for code blocks that are scheduled for execution."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
+(defface org-async-pending '((t :inherit org-checkbox :background "dark orange"))
+  "Face for babel results for code blocks that are running."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
+(defface org-async-failure '((t :inherit org-warning))
+  "Face for babel results for code blocks that have failed."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
 (provide 'org-faces)
 
 ;;; org-faces.el ends here
-- 
2.43.0


[-- Attachment #3: 0002-ob-core-async-Add-org-babel-async-tools-2-5.patch --]
[-- Type: text/x-patch, Size: 10450 bytes --]

From 8ecdc2159d85648949b477359793f007005bb7ca Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:32:00 +0100
Subject: [PATCH 2/8] ob-core async: Add org-babel--async tools [2/5]

---
 lisp/ob-core.el | 219 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 219 insertions(+)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index bfeac257b..bb44f91cb 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -792,6 +792,225 @@ (defun org-babel-session-buffer (&optional info)
     (when (org-babel-comint-buffer-livep buffer-name)
       buffer-name)))
 
+(defun org-babel--async-status-face (status)
+  (pcase status
+    (:scheduled 'org-async-scheduled)
+    (:pending   'org-async-pending)
+    (:failure   'org-async-failure)
+    (:success   nil)
+    (_ (error "Not a status"))
+    ))
+
+(defun org-babel--async-make-overlay (beg end)
+  "Create an overlay between positions BEG and END and return it."
+  (let ((overlay (make-overlay beg end))
+        (read-only
+	 (list
+	  (lambda (&rest _)
+	    (user-error
+	     "Cannot modify an area being updated"))))
+        )
+    (cl-flet ((make-read-only
+               (ovl)
+               (overlay-put ovl 'modification-hooks read-only)
+               (overlay-put ovl 'insert-in-front-hooks read-only)
+               (overlay-put ovl 'insert-behind-hooks read-only))
+              )
+      (overlay-put overlay 'org-babel--async-type 'org-babel--async-note)
+      (overlay-put overlay 'face 'secondary-selection)
+      (overlay-put overlay 'help-echo "Pending src block result...")
+      (make-read-only overlay)
+      overlay)))
+
+(defun org-babel--async-result-region (inline-elem &optional info)
+  "Return the region of the results, for the source block at point."
+  (unless info (setq info (org-babel-get-src-block-info)))
+  (save-excursion
+    (when-let ((res-begin  (org-babel-where-is-src-block-result nil info)))
+      (cons res-begin
+            (save-excursion
+              (goto-char res-begin)
+              (if inline-elem
+                  ;; Logic copy/pasted from org-babel-where-is-src-block-result.
+	          (let ((result (org-element-context)))
+		    (and (org-element-type-p result 'macro)
+		         (string= (org-element-property :key result)
+			          "results")
+		         (progn
+			   (goto-char (org-element-end result))
+			   (skip-chars-backward " \t")
+			   (point))))
+                ;; Logic copy/pasted from hide-result
+                (beginning-of-line)
+                (let ((case-fold-search t))
+                  (unless (re-search-forward org-babel-result-regexp nil t)
+	            (error "Not looking at a result line")))
+                (org-babel-result-end)
+                ))))))
+
+(defun org-babel--async-feedbacks (info handle-result
+                                       result-params exec-start-time)
+  "Flag the result as 'scheduled' and return how to handle feedbacks.
+
+Use overlays to report progress and status to the user.  Do not delete
+the existing result unless a new one is available.  When the result is
+available, remove the async overlays and insert the result as usual,
+like for a synchronous result.  In case of failure, use an overlay to
+report the error.
+
+The returned function handles 3 types of feedbacks:
+   - (:success R):   Evaluation is successful; result is R.
+   - (:failure ERR): Evaluation failed; error is ERR.
+   - (:pending P): Outcome still pending; current progress is P."
+  ;; FIXME: INFO CMD ... Nothing is used but handle-result here !!
+  (let (;; copy/pasted from org-babel-insert-result
+        (inline-elem (let ((context (org-element-context)))
+		       (and (memq (org-element-type context)
+			          '(inline-babel-call inline-src-block))
+			    context)))
+        result-indent
+        )
+    (cl-labels
+        ((eot-point (start)
+           "Move to End Of Title after START"
+           (if inline-elem
+               (org-element-end inline-elem)
+             (save-excursion (goto-char start)
+                             (forward-line 1) (point))))
+         (after-indent (pt)
+           "Move after indentation, starting at PT."
+           (save-excursion (goto-char pt) (re-search-forward "[[:blank:]]*")))
+         (mk-result-overlays ()
+           ;; Make 2 overlays to handle the pending result: one title
+           ;; (first line) and one for the body.
+           (pcase-let ((`(,start . ,end) (org-babel--async-result-region
+                                          inline-elem info)))
+             (let ((anchor-end (eot-point start)))
+               (cons (org-babel--async-make-overlay
+                      (after-indent start)
+                      (1- anchor-end))
+                     (org-babel--async-make-overlay
+                      anchor-end end)))))
+         (add-style (status txt)
+           ;; Add the style matching STATUS over the text TXT.
+           (propertize txt 'face (org-babel--async-status-face status)))
+
+         (short-version-of (msg)
+           ;; Compute the short version of MSG, to display in the header.
+           ;; Must return a string.
+           (if msg
+               (car (split-string (format "%s" msg) "\n" :omit-nulls))
+             ""))
+         (update (ovl-title status msg)
+           ;; Update the title overlay to match STATUS and MSG.
+           (let (header)
+             (overlay-put ovl-title
+                          'face
+                          (org-babel--async-status-face status)
+                          )
+             (overlay-put ovl-title
+                          'before-string (pcase status
+                                           (:scheduled "⏱")
+                                           (:pending "⏳")
+                                           (:failure "❌")
+                                           (:success "✔️")))
+
+             (overlay-put ovl-title
+                          'after-string
+                          (propertize (format " |%s|"
+                                              (if (eq :failure status)
+                                                  (if (consp msg) (car msg)
+                                                    (format "%s" msg))
+                                                  (short-version-of msg)))
+                                      'face (org-babel--async-status-face status)))
+             ))
+         (remove-previous-overlays ()
+           ;; Remove previous title and body overlays.
+           (mapc (lambda (ovl)
+                   (when (eq 'org-babel--async-note
+                             (overlay-get ovl 'org-babel--async-type))
+                     (delete-overlay ovl)))
+                 (when-let ((region (org-babel--async-result-region
+                                     inline-elem info)))
+                   ;; Not sure why, but we do need to start before
+                   ;; point min, else, in some cases, some overlays
+                   ;; are not found.
+                   (overlays-in (max (1- (car region)) (point-min))
+                                (cdr region))))))
+
+      (remove-previous-overlays)
+
+      ;; Ensure there is a non-empty region for the result.
+      (save-excursion
+         (unless (org-babel-where-is-src-block-result (not inline-elem) nil nil)
+          (org-babel-insert-result
+           ;; Use " " for the empty result. That cannot be nil, else it's interpreted
+           ;; as a list. We need at least one char, to separate markers if any.
+           " \n"
+           result-params
+           info nil
+           (nth 0 info) ; lang
+           exec-start-time
+           )))
+
+      ;; Create the overlays that span the result title and its body.
+      (pcase-let ((`(,title-ovl . ,body-ovl) (mk-result-overlays)))
+        ;; Flag the result as ":scheduled".
+        (update title-ovl :scheduled nil)
+
+        ;; The callback, that runs in the org buffer at point.
+        (let ((buf (current-buffer))
+              (pt  (point-marker)))
+          (lambda (feedback)
+            (message "ob-core: Handling outcome at %s@%s: %s" pt buf feedback)
+            (with-current-buffer buf
+              (save-excursion
+                (goto-char pt)
+                (pcase feedback
+                  (`(:success ,r)
+                   ;; Visual beep that the result is available.
+                   (update title-ovl :success r)
+                   (sit-for 0.2)
+                   ;; We remove all overlays and let org insert the result
+                   ;; as it would in the synchronous case.
+                   (delete-overlay title-ovl)
+                   (delete-overlay body-ovl)
+                   (funcall handle-result r))
+
+                  (`(:pending ,r)
+                   ;; Still waiting for the outcome. Update our
+                   ;; overlays with the progress info R.
+                   (message "Updating block at %s@%s" pt buf)
+                   (update title-ovl :pending r))
+
+                  (`(:failure ,err)
+                   ;; We didn't get a result. We update our overlays
+                   ;; to report that failure. And unlock the old
+                   ;; result.
+                   (overlay-put title-ovl 'face nil)
+                   (update title-ovl :failure err)
+                   (delete-overlay body-ovl))
+
+                  (_ (error "Invalid outcome"))
+                  )
+                ))
+            nil))))))
+
+
+(cl-defun org-babel--async-p (params &key default)
+  "Return a non-nil value when the execution is asynchronous.
+Get the value of the :nasync argument and convert it."
+  (if-let ((binding (assq :nasync params)))
+      (pcase (cdr binding)
+        ((pred (not stringp))
+         (error "Invalid value for :nasync argument"))
+        ((or "no" "n")  nil)
+        ((or "yes" "y") t)
+        (_ (error "Invalid value for :nasync argument")))
+    default))
+
+
+
 ;;;###autoload
 (defun org-babel-execute-src-block (&optional arg info params executor-type)
   "Execute the current source code block and return the result.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-ob-core-async-Refactor-handle-result-3-5.patch --]
[-- Type: text/x-patch, Size: 5958 bytes --]

From e8fe2f922a992de48ca80c52f530b0aa5fb45ea3 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:32:22 +0100
Subject: [PATCH 3/8] ob-core async: Refactor handle-result [3/5]

lisp/ob-core.el (org-babel-execute-src-block): Refactor the code to
prepare for the next change: move the part handling the result in its
own function `handle-result'.
---
 lisp/ob-core.el | 105 ++++++++++++++++++++++++------------------------
 1 file changed, 53 insertions(+), 52 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index bb44f91cb..f8071a534 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -1092,7 +1092,58 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		      (make-directory d 'parents)
 		      d))))
 		 (cmd (intern (concat "org-babel-execute:" lang)))
-		 result exec-start-time)
+                 (exec-start-time (current-time))
+                 (handle-result
+                  (lambda (result)
+                    (setq result
+		          (if (and (eq (cdr (assq :result-type params)) 'value)
+			           (or (member "vector" result-params)
+				       (member "table" result-params))
+			           (not (listp result)))
+		              (list (list result))
+		            result))
+	            (let ((file (and (member "file" result-params)
+			             (cdr (assq :file params)))))
+		      ;; If non-empty result and :file then write to :file.
+		      (when file
+		        ;; If `:results' are special types like `link' or
+		        ;; `graphics', don't write result to `:file'.  Only
+		        ;; insert a link to `:file'.
+		        (when (and result
+			           (not (or (member "link" result-params)
+				            (member "graphics" result-params))))
+		          (with-temp-file file
+		            (insert (org-babel-format-result
+			             result
+			             (cdr (assq :sep params)))))
+		          ;; Set file permissions if header argument
+		          ;; `:file-mode' is provided.
+		          (when (assq :file-mode params)
+		            (set-file-modes file (cdr (assq :file-mode params)))))
+		        (setq result file))
+		      ;; Possibly perform post process provided its
+		      ;; appropriate.  Dynamically bind "*this*" to the
+		      ;; actual results of the block.
+		      (let ((post (cdr (assq :post params))))
+		        (when post
+		          (let ((*this* (if (not file) result
+				          (org-babel-result-to-file
+				           file
+				           (org-babel--file-desc params result)
+                                           'attachment))))
+		            (setq result (org-babel-ref-resolve post))
+		            (when file
+			        (setq result-params (remove "file" result-params))))))
+	              (unless (member "none" result-params)
+		        (org-babel-insert-result
+		         result result-params info
+                         ;; append/prepend cannot handle hash as we accumulate
+                         ;; multiple outputs together.
+                         (when (member "replace" result-params) new-hash)
+                         lang
+                         (time-subtract (current-time) exec-start-time)))
+	              (run-hooks 'org-babel-after-execute-hook)
+                      result))))
 	    (unless (fboundp cmd)
 	      (error "No org-babel-execute function for %s!" lang))
 	    (message "Executing %s %s %s..."
@@ -1107,57 +1158,7 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		       (if name
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
-	    (setq exec-start-time (current-time)
-                  result
-		  (let ((r (save-current-buffer (funcall cmd body params))))
-		    (if (and (eq (cdr (assq :result-type params)) 'value)
-			     (or (member "vector" result-params)
-				 (member "table" result-params))
-			     (not (listp r)))
-			(list (list r))
-		      r)))
-	    (let ((file (and (member "file" result-params)
-			     (cdr (assq :file params)))))
-	      ;; If non-empty result and :file then write to :file.
-	      (when file
-		;; If `:results' are special types like `link' or
-		;; `graphics', don't write result to `:file'.  Only
-		;; insert a link to `:file'.
-		(when (and result
-			   (not (or (member "link" result-params)
-				  (member "graphics" result-params))))
-		  (with-temp-file file
-		    (insert (org-babel-format-result
-			     result
-			     (cdr (assq :sep params)))))
-		  ;; Set file permissions if header argument
-		  ;; `:file-mode' is provided.
-		  (when (assq :file-mode params)
-		    (set-file-modes file (cdr (assq :file-mode params)))))
-		(setq result file))
-	      ;; Possibly perform post process provided its
-	      ;; appropriate.  Dynamically bind "*this*" to the
-	      ;; actual results of the block.
-	      (let ((post (cdr (assq :post params))))
-		(when post
-		  (let ((*this* (if (not file) result
-				  (org-babel-result-to-file
-				   file
-				   (org-babel--file-desc params result)
-                                   'attachment))))
-		    (setq result (org-babel-ref-resolve post))
-		    (when file
-		      (setq result-params (remove "file" result-params))))))
-	      (unless (member "none" result-params)
-	        (org-babel-insert-result
-	         result result-params info
-                 ;; append/prepend cannot handle hash as we accumulate
-                 ;; multiple outputs together.
-                 (when (member "replace" result-params) new-hash)
-                 lang
-                 (time-subtract (current-time) exec-start-time))))
-	    (run-hooks 'org-babel-after-execute-hook)
-	    result)))))))
+            (funcall handle-result (save-current-buffer (funcall cmd body params))))))))))
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
   "Expand BODY with PARAMS.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-ob-core-async-Handle-nasync-param-4-5.patch --]
[-- Type: text/x-patch, Size: 2620 bytes --]

From c9d64263af8d9d9b8efc83585323cc6d876018dc Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:32:45 +0100
Subject: [PATCH 4/8] ob-core async: Handle :nasync param [4/5]

---
 lisp/ob-core.el | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index f8071a534..0f0a36b70 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -1091,7 +1091,10 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		    (let ((d (file-name-as-directory (expand-file-name dir))))
 		      (make-directory d 'parents)
 		      d))))
-		 (cmd (intern (concat "org-babel-execute:" lang)))
+                 (async (org-babel--async-p params))
+                 (cmd (intern (concat "org-babel-"
+                                      (if async "schedule" "execute")
+                                      ":" lang)))
                  (exec-start-time (current-time))
                  (handle-result
                   (lambda (result)
@@ -1145,8 +1148,8 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 	              (run-hooks 'org-babel-after-execute-hook)
                       result))))
 	    (unless (fboundp cmd)
-	      (error "No org-babel-execute function for %s!" lang))
-	    (message "Executing %s %s %s..."
+	      (error "No org-babel-execute function for %s: %s!" lang (symbol-name cmd)))
+	    (message "Executing %s %s %s %s..."
 		     (capitalize lang)
                      (pcase executor-type
                        ('src-block "code block")
@@ -1154,11 +1157,17 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
                        ('babel-call "call")
                        ('inline-babel-call "inline call")
                        (e (symbol-name e)))
+                     (if async "async" "")
 		     (let ((name (nth 4 info)))
 		       (if name
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
-            (funcall handle-result (save-current-buffer (funcall cmd body params))))))))))
+            (if (not async)
+                (funcall handle-result (save-current-buffer (funcall cmd body params)))
+              (let ((handle-feedback
+                     (org-babel--async-feedbacks info handle-result result-params exec-start-time)))
+                (funcall cmd body params handle-feedback))))))))))
+
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
   "Expand BODY with PARAMS.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-ob-core-async-Add-execute-with-5-5.patch --]
[-- Type: text/x-patch, Size: 2458 bytes --]

From 8b311516505f693488bfef661905f34e7375539c Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:33:09 +0100
Subject: [PATCH 5/8] ob-core async: Add :execute-with [5/5]

---
 lisp/ob-core.el | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 0f0a36b70..1de94330a 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -1092,9 +1092,18 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		      (make-directory d 'parents)
 		      d))))
                  (async (org-babel--async-p params))
-                 (cmd (intern (concat "org-babel-"
-                                      (if async "schedule" "execute")
-                                      ":" lang)))
+                 (execute-with (let ((be (cdr (assq :execute-with params))))
+                                 (when (equal be "none") (setq be nil))
+                                 be))
+                 (cmd (intern (or (and execute-with
+                                       (concat execute-with "-" (if async "schedule" "execute")))
+                                  (concat "org-babel-"
+                                          (if async "schedule" "execute")
+                                          ":" lang))))
+                 (cmd-args (let ((ps (list body params)))
+                             (when execute-with
+                               (setq ps (cons lang ps)))
+                             ps))
                  (exec-start-time (current-time))
                  (handle-result
                   (lambda (result)
@@ -1163,10 +1172,10 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
             (if (not async)
-                (funcall handle-result (save-current-buffer (funcall cmd body params)))
+                (funcall handle-result (save-current-buffer (apply cmd cmd-args)))
               (let ((handle-feedback
                      (org-babel--async-feedbacks info handle-result result-params exec-start-time)))
-                (funcall cmd body params handle-feedback))))))))))
+                (apply cmd (nconc cmd-args (list handle-feedback))))))))))))
 
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #7: 0006-lisp-org-elib-async.el-New-package-about-async-helpe.patch --]
[-- Type: text/x-patch, Size: 14462 bytes --]

From 50ecb37089b96cd0fe4107e4d74357c151540e34 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:33:23 +0100
Subject: [PATCH 6/8] lisp/org-elib-async.el: New package about async helpers

---
 lisp/org-elib-async.el | 323 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 323 insertions(+)
 create mode 100644 lisp/org-elib-async.el

diff --git a/lisp/org-elib-async.el b/lisp/org-elib-async.el
new file mode 100644
index 000000000..ba4e08869
--- /dev/null
+++ b/lisp/org-elib-async.el
@@ -0,0 +1,323 @@
+;;; org-elib-async.el --- Helper to write asynchronous functions -*- lexical-binding: t -*-
+
+;; Copyright (C) 2024 Bruno BARBIER
+
+;; Author: Bruno BARBIER
+;; Version: 0.0.0
+;; Maintainer: Bruno BARBIER
+;; Keywords:
+;; Status: WORK IN PROGRESS.  DO NOT USE.
+;; URL:
+;; Compatibility: GNU Emacs 30.0.50
+;;
+;; This file is NOT (yet) part of GNU Emacs.
+
+;; This program is free software; you can redistribute it and/or
+;; modify it under the terms of the GNU General Public License as
+;; published by the Free Software Foundation; either version 2 of
+;; the License, or (at your option) any later version.
+
+;; This program is distributed in the hope that it will be
+;; useful, but WITHOUT ANY WARRANTY; without even the implied
+;; warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+;; PURPOSE.  See the GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public
+;; License along with this program; if not, write to the Free
+;; Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+;; MA 02111-1307 USA
+
+
+;;; Commentary:
+;; Names with "--" are for functions and variables that are meant to be for
+;; internal use only.
+
+;;;; Description
+;; Some functions to help dealing with asynchronous tasks.
+
+;; The prefix 'org-elib' means that this package should evenutally be
+;; moved into core Emacs.  The functions defined here do NOT depend
+;; nor rely on org itself.
+
+;;; TODOs
+;;
+;; - Keywords
+;;
+
+
+;;; Code:
+
+;;;; Process
+;;
+(cl-defun org-elib-async-process (command &key input callback)
+  "Execute COMMAND.
+
+A quick naive featureless boggus wrapper around `make-process' to
+receive the result when the process is done.
+
+When INPUT is non-nil, use it as the COMMAND standard input.  Let DATA
+be the COMMAND output, if COMMAND succeeds, call CALLBACK with
+'(:success DATA), else, call CALLBACK with '(:failure DATA)."
+  (let* ((stdout-buffer (generate-new-buffer "*org-elib-async-process*"))
+         (get-outcome
+          (lambda (process)
+            (with-current-buffer stdout-buffer
+              (let* ((exit-code (process-exit-status process))
+                     (real-end ;; Getting rid of the user message.
+                      (progn (goto-char (point-max))
+                             (forward-line -1)
+                             (point)))
+                     (txt (string-trim (buffer-substring-no-properties
+                                        (point-min) real-end))))
+                (list (if (eq 0 exit-code) :success :failure)
+                      (if (not (string-empty-p txt)) txt
+                        (and (not (eq 0 exit-code)) exit-code)))))))
+         (process (make-process
+                   :name "*org-elib-async-process*"
+                   :buffer  stdout-buffer
+                   :command command
+                   :connection-type 'pipe))
+         (sentinel
+          (lambda (&rest _whatever)
+            (pcase (process-status process)
+              ('run )
+              ('stop)
+              ((or 'exit 'signal)
+               (funcall callback (funcall get-outcome process)))
+              (_ (error "Not a real process"))))))
+    (add-function :after (process-sentinel process) sentinel)
+    (when input
+      (process-send-string process input)
+      (process-send-eof process))
+    process))
+;; (org-elib-async-process (list "date") :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "false") :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "true") :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash" "-c" "bash") :input "date" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "date" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "false" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "true" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "sleep 2; date" :callback (lambda (o) (message "outcome: %S" o)))
+
+
+;;;; Wait for a process until some condition becomes true.
+
+(define-error 'org-elib-async-timeout-error
+              "Timeout waiting for a process.")
+
+(cl-defun org-elib-async-wait-condition ( cond-p
+                                          &key
+                                          (tick .3) (message "Waiting")
+                                          (nb_secs_between_messages 5)
+                                          timeout)
+  "Wait until the condition COND-P returns non-nil.
+Repeatedly call COND-P with no arguments, about every TICK seconds,
+until it returns a non-nil value.  Return that non-nil value.  When
+TIMEOUT (seconds) is non-nil, raise an `org-elib-async-timeout-error' if
+the COND-P is still nil after TIMEOUT seconds.  Assume COND-P calls cost
+0s.  Do NOT block display updates.  Do not block process outputs.  Do
+not block idle timers.  Do block the user, letting him/her know why, but
+do not display more messages than one every NB_SECS_BETWEEN_MESSAGES.
+Default MESSAGE is \"Waiting\".  Use 0.3s as the default for TICK."
+  ;; FIXME: Still not sure if it's possible to write such a function.
+  (let ((keep-waiting t)
+        (result nil)
+        (start (float-time))
+        elapsed
+        last-elapsed
+        )
+    (while keep-waiting
+      (setq result (funcall cond-p))
+      (if result
+          (setq keep-waiting nil)
+        (sleep-for 0.01)
+        (redisplay :force)
+        (setq elapsed (- (float-time) start))
+        (when (and timeout (> elapsed timeout))
+          (signal 'org-timeout-error message elapsed))
+        ;; Let the user know, without flooding the message area.
+        (if (and last-elapsed (> (- elapsed last-elapsed) nb_secs_between_messages))
+            (message (format "%s ...(%.1fs)" message elapsed)))
+        (unless (sit-for tick :redisplay)
+          ;; Emacs has something to do; let it process new
+          ;; sub-processes outputs in case there are some.
+          (accept-process-output nil 0.01))))
+    result))
+
+
+
+;;;; Comint: a FIFO queue of tasks with callbacks
+;; org-elib-async-comint-queue executes tasks in a FIFO order. For each
+;; task, it identifies the text output for that
+;; task. org-elib-async-comint-queue does NOT remove prompts, or other
+;; useless texts; this is the responsibility of the user.  Currently,
+;; org-elib-async-comint-queue assume it has the full control of the
+;; session: no user interaction, no other direct modifications.
+
+(defvar-local org-elib-async-comint-queue--todo :NOT-SET
+  "A FIFO queue of pending executions.")
+
+
+(defvar-local org-elib-async-comint-queue--unused-output ""
+  "Process output that has not been used yet.")
+
+(defvar-local org-elib-async-comint-queue--incoming-text ""
+  "Newly incoming text, added by the process filter, not yet handled.")
+
+(defvar-local org-elib-async-comint-queue--current-task nil
+  "The task that is currently running.")
+
+(defvar-local org-elib-async-comint-queue--process-filter-running nil
+  "non-nil when filter is running.")
+
+(defvar-local org-elib-async-comint-queue--incoming-timer nil
+  "A timer, when handling incoming text is scheduled or running.")
+
+
+(defvar-local org-elib-async-comint-queue--handle-incoming-running
+    nil
+  "True when the incoming text handler is running.")
+
+(defun org-elib-async-comint-queue--handle-incoming ()
+  (when org-elib-async-comint-queue--handle-incoming-running
+    (error "Bad filter call detected: Kill buffer %s!" (current-buffer)))
+  (setq org-elib-async-comint-queue--handle-incoming-running t)
+
+  ;; Take the incoming text.
+  (setq org-elib-async-comint-queue--unused-output
+        (concat org-elib-async-comint-queue--unused-output
+                org-elib-async-comint-queue--incoming-text))
+  (setq org-elib-async-comint-queue--incoming-text "")
+
+  ;; Process the unused text with the queued tasks
+  (unless org-elib-async-comint-queue--current-task
+    (when org-elib-async-comint-queue--todo
+      (setq org-elib-async-comint-queue--current-task (pop org-elib-async-comint-queue--todo))))
+  (when-let ((task org-elib-async-comint-queue--current-task))
+    (let ((unused org-elib-async-comint-queue--unused-output)
+          (session-buffer (current-buffer))
+          task-start)
+      (setq org-elib-async-comint-queue--unused-output
+            (with-temp-buffer
+              (insert unused)
+              (goto-char (point-min))
+              (while (and task
+                          (setq task-start (point))
+                          (search-forward (car task) nil t))
+                (when (cdr task)
+                  (let ((txt (buffer-substring-no-properties task-start
+                                                             (- (point) (length (car task))))))
+                    (save-excursion (funcall (cdr task) txt))))
+                (setq task (and (buffer-live-p session-buffer)
+                                (with-current-buffer session-buffer (pop org-elib-async-comint-queue--todo)))))
+              (buffer-substring (point) (point-max))))
+      (setq org-elib-async-comint-queue--current-task task)))
+
+  ;; Signal that we are done. If we already have some new incoming text,
+  ;; reschedule to run.
+  (setq org-elib-async-comint-queue--incoming-timer
+        (if (string-empty-p org-elib-async-comint-queue--incoming-text)
+            nil
+          (org-elib-async-comint-queue--wake-up-handle-incoming)))
+
+  ;; We reset it only on success. If it failed for some reason, the
+  ;; comint buffer is in an unknown state: you'll need to kill that
+  ;; buffer.
+  (setq org-elib-async-comint-queue--handle-incoming-running nil))
+
+
+(defun org-elib-async-comint-queue--wake-up-handle-incoming ()
+  "Wake up the handling of incoming chunks of text.
+Assume we are called from the comint buffer."
+  (setq org-elib-async-comint-queue--incoming-timer
+        (run-with-timer
+         0.01 nil
+         (let ((comint-buffer (current-buffer)))
+           (lambda ()
+             (with-local-quit
+               (with-current-buffer comint-buffer
+                 (org-elib-async-comint-queue--handle-incoming))))))))
+
+
+(defun org-elib-async-comint-queue--process-filter (chunk)
+  "Accept the arbitrary CHUNK of text."
+  (setq org-elib-async-comint-queue--incoming-text
+        (concat org-elib-async-comint-queue--incoming-text
+                chunk))
+  :; We delegate the real work outside the process filter, as it is
+   ; not reliable to do anything here.
+  (unless org-elib-async-comint-queue--incoming-timer
+    (org-elib-async-comint-queue--wake-up-handle-incoming)))
+
+
+
+(define-error 'org-elib-async-comint-queue-task-error
+              "Task failure.")
+
+(cl-defun org-elib-async-comint-queue--push (exec &key handle-feedback)
+  "Push the execution of EXEC into the FIFO queue.
+When the task completed, call HANDLE-FEEDBACK with its outcome.  Return
+a function that waits for and return the result on succes, raise on
+failure."
+  (let* ((tid (org-id-uuid))
+         (start-tag (format "ORG-ELIB-ASYNC_START_%s" tid))
+         (end-tag (format "ORG-ELIB-ASYNC_END___%s" tid))
+         (result-sb (make-symbol "result"))
+         (on-start
+          (lambda (_)
+            ;; TODO: Use (point) in session to link back to it.
+            (when handle-feedback
+              (funcall handle-feedback '(:pending "running")))))
+         (on-result
+          (lambda (result)
+            ;; Get the result, and report success using HANDLE-FEEDBACK.
+            ;; If something fails, report failure using HANDLE-FEEDBACK.
+            (unwind-protect
+                (let ((outcome
+                       (condition-case-unless-debug exc
+                           (list :success (funcall exec :post-process result))
+                         (error (list :failure exc)))))
+                  (when handle-feedback (save-excursion (funcall handle-feedback outcome)))
+                  (set result-sb outcome))
+              (funcall exec :finally)))))
+
+    (let ((comint-buffer (funcall exec :get-comint-buffer)))
+      (with-current-buffer comint-buffer
+        (setq org-elib-async-comint-queue--todo
+              (nconc org-elib-async-comint-queue--todo
+                     (list (cons start-tag on-start)
+                           (cons end-tag on-result))))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instrs-to-enter))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instr-to-emit-tag start-tag))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :get-code))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instr-to-emit-tag end-tag))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instrs-to-exit))
+
+        (lambda ()
+          (org-elib-async-wait-condition (lambda ()
+                                           (boundp result-sb)))
+          (pcase (symbol-value result-sb)
+            (`(:success ,r) r)
+            (`(:failure ,err) (signal (car err) (cdr err)))))
+        ))))
+
+
+
+(defun org-elib-async-comint-queue-init-if-needed (buffer)
+  "Initialize the FIFO queue in BUFFER if needed."
+  (with-current-buffer buffer
+    (unless (local-variable-p 'org-elib-async-comint-queue--todo)
+      (setq-local org-elib-async-comint-queue--todo nil)
+      (add-hook 'comint-output-filter-functions
+                #'org-elib-async-comint-queue--process-filter nil :local))))
+
+
+
+;;;; Provide
+(provide 'org-elib-async)
+;;; org-elib-async.el ends here
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #8: 0007-lisp-ob-core.el-Notify-when-execution-fails.patch --]
[-- Type: text/x-patch, Size: 3038 bytes --]

From 8db393a06efc3e8a4a3a101be2e59125d1666817 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:33:40 +0100
Subject: [PATCH 7/8] lisp/ob-core.el: Notify when execution fails

lisp/ob-core.el (org-babel-popup-failure-details): New function.
(org-babel-execute-src-block): For synchronous execution, use
`org-babel-popup-failure-details' on failure.
(org-babel--async-feedbacks): Add call to
`org-babel-popup-failure-details' on user request.
---
 lisp/ob-core.el | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 1de94330a..c0f6a398a 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -923,6 +923,16 @@ (defun org-babel--async-feedbacks (info handle-result
                                                     (format "%s" msg))
                                                   (short-version-of msg)))
                                       'face (org-babel--async-status-face status)))
+             (when (eq :failure status)
+               (overlay-put ovl-title
+                            'keymap
+                            (let ((km (make-sparse-keymap)))
+                              (define-key km (kbd "<mouse-1>")
+                                          (lambda ()
+                                            "Display failure details."
+                                            (interactive)
+                                            (org-babel-popup-failure-details msg)))
+                              km)))
              ))
          (remove-previous-overlays ()
            ;; Remove previous title and body overlays.
@@ -1172,11 +1182,26 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
             (if (not async)
-                (funcall handle-result (save-current-buffer (apply cmd cmd-args)))
+                (let ((res-sb (make-symbol "result")))
+                  (condition-case exc
+                      (set res-sb (save-current-buffer (apply cmd cmd-args)))
+                    (error (org-babel-popup-failure-details exc)))
+                  (when (boundp res-sb)
+                    (funcall handle-result (symbol-value res-sb))))
               (let ((handle-feedback
                      (org-babel--async-feedbacks info handle-result result-params exec-start-time)))
                 (apply cmd (nconc cmd-args (list handle-feedback))))))))))))
 
+(defun org-babel-popup-failure-details (exc)
+  "Notify/display"
+  (when-let ((buf (get-buffer org-babel-error-buffer-name)))
+    (with-current-buffer buf (erase-buffer)))
+  (org-babel-eval-error-notify
+   127 ; Don't have exit-code
+   (if (consp exc)
+       (format "%s\n%s\n" (car exc) (cdr exc))
+     (format "%s\n" exc))))
+
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
   "Expand BODY with PARAMS.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #9: 0008-scratch-bba-ob-core-async-Some-temporary-test-files.patch --]
[-- Type: text/x-patch, Size: 20479 bytes --]

From 69db5eef49c69287270b1f171e53ddb5e76625dd Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:33:33 +0100
Subject: [PATCH 8/8] scratch/bba-ob-core-async: Some temporary test files

---
 scratch/bba-ob-core-async/my-async-tests.el  | 172 +++++++
 scratch/bba-ob-core-async/my-async-tests.org | 484 +++++++++++++++++++
 2 files changed, 656 insertions(+)
 create mode 100644 scratch/bba-ob-core-async/my-async-tests.el
 create mode 100644 scratch/bba-ob-core-async/my-async-tests.org

diff --git a/scratch/bba-ob-core-async/my-async-tests.el b/scratch/bba-ob-core-async/my-async-tests.el
new file mode 100644
index 000000000..3550fc88f
--- /dev/null
+++ b/scratch/bba-ob-core-async/my-async-tests.el
@@ -0,0 +1,172 @@
+;;; my-async-tests.el --- Scratch/temporary file: some tests about async -*- lexical-binding: t -*-
+
+;; Copyright (C) 2024 Bruno BARBIER
+
+;; Author: Bruno BARBIER
+;; Status: Temporary tests.
+;; Compatibility: GNU Emacs 30.0.50
+;;
+;; This file is NOT part of GNU Emacs.
+
+;; This program is free software; you can redistribute it and/or
+;; modify it under the terms of the GNU General Public License as
+;; published by the Free Software Foundation; either version 2 of
+;; the License, or (at your option) any later version.
+
+;; This program is distributed in the hope that it will be
+;; useful, but WITHOUT ANY WARRANTY; without even the implied
+;; warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+;; PURPOSE.  See the GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public
+;; License along with this program; if not, write to the Free
+;; Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+;; MA 02111-1307 USA
+
+(require 'org)
+(require 'org-elib-async)
+
+(defun my-shell-babel-schedule (lang body params handle-feedback)
+  "Execute the bash script BODY.
+Execute the shell script BODY using bash.  Use HANDLE-FEEDBACK to report
+the outcome (success or failure)."
+  (unless (equal "bash" lang)
+    (error "Only for bash"))
+  (funcall handle-feedback (list :pending "started"))
+  (org-elib-async-process (list "bash") :input body :callback handle-feedback))
+
+
+(defun my-org-babel-python-how-to-execute (body params)
+  "Return how to execute BODY using python.
+Return how to execute, as expected by
+`org-elib-async-comint-queue--execution'."
+  ;; Code mostly extracted from ob-python, following
+  ;; `org-babel-python-evaluate-session'.
+  ;; Results are expected to differ from ob-python as we follow the
+  ;; same process for all execution paths: asynchronous or not, with
+  ;; session or without.
+  (let* ((org-babel-python-command
+                (or (cdr (assq :python params))
+                    org-babel-python-command))
+               (session-key (org-babel-python-initiate-session
+                                   (cdr (assq :session params))))
+               (graphics-file (and (member "graphics" (assq :result-params params))
+                                               (org-babel-graphical-output-file params)))
+         (result-params (cdr (assq :result-params params)))
+         (result-type (cdr (assq :result-type params)))
+         (results-file (when (eq 'value result-type)
+                         (or graphics-file
+                             (org-babel-temp-file "python-"))))
+               (return-val (when (eq result-type 'value)
+                                   (cdr (assq :return params))))
+               (preamble (cdr (assq :preamble params)))
+         (full-body
+                (concat
+                 (org-babel-expand-body:generic
+                  body params
+                  (org-babel-variable-assignments:python params))
+                 (when return-val
+                   (format "\n%s" return-val))))
+         (post-process
+          (lambda (r)
+            (setq r (string-trim r))
+            (when (string-prefix-p "Traceback (most recent call last):" r)
+              (signal 'user-error (list r)))
+            (when (eq 'value result-type)
+              (setq r (org-babel-eval-read-file results-file)))
+            (org-babel-reassemble-table
+             (org-babel-result-cond result-params
+               r
+               (org-babel-python-table-or-string r))
+             (org-babel-pick-name (cdr (assq :colname-names params))
+                                                    (cdr (assq :colnames params)))
+             (org-babel-pick-name (cdr (assq :rowname-names params))
+                                                    (cdr (assq :rownames params))))))
+         (tmp-src-file (org-babel-temp-file "python-"))
+         (session-body
+          ;; The real code we evaluate in the session.
+          (pcase result-type
+                  (`output
+                   (format (string-join
+                      (list "with open('%s') as f:\n"
+                            "    exec(compile(f.read(), f.name, 'exec'))\n"))
+                                             (org-babel-process-file-name
+                                              tmp-src-file 'noquote)))
+            (`value
+             ;; FIXME: In this case, any output is an error.
+                                           (org-babel-python-format-session-value
+                                tmp-src-file results-file result-params))))
+         comint-buffer
+         finally)
+
+
+    (unless session-key
+      ;; No session. We create a temporary one and use 'finally' to
+      ;; destroy it once we are done.
+      ;;
+      ;; FIXME: This session code should be refactored and moved into
+      ;;        ob-core.
+      (setq session-key (org-babel-python-initiate-session
+                        ;; We can't use a simple `generate-new-buffer'
+                        ;; due to the earmuffs game.
+                         (org-babel-python-without-earmuffs
+                         (format "*ob-python-no-session-%s*" (org-id-uuid)))))
+      (setq finally (lambda ()
+                      (when-let ((s-buf
+                                  (get-buffer (org-babel-python-with-earmuffs session-key))))
+                        ;; We cannot delete it immediately as we are called from it.
+                        (run-with-idle-timer
+                         0.1 nil
+                         (lambda ()
+                           (when (buffer-live-p s-buf)
+                             (let ((kill-buffer-query-functions nil)
+                                   (kill-buffer-hook nil))
+                               (kill-buffer s-buf)))))))))
+
+    (org-elib-async-comint-queue-init-if-needed session-key)
+    (setq comint-buffer
+          (get-buffer (org-babel-python-with-earmuffs session-key)))
+    (with-temp-file tmp-src-file
+      (insert (if (and graphics-file (eq result-type 'output))
+                  (format org-babel-python--output-graphics-wrapper
+                          full-body graphics-file)
+                full-body)))
+
+    (lambda (&rest q)
+      (pcase q
+        (`(:instrs-to-enter)
+         ;; FIXME: This is wrong.
+         "import sys; sys.ps1=''; sys.ps2=''")
+        (`(:instrs-to-exit))
+        (`(:finally) (when finally (funcall finally)))
+        (`(:instr-to-emit-tag ,tag) (format "print ('%s')" tag))
+        (`(:post-process ,r) (when post-process (funcall post-process r)))
+        (`(:send-instrs-to-session ,code)
+         ;; See org-babel-python-send-string
+         (with-current-buffer comint-buffer
+           (let ((python-shell-buffer-name
+                        (org-babel-python-without-earmuffs session-key)))
+                   (python-shell-send-string (concat code "\n")))))
+        (`(:get-code) session-body)
+        (`(:get-comint-buffer) comint-buffer)
+        (_ (error "Unknown query"))))))
+
+
+
+(defun my-org-babel-schedule (lang body params handle-feedback)
+  "Schedule the execution of BODY according to PARAMS.
+This function is called by `org-babel-execute-src-block'.  Return a
+function that waits and returns the result on success, raise on failure."
+  (cl-assert (equal "python" lang))
+  (let ((exec (my-org-babel-python-how-to-execute body params)))
+    (org-elib-async-comint-queue--push exec :handle-feedback handle-feedback)))
+
+
+(defun my-org-babel-execute (lang body params)
+  "Execute Python BODY according to PARAMS.
+This function is called by `org-babel-execute-src-block'."
+  ;; We just start the asynchronous execution, wait for it, and return
+  ;; the result (or raise the exception). No custom code, and,
+  ;; synchronous and asynchronous should just mix nicely together.
+  (cl-assert (equal "python" lang))
+  (funcall (my-org-babel-schedule lang body params nil)))
diff --git a/scratch/bba-ob-core-async/my-async-tests.org b/scratch/bba-ob-core-async/my-async-tests.org
new file mode 100644
index 000000000..263ca77a2
--- /dev/null
+++ b/scratch/bba-ob-core-async/my-async-tests.org
@@ -0,0 +1,484 @@
+#+PROPERTY: HEADER-ARGS+ :eval no-export :exports both
+* Intro
+
+An org document with code blocks to help test the proposed patches.
+
+You need to load:
+   #+begin_src elisp :results silent
+   (load-file "my-async-tests.el")
+   #+end_src
+
+
+
+Emacs and org versions:
+   #+begin_src elisp
+   (mapcar (lambda (sb) (list sb (symbol-value sb)))
+           '(emacs-version org-version))
+   #+end_src
+
+   #+RESULTS:
+   | emacs-version | 30.0.50 |
+   | org-version   | 9.7-pre |
+
+Note that we've disabled eval on export: export doesn't know it needs
+to wait for asynchronous results.
+
+* A simple bash example
+   :PROPERTIES:
+    :header-args:bash: :execute-with my-shell-babel :nasync yes
+   :END:
+
+The package `my-async-tests.el' contains the function
+`my-shell-babel-schedule' to evaluate shell script asynchronously.
+
+The header-args properties above request asynchronous execution for
+bash (:nasync yes), and, tells ob-core to use the prefix
+`my-shell-babel' when looking for functions to evaluate a source
+block. Thus, org will delegate execution to `my-shell-babel-schedule'.
+We don't have `my-shell-babel-execute', so, in this case, :nasync must
+be yes.
+
+A simple execution:
+    #+begin_src bash
+      date
+    #+end_src
+
+    #+RESULTS:
+    : Fri Feb 16 18:11:08 CET 2024
+
+A tricky computation takes some time:
+    #+begin_src bash
+      sleep 5; date
+    #+end_src
+
+    #+RESULTS:
+    : Fri Feb 16 17:58:23 CET 2024
+
+An example of a failure:
+    #+begin_src bash
+      sleepdd 1; false
+    #+end_src
+
+    #+RESULTS:
+
+* Python
+   :PROPERTIES:
+    :header-args:python:  :execute-with my-org-babel :nasync yes
+    :header-args:python+: :session py-async
+   :END:
+
+Used =header-args= properties:
+   - =:execute-with my-org-babel=:  look for functions with the prefix `my-org-babel' to execute
+     blocks (for the asynchronous case use
+     `my-org-babel-schedule', and, for the synchronous case
+     `my-org-babel-execute'). These functions are defined in [[file:my-async-tests.el]].
+
+   - =:nasync yes=: by default, execute asynchronously (use `my-org-babel-schedule').
+
+   - =:session py-async= by default, use a session named "py-async".
+
+** basic examples
+*** async with a session
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+    #+RESULTS:
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708103472.9s | 1708103473.9s | 1.0s |
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+*** async with no session
+   :PROPERTIES:
+    :header-args:python+: :session none
+   :END:
+
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+    #+RESULTS:
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708083470.9s | 1708083471.9s | 1.0s |
+
+Yes, it failed, as expected. "import time" was done in its own
+temporary session.  The old result is preserved; the error is display
+as an overlay. Click on it to get more info about the error.
+
+
+Let's fix it, adding the import line:
+    #+begin_src python
+    import time
+    start = time.time()
+    time.sleep(1)
+    end = time.time()
+    ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708102948.9s | 1708102949.9s | 1.0s |
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+
+*** sync with a session
+   :PROPERTIES:
+    :header-args:python+: :session py-sync-session :nasync no
+   :END:
+
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708102997.5s | 1708102998.5s | 1.0s |
+
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+*** sync with no session
+   :PROPERTIES:
+    :header-args:python+: :session none :nasync no
+   :END:
+
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708083470.9s | 1708083471.9s | 1.0s |
+
+Yes, that fails (no session), displaying the details in a popup. Let's
+fix it:
+    #+begin_src python
+    import time
+    start = time.time()
+    time.sleep(1)
+    end = time.time()
+    ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708103039.0s | 1708103040.0s | 1.0s |
+
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+** worg examples
+
+Let's import matplotlib in our session.
+
+    #+begin_src python
+    import matplotlib
+    import matplotlib.pyplot as plt
+    #+end_src
+
+    #+RESULTS:
+    : None
+
+A figure in a PDF, asynchronous case.
+    #+begin_src python :results file link
+    fig=plt.figure(figsize=(3,2))
+    plt.plot([1,3,2])
+    fig.tight_layout()
+
+    fname = 'myfig-async.pdf'
+    plt.savefig(fname)
+    fname # return this to org-mode
+    #+end_src
+
+    #+RESULTS:
+    [[file:myfig-async.pdf]]
+
+
+A figure in a PDF, synchronous case.
+    #+begin_src python :results file link :nasync no
+    fig=plt.figure(figsize=(3,2))
+    plt.plot([1,3,2])
+    fig.tight_layout()
+
+    fname = 'myfig-sync.pdf'
+    plt.savefig(fname)
+    fname # return this to org-mode
+    #+end_src
+
+    #+RESULTS:
+    [[file:myfig-sync.pdf]]
+
+
+
+A PNG figure, asynchronous case.
+    #+begin_src python :results graphics file output :file boxplot.png
+      fig=plt.figure(figsize=(3,2))
+      plt.plot([1,3,2])
+      fig.tight_layout()
+      fig
+    #+end_src
+
+    #+RESULTS:
+    [[file:boxplot.png]]
+
+Same, but using the =:return= keyword.
+    #+begin_src python :return "plt.gcf()" :results graphics file output :file boxplot.png
+      fig=plt.figure(figsize=(3,2))
+      plt.plot([1,3,2])
+      fig.tight_layout()
+    #+end_src
+
+    #+RESULTS:
+    [[file:boxplot.png]]
+
+Same, asynchronous but without a session this time.
+    #+begin_src python :return "plt.gcf()" :results graphics file output :file boxplot-no-sess-a-y.png :session none
+      import matplotlib
+      import matplotlib.pyplot as plt
+      fig=plt.figure(figsize=(3,2))
+      plt.plot([1,3,2])
+      fig.tight_layout()
+    #+end_src
+
+    #+RESULTS:
+    [[file:boxplot-no-sess-a-y.png]]
+
+
+Lists are table,
+    #+begin_src python
+      [1,2,3]
+    #+end_src
+
+    #+RESULTS:
+    | 1 | 2 | 3 |
+
+unless requested otherwise.
+    #+begin_src python :results verbatim
+      [1,2,3]
+    #+end_src
+
+    #+RESULTS:
+    : [1, 2, 3]
+
+
+Dictionaries are tables too.
+    #+begin_src python :results table
+      {"a": 1, "b": 2}
+    #+end_src
+
+    #+RESULTS:
+    | a | 1 |
+    | b | 2 |
+
+
+Let's try the example with Panda.
+    #+begin_src python :results none
+      import pandas as pd
+      import numpy as np
+    #+end_src
+
+    #+RESULTS:
+    : None
+
+    #+begin_src python :results table
+      pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                   columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+And the synchronous case?
+
+    #+begin_src python :results table  :nasync no
+      pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                   columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+
+
+Without session ?
+
+    #+begin_src python :results table :session none
+    pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                 columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+Right, we need to import the libraries (no session).
+
+    #+begin_src python :results table :session none
+    import pandas as pd
+    import numpy as np
+    pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                 columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+
+** inline examples
+
+    A simple asynchronous inline src_python{3*2} {{{results(=6=)}}}.
+
+    An other one containing a mistake src_python{2/0} {{{results(=6=)}}}
+    (click on the error to see the details).
+
+
+    Some very slow inline asynchronous computations that all run in
+    the same session. You need to execute the 3 of them at once. Here
+    is the first one src_python[:return "\"OK1\""]{import time;
+    time.sleep(5)} {{{results(=OK1=)}}} and a second one
+    src_python[:return "\"OK1 bis\""]{import time; time.sleep(5)}
+    {{{results(=OK1 bis=)}}} and the third one src_python[:return
+    "\"OK2\""]{import time; time.sleep(5)} {{{results(=OK2=)}}}.
+
+    Yes, the previous paragraph is unreadable; it's on purpose, to
+    check that ob-core can figure it out.
+
+    Let's repeat, in a more readable way, and making the last one
+    synchronous.
+
+    Some very slow inline computations that all run in the same
+    session. Here is the first asynchronous one
+          src_python[:return"\"OK1\""]{import time; time.sleep(5)} {{{results(=None=)}}}
+    and a second one, asynchronous too:
+          src_python[:return "\"OK1 bis\""]{import time; time.sleep(5)} {{{results(=OK1 bis=)}}}
+    and finally, a third one, synchronous this one:
+          src_python[:nasync no :return "\"OK2\""]{import time; time.sleep(5)} {{{results(=OK2=)}}}.
+
+    Note that, once the user executes the last synchronous block, the
+    user is blocked until the synchronous execution can start
+    (i.e. all previous asynchronous executions are done) and until
+    it's done.  The display is updated though, to see the asynchronous
+    progress.
-- 
2.43.0

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Matt]

[-- Attachment #1: Type: text/plain, Size: 7215 bytes --]


 ---- On Fri, 16 Feb 2024 18:52:22 +0100  Bruno Barbier 

 > Sorry for the late reply.  Cleaning the code took me longer than expected.

No need to apologize, we're all volunteers here :)

 > Feedbacks, corrections, critiques, etc are most welcome!

Thank you for sharing!

If I understand correctly, there are several independent topics the code addresses:

| topic            | manner addressed                               |
|------------------+------------------------------------------------|
| execution status | using overlays to communicate execution status |
| locating results | using overlays to locate results               |
| blocking         | making all execution asynchronous              |
|------------------+------------------------------------------------|

I suggest these be discussed in separate threads.

 > > The use of the overlay is a really cool idea!
 > >
 > > I hesitate to say that's a good way to convey success or failure.  If a process failed, I want to see the output which tells me why so that I can correct it.  Or, I might actually want the failure output.  Maybe I want to literally demonstrate what a code failure looks like.  Maybe I want to use that output in another block.  For example, shell blocks have multiple output types.  A shell process may return standard output/error or a failure code.  The result of the failure may trigger something else.
 > 
 > I'm not sure I fully understand what you mean. The API just assumes the backend returns the outcome: either success or failure, where failure means "no result" (the previous result, if it exists, is even preserved in the document).  The backend is free to transform a failure into a success to make that result available though.

You can disregard my hesitation on this point.  I had not run your code yet and had misunderstood how it worked.

Since this thread is dedicated to blocking, let me share my thoughts on that subject.

 > To execute Python blocks, using the proposed async API:
 > 
 >    - I've (re)implemented the "asynchronous with session" case (copying/pasting the relevant part from ob-python).
 >    
 >    - The "synchronous case" is just artificially blocking the user until the asynchronous result is known (which looks incredibly tricky to implement if even possible...).
 >    
 >    - The "no session" case is just about creating a new unique session and throwing it away immediately.

This is an interesting idea, feeding all processes through the same mechanism.

Executing a shell block requires starting a [[https://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html][process]].

Processes are synchronous or asynchronous.

Three primitives exist in Emacs for making processes:

1. make-process (asynchronous)
2. call-process (synchronous)
3. call-process-region (synchronous)

There exist several convenience wrappers for these.  AFAIK, everything reduces to these three primitives.  For example, =async-shell-command= runs =call-process= and prevents blocking by appending a "&" to the command which tells the shell to run the command in the background and return control to the terminal.  This background-foreground distinction is called "job control".

Output from a process typically goes to a buffer.  This may be changed and instead handle output with a filter function.  =call-process= has an option to directly send output to a file.

Subprocesses inherent the =default-directory= and the environment from Emacs.  The environment may be changed using =process-environment=.

There are two types of asynchronous connections: "pty" ("pseudoterminal") and "pipe".  The main difference is that "pty" provides a terminal-like connection which allows for things like job control (=C-c=, =C-z=, etc.).

In my previous message, I divided evaluation into 4 types:

  - non-persistent vs. persistent 
  - synchronous vs. asynchronous

I find the approach of feeding everything through, fundamentally, =make-process= interesting because if we make a chart of the 4 types, we see some ambiguities:

|              | non-persistent | persistent   |
|--------------+----------------+--------------|
| synchronous  | call-process   | ???          |
|--------------+----------------+--------------|
| asynchronous | ???            | make-process |
|--------------+----------------+--------------|

To make a non-persistent asynchronous process, the first thing that comes to mind is =async-shell-command=.  However, as the code shows, another option is to use =make-process= and throw away the state (the process buffer).

I'm not sure how we could make a persistent, synchronous process.  Persistence is achieved, currently, by a process buffer.  Is there another way persistence may be achieved?  Of course, this ignores whether a persistent, synchronous process is even desirable.  Given reliable asynchronous execution with persistence, I can't think of reason why someone would prefer a blocking operation.

All that is mainly academic.  The idea I think most interesting is using a single primitive to handle all evaluation.

It idea reminded me of exploration code I wrote a while back which uses =make-process= to run all code blocks asynchronously (attached).

It works as follows.  I defined a new Babel "language" called "blub".   Blub could be a shell, python, ruby, whatever.  I wanted to test that the implementation could work with different interpreters or compilers.  Note that "blub" doesn't have any relationship to Paul Graham's blub; I just needed a name for a generic language that could be swapped out.

Attached are two files, ob-blub.el and ob-blub-test.org.  Download both to the same directory.  Run the first block in ob-blub-test.org.  This imports ob-blub, loads it into Babel, and sets up blub to be whatever =shell-file-name= is (for example, bash).  If you want to try Python or Ruby, comment out the shell configuration, uncomment the Python or Ruby implementations, and evaluate the block again.  Hopefully ob-blub.el is documented sufficiently for you to experiment.

The blub implementation has the same shortcomings, at least for shells, as the current shell implementation.  It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally.  The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings.  If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil).

The challenge I've found with Babel is figuring out how to make the changes.  My current approach is to address bugs and to make changes that move us toward something like the ob-blub implementation.  I wonder if it might help to discuss the core ideas and use a minimal reference implementation that serves as a guide for the actual changes we make.

Curious to hear other people's thoughts!

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

[-- Attachment #2: ob-blub-test.org --]
[-- Type: application/octet-stream, Size: 2532 bytes --]

#+begin_src emacs-lisp :results silent :var HERE=(buffer-file-name)
;; load blub
(add-to-list 'load-path (file-name-directory HERE))
(require 'ob-blub)
(org-babel-do-load-languages 'org-babel-load-languages '((blub . t)))

;; reset uuid on failed block
(setq my-org-babel-comint--async-uuid nil)

;; configure shell
(setq org-babel-blub-interpreter shell-file-name)
(setq org-babel-blub-remove-prompt-command "PROMPT_COMMAND=;PS1=;PS2=;")
(setq org-babel-blub-output-start-delimiter "echo \"start_%s\"")
(setq org-babel-blub-output-end-delimiter "echo \"end_%s\"")
(setq org-babel-blub-interpreter-args '())

;; configure python
;; (setq org-babel-blub-interpreter "python3")
;; (setq org-babel-blub-remove-prompt-command "import sys;sys.ps1='';sys.ps2='';")
;; (setq org-babel-blub-output-start-delimiter "print(\"start_%s\")")
;; (setq org-babel-blub-output-end-delimiter "print(\"end_%s\")")
;; (setq org-babel-blub-interpreter-args '())

;; configure ruby
;; (setq org-babel-blub-interpreter "ruby")  ; for non-sessions
;; (setq org-babel-blub-interpreter "irb")  ; for sessions
;; (setq org-babel-blub-remove-prompt-command nil)
;; (setq org-babel-blub-output-start-delimiter "puts \"start_%s\"")
;; (setq org-babel-blub-output-end-delimiter "puts \"end_%s\"")
;; (setq org-babel-blub-interpreter-args '("--noprompt" "--noreadline" "--nomultiline"))
#+end_src

* Non-persistent
** shell
#+begin_src blub
echo "hello"
sleep 3
echo "world!"
#+end_src

#+RESULTS:
: hello
: world!

#+begin_src blub
==echo "hello"
sleep 3
echo "world!"
#+end_src

** python
#+begin_src blub
import time
print("hello without session")
time.sleep(3)
print("world")
#+end_src

** ruby
#+begin_src blub
puts "cruel world"
sleep(3)
puts "good-bye"
#+end_src

* Persistent
** shell
#+begin_src blub :session *shell-blubber*
echo "hello"
sleep 3
echo "world!"
#+end_src

#+begin_src blub :session *shell-blubber*
==echo "hello"
sleep 3
echo "world!"
#+end_src

** python
#+begin_src blub :session *python-blubber*
import time
print("hello")
time.sleep(5)
print("world")
#+end_src

#+RESULTS:

#+begin_src blub :session *python-blubber*
import time
print("good-bye")
time.sleep(5)
print("cruel world")
#+end_src

** ruby
#+begin_src blub :session *ruby-blubber*
puts "good-bye"
sleep(3)
puts "cruel world"
#+end_src

* Failures
#+begin_src blub :session *bash-blubber*
    ssh localhost "echo foo>foo_file"
    echo "bar" | tee /tmp/bar.txt
#+end_src

#+begin_src blub :session *shell-blubber* :epilogue echo "bye"
ssh $USER@localhost echo "hi"
#+end_src

[-- Attachment #3: ob-blub.el --]
[-- Type: application/octet-stream, Size: 10821 bytes --]

;; -*- lexical-binding: t -*-

;;; ob-blub.el --- org-babel functions for blub evaluation

;; Copyright (C) Matt Trzcinski

;; Author: Matt Trzcinski
;; Keywords: literate programming, reproducible research
;; Homepage: https://orgmode.org
;; Version: 0.01

;;; License:

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation; either version 3, or (at your option)
;; any later version.
;;
;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs; see the file COPYING.  If not, write to the
;; Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
;; Boston, MA 02110-1301, USA.

;;; Commentary:
;;
;; A new language, blub, is defined using the Org Babel API
;; (`org-babel-execute:blub').  "Blub" is a dummy language.  If blub
;; is interpreted, set the interpreter using
;; `org-babel-blub-interpreter'.  Compiled languages are not
;; demonstrated here although the implementation may be extended to
;; support compiled languages.
;;
;; The code is split into three sections:
;;
;;   1. Eval
;;   2. Comint
;;   3. Blub
;;
;; Sections are based on their current implements in Org Babel.  It's
;; not clear if these are the "right" categorizations.  For example,
;; should comint evaluation be in ob-comint or ob-eval?
;;
;; Set the following:
;;
;;   `org-babel-blub-interpreter'
;;   `org-babel-blub-interpreter-args'
;;   `org-babel-blub-remove-prompt-command'
;;   `org-babel-blub-output-start-delimiter'
;;   `org-babel-blub-output-end-delimiter'
;;
;; Sessions run in a dedicated process buffer.  Output is captured
;; from between delimiters.  This is similar to the current Babel
;; implementation.  However, unlike the current Babel implementation,
;; both delimiters in this proof of concept are custom.  The current
;; Babel implementation uses the prompt for the start delimiter which
;; causes problems when it changes.
;;
;; For sessions, only a single process may run at a time.  The UUID is
;; stored in `my-org-babel-comint--async-uuid' and the associated Org
;; buffer is stored in `my-org-babel-comint--async-org-buffer'.  If an
;; error occurs that prevents the end delimiter from printing,
;; manually the clear `my-org-babel-comint--async-uuid' to run blocks.
;;
;;   (setq my-org-babel-comint--async-uuid nil)
;;
;; Results are extracted from standard output using a regexp.
;;
;; Noted problems:
;;   - running sudo

;;; Requirements:
(require 'ob)

\f
;;; Eval:
(defun my-org-babel-eval-async (command body)
  "Start process with COMMAND, send BODY to process, get
results.

Since results execute asynchronously, a UUID is returned.  The
UUID is later replaced when the async process finishes."
  (setq my-org-babel-comint--async-uuid (org-id-uuid))
  (setq my-org-babel-comint--async-org-buffer (current-buffer))
  (let* ((my-process
          (make-process
           :name "my-org-babel-eval-async"
           :buffer "*my-org-babel-eval-async*"
           :command `(,command)
           :connection-type 'pipe
           :sentinel '(lambda (process msg)
                        (cond ((string= msg "finished\n")
                               (my-org-babel-comint-replace-uuid-with-results
                                my-org-babel-comint--async-org-buffer
                                my-org-babel-comint--async-uuid
                                (with-current-buffer (process-buffer process)
                                  (buffer-substring-no-properties (point-min) (point-max))))
                               (setq my-org-babel-comint--async-uuid nil)
                               (let ((kill-buffer-query-functions nil))
                                 (if (kill-buffer "*my-org-babel-eval-async*")
                                     (setq my-org-babel-comint--async-org-buffer nil)))
                               ))))))
    (process-send-string my-process body)
    (process-send-eof my-process))
  my-org-babel-comint--async-uuid)

\f
;;; Comint:
(defvar my-org-babel-comint--async-uuid nil
  "Placeholder for results.")
(defvar my-org-babel-comint--async-org-buffer nil
  "Buffer containing UUID.")

(defun my-org-babel-comint-send-string (session string)
  "Send STRING to comint SESSION."
  (with-current-buffer session
    (goto-char (process-mark (get-buffer-process session)))
    (insert string)
    (comint-send-input)))

(defun my-org-babel-comint-replace-uuid-with-results (buffer uuid results)
  "Replace UUID string in BUFFER with RESULTS string."
  (with-current-buffer buffer
    (save-excursion
      (goto-char (point-min))
      (when (search-forward uuid nil t)
        (org-babel-previous-src-block)
        (let* ((info (org-babel-get-src-block-info))
               (params (nth 2 info))
               (result-params
                (cdr (assq :result-params params))))
          (org-babel-insert-result
           results result-params info))))))

(defun my-org-babel-comint-send-to-session-async (process-buffer &rest body)
  "Send BODY to PROCESS-BUFFER asynchronously."
  (setq my-org-babel-comint--async-uuid (org-id-uuid))
  (setq my-org-babel-comint--async-org-buffer (current-buffer))

  (defun my-org-babel-comint--async-filter (text)
    "Check TEXT for ending delimiter and replace results held by
`my-org-babel-comint--async-uuid' placeholder."
    (cond ((string-match-p (format "end_%s" my-org-babel-comint--async-uuid) text)
           (remove-hook 'comint-output-filter-functions 'my-org-babel-comint--async-filter)

           ;; replace my-org-babel-comint--async-uuid in Org buffer
           (let ((results
                  (with-current-buffer process-buffer ; e.g. "*blubber*"
                    (goto-char (point-min))
                    (re-search-forward
                     ;; Of course, all the problems with regexp happen here.  The goal is getting
                     ;; the text between the delimiters.
                     ;;
                     ;; Some programs (guix shell?) may reset PS1.  So, we can't always match on
                     ;; start_uuid being at the very start of the line.  Match on the one that
                     ;; doesn't have the quote (that is, the result of the echo).
                     (format "[^\"]start_%s[\r\n]*\\(\\(.*[\r\n]+\\)*.*\\)end_%s$"
                             my-org-babel-comint--async-uuid   ; start
                             my-org-babel-comint--async-uuid)  ; end
                     nil nil 1)
                    (let ((match (match-string 1)))
                      (substring-no-properties
                       match)))))

             (my-org-babel-comint-replace-uuid-with-results
              my-org-babel-comint--async-org-buffer
              my-org-babel-comint--async-uuid
              results)

             (setq my-org-babel-comint--async-uuid nil)
             (setq my-org-babel-comint--async-org-buffer nil)))))

  (let* ((proc (get-buffer-process process-buffer)))
    (with-current-buffer process-buffer
      (add-hook 'comint-output-filter-functions 'my-org-babel-comint--async-filter)

      (goto-char (process-mark proc))
      ;; TODO need better abstraction
      (insert (format org-babel-blub-output-start-delimiter my-org-babel-comint--async-uuid))
      (comint-send-input nil t)

      (goto-char (process-mark proc))
      (insert (car body))
      (comint-send-input nil t)

      (goto-char (process-mark proc))
      ;; TODO need better abstraction
      (insert (format org-babel-blub-output-end-delimiter my-org-babel-comint--async-uuid))
      (comint-send-input nil t)))

  my-org-babel-comint--async-uuid)

\f
;;; Blub:
(defvar org-babel-blub-interpreter
  shell-file-name  ; bash/sh
  ;; "irb"         ; ruby
  ;; "python3"     ; python
  "Blub interpreter command or executable.")

(defvar org-babel-blub-interpreter-args
  '()                                                ; bash
  ;; '("--noprompt" "--noreadline" "--nomultiline")  ; ruby
  "Blub interpreter command or executable arguments.")

(defvar org-babel-blub-remove-prompt-command
  "PROMPT_COMMAND=;PS1=;PS2=;"  ; bash
  ;; "import sys;sys.ps1='';sys.ps2='';"  ; python
  "Command(s) to remove interpreter prompt.")

(defvar org-babel-blub-output-start-delimiter
  "echo \"start_%s\""       ; bash
  ;; "puts \"start_%s\""    ; ruby
  ;; "print(\"start_%s\")"  ; python
  "Format expression for writing the start delimiter to standard
output in blub.")

(defvar org-babel-blub-output-end-delimiter
  "echo \"end_%s\""       ; bash
  ;; "puts \"end_%s\""    ; ruby
  ;; "print(\"end_%s\")"  ; python
  "Format expression for writing the end delimiter to standard
output in blub.")

;; this removes the need for `org-babel-comint-buffer-livep'
(defun org-babel-blub-get-session (session)
  "Return SESSION buffer, making process buffer if none exists."
  (cond ((not (get-buffer-process session))
         (apply
          #'make-comint-in-buffer
          `(,session                     ; process name
            ,session                     ; buffer name
            ,org-babel-blub-interpreter  ; program
            nil                          ; start file
            ,@org-babel-blub-interpreter-args))
         (org-babel-comint-wait-for-output session)
         (if org-babel-blub-remove-prompt-command
             (my-org-babel-comint-send-string
              session
              org-babel-blub-remove-prompt-command))

         ;; Needed for Emacs 23 since the marker is initially
         ;; undefined and the filter functions try to use it without
         ;; checking.
         (with-current-buffer session
           (set-marker comint-last-output-start (point)))

         ;; return shell buffer
         (get-buffer session))
        (t session)))

(defun org-babel-execute:blub (body params)
  "Execute BODY of Blub code with org-babel."
  (let* ((session (cdr (assq :session params))))
    (cond ((and session
                (not (string= session "none"))
                (org-babel-blub-get-session session))
           (if (not my-org-babel-comint--async-uuid)
               (my-org-babel-comint-send-to-session-async session body)
             ;; TODO prompt user to just go ahead with it (clobbering
             ;; the "existing" process)
             (error "Block already running.  Call `(setq my-org-babel-comint--async-uuid nil)' to run a new process")))
          (t (my-org-babel-eval-async org-babel-blub-interpreter (org-trim body))))))

(provide 'ob-blub)
;;; ob-blub.el ends here

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Jack Kamm]

Matt <matt@excalamus.com> writes:

> The challenge I've found with Babel is figuring out how to make the changes.  My current approach is to address bugs and to make changes that move us toward something like the ob-blub implementation.  I wonder if it might help to discuss the core ideas and use a minimal reference implementation that serves as a guide for the actual changes we make.
>
> Curious to hear other people's thoughts!

I don't remember the details, but my past self [1] thought it would be
good to find a way to replace `process-file' with `make-process' in
`org-babel--shell-command-on-region' or `org-babel-eval', and it seems
you are thinking along those lines with `my-org-babel-eval-async'. Hope
you're able to make progress on this and get the improvements into
ob-eval.el eventually.

[1] https://list.orgmode.org/871rczg7bi.fsf@gmail.com/#t

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Ihor Radchenko]

Matt <matt@excalamus.com> writes:

> The blub implementation has the same shortcomings, at least for shells, as the current shell implementation.  It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally.  The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings.  If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil).

Doing everything asynchronously is not always desired.
Consider, for example,

#+begin_src bash
echo "Contents" > /tmp/tmpfile
#+end_src bash

#+begin_src bash
cat /tmp/tmpfile # I must run after /tmp/tmpfile is created!
#+end_src

Also, using "print" statements to create output delimiters might
sometimes be tricky.
If I remember correctly Ruby repl is asynchronous and sometimes produces
unexpected artefacts when you send multiple commands in quick
succession. The commands may be executed in different order than you may expect.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1267 bytes --]

On Tue, Feb 20, 2024 at 10:28:06AM +0000, Ihor Radchenko wrote:
> Matt <matt@excalamus.com> writes:
> 
> > The blub implementation has the same shortcomings, at least for shells, as the current shell implementation.  It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally.  The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings.  If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil).
> 
> Doing everything asynchronously is not always desired.
> Consider, for example,
> 
> #+begin_src bash
> echo "Contents" > /tmp/tmpfile
> #+end_src bash
> 
> #+begin_src bash
> cat /tmp/tmpfile # I must run after /tmp/tmpfile is created!
> #+end_src

You "just" [1] need a way of stating dependencies :-)

Cheers

[1] In quotes, because this opens a vast space of interesting
   and strange worlds. The functional folks have their monads,
   the compiler backend builders have their dependency graphs.
   Definitely doable, but treading with care will be helpful
   to not step into a mess :-)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Ihor Radchenko]

<tomas@tuxteam.de> writes:

>> Doing everything asynchronously is not always desired.
>> Consider, for example,
>> 
>> #+begin_src bash
>> echo "Contents" > /tmp/tmpfile
>> #+end_src bash
>> 
>> #+begin_src bash
>> cat /tmp/tmpfile # I must run after /tmp/tmpfile is created!
>> #+end_src
>
> You "just" [1] need a way of stating dependencies :-)
>
> Cheers
>
> [1] In quotes, because this opens a vast space of interesting
>    and strange worlds. The functional folks have their monads,
>    the compiler backend builders have their dependency graphs.
>    Definitely doable, but treading with care will be helpful
>    to not step into a mess :-)

Let's not jump into this rabbit hole yet before we have async code
working for less complicated scenarios.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[tomas]

[-- Attachment #1: Type: text/plain, Size: 536 bytes --]

On Tue, Feb 20, 2024 at 11:00:28AM +0000, Ihor Radchenko wrote:
> <tomas@tuxteam.de> writes:

[...]

> > You "just" [1] need a way of stating dependencies :-)

[...]

> Let's not jump into this rabbit hole yet before we have async code
> working for less complicated scenarios.

:-)

My take is: it's never too early to *think* about it (and to have
a look what others are doing), but it might yet be too early to
design something.

Other than that... rabbit holes is what makes things interesting.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Bruno Barbier]

Hi Matt,

Thanks for your answer.

Matt <matt@excalamus.com> writes:

> ...
> If I understand correctly, there are several independent topics the code addresses:
>
> | topic            | manner addressed                               |
> |------------------+------------------------------------------------|
> | execution status | using overlays to communicate execution status |
> | locating results | using overlays to locate results               |
> | blocking         | making all execution asynchronous              |
> |------------------+------------------------------------------------|
>
> I suggest these be discussed in separate threads.

Actually my patch is only about reporting execution status and
inserting results, in a generic way, in ob-core.  I provided some demo
code on top of them, just to show how it could be used.

Sorry about the confusion.  

> ...
>
> Since this thread is dedicated to blocking, let me share my thoughts on that subject.

I guess I should start a new thread then :)


> ...
>
>
> Executing a shell block requires starting a [[https://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html][process]].
>
> Processes are synchronous or asynchronous.
>
> Three primitives exist in Emacs for making processes:
>
> 1. make-process (asynchronous)
> 2. call-process (synchronous)
> 3. call-process-region (synchronous)
>
> There exist several convenience wrappers for these.  AFAIK, everything reduces to these three primitives.  For example, =async-shell-command= runs =call-process= and prevents blocking by appending a "&" to the command which tells the shell to run the command in the background and return control to the terminal.  This background-foreground distinction is called "job control".
>

> Output from a process typically goes to a buffer.  This may be changed and instead handle output with a filter function.  =call-process= has an option to directly send output to a file.
>

I've reached the same conclusion. As I wanted one simple
implementation, I was thinking about using 'make-process', but, as we
have to deal with the output manually and use a process sentinel, and,
as offering a REPL to the user is nice, my personal pick is to use a
plain shell with comint.

> Subprocesses inherent the =default-directory= and the environment from Emacs.  The environment may be changed using =process-environment=.
>

One wonderful thing about using =default-directory= is that, if done
right, tramp will automatically handle executions on one or more remote
computers.

> There are two types of asynchronous connections: "pty" ("pseudoterminal") and "pipe".  The main difference is that "pty" provides a terminal-like connection which allows for things like job control (=C-c=, =C-z=, etc.).

I'm personally ignoring the difference between pty and pipe: as I
don't use the terminal features, it doesn't really matter to me.


> ...
> I'm not sure how we could make a persistent, synchronous process.  Persistence is achieved, currently, by a process buffer.  Is there another way persistence may be achieved?

If I understand correcly, "persistent" here means to preserve some
state between executions.  They are definitely other way to preserve
and provide a state.

If you focus on processes only, it's probably OK. But, if you use
threads or some other kinds of asynchronicity, that might be too
limiting to require a buffer and a process.


> Of course, this ignores whether a persistent, synchronous process is even desirable.  Given reliable asynchronous execution with persistence, I can't think of reason why someone would prefer a blocking operation.
>

I'm not so sure.  If the success of an execution is really important,
and, intrinsically unreliable (like a network connection), and, it
only takes a few seconds, I'll probably choose synchronous execution,
just to see that everything is OK, before moving to my next task.



> ...
> Attached are two files, ob-blub.el and ob-blub-test.org.  Download both to the same directory.  Run the first block in ob-blub-test.org.  This imports ob-blub, loads it into Babel, and sets up blub to be whatever =shell-file-name= is (for example, bash).  If you want to try Python or Ruby, comment out the shell configuration, uncomment the Python or Ruby implementations, and evaluate the block again.  Hopefully ob-blub.el is documented sufficiently for you to experiment.

Thanks for sharing.  I didn't have time to try it yet.  But it looks
like you could use the 'execute-with' keyword that my patchs provide.
That way, you may redirect evaluation to use your "blub"
implementation, and still use the real language name for your code
blocks (that way, org knows how to fontify them, edit them, etc).  And, you
will not have to manually modify and reevaluate your elisp when
switching languages.

> The blub implementation has the same shortcomings, at least for shells, as the current shell implementation.  It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally.  The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings.  If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil).

I'll try it. Thanks,


> The challenge I've found with Babel is figuring out how to make the changes.  My current approach is to address bugs and to make changes that move us toward something like the ob-blub implementation.  I wonder if it might help to discuss the core ideas and use a minimal reference implementation that serves as a guide for the actual changes we make.
>
> Curious to hear other people's thoughts!

One think that comes to mind is to check that our testsuite covers
everything we might break.

If 'execute-with' gets added to org (from my patchs), that may provide
an easy way to test/compare several execution engines.  I'm thinking
about modifying my patch to provide a 'feedbacks-with' keyword so that
we can also test/compare different ways to report the execution status
and outcome.

Bruno

Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Hi,

Bruno Barbier <brubar.cs@gmail.com> writes:

> Hi Matt,
[...]
>> Since this thread is dedicated to blocking, let me share my thoughts on that subject.
>
> I guess I should start a new thread then :)


I finally got a new version.  I've renamed the proposed feature "pending
contents", that is, some contents that something will update *later*.
With that feature, source code blocks and dynamic blocks can be made
asynchronous … and more!

I've publish the proposed changes as a branch.  You can fecth that
branch there:
┌────
│ repo:   https://framagit.org/brubar/org-mode-mirror
│ branch: bba-pending-contents
└────

The file [my-async-tests.org] describes the proposed changes, its
implementation and contains examples to play with it (shell, python,
ruby, results{append,prepend,silent}, inline blocks, multithreading,
dynamic asynchronous blocks, source codes that depend on other blocks
{post, stdin}, etc.).

I've tried to fully describe the feature in the new section "Pending
contents", in the file `lisp/org-macs.el'.

Testing it in a bare Emacs should now work (I'm using 30.0.50).

*DO NOT TEST* in your production Emacs: this is *alpha* software and it
hooks itself deeply into Emacs (kill hooks), and (only when trying the
multithread examples) uses thread mutexes&locks.

Here is a simple recipe to test the proposed "pending contents" feature:
┌────
│ git clone -b bba-pending-contents https://framagit.org/brubar/org-mode-mirror
│ cd org-mode-mirror
│ make compile
│ cd scratch/bba-pending-contents/
│ emacs -q -L ../../lisp my-async-tests.org
└────


Thanks you all for your previous comments.  I hope I've addressed most
of them.

Comments, critiques, ideas, corrections are most welcome.

Thanks,

Bruno


[my-async-tests.org]
<https://framagit.org/brubar/org-mode-mirror/-/tree/bba-pending-contents/scratch/bba-pending-contents/my-async-tests.org>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

> I've publish the proposed changes as a branch.  You can fecth that
> branch there:
> ┌────
> │ repo:   https://framagit.org/brubar/org-mode-mirror
> │ branch: bba-pending-contents
> └────

Thanks!

> I've tried to fully describe the feature in the new section "Pending
> contents", in the file `lisp/org-macs.el'.

I have one general concern about the implementation.

Overlays are not transferred when a new indirect buffer is created (for
example, by org-capture, or by user). So, it will be (1) impossible to
see pending overlays in indirect buffers; (2) user edits of pending text
from indirect buffer will not be prevented.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Hi Ihor,


Ihor Radchenko <yantar92@posteo.net> writes:

[...]
>> I've tried to fully describe the feature in the new section "Pending
>> contents", in the file `lisp/org-macs.el'.
>
> I have one general concern about the implementation.
>
> Overlays are not transferred when a new indirect buffer is created (for
> example, by org-capture, or by user). So, it will be (1) impossible to
> see pending overlays in indirect buffers; (2) user edits of pending text
> from indirect buffer will not be prevented.

If I understand correctly, you would prefer a solution that relies on
text properties ? (I didn't leak the overlay details in the API, so, it
should not be too hard to switch to text properties, except for possible
conflicts with other existing properties).

What about markers ? Are they OK ?

Thanks,


Bruno

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

>> Overlays are not transferred when a new indirect buffer is created (for
>> example, by org-capture, or by user). So, it will be (1) impossible to
>> see pending overlays in indirect buffers; (2) user edits of pending text
>> from indirect buffer will not be prevented.
>
> If I understand correctly, you would prefer a solution that relies on
> text properties ? (I didn't leak the overlay details in the API, so, it
> should not be too hard to switch to text properties, except for possible
> conflicts with other existing properties).

Yes, I think. Might also try to mirror overlays, but that's problematic
in practice.

However, you cannot use 'before-string/'after-string for overlays.

> What about markers ? Are they OK ?

Markers always point to a specific buffer.
However, for indirect buffers, the text and text properties are shared.
So, editing in one buffer will be automatically reflected in all the clones.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>>> Overlays are not transferred when a new indirect buffer is created (for
>>> example, by org-capture, or by user). So, it will be (1) impossible to
>>> see pending overlays in indirect buffers; (2) user edits of pending text
>>> from indirect buffer will not be prevented.
>>
>> If I understand correctly, you would prefer a solution that relies on
>> text properties ? (I didn't leak the overlay details in the API, so, it
>> should not be too hard to switch to text properties, except for possible
>> conflicts with other existing properties).
>
> Yes, I think. Might also try to mirror overlays, but that's problematic
> in practice.

I now think that overlays are the right way; the /pending content/ is
attached to one buffer: a base or a clone; this is for the user to
decide.

I will manually add text properties, below the overlay, to mark the text
as /pending/, so that pending contents will be visible and read-only in
all other buffers, base or indirect ones.

Cloning buffers is easy to test. I'm not sure which scenario I should
use to test org-capture though.

I'll update my branch with that improvement soon.


Thanks Ihor!

Bruno

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Hi,

Bruno Barbier <brubar.cs@gmail.com> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> Bruno Barbier <brubar.cs@gmail.com> writes:
>>
>>>> Overlays are not transferred when a new indirect buffer is created (for
>>>> example, by org-capture, or by user). So, it will be (1) impossible to
>>>> see pending overlays in indirect buffers; (2) user edits of pending text
>>>> from indirect buffer will not be prevented.
>>>
[...]
> I now think that overlays are the right way; the /pending content/ is
> attached to one buffer: a base or a clone; this is for the user to
> decide.
>
> I will manually add text properties, below the overlay, to mark the text
> as /pending/, so that pending contents will be visible and read-only in
> all other buffers, base or indirect ones.
>
> Cloning buffers is easy to test. I'm not sure which scenario I should
> use to test org-capture though.
>
> I'll update my branch with that improvement soon.
>
>
> Thanks Ihor!
>
> Bruno



Hi,

After some work, some bug fixes and a few segfaults (using indirect
buffers, see bug#69529), I pushed a new version.  The main changes are:

• Handle indirect buffers: the pending content belongs to the buffer
  that started it (using text properties to mirror overlays).
• Use the fringe to indicate success or failure.
• Describe the pending content (past&present) when the user clicks it
  (pending, success or failure, time, duration, log, etc.).
• Improve the logging API and provide examples (may be used to collect
  stderr for example).

As before, the org file describes how to test it, see the file
[scratch/bba-pending-contents/my-async-tests.org] (direct link below).

Comments, critiques, ideas, corrections are most welcome.

Thanks,

Bruno


[scratch/bba-pending-contents/my-async-tests.org]
<https://framagit.org/brubar/org-mode-mirror/-/tree/bba-pending-contents/scratch/bba-pending-contents/my-async-tests.org>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

>> I now think that overlays are the right way; the /pending content/ is
>> attached to one buffer: a base or a clone; this is for the user to
>> decide.
>>
>> I will manually add text properties, below the overlay, to mark the text
>> as /pending/, so that pending contents will be visible and read-only in
>> all other buffers, base or indirect ones.

Thanks!
I have some minor concerns about implementation, but you clearly
demonstrated the things can be working in general.

Let me now zoom out all the way down to org-pending library design.

While reading the library header and `org-pending' docstring (btw, it
should probably be a separate library, not a part of org-macs), I felt
confused about terminology and also had déjà vu's about what org-pending
does.

More or less, org-pending implements Elisp asynchronous process control,
but the processes are associated with region, not the whole buffer.
Hence, I feel that we should adopt terminology similar to the existing
terminology for processes - process filters, process sentinels, sending
signals to processes, etc. And similar API.

Also, I am not sure if I like the idea of exposing raw PINFO alist and
mutating it. In particular, I have doubts about mutating CONTENT.
What we might do instead is implement PINFO as struct with custom
accessors/setters.

WDYT?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

> Thanks!
> I have some minor concerns about implementation, but you clearly
> demonstrated the things can be working in general.

Thanks!

> While reading the library header and `org-pending' docstring (btw, it
> should probably be a separate library, not a part of org-macs),

I was feeling more and more like squatting the wrong file :-)

Would "lisp/org-pending.el" be OK ?  Or do you see a better place/name ?


> I felt confused about terminology and also had déjà vu's about what
> org-pending does.

> More or less, org-pending implements Elisp asynchronous process control,
> but the processes are associated with region, not the whole buffer.
> Hence, I feel that we should adopt terminology similar to the existing
> terminology for processes - process filters, process sentinels, sending
> signals to processes, etc. And similar API.

The current `org-pending' doesn't know what a process is, or a thread.
The job of org-pending is only to let Emacs know that a given region
will be updated at some point later (much like what
`org-edit-src-code' does, when a source block is being edited
manually).

I've re-read the Elisp manual about asynchronous processes.  The only
thing that seemed obvious to me, was to replace "feedbacks-handler"
with "sentinel".  I also tried to simplify things and use REGION
instead of CONTENT where it make sense.  Thanks.

Do you see something else ?



> Also, I am not sure if I like the idea of exposing raw PINFO alist and
> mutating it. In particular, I have doubts about mutating CONTENT.
> What we might do instead is implement PINFO as struct with custom
> accessors/setters.

I went for this simple alist because it's more flexible (easy to add new
bindings) and less code than using a struct.  And Org already uses lists
like this (params, info, etc.).  Mutating CONTENT (or other essential
data) would be a very bad idea indeed; but it's like that for pretty
much everything with elisp.

I'm OK to rewrite it using a cl-defstruct (to be honnest, I just
flipped a coin to decide between a struct or an alist :) ).

I'll push my new version after relocating everything into its own
file. Let me know if lisp/org-pending.el is OK.

Thanks.

Bruno

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

>> While reading the library header and `org-pending' docstring (btw, it
>> should probably be a separate library, not a part of org-macs),
>
> I was feeling more and more like squatting the wrong file :-)
>
> Would "lisp/org-pending.el" be OK ?  Or do you see a better place/name ?

org-pending sounds fine. Maybe org-pending-text to be more explicit.

>> More or less, org-pending implements Elisp asynchronous process control,
>> but the processes are associated with region, not the whole buffer.
>> Hence, I feel that we should adopt terminology similar to the existing
>> terminology for processes - process filters, process sentinels, sending
>> signals to processes, etc. And similar API.
>
> The current `org-pending' doesn't know what a process is, or a thread.
> The job of org-pending is only to let Emacs know that a given region
> will be updated at some point later (much like what
> `org-edit-src-code' does, when a source block is being edited
> manually).

> I've re-read the Elisp manual about asynchronous processes.  The only
> thing that seemed obvious to me, was to replace "feedbacks-handler"
> with "sentinel".  I also tried to simplify things and use REGION
> instead of CONTENT where it make sense.  Thanks.

> Do you see something else ?

Similar to process API, your library provides means to interact with
process associated with the pending text - create them (40.4 Creating an
Asynchronous Process), handle feedback (40.9.2 Process Filter
Functions), handle on-done/fail/etc (40.10 Sentinels: Detecting Process
Status Changes), kill/cancel execution (40.5 Deleting Processes, 40.8
Sending Signals to Processes), insert-log (strerr in Emacs processes)
list pending (40.6 Process Information), org-pending-on-kill-buffer
(40.11 Querying Before Exit), get information (`process-attributes',
40.12 Accessing Other Processes)

I simply went across 40 Processes section of Elisp manual and I see
parallels for pretty much every subsection; but the terminology _and
API_ are different.

>> Also, I am not sure if I like the idea of exposing raw PINFO alist and
>> mutating it. In particular, I have doubts about mutating CONTENT.
>> What we might do instead is implement PINFO as struct with custom
>> accessors/setters.
>
> I went for this simple alist because it's more flexible (easy to add new
> bindings) and less code than using a struct.  And Org already uses lists
> like this (params, info, etc.).  Mutating CONTENT (or other essential
> data) would be a very bad idea indeed; but it's like that for pretty
> much everything with elisp.
>
> I'm OK to rewrite it using a cl-defstruct (to be honnest, I just
> flipped a coin to decide between a struct or an alist :) ).

I think that I need to elaborate.
It is not necessarily a matter of alist vs. struct, but more about the
API. I do not like the idea of mutating the alist entries directly. I'd
rather expose an API to work with "pending" as an opaque object - send
signals to it, get status, etc.

struct is slightly better here because it automatically defines setters
and getters for all the slots without a need to write extra boilerplate
code.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>> Would "lisp/org-pending.el" be OK ?  Or do you see a better place/name ?
>
> org-pending sounds fine. Maybe org-pending-text to be more explicit.

I've picked: "org-pending"; it's shorter and can by used as a prefix.
Thanks.


> Similar to process API, your library provides means to interact with
> process associated with the pending text - create them (40.4 Creating an
> Asynchronous Process), handle feedback (40.9.2 Process Filter
> Functions), handle on-done/fail/etc (40.10 Sentinels: Detecting Process
> Status Changes), kill/cancel execution (40.5 Deleting Processes, 40.8
> Sending Signals to Processes), insert-log (strerr in Emacs processes)
> list pending (40.6 Process Information), org-pending-on-kill-buffer
> (40.11 Querying Before Exit), get information (`process-attributes',
> 40.12 Accessing Other Processes)
>
> I simply went across 40 Processes section of Elisp manual and I see
> parallels for pretty much every subsection; but the terminology _and
> API_ are different.
>

I rewrote the API, rename many things, moved the code around and
sorted everything into heading/subheading sections.  This is hopefully
less confusing and a lot simpler; and the documentation hopefully
explains better how to use it.

The updated section "Commentary", in org-pending, describes the 3 steps
that are needed to mark and use a pending region: a PENREG (I've renamed
"PINFO" to "PENREG", for PENding REGion, more specific).

I tried to stick with the process terminology when relevant (mostly
kill-query and sentinel).

It doesn't really match the process API, but, now, it should be easier to
understand why.


>> I'm OK to rewrite it using a cl-defstruct (to be honnest, I just
>> flipped a coin to decide between a struct or an alist :) ).
>
> I think that I need to elaborate.
> It is not necessarily a matter of alist vs. struct, but more about the
> API. I do not like the idea of mutating the alist entries directly. I'd
> rather expose an API to work with "pending" as an opaque object - send
> signals to it, get status, etc.
>
> struct is slightly better here because it automatically defines setters
> and getters for all the slots without a need to write extra boilerplate
> code.

Done. Thanks!

I added a function `org-pending-user-edit': it allows to edit a region
using org-pending and `string-edit' (see org-pending.el).

I also added an other execution engine (DEMO_ONLY), that allows to
execute any elisp asynchronously, using callbacks (see
`my-use-callbacks-schedule', in my-async-tests.el), and in
my-async-tests.org, near the multithreading examples.

To test, follow the org file [my-async-tests.org] (see direct link below).

WDYT of this version ?

Thanks!

Bruno

[I've forced push the new version, squashing everything into simple
commits, if you need the old version, just ask and I'll push a copy
there.]


[my-async-tests.org]:
     https://framagit.org/brubar/org-mode-mirror/-/blob/bba-pending-contents/scratch/bba-pending-contents/my-async-tests.org?ref_type=heads

[repo & branch]:
     https://framagit.org/brubar/org-mode-mirror/-/tree/bba-pending-contents?ref_type=heads
     bba-pending-contents

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

> I rewrote the API, rename many things, moved the code around and
> sorted everything into heading/subheading sections.  This is hopefully
> less confusing and a lot simpler; and the documentation hopefully
> explains better how to use it.

Thanks! It does look more clear.

> The updated section "Commentary", in org-pending, describes the 3 steps
> that are needed to mark and use a pending region: a PENREG (I've renamed
> "PINFO" to "PENREG", for PENding REGion, more specific).

I feel that org-pending-penreg (org-pending-<pending region>) is
redundant. Maybe better use org-pending-region?

> WDYT of this version ?

Comments on org-pending-pendreg struct:

1. It is not clear why you need a separate ~virtual~ field. When
   ~region~ is nil it already implies that the pending region is
   virtual.

2. ~id~ field is semi-internal and assumed to be a number.
   Maybe we can do something more similar to Emacs process API:
   
   (make-process &rest ARGS)
   ...
   :name NAME -- NAME is name for process.  It is modified if necessary
   to make it unique.

   We can replace id with human-readable name that can also be supplied
   when creating a pending region

3. ~source~ field must match the ~region~ marker buffer. Then, why do we
   need this field at all? May as well just use (marker-buffer (car region))

On the design of ~org-pending~ and ~org-pending-task-connect~:

1. I feel confused about the overall design of the interaction between
   pending region and the associated task.

   Functions like ~org-pending-task-send-update~ imply that pending
   region is rather decoupled from from associated task and the task
   should arrange manually for sending updates to the pending region
   object.

   On the other hand, there is ~task-connection~ that is used to kill
   associated task/process or to "await" for it. This time, pending
   region is strongly coupled with the task, killing it upon deleting
   the pending region.

   I think that we need more (optional) coupling between pending region
   and the associated task. We should be able to get more information
   about the task from pending region side - get logs, current status,
   exit status, etc.

   More specifically, I think that we need (1) allow to pass task as an
   argument for ~org-pending~. (2) In ~org-pending-task-connect~, we
   should allow the task to be a process or timer object. Then, we can
   automatically arrange retrieving process/timer status from the task:
   
   Use process sentinel (maybe, modifying existing via ~add-function~)
   to arrange process status changes to be automatically submitted to
   the pending region;

   Get log updates via process filter

   Kill process via ~kill-process~

   Similar for timers.

   If the argument to ~org-pending-task-connect~ is a lambda, we can use
   the current approach you implemented on the branch.

2. ~org-pending-task-send-update~ name is confusing - it reads as if we
   send an update _to_ the task. Maybe better ~org-pending-region-update~?
   Then, we might even drop ~-sentinel~ field in org-pending-penreg
   object and instead implement that hard-coded ~update~ lambda from
   ~org-pending~ as a part of ~org-pending-region-update~.

3. I feel that different handling of "owner" and indirect buffers is not
   right.

   From the user perspective, it does not matter whether we run an src
   block from one or another indirect buffers - it makes sense to see
   the status in all the indirect org-mode buffers. Maybe we can hook
   into org-mode's fontification and display pending overlays in all the
   indirect buffers.

   Further, it is very confusing that running src block twice from the
   same buffer is not the same as running the same src block from one
   buffer and then from another indirect buffer. The current
   implementation of ~remove-previous-overlays~ makes such distinction
   for reasons I do not understand.

4. I have questions about ~handle-result~ argument in ~org-pending~.

   It is only called on success, but I can easily see that we need to
   handle things specially on failure as well. For example, insert
   stderr or perform other actions like displaying some buffer.

   Or we may even hook some special action to clicking on status
   overlay. For example, clicking on "failure" status overlay may raise
   stderr log.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Ihor Radchenko <yantar92@posteo.net> writes:

Thanks for your review Ihor!

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> I rewrote the API, rename many things, moved the code around and
>> sorted everything into heading/subheading sections.  This is hopefully
>> less confusing and a lot simpler; and the documentation hopefully
>> explains better how to use it.
>
> Thanks! It does look more clear.
>
>> The updated section "Commentary", in org-pending, describes the 3 steps
>> that are needed to mark and use a pending region: a PENREG (I've renamed
>> "PINFO" to "PENREG", for PENding REGion, more specific).
>
> I feel that org-pending-penreg (org-pending-<pending region>) is
> redundant. Maybe better use org-pending-region?

PENREG is the name of the structure; the "org-pending" is the
mandatory library prefix; this mechanically gives the name.  A PENDREG
object is not a "region" in Emacs sense.

Do you see a better name for the structure PENREG, so that it doesn't
sound like a "region" ?


>> WDYT of this version ?
>
> Comments on org-pending-pendreg struct:
>
> 1. It is not clear why you need a separate ~virtual~ field. When
>    ~region~ is nil it already implies that the pending region is
>    virtual.

It's a constant.  Calling a function looks more like we need to
recompute it each time, and, we could even change it.  And
cl-defstruct writes the function for us.

Do you prefer a manually written function ?


> 2. ~id~ field is semi-internal and assumed to be a number.
>    Maybe we can do something more similar to Emacs process API:
>    
>    (make-process &rest ARGS)
>    ...
>    :name NAME -- NAME is name for process.  It is modified if necessary
>    to make it unique.
>
>    We can replace id with human-readable name that can also be supplied
>    when creating a pending region

Good idea. Done.


> 3. ~source~ field must match the ~region~ marker buffer. Then, why do we
>    need this field at all? May as well just use (marker-buffer (car region))

The "source" is the region requesting the update.  The pending region
is the "target" of the update, i.e. the region that will be updated.


For example, in DEMO_ONLY, with org-babel, these 2 regions are never
the same:
   1. the source is the source code block,
   2. the target (pending region) is the result region.

I tried to improve the documentation of `org-pending-penreg' (source &
region).


> On the design of ~org-pending~ and ~org-pending-task-connect~:
>
> 1. I feel confused about the overall design of the interaction between
>    pending region and the associated task.
>
>    Functions like ~org-pending-task-send-update~ imply that pending
>    region is rather decoupled from from associated task and the task
>    should arrange manually for sending updates to the pending region
>    object.

Exactly: the task implementation must use these ~org-pending-task-XXX~
functions to send updates to one (or more) pending region(s).


>    On the other hand, there is ~task-connection~ that is used to kill
>    associated task/process or to "await" for it. This time, pending
>    region is strongly coupled with the task, killing it upon deleting
>    the pending region.


These are optional features; and only ~org-pending~ will know if and
when those might be useful.  That's why the task needs to provide
callbacks here.
     1. cancel: Emacs may, in exceptional cases only,
     send a "cancel" to the task, meaning, "The user destroyed the
     pending region, and thus, Emacs will not use any update for it".

     2. insert-details: If, and only if, the user decides to
     investigate what happened, Emacs will ask the task if it has any
     details to add, that might help the user (like exit-code for an
     OS process, stderr for an OS process or link to a log file, etc.)

     3. get (await): It's an (unofficial) way, (in the degenerate case
     where the task implementation gives up on asynchronicity) to
     block until the outcome is available.  `org-pending' itself
     doesn't use it; DEMO_ONLY uses it with org-babel to define the
     synchronous executions.

>    I think that we need more (optional) coupling between pending region
>    and the associated task. We should be able to get more information
>    about the task from pending region side - get logs, current status,
>    exit status, etc.


>    More specifically, I think that we need (1) allow to pass task as an
>    argument for ~org-pending~.

That's actually what I started with, but, I couldn't make it work.

Breaking it like this is what allowed me to get the most generic and
simplest API that works for anything: threads, callbacks, OS processes,
timers.

If org-pending takes a "task" as an argument, then, we have to define
a universal API for whatever a "task" might be: threads, processes,
callbacks, timers, etc. and any combination of them.

It looks simpler to say that the "task" (whatever "task" means), MAY
call:
    - org-pending-task-send-update (:progress xxx1)
    - org-pending-task-send-update (:progress xxx2)
    - org-pending-task-send-update (:progress xxx3)
      
then, finally MUST either call:

    - org-pending-task-send-update (:success RESULT)
or:
      org-pending-task-send-update (:failure RESULT)


>    (2) In ~org-pending-task-connect~, we
>    should allow the task to be a process or timer object. Then, we can
>    automatically arrange retrieving process/timer status from the task:
>    Use process sentinel (maybe, modifying existing via ~add-function~)
>    to arrange process status changes to be automatically submitted to
>    the pending region;
>
>    Get log updates via process filter
>
>    Kill process via ~kill-process~

It looks to me like a very specific case (one OS process for one
pending region); and I'm not sure how useful it would be in practice.

But this could easily be implemented on top of the existing API.


>    Similar for timers.

Same, it could easily be defined on top of the existing API.


>    If the argument to ~org-pending-task-connect~ is a lambda, we can use
>    the current approach you implemented on the branch.

> 2. ~org-pending-task-send-update~ name is confusing - it reads as if we
>    send an update _to_ the task. Maybe better ~org-pending-region-update~?

Yes ... I wanted a common prefix for the 3 functions that a "task"
implementation is allowed to use:
    - org-pending-task-connect,
    - org-pending-task-send-update,
    - org-pending-task-not-implemented.

It's not confusing if one ignores the common prefix :-)

I've renamed all these functions from "org-pending-task-" to
"org-pending-ti-" where "ti" stands for "task implementation".


>    Then, we might even drop ~-sentinel~ field in org-pending-penreg
>    object and instead implement that hard-coded ~update~ lambda from
>    ~org-pending~ as a part of ~org-pending-region-update~.

That would require to manually capture (dump/load) the context that
the sentinel closure is automatically capturing.

Why would it be better ? Debugging purposes ?

In this case, the current context is (currently) very small, and there
is an obvious place where to dump/load, so, just tell me if you want
me to eliminate that closure.

    
> 3. I feel that different handling of "owner" and indirect buffers is not
>    right.
>    From the user perspective, it does not matter whether we run an src
>    block from one or another indirect buffers - it makes sense to see
>    the status in all the indirect org-mode buffers.

I just tried to followe Emacs: a buffer owns its overlays; a pending
region is (kind of) an overlay.  Thus, a buffer owns its pending
region.


>    Maybe we can hook into org-mode's fontification and display
>    pending overlays in all the indirect buffers.

Well ... "adding overlays in indirect buffers using font-lock" looks
like a very bumpy road to me ... (being very positive, assuming there
is even a road there ... :-) ). As jit-lock is explicitly disabled in
indirect buffers, I'm not even sure what it would technically mean.


>    Further, it is very confusing that running src block twice from the
>    same buffer is not the same as running the same src block from one
>    buffer and then from another indirect buffer. The current
>    implementation of ~remove-previous-overlays~ makes such distinction
>    for reasons I do not understand.

Technically, the outcomes are overlays too; thus, they belong to
one buffer.

If a user created an indirect buffer to focus on some source blocks,
he should expect to manage everything about them from that buffer.
... that looks to me like a plausible explanation that matches the
technical limitations :-)

We might be able to add some other workarounds for indirect buffers,
to provide seamless switches between buffers.  But would that really
be worth it?

In summary, about indirect buffers, I'm not sure it's a good idea to
try to handle them.  I didn't do much with them, but, I'll already was
able to segfault Emacs.  I would prefer to put the no-clone-indirect
property to the org-mode personally :-)

Couldn't we just to forbid "pending regions" in indirect buffers ?
(pending regions don't exist today, so, that doesn't look that bad, at
least for now)



> 4. I have questions about ~handle-result~ argument in ~org-pending~.
> It is only called on success,

Yes. It means Org needs to handle the exact same result as in the
synchronous case, and, it must handle it in exactly the same way.
That's a design choice.

That's probably why it was easy to write the org-babel examples in
DEMO_ONLY; and they already handle most of the standard Org parameters:
async & sync, session & no session, append/prepend/post/stdin/var, etc.



>  but I can easily see that we need to
> handle things specially on failure as well. For example, insert
> stderr or perform other actions like displaying some buffer.  Or we
> may even hook some special action to clicking on status overlay. For
> example, clicking on "failure" status overlay may raise stderr log.

It's already there, no?

If you click on any result (success or failure, inline block or not,
even dynamic blocks), Emacs pops up a buffer with all the details
(source, start, end, duration, stderr, etc.). The function
`org-pending-describe-penreg' defines what is inserted. A given task is
free to insert log, links, widgets, images, diffs, etc. (by providing
the relevant :insert-details method).

These are design choices that are relevant here:
   1. Do not differ from the synchronous case.
   2. Do not delete a valid result until you know that you have
      something better (where a progress or a failure is not "better").
   3. Do not interrupt the user with popups.

If, in the synchronous case, org-babel writes some logs, then the task
must report a success to org-pending so that Org just behaves the
same.  If the task reports a failure, the user keeps the previous
content.

We could modify how ob-core uses org-pending, adding some options if
some users would like errors to be written down using some Org syntax.

I've pushed my update to the public repo (sorry, forced push again due
to some mistakes).


Bruno

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

>> I feel that org-pending-penreg (org-pending-<pending region>) is
>> redundant. Maybe better use org-pending-region?
>
> PENREG is the name of the structure; the "org-pending" is the
> mandatory library prefix; this mechanically gives the name.  A PENDREG
> object is not a "region" in Emacs sense.
>
> Do you see a better name for the structure PENREG, so that it doesn't
> sound like a "region" ?

Library prefix is also a part of the name and delivers useful
information. "org-pending-region" and "region" and not the same names.

We make use of prefix semantic in various places:
- org-export-backend, implying not just "backend", but also "export"
- org-cite-processor, implying not just "processor", but also "cite"
- org-lint-checker - "org-lint" + "checker"
- org-element-deferred - "org-element" + "deferred"

So, there is no need to duplicate information from the prefix - it is an
integral part of the struct name. Doing otherwise would go again the
existing naming in Org code base.

>> 1. It is not clear why you need a separate ~virtual~ field. When
>>    ~region~ is nil it already implies that the pending region is
>>    virtual.
>
> It's a constant.  Calling a function looks more like we need to
> recompute it each time, and, we could even change it.  And
> cl-defstruct writes the function for us.
>
> Do you prefer a manually written function ?

Either a function or a clear indication in the docstring that ~virtual~
and ~region~ are connected and both read-only.

Also, ~virtual~ field is unused. So, maybe we can even drop it
completely. We can always add new fields in future, if a need arises.

>> 3. ~source~ field must match the ~region~ marker buffer. Then, why do we
>>    need this field at all? May as well just use (marker-buffer (car region))
>
> The "source" is the region requesting the update.

The docstring of `org-pending' states that it is a buffer position:

    The SOURCE is the buffer position that requested this pending region.

> ... The pending region
> is the "target" of the update, i.e. the region that will be updated.
>
>
> For example, in DEMO_ONLY, with org-babel, these 2 regions are never
> the same:
>    1. the source is the source code block,
>    2. the target (pending region) is the result region.

I am wondering why source must be a buffer position.
What if we want to mark a region pending for some task not associated
with a source? And why do we need to know the source at all?

>      2. insert-details: If, and only if, the user decides to
>      investigate what happened, Emacs will ask the task if it has any
>      details to add, that might help the user (like exit-code for an
>      OS process, stderr for an OS process or link to a log file, etc.)

I have to say that I am confused about "insert-details" part. Mostly
because it is not per se connected to the associated task. It is rather
an additional handler used to provide debug information about the task
status and outcome.

AFAIU, it is conceptually very similar to HANDLE-RESULT function.

I think that rather than handing HANDLE-RESULT and also TASK-CONTROL, we
may reduce everything to a single "handler" object that will serve as a
way for PENREG to communicate back to Elisp. That way, we do not need to
have a concept of a "task". Instead, it will be a familiar async API
with ability to (1) create (2) send signals to (3) receive signals from
PENREG object.

`org-pending' will be the entry point to create PENREG object.

`org-pending-ti-send-update' (or maybe simply
`org-pending-send-update'?) will be a way to send data to PENREG object.

HANLDER will be another object we may expose via something like
(org-pending-handler (&key on-success-function on-cancel-function on-await on-insert-logs) ...)
Then, PENREG will call appropriate handler function as needed.

>>    If the argument to ~org-pending-task-connect~ is a lambda, we can use
>>    the current approach you implemented on the branch.
>
>> 2. ~org-pending-task-send-update~ name is confusing - it reads as if we
>>    send an update _to_ the task. Maybe better ~org-pending-region-update~?
>
> Yes ... I wanted a common prefix for the 3 functions that a "task"
> implementation is allowed to use:
>     - org-pending-task-connect,
>     - org-pending-task-send-update,
>     - org-pending-task-not-implemented.
>
> It's not confusing if one ignores the common prefix :-)
>
> I've renamed all these functions from "org-pending-task-" to
> "org-pending-ti-" where "ti" stands for "task implementation".

I still feel confused. As stated above, it might be a good idea to get
rid of the concept of "task" completely.

>>    Then, we might even drop ~-sentinel~ field in org-pending-penreg
>>    object and instead implement that hard-coded ~update~ lambda from
>>    ~org-pending~ as a part of ~org-pending-region-update~.
>
> That would require to manually capture (dump/load) the context that
> the sentinel closure is automatically capturing.
>
> Why would it be better ? Debugging purposes ?

Yes. Lexical context is implicit and harder to debug, while storing
necessary data explicitly in the struct slots is more robust as we are
very clear which context is intended to be captured.

>> 3. I feel that different handling of "owner" and indirect buffers is not
>>    right.
>>    From the user perspective, it does not matter whether we run an src
>>    block from one or another indirect buffers - it makes sense to see
>>    the status in all the indirect org-mode buffers.
>
> I just tried to followe Emacs: a buffer owns its overlays; a pending
> region is (kind of) an overlay.  Thus, a buffer owns its pending
> region.

I do not think that it is a good analogy. Not when we also mark the text
read-only in all the indirect buffers as well.

Let me state my idea differently - if some text in buffer is "pending",
it should be visible in all indirect buffers. Otherwise, as a user, I
may be confused why some parts of the buffer are read-only.

>>    Maybe we can hook into org-mode's fontification and display
>>    pending overlays in all the indirect buffers.
>
> Well ... "adding overlays in indirect buffers using font-lock" looks
> like a very bumpy road to me ... (being very positive, assuming there
> is even a road there ... :-) ). As jit-lock is explicitly disabled in
> indirect buffers, I'm not even sure what it would technically mean.

Right. font-lock is linked to text properties, so font-lock in indirect
buffers is finicky (it relies upon text properties to function).

We might try to brew something with `pre-redisplay-functions', but let's
not dive into that rabbit hole for now.

>>    Further, it is very confusing that running src block twice from the
>>    same buffer is not the same as running the same src block from one
>>    buffer and then from another indirect buffer. The current
>>    implementation of ~remove-previous-overlays~ makes such distinction
>>    for reasons I do not understand.
>
> Technically, the outcomes are overlays too; thus, they belong to
> one buffer.
>
> If a user created an indirect buffer to focus on some source blocks,
> he should expect to manage everything about them from that buffer.
> ... that looks to me like a plausible explanation that matches the
> technical limitations :-)

This sounds like trying to fit (by force) expectations to technical
limitations. As a user, I do not really manage everything from the same
buffer and do not expect that Org mode expects me to do so :)

But let's postpone indirect buffer discussion to later and focus on more
high-level design first.

> ... would prefer to put the no-clone-indirect
> property to the org-mode personally :-)

We cannot do it. People use indirect buffers with Org mode extensively.

> Couldn't we just to forbid "pending regions" in indirect buffers ?
> (pending regions don't exist today, so, that doesn't look that bad, at
> least for now)

This might be ok. But we should be prepared that "read-only" on the
pending regions is not going to be reliable - the regions marked pending
can be changed out of sight.

>>  but I can easily see that we need to
>> handle things specially on failure as well. For example, insert
>> stderr or perform other actions like displaying some buffer.  Or we
>> may even hook some special action to clicking on status overlay. For
>> example, clicking on "failure" status overlay may raise stderr log.
>
> It's already there, no?
>
> If you click on any result (success or failure, inline block or not,
> even dynamic blocks), Emacs pops up a buffer with all the details
> (source, start, end, duration, stderr, etc.). The function
> `org-pending-describe-penreg' defines what is inserted. A given task is
> free to insert log, links, widgets, images, diffs, etc. (by providing
> the relevant :insert-details method).

Your thinking also makes sense, if I use a different definition of
"failure" (in the context of PENREG, not in the context of exit code of
the attached process)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Hi Ihor,

Thanks for your review and detailed explanations.

I've pushed an update that should address most of your comments.

Let me answer point by point below.

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>>> I feel that org-pending-penreg (org-pending-<pending region>) is
>>> redundant. Maybe better use org-pending-region?
>>
>> PENREG is the name of the structure; the "org-pending" is the
>> mandatory library prefix; this mechanically gives the name.  A PENDREG
>> object is not a "region" in Emacs sense.
>>
>> Do you see a better name for the structure PENREG, so that it doesn't
>> sound like a "region" ?
>
> Library prefix is also a part of the name and delivers useful
> information. "org-pending-region" and "region" and not the same names.
>
> We make use of prefix semantic in various places:
> - org-export-backend, implying not just "backend", but also "export"
> - org-cite-processor, implying not just "processor", but also "cite"
> - org-lint-checker - "org-lint" + "checker"
> - org-element-deferred - "org-element" + "deferred"
>
> So, there is no need to duplicate information from the prefix - it is an
> integral part of the struct name. Doing otherwise would go again the
> existing naming in Org code base.

Got it. Thanks for the explanation.

I've found a better name: I'm now calling it a "lock".  So I renamed
"PENREG" into "REGLOCK" as "region lock".  The structure is now
`org-pending-reglock'.


>>> 1. It is not clear why you need a separate ~virtual~ field. When
>>>    ~region~ is nil it already implies that the pending region is
>>>    virtual.
>>
[...]
> Also, ~virtual~ field is unused. So, maybe we can even drop it
> completely. We can always add new fields in future, if a need arises.

The region is not allowed to be nil anymore. All "virtual" issues
are solved! :)


>>> 3. ~source~ field must match the ~region~ marker buffer. Then, why do we
>>>    need this field at all? May as well just use (marker-buffer (car region))
>>
>> The "source" is the region requesting the update.
>
> The docstring of `org-pending' states that it is a buffer position:
>
>     The SOURCE is the buffer position that requested this pending region.

Sorry. It was, yes.

>> ... The pending region
>> is the "target" of the update, i.e. the region that will be updated.
>>
>>
>> For example, in DEMO_ONLY, with org-babel, these 2 regions are never
>> the same:
>>    1. the source is the source code block,
>>    2. the target (pending region) is the result region.
>
> I am wondering why source must be a buffer position.
> What if we want to mark a region pending for some task not associated
> with a source? And why do we need to know the source at all?

Good point. It was to allow the user to quickly jump to the source
code block; I removed it from org-pending.



>>      2. insert-details: If, and only if, the user decides to
>>      investigate what happened, Emacs will ask the task if it has any
>>      details to add, that might help the user (like exit-code for an
>>      OS process, stderr for an OS process or link to a log file, etc.)
>
> I have to say that I am confused about "insert-details" part. Mostly
> because it is not per se connected to the associated task. It is rather
> an additional handler used to provide debug information about the task
> status and outcome.
> AFAIU, it is conceptually very similar to HANDLE-RESULT function.

You're right: it was confusing.  It's now like a hook, that
org-pending-describe-reglock will use.  It should now be clearer that
it's for "information purposes" only.


> I think that rather than handing HANDLE-RESULT and also TASK-CONTROL, we
> may reduce everything to a single "handler" object that will serve as a
> way for PENREG to communicate back to Elisp. That way, we do not need to
> have a concept of a "task".

I removed the task-control, and, the concept of "task".  HANDLE-RESULT
is gone from org-pending.  `org-pending' has a new keyword: :on-outcome
that will allow to do anything, both on success and on failure.

> Instead, it will be a familiar async API
> with ability to (1) create (2) send signals to (3) receive signals from
> PENREG object.
> `org-pending' will be the entry point to create PENREG object.
>
> `org-pending-ti-send-update' (or maybe simply
> `org-pending-send-update'?) will be a way to send data to PENREG object.
>

I've renamed org-pending-ti-send-update to org-pending-send-update
(now that the task control is gone, the prefix becomes useless).


> HANLDER will be another object we may expose via something like
> (org-pending-handler (&key on-success-function on-cancel-function on-await on-insert-logs) ...)
> Then, PENREG will call appropriate handler function as needed.

As the task-control is now gone:
  - get/await is gone,
  - cancel is now a hook/function of REGLOCK,
  - insert-details is now a hook/function too of REGLOCK.


>>>    If the argument to ~org-pending-task-connect~ is a lambda, we can use
>>>    the current approach you implemented on the branch.
>>
>>> 2. ~org-pending-task-send-update~ name is confusing - it reads as if we
>>>    send an update _to_ the task. Maybe better ~org-pending-region-update~?
>>
>> Yes ... I wanted a common prefix for the 3 functions that a "task"
>> implementation is allowed to use:
>>     - org-pending-task-connect,
>>     - org-pending-task-send-update,
>>     - org-pending-task-not-implemented.
>>
>> It's not confusing if one ignores the common prefix :-)
>>
>> I've renamed all these functions from "org-pending-task-" to
>> "org-pending-ti-" where "ti" stands for "task implementation".
>
> I still feel confused. As stated above, it might be a good idea to get
> rid of the concept of "task" completely.

Done.


>>>    Then, we might even drop ~-sentinel~ field in org-pending-penreg
>>>    object and instead implement that hard-coded ~update~ lambda from
>>>    ~org-pending~ as a part of ~org-pending-region-update~.
>>
>> That would require to manually capture (dump/load) the context that
>> the sentinel closure is automatically capturing.
>>
>> Why would it be better ? Debugging purposes ?
>
> Yes. Lexical context is implicit and harder to debug, while storing
> necessary data explicitly in the struct slots is more robust as we are
> very clear which context is intended to be captured.

Done.


>>> 3. I feel that different handling of " [[]] owner" and indirect buffers is not
>>>    right.
>>>    From the user perspective, it does not matter whether we run an src
>>>    block from one or another indirect buffers - it makes sense to see
>>>    the status in all the indirect org-mode buffers.
>>
>> I just tried to followe Emacs: a buffer owns its overlays; a pending
>> region is (kind of) an overlay.  Thus, a buffer owns its pending
>> region.
>
> I do not think that it is a good analogy.
> Not when we also mark the text read-only in all the indirect buffers
> as well.
> Let me state my idea differently - if some text in buffer is "pending",
> it should be visible in all indirect buffers. Otherwise, as a user, I
> may be confused why some parts of the buffer are read-only.

You're describing the current implementation ... at least, what I
tried to do :)

With the current implementation, they should be clearly visible in all
buffers, using the secondary-selection face.  If they are not, it's
because Org is explicitly removing some text properties.

Oh, I see ... sorry: don't test indirect buffers with empty results, as
Org is almost always removing custom faces on "#+RESULT:" (IIUC, Org
doesn't comply with font-lock-face).

[...]

> But let's postpone indirect buffer discussion to later and focus on more
> high-level design first.
>
ok. Thanks.


[...]

>>>  but I can easily see that we need to
>>> handle things specially on failure as well. For example, insert
>>> stderr or perform other actions like displaying some buffer.  Or we
>>> may even hook some special action to clicking on status overlay. For
>>> example, clicking on "failure" status overlay may raise stderr log.
>>
>> It's already there, no?
>>
>> If you click on any result (success or failure, inline block or not,
>> even dynamic blocks), Emacs pops up a buffer with all the details
>> (source, start, end, duration, stderr, etc.). The function
>> `org-pending-describe-penreg' defines what is inserted. A given task is
>> free to insert log, links, widgets, images, diffs, etc. (by providing
>> the relevant :insert-details method).
>
> Your thinking also makes sense, if I use a different definition of
> "failure" (in the context of PENREG, not in the context of exit code of
> the attached process)

org-pending now takes an :on-outcome callback: we can now decide what
to do both on success and on failure, and insert things in the org
document in all cases if needed.


Let me know what you think about this new version,

Thanks again for your help!

Bruno

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

> I've pushed an update that should address most of your comments.

Thanks!

> I've found a better name: I'm now calling it a "lock".  So I renamed
> "PENREG" into "REGLOCK" as "region lock".  The structure is now
> `org-pending-reglock'.

I slightly dislike short names and would prefer region-lock, but not a
big deal - it is just my personal style preference.

> I've renamed org-pending-ti-send-update to org-pending-send-update
> (now that the task control is gone, the prefix becomes useless).

I have a further request on interaction with penreg objects.
I feel that it is not ideal that overlays associated with penreg objects
cannot be fully controlled by the callers.

In particular, I'd like to see some way to

1. Create penreg object without locking the region, so that scheduled-at
   time is not set immediately and status overlay is not displayed.
   Then, `org-pending-send-update' could send :schedule signal to
   perform actual lock.

2. Act on the outcome overlays - there is currently no way to remove
   them using penreg object. Maybe :cancel signal? Canceled penreg
   objects can then be garbage-collected from the manager.

Also, the top-level commentary is getting incomplete and out-of-sync at
this point. May you work towards more detailed top-level description of
the library? This will make your ideas more explicit and make life
easier for me during the further review (now, I have to guess what you
meant by some parts of the code).

>> HANLDER will be another object we may expose via something like
>> (org-pending-handler (&key on-success-function on-cancel-function on-await on-insert-logs) ...)
>> Then, PENREG will call appropriate handler function as needed.
>
> As the task-control is now gone:
>   - get/await is gone,
>   - cancel is now a hook/function of REGLOCK,
>   - insert-details is now a hook/function too of REGLOCK.
>> ...
>> Yes. Lexical context is implicit and harder to debug, while storing
>> necessary data explicitly in the struct slots is more robust as we are
>> very clear which context is intended to be captured.
>
> Done.

Is there any reason why you hide the extra information behind :-alist
filed? Why not directly adding extra fields with proper documentation?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Rudolf Adamkovič]

Ihor Radchenko <yantar92@posteo.net> writes:

>> I've found a better name: I'm now calling it a "lock".  So I renamed
>> "PENREG" into "REGLOCK" as "region lock".  The structure is now
>> `org-pending-reglock'.
>
> I slightly dislike short names and would prefer region-lock, but not a
> big deal - it is just my personal style preference.

+1 for the full name.

Searching 'M-x' for 'region' gives 229 results on my Emacs, so there is
a precedent.  In fact, when I first read the name 'reglock', I took
'reg' for *not* a region, but register or registry, precisely because
Emacs consistently spells the word out in full.

Rudy
-- 
"I love deadlines.  I love the whooshing noise they make as they go by."
--- Douglas Adams, The Salmon of Doubt, 2002

Rudolf Adamkovič <rudolf@adamkovic.org> [he/him]
Studenohorská 25, 84103 Bratislava, Slovakia, European Union

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Rudolf Adamkovič <rudolf@adamkovic.org> writes:

>
> +1 for the full name.
>
> Searching 'M-x' for 'region' gives 229 results on my Emacs, so there is
> a precedent.  In fact, when I first read the name 'reglock', I took
> 'reg' for *not* a region, but register or registry, precisely because
> Emacs consistently spells the word out in full.
>

Thanks for your input.

Note that "reglock" is the current technical name, an implementation
detail, and for developer only.  The only current command using that
term (that shows up using M-x) is a debugging internal command
(org-pending--describe-reglock-at-point).

I'm thinking about using just "lock" but I'll keep using "reglock" for
now, as it's easier to search/replace, until the public API is
finalized.

Bruno

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> I've pushed an update that should address most of your comments.
>
> Thanks!
>
>> I've found a better name: I'm now calling it a "lock".  So I renamed
>> "PENREG" into "REGLOCK" as "region lock".  The structure is now
>> `org-pending-reglock'.
>
> I slightly dislike short names and would prefer region-lock, but not a
> big deal - it is just my personal style preference.

It's more about being one word than being short, and being a new and
opaque word.  REGLOCK is a new technical term; it's definition is the
cl-defstruct.  For normal use, this is just a black box to pass
around.

I'll probably switch to using just "lock".



>> I've renamed org-pending-ti-send-update to org-pending-send-update
>> (now that the task control is gone, the prefix becomes useless).
>
> I have a further request on interaction with penreg objects.
> I feel that it is not ideal that overlays associated with penreg objects
> cannot be fully controlled by the callers.

I'm trying to limit the public API surface.  I don't think we should
leak that we are currently using a mix of overlays and text
properties.


> In particular, I'd like to see some way to
>
> 1. Create penreg object without locking the region, so that scheduled-at
>    time is not set immediately and status overlay is not displayed.
>    Then, `org-pending-send-update' could send :schedule signal to
>    perform actual lock.

Using the term "region" was confusing, sorry.  That's why I switched
to region "lock".  I don't think there is a use to create a lock that
doesn't lock.

Also, that might be tricky to implement: `org-pending-send-update' is
called asynchronously, from the user point of view.  Having regions
that suddenly become locked, independently of what the user is
currently doing (if we implement the :schedule message), might be
difficult.

What use do you have in mind ?


> 2. Act on the outcome overlays - there is currently no way to remove
>    them using penreg object.

I've added a funcion `org-pending-delete-outcome-marks' to manually
delete outcome marks that are in a given region.

Else, everything is handled automatically. Once the outcome is known,
the reglock is dead (not live-p).  org-pending may leave outcome marks
about the outcomes (outcome marks are optional).  The outcome marks
automatically disappear if the user remove the section, or, if a new
lock is created for the same region.


>    Maybe :cancel signal? Canceled penreg
>    objects can then be garbage-collected from the manager.

Cancel is handled by sending a failure message (see
 `org-pending-cancel').  It's customizable using the reglock field
 ~org-pending-reglock-user-cancel-function~, which can decide what to
 do (like kill a process) and which can send a better outcome.
 Standard 'cancel' leaves a failure outcome mark.
 
I've added garbage collections of useless reglocks (success or
failure): see `org-pending--mgr-garbage-collect'.



> Also, the top-level commentary is getting incomplete and out-of-sync at
> this point. May you work towards more detailed top-level description of
> the library?

Sorry. I tried hard to keep it in sync with all the modifications.

I found it too much work, and, possibly overwhelming for the reader,
to explain everything in the top-level "Commentary" section.

That's why I deleted everything that wasn't mandatory to understand
the core features.

Everything should be documented as elisp documentation strings,
following the documentation of `org-pending' and
`org-pending-send-update', and, from code comments.


> May you work towards more detailed top-level description of
> the library?
> This will make your ideas more explicit and make life
> easier for me during the further review (now, I have to guess what you
> meant by some parts of the code).

Sorry, and thank you again for your time.

I tried to improve the overall documentation.  I hope it's going to be
easier for you, and others.


>>> HANLDER will be another object we may expose via something like
>>> (org-pending-handler (&key on-success-function on-cancel-function on-await on-insert-logs) ...)
>>> Then, PENREG will call appropriate handler function as needed.
>>
>> As the task-control is now gone:
>>   - get/await is gone,
>>   - cancel is now a hook/function of REGLOCK,
>>   - insert-details is now a hook/function too of REGLOCK.
>>> ...
>>> Yes. Lexical context is implicit and harder to debug, while storing
>>> necessary data explicitly in the struct slots is more robust as we are
>>> very clear which context is intended to be captured.
>>
>> Done.
>
> Is there any reason why you hide the extra information behind :-alist
> filed? Why not directly adding extra fields with proper documentation?

To hide them, indeed :)

The API for 'get-status and 'get-live-p are
`org-pending-reglock-status' and `org-pending-reglock-live-p' (they
are read-only).  The API for the new `useless-p' is
`org-pending-reglock-useless-p' (it's read-only too).

The fields anchor-ovl, region-ovl, on-outcome, set-status and
creation-point are the dump of the closure context, so that
org-pending doesn't rely anymore on a closure to handle updates; I've
rewritten that recently.  Nobody is supposed to use or change those
values, except the update process.

IMHO, dumping those as fields in the lock structure would be more
confusing and fragile than keeping those out of sight.  I could add
comments when they are created/used in the code to help understand how
they are used.

I've added a new macro `org-pending-updating-region' that locks a
region while executing its body.

I've pushed a new version.  The documentation in org-pending should
now be in sync, sorry.

Let me know what you think,
Thanks,

Bruno

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

>> I have a further request on interaction with penreg objects.
>> I feel that it is not ideal that overlays associated with penreg objects
>> cannot be fully controlled by the callers.
>
> I'm trying to limit the public API surface.  I don't think we should
> leak that we are currently using a mix of overlays and text
> properties.

Let me rephrase my concern - I do not like that after reglock is no
longer live (got success/failure signal), there is no way to clean up the
visual hints associated with this particular reglock.

>> In particular, I'd like to see some way to
>>
>> 1. Create penreg object without locking the region, so that scheduled-at
>>    time is not set immediately and status overlay is not displayed.
>>    Then, `org-pending-send-update' could send :schedule signal to
>>    perform actual lock.
>
> Using the term "region" was confusing, sorry.  That's why I switched
> to region "lock".  I don't think there is a use to create a lock that
> doesn't lock.
>
> Also, that might be tricky to implement: `org-pending-send-update' is
> called asynchronously, from the user point of view.  Having regions
> that suddenly become locked, independently of what the user is
> currently doing (if we implement the :schedule message), might be
> difficult.
>
> What use do you have in mind ?

I was mostly confused about linkage between "visual hints" in buffer and
how they are connected with reglock object. You may discard this
comment of mine.

>> 2. Act on the outcome overlays - there is currently no way to remove
>>    them using penreg object.
>
> I've added a funcion `org-pending-delete-outcome-marks' to manually
> delete outcome marks that are in a given region.
>
> Else, everything is handled automatically. Once the outcome is known,
> the reglock is dead (not live-p).  org-pending may leave outcome marks
> about the outcomes (outcome marks are optional).  The outcome marks
> automatically disappear if the user remove the section, or, if a new
> lock is created for the same region.

I do not like this.
I'd like the Elisp program that creates the reglock to be able to
clean up any visual hints associated with it. A function doing it for a
given region cannot do this AFAIU.

>>    Maybe :cancel signal? Canceled penreg
>>    objects can then be garbage-collected from the manager.
>
> Cancel is handled by sending a failure message (see
>  `org-pending-cancel').  It's customizable using the reglock field
>  ~org-pending-reglock-user-cancel-function~, which can decide what to
>  do (like kill a process) and which can send a better outcome.
>  Standard 'cancel' leaves a failure outcome mark.

Note that this function is not documented anywhere other than in reglock
class documentation. In general, I am confused about your overall design
of the user interaction with the locks.

The updated top commentary explains well how Elisp programs can send
data to the locks, but it does not say anything about how Elisp programs
can receive the data.

Also, I'd like to see more information in the top commentary about what
are the "visual hints" displayed to the user and how to configure them.
  
>> Also, the top-level commentary is getting incomplete and out-of-sync at
>> this point. May you work towards more detailed top-level description of
>> the library?
>
> Sorry. I tried hard to keep it in sync with all the modifications.
>
> I found it too much work, and, possibly overwhelming for the reader,
> to explain everything in the top-level "Commentary" section.
>
> That's why I deleted everything that wasn't mandatory to understand
> the core features.
>
> Everything should be documented as elisp documentation strings,
> following the documentation of `org-pending' and
> `org-pending-send-update', and, from code comments.

I agree, but I'd like to see the core concepts explained on top:
1. How to create region lock (done)
2. What the region lock does (prohibit modifications, done)
3. How the lock is presented to the user and how to control the presentation (not done)
4. What can user do with the lock and how it is reflected in Elisp level (not done)
5. What can Elisp do with the lock (done)

> I tried to improve the overall documentation.  I hope it's going to be
> easier for you, and others.

Yes! Thanks!

>> Is there any reason why you hide the extra information behind :-alist
>> filed? Why not directly adding extra fields with proper documentation?
>
> To hide them, indeed :)

> The API for 'get-status and 'get-live-p are
> `org-pending-reglock-status' and `org-pending-reglock-live-p' (they
> are read-only).  The API for the new `useless-p' is
> `org-pending-reglock-useless-p' (it's read-only too).

We usually "hide" fields by declaring them private.
Hiding them from the type docs is not a good idea because it defeats the
purpose of type documentation in general.

> The fields anchor-ovl, region-ovl, on-outcome, set-status and
> creation-point are the dump of the closure context, so that
> org-pending doesn't rely anymore on a closure to handle updates; I've
> rewritten that recently.  Nobody is supposed to use or change those
> values, except the update process.
>
> IMHO, dumping those as fields in the lock structure would be more
> confusing and fragile than keeping those out of sight.  I could add
> comments when they are created/used in the code to help understand how
> they are used.

I disagree. In particular, I dislike the fact that they are not
documented anywhere and one has to read the internals of the code to
understand their purpose.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Hi Ihor,

Thanks for the review.

I've pushed a new version, hoping to decrease the number of dislikes
;-)

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>>> I have a further request on interaction with penreg objects.
>>> I feel that it is not ideal that overlays associated with penreg objects
>>> cannot be fully controlled by the callers.
>>
>> I'm trying to limit the public API surface.  I don't think we should
>> leak that we are currently using a mix of overlays and text
>> properties.
>
> Let me rephrase my concern - I do not like that after reglock is no
> longer live (got success/failure signal), there is no way to clean up the
> visual hints associated with this particular reglock.
[....]

For the org-pending library, "live" means "locked".  Once the outcome
is known, it can't be "live" anymore (it's unlocked); as it's not
reusable, it's "dead".

As the region is not locked anymore, the lock properties/fields can't
be trusted anymore.

But see below about removing the visual outcome hints of a given
reglock.

>>> 2. Act on the outcome overlays - there is currently no way to remove
>>>    them using penreg object.
>>
>> I've added a funcion `org-pending-delete-outcome-marks' to manually
>> delete outcome marks that are in a given region.
>>
>> Else, everything is handled automatically. Once the outcome is known,
>> the reglock is dead (not live-p).  org-pending may leave outcome marks
>> about the outcomes (outcome marks are optional).  The outcome marks
>> automatically disappear if the user remove the section, or, if a new
>> lock is created for the same region.
>
> I do not like this.
> I'd like the Elisp program that creates the reglock to be able to
> clean up any visual hints associated with it. > A function doing it for a
> given region cannot do this AFAIU.

ok. I've added the function `org-pending-reglock-delete-outcome-marks,
that will delete the outcome visual hints for a given reglock, if
there are some.

I updated how the lock is described to the user
(org-pending-describe-reglock): I added a button "Forget" (if the lock
is dead, that removes the outcome marks), and I added a "Cancel"
button if the lock is still live.


>>>    Maybe :cancel signal? Canceled penreg
>>>    objects can then be garbage-collected from the manager.
>>
>> Cancel is handled by sending a failure message (see
>>  `org-pending-cancel').  It's customizable using the reglock field
>>  ~org-pending-reglock-user-cancel-function~, which can decide what to
>>  do (like kill a process) and which can send a better outcome.
>>  Standard 'cancel' leaves a failure outcome mark.
>
> Note that this function is not documented anywhere other than in reglock
> class documentation.

Thanks. I've improved the documentation of `org-pending' to mention
that one may want to customize the following fields of a reglock:
before-kill-function, user-cancel-function and
insert-details-function.  And, also, I added that one can attach
custom properties using the "properties" field.

> In general, I am confused about your overall design
> of the user interaction with the locks.
> The updated top commentary explains well how Elisp programs can send
> data to the locks, but it does not say anything about how Elisp programs
> can receive the data.

An elisp program, that uses org-pending, must update the locks using
`org-pending-send-update'.  That program does not receive any data
from the lock; it may customize Emacs behavior using the reglock
fields mentioned above: before-kill-function, user-cancel-function and
insert-details-function.

Hopefully, it's clearer now with the improved documentation of the
org-pending function.

Just let me know if you still think that the top commentary should
explain this.  Thanks.


> Also, I'd like to see more information in the top commentary about what
> are the "visual hints" displayed to the user and how to configure them.

If you think the current "visual hints" are good enough and could be
shipped as-is, in a first version (indirect buffers, etc.); I could
work on documenting them better.  What kind of configuration are you
thinking about ? just the faces ? or more advanced configurations ?


[...]

>>> Is there any reason why you hide the extra information behind :-alist
>>> filed? Why not directly adding extra fields with proper documentation?
>>
>> To hide them, indeed :)
>
>> The API for 'get-status and 'get-live-p are
>> `org-pending-reglock-status' and `org-pending-reglock-live-p' (they
>> are read-only).  The API for the new `useless-p' is
>> `org-pending-reglock-useless-p' (it's read-only too).
>
> We usually "hide" fields by declaring them private.
> Hiding them from the type docs is not a good idea because it defeats the
> purpose of type documentation in general.
>
>> The fields anchor-ovl, region-ovl, on-outcome, set-status and
>> creation-point are the dump of the closure context, so that
>> org-pending doesn't rely anymore on a closure to handle updates; I've
>> rewritten that recently.  Nobody is supposed to use or change those
>> values, except the update process.
>>
>> IMHO, dumping those as fields in the lock structure would be more
>> confusing and fragile than keeping those out of sight.  I could add
>> comments when they are created/used in the code to help understand how
>> they are used.
>
> I disagree. In particular, I dislike the fact that they are not
> documented anywhere and one has to read the internals of the code to
> understand their purpose.

Done.  I hope the minimal documentation is enough.

Thanks again for your reviews and your comments,


Bruno

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

> Thanks for the review.
>
> I've pushed a new version, hoping to decrease the number of dislikes
> ;-)

Thanks!

>>> Cancel is handled by sending a failure message (see
>>>  `org-pending-cancel').  It's customizable using the reglock field
>>>  ~org-pending-reglock-user-cancel-function~, which can decide what to
>>>  do (like kill a process) and which can send a better outcome.
>>>  Standard 'cancel' leaves a failure outcome mark.
>>
>> Note that this function is not documented anywhere other than in reglock
>> class documentation.
>
> Thanks. I've improved the documentation of `org-pending' to mention
> that one may want to customize the following fields of a reglock:
> before-kill-function, user-cancel-function and
> insert-details-function.  And, also, I added that one can attach
> custom properties using the "properties" field.

Thanks, but I still feel confused.
May you:

1. Explain what does "kill" and "cancel" mean in the context of the
   REGLOCK.

2. Add information about visual hints, "kill", and "cancel" to the
   top-level commentary.

For now, when reading the top commentary:

    ;; This library contains an API to lock a region while it is "being
    ;; updated"; the content of the region is "pending" and cannot be
    ;; modified.  It will be updated, later, when the new content is
    ;; available.

I have an impression that the only side effect of "locking" is that the
region becomes read-only. It is not clear at all that any other
visual indication will be provided.

>> In general, I am confused about your overall design
>> of the user interaction with the locks.
>> The updated top commentary explains well how Elisp programs can send
>> data to the locks, but it does not say anything about how Elisp programs
>> can receive the data.
>
> An elisp program, that uses org-pending, must update the locks using
> `org-pending-send-update'.  That program does not receive any data
> from the lock; it may customize Emacs behavior using the reglock
> fields mentioned above: before-kill-function, user-cancel-function and
> insert-details-function.
>
> Hopefully, it's clearer now with the improved documentation of the
> org-pending function.
>
> Just let me know if you still think that the top commentary should
> explain this.  Thanks.

Yes, I do think that top commentary should explain this.
The very idea that lock may be "canceled" or "killed" is important when
designing Elisp code that uses the org-pending library.

>> Also, I'd like to see more information in the top commentary about what
>> are the "visual hints" displayed to the user and how to configure them.
>
> If you think the current "visual hints" are good enough and could be
> shipped as-is, in a first version (indirect buffers, etc.); I could
> work on documenting them better.  What kind of configuration are you
> thinking about ? just the faces ? or more advanced configurations ?

I am not thinking about details yet.
I just want insert-details-function to be described in the top
commentary. At high level. Something like


* Visual indication of the "pending" regions

While the region is locked, it is visually highlighted in the buffer,
providing information about the lock status as an overlay. The status
can be:

1. :scheduled - the region is "being updated", but the computation has
   not yet started.
2. :progress  - the computation is in progress

After unlocking the region, the visual indication is not necessarily
removed. Instead, the outcome is indicated in an overlay. The outcome
may be:

1. :success - the computation has been successful and the region text
   has been updated
2. :failure - the computation failed

The lock status is updated according to the data submitted to REGLOCK
object via `org-pending-send-update'.

* User interaction with pending regions

For any pending region, users may request detailed description to be
displayed. The description includes the pending region status, creation
time, outcome, duration, results, errors, etc.

Elisp code using this library may also supply additional details about
any given reglock, via `insert-details-function' field of REGLOCK
object. For example, process logs can be displayed this way.

--------

I hope that the above clarifies what I am looking for.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Thanks for the review Ihor!

>> Thanks. I've improved the documentation of `org-pending' to mention
>> that one may want to customize the following fields of a reglock:
>> before-kill-function, user-cancel-function and
>> insert-details-function.  And, also, I added that one can attach
>> custom properties using the "properties" field.
>
> Thanks, but I still feel confused.
> May you:
>
> 1. Explain what does "kill" and "cancel" mean in the context of the
>    REGLOCK.

I tried to document that as part of the function `org-pending. That is:

   |    - Emacs may have to kill your locks; see the field
   |      `before-kill-function' if you wish to do something before your
   |      lock is killed.

In other words, Emacs has to kill the locks, because the user decided
to kill Emacs or some buffers.  The user has no say about this.  This
is when org-pending calls the private function
`org-pending--forced-kill'.
   
   
   |    - The user may ask Emacs to cancel your lock; see the field
   |      `user-cancel-function' to override the default cancel function.

The user is not interested anymore by the lock.  He makes a polite
request to the library that locked the region to unlock it, the sooner
the better.

I added some documentation about them to the top level commentary.
   
> 2. Add information about visual hints, "kill", and "cancel" to the
>    top-level commentary.
>
> For now, when reading the top commentary:
>
>     ;; This library contains an API to lock a region while it is "being
>     ;; updated"; the content of the region is "pending" and cannot be
>     ;; modified.  It will be updated, later, when the new content is
>     ;; available.
>
> I have an impression that the only side effect of "locking" is that the
> region becomes read-only. It is not clear at all that any other
> visual indication will be provided.

I added a description of the user interface at top-level.


[...]
> Yes, I do think that top commentary should explain this.
> The very idea that lock may be "canceled" or "killed" is important when
> designing Elisp code that uses the org-pending library.
>

Both "killed" and "canceled" are now described in the user interface
top-level section.


[...]
>
> I hope that the above clarifies what I am looking for.

I think it did.  And I even hope the improved top-level documentation
shows it :)


I've pushed the modification to my branch.

Thanks again for the review, your time and useful comments,


Bruno

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

> I've pushed the modification to my branch.

Thanks! Let's work further on the top comment.

> ;; To lock a region, you need to do something like this:

I think "something like this" can be just dropped.

> ;;    1. Call the function `org-pending' with the region to lock; use
> ;;       the ON-OUTCOME argument to tell Emacs how to update the
> ;;       region.  Keep the returned REGLOCK (you'll need it to send
> ;;       updates).

It would be nice to provide examples of `org-pending' call right in the
top comment. Ideally, the example should also show how to modify REGLOCK
fields to customize its behaviour.

> ;;    2. Start "something" that computes the new content.  That
> ;;       "something" may be a thread, a timer, a notification, a
> ;;       process, etc.  That "something" must eventually send a
> ;;       :success or :failure message (using
> ;;       `org-pending-send-update'): Emacs will update the pending
> ;;       region (using your ON-OUTCOME) and unlock it; at this point
> ;;       the lock is "dead" (not live-p).

Please also add information about sending updates to the REGLOCK, with
examples. Otherwise, "receiving progress" in the next section is
surprising.

> ... (not live-p)

Please name `org-pending-reglock-live-p' - it is more clear than forcing
the readers search it themselves.

> ;;;; Interface provided to the Emacs user
> ;;
> ;; The library makes locks visible to the user using text properties
> ;; and/or overlays.  It diplays and updates the status while the
> ;; region is locked: the initial status is "scheduled", then, when

It would be nice to name `org-pending-reglock-status' here and use the
actual status values - :scheduled, :pending, :success, :failure.
Ideally, in table/list explaining what happens with buffer text,
overlays, and user interaction, when REGLOCK has each of the listed
status values.

> ;; receiving progress it becomes "pending" (with progress information
> ;; if any).  Emacs allows to diplay a description of the lock.  From
> ;; that description, the user may request to cancel that lock; see the
> ;; field `user-cancel-function' of the REGLOCK object if you need to
> ;; customize what to do on cancel.

It is not very clear how user can interact with "description". Is it a
tooltip? A window? Something else? Please give a bit more details.

Also, when "cancel" is requested, it is a good idea to state that
`user-cancel-function' is called and describe what it does by default.

> ;; When receiving the outcome (success or failure), after unlocking
> ;; the region, the library may leave information about the outcome

"may"?? Or does it always leave the information (by default)?

> ;; (using text properties/overlays).  If that outcome information is
> ;; (still) displayed, Emacs allows to display a description of that
> ;; lock.  From that description, the user may decide to "forget" that
> ;; lock; "forgetting the lock" removes the outcome visual marks, and,

Is "forgetting" customizeable like `user-cancel-function'?

> ;; it allows Emacs to discard any information related to this lock.

What does it mean?

> ;; The description of a lock (live or dead) provides information like
> ;; the schedule time, the duration, the outcome time, the result (in
> ;; case of success), the error (in case of failure), etc.  Customize
> ;; the field `insert-details-function' of REGLOCK object to add your
> ;; own information.

Please show an example how to do it here.

> ;; If the user kills a buffer, or, kills Emacs, some locks may have to
> ;; be killed too be killed too.  The library will ask the user to
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
typo

> ;; confirm if an operation requires to kill some locks.  See the field
> ;; `before-kill-function' of REGLOCK object, if you need to do
> ;; something before a lock is really killed.

Again, an example would be helpful.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> I've pushed the modification to my branch.
>
> Thanks! Let's work further on the top comment.
>
>> ;; To lock a region, you need to do something like this:
>
> I think "something like this" can be just dropped.

Sorry, I'm failing to find a simpler accurate sentence.


>> ;;    1. Call the function `org-pending' with the region to lock; use
>> ;;       the ON-OUTCOME argument to tell Emacs how to update the
>> ;;       region.  Keep the returned REGLOCK (you'll need it to send
>> ;;       updates).
>
> It would be nice to provide examples of `org-pending' call right in the
> top comment. Ideally, the example should also show how to modify REGLOCK
> fields to customize its behaviour.

I added an example below the 2 steps.

>> ;;    2. Start "something" that computes the new content.  That
>> ;;       "something" may be a thread, a timer, a notification, a
>> ;;       process, etc.  That "something" must eventually send a
>> ;;       :success or :failure message (using
>> ;;       `org-pending-send-update'): Emacs will update the pending
>> ;;       region (using your ON-OUTCOME) and unlock it; at this point
>> ;;       the lock is "dead" (not live-p).
>
> Please also add information about sending updates to the REGLOCK, with
> examples. Otherwise, "receiving progress" in the next section is
> surprising.

Done in the example too.

>> ... (not live-p)
>
> Please name `org-pending-reglock-live-p' - it is more clear than forcing
> the readers search it themselves.

Done.

>
>> ;;;; Interface provided to the Emacs user
>> ;;
>> ;; The library makes locks visible to the user using text properties
>> ;; and/or overlays.  It diplays and updates the status while the
>> ;; region is locked: the initial status is "scheduled", then, when
>
> It would be nice to name `org-pending-reglock-status' here and use the
> actual status values - :scheduled, :pending, :success, :failure.
> Ideally, in table/list explaining what happens with buffer text,
> overlays, and user interaction, when REGLOCK has each of the listed
> status values.

I added a table describing the possible status values, the valid
updates, etc in the section above.


>
>> ;; receiving progress it becomes "pending" (with progress information
>> ;; if any).  Emacs allows to diplay a description of the lock.  From
>> ;; that description, the user may request to cancel that lock; see the
>> ;; field `user-cancel-function' of the REGLOCK object if you need to
>> ;; customize what to do on cancel.
>
> It is not very clear how user can interact with "description".
> Is it a  tooltip? A window? Something else? Please give a bit more details.

I added that it's like the function `describe-package'.

> Also, when "cancel" is requested, it is a good idea to state that
> `user-cancel-function' is called and describe what it does by default.

The function `user-cancel-function' is not always called; Emacs will
only call it if the lock is still live (see `org-pending-cancel').

I added a short sentence describing what is done by default (the
documentation of the field `user-cancel-function' of the reglock
object already fully document `user-cancel-function').


>> ;; When receiving the outcome (success or failure), after unlocking
>> ;; the region, the library may leave information about the outcome
>
> "may"?? Or does it always leave the information (by default)?

To mark the outcome, org-pending needs to know where it is.  If the
ON-OUTCOME function (see `org-pending` documentation) returns the
outcome region, org-pending will add a mark, else not.  By default,
ON-OUTCOME is nil, meaning no outcome region and no outcome marks.  I
added a sentence about that.


>> ;; (using text properties/overlays).  If that outcome information is
>> ;; (still) displayed, Emacs allows to display a description of that
>> ;; lock.  From that description, the user may decide to "forget" that
>> ;; lock; "forgetting the lock" removes the outcome visual marks, and,
>
> Is "forgetting" customizeable like `user-cancel-function'?

No.

>> ;; it allows Emacs to discard any information related to this lock.
>
> What does it mean?

Emacs is free to delete any data related to this lock.  This lock will
not be available anywhere anymore for the user. Concretely, this lock
is remove from the list of known locks, and, eventually, Emacs should
hopefully garbage collect the related data.

>
>> ;; The description of a lock (live or dead) provides information like
>> ;; the schedule time, the duration, the outcome time, the result (in
>> ;; case of success), the error (in case of failure), etc.  Customize
>> ;; the field `insert-details-function' of REGLOCK object to add your
>> ;; own information.
>
> Please show an example how to do it here.

Done.

>
>> ;; If the user kills a buffer, or, kills Emacs, some locks may have to
>> ;; be killed too be killed too.  The library will ask the user to
>      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
> typo

Thanks.

>> ;; confirm if an operation requires to kill some locks.  See the field
>> ;; `before-kill-function' of REGLOCK object, if you need to do
>> ;; something before a lock is really killed.
>
> Again, an example would be helpful.


Done.

I've pushed the update to my public branch.

Thanks,
Bruno

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 3053 bytes --]

Bruno Barbier <brubar.cs@gmail.com> writes:

> I've pushed the update to my public branch.

Thanks!

I am attaching some minor edits I'd like to propose on top of your
latest branch.

Also, some more questions.

> ;;        (setq my-rlock
> ;;              (org-pending (cons (point) (mark))
> ;;                           (lambda (outcome)
> ;;                             (pcase outcome
> ;;                               (`(:success ,result) (goto-char END) (insert result))
> ;;                               (`(:failure ,err) (message "Failed: %s" err))))))

1. It is more natural in Elisp to pass regions as two parameters: BEG
   and END, not a cons cell.

2. Note that (point) may be _after_ (mark).  AFAIU, you code assumes
   that point is always before the mark.  You may want to address this.

3. ON-OUTCOME is optional. What happens if none is provided?

4. In the `org-pending' docstring you say "ON-OUTCOME is non-nil, call
   it with the reglock and the outcome", but the example shows a lambda
   accepting a single argument. Something is off. I'm afraid that this
   example will not work if copy-pasted.

> ;;        (org-pending-send-update my-rlock  (list :progress "Not ready yet."))
> ;;        (org-pending-send-update my-rlock  (list :progress "Coming soon."))

Should the progress message always be a string?

> ;;        (org-pending-send-update my-rlock (list :success 1))

What will org-pending do with :success 1? Will it replace region with
"1" or will it do something else?

> ;;        (org-pending-send-update my-rlock (list :failure "Some error!"))

I am slightly confused by this calling convention. Why not simply

(org-pending-send-update my-rlock :failure "Some error!")

> ;;    (setf (org-pending-reglock-insert-details-function my-reglock)
> ;;          (lambda (rl _start _end)
> ;;            (insert (format "%s" (org-pending-reglock-property rl :my-prop)))))

Are there any standard properties? It would be nice to list them in a
table as well.

Also, you can show an example of _setting_ :my-prop property.

> ;; If the user kills a buffer, or, kills Emacs, some locks may have to
> ;; be killed too.  The library will ask the user to confirm if an
> ;; operation requires to kill some locks.  See the field
> ;; `before-kill-function' of REGLOCK object, if you need to do
> ;; something before a lock is really killed.  For example, if you like
> ;; to kill a MY-BUFFER before MY-LOCK is killed, you can do:
> ;;
> ;;    (setf (org-pending-reglock-before-kill-function my-reglock)
> ;;          (lambda (_rl) (kill-buffer my-buffer)))

It would be nice to have an example that will also send a signal to
process, as it is probably the most commonly used way to utilize
org-pending.


From `org-pending' docstring:

> If ON-OUTCOME returns
> a region (a pair (start position . end position)), use it to report the
> success/failure using visual hints on that region.  If ON-OUTCOME
> returns nothing, don't display outcome marks.

What if ON-OUTCOME returns something that is not a cons cell and not nil?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: org-pending.diff --]
[-- Type: text/x-patch, Size: 5019 bytes --]

diff --git a/lisp/org-pending.el b/lisp/org-pending.el
index 2530a95b3..973b5e17b 100644
--- a/lisp/org-pending.el
+++ b/lisp/org-pending.el
@@ -26,11 +26,18 @@ ;;; Commentary:
 ;;;; Overview
 ;;
 ;;
-;; This library contains an API to lock a region while it is "being
+;; This library provides an API to lock a region while it is "being
 ;; updated"; the content of the region is "pending" and cannot be
 ;; modified.  It will be updated, later, when the new content is
 ;; available.
 ;;
+;; While region is "pending", the library will mark it for the user,
+;; displaying the current update progress.
+;;
+;; The update may yield success or failure.  On success, the region
+;; content will be updated, and the update summary will be indicated.
+;; On failure, the error log will be displayed.
+;;
 ;; Locking regions is useful when the update is computed
 ;; asynchronously and/or depends on external events.
 ;;
@@ -38,7 +45,7 @@ ;;;; Overview
 ;;;; How to use locks in your library
 ;;
 
-;; To lock a region, you need to do something like this:
+;; To lock a region, you need to:
 ;;
 ;;    1. Call the function `org-pending' with the region to lock; use
 ;;       the ON-OUTCOME argument to tell Emacs how to update the
@@ -47,18 +54,19 @@ ;;;; How to use locks in your library
 ;;
 ;;    2. Start "something" that computes the new content.  That
 ;;       "something" may be a thread, a timer, a notification, a
-;;       process, etc.  That "something" must eventually send a
-;;       :success or :failure message (using
-;;       `org-pending-send-update'): Emacs will update the pending
-;;       region (using your ON-OUTCOME) and unlock it; at this point
-;;       the lock is "dead" (see `org-pending-reglock-live-p').
+;;       process, etc.  That "something" might optionally report
+;;       :progress, and must eventually send a :success or :failure
+;;       message (using `org-pending-send-update'): org-pending will
+;;       update the pending region (using your ON-OUTCOME) and unlock
+;;       it; at this point the lock is "dead" (see
+;;       `org-pending-reglock-live-p').
 ;;
 ;; A lock is "live" (blocking its region) from when it's created until
-;; it receives its outcome (success or failure).  Once the lock
+;; it receives its outcome (:success or :failure).  Once the lock
 ;; receives its outcome, it's dead.
 ;;
 ;; You may read the current status using `org-pending-reglock-status'.
-;; The status is automatically updated when you send updates using
+;; The status is updated when you send updates using
 ;; `org-pending-send-update'.
 ;;
 ;; | Status    | Type     | Region   | Live ? | Possible updates              | Outcome available | Outcome marks                  |
@@ -99,23 +107,23 @@ ;;;; Interface provided to the Emacs user
 ;; and/or overlays.  It diplays and updates the status while the
 ;; region is locked: the initial status is :scheduled, then, when
 ;; receiving progress it becomes :pending (with progress information
-;; if any).  Emacs allows to diplay a description of the lock in a new
-;; buffer, like, for example, `describe-package'.  From that
+;; if any).  org-pending allows to diplay a description of the lock in
+;; a new buffer, like, for example, `describe-package'.  From that
 ;; description buffer, the user may request to cancel that lock; see
 ;; the field `user-cancel-function' of the REGLOCK object if you need
-;; to customize what to do on cancel.  By default, Emacs will just
-;; send the update (list :failure 'org-pending-user-cancel) so that
-;; the region is unlocked.
+;; to customize what to do on cancel.  By default, org-pending will
+;; just send the update (list :failure 'org-pending-user-cancel) so
+;; that the region is unlocked.
 ;;
-;; When receiving the outcome (success or failure), after unlocking
+;; When receiving the outcome (:success or :failure), after unlocking
 ;; the region, the library may leave information about the outcome
 ;; (using text properties/overlays); it will leave an outcome mark
 ;; only if the ON-OUTCOME function returns the outcome region (see
-;; `org-pending`). If that outcome information is (still) displayed,
+;; `org-pending').  If that outcome information is (still) displayed,
 ;; Emacs allows to display a description of that lock.  From that
 ;; description, the user may decide to "forget" that lock; "forgetting
-;; the lock" removes the outcome visual marks, and, it allows Emacs to
-;; discard any information related to this lock.
+;; the lock" removes the outcome visual marks, and, it allows
+;; org-pending to discard any information related to this lock.
 
 ;; Note that the visual marks of an outcome are silently removed if
 ;; the library needs to (like when creating a new lock, or when
@@ -172,7 +180,7 @@ ;;;; Content of this file
 ;; and block the user.  The section "Dev & debug" contains tools that
 ;; are useful only for development and debugging.
 ;;
-;; This file does *NOT* depend on Org.
+;; This file does *NOT* depend on Org mode.
 
 ;;; Code:
 

[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
> I am attaching some minor edits I'd like to propose on top of your
> latest branch.

I've applied your patch. Thanks.

> Also, some more questions.
>
>> ;;        (setq my-rlock
>> ;;              (org-pending (cons (point) (mark))
>> ;;                           (lambda (outcome)
>> ;;                             (pcase outcome
>> ;;                               (`(:success ,result) (goto-char END) (insert result))
>> ;;                               (`(:failure ,err) (message "Failed: %s" err))))))
>
> 1. It is more natural in Elisp to pass regions as two parameters: BEG
>    and END, not a cons cell.

I rewrote the example so that it's complete and can be uncommented and
evaluated.

With the new example, it's less obvious which is more natural.



> 2. Note that (point) may be _after_ (mark).  AFAIU, you code assumes
>    that point is always before the mark.  You may want to address this.

Right. I'm not using those in the new example: problem solved ;)



> 3. ON-OUTCOME is optional. What happens if none is provided?

No function is called. And thus, no value is returned, so no mark.
It's like calling a function that does nothing and return nil.



> 4. In the `org-pending' docstring you say "ON-OUTCOME is non-nil, call
>    it with the reglock and the outcome", but the example shows a lambda
>    accepting a single argument. Something is off. I'm afraid that this
>    example will not work if copy-pasted.
>

You're right. Sorry. I'm now using using a fully working example.



>> ;;        (org-pending-send-update my-rlock  (list :progress "Not ready yet."))
>> ;;        (org-pending-send-update my-rlock  (list :progress "Coming soon."))
>
> Should the progress message always be a string?

No.  It may currently be any data.  org-pending will format it as a
string that fits on one line.


>> ;;        (org-pending-send-update my-rlock (list :success 1))
>
> What will org-pending do with :success 1? Will it replace region with
> "1" or will it do something else?

That's the job on ON-OUTCOME to convert/format/append/prepend/replace
some content if needed, on :success and/or on :failure.


>> ;;        (org-pending-send-update my-rlock (list :failure "Some error!"))
>
> I am slightly confused by this calling convention. Why not simply
>
> (org-pending-send-update my-rlock :failure "Some error!")

Because one message update is one value, either:
     (list :success value)
or
     (list :failure err)
or
     (list :progress data)

On :success/:failure, this message is also a valid outcome, and it's
the same value that ON-OUTCOME will receive, as one single entity.

When listing artificial calls like this, it does look suboptimal; but,
in general, it's simpler to have a real value we can store, pass
around and return.

>
>> ;;    (setf (org-pending-reglock-insert-details-function my-reglock)
>> ;;          (lambda (rl _start _end)
>> ;;            (insert (format "%s" (org-pending-reglock-property rl :my-prop)))))
>
> Are there any standard properties? It would be nice to list them in a
> table as well.

There are no standard properties.  All required properties have been
hard-coded in the cl-defstruct.

> Also, you can show an example of _setting_ :my-prop property.

I removed the use of `org-pending-reglock-property' from the example; I
think it was a mistake to introduce that advanced feature here.  The
documentation of the function `org-pending-reglock-property' explains
how to set a property.


>> ;; If the user kills a buffer, or, kills Emacs, some locks may have to
>> ;; be killed too.  The library will ask the user to confirm if an
>> ;; operation requires to kill some locks.  See the field
>> ;; `before-kill-function' of REGLOCK object, if you need to do
>> ;; something before a lock is really killed.  For example, if you like
>> ;; to kill a MY-BUFFER before MY-LOCK is killed, you can do:
>> ;;
>> ;;    (setf (org-pending-reglock-before-kill-function my-reglock)
>> ;;          (lambda (_rl) (kill-buffer my-buffer)))
>
> It would be nice to have an example that will also send a signal to
> process, as it is probably the most commonly used way to utilize
> org-pending.

For my many use cases, that would always be a mistake to kill the
process: an OS process is always in charge of many locks.

More importantly, to find a self-contained working readable example
might be a challenge.

We could add a function 'org-pending-shell-command-on-region' in
org-pending, that could be used as an implementation example, like
`org-pending-user-edit', `org-babel-execute-src-block', etc.


> From `org-pending' docstring:
>
>> If ON-OUTCOME returns
>> a region (a pair (start position . end position)), use it to report the
>> success/failure using visual hints on that region.  If ON-OUTCOME
>> returns nothing, don't display outcome marks.
>
> What if ON-OUTCOME returns something that is not a cons cell and not nil?
>

Then org-pending would have raised some error at some point, as it's not
a valid value; the doc says that ON-OUTCOME may return a region or
nothing.

I added a test so that org-pending will now scream immediately if
ON-OUTCOME returns an invalid value.

I've updated my public branch,

Let me know what you think,

Thanks,

Bruno

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

> ... I'm now using using a fully working example.

Thanks!
I will first reply to you email inline and then common further on your
changes from the branch.

>>> ;;        (org-pending-send-update my-rlock  (list :progress "Not ready yet."))
>>> ;;        (org-pending-send-update my-rlock  (list :progress "Coming soon."))
>>
>> Should the progress message always be a string?
>
> No.  It may currently be any data.  org-pending will format it as a
> string that fits on one line.

Please say this in the docstring of `org-pending-send-update'.

>>> ;;        (org-pending-send-update my-rlock (list :success 1))
>>
>> What will org-pending do with :success 1? Will it replace region with
>> "1" or will it do something else?
>
> That's the job on ON-OUTCOME to convert/format/append/prepend/replace
> some content if needed, on :success and/or on :failure.

Fair. Although, it feels like a common use case to replace the region
with :success value. Maybe the library should provide some ready-to-use
functions that can be used as the value of :on-outcome.

>> It would be nice to have an example that will also send a signal to
>> process, as it is probably the most commonly used way to utilize
>> org-pending.
>
> For my many use cases, that would always be a mistake to kill the
> process: an OS process is always in charge of many locks.
>
> More importantly, to find a self-contained working readable example
> might be a challenge.
>
> We could add a function 'org-pending-shell-command-on-region' in
> org-pending, that could be used as an implementation example, like
> `org-pending-user-edit', `org-babel-execute-src-block', etc.

Yes, having pre-cooked wrappers for `org-pending' or pre-defined values
for :on-outcome/:befire-kill-function/:user-cancel-function/etc would be useful.

;;     ;; We lock the 'region', defining how to update it when the
;;     ;; outcome is available.
;;     (setq my-lock (org-pending
;;                    region
;;                    :on-outcome
;;                    (lambda (_rl outcome)
;;                      (pcase outcome
;;                        (`(:success ,value)
;;                         ;; On :success, we replace the region with the
;;                         ;; value.
;;                         (let ((tmp-end (cdr region)))
;;                           (goto-char tmp-end)
;;                           (insert (format "%s\n" value))
;;                           (setcdr region (point))
;;                           (delete-region (car region) tmp-end)))
;; ...

This example has a major problem if user edits the buffer text _before_
locked region before the outcome is available.
(car region) and (cdr region) will no longer be accurate, and your code
will replace text in places you do not expect.

I believe that it will be better to query region-lock object about the
region location where we need to replace text:

(setq region (org-pending-reglock-region rl))

Same for reglock buffer in other examples.

Then, we will keep the possibility open for org-pending to handle cases
like killing/yanking text containing reglocks (org-refile) - org-pending
may in future keep track of them.

;;     (setf (org-pending-reglock-user-cancel-function my-lock)
;;           (let ((this-timer my-timer))
;;             (lambda (_rl)
;;               (cancel-timer this-timer)
;; ...

;;     (setf (org-pending-reglock-before-kill-function my-lock)
;;           (let ((this-timer my-timer))
;;             (lambda (_rl)
;;               (message "Killing %s" this-timer)
;;               (cancel-timer this-timer))))

What is the difference between "canceling" and "killing" the reglock?
Do they need to be separate?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents

[Bruno Barbier]

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>>>> ;;        (org-pending-send-update my-rlock  (list :progress "Not ready yet."))
>>>> ;;        (org-pending-send-update my-rlock  (list :progress "Coming soon."))
>>>
>>> Should the progress message always be a string?
>>
>> No.  It may currently be any data.  org-pending will format it as a
>> string that fits on one line.
>
> Please say this in the docstring of `org-pending-send-update'.

Done.



>>>> ;;        (org-pending-send-update my-rlock (list :success 1))
>>>
>>> What will org-pending do with :success 1? Will it replace region with
>>> "1" or will it do something else?
>>
>> That's the job on ON-OUTCOME to convert/format/append/prepend/replace
>> some content if needed, on :success and/or on :failure.
>
> Fair. Although, it feels like a common use case to replace the region
> with :success value. Maybe the library should provide some ready-to-use
> functions that can be used as the value of :on-outcome.

I've recycled the old function used by `org-pending-user-edit',
improved it and made it the default :on-outcome handler: see
`org-pending-on-outcome-replace'.  I've simplified the example
accordingly, removing the custom :on-outcome.

I don't know any safe way to replace some text, but, I hope that will
be a good enough default.



>>> It would be nice to have an example that will also send a signal to
>>> process, as it is probably the most commonly used way to utilize
>>> org-pending.
>>
>> For my many use cases, that would always be a mistake to kill the
>> process: an OS process is always in charge of many locks.
>>
>> More importantly, to find a self-contained working readable example
>> might be a challenge.
>>
>> We could add a function 'org-pending-shell-command-on-region' in
>> org-pending, that could be used as an implementation example, like
>> `org-pending-user-edit', `org-babel-execute-src-block', etc.
>
> Yes, having pre-cooked wrappers for `org-pending' or pre-defined values
> for :on-outcome/:befire-kill-function/:user-cancel-function/etc would be useful.

:on-outcome now has a better default: `org-pending-on-outcome-replace'
(see above).

The predefined values for :before-kill-function and
:user-cancel-function seem OK to me.  We will see, when using
org-pending, if some patterns need to be included in org-pending.

From the many examples provided in the branch, do you see any that
should be included in the library as an other precooked-wrapper, that
should be included in the section "Basic use of locks" ?

I've added 'org-pending-shell-command-on-region' to my todo list.


>
> ;;     ;; We lock the 'region', defining how to update it when the
> ;;     ;; outcome is available.
> ;;     (setq my-lock (org-pending
> ;;                    region
> ;;                    :on-outcome
> ;;                    (lambda (_rl outcome)
> ;;                      (pcase outcome
> ;;                        (`(:success ,value)
> ;;                         ;; On :success, we replace the region with the
> ;;                         ;; value.
> ;;                         (let ((tmp-end (cdr region)))
> ;;                           (goto-char tmp-end)
> ;;                           (insert (format "%s\n" value))
> ;;                           (setcdr region (point))
> ;;                           (delete-region (car region) tmp-end)))
> ;; ...
>
> This example has a major problem if user edits the buffer text _before_
> locked region before the outcome is available.
> (car region) and (cdr region) will no longer be accurate, and your code
> will replace text in places you do not expect.
> I believe that it will be better to query region-lock object about the
> region location where we need to replace text:
>
> (setq region (org-pending-reglock-region rl))
>
> Same for reglock buffer in other examples.
>
> Then, we will keep the possibility open for org-pending to handle cases
> like killing/yanking text containing reglocks (org-refile) - org-pending
> may in future keep track of them.

I see. Good point!

But note that the region is a "read-only constant" field;
archiving/refiling live locks is forbidden.

I modified the example to rely on the reglock when possible (as
opposed to values kept from the creation time).


>
> ;;     (setf (org-pending-reglock-user-cancel-function my-lock)
> ;;           (let ((this-timer my-timer))
> ;;             (lambda (_rl)
> ;;               (cancel-timer this-timer)
> ;; ...
>
> ;;     (setf (org-pending-reglock-before-kill-function my-lock)
> ;;           (let ((this-timer my-timer))
> ;;             (lambda (_rl)
> ;;               (message "Killing %s" this-timer)
> ;;               (cancel-timer this-timer))))
>
> What is the difference between "canceling" and "killing" the reglock?
> Do they need to be separate?

If you cut out, from the example, the part where they differ, they do
look the same indeed :)

I'm apparently failing to explain and document this correctly, as it
looks like a recurring topic, sorry.

Yes, they need to be separate as they are two different operations.

 - cancel: The *user* may request a *cancel*; it's a polite way to
   tell org-pending that the user doesn't care anymore about the
   outcome.  A valid implementation is to ignore the user request.
   The default implementation is to unlock the region (sending a
   cancel :failure using 'org-pending-send-update'): it unlocks the
   region, ignoring why it was locked..

 - kill: *Emacs* may have to *kill* some locks, because Emacs is
   killed, or the lock buffer is killed.  org-pending will intercept
   the operations of this kind, ask the user to confirm the
   destruction, and, if confirmed, it will give a chance to the lock
   to do some cleanup by calling the 'before-kill-function'.


I've made some improvements about the kill behaviour and
documentation.

I've pushed my changes to my public branch.

Bruno

Re: Pending contents in org documents

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

>> Fair. Although, it feels like a common use case to replace the region
>> with :success value. Maybe the library should provide some ready-to-use
>> functions that can be used as the value of :on-outcome.
>
> I've recycled the old function used by `org-pending-user-edit',
> improved it and made it the default :on-outcome handler: see
> `org-pending-on-outcome-replace'.  I've simplified the example
> accordingly, removing the custom :on-outcome.

Thanks!
I have one suggestion though. You now do

    Use the function ON-OUTCOME to update the region with the outcome; if it
    is nil, set it to the function `org-pending-on-outcome-replace'.

However, `org-pending' is defined via `cl-defun', so you can instead
just put the default value for :on-outcome key and mention that it is
the default in the docstring. Then, you do not need to do any additional
checks in the function body; `cl-defun' will take care about assigning
the default value.

> From the many examples provided in the branch, do you see any that
> should be included in the library as an other precooked-wrapper, that
> should be included in the section "Basic use of locks" ?

Not sure (I did not look deep into the implementation yet to keep my
perspective closer to end user who first encountered the library)

As a general rule, we should (1) provide simple examples that are easy
to understand and copy/paste; (2) not-so-simple, but useful examples
that are ready to use in practice. These examples should not be complex
either, with all the complexity hidden behind library API (API should be
modified if complexity is unavoidable).

I tried to run your example and have several observations:

1. On failure, it is not obvious that failure happened:
   - The failure overlay disappear very quickly, and is not visible at
      all if I happen to look elsewhere in the buffer. Maybe we can
      simply keep it and remove the overlay on click
   - After failure, the "!" fringe indicator is visible, but it is not
     obvious at all that user can click to get details
     I first tried to click on the fringe itself to no avail. Then, I
     randomly clicked on the text and got the description buffer; but
     that was unexpected - the text I clicked did not have any
     indication of its "clickability" - neither some kind of underline
     face, nor an overlay or a mouse hint.

2. I tried to do M-x undo while the reglock was active, and got an
   error. I'd expect that undo would work, skipping the region.

3. I tried M-x undo _after_ reglock was unlocked, and I got "TO REPLACE"
   word highlighted. I did not expect it to be highlighted.

4. If I try to cancel the reglock, it does get canceled, but *Region
   Lock* buffer is not updated - Live? still displays "yes Cancel".


>> What is the difference between "canceling" and "killing" the reglock?
>> Do they need to be separate?
>
> If you cut out, from the example, the part where they differ, they do
> look the same indeed :)
>
> I'm apparently failing to explain and document this correctly, as it
> looks like a recurring topic, sorry.
>
> Yes, they need to be separate as they are two different operations.
>
>  - cancel: The *user* may request a *cancel*; it's a polite way to
>    tell org-pending that the user doesn't care anymore about the
>    outcome.  A valid implementation is to ignore the user request.
>    The default implementation is to unlock the region (sending a
>    cancel :failure using 'org-pending-send-update'): it unlocks the
>    region, ignoring why it was locked..
>
>  - kill: *Emacs* may have to *kill* some locks, because Emacs is
>    killed, or the lock buffer is killed.  org-pending will intercept
>    the operations of this kind, ask the user to confirm the
>    destruction, and, if confirmed, it will give a chance to the lock
>    to do some cleanup by calling the 'before-kill-function'.

Does it mean that clicking "cancel" does not guarantee that the region
will not be updated by the running process/timer?

In my eyes, there is no difference between user request and "kill". If
users asks things to stop, no modifications should be allowed to
the region.

> I modified the example to rely on the reglock when possible (as
> opposed to values kept from the creation time).

I tried to simplify your example as the following:

(cl-defstruct my/counter (state 0) timer)
(defun my/counter-update (counter reglock &optional force-landing)
  "Increase COUNTER and send update to REGLOCK.
At the end of sequence, cancel COUNTER timer.
When FORCE-LANDING is symbol `land', report :success \"Landed early\" and cancel
the timer."
  (org-pending-send-update
   reglock
   (pcase (my/counter-state counter)
     ((guard (eq force-landing 'land))
      (when (timerp (my/counter-timer counter))
	(cancel-timer (my/counter-timer counter)))
      '(:success "Landed early"))
     ((guard (eq force-landing 'crashed))
      (when (timerp (my/counter-timer counter))
	(cancel-timer (my/counter-timer counter)))
      '(:failure "Crashed"))
     (0 '(:progress "Taking off..."))
     (1 '(:progress "Flying..."))
     (2 '(:progress "Landing..."))
     (_
      (when (timerp (my/counter-timer counter))
	(cancel-timer (my/counter-timer counter)))
      (if (= 0 (random 2))
	  '(:success "Landed successfully")
        '(:failure "Landing malfunction")))))
  (cl-incf (my/counter-state counter)))

(let ((lock-buffer (generate-new-buffer "*Pending region example*"))
      reglock state)
  (with-current-buffer lock-buffer
    (insert "
Buffer displaying pending content.
OUTPUT>>>

TO REPLACE

<<<OUTPUT
More text.
")

    (goto-char (point-min))
    (re-search-forward "TO REPLACE")

    ;; We lock the 'region', defining how to update it when the
    ;; outcome is available.
    (setq reglock (org-pending (cons (match-beginning 0) (match-end 0))))

    (pop-to-buffer lock-buffer)

    (setq state (make-my/counter))
    ;; We create a timer to update our state every few seconds.
    (setf (my/counter-timer state)
	  (run-with-timer 2 2 #'my/counter-update state reglock)))

  (setf (org-pending-reglock-user-cancel-function reglock)
        `(lambda (rlock)
           (warn "Initiating emergency landing...")
           (sleep-for 1)
           (my/counter-update ,state rlock 'land)
           (warn "Initiating emergency landing... done")))

  (setf (org-pending-reglock-before-kill-function reglock)
        `(lambda (_rlock)
           (cancel-timer (my/counter-timer ,state))
           (warn "Transponder signal lost")))

  (setf (org-pending-reglock-insert-details-function reglock)
        `(lambda (rlock _start _end)
           (insert (format "State: %s\n"
                           (my/counter-state ,state)))))

  )

In the above, it is not fully clear for me what BEG and END arguments in
`org-pending-reglock-insert-details-function' mean and where the
insertion happens.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
> I have one suggestion though. You now do
>
>     Use the function ON-OUTCOME to update the region with the outcome; if it
>     is nil, set it to the function `org-pending-on-outcome-replace'.
>
> However, `org-pending' is defined via `cl-defun', so you can instead
> just put the default value for :on-outcome key and mention that it is
> the default in the docstring. Then, you do not need to do any additional
> checks in the function body; `cl-defun' will take care about assigning
> the default value.

Done.


> I tried to run your example and have several observations:
>
> 1. On failure, it is not obvious that failure happened:
>    - The failure overlay disappear very quickly, and is not visible at
>       all if I happen to look elsewhere in the buffer. Maybe we can
>       simply keep it and remove the overlay on click

The current method was to 'sit-for' 0.2 seconds; the new default is to
do nothing (the overlay disappears immediately), but it's now
configurable, see below.


>    - After failure, the "!" fringe indicator is visible, but it is not
>      obvious at all that user can click to get details
>      I first tried to click on the fringe itself to no avail. Then, I
>      randomly clicked on the text and got the description buffer; but
>      that was unexpected - the text I clicked did not have any
>      indication of its "clickability" - neither some kind of underline
>      face, nor an overlay or a mouse hint.

I couldn't find a way to answer a click on a fringe.  Is there a way?

About how much to decorate, it depends on the user, I guess.  For
example, when org-pending is used for org babel, it should be obvious
that one has to click on "#+RESULTS:".

The current decoration is not the best for discoverability, indeed.
But decorating the whole outcome region would be too much IMHO, and,
it might interfer with other buffer fontifications.

I've refactored the code and added two variables so that it's now
configurable, see 'org-pending-outcome-pre-display-function' and
'org-pending-outcome-post-display-function'.

I'll be happy to commit a patch that provide better defaults.

> 2. I tried to do M-x undo while the reglock was active, and got an
>    error. I'd expect that undo would work, skipping the region.

I'm not sure exactly what you did, nor which error you got.  I've
noticed that the hacks (to handle indirect buffers) were flagging the
buffer as "modified" (using text properties).  I've fixed that.

Is the problem solved now ?

>
> 3. I tried M-x undo _after_ reglock was unlocked, and I got "TO REPLACE"
>    word highlighted. I did not expect it to be highlighted.

I couldn't get that behavior, but the undo wasn't correct either.

org-pending is now directly instrumenting the buffer-undo-list, and
manually adding an undo-boundary.

Do you still see your problem ?


> 4. If I try to cancel the reglock, it does get canceled, but *Region
>    Lock* buffer is not updated - Live? still displays "yes Cancel".

It's by design.

The function `org-pending-describe-reglock' works like
`describe-variable' and other similar functions.  You need to revert
the buffer (g) to update it.

The reglock is live in its buffer.


>>> What is the difference between "canceling" and "killing" the reglock?
>>> Do they need to be separate?
>>
>> If you cut out, from the example, the part where they differ, they do
>> look the same indeed :)
>>
>> I'm apparently failing to explain and document this correctly, as it
>> looks like a recurring topic, sorry.
>>
>> Yes, they need to be separate as they are two different operations.
>>
>>  - cancel: The *user* may request a *cancel*; it's a polite way to
>>    tell org-pending that the user doesn't care anymore about the
>>    outcome.  A valid implementation is to ignore the user request.
>>    The default implementation is to unlock the region (sending a
>>    cancel :failure using 'org-pending-send-update'): it unlocks the
>>    region, ignoring why it was locked..
>>
>>  - kill: *Emacs* may have to *kill* some locks, because Emacs is
>>    killed, or the lock buffer is killed.  org-pending will intercept
>>    the operations of this kind, ask the user to confirm the
>>    destruction, and, if confirmed, it will give a chance to the lock
>>    to do some cleanup by calling the 'before-kill-function'.
>
> Does it mean that clicking "cancel" does not guarantee that the region
> will not be updated by the running process/timer?

Yes, org-pending does not enforce that; and it should not, else it
would forbid valid use cases of org-pending. A given user of
org-pending may decide to garantuee that though (using a suitable
function for :on-outcome).


> In my eyes, there is no difference between user request and "kill". If
> users asks things to stop, no modifications should be allowed to
> the region.

There is no relation between "kill" and "cancel".

For "kill", *Emacs* is killing the owner of the lock; there is nothing
to update.  This is synchronous, immediate and definitive.

For "cancel", the *user* is sending an asynchronous message to
org-pending that it would be nice to release this particular lock
sooner, if possible; that message implies that the user doesn't care
about the outcome, but, if that outcome is available, then, just don't
waste it: insert it in the document.

Should I rename "kill" and "cancel" to something better ?



>> I modified the example to rely on the reglock when possible (as
>> opposed to values kept from the creation time).
>
> I tried to simplify your example as the following:
>
[...]
>

Nice! Thanks.

I simplified the pcase.  I switched back to the "my-" prefix.  I'm not
sure why you're using quoted lambdas, as if we were in 2011 ;) I guess
it's so that this example works when copy/pasted to scratch for
evaluation in a non lexical-binding buffer.

I've left the 'warn' messages, but, I'm not sure we should. In my
case, they are just killing my window configuration, even stealing the
window where the lock itself was.



> In the above, it is not fully clear for me what BEG and END arguments in
> `org-pending-reglock-insert-details-function' mean and where the
> insertion happens.

I've removed '_start' and '_end' from the example: they are advanced
features (see the documentation of the field insert-details-function).

The insertion happens in the description buffer, at a position deemed
suitable (the current implementation inserts it at the end of the
buffer).

Thanks!

Bruno

Re: Pending contents in org documents

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

>>    - After failure, the "!" fringe indicator is visible, but it is not
>>      obvious at all that user can click to get details
>>      I first tried to click on the fringe itself to no avail. Then, I
>>      randomly clicked on the text and got the description buffer; but
>>      that was unexpected - the text I clicked did not have any
>>      indication of its "clickability" - neither some kind of underline
>>      face, nor an overlay or a mouse hint.
>
> I couldn't find a way to answer a click on a fringe.  Is there a way?

Well. I do not know a way either :)

> About how much to decorate, it depends on the user, I guess.  For
> example, when org-pending is used for org babel, it should be obvious
> that one has to click on "#+RESULTS:".
>
> The current decoration is not the best for discoverability, indeed.
> But decorating the whole outcome region would be too much IMHO, and,
> it might interfer with other buffer fontifications.

We may do what flycheck/flyspell do. Maybe fontifying not the whole
region, but at least the region part where the indication was placed.

I really find the current behavior unintuitive even for experienced Emacs
users.

> I've refactored the code and added two variables so that it's now
> configurable, see 'org-pending-outcome-pre-display-function' and
> 'org-pending-outcome-post-display-function'.

Makes sense. Although, "display" part in the names is very confusing -
it sounds way too similar to `pre-redisplay-functions', which is
something entirely different.

What about removing org-pending-output-pre-display-function entirely (we
can add it later, if necessary) and renaming
org-pending-outcome-post-display-function to

`org-pending-on-outcome-functions' - an abnormal hook executed after
ON-OUTCOME action.

>> 2. I tried to do M-x undo while the reglock was active, and got an
>>    error. I'd expect that undo would work, skipping the region.
>
> I'm not sure exactly what you did, nor which error you got.  I've
> noticed that the hacks (to handle indirect buffers) were flagging the
> buffer as "modified" (using text properties).  I've fixed that.
>
> Is the problem solved now ?

No.

What I did is:

1. make repro
2. Insert the example code from the top comment and uncomment it
3. M-x eval-buffer
4. *While regloc is active*, press C-x u
5. Observe

Debugger entered--Lisp error: (org-pending-error "Cannot modify a region containing pending content")
  signal(org-pending-error ("Cannot modify a region containing pending content"))
  (closure (t) (&rest _) (signal 'org-pending-error (list "Cannot modify a region containing pending content")))(1 81)
  primitive-undo(1 ((1 . 81) (t . 0)))
  undo-more(1)
  undo(nil)
  funcall-interactively(undo nil)
  command-execute(undo)

>> 3. I tried M-x undo _after_ reglock was unlocked, and I got "TO REPLACE"
>>    word highlighted. I did not expect it to be highlighted.
>
> I couldn't get that behavior, but the undo wasn't correct either.
>
> org-pending is now directly instrumenting the buffer-undo-list, and
> manually adding an undo-boundary.
>
> Do you still see your problem ?

No, this problem is solved now.

>> 4. If I try to cancel the reglock, it does get canceled, but *Region
>>    Lock* buffer is not updated - Live? still displays "yes Cancel".
>
> It's by design.
>
> The function `org-pending-describe-reglock' works like
> `describe-variable' and other similar functions.  You need to revert
> the buffer (g) to update it.

I still find it confusing.
Mostly because it is not clear if pressing "cancel" does anything.

> The reglock is live in its buffer.

What do you mean by that??
How can reglock be active in the buffer that does not contain the locked
text? How can even have different state in different buffers?

>> Does it mean that clicking "cancel" does not guarantee that the region
>> will not be updated by the running process/timer?
>
> Yes, org-pending does not enforce that; and it should not, else it
> would forbid valid use cases of org-pending. A given user of
> org-pending may decide to garantuee that though (using a suitable
> function for :on-outcome).

Imagine that the process that locked the region hangs. As a user, I'd
like to be able to edit text in buffer if I see that the lock is taking
too long. How can I do it without closing the buffer?

>> In my eyes, there is no difference between user request and "kill". If
>> users asks things to stop, no modifications should be allowed to
>> the region.
>
> There is no relation between "kill" and "cancel".
>
> For "kill", *Emacs* is killing the owner of the lock; there is nothing
> to update.  This is synchronous, immediate and definitive.
>
> For "cancel", the *user* is sending an asynchronous message to
> org-pending that it would be nice to release this particular lock
> sooner, if possible; that message implies that the user doesn't care
> about the outcome, but, if that outcome is available, then, just don't
> waste it: insert it in the document.
>
> Should I rename "kill" and "cancel" to something better ?

See my example above. For me, users should have access to "kill" - to
unlock the pending region and make sure that nothing unexpected happens
henceforth with that text. The unrelying Elisp that is creating the lock
should indeed be able to intercept such user request and process it, but
not refuse it, keeping the region locked.

> I simplified the pcase.  I switched back to the "my-" prefix.  I'm not
> sure why you're using quoted lambdas, as if we were in 2011 ;) I guess
> it's so that this example works when copy/pasted to scratch for
> evaluation in a non lexical-binding buffer.

Yes.

> I've left the 'warn' messages, but, I'm not sure we should. In my
> case, they are just killing my window configuration, even stealing the
> window where the lock itself was.

You can use (message ...) instead. It is also fine.

>> In the above, it is not fully clear for me what BEG and END arguments in
>> `org-pending-reglock-insert-details-function' mean and where the
>> insertion happens.
>
> I've removed '_start' and '_end' from the example: they are advanced
> features (see the documentation of the field insert-details-function).

Please link to that documentation somewhere in top comment, linking to
the defstruct. Maybe something like:

   Elisp programs can further alter various fields of REGLOCK object to
   alter its behavior. See the docstrings in `org-pending-reglock'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> About how much to decorate, it depends on the user, I guess.  For
>> example, when org-pending is used for org babel, it should be obvious
>> that one has to click on "#+RESULTS:".
>>
>> The current decoration is not the best for discoverability, indeed.
>> But decorating the whole outcome region would be too much IMHO, and,
>> it might interfer with other buffer fontifications.
>
> We may do what flycheck/flyspell do. Maybe fontifying not the whole
> region, but at least the region part where the indication was placed.
>
> I really find the current behavior unintuitive even for experienced Emacs
> users.

I added 2 faces;  org-pending is now using those faces for the
whole outcome region.  I personally don't like it and won't use it
though :)


>> I've refactored the code and added two variables so that it's now
>> configurable, see 'org-pending-outcome-pre-display-function' and
>> 'org-pending-outcome-post-display-function'.
>
> Makes sense. Although, "display" part in the names is very confusing -
> it sounds way too similar to `pre-redisplay-functions', which is
> something entirely different.

I've renamed them to pre-insert-outcome/post-insert-outcome.


> What about removing org-pending-output-pre-display-function entirely (we
> can add it later, if necessary) and renaming
> org-pending-outcome-post-display-function to
> `org-pending-on-outcome-functions' - an abnormal hook executed after
> ON-OUTCOME action.

I am using it, So, I would prefer to keep it :)

The timing does matter, so I would prefer to keep being explicit about
it (i.e. keep the pre/post prefix).

These 2 functions define the implementation; they are not hooks.


>>> 2. I tried to do M-x undo while the reglock was active, and got an
>>>    error. I'd expect that undo would work, skipping the region.
>>
>> I'm not sure exactly what you did, nor which error you got.  I've
>> noticed that the hacks (to handle indirect buffers) were flagging the
>> buffer as "modified" (using text properties).  I've fixed that.
>>
>> Is the problem solved now ?
>
> No.
>
> What I did is:
>
> 1. make repro
> 2. Insert the example code from the top comment and uncomment it
> 3. M-x eval-buffer
> 4. *While regloc is active*, press C-x u
> 5. Observe
>
> Debugger entered--Lisp error: (org-pending-error "Cannot modify a region containing pending content")
>   signal(org-pending-error ("Cannot modify a region containing pending content"))
>   (closure (t) (&rest _) (signal 'org-pending-error (list "Cannot modify a region containing pending content")))(1 81)
>   primitive-undo(1 ((1 . 81) (t . 0)))
>   undo-more(1)
>   undo(nil)
>   funcall-interactively(undo nil)
>   command-execute(undo)

Thanks for the details.

This is what is supposed to happen: the 'undo' is trying to erase the
whole buffer content, but that buffer contains a reglock, thus,
org-pending is explicitly interrupting that operation and raising a
meaningful error instead.

What other behaviour are you expecting here ?



>>> 3. I tried M-x undo _after_ reglock was unlocked, and I got "TO REPLACE"
>>>    word highlighted. I did not expect it to be highlighted.
>>
>> I couldn't get that behavior, but the undo wasn't correct either.
>>
>> org-pending is now directly instrumenting the buffer-undo-list, and
>> manually adding an undo-boundary.
>>
>> Do you still see your problem ?
>
> No, this problem is solved now.

Perfect! Thanks for testing.

(I removed the undo-boundary from org-pending; I moved it to the
example)

>>> 4. If I try to cancel the reglock, it does get canceled, but *Region
>>>    Lock* buffer is not updated - Live? still displays "yes Cancel".
>>
>> It's by design.
>>
>> The function `org-pending-describe-reglock' works like
>> `describe-variable' and other similar functions.  You need to revert
>> the buffer (g) to update it.
>
> I still find it confusing.
> Mostly because it is not clear if pressing "cancel" does anything.

I added a header to make it clear that the info of the buffer is a
snapshot at a given time.  And, that the user needs to hit the usual
key 'g' to revert the buffer.

When clicking "Cancel", org-pending now aknowledges that the cancel
request has been sent, using a message.


>> The reglock is live in its buffer.
>
> What do you mean by that??
> How can reglock be active in the buffer that does not contain the locked
> text? How can even have different state in different buffers?

Sorry, I meant:

   The reglock displays its status, and keeps it up-to-date, in the
   buffer containing the locked content.


>>> Does it mean that clicking "cancel" does not guarantee that the region
>>> will not be updated by the running process/timer?
>>
>> Yes, org-pending does not enforce that; and it should not, else it
>> would forbid valid use cases of org-pending. A given user of
>> org-pending may decide to garantuee that though (using a suitable
>> function for :on-outcome).
>
> Imagine that the process that locked the region hangs. As a user, I'd
> like to be able to edit text in buffer if I see that the lock is taking
> too long. How can I do it without closing the buffer?

As a user, I would tell the process to hurry up, possibly throwing
data away, as I need to edit that region (i.e. click "cancel").

Whoever started the process (that locked the region) should provide you a
way to stop that process, by answering the cancel request and/or
by providing another suitable interface.

The default implementation will unlock the region immediately,
completely disregarding any "process" and thus, will allow you to
immediately edit the content.


>>> In my eyes, there is no difference between user request and "kill". If
>>> users asks things to stop, no modifications should be allowed to
>>> the region.
>>
>> There is no relation between "kill" and "cancel".
>>
>> For "kill", *Emacs* is killing the owner of the lock; there is nothing
>> to update.  This is synchronous, immediate and definitive.
>>
>> For "cancel", the *user* is sending an asynchronous message to
>> org-pending that it would be nice to release this particular lock
>> sooner, if possible; that message implies that the user doesn't care
>> about the outcome, but, if that outcome is available, then, just don't
>> waste it: insert it in the document.
>>
>> Should I rename "kill" and "cancel" to something better ?
>
> See my example above. For me, users should have access to "kill" - to
> unlock the pending region and make sure that nothing unexpected happens
> henceforth with that text. The unrelying Elisp that is creating the lock
> should indeed be able to intercept such user request and process it, but
> not refuse it, keeping the region locked.

org-pending is just the messenger here. It doesn't start any "process"
and it doesn't refuse anything, it fully cooperates :)

Your "kill" definition looks like the current default "cancel"
implementation.

To avoid further confusion, I'm not using the word "kill" anymore
about reglocks in org-pending.

I added a function 'org-pending-unlock-NOW!' which unlock the region
immediately.  The uppercase "NOW!" emphasizes that it's not the
"safe" way to unlock a region.


>> I simplified the pcase.  I switched back to the "my-" prefix.  I'm not
>> sure why you're using quoted lambdas, as if we were in 2011 ;) I guess
>> it's so that this example works when copy/pasted to scratch for
>> evaluation in a non lexical-binding buffer.
>
> Yes.

ok. I'll try to keep that in mind. Thanks.


>> I've left the 'warn' messages, but, I'm not sure we should. In my
>> case, they are just killing my window configuration, even stealing the
>> window where the lock itself was.
>
> You can use (message ...) instead. It is also fine.

Good. Let's not promote the idea that raising popups from background
jobs at unpredictable time is a good idea: I replaced 'warn' with
'message'.

I've missed that you added a 1s sleep in the custom cancel function.
This was wrong: the documentation says that this function must return
*immediately* (see documentation of the field 'user-cancel-function').
I fixed it.  I also restored the fact that, the outcome of a cancel
request may be either a success or a failure.



>
>>> In the above, it is not fully clear for me what BEG and END arguments in
>>> `org-pending-reglock-insert-details-function' mean and where the
>>> insertion happens.
>>
>> I've removed '_start' and '_end' from the example: they are advanced
>> features (see the documentation of the field insert-details-function).
>
> Please link to that documentation somewhere in top comment, linking to
> the defstruct. Maybe something like:
>
>    Elisp programs can further alter various fields of REGLOCK object to
>    alter its behavior. See the docstrings in `org-pending-reglock'.

Done.


I've just pushed a new version.

Thanks again for your patience and the many reviews.

Bruno

Re: Pending contents in org documents

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

> I've just pushed a new version.

Thanks!

> I added a function 'org-pending-unlock-NOW!' which unlock the region
> immediately.  The uppercase "NOW!" emphasizes that it's not the
> "safe" way to unlock a region.

I expect to see this function called by some kind of button in the
details buffer, so that users can actually call it.

Also, a few more small comments on the commentary section:

> ;; The library makes locks visible to the user using text properties
> ;; and/or overlays.  It diplays and updates the status while the
* displays

> ;; If the user kills a buffer, or, kills Emacs, some locks may have to
> ;; be destroyed.  The library will ask the user to confirm if an
> ;; operation requires to destroy some locks.  See the field
> ;; `before-destroy-function' of REGLOCK object, if you need to do
> ;; something before a lock is destroyed.

We should provide a user option to suppress the query. Something like
what `confirm-kill-processes' does. Maybe even support something akin
`process-query-on-exit-flag', but for reglocks.

> ... feel free to look at the docstrings of the
> ;; cl-defstruct `org-pending-reglock' for the full documentation.

Maybe better refer to M-x cl-describe-type org-pending-reglock. It will
look nicer.

---

I have no other comments on your replies, so I am proceeding with the
code review. Next step is focusing on the core API functions :)

> (cl-defun org-pending (region
> ...
> On receiving the outcome (a :success or :failure message, sent with
> `org-pending-send-update'), remove the region protection.  Call
> ON-OUTCOME with the reglock and the outcome, from the position from
> where the REGLOCK was created.  If ON-OUTCOME returns a region (a
> pair (start position . end position)), use it to report the
> success/failure using visual hints on that region.  If ON-OUTCOME
> returns nothing, don't display outcome marks.

Please describe what the default value of ON-OUTCOME (when ON-OUTCOME is
not explicitly provided) does right in the docstring.

> You may set/update the following fields of your reglock to customize its
> behavior:
>    - Emacs may have to destroy your locks; see the field
>      `before-destroy-function' if you wish to do something before your
>      lock is destroyed.
>    - The user may ask Emacs to cancel your lock; see the field
>      `user-cancel-function' to override the default cancel function.
>    - The user may request a description of the lock; see the the field
>      `insert-details-function' to add custom information when your
>      lock is displayed to the user.
> 
> You may add/update your own properties to your reglock using the field
> `properties', which is an association list.

Here, we may also refer to cl-describe-type for more details about the fields.

>   ( user-cancel-function nil
>     :documentation
>     "Function called when the user wish to cancel this REGLOCK,
> with the REGLOCK as argument.  This function must return immediately; it
> may, asynchronously, stop some processing and release resources; and,
> once this is done, it should send the outcome to the REGLOCK (using
> `org-pending-send-update', so that the region is unlocked and the
> REGLOCK destroyed). The default value is
> `org-pending--user-cancel-default'" )

How come the default value is `org-pending--user-cancel-default' when it
is nil?

> (cl-defstruct (org-pending-reglock
> ...

It would be nice to define slot types for each slot.

> 	insert-details-function		nil
>     When non-nil, function called to insert custom details at the end of
>     ‘org-pending-describe-reglock’.  The function is called with a REGLOCK,
>     a START position and an END position, it must insert details at
>     point. Assuming B is a (virtual) buffer containing all detailed human
>     readable information, insert at point details from START to END.  Handle
>     cases where START, END are nil or out of bounds without raising an
>     error.  The function may use text properties, overlays, etc.  See
>     ‘org-pending-describe-reglock’

From this description, my understanding is that the existing text
between START and END must be replaced? Or is it something else? It is
also not clear what the custom function is supposed to do when START/END
are nil.

I looked up the docstring of `org-pending-describe-reglock', but it does
not contain anything useful wrt setting insert-details-function slot.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> I added a function 'org-pending-unlock-NOW!' which unlock the region
>> immediately.  The uppercase "NOW!" emphasizes that it's not the
>> "safe" way to unlock a region.
>
> I expect to see this function called by some kind of button in the
> details buffer, so that users can actually call it.

This function should not be used, not even in code, except to work
around bugs.  I would prefer not to provide a button for it.


> Also, a few more small comments on the commentary section:
>
>> ;; The library makes locks visible to the user using text properties
>> ;; and/or overlays.  It diplays and updates the status while the
> * displays

Thanks.


>> ;; If the user kills a buffer, or, kills Emacs, some locks may have to
>> ;; be destroyed.  The library will ask the user to confirm if an
>> ;; operation requires to destroy some locks.  See the field
>> ;; `before-destroy-function' of REGLOCK object, if you need to do
>> ;; something before a lock is destroyed.
>
> We should provide a user option to suppress the query. Something like
> what `confirm-kill-processes' does. Maybe even support something akin
> `process-query-on-exit-flag', but for reglocks.

IMHO, it's the package that uses org-pending that should provide such a
feature: I may not care about loosing data by ignoring on-going code
block executions, but, I'll probably care if Emacs aborts some money
transfers.


>
>> ... feel free to look at the docstrings of the
>> ;; cl-defstruct `org-pending-reglock' for the full documentation.
>
> Maybe better refer to M-x cl-describe-type org-pending-reglock. It will
> look nicer.

Thanks. I didn't know how to display that documentation actually :)

As this library is for developpers, I've used the syntax:
     (cl-describe-type 'org-pending-reglock)
so that the documentation is only one click away.


>> (cl-defun org-pending (region
>> ...
>> On receiving the outcome (a :success or :failure message, sent with
>> `org-pending-send-update'), remove the region protection.  Call
>> ON-OUTCOME with the reglock and the outcome, from the position from
>> where the REGLOCK was created.  If ON-OUTCOME returns a region (a
>> pair (start position . end position)), use it to report the
>> success/failure using visual hints on that region.  If ON-OUTCOME
>> returns nothing, don't display outcome marks.
>
> Please describe what the default value of ON-OUTCOME (when ON-OUTCOME is
> not explicitly provided) does right in the docstring.

Done.



>> You may set/update the following fields of your reglock to customize its
>> behavior:
>>    - Emacs may have to destroy your locks; see the field
>>      `before-destroy-function' if you wish to do something before your
>>      lock is destroyed.
>>    - The user may ask Emacs to cancel your lock; see the field
>>      `user-cancel-function' to override the default cancel function.
>>    - The user may request a description of the lock; see the the field
>>      `insert-details-function' to add custom information when your
>>      lock is displayed to the user.
>> 
>> You may add/update your own properties to your reglock using the field
>> `properties', which is an association list.
>
> Here, we may also refer to cl-describe-type for more details about the fields.

Done.



>>   ( user-cancel-function nil
>>     :documentation
>>     "Function called when the user wish to cancel this REGLOCK,
>> with the REGLOCK as argument.  This function must return immediately; it
>> may, asynchronously, stop some processing and release resources; and,
>> once this is done, it should send the outcome to the REGLOCK (using
>> `org-pending-send-update', so that the region is unlocked and the
>> REGLOCK destroyed). The default value is
>> `org-pending--user-cancel-default'" )
>
> How come the default value is `org-pending--user-cancel-default' when it
> is nil?

The only way to build a lock is to use the function 'org-pending'; and
that function was setting it to that value.  I've moved the setting
to the defstruct definition.


>> (cl-defstruct (org-pending-reglock
>> ...
>
> It would be nice to define slot types for each slot.

I added that to my todo list: I'll have to learn how to write types in
elisp (or common lisp?) to be able to do that (... the very bottom of my
list :)
)


>> 	insert-details-function		nil
>>     When non-nil, function called to insert custom details at the end of
>>     ‘org-pending-describe-reglock’.  The function is called with a REGLOCK,
>>     a START position and an END position, it must insert details at
>>     point. Assuming B is a (virtual) buffer containing all detailed human
>>     readable information, insert at point details from START to END.  Handle
>>     cases where START, END are nil or out of bounds without raising an
>>     error.  The function may use text properties, overlays, etc.  See
>>     ‘org-pending-describe-reglock’
>
> From this description, my understanding is that the existing text
> between START and END must be replaced? Or is it something else?

No. That description says:
   "it must insert details at point."


> It is
> also not clear what the custom function is supposed to do when START/END
> are nil.
>
> I looked up the docstring of `org-pending-describe-reglock', but it does
> not contain anything useful wrt setting insert-details-function slot.

Yeah ... that feature was smelling more and more like an overdesign :)

I'm dropping that feature altogether: no more start/end; problem solved
;)

For the record, the idea was to allow to insert a small view from a
10GB log; i.e. to insert *at point* in the description buffer, the
text that begins at START in that log file and ends at END in that log
file; the current default implementation would just display the first
few lines.  That API would allow:
   1. to configure how much is displayed,
   2. to display the tail if needed,
   2. and even to implement paging.

As I removed the feature, the org-pending user is now responsible to
insert at point only a reasonable amount of text.

I've pushed the changes to my repository.

Thanks again for the review.

Bruno

Re: Pending contents in org documents

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

>>> I added a function 'org-pending-unlock-NOW!' which unlock the region
>>> immediately.  The uppercase "NOW!" emphasizes that it's not the
>>> "safe" way to unlock a region.
>>
>> I expect to see this function called by some kind of button in the
>> details buffer, so that users can actually call it.
>
> This function should not be used, not even in code, except to work
> around bugs.  I would prefer not to provide a button for it.

Then, there is no point in this function - users will never know about
it. Maybe you do expose it as a button, but also supply a yes/no prompt
asking for confirmation?

>> We should provide a user option to suppress the query. Something like
>> what `confirm-kill-processes' does. Maybe even support something akin
>> `process-query-on-exit-flag', but for reglocks.
>
> IMHO, it's the package that uses org-pending that should provide such a
> feature: I may not care about loosing data by ignoring on-going code
> block executions, but, I'll probably care if Emacs aborts some money
> transfers.

I kindly disagree. I am not proposing anything dramatically different
from what Emacs already does.

If you look into `confirm-kill-process' docstring, you will see that
Emacs does, in fact, have mechanisms for Elisp packages to tell whether a
given process is important or not (`process-query-on-exit-flag').

Yet, `confirm-kill-processes' overrides that mechanism if Emacs user
wishes to and sets the value to non-default nil.

There is nothing principally different in org-pending users compared to
processes. So, in order to conform with the rest of Emacs API, we do
need to provide an equivalent user customization.
We will not suppress queries by default, but we ought to give users an
option to suppress these queries. Maybe even follow
`confirm-kill-process' setting in how org-pending behaves.

>> Please describe what the default value of ON-OUTCOME (when ON-OUTCOME is
>> not explicitly provided) does right in the docstring.
>
> Done.

Thanks!
There is one typo in the relevant commit:

+ The default ON-OUTCOME
+function replaces the region on success and ignore failures; in all
                                            ^ignores

> I've pushed the changes to my repository.

Continuing the code review.

> (defface org-pending-outcome-failure
> ...
> (defface org-pending-outcome-success

If possible, please derive the faces from either org-faces.el or from
built-in faces like the ones listed in `modus-themes-faces' variable.

Ideally, we should have no hard-coded color names.

> :version "30.1"

Please, use :package-version instead of :version.

> (defun org-pending--status-still-pending-p (status)

This function is trivial and used only once. Maybe you can just inline
it.

> (defvar org-pending--outcome-keymap
> (defvar org-pending--pending-keymap

Why are these keymaps internal? May we better expose them to users for
modification?

It may also be a good idea to add some actual keyboard binding to this
map.

> (defun org-pending--popup-failure-details (exc)

This _internal_ function is unused in org-pending.el, but instead used
in ob-core.el.  I am not yet looking into the changes outside
org-pending, but it is generally not ideal when library users need to
call an internal function from the library.

Also, this particular function does not look relevant to org-pending
library per se. It is rather generic and may be inlined.

> (defun org-pending-reglock-live-p (reglock)

You introduce this API function, but yet using
`org-pending-reglock--get-live-p' directly in other places in the
library. It will be more readable to use `org-pending-reglock-live-p'
everywhere.

> (defun org-pending-reglock-duration (reglock)

The docstring does not mention REGLOCK argument.

> (defun org-pending-reglock-property (reglock prop)

... PROP argument.

> (defun org-pending-reglock-set-property (reglock prop val)
>   "See `org-pending-reglock-property'."

May as well write the full docstring.

> (defun org-pending-reglock-delete-outcome-marks (reglock)
>   "Delete visual hints of the outcome for this REGLOCK, if any.
> Do nothing if the outcome is not known. Do nothing if there are no

We use double space between sentences in the docstrings.

> (cl-defun org-pending (region
>                        &key anchor name
>                        (on-outcome #'org-pending-on-outcome-replace))

NAME key is not documented in the docstring.
(and, more generally, please do M-x checkdoc on org-pending.el)

>                  (user-error "This region lock has been destroyed."))))

Error messages should not end with period.

>   (let ((buffer (get-buffer-create "*Region Lock*")))

Please avoid using constant strings. Instead, declare them as defvar.

> (defun org-pending-describe-reglock (reglock)
>   "Describe REGLOCK in a buffer.
> 
> Describe position REGLOCK.
> The information is displayed in new buffer.

in *a* new buffer... except that it is not really a new buffer, when
another REGLOCK buffer is already being displayed.

May users want to see info about multiple reglocks at the same time?

> (setq-local header-line-format
>                       (format "Lock info (at %s), hit 'g' to update."
>                               (format-time-string "%T")))

Please, avoid using hard-coded bindings.  Instead, see
`substitute-command-keys' (also see how org-capture.el sets up header
line)

> (defun org-pending--describe-reglock-at-point ()

Why internal? It is a command users may want to call via M-x.

> (defun org-pending-pre-insert-outcome-default (_lock _message)
>   "Default value for `org-pending-pre-insert-outcome-function'.")

May as well just use #'ignore

> (defvar org-pending-pre-insert-outcome-function
> (defvar org-pending-post-insert-outcome-function

Should these be defcustom ?

> (overlay-put outcome-ovl
>                  'help-echo
>                  "Last lock outcome, click to popup its full description.")

This assumes fixed key bindings in the lock overlay keymap. Which may or
may not be the case. Please compute 'help-echo programmatically, based on
the key map. (using `substitute-command-keys').

> (cl-defun org-pending--update (reglock status data)

No docstring.  Please, add.

>     (let* ((reg  (org-pending-reglock-region reglock))
>            (start (car reg))
>            (end (cdr reg))
>            (buf (marker-buffer start)))
>       (when (buffer-live-p buf)
>         (with-current-buffer buf
>           (save-excursion
>             (if (> (- end start) 1)

What if the buffer is narrowed and reglock is outside the narrowing?

>   (with-current-buffer buffer
>     (without-restriction

`without-restriction' is only available since Emacs 29. However, we
still support Emacs 27 and will support Emacs 28 for quite a while.
So, please avoid `without-restriction' if possible.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>>>> I added a function 'org-pending-unlock-NOW!' which unlock the region
>>>> immediately.  The uppercase "NOW!" emphasizes that it's not the
>>>> "safe" way to unlock a region.
>>>
>>> I expect to see this function called by some kind of button in the
>>> details buffer, so that users can actually call it.
>>
>> This function should not be used, not even in code, except to work
>> around bugs.  I would prefer not to provide a button for it.
>
> Then, there is no point in this function - users will never know about
> it. Maybe you do expose it as a button, but also supply a yes/no prompt
> asking for confirmation?
>

The function 'org-pending-unlock-NOW!' is part of the API, it's not a
command Emacs end users.

If we make it a command and display that button by default, we'll need
also an option to not display it by default, and, probably an other
option to not ask for confirmation.  Let see later if we really need
to provide all this.


>>> We should provide a user option to suppress the query. Something like
>>> what `confirm-kill-processes' does. Maybe even support something akin
>>> `process-query-on-exit-flag', but for reglocks.
>>
>> IMHO, it's the package that uses org-pending that should provide such a
>> feature: I may not care about loosing data by ignoring on-going code
>> block executions, but, I'll probably care if Emacs aborts some money
>> transfers.
>
> I kindly disagree. I am not proposing anything dramatically different
> from what Emacs already does.
>
> If you look into `confirm-kill-process' docstring, you will see that
> Emacs does, in fact, have mechanisms for Elisp packages to tell whether a
> given process is important or not (`process-query-on-exit-flag').
>
> Yet, `confirm-kill-processes' overrides that mechanism if Emacs user
> wishes to and sets the value to non-default nil.
>
> There is nothing principally different in org-pending users compared to
> processes. So, in order to conform with the rest of Emacs API, we do
> need to provide an equivalent user customization.
> We will not suppress queries by default, but we ought to give users an
> option to suppress these queries. Maybe even follow
> `confirm-kill-process' setting in how org-pending behaves.

In my view, reglocks are more about regions that are being updated, than
processes; they are kind of _planed_ buffer modifications.

But, anyway, I added a user option
`org-pending-confirm-ignore-reglocks-on-exit' that allows to ignore
pending locks when exiting.


>>> Please describe what the default value of ON-OUTCOME (when ON-OUTCOME is
>>> not explicitly provided) does right in the docstring.
>>
>> Done.
>
> Thanks!
> There is one typo in the relevant commit:
>
> + The default ON-OUTCOME
> +function replaces the region on success and ignore failures; in all
>                                             ^ignores
>
Thanks.


>> (defface org-pending-outcome-failure
>> ...
>> (defface org-pending-outcome-success
>
> If possible, please derive the faces from either org-faces.el or from
> built-in faces like the ones listed in `modus-themes-faces' variable.
>
> Ideally, we should have no hard-coded color names.

They were derived from built-in ones (error, success, org-tag, etc.).

I've redesigned the faces and put them all in org-pending, so that
org-pending is indenpendent of Org.

I'm now computing some colours from built-in faces to avoid colour
names. I'm not sure it's what you meant, as even Org itself doesn't do
this.


>> :version "30.1"
> Please, use :package-version instead of :version.

I'm now using:
   :package-version '(Org . "9.7")


>> (defun org-pending--status-still-pending-p (status)
>
> This function is trivial and used only once. Maybe you can just inline
> it.

Right.  But it also defines what "still pending" means.  I prefer to
keep it.


>
>> (defvar org-pending--outcome-keymap
>> (defvar org-pending--pending-keymap
>
> Why are these keymaps internal? May we better expose them to users for
> modification?

They are now part of the public API.



> It may also be a good idea to add some actual keyboard binding to this
> map.

Right.  I've improved the keymaps, trying to work like a button when
the text is read-only.



>> (defun org-pending--popup-failure-details (exc)
>
> This _internal_ function is unused in org-pending.el, but instead used
> in ob-core.el.  I am not yet looking into the changes outside
> org-pending, but it is generally not ideal when library users need to
> call an internal function from the library.
> Also, this particular function does not look relevant to org-pending
> library per se. It is rather generic and may be inlined.

Indeed.  I moved it into ob-core, where it is used.


>> (defun org-pending-reglock-live-p (reglock)
>
> You introduce this API function, but yet using
> `org-pending-reglock--get-live-p' directly in other places in the
> library. It will be more readable to use `org-pending-reglock-live-p'
> everywhere.
>
Done. Thanks.


>> (defun org-pending-reglock-duration (reglock)
>
> The docstring does not mention REGLOCK argument.

Done.


>> (defun org-pending-reglock-property (reglock prop)
>
> ... PROP argument.

Done.


>> (defun org-pending-reglock-set-property (reglock prop val)
>>   "See `org-pending-reglock-property'."
>
> May as well write the full docstring.

Done.


>> (defun org-pending-reglock-delete-outcome-marks (reglock)
>>   "Delete visual hints of the outcome for this REGLOCK, if any.
>> Do nothing if the outcome is not known. Do nothing if there are no
>
> We use double space between sentences in the docstrings.

Oops. Sorry.

>
>> (cl-defun org-pending (region
>>                        &key anchor name
>>                        (on-outcome #'org-pending-on-outcome-replace))
>
> NAME key is not documented in the docstring.
> (and, more generally, please do M-x checkdoc on org-pending.el)

Done. Thanks.


>
>>                  (user-error "This region lock has been destroyed."))))
>
> Error messages should not end with period.

Oops. Sorry.


>>   (let ((buffer (get-buffer-create "*Region Lock*")))
>
> Please avoid using constant strings. Instead, declare them as defvar.

Done.


>> (defun org-pending-describe-reglock (reglock)
>>   "Describe REGLOCK in a buffer.
>> 
>> Describe position REGLOCK.
>> The information is displayed in new buffer.
>
> in *a* new buffer... except that it is not really a new buffer, when
> another REGLOCK buffer is already being displayed.

Thanks.  And I've fixed the doc to more accurately describe what's happening.


> May users want to see info about multiple reglocks at the same time?

The function `org-pending-describe-reglock' works like
`describe-variable' and other Emacs related functions: the user needs
to rename the buffer manually if he wants to keep it.


>> (setq-local header-line-format
>>                       (format "Lock info (at %s), hit 'g' to update."
>>                               (format-time-string "%T")))
>
> Please, avoid using hard-coded bindings.  Instead, see
> `substitute-command-keys' (also see how org-capture.el sets up header
> line)

Done. Thanks for the references to the command and to one example.


>> (defun org-pending--describe-reglock-at-point ()
>
> Why internal? It is a command users may want to call via M-x.

OK.


>> (defun org-pending-pre-insert-outcome-default (_lock _message)
>>   "Default value for `org-pending-pre-insert-outcome-function'.")
>
> May as well just use #'ignore

OK. Done.


>
>> (defvar org-pending-pre-insert-outcome-function
>> (defvar org-pending-post-insert-outcome-function
>
> Should these be defcustom ?

No. This is part of the developer API, not for the end user.


>
>> (overlay-put outcome-ovl
>>                  'help-echo
>>                  "Last lock outcome, click to popup its full description.")
>
> This assumes fixed key bindings in the lock overlay keymap. Which may or
> may not be the case. Please compute 'help-echo programmatically, based on
> the key map. (using `substitute-command-keys').

Done.


>> (cl-defun org-pending--update (reglock status data)
>
> No docstring.  Please, add.

I added some basic documentation; I hope this is enough (this is an
internal function).


>
>>     (let* ((reg  (org-pending-reglock-region reglock))
>>            (start (car reg))
>>            (end (cdr reg))
>>            (buf (marker-buffer start)))
>>       (when (buffer-live-p buf)
>>         (with-current-buffer buf
>>           (save-excursion
>>             (if (> (- end start) 1)
>
> What if the buffer is narrowed and reglock is outside the narrowing?

Good point. I'm now using an inlined without-restriction.


>>   (with-current-buffer buffer
>>     (without-restriction
>
> `without-restriction' is only available since Emacs 29. However, we
> still support Emacs 27 and will support Emacs 28 for quite a while.
> So, please avoid `without-restriction' if possible.

Done. I've inlined its definition.

I've pushed the changes to my branch.

Thanks again for your review and time,


Bruno

Re: Pending contents in org documents

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

>> Then, there is no point in this function - users will never know about
>> it. Maybe you do expose it as a button, but also supply a yes/no prompt
>> asking for confirmation?
>>
>
> The function 'org-pending-unlock-NOW!' is part of the API, it's not a
> command Emacs end users.
>
> If we make it a command and display that button by default, we'll need
> also an option to not display it by default, and, probably an other
> option to not ask for confirmation.  Let see later if we really need
> to provide all this.

Ok.

>> Ideally, we should have no hard-coded color names.
>
> They were derived from built-in ones (error, success, org-tag, etc.).
>
> I've redesigned the faces and put them all in org-pending, so that
> org-pending is indenpendent of Org.
>
> I'm now computing some colours from built-in faces to avoid colour
> names. I'm not sure it's what you meant, as even Org itself doesn't do
> this.

Sorry, that's not what I meant.
I was hoping that you can make use of inherit face attribute to define faces.
Using just a part of an existing face is not a good idea - faces can be
changed to make the _full_ combination of foreground/background/etc
legit, but not necessarily to make individual color values reusable.

If you have no ideas about faces to inherit from, better keep hard-coded
colors.

(Also, this is not too critical; just something nice to have for better
inter-operability with Emacs themes)

>>> (cl-defun org-pending--update (reglock status data)
>>
>> No docstring.  Please, add.
>
> I added some basic documentation; I hope this is enough (this is an
> internal function).

I prefer to have docstrings for every function, including internal
functions. This will make life easier for future contributors when
diagnosing whether a given function behaves as it supposed to.

Ideally, we should detail what the function is expected to do by its
callers in its docstring.  This way, we have a way to check if the code
behaves as it should in the future, after situations like external API
change or unexpected change in another internal API.

> I've pushed the changes to my branch.

Thanks!
Next set of comments :)

>   (let ((map (make-sparse-keymap)))
>     (dolist (k `([touchscreen-down] [mouse-2] [mouse-1]))
>       (define-key map k 'org-pending-describe-reglock-at-point))
>     (when read-only
>       (define-key map [13] 'org-pending-describe-reglock-at-point))
>     map))

Nitpick: #'org-pending... - this will help compiler to catch problems
if something happens to `org-pending-describe-reglock-at-point' (like
rename).

Also, [13] is not very readable.  Better use ?\C-m (or what did you mean?)

> (defun org-pending--make-overlay (type beg-end)
> ...
>   (let ((overlay (make-overlay (car beg-end) (cdr beg-end)))
>         (read-only
> 	 (list
> 	  (lambda (&rest _)
> 	    (signal 'org-pending-error
> 	            (list "Cannot modify a region containing pending content"))))))

You can factor this lambda out into an internal constant.

> (defun org-pending--make-overlay (type beg-end)
>   "Create a pending overlay of type TYPE between BEG-END.

> The pair BEG-END contains 2 positions (BEG . END).
> Create an overlay between BEGIN and END.  Return it.

> See `org-pending--delete-overlay' to delete it."

It would be nice to have the docstring detail what kinds of properties
are applied to the overlay: (1) that it is read-only; (2)
org-pending--owner; (3) org-pending--before-delete; (4) keymap.
It is especially important to document properties that other functions
make use of.

>   (let ((overlay (make-overlay (car beg-end) (cdr beg-end)))
>         (read-only
> 	 (list
> 	  (lambda (&rest _)
> 	    (signal 'org-pending-error
> 	            (list "Cannot modify a region containing pending content"))))))

May you factor this lambda out into an internal constant?

>     (cl-flet ((make-read-only (ovl)
>                 "Make the overly OVL read-only."
>                (overlay-put ovl 'modification-hooks read-only)
>                (overlay-put ovl 'insert-in-front-hooks read-only)
>                (overlay-put ovl 'insert-behind-hooks read-only)))

Or maybe even factor out make-read-only into an internal function that
can mark/unmark overlay/region with read-only text properties.

>       (overlay-put overlay 'org-pending type)
>       (unless (memq type '(:success :failure))
>         (overlay-put overlay 'face 'secondary-selection)

Better use a new org-pending-specific face (inheriting from
secondary-selection).

>         (overlay-put
>          overlay 'help-echo
>          (substitute-command-keys
>           (concat "\\<org-pending-pending-keymap>"
>                   "This content is pending. "
>                   "\\[org-pending-describe-reglock-at-point]"
>                   " to know more."))))

You set a similar help-echo string in two places.  It is a good
candidate to factor things out into a helper function.

>       ;; Hack to detect if our overlay has been copied into an other
>       ;; buffer.
>       (overlay-put overlay 'org-pending--owner (overlay-buffer overlay))

AFAIU, the sole purpose of this is

  > ;; Try to detect and delete overlays that have been wrongly copied
  > ;; from other buffers.

... but cloning buffer does not copy its overlays. Or do I miss something?
Also, "ownder" buffer is reachable via the associated REGLOCK object
(`org-pending-reglock-owner').

>       (when (memq type '(:success :failure))
>         ;; Add a link to the outcome overlay so that we may remove it
>         ;; from any buffer.
>         (with-silent-modifications
>           (add-text-properties (car beg-end) (cdr beg-end)
>                                (list 'org-pending--outcome-overlay overlay)))

Do I understand correctly that this is to support indirect buffers? If
so, why is it not a part of `org-pending--add-overlay-projection'?

>         (overlay-put overlay 'org-pending--before-delete
>                      (lambda ()
>                        (let ((inhibit-modification-hooks t)
>                              (inhibit-read-only t))
>                          (overlay-put overlay 'modification-hooks nil)
>                          (overlay-put overlay 'insert-in-front-hooks nil)
>                          (overlay-put overlay 'insert-behind-hooks nil)
>                          (org-pending--remove-overlay-projection overlay)
>                          ;; Force refontification of the result
>                          ;; (working around indirect buffers hacks).
>                          (let ((start (overlay-start overlay))
>                                (end   (overlay-end overlay)))
>                            (remove-text-properties start end
>                                                    (list 'fontified :not-used
>                                                          'font-lock-face :not-used
>                                                          'font-lock-fontified :not-used))
>                            (font-lock-flush start end)))))

This feels like overengineering.  I'd rather put the overlay removal
code into `org-pending--delete-overlay'.  I see no reason to increase
memory used to store text/overlay properties unless we need to.

Also, a more general question on `org-pending--make-overlay' design: Why
also not setting 'org-pending-reglock property right within this
function? Same for other places that modify the overlay in place after
creating. It feels like REGLOCK, face, help-echo (or even part of it),
and before-string can easily be parameters of
`org-pending--make-overlay'.  At least, REGLOCK.  Other properties may
be simply passed as some kind of (... &rest other-properties) argument
for `org-pending--make-overlay'.
IMHO, it would be nice to keep everything related to creating the
overlay in one function rather than spreading it all over the place.

> (defun org-pending-reglock-status (reglock)
>   "Return the status of REGLOCK.
> The possible status are, in chronological order:
>   :scheduled =>
>      :pending =>
>          :success
>          or :failure."
>   (org-pending-reglock--status reglock))

The return value is a symbol, right? You need to document this fact.

> (defun org-pending-reglock-live-p (reglock)
>   "Return non-nil if REGLOCK is still live.
> A REGLOCK stays live until it receives its outcome: :success or :failure."
>   (funcall (org-pending-reglock--get-live-p reglock)))

Here as well.  I suggest marking the outcome types in the docstrings as
`:success' or `:failure'.

> (defun org-pending-reglock-duration (reglock)
>   "Return REGLOCK duration between its scheduling and its outcome.
> If the outcome is not known, use the current time."
>   (let ((start (org-pending-reglock-scheduled-at reglock))
>         (end (or (org-pending-reglock-outcome-at reglock)
>                  (float-time))))
>     (- end start)))

Is the return value in seconds? Internal time representation?

> (defun org-pending-reglock-set-property (reglock prop val)
>   "Set the value of the property PROP for this REGLOCK.
> See also `org-pending-reglock-property'."
>   (if-let ((b (assq prop (org-pending-reglock-properties reglock))))
>       (setcdr b val)
>     (push (cons prop val)
>           (org-pending-reglock-properties reglock))))

`alist-get' is setf-able as well :)

> (defun org-pending--user-cancel-default (reglock)
>   "Send a cancel message to REGLOCK to close it.
> Default value for `org-pending-reglock-user-cancel-function'."
>   (org-pending-send-update
>    reglock (list :failure (list 'org-pending-user-cancel
>                                 "Canceled"))))

What is the purpose of 'org-pending-user-cancel?
Also, '(:failure (org-pending-user-cancel "Canceled)) would be shorter.

> (defun org-pending-reglock-delete-outcome-marks (reglock)

Not very related, but I am wondering if an API function to retrieve
reglock at point could be useful. WDYT?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Pending contents in org documents

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
[...]
>>> Ideally, we should have no hard-coded color names.
>>
[...]
> If you have no ideas about faces to inherit from, better keep hard-coded
> colors.
>
> (Also, this is not too critical; just something nice to have for better
> inter-operability with Emacs themes)

I've simplified everything, inheriting only basic faces that come
preloaded with emacs.  The display looks good enough, and, the code
will be easier to maintain.  Thanks.



>
>>>> (cl-defun org-pending--update (reglock status data)
>>>
>>> No docstring.  Please, add.
>>
>> I added some basic documentation; I hope this is enough (this is an
>> internal function).
>
> I prefer to have docstrings for every function, including internal
> functions. This will make life easier for future contributors when
> diagnosing whether a given function behaves as it supposed to.
>
> Ideally, we should detail what the function is expected to do by its
> callers in its docstring.  This way, we have a way to check if the code
> behaves as it should in the future, after situations like external API
> change or unexpected change in another internal API.

Ideally, I agree :)

Though, IME, for internal details of experimental code, improving the
code readability and adding inline comments is more effective.



>>   (let ((map (make-sparse-keymap)))
>>     (dolist (k `([touchscreen-down] [mouse-2] [mouse-1]))
>>       (define-key map k 'org-pending-describe-reglock-at-point))
>>     (when read-only
>>       (define-key map [13] 'org-pending-describe-reglock-at-point))
>>     map))
>
> Nitpick: #'org-pending... - this will help compiler to catch problems
> if something happens to `org-pending-describe-reglock-at-point' (like
> rename).

Done.



> Also, [13] is not very readable.  Better use ?\C-m (or what did you mean?)

I just blindly pasted those keys from the Emacs source code, to get
the button-like behavior.  I've replaced it, using keymap-set and
"RET".

To me, '?\C-m' doesn't look simpler than the ASCII code 13 :)



>> (defun org-pending--make-overlay (type beg-end)
>> ...
>>   (let ((overlay (make-overlay (car beg-end) (cdr beg-end)))
>>         (read-only
>>          (list
>>           (lambda (&rest _)
>>             (signal 'org-pending-error
>>                     (list "Cannot modify a region containing pending content"))))))
>
> You can factor this lambda out into an internal constant.

Done.


>> (defun org-pending--make-overlay (type beg-end)
>>   "Create a pending overlay of type TYPE between BEG-END.
>
>> The pair BEG-END contains 2 positions (BEG . END).
>> Create an overlay between BEGIN and END.  Return it.
>
>> See `org-pending--delete-overlay' to delete it."
>
> It would be nice to have the docstring detail what kinds of properties
> are applied to the overlay: (1) that it is read-only; (2)
> org-pending--owner; (3) org-pending--before-delete; (4) keymap.

I tried to improve the current documentation (only ':region' overlays
are read-only). Almost every line of code is now in the documentation:
that internal function is now officially declared "carved in stone",
and, it should ship it as-is :)


I've renamed `org-pending--owner' to `org-pending--real-owner', and I
improved the comment about it in the code.  The property
`org-pending--before-delete' is now gone.


> It is especially important to document properties that other functions
> make use of.

That's why I like closures so much: keep the internal details internal
in one place; no need to split the logic and drop many parts in many
different places :)



>>   (let ((overlay (make-overlay (car beg-end) (cdr beg-end)))
>>         (read-only
>>          (list
>>           (lambda (&rest _)
>>             (signal 'org-pending-error
>>                     (list "Cannot modify a region containing pending content"))))))
>
> May you factor this lambda out into an internal constant?

I think it's the same as above, or I missed something.
Done ? :)


>
>>     (cl-flet ((make-read-only (ovl)
>>                 "Make the overly OVL read-only."
>>                (overlay-put ovl 'modification-hooks read-only)
>>                (overlay-put ovl 'insert-in-front-hooks read-only)
>>                (overlay-put ovl 'insert-behind-hooks read-only)))
>
> Or maybe even factor out make-read-only into an internal function that
> can mark/unmark overlay/region with read-only text properties.

Then, I'll have to document it ... No, thanks :)

More seriously, org-pending doesn't need to mark/unmark.  If an
overlay has been created read-only, it just needs a way to delete it,
and to also delete the attached hacks and, possibly try to undo their
effects.



>>       (overlay-put overlay 'org-pending type)
>>       (unless (memq type '(:success :failure))
>>         (overlay-put overlay 'face 'secondary-selection)
>
> Better use a new org-pending-specific face (inheriting from
> secondary-selection).

Right. The new face is `org-pending-locked'. Thanks.


>
>>         (overlay-put
>>          overlay 'help-echo
>>          (substitute-command-keys
>>           (concat "\\<org-pending-pending-keymap>"
>>                   "This content is pending. "
>>                   "\\[org-pending-describe-reglock-at-point]"
>>                   " to know more."))))
>
> You set a similar help-echo string in two places.  It is a good
> candidate to factor things out into a helper function.

I'm not sure I understand.  The two messages for the user are not the
same, the keymaps are not the same.  I'll end up with something like this:

  (defun overlay-set-help-with-keys (overlay message)
     ...)

Is this really what you meant ?



>>       ;; Hack to detect if our overlay has been copied into an other
>>       ;; buffer.
>>       (overlay-put overlay 'org-pending--owner (overlay-buffer overlay))
>
> AFAIU, the sole purpose of this is
>
>   > ;; Try to detect and delete overlays that have been wrongly copied
>   > ;; from other buffers.
>
> ... but cloning buffer does not copy its overlays. Or do I miss something?

If the function 'make-indirect-buffer' is called with a non-nil CLONE
argument (like `org-tree-to-indirect-buffer' does), the buffer initial
values are copied, and that includes the copy of overlays.

Org-pending overlays work only in the buffer that owns the reglock:
so, to support indirect buffers, org-pending needs to detect and
remove such unwanted copies.  Emacs should probably allow us to flag
our overlays as "not clonable into other buffers".


You can see what happens if you remove this hack and run the provided
examples (that's how I discovered the problem a while ago):

     in scratch/bba-pending-contents/my-async-tests.org
     Section "Test cases" (near the end)


> Also, "ownder" buffer is reachable via the associated REGLOCK object
> (`org-pending-reglock-owner').
>

Right. But it's a hack about overlays and indirect buffers: I prefer
to keep this hack independent of reglocks.



>>       (when (memq type '(:success :failure))
>>         ;; Add a link to the outcome overlay so that we may remove it
>>         ;; from any buffer.
>>         (with-silent-modifications
>>           (add-text-properties (car beg-end) (cdr beg-end)
>>                                (list 'org-pending--outcome-overlay overlay)))
>
> Do I understand correctly that this is to support indirect buffers? If
> so, why is it not a part of `org-pending--add-overlay-projection'?

Correct: this is to support indirect buffers.  I improved the inline
comment.

Only the ':region' overlay is projected with
org-pending--add-overlay-projection, to protect the region from being
modified.

Here, this is about an *outcome* overlay (:success or :failure). That
could be seen as another kind of "projection" indeed, but this one is
used only in `org-pending-delete-outcome-marks', and it's only using
one text property to create a link.



>
>>         (overlay-put overlay 'org-pending--before-delete
>>                      (lambda ()
>>                        (let ((inhibit-modification-hooks t)
>>                              (inhibit-read-only t))
>>                          (overlay-put overlay 'modification-hooks nil)
>>                          (overlay-put overlay 'insert-in-front-hooks nil)
>>                          (overlay-put overlay 'insert-behind-hooks nil)
>>                          (org-pending--remove-overlay-projection overlay)
>>                          ;; Force refontification of the result
>>                          ;; (working around indirect buffers hacks).
>>                          (let ((start (overlay-start overlay))
>>                                (end   (overlay-end overlay)))
>>                            (remove-text-properties start end
>>                                                    (list 'fontified :not-used
>>                                                          'font-lock-face :not-used
>>                                                          'font-lock-fontified :not-used))
>>                            (font-lock-flush start end)))))
>
> This feels like overengineering.  I'd rather put the overlay removal
> code into `org-pending--delete-overlay'.  I see no reason to increase
> memory used to store text/overlay properties unless we need to.

I moved the logic inside `org-pending--delete-overlay'. I reduced the
memory usage (for remove-text-properties calls).  I'm not sure if it's
still "overengineered" though.



> Also, a more general question on `org-pending--make-overlay' design: Why
> also not setting 'org-pending-reglock property right within this
> function? Same for other places that modify the overlay in place after
> creating. It feels like REGLOCK, face, help-echo (or even part of it),
> and before-string can easily be parameters of
> `org-pending--make-overlay'.  At least, REGLOCK.  Other properties may
> be simply passed as some kind of (... &rest other-properties) argument
> for `org-pending--make-overlay'.
> IMHO, it would be nice to keep everything related to creating the
> overlay in one function rather than spreading it all over the place.

You're right that `org-pending--make-overlay' should set the
'org-pending-reglock' property. Done.

I've also moved most of the configuration for outcome overlays from
`org-pending-post-insert-outcome-default' into
'org-pending--make-overlay'.  It makes more sense like that,
indeed. Thanks.


>> (defun org-pending-reglock-status (reglock)
>>   "Return the status of REGLOCK.
>> The possible status are, in chronological order:
>>   :scheduled =>
>>      :pending =>
>>          :success
>>          or :failure."
>>   (org-pending-reglock--status reglock))
>
> The return value is a symbol, right? You need to document this fact.

Done.


>> (defun org-pending-reglock-live-p (reglock)
>>   "Return non-nil if REGLOCK is still live.
>> A REGLOCK stays live until it receives its outcome: :success or :failure."
>>   (funcall (org-pending-reglock--get-live-p reglock)))
>
> Here as well.  I suggest marking the outcome types in the docstrings as
> `:success' or `:failure'.

That function just returns non-nil when live-p, no other promises.
I've quoted the keywords.



>> (defun org-pending-reglock-duration (reglock)
>>   "Return REGLOCK duration between its scheduling and its outcome.
>> If the outcome is not known, use the current time."
>>   (let ((start (org-pending-reglock-scheduled-at reglock))
>>         (end (or (org-pending-reglock-outcome-at reglock)
>>                  (float-time))))
>>     (- end start)))
>
> Is the return value in seconds? Internal time representation?

Ooops: seconds. Fixed. Thanks!



>> (defun org-pending-reglock-set-property (reglock prop val)
>>   "Set the value of the property PROP for this REGLOCK.
>> See also `org-pending-reglock-property'."
>>   (if-let ((b (assq prop (org-pending-reglock-properties reglock))))
>>       (setcdr b val)
>>     (push (cons prop val)
>>           (org-pending-reglock-properties reglock))))
>
> `alist-get' is setf-able as well :)

Indeed, way simpler! :) Thanks!


>> (defun org-pending--user-cancel-default (reglock)
>>   "Send a cancel message to REGLOCK to close it.
>> Default value for `org-pending-reglock-user-cancel-function'."
>>   (org-pending-send-update
>>    reglock (list :failure (list 'org-pending-user-cancel
>>                                 "Canceled"))))
>
> What is the purpose of 'org-pending-user-cancel?

`org-pending-user-cancel' is the error/condition symbol, defined using
'define-error'.  It's designed so that's easy to "unwrap" a message:

   (pcase outcome
      (`(:failure ,err) (signal (car err) (cdr err)))
      (`(:success ,v) v))


> Also, '(:failure (org-pending-user-cancel "Canceled)) would be shorter.

Shorter ... but that message is sent to unknown consumers.  With
'list', every consumer gets its own copy (they are free to mutate it
if they so wish).  With the "quote" version, every consumer will share
the same message, and, if one consumer mutates it, he will change it
for all other consumers, past, present and future, until Emacs is
restarted.  Definitely not the kind of bugs I would like to
investigate.

I added a comment about this choice in the code.  Thanks.


>> (defun org-pending-reglock-delete-outcome-marks (reglock)
>
> Not very related, but I am wondering if an API function to retrieve
> reglock at point could be useful. WDYT?

I added the function `org-pending-lock-or-outcome-at-point', which
will, indeed, be required if anybody would like to customize the
keymaps of pending or outcome overlays. Thanks!


OK, that's it to answer your last review! Thanks!


Just in case, don't jump to a detailed code review of my changes in
ob-core yet, I've some planed changes there.

 - (already applied) I've changed the API in ob-core to fix some
   corner cases (results silent/none/etc.) and error handling: I'm
   moved back to using a callback to handle the outcome: it's a
   simpler and more flexible API than a reglock, plus, the reglock
   doesn't always exist.

 - (incoming) I'm going to redesign the ':nasync' parameter (I've been
   using asynchronous executions for everything by default (but elisp)
   for a few months now, and I realised async = "yes" or "no" is not
   enough for my needs). The plan is to have 3 execution modes:

   - schedule: schedule the execution, return immediately, report the
     outcome when it is known (i.e. async),
     
   - execute: schedule the execution, block the user until the outcome is
     known and handled (i.e. sync), raise on failure, return the result
     on success.
     
   - send: schedule the execution, return immediately, and forget
     about it (do not report success nor failure).

I'm thinking about using ':execution-mode' as the new parameter name
(to replace ':nasync').

After these changes, I will have to recheck everything thoroughly; so,
that will be a good time to fix the `org-pending' interface like you
requested a long time ago.

It's now:

   (cl-defun org-pending (region ...)
   
and, as you said, using two positions will better match the Emacs way,
so it will become:

   (cl-defun org-pending (start end ...)



Next set of comments about the org-pending library itself is very
welcome (as it's mostly independent of it's integration with org-babel).

I've updated my public branch.

Thanks again Ihor, for your time and your review,

Bruno

Re: Pending contents in org documents

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

>>>           (concat "\\<org-pending-pending-keymap>"
>>>                   "This content is pending. "
>>>                   "\\[org-pending-describe-reglock-at-point]"
>>>                   " to know more."))))
>>
>> You set a similar help-echo string in two places.  It is a good
>> candidate to factor things out into a helper function.
>
> I'm not sure I understand.  The two messages for the user are not the
> same, the keymaps are not the same.  I'll end up with something like this:
>
>   (defun overlay-set-help-with-keys (overlay message)
>      ...)
>
> Is this really what you meant ?

Hmm. Not really. I misread the strings a bit and thought that they are
more similar than they actually are.

Aside: it looks like 'help-echo uses `substitute-command-keys' without a
need to run it manually. So, we don't really need to do it:

33.19.4 Properties with Special Meanings (Elisp manual)

    ‘help-echo’
         If text has a string as its ‘help-echo’ property, then when you
         move the mouse onto that text, Emacs displays that string in the
         echo area, or in the tooltip window (*note Tooltips::), after
         passing it through ‘substitute-command-keys’.

>>>       ;; Hack to detect if our overlay has been copied into an other
>>>       ;; buffer.
>>>       (overlay-put overlay 'org-pending--owner (overlay-buffer overlay))
>>
>> AFAIU, the sole purpose of this is
>>
>>   > ;; Try to detect and delete overlays that have been wrongly copied
>>   > ;; from other buffers.
>>
>> ... but cloning buffer does not copy its overlays. Or do I miss something?
>
> If the function 'make-indirect-buffer' is called with a non-nil CLONE
> argument (like `org-tree-to-indirect-buffer' does), the buffer initial
> values are copied, and that includes the copy of overlays.
>
> Org-pending overlays work only in the buffer that owns the reglock:
> so, to support indirect buffers, org-pending needs to detect and
> remove such unwanted copies.  Emacs should probably allow us to flag
> our overlays as "not clonable into other buffers".

Then, may you re-iterate what exactly is the problem if we do allow the
overlays to be copied? Maybe we do not really need all the hassle with
text properties?

>>> (defun org-pending--user-cancel-default (reglock)
>>>   "Send a cancel message to REGLOCK to close it.
>>> Default value for `org-pending-reglock-user-cancel-function'."
>>>   (org-pending-send-update
>>>    reglock (list :failure (list 'org-pending-user-cancel
>>>                                 "Canceled"))))
>>
>> What is the purpose of 'org-pending-user-cancel?
>
> `org-pending-user-cancel' is the error/condition symbol, defined using
> 'define-error'.  It's designed so that's easy to "unwrap" a message:
>
>    (pcase outcome
>       (`(:failure ,err) (signal (car err) (cdr err)))
>       (`(:success ,v) v))

But I do not see any calls to `signal' in such context. Do I miss something?

> Just in case, don't jump to a detailed code review of my changes in
> ob-core yet, I've some planed changes there.

Sure. That will be after we tidy up the core library.

> Next set of comments about the org-pending library itself is very
> welcome (as it's mostly independent of it's integration with org-babel).

Here it is :)

> (cl-defun org-pending--new-button-like-keymap (&key read-only)
>   "Return a new keymap for use on reglock overlays.
> If READ-ONLY is non-nil, add bindings for read-only text else for
> editable text."
>   (let ((map (make-sparse-keymap)))
>     (dolist (k `([touchscreen-down] [mouse-2] [mouse-1]))
>       (define-key map k #'org-pending-describe-reglock-at-point))
>     (when read-only
>       (keymap-set map "RET" #'org-pending-describe-reglock-at-point))
>     map))

> (defvar org-pending-outcome-keymap
>   (org-pending--new-button-like-keymap :read-only nil)
>   "Keymap for outcome overlays.")

> (defvar org-pending-pending-keymap
>   (org-pending--new-button-like-keymap :read-only t)
>   "Keymap for pending region overlays.")

Maybe we can make `org-pending-pending-keymap' inherit from
`org-pending-outcome-keymap'? This way, if the latter is customized, the
former will automatically update the bindings.

> (defun org-pending--make-overlay (reglock type begin-end)
> ...
> (cl-flet ((make-read-only (ovl)
>             "Make the overly OVL read-only."
>            (overlay-put ovl 'modification-hooks read-only)
>            (overlay-put ovl 'insert-in-front-hooks read-only)
>            (overlay-put ovl 'insert-behind-hooks read-only)))

You call `make-read-only' exactly once.  cl-flet is redundant.

>  ...
> (when (memq type '(:success :failure))
>   (let ((bitmap (pcase type
>                   (:success 'large-circle)
>                   (:failure 'exclamation-mark)))
>         (face (pcase type
>                 (:success 'org-done)
>                 (:failure 'org-todo)))

Bitmap and fringe face should be customizeable.

>   ( outcome nil
>     :documentation
>     "The outcome. nil when not known yet. Else a list: (:success RESULT)
> or (:failure ERROR)")

Nitpick: Here, and in a couple of nearby docstrings, there is a missing "." at
the very end

> ( properties nil
>   :documentation
>   "A alist of properties.  Useful to attach custom features to this REGLOCK." )

Nitpick: Usually, "properties" imply plist.  You might consider using a
plist instead of alist or changing the field name to properties-alist.

> ( -delete-outcome-marks (lambda ())

Can just use #'ignore as the default value.

> (defun org-pending-reglock-useless-p (reglock)
>   "Return non-nil if REGLOCK is useless.
> When a REGLOCK becomes useless, org-pending will, at some point, forget
> about it."
>   (funcall (org-pending-reglock--useless-p reglock)))

I feel like this idea of -useless-p is a kludge.  We should use GC
functionality instead, so that Emacs can take care about cleaning up
things.

Note that Emacs exposes custom GC handlers to Elisp level.
Check out "2.4.20 Finalizer Type" section of Elisp manual.

Here is a small demo:

(setq bar 'non-nil)
(with-temp-buffer
  (setq-local foo (list 1 (make-finalizer (lambda () (setq bar 'it-hapenned))))))
# may need to do GC several times, until it actually decides to clean up the killed buffer
(garbage-collect)
(garbage-collect)
bar

> (cl-defun org-pending (region
>                        &key anchor (name "REGLOCK")
>                        (on-outcome #'org-pending-on-outcome-replace))
> ...
> ... The default ON-OUTCOME
> function replaces the region on success and ignores failures; in all
> cases, it returns the outcome region (see the function
> `org-pending-on-outcome-replace').

It is not clear what is used as the replacement by the default
ON-OUTCOME.

> You may set/update the following fields of your reglock to customize its
> behavior:
>    - Emacs may have to destroy your locks; see the field
>      `before-destroy-function' if you wish to do something before your
>      lock is destroyed.

It is not clear which "field" you are referring to when reading from top
to bottom.  Maybe, we can move "(cl-describe-type
\\='org-pending-reglock)" a bit earlier to avoid confusion.

> You may add/update your own properties to your reglock using the field
> `properties', which is an association list.

I think that we should refer to `org-pending-reglock-property' here.

> (let ((to-marker (lambda (p)
>                    ;; Make sure P is a marker.
>                    (or (and (markerp p) p)
>                        (save-excursion (goto-char p) (point-marker)))))

(copy-marker p) is shorter and does not involve moving point around.

Also, we may want to copy these markers unconditionally, even when p is
already the marker, it may have funny insertion type that slurps text
inserted _after_ the marker (and locked region).  So, maybe a simple
unconditional (copy-marker p) is what we want.

    > (save-excursion
    >   (setq anchor
    >         (if (not anchor)
    >             (let ((abeg ;; First non-blank point in region.

The fact that ANCHOR does not include indentation is missing from the
docstring.

    >                    (save-excursion (goto-char (car region))
    >                                    (re-search-forward "[[:blank:]]*")

`skip-chars-forward' would be slightly faster, as we do not need
match-data here.

    >                                    (point-marker)))
    >                   (aend ;; Last position on first line
    >                    (save-excursion (goto-char (car region))
    >                                    (end-of-line)

This will behave funny inside invisible text, because it honors point
adjustments. Better use `line-end-position' + `copy-marker'.

> (setq reglock (org-pending--make
>                :scheduled-at (float-time)
>                :-creation-point (point-marker)

What is the purpose of -creation-point? It does not look like it is
documented in `org-pending' docstring. Are there implicit assumptions
about it in the other internal functions?

>                :-on-outcome on-outcome
>                ;; useless-p returns non-nil when a reglock becomes
>                ;; useless and we may forget about it.  We'll update the
>                ;; function when we get the outcome.
>                :-useless-p (lambda () nil)

#ignore

> (defun org-pending-describe-reglock (reglock)
>   "Describe REGLOCK in a buffer.

Maybe "Display a popup buffer describing REGLOCK."?

> (one-line "Duration"
>           ;; TODO nice human format like 1m27s
>           (format "%.1fs" (org-pending-reglock-duration reglock)))

`org-duration-from-minutes'

> (defun org-pending-post-insert-outcome-default (lock message outcome-region)
>   "Default value for `org-pending-post-insert-outcome-function'."
>   ;; We add some outcome decorations to let the user know what
>   ;; happened and allow him to explore the details.
>   (let* ((outcome-ovl (org-pending--make-overlay lock (car message)
>                                                  outcome-region)))
>     (push `(apply delete-overlay ,outcome-ovl) buffer-undo-list)

This will err when undo is disabled in buffer (and `buffer-undo-list' is
set to t).

> (cl-defun org-pending--update (reglock status data)
>   "Update REGLOCK to its new STATUS, using DATA.
> Update the REGLOCK display to match the status STATUS (:scheduled,
> :progress, :success, :failure).  Also update the REGLOCK as needed.
> Return nothing."
>   (cl-labels
>       ((add-style (status txt)
>          "Add the style matching STATUS over the text TXT."
>          (propertize txt 'face (org-pending-status-face status)))

`add-style' is not used anywhere.

> (short-version-of (msg)
>          "Compute the short version of MSG, to display on the anchor.
>          Must return a string."
>          (if msg
>              (car (split-string (format "%s" msg) "\n" :omit-nulls))
>            ""))

Maybe `truncate-string-to-width'?

> (overlay-put anchor-ovl
>              'before-string
> ...
>              (propertize
>               (pcase status
>                 (:scheduled "⏱")
>                 (:pending "⏳")
>                 (:failure "❌")
>                 (:success "✔️"))

1. indicator strings should be customizeable.
2. We need ASCII fallbacks of these symbols, if they cannot be displayed

> (unless (and (consp outcome-region)
>              (or (integerp (car outcome-region))
>                  (markerp (car outcome-region)))
>              (or (integerp (cdr outcome-region))
>                  (markerp (cdr outcome-region))))
>   (error "Not a region"))))

`integer-or-marker-p'

> (if (not outcome-region)
>     (setf (org-pending-reglock--useless-p reglock)
>           (lambda () t))

#'always

> (let* ((pt (org-pending-reglock--creation-point reglock))
>        (buf (marker-buffer pt)))
>   (message "org-pending: Handling update message at %s@%s: %s"
>            pt buf upd-message)
>   (save-excursion
>     (with-current-buffer buf
>       (save-excursion
>         (goto-char pt)

It will be shorter to use `org-with-point-at' from org-macs.el
Also, when marker is outside, narrowing, your version will behave
unexpectedly.

> (with-current-buffer buf
>   (save-excursion
>     (save-restriction
>       (widen)

Or just (org-with-point-at start ...)

> (if (> (- end start) 1)
>                    ;; Insert in the middle as it's more robust to
>                    ;; keep existing data (text properties, markers,
>                    ;; overlays).
> ...

See `replace-region-contents'

> (defun org-pending-locks-in (start end &optional owned)
>   "Return the list of locks in START..END.

> Return the list of REGLOCK(s) that are alive between START and END, in
> the current buffer.

> When OWNED is non-nil, ignore locks that are not owned by this buffer.

> See also `org-pending-locks-in-buffer-p'."
>   (let ((reglocks)
>         (here nil)
>         ovl)
>     (while (and (< start end)

Mind the narrowing.

> (defun org-pending--mgr-handle-reglock-update (reglock update)
>   "Handle the update UPDATE for this REGLOCK.
> Return nothing."
>   (message "org-pending: reglock update for id=%s: %s"
>            (org-pending-reglock-id reglock) update))

Is it always desired to display a message each time a reglock is updated?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Jack Kamm]

Bruno Barbier <brubar.cs@gmail.com> writes:

> I'm not using it with official org backends (yet).  I'm using it with
> several custom backends that I'm working on.  One of the backend
> delegate the block executions to emacs subprocesses: so I have a kind of
> asynchronous executions for free for any language, including elisp
> itself.

For sessions, wouldn't running in a subprocess prevent the user from
directly interacting with the REPL outside of Org? If so, that's a
problem. Org-babel sessions need to play nicely with inferior Python,
inferior ESS, and other interactive comint modes.

> So, here we go.  You'll find attach a set of patchs.  It works for me with
> Emacsc 30.50 and 9.7-pre (from today).

I suggest to keep these patches on a public branch somewhere, see:
https://orgmode.org/worg/org-contribute.html#patches

  "When discussing important changes, it is sometimes not so useful to
  send long and/or numerous patches.
  
  In this case, you can maintain your changes on a public branch of a
  public clone of Org and send a link to the diff between your changes
  and the latest Org commit that sits in your clone."

I tried running your example on emacs29 using

  emacs -q -L  /path/to/org-mode/lisp my-async-tests.org

but it fails with the error below. Also "make" gives a bunch of
compilation warnings (which I've put at the bottom).

> +You need to load:
> +   #+begin_src elisp :results silent
> +   (load-file "my-async-tests.el")
> +   #+end_src

This raises the following error in *Org-Babel Error Output*

  void-variable
  (org-elib-async-process)
  [ Babel evaluation exited with code 127 ]

All the subsequent blocks don't work because of that, for example:

> +A simple execution:
> +    #+begin_src bash
> +      date
> +    #+end_src

yields:

  org-babel-execute-src-block: No org-babel-execute function for bash: my-shell-babel-schedule!

Finally here are the warnings when running "make":

Compiling single /home/jack/src/org-mode/2024-02-brubar-async/lisp/ob-core.el...

In org-babel--async-feedbacks:
ob-core.el:851:2: Warning: docstring has wrong usage of unescaped single quotes (use \= or different quoting)
ob-core.el:871:9: Warning: Unused lexical variable `result-indent'
ob-core.el:906:18: Warning: Unused lexical variable `header'

Compiling single /home/jack/src/org-mode/2024-02-brubar-async/lisp/org-elib-async.el...

In toplevel form:
org-elib-async.el:52:11: Warning: reference to free variable ‘org-elib-async-process’
org-elib-async.el:52:43: Warning: reference to free variable ‘&key’
org-elib-async.el:52:48: Warning: reference to free variable ‘input’
org-elib-async.el:52:54: Warning: reference to free variable ‘callback’
org-elib-async.el:78:29: Warning: reference to free variable ‘command’
org-elib-async.el:127:9: Warning: Variable ‘last-elapsed’ left uninitialized

In org-elib-async-wait-condition:
org-elib-async.el:137:12: Warning: ‘signal’ called with 3 arguments, but accepts only 2

In end of data:
org-elib-async.el:262:16: Warning: the function ‘org-id-uuid’ is not known to be defined.
org-elib-async.el:52:35: Warning: the function ‘command’ is not known to be defined.
org-elib-async.el:52:2: Warning: the function ‘cl-defun’ might not be defined at runtime.

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Bruno Barbier]

[-- Attachment #1: Type: text/plain, Size: 2427 bytes --]

Hi Jack,

Jack Kamm <jackkamm@gmail.com> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> I'm not using it with official org backends (yet).  I'm using it with
>> several custom backends that I'm working on.  One of the backend
>> delegate the block executions to emacs subprocesses: so I have a kind of
>> asynchronous executions for free for any language, including elisp
>> itself.
>
> For sessions, wouldn't running in a subprocess prevent the user from
> directly interacting with the REPL outside of Org?

Good point. The REPL should be created in the same subprocess; the
REPL display and interaction must happen in the user main emacs.  If
the REPL is based on comint, it should be relatively easy to
implement.


> If so, that's a problem. Org-babel sessions need to play nicely with
> inferior Python, inferior ESS, and other interactive comint modes.

With this solution, the user and the REPL/execution will be in
separate processes; so there will be disavantages.  For basic
interactions, mostly based on text input/output, it should work well.


>> So, here we go.  You'll find attach a set of patchs.  It works for me with
>> Emacsc 30.50 and 9.7-pre (from today).
>
> I suggest to keep these patches on a public branch somewhere, see:
> https://orgmode.org/worg/org-contribute.html#patches
>
>   "When discussing important changes, it is sometimes not so useful to
>   send long and/or numerous patches.
>   
>   In this case, you can maintain your changes on a public branch of a
>   public clone of Org and send a link to the diff between your changes
>   and the latest Org commit that sits in your clone."

Good point.  I'll switch to such a solution as soon as possible.


> I tried running your example on emacs29 using
>
>   emacs -q -L  /path/to/org-mode/lisp my-async-tests.org
>
> but it fails with the error below. Also "make" gives a bunch of
> compilation warnings (which I've put at the bottom).
> ...

My bad: I should have compiled the demo code in a standalone emacs.
I forgot to require some libraries: cl-lib and org-id.

I've now tested with your command line (thanks).  It should now
work. Sorry about that.


> Finally here are the warnings when running "make":
I should have fixed everything; no more (new) warnings.
Thanks!


Please find attached the new set of patchs.  I'll switch to using a
clone and a branch soon, in case if you prefer to wait.

Thanks again!


Bruno




[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-core-async-Add-faces-1-5.patch --]
[-- Type: text/x-patch, Size: 1367 bytes --]

From f67829454ac0d3cd142da1bd0006efa37acce588 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:31:36 +0100
Subject: [PATCH 1/8] ob-core async: Add faces [1/5]

lisp/org-faces.el (org-async-scheduled, org-async-pending,
org-async-failure): new faces
---
 lisp/org-faces.el | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/lisp/org-faces.el b/lisp/org-faces.el
index 0e20de51a..5a8a8fd51 100644
--- a/lisp/org-faces.el
+++ b/lisp/org-faces.el
@@ -736,6 +736,24 @@ (defface org-mode-line-clock-overrun
   "Face used for clock display for overrun tasks in mode line."
   :group 'org-faces)
 
+(defface org-async-scheduled '((t :inherit org-tag :background "gray"))
+  "Face for babel results for code blocks that are scheduled for execution."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
+(defface org-async-pending '((t :inherit org-checkbox :background "dark orange"))
+  "Face for babel results for code blocks that are running."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
+(defface org-async-failure '((t :inherit org-warning))
+  "Face for babel results for code blocks that have failed."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
 (provide 'org-faces)
 
 ;;; org-faces.el ends here
-- 
2.43.0


[-- Attachment #3: 0001-ob-core-async-Add-faces-1-5.patch --]
[-- Type: text/x-patch, Size: 10343 bytes --]

From 9f135bd5e8e153323bed5a3274851fa78f246b83 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:32:00 +0100
Subject: [PATCH 2/8] ob-core async: Add org-babel--async tools [2/5]

---
 lisp/ob-core.el | 213 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 213 insertions(+)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index bfeac257b..d98626fe8 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -792,6 +792,219 @@ (defun org-babel-session-buffer (&optional info)
     (when (org-babel-comint-buffer-livep buffer-name)
       buffer-name)))
 
+(defun org-babel--async-status-face (status)
+  (pcase status
+    (:scheduled 'org-async-scheduled)
+    (:pending   'org-async-pending)
+    (:failure   'org-async-failure)
+    (:success   nil)
+    (_ (error "Not a status"))
+    ))
+
+(defun org-babel--async-make-overlay (beg end)
+  "Create an overlay between positions BEG and END and return it."
+  (let ((overlay (make-overlay beg end))
+        (read-only
+	 (list
+	  (lambda (&rest _)
+	    (user-error
+	     "Cannot modify an area being updated"))))
+        )
+    (cl-flet ((make-read-only
+               (ovl)
+               (overlay-put ovl 'modification-hooks read-only)
+               (overlay-put ovl 'insert-in-front-hooks read-only)
+               (overlay-put ovl 'insert-behind-hooks read-only))
+              )
+      (overlay-put overlay 'org-babel--async-type 'org-babel--async-note)
+      (overlay-put overlay 'face 'secondary-selection)
+      (overlay-put overlay 'help-echo "Pending src block result...")
+      (make-read-only overlay)
+      overlay)))
+
+(defun org-babel--async-result-region (inline-elem &optional info)
+  "Return the region of the results, for the source block at point."
+  (unless info (setq info (org-babel-get-src-block-info)))
+  (save-excursion
+    (when-let ((res-begin  (org-babel-where-is-src-block-result nil info)))
+      (cons res-begin
+            (save-excursion
+              (goto-char res-begin)
+              (if inline-elem
+                  ;; Logic copy/pasted from org-babel-where-is-src-block-result.
+	          (let ((result (org-element-context)))
+		    (and (org-element-type-p result 'macro)
+		         (string= (org-element-property :key result)
+			          "results")
+		         (progn
+			   (goto-char (org-element-end result))
+			   (skip-chars-backward " \t")
+			   (point))))
+                ;; Logic copy/pasted from hide-result
+                (beginning-of-line)
+                (let ((case-fold-search t))
+                  (unless (re-search-forward org-babel-result-regexp nil t)
+	            (error "Not looking at a result line")))
+                (org-babel-result-end)
+                ))))))
+
+(defun org-babel--async-feedbacks (info handle-result
+                                       result-params exec-start-time)
+  "Flag the result as \='scheduled\=' and return how to handle feedbacks.
+
+Use overlays to report progress and status to the user.  Do not delete
+the existing result unless a new one is available.  When the result is
+available, remove the async overlays and insert the result as usual,
+like for a synchronous result.  In case of failure, use an overlay to
+report the error.
+
+The returned function handles 3 types of feedbacks:
+   - (:success R):   Evaluation is successful; result is R.
+   - (:failure ERR): Evaluation failed; error is ERR.
+   - (:pending P): Outcome still pending; current progress is P."
+  ;; FIXME: INFO CMD ... Nothing is used but handle-result here !!
+  (let (;; copy/pasted from org-babel-insert-result
+        (inline-elem (let ((context (org-element-context)))
+		       (and (memq (org-element-type context)
+			          '(inline-babel-call inline-src-block))
+			    context))))
+    (cl-labels
+        ((eot-point (start)
+           "Move to End Of Title after START"
+           (if inline-elem
+               (org-element-end inline-elem)
+             (save-excursion (goto-char start)
+                             (forward-line 1) (point))))
+         (after-indent (pt)
+           "Move after indentation, starting at PT."
+           (save-excursion (goto-char pt) (re-search-forward "[[:blank:]]*")))
+         (mk-result-overlays ()
+           ;; Make 2 overlays to handle the pending result: one title
+           ;; (first line) and one for the body.
+           (pcase-let ((`(,start . ,end) (org-babel--async-result-region
+                                          inline-elem info)))
+             (let ((anchor-end (eot-point start)))
+               (cons (org-babel--async-make-overlay
+                      (after-indent start)
+                      (1- anchor-end))
+                     (org-babel--async-make-overlay
+                      anchor-end end)))))
+         (add-style (status txt)
+           ;; Add the style matching STATUS over the text TXT.
+           (propertize txt 'face (org-babel--async-status-face status)))
+
+         (short-version-of (msg)
+           ;; Compute the short version of MSG, to display in the header.
+           ;; Must return a string.
+           (if msg
+               (car (split-string (format "%s" msg) "\n" :omit-nulls))
+             ""))
+         (update (ovl-title status msg)
+           ;; Update the title overlay to match STATUS and MSG.
+             (overlay-put ovl-title
+                          'face
+                          (org-babel--async-status-face status))
+             (overlay-put ovl-title
+                          'before-string (pcase status
+                                           (:scheduled "⏱")
+                                           (:pending "⏳")
+                                           (:failure "❌")
+                                           (:success "✔️")))
+             (overlay-put ovl-title
+                          'after-string
+                          (propertize (format " |%s|"
+                                              (if (eq :failure status)
+                                                  (if (consp msg) (car msg)
+                                                    (format "%s" msg))
+                                                  (short-version-of msg)))
+                                      'face (org-babel--async-status-face status))))
+         (remove-previous-overlays ()
+           ;; Remove previous title and body overlays.
+           (mapc (lambda (ovl)
+                   (when (eq 'org-babel--async-note
+                             (overlay-get ovl 'org-babel--async-type))
+                     (delete-overlay ovl)))
+                 (when-let ((region (org-babel--async-result-region
+                                     inline-elem info)))
+                   ;; Not sure why, but we do need to start before
+                   ;; point min, else, in some cases, some overlays
+                   ;; are not found.
+                   (overlays-in (max (1- (car region)) (point-min))
+                                (cdr region))))))
+
+      (remove-previous-overlays)
+
+      ;; Ensure there is a non-empty region for the result.
+      (save-excursion
+         (unless (org-babel-where-is-src-block-result (not inline-elem) nil nil)
+          (org-babel-insert-result
+           ;; Use " " for the empty result. That cannot be nil, else it's interpreted
+           ;; as a list. We need at least one char, to separate markers if any.
+           " \n"
+           result-params
+           info nil
+           (nth 0 info) ; lang
+           exec-start-time
+           )))
+
+      ;; Create the overlays that span the result title and its body.
+      (pcase-let ((`(,title-ovl . ,body-ovl) (mk-result-overlays)))
+        ;; Flag the result as ":scheduled".
+        (update title-ovl :scheduled nil)
+
+        ;; The callback, that runs in the org buffer at point.
+        (let ((buf (current-buffer))
+              (pt  (point-marker)))
+          (lambda (feedback)
+            (message "ob-core: Handling outcome at %s@%s: %s" pt buf feedback)
+            (with-current-buffer buf
+              (save-excursion
+                (goto-char pt)
+                (pcase feedback
+                  (`(:success ,r)
+                   ;; Visual beep that the result is available.
+                   (update title-ovl :success r)
+                   (sit-for 0.2)
+                   ;; We remove all overlays and let org insert the result
+                   ;; as it would in the synchronous case.
+                   (delete-overlay title-ovl)
+                   (delete-overlay body-ovl)
+                   (funcall handle-result r))
+
+                  (`(:pending ,r)
+                   ;; Still waiting for the outcome. Update our
+                   ;; overlays with the progress info R.
+                   (message "Updating block at %s@%s" pt buf)
+                   (update title-ovl :pending r))
+
+                  (`(:failure ,err)
+                   ;; We didn't get a result. We update our overlays
+                   ;; to report that failure. And unlock the old
+                   ;; result.
+                   (overlay-put title-ovl 'face nil)
+                   (update title-ovl :failure err)
+                   (delete-overlay body-ovl))
+
+                  (_ (error "Invalid outcome"))
+                  )
+                ))
+            nil))))))
+
+
+(cl-defun org-babel--async-p (params &key default)
+  "Return a non-nil value when the execution is asynchronous.
+Get the value of the :nasync argument and convert it."
+  (if-let ((binding (assq :nasync params)))
+      (pcase (cdr binding)
+        ((pred (not stringp))
+         (error "Invalid value for :nasync argument"))
+        ((or "no" "n")  nil)
+        ((or "yes" "y") t)
+        (_ (error "Invalid value for :nasync argument")))
+    default))
+
+
+
 ;;;###autoload
 (defun org-babel-execute-src-block (&optional arg info params executor-type)
   "Execute the current source code block and return the result.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-ob-core-async-Refactor-handle-result-3-5.patch --]
[-- Type: text/x-patch, Size: 5958 bytes --]

From b0cdd3f5a9bd6e4e72adeac91f968741da95f98f Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:32:22 +0100
Subject: [PATCH 3/8] ob-core async: Refactor handle-result [3/5]

lisp/ob-core.el (org-babel-execute-src-block): Refactor the code to
prepare for the next change: move the part handling the result in its
own function `handle-result'.
---
 lisp/ob-core.el | 105 ++++++++++++++++++++++++------------------------
 1 file changed, 53 insertions(+), 52 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index d98626fe8..d1adba61c 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -1086,7 +1086,58 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		      (make-directory d 'parents)
 		      d))))
 		 (cmd (intern (concat "org-babel-execute:" lang)))
-		 result exec-start-time)
+                 (exec-start-time (current-time))
+                 (handle-result
+                  (lambda (result)
+                    (setq result
+		          (if (and (eq (cdr (assq :result-type params)) 'value)
+			           (or (member "vector" result-params)
+				       (member "table" result-params))
+			           (not (listp result)))
+		              (list (list result))
+		            result))
+	            (let ((file (and (member "file" result-params)
+			             (cdr (assq :file params)))))
+		      ;; If non-empty result and :file then write to :file.
+		      (when file
+		        ;; If `:results' are special types like `link' or
+		        ;; `graphics', don't write result to `:file'.  Only
+		        ;; insert a link to `:file'.
+		        (when (and result
+			           (not (or (member "link" result-params)
+				            (member "graphics" result-params))))
+		          (with-temp-file file
+		            (insert (org-babel-format-result
+			             result
+			             (cdr (assq :sep params)))))
+		          ;; Set file permissions if header argument
+		          ;; `:file-mode' is provided.
+		          (when (assq :file-mode params)
+		            (set-file-modes file (cdr (assq :file-mode params)))))
+		        (setq result file))
+		      ;; Possibly perform post process provided its
+		      ;; appropriate.  Dynamically bind "*this*" to the
+		      ;; actual results of the block.
+		      (let ((post (cdr (assq :post params))))
+		        (when post
+		          (let ((*this* (if (not file) result
+				          (org-babel-result-to-file
+				           file
+				           (org-babel--file-desc params result)
+                                           'attachment))))
+		            (setq result (org-babel-ref-resolve post))
+		            (when file
+			        (setq result-params (remove "file" result-params))))))
+	              (unless (member "none" result-params)
+		        (org-babel-insert-result
+		         result result-params info
+                         ;; append/prepend cannot handle hash as we accumulate
+                         ;; multiple outputs together.
+                         (when (member "replace" result-params) new-hash)
+                         lang
+                         (time-subtract (current-time) exec-start-time)))
+	              (run-hooks 'org-babel-after-execute-hook)
+                      result))))
 	    (unless (fboundp cmd)
 	      (error "No org-babel-execute function for %s!" lang))
 	    (message "Executing %s %s %s..."
@@ -1101,57 +1152,7 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		       (if name
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
-	    (setq exec-start-time (current-time)
-                  result
-		  (let ((r (save-current-buffer (funcall cmd body params))))
-		    (if (and (eq (cdr (assq :result-type params)) 'value)
-			     (or (member "vector" result-params)
-				 (member "table" result-params))
-			     (not (listp r)))
-			(list (list r))
-		      r)))
-	    (let ((file (and (member "file" result-params)
-			     (cdr (assq :file params)))))
-	      ;; If non-empty result and :file then write to :file.
-	      (when file
-		;; If `:results' are special types like `link' or
-		;; `graphics', don't write result to `:file'.  Only
-		;; insert a link to `:file'.
-		(when (and result
-			   (not (or (member "link" result-params)
-				  (member "graphics" result-params))))
-		  (with-temp-file file
-		    (insert (org-babel-format-result
-			     result
-			     (cdr (assq :sep params)))))
-		  ;; Set file permissions if header argument
-		  ;; `:file-mode' is provided.
-		  (when (assq :file-mode params)
-		    (set-file-modes file (cdr (assq :file-mode params)))))
-		(setq result file))
-	      ;; Possibly perform post process provided its
-	      ;; appropriate.  Dynamically bind "*this*" to the
-	      ;; actual results of the block.
-	      (let ((post (cdr (assq :post params))))
-		(when post
-		  (let ((*this* (if (not file) result
-				  (org-babel-result-to-file
-				   file
-				   (org-babel--file-desc params result)
-                                   'attachment))))
-		    (setq result (org-babel-ref-resolve post))
-		    (when file
-		      (setq result-params (remove "file" result-params))))))
-	      (unless (member "none" result-params)
-	        (org-babel-insert-result
-	         result result-params info
-                 ;; append/prepend cannot handle hash as we accumulate
-                 ;; multiple outputs together.
-                 (when (member "replace" result-params) new-hash)
-                 lang
-                 (time-subtract (current-time) exec-start-time))))
-	    (run-hooks 'org-babel-after-execute-hook)
-	    result)))))))
+            (funcall handle-result (save-current-buffer (funcall cmd body params))))))))))
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
   "Expand BODY with PARAMS.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-ob-core-async-Handle-nasync-param-4-5.patch --]
[-- Type: text/x-patch, Size: 2620 bytes --]

From 019a0d2d8ba042606632e13976f8dfaeb37a8e74 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:32:45 +0100
Subject: [PATCH 4/8] ob-core async: Handle :nasync param [4/5]

---
 lisp/ob-core.el | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index d1adba61c..262218923 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -1085,7 +1085,10 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		    (let ((d (file-name-as-directory (expand-file-name dir))))
 		      (make-directory d 'parents)
 		      d))))
-		 (cmd (intern (concat "org-babel-execute:" lang)))
+                 (async (org-babel--async-p params))
+                 (cmd (intern (concat "org-babel-"
+                                      (if async "schedule" "execute")
+                                      ":" lang)))
                  (exec-start-time (current-time))
                  (handle-result
                   (lambda (result)
@@ -1139,8 +1142,8 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 	              (run-hooks 'org-babel-after-execute-hook)
                       result))))
 	    (unless (fboundp cmd)
-	      (error "No org-babel-execute function for %s!" lang))
-	    (message "Executing %s %s %s..."
+	      (error "No org-babel-execute function for %s: %s!" lang (symbol-name cmd)))
+	    (message "Executing %s %s %s %s..."
 		     (capitalize lang)
                      (pcase executor-type
                        ('src-block "code block")
@@ -1148,11 +1151,17 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
                        ('babel-call "call")
                        ('inline-babel-call "inline call")
                        (e (symbol-name e)))
+                     (if async "async" "")
 		     (let ((name (nth 4 info)))
 		       (if name
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
-            (funcall handle-result (save-current-buffer (funcall cmd body params))))))))))
+            (if (not async)
+                (funcall handle-result (save-current-buffer (funcall cmd body params)))
+              (let ((handle-feedback
+                     (org-babel--async-feedbacks info handle-result result-params exec-start-time)))
+                (funcall cmd body params handle-feedback))))))))))
+
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
   "Expand BODY with PARAMS.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-ob-core-async-Add-execute-with-5-5.patch --]
[-- Type: text/x-patch, Size: 2458 bytes --]

From c17aaea885e3aa6087563d55045e74ce71557dbf Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:33:09 +0100
Subject: [PATCH 5/8] ob-core async: Add :execute-with [5/5]

---
 lisp/ob-core.el | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 262218923..64f434f71 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -1086,9 +1086,18 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		      (make-directory d 'parents)
 		      d))))
                  (async (org-babel--async-p params))
-                 (cmd (intern (concat "org-babel-"
-                                      (if async "schedule" "execute")
-                                      ":" lang)))
+                 (execute-with (let ((be (cdr (assq :execute-with params))))
+                                 (when (equal be "none") (setq be nil))
+                                 be))
+                 (cmd (intern (or (and execute-with
+                                       (concat execute-with "-" (if async "schedule" "execute")))
+                                  (concat "org-babel-"
+                                          (if async "schedule" "execute")
+                                          ":" lang))))
+                 (cmd-args (let ((ps (list body params)))
+                             (when execute-with
+                               (setq ps (cons lang ps)))
+                             ps))
                  (exec-start-time (current-time))
                  (handle-result
                   (lambda (result)
@@ -1157,10 +1166,10 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
             (if (not async)
-                (funcall handle-result (save-current-buffer (funcall cmd body params)))
+                (funcall handle-result (save-current-buffer (apply cmd cmd-args)))
               (let ((handle-feedback
                      (org-babel--async-feedbacks info handle-result result-params exec-start-time)))
-                (funcall cmd body params handle-feedback))))))))))
+                (apply cmd (nconc cmd-args (list handle-feedback))))))))))))
 
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #7: 0006-lisp-org-elib-async.el-New-package-about-async-helpe.patch --]
[-- Type: text/x-patch, Size: 14628 bytes --]

From d5766eada2c31d0886f514f5ac6b38ad342e158f Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:33:23 +0100
Subject: [PATCH 6/8] lisp/org-elib-async.el: New package about async helpers

---
 lisp/org-elib-async.el | 327 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 327 insertions(+)
 create mode 100644 lisp/org-elib-async.el

diff --git a/lisp/org-elib-async.el b/lisp/org-elib-async.el
new file mode 100644
index 000000000..f0a1e4432
--- /dev/null
+++ b/lisp/org-elib-async.el
@@ -0,0 +1,327 @@
+;;; org-elib-async.el --- Helper to write asynchronous functions -*- lexical-binding: t -*-
+
+;; Copyright (C) 2024 Bruno BARBIER
+
+;; Author: Bruno BARBIER
+;; Version: 0.0.0
+;; Maintainer: Bruno BARBIER
+;; Keywords:
+;; Status: WORK IN PROGRESS.  DO NOT USE.
+;; URL:
+;; Compatibility: GNU Emacs 30.0.50
+;;
+;; This file is NOT (yet) part of GNU Emacs.
+
+;; This program is free software; you can redistribute it and/or
+;; modify it under the terms of the GNU General Public License as
+;; published by the Free Software Foundation; either version 2 of
+;; the License, or (at your option) any later version.
+
+;; This program is distributed in the hope that it will be
+;; useful, but WITHOUT ANY WARRANTY; without even the implied
+;; warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+;; PURPOSE.  See the GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public
+;; License along with this program; if not, write to the Free
+;; Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+;; MA 02111-1307 USA
+
+
+;;; Commentary:
+;; Names with "--" are for functions and variables that are meant to be for
+;; internal use only.
+
+;;;; Description
+;; Some functions to help dealing with asynchronous tasks.
+
+;; The prefix 'org-elib' means that this package should evenutally be
+;; moved into core Emacs.  The functions defined here do NOT depend
+;; nor rely on org itself.
+
+;;; TODOs
+;;
+;; - Keywords
+;;
+
+
+;;; Code:
+;;
+(require 'cl-lib)
+(require 'org-id)
+
+;;;; Process
+;;
+(cl-defun org-elib-async-process (command &key input callback)
+  "Execute COMMAND.
+
+A quick naive featureless boggus wrapper around `make-process' to
+receive the result when the process is done.
+
+When INPUT is non-nil, use it as the COMMAND standard input.  Let DATA
+be the COMMAND output, if COMMAND succeeds, call CALLBACK with
+(:success DATA), else, call CALLBACK with (:failure DATA)."
+  (let* ((stdout-buffer (generate-new-buffer "*org-elib-async-process*"))
+         (get-outcome
+          (lambda (process)
+            (with-current-buffer stdout-buffer
+              (let* ((exit-code (process-exit-status process))
+                     (real-end ;; Getting rid of the user message.
+                      (progn (goto-char (point-max))
+                             (forward-line -1)
+                             (point)))
+                     (txt (string-trim (buffer-substring-no-properties
+                                        (point-min) real-end))))
+                (list (if (eq 0 exit-code) :success :failure)
+                      (if (not (string-empty-p txt)) txt
+                        (and (not (eq 0 exit-code)) exit-code)))))))
+         (process (make-process
+                   :name "*org-elib-async-process*"
+                   :buffer  stdout-buffer
+                   :command command
+                   :connection-type 'pipe))
+         (sentinel
+          (lambda (&rest _whatever)
+            (pcase (process-status process)
+              ('run )
+              ('stop)
+              ((or 'exit 'signal)
+               (funcall callback (funcall get-outcome process)))
+              (_ (error "Not a real process"))))))
+    (add-function :after (process-sentinel process) sentinel)
+    (when input
+      (process-send-string process input)
+      (process-send-eof process))
+    process))
+;; (org-elib-async-process (list "date") :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "false") :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "true") :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash" "-c" "bash") :input "date" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "date" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "false" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "true" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "sleep 2; date" :callback (lambda (o) (message "outcome: %S" o)))
+
+
+;;;; Wait for a process until some condition becomes true.
+
+(define-error 'org-elib-async-timeout-error
+              "Timeout waiting for a process.")
+
+(cl-defun org-elib-async-wait-condition ( cond-p
+                                          &key
+                                          (tick .3) (message "Waiting")
+                                          (nb_secs_between_messages 5)
+                                          timeout)
+  "Wait until the condition COND-P returns non-nil.
+Repeatedly call COND-P with no arguments, about every TICK seconds,
+until it returns a non-nil value.  Return that non-nil value.  When
+TIMEOUT (seconds) is non-nil, raise an `org-elib-async-timeout-error' if
+the COND-P is still nil after TIMEOUT seconds.  Assume COND-P calls cost
+0s.  Do NOT block display updates.  Do not block process outputs.  Do
+not block idle timers.  Do block the user, letting him/her know why, but
+do not display more messages than one every NB_SECS_BETWEEN_MESSAGES.
+Default MESSAGE is \"Waiting\".  Use 0.3s as the default for TICK."
+  ;; FIXME: Still not sure if it's possible to write such a function.
+  (let ((keep-waiting t)
+        (result nil)
+        (start (float-time))
+        elapsed
+        last-elapsed)
+    (while keep-waiting
+      (setq result (funcall cond-p))
+      (if result
+          (setq keep-waiting nil)
+        (sleep-for 0.01)
+        (redisplay :force)
+        (setq elapsed (- (float-time) start))
+        (when (and timeout (> elapsed timeout))
+          (signal 'org-timeout-error (list message elapsed)))
+        ;; Let the user know, without flooding the message area.
+        (if (and last-elapsed (> (- elapsed last-elapsed) nb_secs_between_messages))
+            (message (format "%s ...(%.1fs)" message elapsed)))
+        (unless (sit-for tick :redisplay)
+          ;; Emacs has something to do; let it process new
+          ;; sub-processes outputs in case there are some.
+          (accept-process-output nil 0.01))
+        (setq last-elapsed elapsed)))
+    result))
+
+
+
+;;;; Comint: a FIFO queue of tasks with callbacks
+;; org-elib-async-comint-queue executes tasks in a FIFO order. For each
+;; task, it identifies the text output for that
+;; task. org-elib-async-comint-queue does NOT remove prompts, or other
+;; useless texts; this is the responsibility of the user.  Currently,
+;; org-elib-async-comint-queue assume it has the full control of the
+;; session: no user interaction, no other direct modifications.
+
+(defvar-local org-elib-async-comint-queue--todo :NOT-SET
+  "A FIFO queue of pending executions.")
+
+
+(defvar-local org-elib-async-comint-queue--unused-output ""
+  "Process output that has not been used yet.")
+
+(defvar-local org-elib-async-comint-queue--incoming-text ""
+  "Newly incoming text, added by the process filter, not yet handled.")
+
+(defvar-local org-elib-async-comint-queue--current-task nil
+  "The task that is currently running.")
+
+(defvar-local org-elib-async-comint-queue--process-filter-running nil
+  "non-nil when filter is running.")
+
+(defvar-local org-elib-async-comint-queue--incoming-timer nil
+  "A timer, when handling incoming text is scheduled or running.")
+
+
+(defvar-local org-elib-async-comint-queue--handle-incoming-running
+    nil
+  "True when the incoming text handler is running.")
+
+(defun org-elib-async-comint-queue--handle-incoming ()
+  (when org-elib-async-comint-queue--handle-incoming-running
+    (error "Bad call to handle-incoming: kill buffer %s!" (current-buffer)))
+  (setq org-elib-async-comint-queue--handle-incoming-running t)
+
+  ;; Take the incoming text.
+  (setq org-elib-async-comint-queue--unused-output
+        (concat org-elib-async-comint-queue--unused-output
+                org-elib-async-comint-queue--incoming-text))
+  (setq org-elib-async-comint-queue--incoming-text "")
+
+  ;; Process the unused text with the queued tasks
+  (unless org-elib-async-comint-queue--current-task
+    (when org-elib-async-comint-queue--todo
+      (setq org-elib-async-comint-queue--current-task (pop org-elib-async-comint-queue--todo))))
+  (when-let ((task org-elib-async-comint-queue--current-task))
+    (let ((unused org-elib-async-comint-queue--unused-output)
+          (session-buffer (current-buffer))
+          task-start)
+      (setq org-elib-async-comint-queue--unused-output
+            (with-temp-buffer
+              (insert unused)
+              (goto-char (point-min))
+              (while (and task
+                          (setq task-start (point))
+                          (search-forward (car task) nil t))
+                (when (cdr task)
+                  (let ((txt (buffer-substring-no-properties task-start
+                                                             (- (point) (length (car task))))))
+                    (save-excursion (funcall (cdr task) txt))))
+                (setq task (and (buffer-live-p session-buffer)
+                                (with-current-buffer session-buffer (pop org-elib-async-comint-queue--todo)))))
+              (buffer-substring (point) (point-max))))
+      (setq org-elib-async-comint-queue--current-task task)))
+
+  ;; Signal that we are done. If we already have some new incoming text,
+  ;; reschedule to run.
+  (setq org-elib-async-comint-queue--incoming-timer
+        (if (string-empty-p org-elib-async-comint-queue--incoming-text)
+            nil
+          (org-elib-async-comint-queue--wake-up-handle-incoming)))
+
+  ;; We reset it only on success. If it failed for some reason, the
+  ;; comint buffer is in an unknown state: you'll need to kill that
+  ;; buffer.
+  (setq org-elib-async-comint-queue--handle-incoming-running nil))
+
+
+(defun org-elib-async-comint-queue--wake-up-handle-incoming ()
+  "Wake up the handling of incoming chunks of text.
+Assume we are called from the comint buffer."
+  (setq org-elib-async-comint-queue--incoming-timer
+        (run-with-timer
+         0.01 nil
+         (let ((comint-buffer (current-buffer)))
+           (lambda ()
+             (with-local-quit
+               (with-current-buffer comint-buffer
+                 (org-elib-async-comint-queue--handle-incoming))))))))
+
+
+(defun org-elib-async-comint-queue--process-filter (chunk)
+  "Accept the arbitrary CHUNK of text."
+  (setq org-elib-async-comint-queue--incoming-text
+        (concat org-elib-async-comint-queue--incoming-text
+                chunk))
+  :; We delegate the real work outside the process filter, as it is
+   ; not reliable to do anything here.
+  (unless org-elib-async-comint-queue--incoming-timer
+    (org-elib-async-comint-queue--wake-up-handle-incoming)))
+
+
+
+(define-error 'org-elib-async-comint-queue-task-error
+              "Task failure.")
+
+(cl-defun org-elib-async-comint-queue--push (exec &key handle-feedback)
+  "Push the execution of EXEC into the FIFO queue.
+When the task completed, call HANDLE-FEEDBACK with its outcome.  Return
+a function that waits for and return the result on succes, raise on
+failure."
+  (let* ((tid (org-id-uuid))
+         (start-tag (format "ORG-ELIB-ASYNC_START_%s" tid))
+         (end-tag (format "ORG-ELIB-ASYNC_END___%s" tid))
+         (result-sb (make-symbol "result"))
+         (on-start
+          (lambda (_)
+            ;; TODO: Use (point) in session to link back to it.
+            (when handle-feedback
+              (funcall handle-feedback '(:pending "running")))))
+         (on-result
+          (lambda (result)
+            ;; Get the result, and report success using HANDLE-FEEDBACK.
+            ;; If something fails, report failure using HANDLE-FEEDBACK.
+            (unwind-protect
+                (let ((outcome
+                       (condition-case-unless-debug exc
+                           (list :success (funcall exec :post-process result))
+                         (error (list :failure exc)))))
+                  (when handle-feedback (save-excursion (funcall handle-feedback outcome)))
+                  (set result-sb outcome))
+              (funcall exec :finally)))))
+
+    ;; TODO: Add detect-properties => alist of properties that can be used: PS1 and PS2
+    (let ((comint-buffer (funcall exec :get-comint-buffer)))
+      (with-current-buffer comint-buffer
+        (setq org-elib-async-comint-queue--todo
+              (nconc org-elib-async-comint-queue--todo
+                     (list (cons start-tag on-start)
+                           (cons end-tag on-result))))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instrs-to-enter))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instr-to-emit-tag start-tag))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :get-code))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instr-to-emit-tag end-tag))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instrs-to-exit))
+
+        (lambda ()
+          (org-elib-async-wait-condition (lambda ()
+                                           (boundp result-sb)))
+          (pcase (symbol-value result-sb)
+            (`(:success ,r) r)
+            (`(:failure ,err) (signal (car err) (cdr err)))))
+        ))))
+
+
+
+(defun org-elib-async-comint-queue-init-if-needed (buffer)
+  "Initialize the FIFO queue in BUFFER if needed."
+  (with-current-buffer buffer
+    (unless (local-variable-p 'org-elib-async-comint-queue--todo)
+      (setq-local org-elib-async-comint-queue--todo nil)
+      (add-hook 'comint-output-filter-functions
+                #'org-elib-async-comint-queue--process-filter nil :local))))
+
+
+
+;;;; Provide
+(provide 'org-elib-async)
+;;; org-elib-async.el ends here
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #8: 0007-lisp-ob-core.el-Notify-when-execution-fails.patch --]
[-- Type: text/x-patch, Size: 3220 bytes --]

From 1129770cc62b0580c71ba2a3f6a94f6fd18574b3 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:33:40 +0100
Subject: [PATCH 7/8] lisp/ob-core.el: Notify when execution fails

lisp/ob-core.el (org-babel-popup-failure-details): New function.
(org-babel-execute-src-block): For synchronous execution, use
`org-babel-popup-failure-details' on failure.
(org-babel--async-feedbacks): Add call to
`org-babel-popup-failure-details' on user request.
---
 lisp/ob-core.el | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 64f434f71..e8f9e9ad9 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -917,7 +917,17 @@ (defun org-babel--async-feedbacks (info handle-result
                                                   (if (consp msg) (car msg)
                                                     (format "%s" msg))
                                                   (short-version-of msg)))
-                                      'face (org-babel--async-status-face status))))
+                                      'face (org-babel--async-status-face status)))
+             (when (eq :failure status)
+               (overlay-put ovl-title
+                            'keymap
+                            (let ((km (make-sparse-keymap)))
+                              (define-key km (kbd "<mouse-1>")
+                                          (lambda ()
+                                            "Display failure details."
+                                            (interactive)
+                                            (org-babel-popup-failure-details msg)))
+                              km))))
          (remove-previous-overlays ()
            ;; Remove previous title and body overlays.
            (mapc (lambda (ovl)
@@ -1166,11 +1176,26 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
             (if (not async)
-                (funcall handle-result (save-current-buffer (apply cmd cmd-args)))
+                (let ((res-sb (make-symbol "result")))
+                  (condition-case exc
+                      (set res-sb (save-current-buffer (apply cmd cmd-args)))
+                    (error (org-babel-popup-failure-details exc)))
+                  (when (boundp res-sb)
+                    (funcall handle-result (symbol-value res-sb))))
               (let ((handle-feedback
                      (org-babel--async-feedbacks info handle-result result-params exec-start-time)))
                 (apply cmd (nconc cmd-args (list handle-feedback))))))))))))
 
+(defun org-babel-popup-failure-details (exc)
+  "Notify/display"
+  (when-let ((buf (get-buffer org-babel-error-buffer-name)))
+    (with-current-buffer buf (erase-buffer)))
+  (org-babel-eval-error-notify
+   127 ; Don't have exit-code
+   (if (consp exc)
+       (format "%s\n%s\n" (car exc) (cdr exc))
+     (format "%s\n" exc))))
+
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
   "Expand BODY with PARAMS.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #9: 0007-lisp-ob-core.el-Notify-when-execution-fails.patch --]
[-- Type: text/x-patch, Size: 33567 bytes --]

From 2078be0ebd741127c383a386f672dbc3d741206d Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Wed, 21 Feb 2024 15:54:05 +0100
Subject: [PATCH 8/8] scratch/bba-ob-core-async: Some temporary test files

---
 scratch/bba-ob-core-async/my-async-tests.el  | 339 ++++++++
 scratch/bba-ob-core-async/my-async-tests.org | 820 +++++++++++++++++++
 2 files changed, 1159 insertions(+)
 create mode 100644 scratch/bba-ob-core-async/my-async-tests.el
 create mode 100644 scratch/bba-ob-core-async/my-async-tests.org

diff --git a/scratch/bba-ob-core-async/my-async-tests.el b/scratch/bba-ob-core-async/my-async-tests.el
new file mode 100644
index 000000000..918a4b142
--- /dev/null
+++ b/scratch/bba-ob-core-async/my-async-tests.el
@@ -0,0 +1,339 @@
+;;; my-async-tests.el --- Scratch/temporary file: some tests about async -*- lexical-binding: t -*-
+
+;; Copyright (C) 2024 Bruno BARBIER
+
+;; Author: Bruno BARBIER
+;; Status: Temporary tests.
+;; Compatibility: GNU Emacs 30.0.50
+;;
+;; This file is NOT part of GNU Emacs.
+
+;; This program is free software; you can redistribute it and/or
+;; modify it under the terms of the GNU General Public License as
+;; published by the Free Software Foundation; either version 2 of
+;; the License, or (at your option) any later version.
+
+;; This program is distributed in the hope that it will be
+;; useful, but WITHOUT ANY WARRANTY; without even the implied
+;; warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+;; PURPOSE.  See the GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public
+;; License along with this program; if not, write to the Free
+;; Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+;; MA 02111-1307 USA
+
+(require 'cl-lib)
+(require 'org)
+(require 'org-elib-async)
+
+(require 'ob-shell)
+(require 'ob-python)
+
+;;; Shells
+;;
+
+;;;; One shell script
+
+;; Standalone direct asynchronous execution.
+
+(defun my-shell-babel-schedule (lang body _params handle-feedback)
+  "Execute the bash script BODY.
+Execute the shell script BODY using bash.  Use HANDLE-FEEDBACK to report
+the outcome (success or failure)."
+  (unless (equal "bash" lang)
+    (error "Only for bash"))
+  (funcall handle-feedback (list :pending "started"))
+  (org-elib-async-process (list "bash") :input body :callback handle-feedback))
+
+
+;;;; Asynchronous using ob-shell
+
+(defun my-org-babel-shell-how-to-execute (body params)
+  "Return how to execute BODY using a POSIX shell.
+Return how to execute, as expected by
+`org-elib-async-comint-queue--execution'."
+  ;; Code mostly extracted from ob-shell, following
+  ;; `org-babel-execute:shell' and `org-babel-sh-evaluate'.
+  ;; Results are expected to differ from ob-shell as we follow the
+  ;; same process for all execution paths: asynchronous or not, with
+  ;; session or without.
+  (let* ((session (org-babel-sh-initiate-session
+		   (cdr (assq :session params))))
+	 (stdin (let ((stdin (cdr (assq :stdin params))))
+                  (when stdin (org-babel-sh-var-to-string
+                               (org-babel-ref-resolve stdin)))))
+	 (result-params (cdr (assq :result-params params)))
+	 (value-is-exit-status
+	  (or (and
+	       (equal '("replace") result-params)
+	       (not org-babel-shell-results-defaults-to-output))
+	      (member "value" result-params)))
+	 (cmdline (cdr (assq :cmdline params)))
+         (shebang (cdr (assq :shebang params)))
+         (full-body (concat
+		     (org-babel-expand-body:generic
+		      body params (org-babel-variable-assignments:shell params))
+		     (when value-is-exit-status "\necho $?")))
+         (post-process
+          (lambda (r)
+            (setq r (org-trim r))
+            (org-babel-reassemble-table
+             (org-babel-result-cond result-params
+               r
+               (let ((tmp-file (org-babel-temp-file "sh-")))
+                 (with-temp-file tmp-file (insert r))
+                 (org-babel-import-elisp-from-file tmp-file)))
+             (org-babel-pick-name
+              (cdr (assq :colname-names params)) (cdr (assq :colnames params)))
+             (org-babel-pick-name
+              (cdr (assq :rowname-names params)) (cdr (assq :rownames params))))))
+         comint-buffer
+         finally
+         to-run)
+
+    (setq comint-buffer
+          (if session session
+            ;; No session. We create a temporary one and use 'finally' to
+            ;; destroy it once we are done.
+            ;;
+            ;; FIXME: This session code should be refactored and moved into
+            ;;        ob-core.
+            (let ((s-buf (org-babel-sh-initiate-session
+                          (generate-new-buffer-name (format "*ob-shell-no-session*")))))
+              (setq finally (lambda ()
+                                ;; We cannot delete it immediately as we are called from it.
+                                (run-with-idle-timer
+                                 0.1 nil
+                                 (lambda ()
+                                   (when (buffer-live-p s-buf)
+                                     (let ((kill-buffer-query-functions nil)
+                                           (kill-buffer-hook nil))
+                                       (kill-buffer s-buf)))))))
+              s-buf)))
+
+    (org-elib-async-comint-queue-init-if-needed comint-buffer)
+
+    (setq to-run
+          (cond
+           ((or stdin cmdline)	       ; external shell script w/STDIN
+            (let ((script-file (org-babel-temp-file "sh-script-"))
+	          (stdin-file (org-babel-temp-file "sh-stdin-"))
+	          (padline (not (string= "no" (cdr (assq :padline params))))))
+	      (with-temp-file script-file
+	        (when shebang (insert shebang "\n"))
+	        (when padline (insert "\n"))
+	        (insert full-body))
+	      (set-file-modes script-file #o755)
+	      (with-temp-file stdin-file (insert (or stdin "")))
+	      (with-temp-buffer
+                (with-connection-local-variables
+                 (concat
+                  (mapconcat #'shell-quote-argument
+                             (cons (if shebang (file-local-name script-file)
+                                     shell-file-name)
+                                   (if shebang (when cmdline (list cmdline))
+                                     (list shell-command-switch
+                                           (concat (file-local-name script-file)  " " cmdline))))
+                             " ")
+                  "<" (shell-quote-argument stdin-file))))))
+           (session ; session evaluation
+            full-body)
+           ;; External shell script, with or without a predefined
+           ;; shebang.
+           ((org-string-nw-p shebang)
+            (let ((script-file (org-babel-temp-file "sh-script-"))
+	          (padline (not (equal "no" (cdr (assq :padline params))))))
+	      (with-temp-file script-file
+	        (insert shebang "\n")
+	        (when padline (insert "\n"))
+	        (insert full-body))
+	      (set-file-modes script-file #o755)
+              (if (file-remote-p script-file)
+                  ;; Run remote script using its local path as COMMAND.
+                  ;; The remote execution is ensured by setting
+                  ;; correct `default-directory'.
+                  (let ((default-directory (file-name-directory script-file)))
+                    (file-local-name script-file)
+	            script-file ""))))
+           (t
+            (let ((script-file (org-babel-temp-file "sh-script-")))
+	      (with-temp-file script-file
+	        (insert full-body))
+	      (set-file-modes script-file #o755)
+              (mapconcat #'shell-quote-argument
+                         (list shell-file-name
+                               shell-command-switch
+                               (if (file-remote-p script-file)
+                                   (file-local-name script-file)
+                                 script-file))
+                         " ")))))
+    ;; TODO: How to handle `value-is-exit-status'?
+    (lambda (&rest q)
+      (pcase q
+        (`(:instrs-to-enter)
+         ;; FIXME: This is wrong.
+         "export PS1=''; export PS2='';")
+        (`(:instrs-to-exit))
+        (`(:finally) (when finally (funcall finally)))
+        (`(:instr-to-emit-tag ,tag) (format "printf '%s\\n'" tag))
+        (`(:post-process ,r) (when post-process (funcall post-process r)))
+        (`(:send-instrs-to-session ,code)
+         (with-current-buffer comint-buffer
+           (when code
+             (goto-char (point-max))
+             (insert code) (insert "\n")
+             (comint-send-input nil t))))
+        (`(:get-code) to-run)
+        (`(:get-comint-buffer) comint-buffer)
+        (_ (error "Unknown query"))))))
+
+
+
+
+
+;;; Python
+;;
+
+;;;; Asynchronous using ob-python
+
+(defun my-org-babel-python-how-to-execute (body params)
+  "Return how to execute BODY using python.
+Return how to execute, as expected by
+`org-elib-async-comint-queue--execution'."
+  ;; Code mostly extracted from ob-python, following
+  ;; `org-babel-python-evaluate-session'.
+  ;; Results are expected to differ from ob-python as we follow the
+  ;; same process for all execution paths: asynchronous or not, with
+  ;; session or without.
+  (let* ((org-babel-python-command
+          (or (cdr (assq :python params))
+              org-babel-python-command))
+         (session-key (org-babel-python-initiate-session
+                       (cdr (assq :session params))))
+         (graphics-file (and (member "graphics" (assq :result-params params))
+                             (org-babel-graphical-output-file params)))
+         (result-params (cdr (assq :result-params params)))
+         (result-type (cdr (assq :result-type params)))
+         (results-file (when (eq 'value result-type)
+                         (or graphics-file
+                             (org-babel-temp-file "python-"))))
+         (return-val (when (eq result-type 'value)
+                       (cdr (assq :return params))))
+         (full-body
+          (concat
+           (org-babel-expand-body:generic
+            body params
+            (org-babel-variable-assignments:python params))
+           (when return-val
+             (format "\n%s" return-val))))
+         (post-process
+          (lambda (r)
+            (setq r (string-trim r))
+            (when (string-prefix-p "Traceback (most recent call last):" r)
+              (signal 'user-error (list r)))
+            (when (eq 'value result-type)
+              (setq r (org-babel-eval-read-file results-file)))
+            (org-babel-reassemble-table
+             (org-babel-result-cond result-params
+               r
+               (org-babel-python-table-or-string r))
+             (org-babel-pick-name (cdr (assq :colname-names params))
+                                  (cdr (assq :colnames params)))
+             (org-babel-pick-name (cdr (assq :rowname-names params))
+                                  (cdr (assq :rownames params))))))
+         (tmp-src-file (org-babel-temp-file "python-"))
+         (session-body
+          ;; The real code we evaluate in the session.
+          (pcase result-type
+            (`output
+             (format (string-join
+                      (list "with open('%s') as f:\n"
+                            "    exec(compile(f.read(), f.name, 'exec'))\n"))
+                     (org-babel-process-file-name
+                      tmp-src-file 'noquote)))
+            (`value
+             ;; FIXME: In this case, any output is an error.
+             (org-babel-python-format-session-value
+              tmp-src-file results-file result-params))))
+         comint-buffer
+         finally)
+
+
+    (unless session-key
+      ;; No session. We create a temporary one and use 'finally' to
+      ;; destroy it once we are done.
+      ;;
+      ;; FIXME: This session code should be refactored and moved into
+      ;;        ob-core.
+      (setq session-key (org-babel-python-initiate-session
+                         ;; We can't use a simple `generate-new-buffer'
+                         ;; due to the earmuffs game.
+                         (org-babel-python-without-earmuffs
+                          (format "*ob-python-no-session-%s*" (org-id-uuid)))))
+      (setq finally (lambda ()
+                      (when-let ((s-buf
+                                  (get-buffer (org-babel-python-with-earmuffs session-key))))
+                        ;; We cannot delete it immediately as we are called from it.
+                        (run-with-idle-timer
+                         0.1 nil
+                         (lambda ()
+                           (when (buffer-live-p s-buf)
+                             (let ((kill-buffer-query-functions nil)
+                                   (kill-buffer-hook nil))
+                               (kill-buffer s-buf)))))))))
+
+    (setq comint-buffer
+          (get-buffer (org-babel-python-with-earmuffs session-key)))
+    (org-elib-async-comint-queue-init-if-needed comint-buffer)
+    (with-temp-file tmp-src-file
+      (insert (if (and graphics-file (eq result-type 'output))
+                  (format org-babel-python--output-graphics-wrapper
+                          full-body graphics-file)
+                full-body)))
+
+    (lambda (&rest q)
+      (pcase q
+        (`(:instrs-to-enter)
+         ;; FIXME: This is wrong.
+         "import sys; sys.ps1=''; sys.ps2=''")
+        (`(:instrs-to-exit))
+        (`(:finally) (when finally (funcall finally)))
+        (`(:instr-to-emit-tag ,tag) (format "print ('%s')" tag))
+        (`(:post-process ,r) (when post-process (funcall post-process r)))
+        (`(:send-instrs-to-session ,code)
+         ;; See org-babel-python-send-string
+         (with-current-buffer comint-buffer
+           (let ((python-shell-buffer-name
+                  (org-babel-python-without-earmuffs session-key)))
+             (python-shell-send-string (concat code "\n")))))
+        (`(:get-code) session-body)
+        (`(:get-comint-buffer) comint-buffer)
+        (_ (error "Unknown query"))))))
+
+
+;;; Org babel 'execute-with'.
+
+
+;;;; Asynchronous
+;;
+(defun my-org-babel-schedule (lang body params handle-feedback)
+  "Schedule the execution of BODY according to PARAMS.
+This function is called by `org-babel-execute-src-block'.  Return a
+function that waits and returns the result on success, raise on failure."
+  (let ((exec (pcase lang
+                ("python" (my-org-babel-python-how-to-execute body params))
+                ("bash"   (my-org-babel-shell-how-to-execute body params))
+                (_ (error "Not handled (yet): %s" lang)))))
+    (org-elib-async-comint-queue--push exec :handle-feedback handle-feedback)))
+
+
+;;;; Synchronous
+;;
+(defun my-org-babel-execute (lang body params)
+  "Execute Python BODY according to PARAMS.
+This function is called by `org-babel-execute-src-block'."
+  ;; We just start the asynchronous execution, wait for it, and return
+  ;; the result (or raise the exception). No custom code, and,
+  ;; synchronous and asynchronous should just mix nicely together.
+  (funcall (my-org-babel-schedule lang body params nil)))
diff --git a/scratch/bba-ob-core-async/my-async-tests.org b/scratch/bba-ob-core-async/my-async-tests.org
new file mode 100644
index 000000000..08284c470
--- /dev/null
+++ b/scratch/bba-ob-core-async/my-async-tests.org
@@ -0,0 +1,820 @@
+#+PROPERTY: HEADER-ARGS+ :eval no-export :exports both
+* Intro
+
+An org document with code blocks to help test the proposed patchs.
+
+See [[*On top of ob-shell][On top of ob-shell]] for asynchronous shell scripts using ob-shell.
+
+See [[*On top of ob-python][On top of ob-python]] for asynchronous python scripts using ob-python.
+
+You need to load:
+   #+begin_src elisp :results silent
+   (setq-local org-confirm-babel-evaluate nil)
+   (load-file "my-async-tests.el")
+   #+end_src
+
+
+Emacs and org versions:
+   #+begin_src elisp
+   (mapcar (lambda (sb) (list sb (symbol-value sb)))
+           '(emacs-version org-version))
+   #+end_src
+
+   #+RESULTS:
+   | emacs-version | 30.0.50 |
+   | org-version   | 9.7-pre |
+
+Note that we've disabled eval on export: export doesn't know it needs
+to wait for asynchronous results.
+
+* POSIX shells
+** A simple bash example
+   :PROPERTIES:
+    :header-args:bash: :execute-with my-shell-babel :nasync yes
+   :END:
+
+The package `my-async-tests.el' contains the function
+`my-shell-babel-schedule' to evaluate shell script asynchronously.
+
+The header-args properties above request asynchronous execution for
+bash (:nasync yes), and, tells ob-core to use the prefix
+`my-shell-babel' when looking for functions to evaluate a source
+block. Thus, org will delegate execution to `my-shell-babel-schedule'.
+We don't have `my-shell-babel-execute', so, in this case, :nasync must
+be yes.
+
+Examples taken from the org mailing list and from worg.
+
+A simple execution:
+    #+begin_src bash
+      date
+    #+end_src
+
+    #+RESULTS:
+    : Wed Feb 21 15:56:23 CET 2024
+
+A tricky computation takes some time:
+    #+begin_src bash
+      sleep 1; date
+    #+end_src
+
+    #+RESULTS:
+    : Wed Feb 21 15:56:25 CET 2024
+
+An example of a failure:
+    #+begin_src bash
+      sleepdd 1; false
+    #+end_src
+
+    #+RESULTS:
+
+** On top of ob-shell
+   :PROPERTIES:
+    :header-args:bash:  :execute-with my-org-babel :nasync yes
+    :header-args:bash+: :session sh-async
+   :END:
+
+   #+begin_src bash
+   sleep 1; date
+   #+end_src
+
+   #+RESULTS:
+   : Wed Feb 21 15:56:42 CET 2024
+
+
+   #+begin_src bash :results output :session *test* :nasync yes
+   cd /tmp
+   echo "hello world"
+   #+end_src
+
+   #+RESULTS:
+   : hello world
+
+   #+begin_src bash :results output
+   # comment
+   # comment
+   #+end_src
+
+   #+RESULTS:
+
+    #+begin_src bash :results output
+    # print message
+    echo \"hello world\"
+    #+end_src
+
+    #+RESULTS:
+    : "hello world"
+
+    #+begin_src bash :results output
+    echo "hello"
+    echo "world"
+    #+end_src
+
+    #+RESULTS:
+    : hello
+    : world
+
+
+    #+begin_src bash :results output
+      echo PID: "$$"
+    #+end_src
+
+    #+RESULTS:
+    : PID: 22212
+
+    #+begin_src bash :results output
+      echo PID: "$$"
+    #+end_src
+
+    #+RESULTS:
+    : PID: 22212
+
+
+    #+begin_src bash :results output :session shared
+      echo PID: "$$"
+      X=5
+    #+end_src
+
+    #+RESULTS:
+    : PID: 22218
+
+    #+begin_src bash :results output :session shared
+      echo PID: "$$"
+      echo X was set to "$X"
+    #+end_src
+
+    #+RESULTS:
+    : PID: 22218
+    : X was set to 5
+
+    #+begin_src bash :nasync yes :results value scalar
+    echo "Execute session blocks in the background"
+    sleep 3
+    echo "Using the :async header"
+    #+end_src
+
+    #+RESULTS:
+    : Execute session blocks in the background
+    : Using the :async header
+    : 0
+
+
+
+    #+name: their-os
+    Linux
+
+
+    #+begin_src bash :results output :shebang #!/usr/bin/env bash :stdin their-os :cmdline RMS :tangle ask_for_os.sh
+
+      # call as ./ask_for_os.sh NAME, where NAME is who to ask
+
+      if [ -z "$1" ]; then
+          asked="$USER"
+      else
+          asked="$1"
+      fi
+
+      echo Hi, "$asked"! What operating system are you using?
+      read my_os
+
+      if [ "$asked" = "RMS" ]; then
+          echo You\'re using GNU/"$my_os"!
+      elif [ "$asked" = "Linus" ]; then
+          echo You\'re using "$my_os"!
+      else
+          echo You\'re using `uname -o`!
+      fi
+    #+end_src
+
+    #+RESULTS:
+    : Hi, RMS! What operating system are you using?
+    : You're using GNU/Linux!
+
+
+
+    #+begin_src bash
+    declare -a array
+
+    m=4
+    n=3
+    for ((i=0; i<m; i++))
+    do
+        for ((j=0; j<n; j++))
+        do
+            a[${i},${j}]=$RANDOM
+        done
+    done
+    for ((i=0; i<m; i++))
+    do
+        for ((j=0; j<n; j++))
+        do
+            echo -ne "${a[${i},${j}]}\t"
+        done
+        echo
+    done
+    #+end_src
+
+    #+RESULTS:
+    | 7592 | 13920 | 4911 |
+    | 7592 | 13920 | 4911 |
+    | 7592 | 13920 | 4911 |
+    | 7592 | 13920 | 4911 |
+
+
+    #+begin_src bash :results list
+    declare -a array
+
+    m=4
+    n=3
+    for ((i=0; i<m; i++))
+    do
+        for ((j=0; j<n; j++))
+        do
+            a[${i},${j}]=$RANDOM
+        done
+    done
+    for ((i=0; i<m; i++))
+    do
+        for ((j=0; j<n; j++))
+        do
+            echo -ne "${a[${i},${j}]}\t"
+        done
+        echo
+    done
+    #+end_src
+
+    #+RESULTS:
+    - 17593
+      384
+      1439
+    - 17593
+      384
+      1439
+    - 17593
+      384
+      1439
+    - 17593
+      384
+      1439
+
+    #+begin_src bash :results file :file my_output.txt
+    declare -a array
+
+    m=4
+    n=3
+    for ((i=0; i<m; i++))
+    do
+        for ((j=0; j<n; j++))
+        do
+            a[${i},${j}]=$RANDOM
+        done
+    done
+    for ((i=0; i<m; i++))
+    do
+        for ((j=0; j<n; j++))
+        do
+            echo -ne "${a[${i},${j}]}\t"
+        done
+        echo
+    done
+    #+end_src
+
+    #+RESULTS:
+    [[file:my_output.txt]]
+
+
+    #+begin_src bash :results output
+      cat my_output.txt
+    #+end_src
+
+    #+RESULTS:
+    : 30132	16194	19934	
+    : 30132	16194	19934	
+    : 30132	16194	19934	
+    : 30132	16194	19934
+
+
+    #+begin_src bash :results output :dir /ssh:phone: :session none
+    if [ ! -e  "foo_file" ];
+    then
+        echo "foo" > foo_file
+        echo "Created foo_file"
+    else
+        echo "foo_file already exists!"
+    fi
+    #+end_src
+
+    #+RESULTS:
+    : foo_file already exists!
+
+
+    #+begin_src bash :results output :dir /ssh:phone: :session *remote*
+    if [ ! -e  "foo_file" ];
+    then
+        echo "foo" > foo_file
+        echo "Created foo_file"
+    else
+        echo "foo_file already exists!"
+    fi
+    #+end_src
+
+    #+RESULTS:
+    : Created foo_file
+
+
+    #+RESULTS:
+
+
+    #+begin_src bash :results none :session *my-session*
+      X=1
+    #+end_src
+
+    #+RESULTS:
+
+    #+begin_src bash :results output :session *my-session*
+      echo X was set to "$X"
+    #+end_src
+
+    #+RESULTS:
+    : X was set to 1
+
+    #+begin_src bash :results output :session *another-session*
+      echo X was set to "$X"
+    #+end_src
+
+    #+RESULTS:
+    : X was set to
+
+    #+RESULTS:
+
+
+    #+begin_src bash :results output
+    echo "Hello, world!"
+    sleep 3
+    echo "Good-bye, cruel World..."
+    #+end_src
+
+    #+RESULTS:
+    : Hello, world!
+    : Good-bye, cruel World...
+
+
+
+    #+begin_src bash :var by_two=0  x=3 :session none
+      if [ "$by_two" = "0" ]; then
+        echo $(($x * 2))
+      else
+        echo $(($x * 3))
+      fi
+    #+end_src
+
+    #+RESULTS:
+    : 6
+
+
+    #+begin_src bash :results output :var arr='("apple" "banana" "cherry")
+      echo The first element is...
+      echo \"${arr[1]}\"
+    #+end_src
+
+    #+RESULTS:
+    : The first element is...
+    : "banana"
+
+*** TODO Doesn't work yet: asynchronous that depends from other blocks
+    #+name: multiply_by_2
+    #+begin_src bash :var data="" :results output
+      echo $(($data * 2))
+    #+end_src
+
+    #+RESULTS: multiply_by_2
+    : bash: * 2: syntax error: operand expected (error token is "* 2")
+
+    #+begin_src bash :post multiply_by_2(data=*this*)
+      echo 3
+    #+end_src
+
+    #+results:
+
+
+* On top of ob-python
+   :PROPERTIES:
+    :header-args:python:  :execute-with my-org-babel :nasync yes
+    :header-args:python+: :session py-async
+   :END:
+
+Used =header-args= properties:
+   - =:execute-with my-org-babel=:  look for functions with the prefix `my-org-babel' to execute
+     blocks (for the asynchronous case use
+     `my-org-babel-schedule', and, for the synchronous case
+     `my-org-babel-execute'). These functions are defined in [[file:my-async-tests.el]].
+
+   - =:nasync yes=: by default, execute asynchronously (use `my-org-babel-schedule').
+
+   - =:session py-async= by default, use a session named "py-async".
+
+** basic examples
+*** async with a session
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+    #+RESULTS:
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708515666.8s | 1708515667.8s | 1.0s |
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+*** async with no session
+   :PROPERTIES:
+    :header-args:python+: :session none
+   :END:
+
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+    #+RESULTS:
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708083470.9s | 1708083471.9s | 1.0s |
+
+Yes, it failed, as expected. "import time" was done in its own
+temporary session.  The old result is preserved; the error is display
+as an overlay. Click on it to get more info about the error.
+
+
+Let's fix it, adding the import line:
+    #+begin_src python
+    import time
+    start = time.time()
+    time.sleep(1)
+    end = time.time()
+    ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708515915.0s | 1708515916.0s | 1.0s |
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+
+*** sync with a session
+   :PROPERTIES:
+    :header-args:python+: :session py-sync-session :nasync no
+   :END:
+
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708102997.5s | 1708102998.5s | 1.0s |
+
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+*** sync with no session
+   :PROPERTIES:
+    :header-args:python+: :session none :nasync no
+   :END:
+
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708083470.9s | 1708083471.9s | 1.0s |
+
+Yes, that fails (no session), displaying the details in a popup. Let's
+fix it:
+    #+begin_src python
+    import time
+    start = time.time()
+    time.sleep(1)
+    end = time.time()
+    ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708103039.0s | 1708103040.0s | 1.0s |
+
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+** worg examples
+
+Let's import matplotlib in our session.
+
+    #+begin_src python
+    import matplotlib
+    import matplotlib.pyplot as plt
+    #+end_src
+
+    #+RESULTS:
+    : None
+
+A figure in a PDF, asynchronous case.
+    #+begin_src python :results file link
+    fig=plt.figure(figsize=(3,2))
+    plt.plot([1,3,2])
+    fig.tight_layout()
+
+    fname = 'myfig-async.pdf'
+    plt.savefig(fname)
+    fname # return this to org-mode
+    #+end_src
+
+    #+RESULTS:
+    [[file:myfig-async.pdf]]
+
+
+A figure in a PDF, synchronous case.
+    #+begin_src python :results file link :nasync no
+    fig=plt.figure(figsize=(3,2))
+    plt.plot([1,3,2])
+    fig.tight_layout()
+
+    fname = 'myfig-sync.pdf'
+    plt.savefig(fname)
+    fname # return this to org-mode
+    #+end_src
+
+    #+RESULTS:
+    [[file:myfig-sync.pdf]]
+
+
+
+A PNG figure, asynchronous case.
+    #+begin_src python :results graphics file output :file boxplot.png
+      fig=plt.figure(figsize=(3,2))
+      plt.plot([1,3,2])
+      fig.tight_layout()
+      fig
+    #+end_src
+
+    #+RESULTS:
+    [[file:boxplot.png]]
+
+Same, but using the =:return= keyword.
+    #+begin_src python :return "plt.gcf()" :results graphics file output :file boxplot.png
+      fig=plt.figure(figsize=(3,2))
+      plt.plot([1,3,2])
+      fig.tight_layout()
+    #+end_src
+
+    #+RESULTS:
+    [[file:boxplot.png]]
+
+Same, asynchronous but without a session this time.
+    #+begin_src python :return "plt.gcf()" :results graphics file output :file boxplot-no-sess-a-y.png :session none
+      import matplotlib
+      import matplotlib.pyplot as plt
+      fig=plt.figure(figsize=(3,2))
+      plt.plot([1,3,2])
+      fig.tight_layout()
+    #+end_src
+
+    #+RESULTS:
+    [[file:boxplot-no-sess-a-y.png]]
+
+
+Lists are table,
+    #+begin_src python
+      [1,2,3]
+    #+end_src
+
+    #+RESULTS:
+    | 1 | 2 | 3 |
+
+unless requested otherwise.
+    #+begin_src python :results verbatim
+      [1,2,3]
+    #+end_src
+
+    #+RESULTS:
+    : [1, 2, 3]
+
+
+Dictionaries are tables too.
+    #+begin_src python :results table
+      {"a": 1, "b": 2}
+    #+end_src
+
+    #+RESULTS:
+    | a | 1 |
+    | b | 2 |
+
+
+Let's try the example with Panda.
+    #+begin_src python :results none
+      import pandas as pd
+      import numpy as np
+    #+end_src
+
+    #+RESULTS:
+    : None
+
+    #+begin_src python :results table
+      pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                   columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+And the synchronous case?
+
+    #+begin_src python :results table  :nasync no
+      pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                   columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+
+
+Without session ?
+
+    #+begin_src python :results table :session none
+    pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                 columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+Right, we need to import the libraries (no session).
+
+    #+begin_src python :results table :session none
+    import pandas as pd
+    import numpy as np
+    pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                 columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+
+** inline examples
+
+    A simple asynchronous inline src_python{3*2} {{{results(=6=)}}}.
+
+    An other one containing a mistake src_python{2/0} {{{results(=6=)}}}
+    (click on the error to see the details).
+
+
+    Some very slow inline asynchronous computations that all run in
+    the same session. You need to execute the 3 of them at once. Here
+    is the first one src_python[:return "\"OK1\""]{import time;
+    time.sleep(5)} {{{results(=OK1=)}}} and a second one
+    src_python[:return "\"OK1 bis\""]{import time; time.sleep(5)}
+    {{{results(=OK1 bis=)}}} and the third one src_python[:return
+    "\"OK2\""]{import time; time.sleep(5)} {{{results(=OK2=)}}}.
+
+    Yes, the previous paragraph is unreadable; it's on purpose, to
+    check that ob-core can figure it out.
+
+    Let's repeat, in a more readable way, and making the last one
+    synchronous.
+
+    Some very slow inline computations that all run in the same
+    session. Here is the first asynchronous one
+          src_python[:return"\"OK1\""]{import time; time.sleep(5)} {{{results(=None=)}}}
+    and a second one, asynchronous too:
+          src_python[:return "\"OK1 bis\""]{import time; time.sleep(5)} {{{results(=OK1 bis=)}}}
+    and finally, a third one, synchronous this one:
+          src_python[:nasync no :return "\"OK2\""]{import time; time.sleep(5)} {{{results(=OK2=)}}}.
+
+    Note that, once the user executes the last synchronous block, the
+    user is blocked until the synchronous execution can start
+    (i.e. all previous asynchronous executions are done) and until
+    it's done.  The display is updated though, to see the asynchronous
+    progress.
-- 
2.43.0

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

> Hi Matt, Jack, Ihor,
>
> Sorry for the late reply.  Cleaning the code took me longer than
> expected.

Thanks for the code!
It is a lot more that I expected.
I have many questions ;)

> The new API itself is more about how to wait for and display one block
> result.  So, it's not really aware of sessions.

I think that it is important to think about sessions. For non-sessions,
we may execute the code in parallel, disregarding already running
execution. In contrast, for sessions, we need to maintain some queue and
wait until previous execution finishes before running next (if multiple
session blocks are executed in quick succession).

It may also be necessary to provide an UI listing the queued and running
execution. Users should be able to stop "stale" processes if they are
defunc (consider infinite loop).

>> Also interesting, I think it's worth exploring/testing this overlay idea
>> out. Does that mean that output is asynchronously printing into the Org
>> buffer? It sounds cool but I wonder if it might cause problems while
>> trying to edit another part of the buffer.
>
> Currently, I've limited the progress feedback to fit on one line to
> avoid anoying vertical display jumps.  When the computation is
> successful, the result is inserted as usual, and, that may be annoying;
> when it updates a previous result of the same height, it's ok.  Else, we
> could fold it to stay on one line too (using the overlay), until the
> user explicitly request to see it.

Let's not worry about "jumps" yet. It is a minor issue that can be
easily solved.

What is more important is when users, for example, remove the whole
subtree containing the place where the execution result is to be
inserted. Or when users perform edits at or around that place where the
result is to be inserted. Or consider subtree with pending result
refiled elsewhere (different file or different place in the same file);
or archived.

Or consider user running an src block, quickly editing it, and then
running again. What should we do then? Should we stop the first running
process?

> So, here we go.  You'll find attach a set of patchs.  It works for me with
> Emacsc 30.50 and 9.7-pre (from today).

I have several general questions:

- what if append/prepend to result asynchronously?
- what if we replace the result, call async evaluation two times, and they arrive in opposite order (first called is calculated after the second)?
- on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications?
  Example: try running
  #+begin_src bash
  idontexist
  #+end_src

> I didn't check yet that the code and the commits follow all the
> guidelines.  This is just a preliminary version for feedbacks.
> Corrections/critiques are welcome, but don't check the details until I
> check them myself.
> So, I decided to rewrite the whole thing, taking code from the
> synchronous case (following `org-babel-python-evaluate-session').  I
> also created a package that contains all the functions that should be
> reusable for any language.  The patch [1] adds some generic functions to
> help dealing with asynchronicity (process, comint, etc.). The patch [2]
> shows a new possible way to execute python code blocks, both
> synchronously and asynchronously, with or without sessions.  You should
> just need to open [3] and follow what's written there, and execute the
> existing bash and python code blocks.

Note that we already have a WIP an asynchronous API in the works.
Check out `org-async-call' in
https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377
If possible, we should reuse it.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Matt]

 ---- On Mon, 19 Feb 2024 01:31:42 +0100  Jack Kamm  wrote --- 

 > I don't remember the details, but my past self [1] thought it would be
 > good to find a way to replace `process-file' with `make-process' in
 > `org-babel--shell-command-on-region' or `org-babel-eval', and it seems
 > you are thinking along those lines with `my-org-babel-eval-async'. Hope
 > you're able to make progress on this and get the improvements into
 > ob-eval.el eventually.
 > 
 > [1] https://list.orgmode.org/871rczg7bi.fsf@gmail.com/#t

AFAIK, aside from appending "&", =make-process= is the only way.  It would probably make sense to continue using =shell= though.  If I understand correctly, you (and others) jump between the Org buffer block and the comint buffer?

 ---- On Mon, 19 Feb 2024 10:03:36 +0100  Ihor Radchenko  wrote --- 

 > Note that we already have a WIP an asynchronous API in the works.
 > Check out `org-async-call' in
 > https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377
 > If possible, we should reuse it.

:O

- What's the status? 
- Anything I could help with?  
- Are there any notes or supplements that go along with it?

> I have several general questions:
>
> - what if append/prepend to result asynchronously?
> - what if we replace the result, call async evaluation two times, and they arrive in opposite order (first called is calculated after the second)?
> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications?
> Example: try running
> #+begin_src bash
> idontexist
> #+end_src

Are these open questions for the `org-async-call' implementation?

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Ihor Radchenko]

Matt <matt@excalamus.com> writes:

>  > Note that we already have a WIP an asynchronous API in the works.
>  > Check out `org-async-call' in
>  > https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377
>  > If possible, we should reuse it.
>
> :O
>
> - What's the status? 

Finalizing.

> - Anything I could help with?

Check https://github.com/tecosaur/org-latex-preview-todos/issues/

> - Are there any notes or supplements that go along with it?

Docstring should detail the API.

>> I have several general questions:
>>
>> - what if append/prepend to result asynchronously?
>> - what if we replace the result, call async evaluation two times, and they arrive in opposite order (first called is calculated after the second)?
>> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications?
>> Example: try running
>> #+begin_src bash
>> idontexist
>> #+end_src
>
> Are these open questions for the `org-async-call' implementation?

No. These are about the patch. And have nothing to do with
`org-async-call', which is just a way to call and queue async processes.
`org-async-call' was written for async latex preview, but its interface
should be generic enough to be reused for babel.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Ihor Radchenko]

Matt <matt@excalamus.com> writes:

> - Anything I could help with?  

https://github.com/tecosaur/org-latex-preview-todos/issues/21
It looks like the main remaining blocker is the lack of LaTeX snippet
export for ox-odt.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Jack Kamm]

Matt <matt@excalamus.com> writes:

> AFAIK, aside from appending "&", =make-process= is the only way.  It would probably make sense to continue using =shell= though.  If I understand correctly, you (and others) jump between the Org buffer block and the comint buffer?

Yes, I typically use ob-R and ob-python in this way. Aside from the
convenience of interacting directly with the REPL, the comint mode may
provide other useful facilities, for example in ob-R the inferior ESS
mode provides completion that is aware of the objects in the current
session.

However I think modifying `org-babel-eval' will mainly be useful to
provide async to the nonsession users. Comint-based sessions mainly use
functionality from ob-comint.el, not ob-eval.el, and I think that the
use cases are different enough that they shouldn't be merged into a
single implementation.

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

>
> Thanks for the code!
> It is a lot more that I expected.

Note that only the first 5 patchs are real patchs.  The remaining things
are just a demo how it could be used.  The current async (ob-comint)
depends on writing UUIDs in org files, and, that's why I couldn't use it
as a demo.  I'm thinking about dropping the patch:

   'ob-core async: Add org-babel--async tools [2/5]'

and just provide an other new keyword (feedbacks-with) so that anybody
may plug its own feedback process.

> I have many questions ;)

Thanks! They are welcome :-)


>> The new API itself is more about how to wait for and display one block
>> result.  So, it's not really aware of sessions.
>
> I think that it is important to think about sessions. For non-sessions,
> we may execute the code in parallel, disregarding already running
> execution. In contrast, for sessions, we need to maintain some queue and
> wait until previous execution finishes before running next (if multiple
> session blocks are executed in quick succession).

I agree.  My proposed patch leaves this open to further work.

My proposed patch allows to have one way, in ob-core, to display
progress of asynchronous executions: like not started, 10%, success,
failed; and a way to plug in asynchronous executions in ob-core.

I've included some demo code just to show how that API could be used.


> It may also be necessary to provide an UI listing the queued and running
> execution. Users should be able to stop "stale" processes if they are
> defunc (consider infinite loop).

That looks nice but that may be too limiting.  Like it will probably forbid
to execute something asynchronously using elisp threads.  Anyway, that
outside the intended scope of my set of patchs.


> What is more important is when users, for example, remove the whole
> subtree containing the place where the execution result is to be
> inserted. Or when users perform edits at or around that place where the
> result is to be inserted. Or consider subtree with pending result
> refiled elsewhere (different file or different place in the same file);
> or archived.

My patch definitely needs to address those, even if only by raising an
error.

> Or consider user running an src block, quickly editing it, and then
> running again. What should we do then? Should we stop the first running
> process?

To keep things simple and generic, I would just forbid the user to
reschedule the task until the previous outcome is available.


> I have several general questions:
>
> - what if append/prepend to result asynchronously?

You mean if org is executing several times the same code concurrently?
I think we should forbid it.

> - what if we replace the result, call async evaluation two times, and they arrive in opposite order (first called is calculated after the second)?

"One execution at a given time" will solve this too :-)

> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications?

In the asynchronous case, I've decided to leave the overlay in place
in case of errors, with the error description.  Clicking on that error
pops up the same kind of popups as in the synchronous case.  You can
give it a try using examples of errors my demo file:

   scratch/bba-ob-core-async/my-async-tests.org

(Please use the updated set of patchs that I've sent today, else, you
will have to fix the required libraries).


> Note that we already have a WIP an asynchronous API in the works.
> Check out `org-async-call' in
> https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377
> If possible, we should reuse it.

Thanks. I will have a look at it.


Bruno

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

> Note that only the first 5 patchs are real patchs.  The remaining things
> are just a demo how it could be used.  The current async (ob-comint)
> depends on writing UUIDs in org files, and, that's why I couldn't use it
> as a demo.  I'm thinking about dropping the patch:
>
>    'ob-core async: Add org-babel--async tools [2/5]'
>
> and just provide an other new keyword (feedbacks-with) so that anybody
> may plug its own feedback process.

May you please clarify if adding the new code block parameter that
defines custom execute function is something you want to add to Org mode
or just a helper code to demo you main patch?

>> Or consider user running an src block, quickly editing it, and then
>> running again. What should we do then? Should we stop the first running
>> process?
>
> To keep things simple and generic, I would just forbid the user to
> reschedule the task until the previous outcome is available.

This may not be as trivial as one might think.
Consider

#+name: foo
#+begin_src bash :async yes
...
#+end_src

#+results: foo
<pending results>

Another code block with the same name will write results under
#+results: foo
#+name: foo
#+begin_src bash :async yes
...
#+end_src

We can currently have multiple code blocks writing to the same place.
And async execution should not only check is the current src block is
scheduled, but also whether we are attempting to write to a place where
no other running block is scheduled to write.

>> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications?
>
> In the asynchronous case, I've decided to leave the overlay in place
> in case of errors, with the error description.  Clicking on that error
> pops up the same kind of popups as in the synchronous case.

Error buffer does not necessarily appear on failure. When the code leads
to process writing to stderr, we also display error buffer. Even if the
process exits with 0 code.


Also, your code currently creates overlays unconditionally.
However, when we have :results silent or :results none, Org babel must
not modify buffer.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Bruno Barbier]

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> Note that only the first 5 patchs are real patchs.  The remaining things
>> are just a demo how it could be used.  The current async (ob-comint)
>> depends on writing UUIDs in org files, and, that's why I couldn't use it
>> as a demo.  I'm thinking about dropping the patch:
>>
>>    'ob-core async: Add org-babel--async tools [2/5]'
>>
>> and just provide an other new keyword (feedbacks-with) so that anybody
>> may plug its own feedback process.
>
> May you please clarify if adding the new code block parameter that
> defines custom execute function is something you want to add to Org mode
> or just a helper code to demo you main patch?

Yes. (I thought about adding it in a separate request;  but that would be
simpler to include it.)


>>> Or consider user running an src block, quickly editing it, and then
>>> running again. What should we do then? Should we stop the first running
>>> process?
>>
>> To keep things simple and generic, I would just forbid the user to
>> reschedule the task until the previous outcome is available.
>
> This may not be as trivial as one might think.
> Consider
>
> #+name: foo
> #+begin_src bash :async yes
> ...
> #+end_src
>
> #+results: foo
> <pending results>
>
> Another code block with the same name will write results under
> #+results: foo
> #+name: foo
> #+begin_src bash :async yes
> ...
> #+end_src

> We can currently have multiple code blocks writing to the same place.
> And async execution should not only check is the current src block is
> scheduled, but also whether we are attempting to write to a place where
> no other running block is scheduled to write.

Today, the asynchronous execution is attached to the result; the source
code itself is not locked and stays editable.

With the current implementation, trying your example, the second execution fails with:
    Error running timer: (user-error "Cannot modify an area being
    updated")

So, it's already OK I guess.

I should improve the behavior (to fail before launching the second
execution).  But, the current design should work in that case.


>
>>> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications?
>>
>> In the asynchronous case, I've decided to leave the overlay in place
>> in case of errors, with the error description.  Clicking on that error
>> pops up the same kind of popups as in the synchronous case.
>
> Error buffer does not necessarily appear on failure. When the code leads
> to process writing to stderr, we also display error buffer. Even if the
> process exits with 0 code.

Got it.  The current design adds the popup on failure; I should make it
more flexible to allow to configure a popup on success too.


> Also, your code currently creates overlays unconditionally.
> However, when we have :results silent or :results none, Org babel must
> not modify buffer.

Yes. I'll fix it (and that one was in my TODO list :-))

But if the execution fails, I guess I'll need to provide some feedbacks
anyway, but I don't know yet how (asynchronous popups are not an option,
modifying the file neither).

Thanks for your review and questions,

Bruno


>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

>> May you please clarify if adding the new code block parameter that
>> defines custom execute function is something you want to add to Org mode
>> or just a helper code to demo you main patch?
>
> Yes. (I thought about adding it in a separate request;  but that would be
> simpler to include it.)

I have doubts about how useful such parameter would be for _users_. It
is certainly useful for developers of new babel backends, but I do not
see how users will use it.

I'd prefer to discuss this in a separate thread.

>> Error buffer does not necessarily appear on failure. When the code leads
>> to process writing to stderr, we also display error buffer. Even if the
>> process exits with 0 code.
>
> Got it.  The current design adds the popup on failure; I should make it
> more flexible to allow to configure a popup on success too.

By popup, do you mean an overlay where clicking on it will raise the
usual `org-babel-eval-error-notify' buffer?
If yes, what happens if a user executes a code block, it fails, and the
user executes it again without looking at the error? (Second execution
may be triggered from a different place, indirectly, via noweb ref or
similar).

> But if the execution fails, I guess I'll need to provide some feedbacks
> anyway, but I don't know yet how (asynchronous popups are not an option,
> modifying the file neither).

We need something more than just an overlay. Maybe some kind of babel
execute history (in tabulated-list-mode buffer), keeping information
about execution stats and stderr data.

We may provide a mode-line or fringe indicator that will warn user if
something went wrong with recent executions. Something akin compile-mode
mode line indicator listing errors and warnings during compilation.

(Handling asynchronous messages is actually a complex topic. It has been
previously discussed on emacs-devel as well. See
<https://yhetil.org/emacs-devel/838t59j821.fsf@gnu.org/>)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Bruno Barbier]

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>>> May you please clarify if adding the new code block parameter that
>>> defines custom execute function is something you want to add to Org mode
>>> or just a helper code to demo you main patch?
>>
>> Yes. (I thought about adding it in a separate request;  but that would be
>> simpler to include it.)
>
> I have doubts about how useful such parameter would be for _users_. It
> is certainly useful for developers of new babel backends, but I do not
> see how users will use it.
>
> I'd prefer to discuss this in a separate thread.

Sure. I'll keep it for an other thread.


>>> Error buffer does not necessarily appear on failure. When the code leads
>>> to process writing to stderr, we also display error buffer. Even if the
>>> process exits with 0 code.
>>
>> Got it.  The current design adds the popup on failure; I should make it
>> more flexible to allow to configure a popup on success too.
>
> By popup, do you mean an overlay where clicking on it will raise the
> usual `org-babel-eval-error-notify' buffer?
> If yes, what happens if a user executes a code block, it fails, and the
> user executes it again without looking at the error? (Second execution
> may be triggered from a different place, indirectly, via noweb ref or
> similar).

Current behavior is to remove the error, and, then re-execute: the
previous errors are gone.  My guess is that this could be improve later,
without breaking the API.


>> But if the execution fails, I guess I'll need to provide some feedbacks
>> anyway, but I don't know yet how (asynchronous popups are not an option,
>> modifying the file neither).
>
> We need something more than just an overlay. Maybe some kind of babel
> execute history (in tabulated-list-mode buffer), keeping information
> about execution stats and stderr data.
>
> We may provide a mode-line or fringe indicator that will warn user if
> something went wrong with recent executions. Something akin compile-mode
> mode line indicator listing errors and warnings during compilation.

I was thinking about using a fringe indicator, for warnings on success (as I'm
removing all overlays on success).

Keeping a buffer of all asynchronous executions looks like a good idea.
I'll think about it.


> (Handling asynchronous messages is actually a complex topic. It has been
> previously discussed on emacs-devel as well. See
> <https://yhetil.org/emacs-devel/838t59j821.fsf@gnu.org/>)

I'll look at this.  Thanks!

I'll publish a branch soon; it will be a major rewrite of my current
proposal.  It should be less confusing and, I hope, address some of your
comments.

Thanks again for your questions and feedbacks,

Bruno

>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Matt]

 ---- On Sat, 24 Feb 2024 17:42:54 +0100  Bruno Barbier 

 > I'll publish a branch soon; it will be a major rewrite of my current
 > proposal.  It should be less confusing and, I hope, address some of your
 > comments.
 > 
 > Thanks again for your questions and feedbacks,

You're welcome and thanks for sharing your ideas.

Any lack of comments from me recently is just limited time and trying to focus on maintenance.

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Bruno Barbier]

Hi Matt,

Matt <matt@excalamus.com> writes:

>  ---- On Sat, 24 Feb 2024 17:42:54 +0100  Bruno Barbier 
>
>  > I'll publish a branch soon; it will be a major rewrite of my current
>  > proposal.  It should be less confusing and, I hope, address some of your
>  > comments.
>  > 
>  > Thanks again for your questions and feedbacks,
>
> You're welcome and thanks for sharing your ideas.
>
> Any lack of comments from me recently is just limited time and trying to focus on maintenance.

Thanks for letting me know.

No worries. Take care.

Bruno

Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Jack Kamm]

Ihor Radchenko <yantar92@posteo.net> writes:

>> I agree that it would be good to redesign it, but am not sure where to
>> start.
>
> For example,
>
> 1. Change `org-babel-comint-async-register' to return UUID and to store
>    PARAMS as passed by the backend (current approach with PARAMS being
>    derived from src blocks prevents backends to transform src block
>    PARAMS dynamically).
> 2. Change `org-babel-insert-result' to handle :async t specially,
>    inserting something reliable, like #+async: <UUID> in place of result
>    without performing extra transformations.
> 3. Change `org-babel-insert-result' to accept an internal parameter
>    that will make it replace #+async: <UUID> keyword rather than perform
>    normal result insertion.
> 4. Change `org-babel-comint-async-filter' to use the previously passed
>    PARAMS, remove :async t from them, and arrange the call to
>    `org-babel-insert-result' to replace the #+async: <UUID> keyword.

That all sounds reasonable...if you work on this, let me know if you
want any help with testing.