From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id MBG2HJJK1mVRrgAAe85BDQ:P1 (envelope-from ) for ; Wed, 21 Feb 2024 20:10:10 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id MBG2HJJK1mVRrgAAe85BDQ (envelope-from ) for ; Wed, 21 Feb 2024 20:10:10 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=QqTPHPhz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1708531458; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=QDp8gTYktAECKlLXGTUTVBE5hVoGky9yU51d9LP1ZIg=; b=f8WoRipfIZtsXa40Q8H7RmhZ2sQ7IOQ5oceZ/Vep/JcKn9fhBUnjff4hY8oy2Mdyxac7zg 4nhrc8IY8+yD2yksLhFTAeKSyMYGlE5ADFpph0e3h1Djoj3mSw0dYRv/i2CmffX/81sfT1 i7g4msmcEOlC0HtODS8jYExc9mr0J7HcDLBkZWHa5rZUAyrntRzqkN7hjl10gKSdGl5opG FqY3hUA+C+t81lnCPKMrCuwCvGoKyUPe61yDPJMYIOyLFinMCX7cspXml96MpGa4La3C6k /kFZKAcipUzNTCnuxB5imqHQdgTJnkscTYGEloorBe+LHGIBCYWv9tSs+5QBnw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=QqTPHPhz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1708531458; a=rsa-sha256; cv=none; b=Mi/qZ2deskNqeArm/8tOOxts6LBUuAT2OZnDFmWxiCb8QKxpNuEEfJCk6f6pGxUEqh/1bi u0vNhZ4fOfWAk4sW5UFeKWlvnXZv2b3xR8PMXlkbwXRj4BW0Bd/c+I95OIIGtyCeSXxGz5 EkvauaQBczoU+7A3FML/1Jmdf+l1UvBzrgyBbgQ9cxBPZsKEfT//RCZPQsSBcR8eilWbXt jaMeq0VhkR3c7zu8DARIbh+r29PYJ2U3ZCnRYliA01H918GU1l4F9BY8T/jvENEWOz+Bap 7BFPoXUKq/9YgEesT9HTlF3mnrCPFBNI+IKrZErbFZKGrDgKDAQ1kVPYHggO+A== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C110D403C3 for ; Wed, 21 Feb 2024 17:04:18 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rcp4B-0000Sy-Bc; Wed, 21 Feb 2024 11:03:27 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rcp47-0000RW-I8 for emacs-orgmode@gnu.org; Wed, 21 Feb 2024 11:03:24 -0500 Received: from mail-wm1-x32b.google.com ([2a00:1450:4864:20::32b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rcp45-00078V-L5 for emacs-orgmode@gnu.org; Wed, 21 Feb 2024 11:03:23 -0500 Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-41275d2edbcso6130225e9.2 for ; Wed, 21 Feb 2024 08:03:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708531400; x=1709136200; darn=gnu.org; h=mime-version:date:references:in-reply-to:subject:cc:to:from :message-id:from:to:cc:subject:date:message-id:reply-to; bh=QDp8gTYktAECKlLXGTUTVBE5hVoGky9yU51d9LP1ZIg=; b=QqTPHPhzo67vCpZpEr3xqIUjaEGtSXDNZ0+7ce9F2MBUHrRBfdZkWvylKSBT29dsWL UoKyKPvDKbJNNZmdFwDIwwoOMt3P6JEDjYTgW4aJQsbQF/j1Zmxsgv1RQlsj2fWtXm6E nrWERW1oIsowpNq7EEYGVl5ESWvuVzmKVzsUNsaQEkkLIceggRaKaSg1aNlMtFAvMEki DlITPlBJ/oc9KBVbVhOwjU4fEelphftrEMc3eY7owLCY9Ro/443BgJBO0IvCeeM/3Xhe p6XUOo8JrTmTUQ+iSqe9c84Q+hTNwqg15PHS4RCLkhpQ7XYn2ZxXCfjosuSMlZ/CE3lF CIMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708531400; x=1709136200; h=mime-version:date:references:in-reply-to:subject:cc:to:from :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QDp8gTYktAECKlLXGTUTVBE5hVoGky9yU51d9LP1ZIg=; b=oHAG8LA8kwXYVgajRPJk228hE2ZJZYIsazN+/jt2ONFw+EGFpEsP3yghjVlSdwaHaS ufNZxoxyFRj8/yrbOY36cNX93gIn31ETHEWKVjQ7FEYlxn4e1gbg4vv94ItMKiv8bChn ka1fdbgUjG9U6jYx90jQ2m53B1PtVyDHElQWCENSiPPDp7hzCxkLWWeNd5WqNhU5T7mM o5L/Y1e5wbn39bJyScq4mUcRxgiJjYDqtmHGxFEIJsomaSXsP4+ZG2nkWj3hLrKKfzmY dDr9xfeHc6kphWSRcAkLc1EPKXraD3LBF63XcQIHxrzV33ETiRpGqd2+cuV09sGso8wG U37w== X-Forwarded-Encrypted: i=1; AJvYcCWcFyNfvxVvFrttEwuNXAuUFxd+4/2t/OUHF2xbaVLjvUtltWnHoY69Bf4SgUWocpjoiPvqCbc265FyfYE2CIu4yzDlMPQ= X-Gm-Message-State: AOJu0Yw6d+tiEqlsziMcI8BIyLceq2Wp19LSCfOAJWx55bR5x70XDsbD JEEnz1ZbjJHyBBVx1MStlc7nYWn0EFUGE52hCNyBC0XwgpQO3CSx X-Google-Smtp-Source: AGHT+IHGhnaRoMVemMasIkrz/naWmNdR97UWXRAjue31XxUrMK9p++048jXNtlOlBWJGRUpQyJwE+g== X-Received: by 2002:a05:600c:1553:b0:412:a6b:f3a5 with SMTP id f19-20020a05600c155300b004120a6bf3a5mr13938162wmg.5.1708531399631; Wed, 21 Feb 2024 08:03:19 -0800 (PST) Received: from keynux ([2a01:e0a:505:3460:1c18:688d:ece4:372e]) by smtp.gmail.com with ESMTPSA id k25-20020a05600c0b5900b00410dd253008sm2941464wmr.42.2024.02.21.08.03.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 08:03:19 -0800 (PST) Message-ID: <65d61ec7.050a0220.999f7.a39b@mx.google.com> Received: by keynux (sSMTP sendmail emulation); Wed, 21 Feb 2024 17:03:17 +0100 From: Bruno Barbier To: Ihor Radchenko Cc: Matt , Jack Kamm , emacs-orgmode Subject: Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) In-Reply-To: <87frxohlgo.fsf@localhost> References: <87o7d0mm54.fsf@localhost> <65bbb108.050a0220.b60fd.6790@mx.google.com> <87jznm8hcu.fsf@gmail.com> <875xz42rp9.fsf@localhost> <874jen8zec.fsf@gmail.com> <87o7cv9e80.fsf@localhost> <65c2875f.050a0220.caf6d.8291@mx.google.com> <18dae5cab1d.bf1c7563863897.4896289306902277373@excalamus.com> <65cfa0d8.050a0220.cb569.ce34@mx.google.com> <87frxohlgo.fsf@localhost> Date: Wed, 21 Feb 2024 17:03:17 +0100 MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2a00:1450:4864:20::32b; envelope-from=brubar.cs@gmail.com; helo=mail-wm1-x32b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -8.00 X-Spam-Score: -8.00 X-Migadu-Queue-Id: C110D403C3 X-Migadu-Scanner: mx13.migadu.com X-TUID: vfgKGaoOUKq/ Hi Ihor, Ihor Radchenko writes: > > Thanks for the code! > It is a lot more that I expected. Note that only the first 5 patchs are real patchs. The remaining things are just a demo how it could be used. The current async (ob-comint) depends on writing UUIDs in org files, and, that's why I couldn't use it as a demo. I'm thinking about dropping the patch: 'ob-core async: Add org-babel--async tools [2/5]' and just provide an other new keyword (feedbacks-with) so that anybody may plug its own feedback process. > I have many questions ;) Thanks! They are welcome :-) >> The new API itself is more about how to wait for and display one block >> result. So, it's not really aware of sessions. > > I think that it is important to think about sessions. For non-sessions, > we may execute the code in parallel, disregarding already running > execution. In contrast, for sessions, we need to maintain some queue and > wait until previous execution finishes before running next (if multiple > session blocks are executed in quick succession). I agree. My proposed patch leaves this open to further work. My proposed patch allows to have one way, in ob-core, to display progress of asynchronous executions: like not started, 10%, success, failed; and a way to plug in asynchronous executions in ob-core. I've included some demo code just to show how that API could be used. > It may also be necessary to provide an UI listing the queued and running > execution. Users should be able to stop "stale" processes if they are > defunc (consider infinite loop). That looks nice but that may be too limiting. Like it will probably forbid to execute something asynchronously using elisp threads. Anyway, that outside the intended scope of my set of patchs. > What is more important is when users, for example, remove the whole > subtree containing the place where the execution result is to be > inserted. Or when users perform edits at or around that place where the > result is to be inserted. Or consider subtree with pending result > refiled elsewhere (different file or different place in the same file); > or archived. My patch definitely needs to address those, even if only by raising an error. > Or consider user running an src block, quickly editing it, and then > running again. What should we do then? Should we stop the first running > process? To keep things simple and generic, I would just forbid the user to reschedule the task until the previous outcome is available. > I have several general questions: > > - what if append/prepend to result asynchronously? You mean if org is executing several times the same code concurrently? I think we should forbid it. > - what if we replace the result, call async evaluation two times, and they arrive in opposite order (first called is calculated after the second)? "One execution at a given time" will solve this too :-) > - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications? In the asynchronous case, I've decided to leave the overlay in place in case of errors, with the error description. Clicking on that error pops up the same kind of popups as in the synchronous case. You can give it a try using examples of errors my demo file: scratch/bba-ob-core-async/my-async-tests.org (Please use the updated set of patchs that I've sent today, else, you will have to fix the required libraries). > Note that we already have a WIP an asynchronous API in the works. > Check out `org-async-call' in > https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377 > If possible, we should reuse it. Thanks. I will have a look at it. Bruno