From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id GNEyER4H/GWIdwEAqHPOHw:P1 (envelope-from ) for ; Thu, 21 Mar 2024 11:08:30 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0.migadu.com with LMTPS id GNEyER4H/GWIdwEAqHPOHw (envelope-from ) for ; Thu, 21 Mar 2024 11:08:30 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=kGRcdKEX; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1711015709; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=i8hFp0IBmD0a9KN270WhkBOo/pwQB9CP54DGMsubDxo=; b=StOM3dEpOaQYhyoStO71MWZffCTbCRleSx00sE4LxXpmmcD4ZdYSd3nd2P32tTpiwdZvET mRuwU2+lGJMTdQOjBW8VSucCUEHj1xCv5anO2w/5dZlx4WxtNkmgp3GCOv1gIjhorWusWZ 0JjMStJ/LT162AKDe4Y8Qn935r4dkfH2Tjz6tPfEn0GH2gw81ta7Fo23pFapKgxdVe2RFv BgaMAwxtKXt+gX3hGlPBwFGJqOXKSsbQaJj6vuA0jb+bNsEfomgKCvqC0WqE1IucX/R/uA wzIy/TJ26VLmzb33j5q+BiKm5ebtLl5hKTSiEZJKiW74jKeDa2Y+vXTmh+OO9A== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=kGRcdKEX; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1711015709; a=rsa-sha256; cv=none; b=qxrpvHD0prLNQH6IXt2ZuQKaoHLKv8jqKjZy3vw10KerD1lZEz2Ew65fZBMYDeyzKwAA94 bfYOyY6Li5Kvd8zA7CwwKTYktxAp37usvBx6UNh1OSQ+y4pMHswY7J5bXk/8qZZjZecCWe 72J2BWeD92x2E4AOzvkxX6e80BJmxrsMTLl1qFonizxCGIW+gV8ETakwC2+0XtNd93Apon lHLUK4wyeOkRw1gkx5C4ea/+TtLe0winS+vAzxlmGXl3dLWCMeHUgPirhAAswlyesttvHQ +tv4ZaVxFubwovbZF/IIbWDriWrOeMPa/c7lC1d99V75WKWRczcDGDjIdTVqMQ== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B70DE129D8 for ; Thu, 21 Mar 2024 11:08:29 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rnFKe-0004Qa-0h; Thu, 21 Mar 2024 06:07:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rnFKF-0003kJ-BW for emacs-orgmode@gnu.org; Thu, 21 Mar 2024 06:07:10 -0400 Received: from mail-wm1-x335.google.com ([2a00:1450:4864:20::335]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rnFK8-0007yj-6Q for emacs-orgmode@gnu.org; Thu, 21 Mar 2024 06:07:07 -0400 Received: by mail-wm1-x335.google.com with SMTP id 5b1f17b1804b1-414701303f7so7491575e9.2 for ; Thu, 21 Mar 2024 03:06:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711015618; x=1711620418; darn=gnu.org; h=mime-version:date:references:in-reply-to:subject:cc:to:from :message-id:from:to:cc:subject:date:message-id:reply-to; bh=i8hFp0IBmD0a9KN270WhkBOo/pwQB9CP54DGMsubDxo=; b=kGRcdKEXVuVW2PnsOSO+t/z6j2Ufq0WznXzuP5GShotfiR2cOIO3GlLWBDb5FG4SFe 9zjQYzfrD7UIyXGSh1N8ybzPDPc0LRSNxyhTatSAP4g0k52jrfGM7vBhuG7HbV7iGtVf +v7sCnHLbnco7sKoxRiq099kxic7CeB9IkqB+nuBsRH+OQHpSz2NrMfR1gykeLN+3HNE omoIjTL4R19rjAbyD9uc7zTaxM2yUgQnfIJwH9IbwGKU5q62dsOFEnmfHUoU6OLKz0Q2 MLNjUtxHwO950XCRL/980Ok1AIeSAJ7jH6Rv5eYQhwal+BWoLs8BbBWVDsRXGpZZzvXN /Xzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711015618; x=1711620418; h=mime-version:date:references:in-reply-to:subject:cc:to:from :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=i8hFp0IBmD0a9KN270WhkBOo/pwQB9CP54DGMsubDxo=; b=ZqhvuAiYISx+CWOayzHy+N5MFPwA76u8LPg4Njz7xnz084/fHaQwYyknRlJNNbZ5kn pMb4/VxFYhiStAtKwDcrQN+9bOxp/cDLA23JVe3btq7rdOi6Zk/rzcXgV9wyVk0rpKZX P8uhYSsv9DWIhb0iMwLE87/tNucLJ0Jtb8UzZWvTzzhz5+pk21SmtfHGMXXKToVow2Ql kV9kMRQLWs8GEyyeh+2CxJctHgoJKfL1DmyaaauU36T8zttKOrMmNgkefkLJzW05SIb5 7TVMcz1C9vhskJBOSMMuS8D5xJ+LJt0U5OAHEM+zzckdA5jwHY5jlG6PTx7JDYrkzzJ2 cFFw== X-Gm-Message-State: AOJu0YyHfPPuWqjGqir+0IOhWvUZvlsU5Ap8mrpd5WphmmVHcjcs4szY pgT5DevypDkp0eIw0o+OI6ynxSJUJcDepzbErIAvnepQS9HqfAXq3OII19Gx X-Google-Smtp-Source: AGHT+IES3RqVPlmSNfh1+grPTSDTxkn+kCc9y0yAuWTcR9V6n+KGIRlYuL/Zk+aT9fRwxh5X1Ub82Q== X-Received: by 2002:a5d:4243:0:b0:33e:1a96:2be7 with SMTP id s3-20020a5d4243000000b0033e1a962be7mr1172577wrr.11.1711015617338; Thu, 21 Mar 2024 03:06:57 -0700 (PDT) Received: from keynux ([2a01:e0a:505:3460:1c18:688d:ece4:372e]) by smtp.gmail.com with ESMTPSA id n2-20020a5d4002000000b0033e93e00f68sm16776212wrp.61.2024.03.21.03.06.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 03:06:57 -0700 (PDT) Message-ID: <65fc06c1.5d0a0220.0d53.efdc@mx.google.com> Received: by keynux (sSMTP sendmail emulation); Thu, 21 Mar 2024 11:06:55 +0100 From: Bruno Barbier To: Ihor Radchenko Cc: emacs-orgmode , Jack Kamm , Matt Subject: Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) In-Reply-To: <87plvpjj76.fsf@localhost> References: <87o7d0mm54.fsf@localhost> <65bbb108.050a0220.b60fd.6790@mx.google.com> <87jznm8hcu.fsf@gmail.com> <875xz42rp9.fsf@localhost> <874jen8zec.fsf@gmail.com> <87o7cv9e80.fsf@localhost> <65c2875f.050a0220.caf6d.8291@mx.google.com> <18dae5cab1d.bf1c7563863897.4896289306902277373@excalamus.com> <65cfa0d8.050a0220.cb569.ce34@mx.google.com> <18dbe11968a.12c0800a31425096.5114791462107560324@excalamus.com> <65df0895.df0a0220.a68c8.0966@mx.google.com> <87edct0x2w.fsf@localhost> <65e30609.050a0220.89c06.1798@mx.google.com> <87a5ng7uoq.fsf@localhost> <65e9f49b.df0a0220.11103.1c10@mx.google.com> <87ttlhki9e.fsf@localhost> <65eb1e60.050a0220.337b2.a0f4@mx.google.com> <87frwuxy1b.fsf@localhost> <65f95bf2.050a0220.6d051.c8b1@mx.google.com> <87plvpjj76.fsf@localhost> Date: Thu, 21 Mar 2024 11:06:55 +0100 MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2a00:1450:4864:20::335; envelope-from=brubar.cs@gmail.com; helo=mail-wm1-x335.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_FILL_THIS_FORM_SHORT=0.01, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -9.60 X-Spam-Score: -9.60 X-Migadu-Queue-Id: B70DE129D8 X-Migadu-Scanner: mx12.migadu.com X-TUID: gP8freGOgpBv Ihor Radchenko writes: Thanks for your review Ihor! > Bruno Barbier writes: > >> I rewrote the API, rename many things, moved the code around and >> sorted everything into heading/subheading sections. This is hopefully >> less confusing and a lot simpler; and the documentation hopefully >> explains better how to use it. > > Thanks! It does look more clear. > >> The updated section "Commentary", in org-pending, describes the 3 steps >> that are needed to mark and use a pending region: a PENREG (I've renamed >> "PINFO" to "PENREG", for PENding REGion, more specific). > > I feel that org-pending-penreg (org-pending-) is > redundant. Maybe better use org-pending-region? PENREG is the name of the structure; the "org-pending" is the mandatory library prefix; this mechanically gives the name. A PENDREG object is not a "region" in Emacs sense. Do you see a better name for the structure PENREG, so that it doesn't sound like a "region" ? >> WDYT of this version ? > > Comments on org-pending-pendreg struct: > > 1. It is not clear why you need a separate ~virtual~ field. When > ~region~ is nil it already implies that the pending region is > virtual. It's a constant. Calling a function looks more like we need to recompute it each time, and, we could even change it. And cl-defstruct writes the function for us. Do you prefer a manually written function ? > 2. ~id~ field is semi-internal and assumed to be a number. > Maybe we can do something more similar to Emacs process API: > > (make-process &rest ARGS) > ... > :name NAME -- NAME is name for process. It is modified if necessary > to make it unique. > > We can replace id with human-readable name that can also be supplied > when creating a pending region Good idea. Done. > 3. ~source~ field must match the ~region~ marker buffer. Then, why do we > need this field at all? May as well just use (marker-buffer (car region)) The "source" is the region requesting the update. The pending region is the "target" of the update, i.e. the region that will be updated. For example, in DEMO_ONLY, with org-babel, these 2 regions are never the same: 1. the source is the source code block, 2. the target (pending region) is the result region. I tried to improve the documentation of `org-pending-penreg' (source & region). > On the design of ~org-pending~ and ~org-pending-task-connect~: > > 1. I feel confused about the overall design of the interaction between > pending region and the associated task. > > Functions like ~org-pending-task-send-update~ imply that pending > region is rather decoupled from from associated task and the task > should arrange manually for sending updates to the pending region > object. Exactly: the task implementation must use these ~org-pending-task-XXX~ functions to send updates to one (or more) pending region(s). > On the other hand, there is ~task-connection~ that is used to kill > associated task/process or to "await" for it. This time, pending > region is strongly coupled with the task, killing it upon deleting > the pending region. These are optional features; and only ~org-pending~ will know if and when those might be useful. That's why the task needs to provide callbacks here. 1. cancel: Emacs may, in exceptional cases only, send a "cancel" to the task, meaning, "The user destroyed the pending region, and thus, Emacs will not use any update for it". 2. insert-details: If, and only if, the user decides to investigate what happened, Emacs will ask the task if it has any details to add, that might help the user (like exit-code for an OS process, stderr for an OS process or link to a log file, etc.) 3. get (await): It's an (unofficial) way, (in the degenerate case where the task implementation gives up on asynchronicity) to block until the outcome is available. `org-pending' itself doesn't use it; DEMO_ONLY uses it with org-babel to define the synchronous executions. > I think that we need more (optional) coupling between pending region > and the associated task. We should be able to get more information > about the task from pending region side - get logs, current status, > exit status, etc. > More specifically, I think that we need (1) allow to pass task as an > argument for ~org-pending~. That's actually what I started with, but, I couldn't make it work. Breaking it like this is what allowed me to get the most generic and simplest API that works for anything: threads, callbacks, OS processes, timers. If org-pending takes a "task" as an argument, then, we have to define a universal API for whatever a "task" might be: threads, processes, callbacks, timers, etc. and any combination of them. It looks simpler to say that the "task" (whatever "task" means), MAY call: - org-pending-task-send-update (:progress xxx1) - org-pending-task-send-update (:progress xxx2) - org-pending-task-send-update (:progress xxx3) then, finally MUST either call: - org-pending-task-send-update (:success RESULT) or: org-pending-task-send-update (:failure RESULT) > (2) In ~org-pending-task-connect~, we > should allow the task to be a process or timer object. Then, we can > automatically arrange retrieving process/timer status from the task: > Use process sentinel (maybe, modifying existing via ~add-function~) > to arrange process status changes to be automatically submitted to > the pending region; > > Get log updates via process filter > > Kill process via ~kill-process~ It looks to me like a very specific case (one OS process for one pending region); and I'm not sure how useful it would be in practice. But this could easily be implemented on top of the existing API. > Similar for timers. Same, it could easily be defined on top of the existing API. > If the argument to ~org-pending-task-connect~ is a lambda, we can use > the current approach you implemented on the branch. > 2. ~org-pending-task-send-update~ name is confusing - it reads as if we > send an update _to_ the task. Maybe better ~org-pending-region-update~? Yes ... I wanted a common prefix for the 3 functions that a "task" implementation is allowed to use: - org-pending-task-connect, - org-pending-task-send-update, - org-pending-task-not-implemented. It's not confusing if one ignores the common prefix :-) I've renamed all these functions from "org-pending-task-" to "org-pending-ti-" where "ti" stands for "task implementation". > Then, we might even drop ~-sentinel~ field in org-pending-penreg > object and instead implement that hard-coded ~update~ lambda from > ~org-pending~ as a part of ~org-pending-region-update~. That would require to manually capture (dump/load) the context that the sentinel closure is automatically capturing. Why would it be better ? Debugging purposes ? In this case, the current context is (currently) very small, and there is an obvious place where to dump/load, so, just tell me if you want me to eliminate that closure. > 3. I feel that different handling of "owner" and indirect buffers is not > right. > From the user perspective, it does not matter whether we run an src > block from one or another indirect buffers - it makes sense to see > the status in all the indirect org-mode buffers. I just tried to followe Emacs: a buffer owns its overlays; a pending region is (kind of) an overlay. Thus, a buffer owns its pending region. > Maybe we can hook into org-mode's fontification and display > pending overlays in all the indirect buffers. Well ... "adding overlays in indirect buffers using font-lock" looks like a very bumpy road to me ... (being very positive, assuming there is even a road there ... :-) ). As jit-lock is explicitly disabled in indirect buffers, I'm not even sure what it would technically mean. > Further, it is very confusing that running src block twice from the > same buffer is not the same as running the same src block from one > buffer and then from another indirect buffer. The current > implementation of ~remove-previous-overlays~ makes such distinction > for reasons I do not understand. Technically, the outcomes are overlays too; thus, they belong to one buffer. If a user created an indirect buffer to focus on some source blocks, he should expect to manage everything about them from that buffer. ... that looks to me like a plausible explanation that matches the technical limitations :-) We might be able to add some other workarounds for indirect buffers, to provide seamless switches between buffers. But would that really be worth it? In summary, about indirect buffers, I'm not sure it's a good idea to try to handle them. I didn't do much with them, but, I'll already was able to segfault Emacs. I would prefer to put the no-clone-indirect property to the org-mode personally :-) Couldn't we just to forbid "pending regions" in indirect buffers ? (pending regions don't exist today, so, that doesn't look that bad, at least for now) > 4. I have questions about ~handle-result~ argument in ~org-pending~. > It is only called on success, Yes. It means Org needs to handle the exact same result as in the synchronous case, and, it must handle it in exactly the same way. That's a design choice. That's probably why it was easy to write the org-babel examples in DEMO_ONLY; and they already handle most of the standard Org parameters: async & sync, session & no session, append/prepend/post/stdin/var, etc. > but I can easily see that we need to > handle things specially on failure as well. For example, insert > stderr or perform other actions like displaying some buffer. Or we > may even hook some special action to clicking on status overlay. For > example, clicking on "failure" status overlay may raise stderr log. It's already there, no? If you click on any result (success or failure, inline block or not, even dynamic blocks), Emacs pops up a buffer with all the details (source, start, end, duration, stderr, etc.). The function `org-pending-describe-penreg' defines what is inserted. A given task is free to insert log, links, widgets, images, diffs, etc. (by providing the relevant :insert-details method). These are design choices that are relevant here: 1. Do not differ from the synchronous case. 2. Do not delete a valid result until you know that you have something better (where a progress or a failure is not "better"). 3. Do not interrupt the user with popups. If, in the synchronous case, org-babel writes some logs, then the task must report a success to org-pending so that Org just behaves the same. If the task reports a failure, the user keeps the previous content. We could modify how ob-core uses org-pending, adding some options if some users would like errors to be written down using some Org syntax. I've pushed my update to the public repo (sorry, forced push again due to some mistakes). Bruno