From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id ODh+KqTt7WCHZAEAgWs5BA (envelope-from ) for ; Tue, 13 Jul 2021 21:46:44 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id CBcLJqTt7WDTOwAAbx9fmQ (envelope-from ) for ; Tue, 13 Jul 2021 19:46:44 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 175C079A2 for ; Tue, 13 Jul 2021 21:46:44 +0200 (CEST) Received: from localhost ([::1]:60918 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m3OMd-0002Lo-34 for larch@yhetil.org; Tue, 13 Jul 2021 15:46:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54962) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m3OIM-0004Ct-6X for emacs-orgmode@gnu.org; Tue, 13 Jul 2021 15:42:18 -0400 Received: from mail-oi1-x234.google.com ([2607:f8b0:4864:20::234]:40800) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1m3OIJ-0007CL-Pn for emacs-orgmode@gnu.org; Tue, 13 Jul 2021 15:42:17 -0400 Received: by mail-oi1-x234.google.com with SMTP id l26so29913847oic.7 for ; Tue, 13 Jul 2021 12:42:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:reply-to:from:date:message-id :subject:to:cc; bh=ovoR6bukxaTKoLvc9Mz1AmuwY4FYAniGokpyy0dtplM=; b=LzygLjvusYLOMyusj4JbJyfflpEcQw5djnjcqePmZRJ/lqiJ43BWj6nHBHymdUdr8J DjdINp1NOo5/vRItMOPUn2UO1tNzOhg6LgN/ruKzyoQE4IGIe1Bs5dO6ZpswGq+8E+qB lBVQcHb8/zFc0sAEgUKMS9V1pOIc+8oMgjwNYvxkEC5bxctC6r31X4XKK/le+7yb0oJh hryC6MUNPImSI8/XYvoDQSdACAeWmQIqSyOuK/q4Pf+pSrfkb6RA+pe9yzewOqE53now mCQRS8fnP9bqkROLV9cWrlbXvB81V5XsoMVLP26OV2Vwyb7IPhyA21jg3m+umR/lEApk X2pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to:cc; bh=ovoR6bukxaTKoLvc9Mz1AmuwY4FYAniGokpyy0dtplM=; b=rfSV40uSfd8smUM+eNBD6e7lIMRYSQbLZVsHaVBq0Dc+oHElePcFUx4V72ZgeH+loJ tOo2KrpmOccOUUwydg7lI+wtaWihxgUn+UXAMfhRCPAKXD80lA6E//bLTrPKjSY/jqz/ CuWDxqj2lhAOiQs3Ahkin3B9VC5j8meg9xgDnKqiETvjXp/0awll3ELaD+dUY9WuY190 N9UDf3CLmT4Chl55FWLLezGulLLar3kosAStiqqYGlh6QNj995oDwevvCO4MaJLq29j8 8jK2r1+4yZFYc0roCSQ/D7tJ+dPnta1DII+YYoQI9LBn45Q+TFTwPyEpypaCNiBvhQiI j80g== X-Gm-Message-State: AOAM531BesaDwSKhirhFe+vM+78ZE8GLByH9jw4VdcpStcT8IzRYrlvL WrfZPS23LBi9n8esO2XD9BffcTUUjHS+cAZNAUJTfjpIDuVmuA== X-Google-Smtp-Source: ABdhPJz2956edeqNnoY9Fq3xjoD5ldiqhBtg6Ub8b5thy9wldldUqbV1KlaFHP7jZuqk1Xd5PJu4fdogcCaM8I8Vu5Y= X-Received: by 2002:aca:ebc3:: with SMTP id j186mr777219oih.153.1626205333387; Tue, 13 Jul 2021 12:42:13 -0700 (PDT) MIME-Version: 1.0 References: <95_WQAdCOQRo2u4VTanEqQ9QrbmOhFNk4aTCGlvgNU8PkVz3j7SQfwIZuwoxtQj0v5bGnRnP6NPf-Z6ImfWKpg_pHF6sTFHMgw0KVfir06U=@protonmail.ch> <9F8F3589-6E58-45AE-A0E6-F359ACA2A4D9@condition-alpha.com> In-Reply-To: From: George Mauer Date: Tue, 13 Jul 2021 14:42:02 -0500 Message-ID: Subject: Re: A requires/provides approach to linking source code blocks To: emacs-orgmode Content-Type: multipart/alternative; boundary="0000000000003a813905c7066fe1" Received-SPF: pass client-ip=2607:f8b0:4864:20::234; envelope-from=gmauer@gmail.com; helo=mail-oi1-x234.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: gmauer@gmail.com Cc: Tom Gillespie Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1626205604; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=ovoR6bukxaTKoLvc9Mz1AmuwY4FYAniGokpyy0dtplM=; b=VUtHl6sgcibnH7Ab6POEMYX7NvGt+J8anSp4TJ4iLAYH2gzUX7tuc6IfKOAp8q6It/YNwr 8dqdBtmPB2dB6m0TcJKCImCF3nCgGeHDEszcGEDl54VH60NH5bMoiu0uk4208rSYnTmBEn hQuu8l4YzIWAWXnhr8t1iFDp6kuTtbyu6aUCYOGo0zllr9yr0xTCS4VkWuUsLrXVyKuTtu pUQRZhMpHbdP1fusBT6U03h7Z9sIubNnP9GogLVzKiFq9mb+Dyz4IDm4VJuOoElaPIRNAn VuVrdZAjUVdQbJ2Pi2dgx5rDpUZXWlstm7uYuhLLzQwfk2P/Hu35Cv1eyOsmBQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1626205604; a=rsa-sha256; cv=none; b=gXdS9zsmQVd64k0xFpfPSbsKv3o5MVNvRJTIeJVJ/DTy4RGHYoX71bELSzN5/lkZA1BqSk FxUuuXtxA1iLEF4kBZZ636+gtVgs4xRlmPYdc/fURhmY7TBDrvpCk7Ic2l+/l45kWDWnAg uOQWJskPyMQkfP0xC2iecz1aCSLmND4y3NlVzJ0k3hTVhrVzSmCJghDraH8VoXGxtqT+B7 Ilm8YYNDE1DgPJ0T1amRGO2uY+IyjLFYRKHHM4q2O1aPBo+r/w3OX60v0NWCQ5HAMykR2R 2NITxYOum6Frz8I59PX10hgl9XgoBDhhRJKKkyd91/nsQHXMJdUecmAF4Eb4iA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=LzygLjvu; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Migadu-Spam-Score: -3.10 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=LzygLjvu; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Migadu-Queue-Id: 175C079A2 X-Spam-Score: -3.10 X-Migadu-Scanner: scn0.migadu.com X-TUID: Jp2DqiKHZL8u --0000000000003a813905c7066fe1 Content-Type: text/plain; charset="UTF-8" In implementing an extension you might consider doing it as a generalized form of what I did with ob-racket (https://github.com/togakangaroo/ob-racket ). I think it is best to rely on the import/require/include mechanism of the language you're using. Pretty much all of them support adjacent files. The problem with code block execution is that it expands the block and writes it to a temporary directory in a non-predictable location. If you want to reference other blocks, you need them to *also* write into files in the same folder. What I do in ob-racket is support a header :adjacent-file which allows you to pass in the name of another block and then writes it into the same folder as the file to be executed with a predictable name (the name of the block) So something like this #+begin_src racket :adjacent-file stacker-reader-expander.rkt #lang reader "./stacker-reader-expander.rkt" Will make sure the block named `stacker-reader-expander.rkt` is written into that same temporary directory with that same name. There's a few rough edges but for the most part it works well. Would love to see someone take this and generalize it into its own extension. On Tue, Jul 13, 2021 at 2:18 PM Tom Gillespie wrote: > We have been receiving many new feature suggestions and requests > coming in for org babel. I think that Tim's suggestion is the right > one. Nearly all of these need to be implemented as an extension first > and tested independently. Further, even if this is done, it should be > clear that there is zero expectation that such extensions will be > incorporated. > > Once I wrap up the formal grammar for org, one of the next things I > plan to work on is a clear specification for org babel. This is > critical because so many of the suggestions that come in deal with > individuals' specific problems and thus fail to account for how such > features interact with existing features and how the newly proposed > feature would block some other features in the future, confuse users, > etc. Such suggestions also often fail to account for increased > complexity, nor have they been exposed to a sufficient number of > examples to reveal fundamental ambiguities in how they could be > interpreted. The issues with variable behavior between ob langs for > :pre :post :prologue :epilogue etc. are already enough to keep us busy > for quite some time. > > With regard to this thread in particular, it is of some interest, but > there are fundamental issues, including the fact that certain > languages (e.g. racket) expect module code to exist somewhere on the > file system. There are ways around many of these issues, in fact there > are likely many ways around any individual issue, so org babel needs > to systematically consider the issues and provide a clear > specification, or at least a guide for how such cases should be > handled. > > To give an example from one of my continual pain points: I start > writing python or racket in an org src block and then I want it to be > a library so that it can be reused by other code both inside and > outside the org file without having to resort to noweb. > > What is the best way to handle this? I don't know. Right now I tangle > things and resort to awful hacks for the reuse-in-this-org-file case, but > I'm guessing there is a better generic solution which would allow _any_ > org block to be exported as a library instead of nowebbed in. > > Before jumping for any particular suggestion for how to handle this > we need to explore the diversity of cases that various ob langs > present, so that we can find a solution that will work for all of > them. After all, packaging code to a library for reuse is an > extremely common pattern that org babel should be able to > abstract, but it is a major undertaking, not just the addition of a > keyword here and there. > > In short I suggest that we issue a general moratorium on new org babel > feature suggestions until we can stabilize what we already have and > provide a clear specification for correct behavior. Until we have that spec > we could encourage users to create extensions that implement those > features. > > Best, > Tom > > > PS The other next thing that I am working on might be another way out > for this particular feature request. Namely, it is simplifying and > extending org keyword syntax so that new keywords (with options) and > associated keywords can be specified using keyword syntax within a > single org file. This would make it possible to get useful high level > keyword behavior in a single file without burdening the core > implementation with more special cases for associated keywords, and it > would allow users to write small elisp functions that could do some of > what is suggested here, all without need to add anything to the core > org implementation. > > --0000000000003a813905c7066fe1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
In implementing an extension=C2=A0you might consider doing= it as a generalized form of what I did with ob-racket (https://github.com/togakangaroo/ob-racke= t).

I think it is best to rely on the import/require/include me= chanism of the language you're using. Pretty much all of them support a= djacent files. The problem with code block execution is that it expands the= block and writes it to a temporary directory in a non-predictable location= . If you want to reference other blocks, you need them to *also* write into= files in the same folder.

What I do in ob-racket = is support a header :adjacent-file which allows you to pass in the name of = another block and then writes it into the same folder as the file to be exe= cuted with a predictable name (the name of the block)

<= div>So something like this

=C2=A0 #+begin_src rack= et :adjacent-file stacker-reader-expander.rkt
=C2=A0 =C2=A0= =C2=A0#lang reader "./stacker-reader-expander.rkt"

=
Will make sure the block named `stacker-reader-expander.rkt` is = written into that same temporary directory with that same name. There's= a few rough edges but for the most part it works well. Would love to see s= omeone take this and generalize it into its own extension.

<= div class=3D"gmail_quote">
On Tue, Jul= 13, 2021 at 2:18 PM Tom Gillespie <= tgbugs@gmail.com> wrote:
We have been receiving many new feature suggestions and req= uests
coming in for org babel. I think that Tim's suggestion is the right
one. Nearly all of these need to be implemented as an extension first
and tested independently. Further, even if this is done, it should be
clear that there is zero expectation that such extensions will be
incorporated.

Once I wrap up the formal grammar for org, one of the next things I
plan to work on is a clear specification for org babel. This is
critical because so many of the suggestions that come in deal with
individuals' specific problems and thus fail to account for how such features interact with existing features and how the newly proposed
feature would block some other features in the future, confuse users,
etc. Such suggestions also often fail to account for increased
complexity, nor have they been exposed to a sufficient number of
examples to reveal fundamental ambiguities in how they could be
interpreted. The issues with variable behavior between ob langs for
:pre :post :prologue :epilogue etc. are already enough to keep us busy
for quite some time.

With regard to this thread in particular, it is of some interest, but
there are fundamental issues, including the fact that certain
languages (e.g. racket) expect module code to exist somewhere on the
file system. There are ways around many of these issues, in fact there
are likely many ways around any individual issue, so org babel needs
to systematically consider the issues and provide a clear
specification, or at least a guide for how such cases should be
handled.

To give an example from one of my continual pain points: I start
writing python or racket in an org src block and then I want it to be
a library so that it can be reused by other code both inside and
outside the org file without having to resort to noweb.

What is the best way to handle this? I don't know. Right now I tangle things and resort to awful hacks for the reuse-in-this-org-file case, but I'm guessing there is a better generic solution which would allow _any_=
org block to be exported as a library instead of nowebbed in.

Before jumping for any particular suggestion for how to handle this
we need to explore the diversity of cases that various ob langs
present, so that we can find a solution that will work for all of
them. After all, packaging code to a library for reuse is an
extremely common pattern that org babel should be able to
abstract, but it is a major undertaking, not just the addition of a
keyword here and there.

In short I suggest that we issue a general moratorium on new org babel
feature suggestions until we can stabilize what we already have and
provide a clear specification for correct behavior. Until we have that spec=
we could encourage users to create extensions that implement those
features.

Best,
Tom


PS The other next thing that I am working on might be another way out
for this particular feature request. Namely, it is simplifying and
extending org keyword syntax so that new keywords (with options) and
associated keywords can be specified using keyword syntax within a
single org file. This would make it possible to get useful high level
keyword behavior in a single file without burdening the core
implementation with more special cases for associated keywords, and it
would allow users to write small elisp functions that could do some of
what is suggested here, all without need to add anything to the core
org implementation.

--0000000000003a813905c7066fe1--