From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id mAyUNf2LdmbdEwAA62LTzQ:P1 (envelope-from ) for ; Sat, 22 Jun 2024 08:31:58 +0000 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id mAyUNf2LdmbdEwAA62LTzQ (envelope-from ) for ; Sat, 22 Jun 2024 10:31:57 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=posteo.net header.s=2017 header.b=SEyIHRlq; dmarc=pass (policy=none) header.from=posteo.net; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1719045117; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=vh/DelY/p+YSBK44Y4EEH4UZbfdr/Fe5WWdrYlqeCR0=; b=HSXPxZw5reyjVJXCX4mf5cTfBa9ZD0qELfazYK3toFNTzp7pDV4T8BETRIAnuhoT6HYwsK /mpDPTJ+4UfJeDyT8mYYM5kRGRmJZ2VmkD36SKYvJDfAGmT/AlVlloaiEWOuXxptJU58kh sb4j9UblshrfzeIma63e+YkSQRp7iXhDSiLtTa77kmD6Hm2Gfep8VA+FmU33eARJ4tOXlO EhAHXaAV0YmiLln+skjIj9PaO8RzkYzSIgwaTc1wNMF1TZWiE6gn90+GVxW0KdZnt+AwUk 030GoRI4nICf6u+sBdCNaV51MEKVE2UF1eQuacbq2ljXeWTp2sJP/7AaxsxuYA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1719045117; a=rsa-sha256; cv=none; b=Y4udoL3VOyzYjgPWQp+Bwd/6BC7mYlNOW5ArQIXrV6L7D/u6jW3/GIKDN+T0Od6yT9aeLc YzMRDwNzwQkNM8Gdd/7iIqic4WOFxKV2lvsOXWXoKyYkGik1aNUsPxOTxJgj0cXhzSY+A1 rL+1KTg/Z5JTAVP/OtpkyZOwJnoe/aV+jHtwyLYCGmsHybv/lOngHWEHJGOlwNc0C3NIdh Vp7DZQmG8/b6fgxXqTRWSqGz3O12XKAZMxwUCcYAdGWdf+mi71kg97JKXGFgbXwNaQ8Vse GsUdfkZbVPjrpzkCBWHI1ZMAyJHPV+n+7x7k2ufoODEphX3OhgB3JZGoVhyfug== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=posteo.net header.s=2017 header.b=SEyIHRlq; dmarc=pass (policy=none) header.from=posteo.net; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 432AC65089 for ; Sat, 22 Jun 2024 10:31:57 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sKw9X-0004WO-0i; Sat, 22 Jun 2024 04:31:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sKw9R-0004VM-6C for emacs-orgmode@gnu.org; Sat, 22 Jun 2024 04:31:13 -0400 Received: from mout02.posteo.de ([185.67.36.66]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sKw9H-0003m8-6y for emacs-orgmode@gnu.org; Sat, 22 Jun 2024 04:31:11 -0400 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 1926D240104 for ; Sat, 22 Jun 2024 10:30:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1719045057; bh=L9N1nWFg+IxNnfAivCJ8/pbEdZKwgfzCS5mYTTdSHlA=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type: Content-Transfer-Encoding:From; b=SEyIHRlqJHEYJ39rNXs68AIfuqEL/O2viQzwTh7gLy/oVTxnPcjc1DzO06TS9iEif jb2W2Hx/6utKY0+850n9Gvzfemq0J3nssAbMZGUraBA22WkRKJXgz1g+v8gcRhny9f lRQcSNBartpcvG3xBfLnwK6m1BdMoOtk28HZOymjWVAPkcoowiH2aRHZYmIsho6Q9Y Cq0FvQE7Kp8N0phWUXyaCzcXlS3vA9EWDtZF826uav8I9uiOo3+1H0paKZaDz0C2Ac sbaad2Q4OzouF+HuOJks5SyO/3GDdK7xmULRZdcLYl3V0ktsDPUyW6n9Z9dF5aGWVw WrgjoxGKsjmjQ== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4W5nVm2jBfz9rxK for ; Sat, 22 Jun 2024 10:30:56 +0200 (CEST) From: Ihor Radchenko To: emacs-orgmode@gnu.org Subject: [SUMMARY] #8 [[bbb:OrgMeetup]] on Wed, June 12, 19:00 UTC+3 In-Reply-To: <8734q9kjg0.fsf@localhost> References: <8734q9kjg0.fsf@localhost> Date: Sat, 22 Jun 2024 08:32:40 +0000 Message-ID: <87h6dlidhz.fsf@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=185.67.36.66; envelope-from=yantar92@posteo.net; helo=mout02.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -6.58 X-Spam-Score: -6.58 X-Migadu-Queue-Id: 432AC65089 X-Migadu-Scanner: mx11.migadu.com X-TUID: tpm1X2SrOm48 It was a long discussion this time (almost 3 hours). We ended up talking in-depth about Org mode development and some very technical aspects of it. - As usual, we started from posting Sacha's News, but with a bit of a twist - Since recently, https://orgmode.org shows Org mode-related news on the = front page - Mailing list feature discussions, polls, and announcements - Org mode section of Sacha's News - Suhail asked about the status of 3 bugs reports posted on Org mailing lis= t recently - https://yhetil.org/orgmode/87h6e134sp.fsf@gmail.com/ - This one is about completion of :async header argument in "shell" blocks, where "shell" is not literally shell, but "bash", "csh", etc. The completion only works for "shell" blocks, but not for other blocks. - This is simply an omission in ob-shell.el - ob-shell is implemented in the following way 1. Generic "shell" backend is defined 2. Other backends like "bash", "csh", "fish" (full list in ~org-babel-shell-names~) are defined based on the generic backend in ~org-babel-shell-initialize~ 3. ~org-babel-shell-initialize~ fails to define ~org-babel-header-args:~ - variable used to compute completions. 4. The fix is easy; just need time to get there. - [2024-06-22 Sat] Fixed. - https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=3De= 666660c - https://yhetil.org/orgmode/87cyop3445.fsf@gmail.com/ - This is a similar issue related to ~org-babel-shell-initialize~ - This time, ~org-babel--initiate-session~ is not defined for specific shells - The problem is more severe than completion though. It renders =3DC-u C-c '=3D (~org-edit-special~) unusable. - ~org-edit-special~ /without/ prefix argument opens code block at po= int for editing - ~org-edit-special~ /with/ prefix argument should instead open associated session for code block with =3D:session ...=3D header arg - Because of the bug, this does not currently work for shell blocks - [2024-06-22 Sat] Fixed. - https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=3D5b366a= 73 - https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=3D5e9ac1= 46 - https://yhetil.org/orgmode/87cyonhuq3.fsf@gmail.com/ - This report is about regression of =3D#+bind: ...=3D keywords in the = new Org 9.7 - Fixed. https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?i= d=3D96113f3b5 - Karl Voit (publicvoit) shared his recent bug report for helm-org-rifle - https://github.com/alphapapa/org-rifle/issues/85 - The problem he experiences is with parts of outline path being invisible in the candidates buffer as long as they are folded - The likely reason with the way folding is done in Org mode 9.6. Org 9.6 (and Org 9.7 for Emacs <29) use 'invisible text property to fold headings. If such folded headings are copied verbatim from the Org buffer, they may remain invisible. - If my guess is right, setting ~org-fold-core-style~ to ~overlays~ should make things work again in helm-org-rifle - Note that Org 9.7 now moves towards restoring the old approach - using overlays to fold staff. This is because built-in isearch.el, query-replace-regexp, and a number of external packages (like evil) only support searching (and revealing!) invisible text when it is hidden via overlays. (I was hoping that it can be possible to work around such limitation, but constant stream of bug reports and growing pile of fragile workarounds proved that text hidden via text properties cannot reliably searched in practice; alas...) - As an unfortunate side effect of this, I had to revert feature introduced in Org 9.6 - searching inside hidden parts of the links - See https://orgmode.org/Changes.html (It is no longer possible to reveal hidden parts of the links during isearch) - We went on talking about org-ql vs. helm-org-rifle - org-ql is generally more flexible as it allows more than just plain text + outline search - In org-ql, one can quickly match headings by specific properties, tags, timestamps, etc - In addition, org-ql, since recently, has integration with built-in completion frameworks, and their extensions like vertico. - Of course, helm interface is also there - helm-org-ql - However, as it turns out, org-ql lacks one important feature Karl particularly likes. =20=20=20=20 In helm-org-rifle, when you enter search string interactively, if the match is inside a heading, the line containing match is displayed in addition to the outline path to the heading. This provides a valuable additional context - you can briefly look what exactly inside a heading is matched. - [2024-06-15 Sat] alphapapa, in https://github.com/alphapapa/org-rifle/issues/85, pointed out that one can set ~org-ql-completing-read-snippet-function~ to ~org-ql-completing-read-snippet-function~ - this will replicate what helm-org-rifle does in ~org-ql-completing-read~ (but not in ~helm-org-ql~). - For ~helm-org-ql~ one may need to fiddle with the source code of ~helm-org-ql-source~ to add this kind of feature =20=20=20=20 - Tim continued his struggle with an elusive bug he experiences when setting tags (C-c C-q) in large Org buffers - setting tags takes forever for him. (There is infinite loop likely lurking somewhere) - His last observation was that setting ~org-element-use-cache~ to nil or running =3DM-x org-element-cache-reset=3D makes settings tags work again and not hang (for some time, at least) - I can guess two possibilities now: 1. Some problem with org-element-cache mishandling headline boundaries 2. Infinite loop in ~org-element-cache-map~ https://list.orgmode.org/orgmode/CH3PR84MB342450FE12DC716A9FA65EADC5= 0D2@CH3PR84MB3424.NAMPRD84.PROD.OUTLOOK.COM/ - For now, I asked Tim to enable cache self-checks (~org-element--cache-s= elf-verify~) - If something shows up there, we got a problem in the cache - Otherwise, it is likely ~org-element-cache-map~ - In any case, ~org-element-cache-map~ should be rewritten and simplifi= ed sooner than later - Karl shared his config snippet that allows updating link description according to its source https://graz.social/@publicvoit/112582948662128779 - For Urls, it fetches the web page title - For Org id: links, it looks up the linked heading title - He had a technical question about how to update the link description fr= om Elisp - Now, he uses interactive : (org-insert-link nil nil myheadingtitle);; add description - The above works, but it is too interactive - this particular call to ~org-insert-link~ will query for the link path, happening to use path at point as the default. So, Karl has to hit needlessly for confirmation. - To make ~org-insert-link~ work non-interactively (from Elisp), one needs to supply two optional arguments: LINK-LOCATION and DESCRIPTION : (org-insert-link &optional COMPLETE-FILE LINK-LOCATION DESCRIPTION) See the docstring for details. - To retrieve link location at point, it is possible to use Org element= API https://orgmode.org/worg/dev/org-element-api.html : (org-element-property :raw-link (org-element-context)) - Also, I noticed that another part of the "auto-description" code uses a function that is not designed for Elisp API: ~org-open-at-point~ - ~org-open-at-point~ fiddles with window arrangement and using is from Elisp may cause unexpected results - Instead, it is better to use ~org-link-open~ that only jumps to the link target (as defined by :follow link parameter): : (org-link-open (org-element-context)) - Since karthink (one of the authors of the WIP async LaTeX preview feature) was around, I shared my major WIP work on Org mode development. (My work will affect their WIP branch) =20=20 For the last half year, I am working on a large refactoring touching all parts of Org mode code. The main objectives are: 1. Avoid situations when loading almost *any* part of Org mode requires loading the *whole Org mode codebase* 2. Make Org mode libraries less scary to contribute to by reducing their size. 3. Minimize uses of global variables when they are not strictly necessary. Or, at least, document them. In practice, it means splitting Org mode into smaller sub-libraries and carefully reviewing all the ~require~, ~declare-function~, and dynamically scoped ~defvar~ statements. (this may sound easy, I initially hoped to spend a few weeks max on this task; but here we are, 6 months later, 400+ commits ahead of main) The WIP branch is available at https://git.sr.ht/~tartan92/org-mode/log/feature/refactor-deeps-v2 (yup, it is already the second attempt; first several tries failed miserably with Org mode in completely broken state) - The most visible changes in the branch are many, *many* new libraries. I separated huge old files like =3Dorg.el=3D, =3Dorg-agenda.= el=3D, =3Dorg-table.el=3D, =3Dorg-list.el=3D, =3Dob-core.el=3D, =3Dorg-clock.e= l=3D, and =3Dorg-compat.el=3D into many smaller helper libraries. - This includes the existing LaTeX preview code being put into a separate =3Dorg-preview-latex.el=3D, which may create problems when merging WIP async preview branch and refactored Org version. - Overall 127 Org libraries will be turned into 263 (WIP!) smaller libraries. The whopping +135 libraries more... - Org major mode will leave in separate =3Dorg-mode.el=3D, which only loads strictly necessary libs, without options user commands or extra Org functionality. All the extras will be autoloaded instead - when a key binding is pressed. - =3Dorg.el=3D will still load everything, so that old packages and configs that do ~(require 'org)~ or ~(use-package org...)~ are not broken. Eventually, we will ask users to avoid loading =3Dorg.el=3D and instead load more specific libraries. =20=20 - Many external optional libraries will also be loaded just-in-time, when their functionality is needed - For example, =3Dol-gnus.el=3D - the library implementing links to GNUS buffers, will no longer load the whole GNUS when Org mode is loaded. It will instead do it when we need to follow an actual gnus: link. - The way it is done is placing ~(require ')~ not directly as a top-level form in =3Dorg-lib.el=3D, but inside individual functions in that lib. - Suhail commented that top-level ~(require ...)~ are treated differently by the byte-compiler implying that something may not be right if we move things inside functions - This is not a problem though. The main effect of top-level require is that byte-compiler loads that library at the compile time, and knows which variables and functions are defined there. - The main effect of putting requires into functions is that we get a bunch of compile-time warnings as byte-compiler does not know functions/variables coming from third-party libs and complains about them - It can be suppressed by ~(defvar )~ or ~(declare-function ...)~ forms. - karthink also commented that runtime ~(require ')~ may hit performance because is must search through the whole ~features~ list (list of all the libraries loaded into current Emacs session) - It is not a big deal in practice though : (benchmark-run 10000 (featurep 'does-not-exist)) : ;; =3D> 0.1 sec or 10 =C2=B5s per require - Even if it were, we could ask Emacs devs to optimize this by caching loading status in the library symbol itself : (put 'foo :loaded t) : (get 'foo :loaded) ; =3D> t - One more aspect of the changes is attempting yet another fix for the everlasting problem with "mixed" Org mode installations we face every major Org release - when some Org libs are loaded from built-in Org mode version while other libs are loaded from newer Org. This often results in random breakage people keep reporting. - Emacs 30 recently introduced a tool to address this problem somewhat: ~require-with-check~. Unlike simple ~require~, ~require-with-check~ checks if a different version of the requested library has been added to the ~load-path~, and re-loads it as necessary. This way, if newer Org gets loaded, it can make sure that the versions of Org sub-libraries are all coming from the correct version of Org mode. - The problem with this idea is that Org codebase does not currently use ~require~ everywhere. Instead, many places in Org simply rely upon other Org libraries already loaded (~declare-function~ statements are everywhere). And we cannot simply add more ~require-with-check~ everywhere - many Org libs struggle from circular dependencies. For example, =3Dorg-agenda.el=3D requires =3Dorg.el=3D to be loaded and= =3Dorg.el=3D expects =3Dorg-agenda.el=3D to be loaded. - Untangling these circular deps has proven to be the hardest part of the ongoing refactoring. - Splitting Org into multiple small libraries is also a step forward toward more modular Org and to converting some Org-specific libraries to proper Emacs libraries, as requested by RMS: https://list.orgmode.org/E1kIkxv-0007iy-Av@fencepost.gnu.org/ - Karthink asked which libraries would be good candidates for upstreami= ng. Here is the list, naming the existing libraries, and the libraries that are a part of WIP branch: - org-agenda-undo :: undoing changes in multiple buffers at once This is the way undo works in agenda buffers, undoing changes in multiple headlines changed from agenda, even when they live in different Org files. - org-capture :: Adding quick notes is not limited to adding them to Org mode; in fact, org-capture is a development of the orig= inal remember.el, which is a part of Emacs core. See https://lists.gnu.org/archive/html/emacs-orgmode/2023-12/msg00412.html - org-clock-notify :: System notifications - org-diary-lib :: Org extensions for diary-lib.el - org-duration :: Library to manipulate durations in human-readable format (like 1d 3h 5min) - org-fold-core :: Generic folding API - org-idle :: Advanced idle time tracking (not just for Emacs idleness, but for system idle time as well) - org-macs :: has many small helper functions; some may be of interest upstream - org-persist :: generic cache API - org-preview-image :: Image preview. Does not have to be limited to Org mode buffers. - org-preview-latex :: Same thing. - org-protocol :: Processing external input into Emacs (from command line, from clicked Urls as url handler). Unlike running commands directly (which is unsafe), =3Dorg-protocol=3D can be used for safe interaction with external processes - org-time :: Extra functions to work with Emacs time data. Some functions might be of interest upstream. - org-timer :: (almost) generic timer for Emacs - org-track-markers :: Keeping relative positions of markers when cutting/pasting text in buffers. This is the way agenda does not get broken by cutting/pasting/refiling the underlying headings around. - In the context of =3Dorg-agenda.el=3D split into multiple smaller libs, Ilya Chernyshov commented about sub-optimality of agenda searches, when agenda looks for timestamps. - Org mode has parser cache, but this cache only goes down to paragraph level. Markup inside paragraphs (like timestamps) is not cached. - Such caching is planned to be done in future, although even now, the agenda search is fairly fast in practice. - In the past, agenda struggled from accounting for property/tag inheritance, but this is no longer a problem because properties and tags are parts of headings, which do get cached. - Ilya also commented that org-ql searches for timestamps very fast - Agenda is actually on par with org-ql these days wrt to timestamps. In fact, AFAIR, the way agenda searches timestamps was one of the things that inspired org-ql. - Ilya further asked about plans to merge org-ql in Org core - We do have such plans - See https://github.com/alphapapa/org-ql/issues/409 - Alphapapa wants to merge ts.el to Emacs first - https://github.com/alphapapa/ts.el/issues/29 - As usual, everything is a subject of free time available to the contributors. So, things are progressing, but slowly. - visuwesh reported a bug with WIP async LaTeX preview branch - It does not work with pdf-tools, because pdf-tools is using ~org-create-formula-image~. The function is supposed to work, (although it will be obsoleted), but it does not. - Karthink suggested reporting the details to Org ML or to TEC's github r= epo - https://github.com/tecosaur/org-latex-preview-todos/issues/34 - Original bug report: https://github.com/vedang/pdf-tools/issues/283 - We looked inside pdf-tools to figure out how pdf-tools has anything to do with latex preview - ~pdf-annot-print-annotation-latex-maybe~ is using ~org-create-formula= -image~ - ... and the way it use it is sub-optimal - It is not the call to ~org-create-formula-image~, but the fact that t= hey have to : (require 'org) ;; org-create-formula-image Load the whole(!) Org mode just for that function. - It clearly shows why splitting Org libraries is a good idea - visuwesh also commented that similar situation is with comint-mime package (https://github.com/astoff/comint-mime): it uses Org's LaTeX preview functionality https://github.com/astoff/comint-mime/blob/ca278064dac431b8c3f72ab1= b860bf9c9b8d8d60/comint-mime.el#L179 - In future, the idea is to discourage ~(require 'org)~ in third-party packages and instead load more specific libraries where the necessary functions live - Jeff Trull asked about using =3Dorg-element=3D in Org tables - I first misunderstood the question and started talking about parsing table formulas (it is done using ancient regexp-based code) and went ahead talking about new parsing features in Emacs like PEG https://elpa.gnu.org/packages/peg.html - PEG is an alternative way to describe syntax, especially recursive syntax (which Org mode is) - And PEG has become a part of Emacs since a few months ago. (it took over 2 years, but we are finally there: https://yhetil.org/emacs-devel/875yvtbbn3.fsf@ericabrahamsen.net/) - However, the actual question was not about table formulas, but about how Org mode edits tables in Org buffers (swap rows/columns; alter cells; re-aligns tables; writes formula results, etc) - Currently, everything is done very manually, hard-coding for the current table syntax and not using Org parser - Ideally, we indeed want to make use of the parser as much as possible, so that we can extend formula syntax more easily - However, Org element API currently lacks sophisticated functions to edit Org text in buffers - We have very rudimentary ~org-element-interpret-data~, but not much more - Various parts of Org instead employ various kludges to edit staff - =3Dorg-list=3D has semi-independent parser and editing API - =3Dorg-table=3D uses old regexp-based approach - headline editing is using parser partially only - So, a proper editing API is necessary before we can properly use Org parser to edit things - One inspiration may be https://github.com/ndwarshuis/org-ml, although there are questions to the performance of that library =20=20=20=20=20=20=20=20 - Karthink asked about technical details of how Org mode parser code is able to respond to C-g even when it is stuck (that is: not just exit, but interact with the user asking to press C-g multiple times in order to force-terminate the parser). - Karthink assumed that Org mode is using ~while-no-input~ Emacs macro - Org mode does not use ~while-no-input~ - ~while-no-input~ terminates on any input event, not just on C-g. Instead, Org mode let-binds ~inhibit-quit~ around critical parts of the parser that may trash the parser state if exited without cleanup. Then, to retain the ability for users to terminate the parser if there is a bug in it (there used to be many bugs related to cache here a few releases back), Org checks ~quit-flag~ to see if C-g was pressed, displays a message, and only runs full parser cleanup when ~org-element--cache-interrupt-C-g-max-count~ is reached. =20=20=20=20 See ~org-element--parse-to~ for the details. - More generally, Org mode parser implements old-school asynchronisity using timers. Curious users may explore the docstrings of ~org-element-cache-sync-idle-time~, ~org-element-cache-sync-duration~, and ~org-element-cache-sync-break~. - Jeff Trull commented that he made a custom extension to Org mode parser last year. We did not go into the details, but went on discussing extending the Org mode syntax - The entry point is the commentary in org-element.el that details what needs to be done - In short, one needs to write a new parser function, a new interpreter function, add the new element/object to org-element-all-elements/objects variable, and modify org-element--current-element to fit the new syntax element into the parser priorities. - We currently have one new major change in Org syntax being discussed: "inline special blocks" (tentative name) https://list.orgmode.org/orgmode/875xwqj4tl.fsf@localhost/ - The idea is to create something similar to special blocks (paragraph-level element) that can represent custom type of paragraph data (notes, sidebars, boxes, alerts, etc: see https://github.com/alhassy/org-special-block-extras that demonstrates how flexible it can be) - Unlike special blocks, inline special blocks are, well..., inline. They represent custom markup - The plan is to allow in-document definition of inline special block exports; custom faces; etc - In a sense, they are like custom link types, but without limitation that links cannot contain other links and that link path is interpreted verbatim - They will also be able to contain attributes, which will solve long-standing problem that we cannot pass export attributes to images on export (like image width) - also, a lot more; all the details in the linked mailing list thread - We already have experimental branch to play around, but still discussing the details of the syntax and the features that will need to be added - (I do not remember whom here, probably Karthink :]) but we also discussed the potential to extend async latex preview to other kinds of text: latex, images, html files, pdf files, typst code snippets, etc - We have one unfinished (but working) attempt to preview html inline in = Org mode https://github.com/alejandrogallo/org-inline-webkit It would be nice is somebody take that repo and finish it to the state ready for merging into Org mode - HTML previews will move towards feature parity with ipython and similar notebooks that can produce interactive graphs and visualizations - Also, the way WIP async preview library works (in its core) is not limited to previewing latex. One can implement other kinds of previews, like typst. All that is needed is a way to qeury typst fragment at point in some way - In fact, we already have some proof of concept code that goes even further and extends LaTeX (for now, just LaTeX) preview functionality _outside_ Org mode, in arbitrary buffers https://list.orgmode.org/orgmode/87edbhljr7.FSF@hyperspace/ - Ideally, the preview might be made almost completely independent from what major mode is currently in use - This might be done using ~thingatpt~ Emacs library (built-in) - A recall Yuan Fu talking about integrating thingatpt with tree-sitter https://yhetil.org/emacs-devel/5E7F2A94-4377-45C0-8541-7F59F3B54BA1@gmai= l.com/ - And we can integrate Org mode parser with thingatpt as well (we already partially do, see ~org--link-at-point~ and ~org--bounds-of-link-at-point~) - P.S. It has been really long discussion :P :comments: [17:44] Ihor Radchenko : Official start in 15 minutes [17:44] Ihor Radchenko : Meanwhile, check out the latest Org news at https:= //orgmode.org/ [17:44] Ihor Radchenko : We added Sacha's News to the front page for conven= ience :) [17:57] Tim : Hi, this is Tim [18:01] suhail : nothing on my side, but if we're short on topics, i'll men= tion 3 recent items i posted in the mailing list: - https://yhetil.org/orgmode/87h6e134sp.fsf@gmail.com/ - https://yhetil.org/orgmode/87cyop3445.fsf@gmail.com/ - https://yhetil.org/orgmode/87cyonhuq3.fsf@gmail.com/ [18:02] suhail : one bug report, and the other 2 it's not clear if they are= bugs or if user-error [18:03] Karl Voit (publicvoit) : I just wrote a feature request for org-rif= le: https://github.com/alphapapa/org-rifle/issues/85 maybe somebody has an = answer so that we could collaboratively close this with a proper answer? [18:11] Ihor Radchenko : org-fold-core-style [18:14] Ihor Radchenko : https://github.com/alphapapa/org-ql/tree/master pr= ovides helm and vertico interfaces [18:14] Ihor Radchenko : and allows tag/outline filtering [18:14] Ihor Radchenko : and much more [18:17] Tim : 78170 99% - command-execute 78124 99% - funcall-interactively 78124 99% - org-set-tags-command 78124 99% - org-get-buffer-tags 68151 86% - org-element-cache-map 38554 49% - org-element-at-point 3839 4% - org-element--parse-to 2030 2% - org-element--cache-find 678 0% # 21 0% org-knuth-hash 99 0% # 44 0% org-element--cache-active-p 117 0% org-element--cache-active-p 101 0% derived-mode-p 41 0% org-element--cache-sync 39 0% org-element--cache-verify-element 2881 3% - # [...] [18:18] Ihor Radchenko : (setq org-element--cache-self-verify t) [18:24] Ihor Radchenko : helm-org-ql-source [18:34] Ilya Chernyshov : Hi [18:35] Ihor Radchenko : https://orgmode.org/worg/dev/org-element-api.html [18:38] Ihor Radchenko : org-link-open [18:40] karthink : I see [18:40] karthink : Can you provide more details about what you're planning = to do? [18:41] karthink : Most of the changes in our feature branch are limited to= the new org-latex-preview.el file [18:41] karthink : M-1 [18:41] karthink : Use M-1 or M-2 [18:43] karthink : This is a great idea [18:43] Karl Voit (publicvoit) : Sorry, I need to leave already - thanks fo= r all the input I need to digest in the next days ;-) [18:44] Ilya Chernyshov : If I remember correctly, org-element hasn't cache= d active/inactive timestamps inside body (except for scheduled, deadline). = Has this been changed in the latest release? That could increase the speed = of subsequent search in org-agenda These problems are already solved in org-ql, the subsequent search results = are instant I also heard that you and alphapapa are planning to merge org-ql to org cor= e. Do you think it will happen this year? [19:02] suhail : i was under the impression that the byte-compiler treats t= op-level requires especially. and so if a require is within a defun, it's h= andled somewhat differently [19:02] suhail : and that the latter is to be avoided (for reasons independ= ent from performance considerations) [19:09] Ihor Radchenko : https://github.com/alphapapa/org-ql/issues/409 [19:09] Ihor Radchenko : https://github.com/alphapapa/ts.el/issues/29 [19:17] oylenshpeegul : I have another meeting coming up. Thanks, Ihor! Tha= nks, all! [19:23] karthink : I can hear you Ihor [19:23] Welcome to [[bbb:OrgMeetup]]!

For help on using B= igBlueButton see these (short) tutorial videos.

To join the au= dio bridge click the phone button. Use a headset to avoid causing backgrou= nd noise for others.

This server is running BigBlueButton. [19:26] visuwesh : Does the org-create-formula-image function in org-compat= in the async latex preview branch need more work? It seems like org--get-= display-dpi is accidentally removed, and there are other incompatibilities = like new format options in org-preview-latex-default-process, fg=3Dauto not= being correctly handled, etc. This function is still used in pdf-tools, and comint-mime. I am not sure h= ow pdf-tools can handle the new async code since it needs to show the image= in a tooltip. [19:35] visuwesh : Yea, its there but it odesnt work [19:35] visuwesh : yes and some more [19:35] visuwesh : The problem is the old user option is an alias to the ne= w option [19:35] visuwesh : so everything breaks [19:36] visuwesh : And you might need to look at handling auto in uhh [19:36] visuwesh : one min [19:36] visuwesh : yea, i will send a mail to the ML [19:37] visuwesh : emacs-orgmode@gnu.org? [19:37] Ihor Radchenko : Yup, just report this to Org mailing list [19:37] Ihor Radchenko : https://orgmode.org/manual/Feedback.html#Feedback [19:37] karthink : Or here: https://github.com/tecosaur/org-latex-preview-todos/issues [19:37] karthink : But mailing list is fine [19:37] visuwesh : ah didn't know about the github repo, thx for the link [19:38] Ihor Radchenko : https://github.com/vedang/pdf-tools/issues/283 [19:38] visuwesh : When creating the tooltip for text annotations [19:38] visuwesh : seee pdf-annot-print-annotation-latex-maybe [19:38] visuwesh : Yea [19:40] visuwesh : Will the async code work nicely with tooltips though? [19:41] visuwesh : AHHH [19:41] visuwesh : okkk, thanks for the hint [19:42] visuwesh : btw, the same problem also exists in comint-mime but the= latex functionality there is disabled by default [19:42] visuwesh : https://github.com/astoff/comint-mime/blob/ca278064dac43= 1b8c3f72ab1b860bf9c9b8d8d60/comint-mime.el#L179 [19:43] visuwesh : yep [19:43] visuwesh : these two were the packages that burnt me when i switche= d to the async code mainly because of the new api [19:47] visuwesh : yea, im happy to not have to wait for like 15 mins when = i need to regen previews for whatever reason [19:48] visuwesh : (when i was using the old sync code) [19:52] visuwesh : afair i dont think so [19:52] visuwesh : its just a temp buffer iirc [19:52] visuwesh : yep [19:54] visuwesh : okay thank you. i will have a look at it when i have th= e itme [19:56] Jeff Trull : I didn't understand everything but I enjoyed it :) [19:57] Jeff Trull : No, interesting things! [19:57] Jeff Trull : magic [19:57] Tim : And again: Thanks for your work! [19:58] Jeff Trull : What's the state of org-element in tables? [19:58] Jeff Trull : It seems like some parts like formulas are treated as = a string IIRC [20:00] Jeff Trull : Also it seemed like the table code used a lot of regex [20:01] karthink : I have a somewhat technical question: where do you use w= hile-no-input in the org-element parsing code? I couldn't find it. (Except= when it gets stuck, creating the parse tree appears to be fully interrupti= ble, it never slows down Emacs) [20:01] Jeff Trull : Would it make sense to integrate element more into org= -table.el? [20:03] Ihor Radchenko : https://elpa.gnu.org/packages/peg.html [20:04] Jeff Trull : ooh I have a completely different use for this. When w= as it added? [20:04] karthink : It was added a couple of months ago I think [20:04] visuwesh : This year iirc [20:04] Ihor Radchenko : https://github.com/ndwarshuis/org-ml [20:05] Jeff Trull : I see. org-element is kind of static [20:05] Jeff Trull : Perfect for exporters, maybe not as good for editing [20:06] Ihor Radchenko : org-element-interpret-data [20:06] Jeff Trull : I got my answer [20:06] Jeff Trull : I made a custom parser for something last year [20:07] Ihor Radchenko : peg is a part of Emacs since few months ago [20:16] Ihor Radchenko : https://list.orgmode.org/orgmode/875xwqj4tl.fsf@lo= calhost/ [20:16] Ihor Radchenko : inline special blocks [20:23] Ihor Radchenko : https://github.com/alejandrogallo/org-inline-webkit [20:23] Ihor Radchenko : preview html [20:23] Ihor Radchenko : inline, with rendering [20:23] Ihor Radchenko : (proof of concept - it would be nice if someone fi= nishes it up to be ready for inclusion) [20:25] Ihor Radchenko : links are supported by thingatpt [20:33] Ihor Radchenko : org--link-at-point [20:33] Ihor Radchenko : org--bounds-of-link-at-point [20:34] Jeff Trull : thanks for the interesting content I'll see you all ne= xt time [20:35] karthink : I have to leave as well, thank you for the meetup Ihor :end: --=20 Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at . Support Org development at , or support my work at