* Using org-latex-preview in other major modes @ 2024-04-07 7:22 Tony Zorman 2024-04-07 11:59 ` Ihor Radchenko 2024-04-07 17:48 ` Karthik Chikmagalur 0 siblings, 2 replies; 11+ messages in thread From: Tony Zorman @ 2024-04-07 7:22 UTC (permalink / raw) To: emacs-orgmode; +Cc: karthikchikmagalur [-- Attachment #1: Type: text/plain, Size: 1632 bytes --] Hi, I'm interested in adjusting org-latex-preview to work in other major modes (most notably LaTeX-mode itself, so one can get rid of preview.el). On the advice of Karthik (Cc'd) I'll move the discussion from private mails to this list, so more people who might be interested can join in. I've prodded the code a little bit and, mostly just following [1], managed to implement basic previews in a relatively straightforward fashion. Attached is a proof-of-concept—aptly named latex-latex-preview.el :) The main user facing functions so far are 'latex-latex-preview' to preview the maths fragment at point, and 'latex-latex-preview-region' to preview all fragments in a region. Both functions currently ignore things like numbered equations and environments, though both do not seem exceptionally difficult to add back in. There's also a stub 'latex-latex-preview-auto-mode' minor mode. Stepping in and out of already rendered environments works fine, with the preview being regenerated if needed; only the implementation of 'org-latex-preview-auto--regenerate-overlay' had to change for that. The mode does not currently feature live-previews. The innards of 'org-latex-preview-auto--detect-fragments-in-change' and 'org-latex-preview-live--setup' look a bit more org-specific, with queries to 'org-element-*' functions all over the place, but all of that looks fixable—at least from afar. I will continue prodding the code a little bit and will report back with any bumps that are hit along the way. Tony [1]: https://abode.karthinks.com/org-latex-preview/latex-preview-everywhere.html [-- Attachment #2: latex-latex-preview.el --] [-- Type: application/emacs-lisp, Size: 7445 bytes --] [-- Attachment #3: Type: text/plain, Size: 44 bytes --] -- Tony Zorman | https://tony-zorman.com/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Using org-latex-preview in other major modes 2024-04-07 7:22 Using org-latex-preview in other major modes Tony Zorman @ 2024-04-07 11:59 ` Ihor Radchenko 2024-04-07 18:06 ` Karthik Chikmagalur 2024-04-07 17:48 ` Karthik Chikmagalur 1 sibling, 1 reply; 11+ messages in thread From: Ihor Radchenko @ 2024-04-07 11:59 UTC (permalink / raw) To: Tony Zorman; +Cc: emacs-orgmode, karthikchikmagalur Tony Zorman <mail@tony-zorman.com> writes: > I'm interested in adjusting org-latex-preview to work in other major > modes (most notably LaTeX-mode itself, so one can get rid of preview.el). > On the advice of Karthik (Cc'd) I'll move the discussion from private > mails to this list, so more people who might be interested can join in. > ... > [1]: https://abode.karthinks.com/org-latex-preview/latex-preview-everywhere.html Abstracting away previews is certainly welcome. RMS explicitly asked Org mode team to work towards this goal: https://list.orgmode.org/E1kIkxv-0007iy-Av@fencepost.gnu.org/ Ideally, we should have Org-independent library that does the previews, and an Org-specific code that re-uses this library. Eventually, we can move the generic library to Emacs core. We may go even further, and extend the previews to be not just for LaTeX. Might as well preview html/image links/pdf links/etc. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Using org-latex-preview in other major modes 2024-04-07 11:59 ` Ihor Radchenko @ 2024-04-07 18:06 ` Karthik Chikmagalur 2024-04-09 14:11 ` Ihor Radchenko 0 siblings, 1 reply; 11+ messages in thread From: Karthik Chikmagalur @ 2024-04-07 18:06 UTC (permalink / raw) To: Ihor Radchenko, Tony Zorman; +Cc: emacs-orgmode > Abstracting away previews is certainly welcome. > RMS explicitly asked Org mode team to work towards this goal: > https://list.orgmode.org/E1kIkxv-0007iy-Av@fencepost.gnu.org/ I agree with RMS about this, and this was on our minds when we wrote org-latex-preview. The basic previewing process is explicitly written to be Org-agnostic, but we weren't as rigorous when adding the fancier features (like live previews). > Ideally, we should have Org-independent library that does the previews, > and an Org-specific code that re-uses this library. Eventually, we can > move the generic library to Emacs core. Here is the plan Timothy and I have discussed: 1. Merge org-latex-preview in its current state and continue to fix bugs/edge cases. 2. Write an external package reimplementing in a more generic way the parts of the API that are Org-specific. This external package will depend (heavily) on org-latex-preview. 3. Solicit from the community integrations of this with other major-modes using this generic API. 4. Once this API is stable, replace the corresponding parts of org-latex-preview. 5. Propose moving everything but the Org-specific parts to a `latex-preview.el' package included with Emacs. From Tony's proof of concept, I think step 2 might be unnecessary and we can incrementally modify org-latex-preview instead. > We may go even further, and extend the previews to be not just for > LaTeX. Might as well preview html/image links/pdf links/etc. I agree in principle but I think this is difficult to do with org-latex-preview because the async process chain and overlay handling are highly tuned for low-latency LaTeX processing.[1] I do think Emacs could use a generic link-preview package, with an org-link-preview adapter for Org mode. Karthik [1]: As low-latency as possible from Emacs without modifying the LaTeX compiler or image renderer, as for example TeXpresso does. https://github.com/let-def/texpresso ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Using org-latex-preview in other major modes 2024-04-07 18:06 ` Karthik Chikmagalur @ 2024-04-09 14:11 ` Ihor Radchenko 0 siblings, 0 replies; 11+ messages in thread From: Ihor Radchenko @ 2024-04-09 14:11 UTC (permalink / raw) To: Karthik Chikmagalur; +Cc: Tony Zorman, emacs-orgmode Karthik Chikmagalur <karthikchikmagalur@gmail.com> writes: >> We may go even further, and extend the previews to be not just for >> LaTeX. Might as well preview html/image links/pdf links/etc. > > I agree in principle but I think this is difficult to do with > org-latex-preview because the async process chain and overlay handling are > highly tuned for low-latency LaTeX processing.[1] > > I do think Emacs could use a generic link-preview package, with an > org-link-preview adapter for Org mode. What I had in mind is something yet more generic: 1. preview.el abstracting away an API to toggle previews in buffer/region and display previews in separate buffer/echo area. It will provide customization on what kinds of previews are to be displayed; and the underlying backends will add themselves to some kind of hook that will do the actual work. 2. latex-preview.el with generic API to preview latex fragments in particular (or maybe even generic fragments that require to run convertor code -> image). 3. org-latex-preview with Org specific code. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Using org-latex-preview in other major modes 2024-04-07 7:22 Using org-latex-preview in other major modes Tony Zorman 2024-04-07 11:59 ` Ihor Radchenko @ 2024-04-07 17:48 ` Karthik Chikmagalur 2024-04-08 6:23 ` Tony Zorman 1 sibling, 1 reply; 11+ messages in thread From: Karthik Chikmagalur @ 2024-04-07 17:48 UTC (permalink / raw) To: Tony Zorman, emacs-orgmode Hi Tony, Just tried it out, it's very promising! Thanks for taking the initiative on this. > I've prodded the code a little bit and, mostly just following [1], > managed to implement basic previews in a relatively straightforward > fashion. Attached is a proof-of-concept—aptly named > latex-latex-preview.el :) The main user facing functions so far are > 'latex-latex-preview' to preview the maths fragment at point, and > 'latex-latex-preview-region' to preview all fragments in a region. Both > functions currently ignore things like numbered equations and > environments, though both do not seem exceptionally difficult to add > back in. There are two issues with numbering: - providing an Org-agnostic API point to attach a numbering table to, and - calculating the new numbering table in LaTeX (or other major-modes). For The first issue, we need a way to provide an updated numbering table during the auto-regeneration of edited fragments. Currently this is done implicitly by calling `org-latex-preview--place-from-elements' from the `--regenerate-overlay` function. The second requires fast numbering table updates. We do it in Org by mapping over the org-element cache (see `org-latex-preview--get-numbered-environments'). Even this is too slow sometimes, so we suspend numbering updates when live-previewing until the cursor exits the fragment. Parsing the LaTeX buffer from point to the end when (re)generating each preview is going to be too slow, so you'll have to create some kind of cache and update it incrementally. > There's also a stub 'latex-latex-preview-auto-mode' minor mode. Stepping > in and out of already rendered environments works fine, with the preview > being regenerated if needed; only the implementation of > 'org-latex-preview-auto--regenerate-overlay' had to change for that. This was a pleasant surprise, I was expecting more trouble here. > The mode does not currently feature live-previews. The innards of > 'org-latex-preview-auto--detect-fragments-in-change' and > 'org-latex-preview-live--setup' look a bit more org-specific, with > queries to 'org-element-*' functions all over the place, but all of > that looks fixable—at least from afar. I will continue prodding the > code a little bit and will report back with any bumps that are hit > along the way. `org-latex-preview-auto--detect-fragments-in-change' is written for speed. It only does quick text-matching and is thus mostly Org-agnostic, except for a call to a numbering calculation near the end. This should be easy to adapt. The problem is the function it calls, `org-latex-preview-auto--maybe-track-element-here', which finds the bounds of the inserted LaTeX fragment using org-element and conditionally sets up a preview overlay. You will need an equivalent of this for LaTeX-mode. Since you have auto-mode working already, live previews should be quite easy to add. From what I can see, you only need to provide your own `org-latex-preview-live--ensure-overlay'. This function creates the preview overlay next to or under the LaTeX fragment. All the other live preview code only changes overlay properties or calls `--regenerate-overlay`, which you've already implemented. Karthik ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Using org-latex-preview in other major modes 2024-04-07 17:48 ` Karthik Chikmagalur @ 2024-04-08 6:23 ` Tony Zorman 2024-04-08 6:36 ` Karthik Chikmagalur 0 siblings, 1 reply; 11+ messages in thread From: Tony Zorman @ 2024-04-08 6:23 UTC (permalink / raw) To: Karthik Chikmagalur, emacs-orgmode On Sun, Apr 07 2024 10:48, Karthik Chikmagalur wrote: > [… 14 lines elided …] > > There are two issues with numbering: > - providing an Org-agnostic API point to attach a numbering table to, and > - calculating the new numbering table in LaTeX (or other major-modes). > > For The first issue, we need a way to provide an updated numbering table > during the auto-regeneration of edited fragments. Currently this is > done implicitly by calling `org-latex-preview--place-from-elements' from > the `--regenerate-overlay` function. > > The second requires fast numbering table updates. We do it in Org by > mapping over the org-element cache (see > `org-latex-preview--get-numbered-environments'). Even this is too slow > sometimes, so we suspend numbering updates when live-previewing until > the cursor exits the fragment. Parsing the LaTeX buffer from point to > the end when (re)generating each preview is going to be too slow, so > you'll have to create some kind of cache and update it incrementally. Thanks for taking a look! It indeed seems that caching will be the most work to properly implement. However, I'm quite happy to not care about numbers for now and make sure the other stuff works correctly first before tackling this. > [… 14 lines elided …] > > `org-latex-preview-auto--detect-fragments-in-change' is written for > speed. It only does quick text-matching and is thus mostly Org-agnostic, > except for a call to a numbering calculation near the end. This should > be easy to adapt. The problem is the function it calls, > `org-latex-preview-auto--maybe-track-element-here', which finds the > bounds of the inserted LaTeX fragment using org-element and > conditionally sets up a preview overlay. You will need an equivalent of > this for LaTeX-mode. > > Since you have auto-mode working already, live previews should be quite > easy to add. From what I can see, you only need to provide your own > `org-latex-preview-live--ensure-overlay'. This function creates the > preview overlay next to or under the LaTeX fragment. All the other live > preview code only changes overlay properties or calls > `--regenerate-overlay`, which you've already implemented. Besides `org-latex-preview-auto--maybe-track-element-here`, I was mostly talking about `org-latex-preview-live--src-buffer-setup` and `org-latex-preview-live--ensure-overlay`, which I think will be the main targets. The latter seems to be quite a straightforward translation (especially my current constraints of not caring about environments), but the former seems to require a bit more work. Then again, maybe I'm being too negative—after taking another quick look at the code I think you're right in that this should not be impossible to overcome. I'll try to conjure up some time this week to get live previews up and running; will report back if I hit any unforeseen major bumps. Tony -- Tony Zorman | https://tony-zorman.com/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Using org-latex-preview in other major modes 2024-04-08 6:23 ` Tony Zorman @ 2024-04-08 6:36 ` Karthik Chikmagalur 2024-04-09 20:06 ` Tony Zorman 0 siblings, 1 reply; 11+ messages in thread From: Karthik Chikmagalur @ 2024-04-08 6:36 UTC (permalink / raw) To: Tony Zorman, emacs-orgmode > Thanks for taking a look! It indeed seems that caching will be the most > work to properly implement. However, I'm quite happy to not care about > numbers for now and make sure the other stuff works correctly first > before tackling this. Sure, this is a disproportionate amount of work for a minor feature. I suggest setting `org-latex-preview-numbered' to nil, FWIW. The previews look better this way. > Besides `org-latex-preview-auto--maybe-track-element-here`, I was mostly > talking about `org-latex-preview-live--src-buffer-setup` and > `org-latex-preview-live--ensure-overlay`, which I think will be the main > targets. The latter seems to be quite a straightforward translation > (especially my current constraints of not caring about environments), You can ignore `org-latex-preview-live--src-buffer-setup', this is for previewing when using `org-edit-special' (C-c '), which does not apply to any other major mode. So `org-latex-preview-live--ensure-overlay' is the only function you need to rewrite, which should be easy. Keep us updated. Good luck! Karthik ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Using org-latex-preview in other major modes 2024-04-08 6:36 ` Karthik Chikmagalur @ 2024-04-09 20:06 ` Tony Zorman 2024-04-21 19:10 ` Tony Zorman 0 siblings, 1 reply; 11+ messages in thread From: Tony Zorman @ 2024-04-09 20:06 UTC (permalink / raw) To: Karthik Chikmagalur, emacs-orgmode On Sun, Apr 07 2024 23:36, Karthik Chikmagalur wrote: > You can ignore `org-latex-preview-live--src-buffer-setup', this is for > previewing when using `org-edit-special' (C-c '), which does not apply > to any other major mode. So `org-latex-preview-live--ensure-overlay' is > the only function you need to rewrite, which should be easy. Oh, indeed, you are right; seems like it only looked intimidating! I quickly hacked this in just now, and it works just fine. As a very brief summary, one currently needs to—in addition to supplying a preamble and a way to recognise maths and environments—patch the following functions: · org-latex-preview--place-from-elements · org-latex-preview-auto--regenerate-overlay · org-latex-preview-auto--maybe-track-element-here · org-latex-preview-live--ensure-overlay The last two are only needed for the "live" part of the live previews to work. I will clean the code up a little bit (hopefully by the weekend, but no promises), and try to add support for LaTeX environments instead of just maths. I reckon that once this is finished one can get a pretty good idea in which direction org-latex-preview.el would need to be nudged to more easily facilitate this kind of integration. After that I guess there's still the issue of caching and numbering, but I'll cross that bridge once we get to it :) Tony -- Tony Zorman | https://tony-zorman.com/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Using org-latex-preview in other major modes 2024-04-09 20:06 ` Tony Zorman @ 2024-04-21 19:10 ` Tony Zorman 2024-04-22 3:41 ` Karthik Chikmagalur 0 siblings, 1 reply; 11+ messages in thread From: Tony Zorman @ 2024-04-21 19:10 UTC (permalink / raw) To: Karthik Chikmagalur, emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 1338 bytes --] On Tue, Apr 09 2024 22:06, Tony Zorman wrote: > As a very brief summary, one currently needs to—in addition to supplying > a preamble and a way to recognise maths and environments—patch the > following functions: > > · org-latex-preview--place-from-elements > · org-latex-preview-auto--regenerate-overlay > · org-latex-preview-auto--maybe-track-element-here > · org-latex-preview-live--ensure-overlay > > The last two are only needed for the "live" part of the live previews to > work. > > I will clean the code up a little bit (hopefully by the weekend, but no > promises), Whoops! Anyways, before I put this off for much longer, there is some more code attached. Live previews (and general environments) work now, and besides the above mentioned points there were no new surprises waiting—at least for getting the basic functionality to work. I did notice that precompilation being a bit flaky. In the end, this was the result of importing local packages; with the precompilation happening somewhere in /tmp/, access to these files wasn't given. I haven't looked too closely into this—and it's probably a use-case that's specific to LaTeX-mode—but it seems worth considering (lots of people carry one big style file around that they include in all of their projects). Tony [-- Attachment #2: latex-latex-preview.el --] [-- Type: application/emacs-lisp, Size: 10856 bytes --] [-- Attachment #3: Type: text/plain, Size: 44 bytes --] -- Tony Zorman | https://tony-zorman.com/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Using org-latex-preview in other major modes 2024-04-21 19:10 ` Tony Zorman @ 2024-04-22 3:41 ` Karthik Chikmagalur 2024-04-22 9:29 ` Tony Zorman 0 siblings, 1 reply; 11+ messages in thread From: Karthik Chikmagalur @ 2024-04-22 3:41 UTC (permalink / raw) To: Tony Zorman, emacs-orgmode > Anyways, before I put this off for much longer, there is some more code > attached. Live previews (and general environments) work now, and besides > the above mentioned points there were no new surprises waiting—at least > for getting the basic functionality to work. Thank you for the update! This looks promising. I'll test it when I have time. > I did notice that precompilation being a bit flaky. In the end, this > was the result of importing local packages; with the precompilation > happening somewhere in /tmp/, access to these files wasn't given. > haven't looked too closely into this—and it's probably a use-case > that's specific to LaTeX-mode—but it seems worth considering Sorry, I should have mentioned this in my original write-up. There is a reason for this, as explained below. This issue should not be happening in most cases though, so I'll need more details to help. > (lots of people carry one big style file around that they include in > all of their projects). This is true of Org mode files that are intended for LaTeX export too, but we avoid this problem in Org. 1. Why we precompile in /tmp: Precompilation is a blocking operation, so we want to do it as infrequently as possible. Imagine if LaTeX previews in every Org buffer/file you opened required a precompile step that blocked Emacs for 3 seconds. Precompiled dump files are identified by a hash that includes the preamble and a default-directory. If we precompile in /tmp (or some other fixed directory), we can reuse dump files for all Org buffers that have the same preamble, irrespective of where they're located. Precompiled files are stored in org-persist and don't expire for a long time. So we'd like to avoid generating loads of them, and reuse them whenever possible. 1a. Why is precompilation a blocking operation? It doesn't have to be, but including it in the async chain is a little annoying. It can be made async in the future. 2. What happens if the user includes a directory-local .sty, .cls or other tex files in the header? When precompiling, we check the header to see if it has an \input{} or \include{} statement. If it does, we assume that there are local dependencies and precompile it in the default-directory. See org-latex-preview--create-tex-file for the implementation of this logic. Is there some other common way that a LaTeX file can have local dependencies? 3. How to force precompilation to occur in the default-directory instead of in /tmp: Based on 1 and 2, the easiest way would be to add an \input{} or \include{} statement to the preamble. The current test is very simple, you can even place it in a LaTeX comment and it should work. Karthik ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Using org-latex-preview in other major modes 2024-04-22 3:41 ` Karthik Chikmagalur @ 2024-04-22 9:29 ` Tony Zorman 0 siblings, 0 replies; 11+ messages in thread From: Tony Zorman @ 2024-04-22 9:29 UTC (permalink / raw) To: Karthik Chikmagalur, emacs-orgmode On Sun, Apr 21 2024 20:41, Karthik Chikmagalur wrote: >> Anyways, before I put this off for much longer, there is some more code >> attached. Live previews (and general environments) work now, and besides >> the above mentioned points there were no new surprises waiting—at least >> for getting the basic functionality to work. > > Thank you for the update! This looks promising. I'll test it when I > have time. > >> I did notice that precompilation being a bit flaky. In the end, this >> was the result of importing local packages; with the precompilation >> happening somewhere in /tmp/, access to these files wasn't given. >> haven't looked too closely into this—and it's probably a use-case >> that's specific to LaTeX-mode—but it seems worth considering > > Sorry, I should have mentioned this in my original write-up. There is a > reason for this, as explained below. This issue should not be happening > in most cases though, so I'll need more details to help. > >> (lots of people carry one big style file around that they include in >> all of their projects). > > This is true of Org mode files that are intended for LaTeX export too, > but we avoid this problem in Org. > > 1. Why we precompile in /tmp: > > Precompilation is a blocking operation, so we want to do it as > infrequently as possible. Imagine if LaTeX previews in every Org > buffer/file you opened required a precompile step that blocked Emacs > for 3 seconds. > > Precompiled dump files are identified by a hash that includes the > preamble and a default-directory. If we precompile in /tmp (or some > other fixed directory), we can reuse dump files for all Org buffers > that have the same preamble, irrespective of where they're located. > > Precompiled files are stored in org-persist and don't expire for a > long time. So we'd like to avoid generating loads of them, and reuse > them whenever possible. > > 1a. Why is precompilation a blocking operation? > > It doesn't have to be, but including it in the async chain is a > little annoying. It can be made async in the future. > > 2. What happens if the user includes a directory-local .sty, .cls or > other tex files in the header? > > When precompiling, we check the header to see if it has an \input{} > or \include{} statement. If it does, we assume that there are local > dependencies and precompile it in the default-directory. See > org-latex-preview--create-tex-file for the implementation of this > logic. > > Is there some other common way that a LaTeX file can have local > dependencies? \usepackage accepts relative paths, although I think it will generate a warning since the package called will provide «pkg» instead of .relative/path/to/«pkg». Either way, it's not uncommon to see things like \usepackage[opt1, opt2=lf]{style/nested/too/deeply/style} Modulo Windows, one might check for slashes in \usepackage invocations, since I don't think TeX packages can have those. (Users *should* really add /path/to/«pkg» to LaTeX's search path, but that will never happen :)) > 3. How to force precompilation to occur in the default-directory instead > of in /tmp: > > Based on 1 and 2, the easiest way would be to add an \input{} or > \include{} statement to the preamble. The current test is very > simple, you can even place it in a LaTeX comment and it should work. This does indeed work; thanks! Tony -- Tony Zorman | https://tony-zorman.com/ ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2024-04-22 9:30 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-04-07 7:22 Using org-latex-preview in other major modes Tony Zorman 2024-04-07 11:59 ` Ihor Radchenko 2024-04-07 18:06 ` Karthik Chikmagalur 2024-04-09 14:11 ` Ihor Radchenko 2024-04-07 17:48 ` Karthik Chikmagalur 2024-04-08 6:23 ` Tony Zorman 2024-04-08 6:36 ` Karthik Chikmagalur 2024-04-09 20:06 ` Tony Zorman 2024-04-21 19:10 ` Tony Zorman 2024-04-22 3:41 ` Karthik Chikmagalur 2024-04-22 9:29 ` Tony Zorman
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).