Large source block causes org-mode to be unusable

Large source block causes org-mode to be unusable

[Léo Ackermann]

[-- Attachment #1.1: Type: text/plain, Size: 918 bytes --]

Dear all,

I am working in an org-file of reasonable size (<2000 lines): my first
paper written in org-mode. Everything fine (and fast) until I started to
add `#+BEGIN_proof / #+END_proof` within my .org to make my .pdf export
prettier. This caused the editing of the proofs to be very slow: navigation
within the proof is fast but adding/removing any char takes around 4s per
char.
It seems that the fontify function is responsible for that (see
screenshot). As far as I understand, this function tries to fontify the
whole block as soon as a single char is modified. In my case, it then tries
to fontify a whole proof (~4 pages in my .pdf, with many LaTeX formulas)
several times per second...

Is there a way to make this fontify function to act "around my cursor" ?

Best,
Leo

--
Here the profiler's report when adding a single char
[image: image.png]
 Spec.
- Emacs 27.2 (doom emacs 2.0.9) (I just updated them)

[-- Attachment #1.2: Type: text/html, Size: 1189 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 115679 bytes --]

Re: Large source block causes org-mode to be unusable

[John Hendy]

[-- Attachment #1.1: Type: text/plain, Size: 2679 bytes --]

Seems known, assuming this is the same thing. I have not experienced the
phenomenon or used proof blocks, so I'm just going by google-fu matching
(which finds a lot):

https://emacs.stackexchange.com/questions/46561/org-mode-9-too-slow-with-long-code-blocks
https://emacs.stackexchange.com/questions/62376/slow-markdown-mode-as-emacs-spends-lots-of-time-fontifying
https://emacs.stackexchange.com/questions/46110/help-needed-to-debug-very-slow-markdown-mode
https://www.reddit.com/r/emacs/comments/c28svb/has_emacs_always_been_this_slow_at_highlighting/
https://lists.gnu.org/archive/html/emacs-orgmode/2015-07/msg00215.html

As with all things like this, I'd work through the old information first to
understand the true problem, if any improvements/workarounds exist, etc.
Then this lands in some bucket of outcomes:
- it's a novel find and therefore a bug (due diligence searching says this
is not the case)
- it's known bad, but not addressed (due to time or inability), but it can
be worked around (nuggets exist in the links above as well as other results
if you search "emacs fontify proof block slow")
- a true "fix" is possible, in which case this would require pinpointing
the exact problem and asking for alternatives or proposing a better/faster
way to fontify


Hope that helps,
John

P.S. This looks independent to Org, so I'm not sure if this is the best
place for the eventual resolution as this may have nothing to do with Org?
I'm no emacs guru, but from my skim I'm guessing org just pulls in some
other package vs. having it's own fontification engine.

On Mon, Jun 21, 2021 at 1:31 PM Léo Ackermann <leo.komba@gmail.com> wrote:

> Dear all,
>
> I am working in an org-file of reasonable size (<2000 lines): my first
> paper written in org-mode. Everything fine (and fast) until I started to
> add `#+BEGIN_proof / #+END_proof` within my .org to make my .pdf export
> prettier. This caused the editing of the proofs to be very slow: navigation
> within the proof is fast but adding/removing any char takes around 4s per
> char.
> It seems that the fontify function is responsible for that (see
> screenshot). As far as I understand, this function tries to fontify the
> whole block as soon as a single char is modified. In my case, it then tries
> to fontify a whole proof (~4 pages in my .pdf, with many LaTeX formulas)
> several times per second...
>
> Is there a way to make this fontify function to act "around my cursor" ?
>
> Best,
> Leo
>
> --
> Here the profiler's report when adding a single char
> [image: image.png]
>  Spec.
> - Emacs 27.2 (doom emacs 2.0.9) (I just updated them)
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 3911 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 115679 bytes --]

Re: Large source block causes org-mode to be unusable

[Sébastien Miquel]

Hi Léo,

Léo Ackermann writes:
> I am working in an org-file of reasonable size (<2000 lines): my first 
> paper written in org-mode. Everything fine (and fast) until I started 
> to add `#+BEGIN_proof / #+END_proof` within my .org to make my .pdf 
> export prettier. This caused the editing of the proofs to be very 
> slow: navigation within the proof is fast but adding/removing any char 
> takes around 4s per char.
> It seems that the fontify function is responsible for that (see 
> screenshot). As far as I understand, this function tries to fontify 
> the whole block as soon as a single char is modified. In my case, it 
> then tries to fontify a whole proof (~4 pages in my .pdf, with many 
> LaTeX formulas) several times per second...
You can try setting org-highlight-latex-and-related to '(latex) instead of
'(native).

Even with this setting, latex fontification in special blocks is slow. The
reason being that the whole block has the `font-lock-multiline' 
property, hence
every char insertion triggers refontification of the whole block.

In the function `org-do-latex-and-related', I commented out the second 
condition
of the `cond' form, which makes calls to `face-at-point'. This yields a
significant speedup, and was enough to make things bearable in my cases. 
You can
also try to simplify the latex regexp.

If you try these, I'd be interested to hear how much of an improvement 
theymake.

Regards,

-- 
Sébastien Miquel

Re: Large source block causes org-mode to be unusable

[John Kitchin]

A quick and dirty way to fix this might be an include file, i.e. move
the block out of your manuscript file into a separate org file, and then
just include it.

Léo Ackermann <leo.komba@gmail.com> writes:

> Dear all,
>
> I am working in an org-file of reasonable size (<2000 lines): my first
> paper written in org-mode. Everything fine (and fast) until I started to
> add `#+BEGIN_proof / #+END_proof` within my .org to make my .pdf export
> prettier. This caused the editing of the proofs to be very slow: navigation
> within the proof is fast but adding/removing any char takes around 4s per
> char.
> It seems that the fontify function is responsible for that (see
> screenshot). As far as I understand, this function tries to fontify the
> whole block as soon as a single char is modified. In my case, it then tries
> to fontify a whole proof (~4 pages in my .pdf, with many LaTeX formulas)
> several times per second...
>
> Is there a way to make this fontify function to act "around my cursor" ?
>
> Best,
> Leo


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his

Re: Large source block causes org-mode to be unusable

[John Hendy]

[-- Attachment #1: Type: text/plain, Size: 1978 bytes --]

Indeed. The top google hit for "emacs fontify proof block slow" I provided
says exactly that :)

https://emacs.stackexchange.com/questions/46561/org-mode-9-too-slow-with-long-code-blocks
"""
That said, I think keeping 2000 lines of source code inside an org src
block is neither a standard use case nor a reasonable idea. You can use the
#+INCLUDE ... src ... directive or split the block into a number of more
reasonably sized cells. If you insist on keeping the entire block, then
disable native highlighting of src code blocks.
"""

On Mon, Jun 21, 2021 at 2:23 PM John Kitchin <jkitchin@andrew.cmu.edu>
wrote:

> A quick and dirty way to fix this might be an include file, i.e. move
> the block out of your manuscript file into a separate org file, and then
> just include it.
>
> Léo Ackermann <leo.komba@gmail.com> writes:
>
> > Dear all,
> >
> > I am working in an org-file of reasonable size (<2000 lines): my first
> > paper written in org-mode. Everything fine (and fast) until I started to
> > add `#+BEGIN_proof / #+END_proof` within my .org to make my .pdf export
> > prettier. This caused the editing of the proofs to be very slow:
> navigation
> > within the proof is fast but adding/removing any char takes around 4s per
> > char.
> > It seems that the fontify function is responsible for that (see
> > screenshot). As far as I understand, this function tries to fontify the
> > whole block as soon as a single char is modified. In my case, it then
> tries
> > to fontify a whole proof (~4 pages in my .pdf, with many LaTeX formulas)
> > several times per second...
> >
> > Is there a way to make this fontify function to act "around my cursor" ?
> >
> > Best,
> > Leo
>
>
> --
> Professor John Kitchin
> Doherty Hall A207F
> Department of Chemical Engineering
> Carnegie Mellon University
> Pittsburgh, PA 15213
> 412-268-7803
> @johnkitchin
> http://kitchingroup.cheme.cmu.edu
> Pronouns: he/him/his
>
>

[-- Attachment #2: Type: text/html, Size: 2738 bytes --]

Re: Large source block causes org-mode to be unusable

[Tom Gillespie]

> That said, I think keeping 2000 lines of source code inside an
> org src block is neither a standard use case nor a reasonable idea.

I would say that it certainly is a standard use case for people who
want to keep everything in a single file (e.g. to simplify
reproducibility and avoid the mess of trying to distribute multiple
files to non-technical users). #+INCLUDE is not a substitute if you
are going to be tangling files, breaks many workflows, and as a result
should rarely be recommended as a solution when src blocks are
involved. Org should definitely be able to handle this case because
there is no reason why performance should be any worse than having a
2000 line file in another buffer.

Org babel has many basic interactivity performance pitfalls that need
to be investigated. I personally have many workarounds for bad emacs
performance degradations related to code executing in the event loop
because I need to get on with the task at hand, but they need to be
fixed, not dismissed.

Re: Large source block causes org-mode to be unusable

[Tim Cross]

Tom Gillespie <tgbugs@gmail.com> writes:

>> That said, I think keeping 2000 lines of source code inside an
>> org src block is neither a standard use case nor a reasonable idea.
>
> I would say that it certainly is a standard use case for people who
> want to keep everything in a single file (e.g. to simplify
> reproducibility and avoid the mess of trying to distribute multiple
> files to non-technical users). #+INCLUDE is not a substitute if you
> are going to be tangling files, breaks many workflows, and as a result
> should rarely be recommended as a solution when src blocks are
> involved. Org should definitely be able to handle this case because
> there is no reason why performance should be any worse than having a
> 2000 line file in another buffer.
>

While I agree with the sentiment, I disagree with the conclusion "there
is no reason why performance should be any worse than having a 2000 line
file in another buffer.". Org mode is pushing right up against the
limits of Emacs' architecture with what it is doing. The way modes and
font-locking work in Emacs was never designed to support multiple modes
and multiple font-locking schemes within a single buffer. The fact org
is able to do this is a testament to the power of Emacs, but this comes
at a cost and that cost is primarily with respect to performance. You
can experience similar performance degradation when you use other
'modes' which try to support multiple modes in a single buffer (like
mmm-mode). This is not specific to org mode, but shows up more due to
the larger user base for org.

> Org babel has many basic interactivity performance pitfalls that need
> to be investigated. I personally have many workarounds for bad emacs
> performance degradations related to code executing in the event loop
> because I need to get on with the task at hand, but they need to be
> fixed, not dismissed.

I think the big challenge here is that the 'fixes' needed by org mode
really need to be at a deeper level inside Emac core architecture. It
isn't something which can be effectively addressed at the org level.
This makes it a much bigger and more complex task which is further
hampered by the need to maintain compatibility with all the existing
modes and libraries.

From what I've seen in the org-devel list, there is on-going work which
is focused on improvements to font-lock efficiency and I think support
for multiple modes is also being considered in some of this work. Other
developments, such as native compilation and pure gtk implementations
may also help in this area. Unfortunately, it may take some time before
we see the benefits of this work as it is complex and non-trivial work. 

In the meantime, we need to consider all possible work-arounds and
tweaks which can help and in some cases, accept some compromise. Having
source blocks which are 2000 lines long is, at this time, a challenge
for org to handle given existing facilities. There are no quick and easy
fixes, but there are some tweaks, such as turning off native
fontification or using include files which can make the system usable.
While not ideal, we have ot work within the limits of the current
architecture while we wait for improvements, which can be expected to be
incremental and slow.  

-- 
Tim Cross

Re: Large source block causes org-mode to be unusable

[Timothy]

[-- Attachment #1.1: Type: text/plain, Size: 1196 bytes --]

Hi Léo,

I think I can actually speak a bit on this. As it stands, babel fontification
operates by:
- sending the src block to a dedicated buffer
- turning on the relevant major mode for the language
- forcing font-lock of the entire buffer
- cloning the font-lock information for the entire buffer back to Org

In essence, every keystroke in a babel block causes font-lock lag equal to
opening the file for the first time. As you can imagine, this adds up.

I think ideally Org would monitor which block you're executing keystrokes in,
and if there are a few in a row in the same org-src block keep the font-lock
buffer alive and incrementally update the content and intelligently copy back
font-lock changes. This has the potential to be much lower latency, but is also
going to be hard to get right.

In the meantime, you could try executing something like this in your Org
buffer ([source]):
,----
| (setq-local jit-lock-defer-time 0.05
|             jit-lock-stealth-time 1)
`----

This sacrifices immediate font-lock accuracy for responsiveness.

All the best,
*Timothy*


[source] <https://tecosaur.github.io/emacs-config/config.html#font-display,code--7>

[-- Attachment #1.2: Type: text/html, Size: 6717 bytes --]

Re: Large source block causes org-mode to be unusable

[Eric S Fraga]

On Monday, 21 Jun 2021 at 14:36, John Hendy wrote:
> [...] or split the block into a number of more reasonably sized
> cells.

This is what I do in practice.  I then use noweb syntax to bring
everything together.  In my case, the long LaTeX blocks tend to be tikz
pictures.

Using (native) for org-highlight-latex-and-related also helps in most
cases.

For me, the remaining performance issue is working with big tables and
I've not found a solution to this.
-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-566-gf0198e
: Latest paper written in org: https://arxiv.org/abs/2106.05096

Re: Large source block causes org-mode to be unusable

[Léo Ackermann]

[-- Attachment #1: Type: text/plain, Size: 1538 bytes --]

Hi all,
Many thanks for those advices!

I am new to org-mode but, is there a reason why #+BEGIN_proof #+END_proof
and other org-latex-special-block are treated as block ?
I mean; those #+... aim, as far as I understand, to give tips to org-export
for prettier exports but nothing else (between those #+... you still need
to write in org). I highlight that org-latex-special-block have nothing to
do (in my point of view) with #+BEGIN_latex / #+END_latex that allow a user
to insert some LaTeX within the document (e.g. Tikz pictures).
Therefore, it might a great idea to take #+BEGIN_proof & co into account
only when it comes to export the document. The proofs/definitions/theorems
will no longer be considered as block, and fast fontification may follow :)

What do you think of this ?

Best,
Leo

Le mar. 22 juin 2021 à 09:55, Eric S Fraga <e.fraga@ucl.ac.uk> a écrit :

> On Monday, 21 Jun 2021 at 14:36, John Hendy wrote:
> > [...] or split the block into a number of more reasonably sized
> > cells.
>
> This is what I do in practice.  I then use noweb syntax to bring
> everything together.  In my case, the long LaTeX blocks tend to be tikz
> pictures.
>
> Using (native) for org-highlight-latex-and-related also helps in most
> cases.
>
> For me, the remaining performance issue is working with big tables and
> I've not found a solution to this.
> --
> : Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-566-gf0198e
> : Latest paper written in org: https://arxiv.org/abs/2106.05096
>
>

[-- Attachment #2: Type: text/html, Size: 2041 bytes --]

Re: Large source block causes org-mode to be unusable

[Eric S Fraga]

On Tuesday, 22 Jun 2021 at 13:20, Léo Ackermann wrote:
> I am new to org-mode but, is there a reason why #+BEGIN_proof #+END_proof
> and other org-latex-special-block are treated as block ?

I am not sure what you are asking.  What specific treatment are you
referring to?

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-567-g22bf80
: Latest paper written in org: https://arxiv.org/abs/2106.05096

Re: Large source block causes org-mode to be unusable

[Léo Ackermann]

[-- Attachment #1: Type: text/plain, Size: 1460 bytes --]

> I am not sure what you are asking.  What specific treatment are you
referring to?

I suggest that special block in LateX export (
https://orgmode.org/manual/Special-blocks-in-LaTeX-export.html) may be
treated differently. Once again, I'm new to org-mode, but here is my
intuition:
- What is happening now. The #+BEGIN_proof / #+END_proof keywords cause my
proof to be seen as a block. Therefore, babel fontification is called to
highlight syntax of my proof within my buffer. Timothy already discussed
here how inefficient the babel fontification is.
- What could happen. Changing (somewhere) the treatment of #BEGIN_proof &
co the following way: Assign them to a new face "latex_export_sugar" and
ignore them while editing. Doing this, the fontification of the org proof
will be made as usual and will be fast enough. While exporting, replace
#+BEGIN_proof by \begin{proof} and so on...

Do you see what I mean ?

Best,

Le mar. 22 juin 2021 à 14:13, Eric S Fraga <e.fraga@ucl.ac.uk> a écrit :

> On Tuesday, 22 Jun 2021 at 13:20, Léo Ackermann wrote:
> > I am new to org-mode but, is there a reason why #+BEGIN_proof #+END_proof
> > and other org-latex-special-block are treated as block ?
>
> I am not sure what you are asking.  What specific treatment are you
> referring to?
>
> --
> : Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-567-g22bf80
> : Latest paper written in org: https://arxiv.org/abs/2106.05096
>

[-- Attachment #2: Type: text/html, Size: 2017 bytes --]

Re: Large source block causes org-mode to be unusable

[Eric S Fraga]

On Tuesday, 22 Jun 2021 at 14:32, Léo Ackermann wrote:
> Do you see what I mean ?

I do but I guess I must have a differently configured system as I do not
see any special fortification due to being in a special block.

However, I see nothing in my configuration that affects special blocks.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-567-g22bf80
: Latest paper written in org: https://arxiv.org/abs/2106.05096

Re: Large source block causes org-mode to be unusable

[Léo Ackermann]

[-- Attachment #1.1: Type: text/plain, Size: 823 bytes --]

Neither do I (concerning the special fontification).
Nevertheless, if you look at the profiler report (1st mail), the
fontification process differ from the usual one (see attached screenshot)
*because* the proof is considered as a block by org-mode.

[image: image.png]

Best,

Le mar. 22 juin 2021 à 15:03, Eric S Fraga <e.fraga@ucl.ac.uk> a écrit :

> On Tuesday, 22 Jun 2021 at 14:32, Léo Ackermann wrote:
> > Do you see what I mean ?
>
> I do but I guess I must have a differently configured system as I do not
> see any special fortification due to being in a special block.
>
> However, I see nothing in my configuration that affects special blocks.
>
> --
> : Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-567-g22bf80
> : Latest paper written in org: https://arxiv.org/abs/2106.05096
>

[-- Attachment #1.2: Type: text/html, Size: 1302 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 91453 bytes --]

Re: Large source block causes org-mode to be unusable

[Maxim Nikulin]

On 22/06/2021 19:32, Léo Ackermann wrote:
> 
> I suggest that special block in LateX export 
> (https://orgmode.org/manual/Special-blocks-in-LaTeX-export.html) may be 
> treated differently.

Generally I agree with you that #+begin_proof should be a thin wrapper 
unlike a source code block. However I am not familiar with 
fontification, so I can not reason if it is feasible. Maybe the code is 
watching if something like #+end_proof appeared, so block finished.

On the other hand my impression was that \begin{proof} environment is 
intended to emphasize a paragraph or two to separate it from regular 
text on the same page. My expectation is that 4-page long proof should 
be formatted as ordinary sections and subsections.

If proof environment is still required, maybe it is possible to 
customize export backend to have proofs as a headings with some property 
in the source document. E.g. beamer backend converts headings to frames. 
I have never tried such tricks, so I am unaware how much work is 
required for such approach.

Re: Large source block causes org-mode to be unusable

[Gennady Uraltsev]

Hello Everyone,

I wanted to chime in to say I encountered a very similar problem. I
author my math papers in LaTeX but I really wanted to use org to have
an interconnected corpus of math notes with definitions, proofs,
ideas, attempted computations,  etc. Then they could be exported to
LaTeX or even better: HTML to make for quite nice interactive notes
(with popups backlinking to definitions of mathematical objects,
etc...) (I was writing the latter in some JS). Unfortunately, HTML
(+mathjax or whatever) is much more flexible than PDFs

>I suggest that special block in LateX export (https://orgmode.org/manual/Special-blocks-in-LaTeX-export.html) may be treated differently. Once again, I'm new to org-mode, but here is my intuition:
>- What is happening now. The #+BEGIN_proof / #+END_proof keywords cause my proof to be seen as a block. Therefore, babel fontification is called to highlight syntax of my proof within my buffer. Timothy already discussed here how inefficient the >babel fontification is.
>- What could happen. Changing (somewhere) the treatment of #BEGIN_proof & co the following way: Assign them to a new face "latex_export_sugar" and ignore them while editing. Doing this, the fontification of the org proof will be made as usual and >will be fast enough. While exporting, replace #+BEGIN_proof by \begin{proof} and so on...

Exactly! Before HTML export becomes relevant, I if I tried to use
#+BEGIN_proof #+END_proof or #+BEGIN_definition #+END_definition
insertion in these environments slowed very quickly down to a crawl.
If I remember correctly the profiler told me that it was due to
fontification.

For me, the slowness in fontification INSIDE actual latex math blocks
(the only place where I need fontification, and I would prefer to use
AucTeX to do that, since it is really good at it) is generally not an
issue since I mostly  have latex previews and use the edit source
command to work on the formulae. It would allow for a greater variety
of workflows if one could specify that some blocks lines should be
ignored for fontification purposes and the contents be treated as
usual org-mode text.

If you think it can be useful I can try to dig up my (abandoned)
attempts and try to provide test cases to improve performance.

> If proof environment is still required, maybe it is possible to
> customize export backend to have proofs as a headings with some property
> in the source document.

Unfortunately this is not really fitting with my workflow (and I think
this is not only my issue). In Org Mode, headings cannot "terminate"
i.e. only a new section can stop a previous one. So, essentially, one
would need to do a very non-elegant trick similar to one used by the
"inline TODOs" i.e. having a very deep level of heading (10?) that
"starts" a proof and then an artificial one that "ends" it.

Also, the point of having "proof" blocks and "example" blocks that
often, when reading math, you do not want to go through the proof on
the first reading.

One issue that many colleagues have agreed when discussing is that it
would be so great to have a "collapsible" environment or block. When
we write notes or explanations of material, often we are tasked with
making a call: do we do computations step by step and risk being
verbose, or do we write only the main steps and leave it to the reader
to fill in the intermediate ones. Both these approaches have
downsides. The former makes seeing the main points of the reasoning
difficult to catch and forces a reader to commit significant effort to
completely read the paper or notes (even if they just want the gist of
it). The other approach may frustrate the reader interested in details
because he/she has to work through the material to fill in all the
gaps, often spending a lot of time on omitted intermediate steps.

 I was actually trying to come up with a natural system through org
and HTML export to avoid this dilemma and have sections that are
hidden by default but can be "expanded" on request.

> On the other hand my impression was that \begin{proof} environment is
> intended to emphasize a paragraph or two to separate it from regular
> text on the same page. My expectation is that 4-page long proof should
> be formatted as ordinary sections and subsections.

Unfortunately this goes against the standard of mathematical writing
(that I at least am familiar with).

Proofs are often long and technical and can span multiple pages. If
there are self-contained results in the proofs you separate them out
in statements (lemmata) with their own (shorter) proofs.
Text outside proofs is usually explanations and "glue" that explains
the process, the idea, but does so on a more "informal" level.

Having "proof" environments is also very useful as a method of having
"namespaces", often you need to introduce auxiliary objects,
expressions, operations, that are relevant only for the proof of the
result but that should not be "polluting" the full paper or writeup.
For example, if you introduce something inside a proof and denote it
with a symbol (e.g. $\phi$) then as soon as you hit the QED symbol at
the end of the proof, that symbol is freed and can be used in other
proofs/constructions. It is very bad form, for example, to reference
objects inside proofs of theorems outside the proofs themselves. If
something is an object of general interest then it should be "defined"
(included in a definition) on its own.


I apologize for going on a tangent.

Thanks again to everyone for the great package that is ORG, that I am
progressively trying to incorporate in more and more of my day-to-day
work and life.


Best!

Gena

--
Gennady Uraltsev
<gennady.uraltsev@gmail.com>
(https://guraltsev.gitlab.io)

Re: Large source block causes org-mode to be unusable

[Eric S Fraga]

On Wednesday, 23 Jun 2021 at 15:42, Gennady Uraltsev wrote:
> In Org Mode, headings cannot "terminate" i.e. only a new section can
> stop a previous one. 

True.

What I do for cases where I want "separation" for visibility etc. is use
headings at the same level but tell the exporter to ignore the actual
headings when exporting.  So I frequently have something like this:

#+begin_src org
  ,* Section heading
  ,** introduction to section                       :ignoreheading:
  ,** statement of problem                          :ignoreheading:
  ,** proof                                         :ignoreheading:
     ,#+latex: \begin{proof}
     proof goes here
     ,#+latex: \end{proof}
  ,** continuation of section                       :ignoreheading:
  ,* Next section
#+end_src

also showing how you can introduce a proof environment without using
special blocks and hence maybe help with speed.  (noting that you could
do the same for HTML.)

I have a filter for export that strips headings that have the
:ignoreheading: tag.  I believe there's something similar built-in now?

This approach makes for easier navigation in the document (for me) and
helps keep that bird's eye view of what I am writing.
-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-571-gc591be
: Latest paper written in org: https://arxiv.org/abs/2106.05096

Re: Large source block causes org-mode to be unusable

[Léo Ackermann]

[-- Attachment #1: Type: text/plain, Size: 2039 bytes --]

Hi,

As I found no convenient solution, I will try to write a small module that
adress this problem. It will be a good occasion to discover Elisp :). It is
thought as an alternative to org-special-latex-block: efficient
fontification using default face, folding environment to hide proofs,
pretty latex export...
@EricSFrada, would you mind sharing your code for your proof sections ?
If you've got ideas for this module, or/and want to participate to its
development, please let me know :)

Best,


Le jeu. 24 juin 2021 à 09:59, Eric S Fraga <e.fraga@ucl.ac.uk> a écrit :

> On Wednesday, 23 Jun 2021 at 15:42, Gennady Uraltsev wrote:
> > In Org Mode, headings cannot "terminate" i.e. only a new section can
> > stop a previous one.
>
> True.
>
> What I do for cases where I want "separation" for visibility etc. is use
> headings at the same level but tell the exporter to ignore the actual
> headings when exporting.  So I frequently have something like this:
>
> #+begin_src org
>   ,* Section heading
>   ,** introduction to section                       :ignoreheading:
>   ,** statement of problem                          :ignoreheading:
>   ,** proof                                         :ignoreheading:
>      ,#+latex: \begin{proof}
>      proof goes here
>      ,#+latex: \end{proof}
>   ,** continuation of section                       :ignoreheading:
>   ,* Next section
> #+end_src
>
> also showing how you can introduce a proof environment without using
> special blocks and hence maybe help with speed.  (noting that you could
> do the same for HTML.)
>
> I have a filter for export that strips headings that have the
> :ignoreheading: tag.  I believe there's something similar built-in now?
>
> This approach makes for easier navigation in the document (for me) and
> helps keep that bird's eye view of what I am writing.
> --
> : Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-571-gc591be
> : Latest paper written in org: https://arxiv.org/abs/2106.05096
>
>

[-- Attachment #2: Type: text/html, Size: 2680 bytes --]

Re: Large source block causes org-mode to be unusable

[Sébastien Miquel]

[-- Attachment #1: Type: text/plain, Size: 493 bytes --]

Hi,

Léo Ackermann writes:
> @EricSFrada, would you mind sharing your code for your proof sections ?
This functionality is now built-in: headings with an `ignore' tag do not get
exported (their contents do). For very large proof, this seems like the 
right
thing to do.

In small to moderate sized blocks, the delay can still be noticeable and 
ought
to be fixed. Attached is a patch that seems to resolve this issue. I haven't
noticed any drawbacks so far.

Regards,

-- 
Sébastien Miquel


[-- Attachment #2: 0001-WIP-do-not-refontify-special-blocks.patch --]
[-- Type: text/x-patch, Size: 6303 bytes --]

From d843bdc5887a6e50a57e349128ebbe032086dc17 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?S=C3=A9bastien=20Miquel?= <sebastien.miquel@posteo.eu>
Date: Sun, 27 Jun 2021 16:24:22 +0200
Subject: [PATCH] WIP : do not refontify special blocks

---
 lisp/org.el | 99 ++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 64 insertions(+), 35 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index 1bd9e02eb..9fd3f8514 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -5265,22 +5265,32 @@ by a #."
       (org-fontify-meta-lines-and-blocks-1 limit)
     (error (message "Org mode fontification error in %S at %d"
 		    (current-buffer)
-		    (line-number-at-pos)))))
+		    (line-number-at-pos))))
+  nil)
 
 (defun org-fontify-meta-lines-and-blocks-1 (limit)
   "Fontify #+ lines and blocks."
-  (let ((case-fold-search t))
-    (when (re-search-forward
-	   (rx bol (group (zero-or-more (any " \t")) "#"
-			  (group (group (or (seq "+" (one-or-more (any "a-zA-Z")) (optional ":"))
-					    (any " \t")
-					    eol))
-				 (optional (group "_" (group (one-or-more (any "a-zA-Z"))))))
-			  (zero-or-more (any " \t"))
-			  (group (group (zero-or-more (not (any " \t\n"))))
-				 (zero-or-more (any " \t"))
-				 (group (zero-or-more any)))))
-	   limit t)
+  (let* ((case-fold-search t)
+         (fl-beg (point))
+         (fl-end limit)
+         (fbeg (when (and (> fl-beg (point-min))
+                          (get-text-property (1- fl-beg) 'font-lock-multiline-block))
+                 (or (previous-single-property-change
+                      fl-beg 'font-lock-multiline-block)
+                     (point-min)))))
+    (when fbeg (goto-char fbeg))
+    (while (and (< (point) limit)
+                (re-search-forward
+	         (rx bol (group (zero-or-more (any " \t")) "#"
+			        (group (group (or (seq "+" (one-or-more (any "a-zA-Z")) (optional ":"))
+					          (any " \t")
+					          eol))
+				       (optional (group "_" (group (one-or-more (any "a-zA-Z"))))))
+			        (zero-or-more (any " \t"))
+			        (group (group (zero-or-more (not (any " \t\n"))))
+				       (zero-or-more (any " \t"))
+				       (group (zero-or-more any)))))
+	         limit t))
       (let ((beg (match-beginning 0))
 	    (end-of-beginline (match-end 0))
 	    ;; Including \n at end of #+begin line will include \n
@@ -5318,7 +5328,7 @@ by a #."
 	      (remove-text-properties beg end-of-endline
 				      '(display t invisible t intangible t)))
 	    (add-text-properties
-	     beg end-of-endline '(font-lock-fontified t font-lock-multiline t))
+	     beg end-of-endline '(font-lock-fontified t font-lock-multiline-block t))
 	    (org-remove-flyspell-overlays-in beg bol-after-beginline)
 	    (org-remove-flyspell-overlays-in nl-before-endline end-of-endline)
 	    (cond
@@ -5327,7 +5337,8 @@ by a #."
 	      (add-text-properties bol-after-beginline block-end '(src-block t)))
 	     (quoting
 	      (add-text-properties
-	       bol-after-beginline beg-of-endline
+	       (max bol-after-beginline (or fl-beg bol-after-beginline))
+               (min beg-of-endline (or fl-end beg-of-endline))
 	       (list 'face
 		     (list :inherit
 			   (let ((face-name
@@ -5426,26 +5437,41 @@ by a #."
 	    (add-text-properties closing-start end '(invisible t)))
 	  t)))))
 
-(defun org-fontify-extend-region (beg end _old-len)
-  (let ((begin-re "\\(\\\\\\[\\|\\(#\\+begin_\\|\\\\begin{\\)\\S-+\\)")
-	(end-re "\\(\\\\\\]\\|\\(#\\+end_\\|\\\\end{\\)\\S-+\\)")
-	(extend
-         (lambda (r1 r2 dir)
-	   (let ((re (replace-regexp-in-string
-                      "\\(begin\\|end\\)" r1
-		      (replace-regexp-in-string
-                       "[][]" r2
-		       (match-string-no-properties 0)))))
-	     (re-search-forward (regexp-quote re) nil t dir)))))
-    (save-match-data
-      (save-excursion
-	(goto-char beg)
-	(back-to-indentation)
-	(cond ((looking-at end-re)
-	       (cons (or (funcall extend "begin" "[" -1) beg) end))
-	      ((looking-at begin-re)
-	       (cons beg (or (funcall extend "end" "]" 1) end)))
-	      (t (cons beg end)))))))
+(defun org-fontify-extend-region (bego endo _old-len)
+  (let* ((beg bego) (end endo)
+        (bol (save-excursion (goto-char beg) (point-at-bol)))
+        (eol (save-excursion (goto-char end) (point-at-eol))))
+    (let ((before-multi (and (> bol (point-min))
+                             (get-text-property (1- bol) 'font-lock-multiline-block)))
+          (after-multi (get-text-property eol 'font-lock-multiline-block)))
+      (if before-multi
+          (unless after-multi
+            (setq beg (or (previous-single-property-change
+                           bol 'font-lock-multiline-block)
+                          (point-min))))
+        (when after-multi
+          (setq end (or (text-property-any eol (point-max)
+                                           'font-lock-multiline-block nil)
+                        (point-max))))))
+    (let ((begin-re "\\(\\\\\\[\\|\\(#\\+begin_\\|\\\\begin{\\)\\S-+\\)")
+	  (end-re "\\(\\\\\\]\\|\\(#\\+end_\\|\\\\end{\\)\\S-+\\)")
+	  (extend
+           (lambda (r1 r2 dir)
+	     (let ((re (replace-regexp-in-string
+                        "\\(begin\\|end\\)" r1
+		        (replace-regexp-in-string
+                         "[][]" r2
+		         (match-string-no-properties 0)))))
+	       (re-search-forward (regexp-quote re) nil t dir)))))
+      (save-match-data
+        (save-excursion
+	  (goto-char beg)
+	  (back-to-indentation)
+	  (cond ((looking-at end-re)
+	         (cons (or (funcall extend "begin" "[" -1) beg) end))
+	        ((looking-at begin-re)
+	         (cons beg (max end (or (funcall extend "end" "]" 1) end))))
+	        (t (cons beg end))))))))
 
 (defun org-activate-footnote-links (limit)
   "Add text properties for footnotes."
@@ -5772,6 +5798,9 @@ needs to be inserted at a specific position in the font-lock sequence.")
 		'(org-font-lock-keywords t nil nil backward-paragraph))
     (setq-local font-lock-extend-after-change-region-function
 		#'org-fontify-extend-region)
+    (setq-local font-lock-extra-managed-props
+                (append font-lock-extra-managed-props
+                        '(font-lock-multiline-block)))
     (kill-local-variable 'font-lock-keywords)
     nil))
 
-- 
2.32.0

Re: Large source block causes org-mode to be unusable

[Eric S Fraga]

On Saturday, 26 Jun 2021 at 16:10, Léo Ackermann wrote:
> @EricSFrada, would you mind sharing your code for your proof sections ?
> If you've got ideas for this module, or/and want to participate to its
> development, please let me know :)

I have no code for proofs.  Sorry!  I was simply illustrating that you
can put in LaTeX code directly should special blocks be too large.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-571-gc591be
: Latest paper written in org: https://arxiv.org/abs/2106.05096

Re: Large source block causes org-mode to be unusable

[Eric S Fraga]

On Monday, 28 Jun 2021 at 08:28, Sébastien Miquel wrote:
> Léo Ackermann writes:
>> @EricSFrada, would you mind sharing your code for your proof sections ?
> This functionality is now built-in: headings with an `ignore' tag do
> not get exported (their contents do). For very large proof, this seems
> like the right thing to do.

Ah, maybe this is what Leo was referring to.  I may have misunderstood.

Yes, it seems that org now does process :ignore: tags, I guess.  I've
been using my own code to ignore headings for so long that I did not
realise this!

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-571-gc591be
: Latest paper written in org: https://arxiv.org/abs/2106.05096