Bug: no math-mode detection for align-environment [9.3.7 (9.3.7-13-ge62ca4-elpaplus @ /home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)]

Bug: no math-mode detection for align-environment [9.3.7 (9.3.7-13-ge62ca4-elpaplus @ /home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)]

[Stefi]

Hi there,

I recently updated my emacs packages and noticed that \begin{align*}
does not trigger math-mode detection any more, so org-cdlatex does not
work. However, begin{equation} still does trigger the math-mode
detection. I have read the documentation of
"org-inside-LaTeX-fragment-p" and found something about matchers, but
nothing about equation or align environment and why it might be treated
differently for math-mode detection.

My test-file.org:
----8<---------8<---------8<---------8<-----

* blabla
 \begin{equation}
\label{eq:1} \frac{}{}
test_{cds} \frac{}{}
\end{equation}

\begin{align}
\label{eq:2}
test$_d$ fr
\end{align}

$\frac{}{} fds_{fdsf} \frac{}{}$

\begin{align*}
fr
\end{align*}


----8<---------8<---------8<---------8<-----

- detection works for: \begin{equation}, $$, \[\] (not in testfile)
- detection does not work for: \begin{align}, begin{align*}; subscript _
  does insert extra $_{}$, which it should not in math mode, and fr does
  not get expanded to \frac{}{}

Emacs  : GNU Emacs 26.3 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.20)
 of 2020-05-19
Package: Org mode version 9.3.7 (9.3.7-13-ge62ca4-elpaplus @
/home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)


Kind regards,
stefi

Re: Bug: no math-mode detection for align-environment [9.3.7 (9.3.7-13-ge62ca4-elpaplus @ /home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)]

[Kyle Meyer]

Stefi writes:

> I recently updated my emacs packages and noticed that \begin{align*}
> does not trigger math-mode detection any more, so org-cdlatex does not
> work. However, begin{equation} still does trigger the math-mode
> detection. I have read the documentation of
> "org-inside-LaTeX-fragment-p" and found something about matchers, but
> nothing about equation or align environment and why it might be treated
> differently for math-mode detection.

The default value for :matches is ("begin" "$1" "$" "$$" "\\(" "\\[").
It looks like the value hasn't been touched in a long time, and that
AFAICT the "begin" should cover "align" just as it covers "equation".
And I'm not spotting any recent changes on Org's end that should affect
this.

> My test-file.org:
> ----8<---------8<---------8<---------8<-----
>
> * blabla
>  \begin{equation}
> \label{eq:1} \frac{}{}
> test_{cds} \frac{}{}
> \end{equation}
>
> \begin{align}
> \label{eq:2}
> test$_d$ fr
> \end{align}
>
> $\frac{}{} fds_{fdsf} \frac{}{}$
>
> \begin{align*}
> fr
> \end{align*}
>
>
> ----8<---------8<---------8<---------8<-----
>
> - detection works for: \begin{equation}, $$, \[\] (not in testfile)
> - detection does not work for: \begin{align}, begin{align*}; subscript _
>   does insert extra $_{}$, which it should not in math mode, and fr does
>   not get expanded to \frac{}{}
>
> Emacs  : GNU Emacs 26.3 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.20)
>  of 2020-05-19
> Package: Org mode version 9.3.7 (9.3.7-13-ge62ca4-elpaplus @
> /home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)

On master, maint, and the commit matching your reported version
(e62ca4), I tried org-inside-LaTeX-fragment-p within each block of your
test file, and it returns non-nil, including in the align and align*
blocks.  So I dunno...

Could some org-cdlatex users out there give the test file a try?

Re: Bug: no math-mode detection for align-environment [9.3.7 (9.3.7-13-ge62ca4-elpaplus @ /home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)]

[Rafael]

Kyle Meyer <kyle@kyleam.com> writes:

> Stefi writes:
>
>> I recently updated my emacs packages and noticed that \begin{align*}
>> does not trigger math-mode detection any more, so org-cdlatex does not
>> work. However, begin{equation} still does trigger the math-mode
>> detection. I have read the documentation of
>> "org-inside-LaTeX-fragment-p" and found something about matchers, but
>> nothing about equation or align environment and why it might be treated
>> differently for math-mode detection.
>
> The default value for :matches is ("begin" "$1" "$" "$$" "\\(" "\\[").
> It looks like the value hasn't been touched in a long time, and that
> AFAICT the "begin" should cover "align" just as it covers "equation".
> And I'm not spotting any recent changes on Org's end that should affect
> this.
>
>> My test-file.org:
>> ----8<---------8<---------8<---------8<-----
>>
>> * blabla
>>  \begin{equation}
>> \label{eq:1} \frac{}{}
>> test_{cds} \frac{}{}
>> \end{equation}
>>
>> \begin{align}
>> \label{eq:2}
>> test$_d$ fr
>> \end{align}
>>
>> $\frac{}{} fds_{fdsf} \frac{}{}$
>>
>> \begin{align*}
>> fr
>> \end{align*}
>>
>>
>> ----8<---------8<---------8<---------8<-----
>>
>> - detection works for: \begin{equation}, $$, \[\] (not in testfile)
>> - detection does not work for: \begin{align}, begin{align*}; subscript _
>>   does insert extra $_{}$, which it should not in math mode, and fr does
>>   not get expanded to \frac{}{}
>>
>> Emacs  : GNU Emacs 26.3 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.20)
>>  of 2020-05-19
>> Package: Org mode version 9.3.7 (9.3.7-13-ge62ca4-elpaplus @
>> /home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)
>
> On master, maint, and the commit matching your reported version
> (e62ca4), I tried org-inside-LaTeX-fragment-p within each block of your
> test file, and it returns non-nil, including in the align and align*
> blocks.  So I dunno...
>
> Could some org-cdlatex users out there give the test file a try?

I can confirm that with
Org mode version 9.3.7 (9.3.7-16-g521d7f-elpaplus @
/home/rafael/Documents/new-emacs.d/elpa/org-plus-contrib-20200803/)
and
GNU Emacs 26.3 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.18.9) of
2019-12-23
and using org-cdlatex, fr does not expand inside align but it does
inside equation, just as Stefi says.

Re: Bug: no math-mode detection for align-environment [9.3.7 (9.3.7-13-ge62ca4-elpaplus @ /home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)]

[Stefi]

Hi Kyle, hi mailing list,

I also evaluated (org-inside-LaTeX-fragment-p) and got non-nil values
for all blocks, however, inside the align* environment, the "fr" does
not get expanded on TAB whereas it does in equation environment. So
either the matchers do not include align/align*, or there is something
different now with cdlatex.

Btw, this is what I got from evaluation:

(#("\\begin{align*} 0 14 (fontified t line-prefix #("    " 0 4 ...)
wrap-prefix # ("    " 0 4 ...))) . 15791)

and

(#("\\begin{equation} 0 16 (fontified t line-prefix #("    " 0 4 ...)
wrap-prefix # ("    " 0 4 ...))) . 15824)

Does someone know how to find and configure the matchers?

I am pretty sure it worked before with align* environment, as I am using
it all the time :)

regards

On 7/29/20 5:12 AM, Kyle Meyer wrote:
> Stefi writes:
> 
>> I recently updated my emacs packages and noticed that \begin{align*}
>> does not trigger math-mode detection any more, so org-cdlatex does not
>> work. However, begin{equation} still does trigger the math-mode
>> detection. I have read the documentation of
>> "org-inside-LaTeX-fragment-p" and found something about matchers, but
>> nothing about equation or align environment and why it might be treated
>> differently for math-mode detection.
> 
> The default value for :matches is ("begin" "$1" "$" "$$" "\\(" "\\[").
> It looks like the value hasn't been touched in a long time, and that
> AFAICT the "begin" should cover "align" just as it covers "equation".
> And I'm not spotting any recent changes on Org's end that should affect
> this.
> 
>> My test-file.org:
>> ----8<---------8<---------8<---------8<-----
>>
>> * blabla
>>  \begin{equation}
>> \label{eq:1} \frac{}{}
>> test_{cds} \frac{}{}
>> \end{equation}
>>
>> \begin{align}
>> \label{eq:2}
>> test$_d$ fr
>> \end{align}
>>
>> $\frac{}{} fds_{fdsf} \frac{}{}$
>>
>> \begin{align*}
>> fr
>> \end{align*}
>>
>>
>> ----8<---------8<---------8<---------8<-----
>>
>> - detection works for: \begin{equation}, $$, \[\] (not in testfile)
>> - detection does not work for: \begin{align}, begin{align*}; subscript _
>>   does insert extra $_{}$, which it should not in math mode, and fr does
>>   not get expanded to \frac{}{}
>>
>> Emacs  : GNU Emacs 26.3 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.20)
>>  of 2020-05-19
>> Package: Org mode version 9.3.7 (9.3.7-13-ge62ca4-elpaplus @
>> /home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)
> 
> On master, maint, and the commit matching your reported version
> (e62ca4), I tried org-inside-LaTeX-fragment-p within each block of your
> test file, and it returns non-nil, including in the align and align*
> blocks.  So I dunno...
> 
> Could some org-cdlatex users out there give the test file a try?
> 

Re: Bug: no math-mode detection for align-environment [9.3.7 (9.3.7-13-ge62ca4-elpaplus @ /home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)]

[Kyle Meyer]

Stefi writes:

> Hi Kyle, hi mailing list,
>
> I also evaluated (org-inside-LaTeX-fragment-p) and got non-nil values
> for all blocks, however, inside the align* environment, the "fr" does
> not get expanded on TAB whereas it does in equation environment. So
> either the matchers do not include align/align*, or there is something
> different now with cdlatex.

I'm leaning towards there being something different with cdlatex.

I've now installed cdlatex from
<https://staff.fnwi.uva.nl/c.dominik/Tools/cdlatex/cdlatex.el>.  The
file header says the version is 4.8, and the file's md5sum is
9246a4e9f30c9589406783ffd5270641.

I tried your example file with this minimal configuration:

    (require 'org)
    (add-hook 'org-mode-hook (lambda () (org-cdlatex-mode 1)))
    (require 'edebug)
    (edebug-instrument-function 'org-try-cdlatex-tab)

As you and Rafael report, "fr => \frac{}{}" is not expanded in the align
or align* environments, while it is expanded in the equation
environment.  However, stepping through the org-try-cdlatex-tab call,
cdlatex-tab is called in all cases.  So it would seem that something has
changed downstream.

In line with this idea, I observed the same behavior with both the
latest commit of the master branch (cdfc404be) and v9.2 (released in Dec
2018).  Also, as I mentioned in earlier in the thread, I don't see any
recent changes on Org's side related to the cdlatex functionality.

Re: Bug: no math-mode detection for align-environment [9.3.7 (9.3.7-13-ge62ca4-elpaplus @ /home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)]

[Stefi]

I can also confirm that org-try-cdlatex-tab outputs "t" in all cases but
does not expand in align environment.

I noticed that cdlatex-env-alist-default does not include the align
environment. I tried to add it to "Cdlatex Env Alist" in the customize
browser but it did not change the behavior to what I expect.

I skimmed through cdlatex.el but did not find anything suspicious. I am
not a hardcore lisp coder though.

On 8/26/20 5:44 AM, Kyle Meyer wrote:
> Stefi writes:
> 
>> Hi Kyle, hi mailing list,
>>
>> I also evaluated (org-inside-LaTeX-fragment-p) and got non-nil values
>> for all blocks, however, inside the align* environment, the "fr" does
>> not get expanded on TAB whereas it does in equation environment. So
>> either the matchers do not include align/align*, or there is something
>> different now with cdlatex.
> 
> I'm leaning towards there being something different with cdlatex.
> 
> I've now installed cdlatex from
> <https://staff.fnwi.uva.nl/c.dominik/Tools/cdlatex/cdlatex.el>.  The
> file header says the version is 4.8, and the file's md5sum is
> 9246a4e9f30c9589406783ffd5270641.
> 
> I tried your example file with this minimal configuration:
> 
>     (require 'org)
>     (add-hook 'org-mode-hook (lambda () (org-cdlatex-mode 1)))
>     (require 'edebug)
>     (edebug-instrument-function 'org-try-cdlatex-tab)
> 
> As you and Rafael report, "fr => \frac{}{}" is not expanded in the align
> or align* environments, while it is expanded in the equation
> environment.  However, stepping through the org-try-cdlatex-tab call,
> cdlatex-tab is called in all cases.  So it would seem that something has
> changed downstream.
> 
> In line with this idea, I observed the same behavior with both the
> latest commit of the master branch (cdfc404be) and v9.2 (released in Dec
> 2018).  Also, as I mentioned in earlier in the thread, I don't see any
> recent changes on Org's side related to the cdlatex functionality.
> 

Re: Bug: no math-mode detection for align-environment [9.3.7 (9.3.7-13-ge62ca4-elpaplus @ /home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)]

[Stefi]

It might be a change to texmathp.el. It is part of Auctex and checks if
math mode is on or off. I could not find align environment in the list
of default environments. Maybe that has changed.

However, I added align and align* to "Texmathp Tex Commands" from the
customize browser (open a .tex file to customize auctex). Tab-expansion
is working now. Can anyone confirm?

In my .emacs, it added to custom-set-variables:
'(texmathp-tex-commands (quote (("align*" env-on) ("align" env-on))))


On 8/26/20 5:44 AM, Kyle Meyer wrote:
> Stefi writes:
> 
>> Hi Kyle, hi mailing list,
>>
>> I also evaluated (org-inside-LaTeX-fragment-p) and got non-nil values
>> for all blocks, however, inside the align* environment, the "fr" does
>> not get expanded on TAB whereas it does in equation environment. So
>> either the matchers do not include align/align*, or there is something
>> different now with cdlatex.
> 
> I'm leaning towards there being something different with cdlatex.
> 
> I've now installed cdlatex from
> <https://staff.fnwi.uva.nl/c.dominik/Tools/cdlatex/cdlatex.el>.  The
> file header says the version is 4.8, and the file's md5sum is
> 9246a4e9f30c9589406783ffd5270641.
> 
> I tried your example file with this minimal configuration:
> 
>     (require 'org)
>     (add-hook 'org-mode-hook (lambda () (org-cdlatex-mode 1)))
>     (require 'edebug)
>     (edebug-instrument-function 'org-try-cdlatex-tab)
> 
> As you and Rafael report, "fr => \frac{}{}" is not expanded in the align
> or align* environments, while it is expanded in the equation
> environment.  However, stepping through the org-try-cdlatex-tab call,
> cdlatex-tab is called in all cases.  So it would seem that something has
> changed downstream.
> 
> In line with this idea, I observed the same behavior with both the
> latest commit of the master branch (cdfc404be) and v9.2 (released in Dec
> 2018).  Also, as I mentioned in earlier in the thread, I don't see any
> recent changes on Org's side related to the cdlatex functionality.
> 

Re: Bug: no math-mode detection for align-environment [9.3.7 (9.3.7-13-ge62ca4-elpaplus @ /home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)]

[Kyle Meyer]

Stefi writes:

> It might be a change to texmathp.el. It is part of Auctex and checks if
> math mode is on or off. I could not find align environment in the list
> of default environments. Maybe that has changed.
>
> However, I added align and align* to "Texmathp Tex Commands" from the
> customize browser (open a .tex file to customize auctex). Tab-expansion
> is working now. Can anyone confirm?
>
> In my .emacs, it added to custom-set-variables:
> '(texmathp-tex-commands (quote (("align*" env-on) ("align" env-on))))

Yes, that results in expansion on my end as well.

It looks like this issue was introduced with AUCTeX's 91701704 (Delete
overhead in extending font lock range of math expression, 2020-06-11).
If you're getting AUCTeX from ELPA, the regression is included the
current version there (12.2.4, 2020-06-29).

At the start of this month, it was fixed in f04a508f (Restore all math
environments in texmathp.el, 2020-08-01), so the issue should go away
with the next update that lands on ELPA.

Re: Bug: no math-mode detection for align-environment [9.3.7 (9.3.7-13-ge62ca4-elpaplus @ /home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)]

[Carsten Dominik]

[-- Attachment #1: Type: text/plain, Size: 1429 bytes --]

Hi,

AUCTeX has taken back this change, realizing that texmathp.el is also used
in contexts where the mechanisms of AUCTeX to find out about LaTeX packages
being used does not work.  I do think this is already fixed in the latest
AUCTeX version.  At least it is fixed in their git.

- Carsten

Carsten

On Fri, Aug 28, 2020 at 7:09 AM Kyle Meyer <kyle@kyleam.com> wrote:

> Stefi writes:
>
> > It might be a change to texmathp.el. It is part of Auctex and checks if
> > math mode is on or off. I could not find align environment in the list
> > of default environments. Maybe that has changed.
> >
> > However, I added align and align* to "Texmathp Tex Commands" from the
> > customize browser (open a .tex file to customize auctex). Tab-expansion
> > is working now. Can anyone confirm?
> >
> > In my .emacs, it added to custom-set-variables:
> > '(texmathp-tex-commands (quote (("align*" env-on) ("align" env-on))))
>
> Yes, that results in expansion on my end as well.
>
> It looks like this issue was introduced with AUCTeX's 91701704 (Delete
> overhead in extending font lock range of math expression, 2020-06-11).
> If you're getting AUCTeX from ELPA, the regression is included the
> current version there (12.2.4, 2020-06-29).
>
> At the start of this month, it was fixed in f04a508f (Restore all math
> environments in texmathp.el, 2020-08-01), so the issue should go away
> with the next update that lands on ELPA.
>
>

[-- Attachment #2: Type: text/html, Size: 1944 bytes --]

Re: Bug: no math-mode detection for align-environment [9.3.7 (9.3.7-13-ge62ca4-elpaplus @ /home/stefi/.emacs.d/elpa/org-plus-contrib-20200713/)]

[Stefi]

Great news! So this problem will just go away by itself then. Until then
I will use the texmathp-tex-commands solution as a workaround.

Thank you everyone for sharing your thoughts!

On 8/28/20 8:55 AM, Carsten Dominik wrote:
> Hi,
> 
> AUCTeX has taken back this change, realizing that texmathp.el is also
> used in contexts where the mechanisms of AUCTeX to find out about LaTeX
> packages being used does not work.  I do think this is already fixed in
> the latest AUCTeX version.  At least it is fixed in their git.
> 
> - Carsten
> 
> Carsten
> 
> On Fri, Aug 28, 2020 at 7:09 AM Kyle Meyer <kyle@kyleam.com
> <mailto:kyle@kyleam.com>> wrote:
> 
>     Stefi writes:
> 
>     > It might be a change to texmathp.el. It is part of Auctex and
>     checks if
>     > math mode is on or off. I could not find align environment in the list
>     > of default environments. Maybe that has changed.
>     >
>     > However, I added align and align* to "Texmathp Tex Commands" from the
>     > customize browser (open a .tex file to customize auctex).
>     Tab-expansion
>     > is working now. Can anyone confirm?
>     >
>     > In my .emacs, it added to custom-set-variables:
>     > '(texmathp-tex-commands (quote (("align*" env-on) ("align" env-on))))
> 
>     Yes, that results in expansion on my end as well.
> 
>     It looks like this issue was introduced with AUCTeX's 91701704 (Delete
>     overhead in extending font lock range of math expression, 2020-06-11).
>     If you're getting AUCTeX from ELPA, the regression is included the
>     current version there (12.2.4, 2020-06-29).
> 
>     At the start of this month, it was fixed in f04a508f (Restore all math
>     environments in texmathp.el, 2020-08-01), so the issue should go away
>     with the next update that lands on ELPA.
>