I accidentally did "reply" instead of "reply all". Re-adding the e-mail between Ihor and I.

Regarding the suggestion to filter line-by-line. I think I understand that, but it will require that we match input lines to echo'd input. However, it's possible that output lines match input and are not from echo'd input. In addition, there's other stripping (filtering) that is needed. For example, removal of the error indicators (<ERRORTXT>). I like the idea of enhancing org-babel-comint-with-output to strip content based on regular expressions and have implemented that.

See attached org-matlab.patch which addresses all feedback. Here's the commit info.

ob-matlab.el: improve MATLAB support

* lisp/ob-matlab.el (header): Update URL for MATLAB

* lisp/ob-octave.el (org-babel-octave-evaluate): Fixed MATLAB support
  - Deprecate variables related to MATLAB Emacs Link and removed the code.
    Emacs Link capability was removed from MATLAB release R2009a, 15 years ago.
  - Fixed the following type of org block evaluation:
    1) #+begin_src matlab :results verbatim
    2) #+begin_src matlab :results output
    3) #+begin_src matlab :results output latex
    4) #+begin_src matlab :results file graphics
    which aid in writing scientific papers.
  - Minor point, the correct spelling of MATLAB when referencing the product is
    all upper case.

* ob-comint.el: enhanced org-babel-comint-with-output
  - The META argument of org-babel-comint-with-output now supports an optional
    STRIP-REGEXPS which can be used to remove content from the returned output.

* etc/ORG-NEWS (New functions and changes in function arguments):
  Added entry "ob-octave: improved MATLAB support"


Thanks
John

-----
Ihor Radchenko <yantar92@posteo.net>Sun, Dec 22, 2024 at 11:24 PM
Hi John,

It has been a month since the last message in this thread.
Do you need any help working on the patch?

--
Ihor Radchenko // yantar92,
Org mode maintainer,
[Quoted text hidden]

John C <john.ciolfi.32@gmail.com>Wed, Dec 25, 2024 at 8:26 AM
To: Ihor Radchenko <yantar92@posteo.net>
My apologies - I’ve been too busy. I'll update the NEWS entry per your suggestion once we resolve the "%-<org-eval>" comment issue you flagged.

Consider this MATLAB code block and expected results.

    #+begin_src matlab :exports both :results output latex
      m = [4*pi, 3*pi; 2*pi, pi];
      result = latex(sym(m));
      disp(result)
    #+end_src
   
    #+RESULTS:
    #+begin_export latex
    \left(\begin{array}{cc} 4\,\pi  & 3\,\pi \\ 2\,\pi  & \pi  \end{array}\right)
    #+end_export

If we remove the proposed "%-<org-eval>" handling from ob-octave.el, we get the incorrect result:

    #+RESULTS:
    #+begin_export latex
    m = [4*pi, 3*pi; 2*pi, pi];
    result = latex(sym(m));
    disp(result)
    \left(\begin{array}{cc} 4\,\pi  & 3\,\pi \\ 2\,\pi  & \pi  \end{array}\right)
    #+end_export

In this case, the *MATLAB* buffer contains:

    >> m = [4*pi, 3*pi; 2*pi, pi];
    result = latex(sym(m));
    disp(result)
    'org_babel_eoe'
    m = [4*pi, 3*pi; 2*pi, pi];
    >> result = latex(sym(m));
    >> disp(result)
    \left(\begin{array}{cc} 4\,\pi  & 3\,\pi \\ 2\,\pi  & \pi  \end{array}\right)
    >> 'org_babel_eoe'
   
    ans =
   
        'org_babel_eoe'
   
    >>


I suspect that the issue is that the MATLAB echo's all commands and that there's noting wrong with comint. For example, matlab-shell buffer, *MATLAB*, the current buffer and at the MATLAB promot, ">>",

    M-: RET (comint-send-string (get-buffer-process (current-buffer)) "pwd\n") RET

Results in:

    #+begin_example
    >> pwd
   
    ans =
   
        '/home/ciolfi/tmp'
    #+end_example

Therefore, the adding "%-<org-eval>" MATLAB comments to the code being evaluated, then stripping the echo'd lines containing the "%-<org-eval>" is probably the best option.

Thanks
John
[Quoted text hidden]

Ihor Radchenko <yantar92@posteo.net>Thu, Dec 26, 2024 at 1:02 AM
John C <john.ciolfi.32@gmail.com> writes:

> My apologies - I’ve been too busy. I'll update the NEWS entry per your
> suggestion once we resolve the "%-<org-eval>" comment issue you flagged.
>
> Consider this MATLAB code block and expected results.
>
>     #+begin_src matlab :exports both :results output latex
>       m = [4*pi, 3*pi; 2*pi, pi];
>       result = latex(sym(m));
>       disp(result)
>     #+end_src
> ...
> In this case, the *MATLAB* buffer contains:
>
>     >> m = [4*pi, 3*pi; 2*pi, pi];
>     result = latex(sym(m));
>     disp(result)
>     'org_babel_eoe'
>     m = [4*pi, 3*pi; 2*pi, pi];
>     >> result = latex(sym(m));
>     >> disp(result)
>     \left(\begin{array}{cc} 4\,\pi  & 3\,\pi \\ 2\,\pi  & \pi
>  \end{array}\right)
>     >> 'org_babel_eoe'
>
>     ans =
>
>         'org_babel_eoe'
>
>     >>

So, MATLAB does not echo the full code, but rather does it line-by-line.
This is not what `org-babel-comint--echo-filter' and
`org-babel-comint-with-output' expect - the usual behavior with REPLs
echoing the input is displaying the whole (multiline) input together.

> ...
> Therefore, the adding "%-<org-eval>" MATLAB comments to the code being
> evaluated, then stripping the echo'd lines containing the "%-<org-eval>" is
> probably the best option.

What I am thinking as an alternative is modifying
`org-babel-comint-with-output' to filter FULL-BODY line by line.
For example, when REMOVE-ECHO is set to 'line.

`org-babel-comint--echo-filter' can then be passed an extra argument
that will make it filter body lines rather than the whole body only.

WDYT?
[Quoted text hidden]

John C <john.ciolfi.32@gmail.com>Fri, Dec 27, 2024 at 8:24 AM
To: Ihor Radchenko <yantar92@posteo.net>
I'm not clear on how I'd update org-babel-comint-with-output to filter line-by-line. Could you give me a start at it? Also, I'm not sure if line-by-line filtering would work. I suspect MATLAB echos when it sees a new line and a full statement. For example,

    m1 = [4*pi, 3*pi; ...
          2*pi, pi];
   
is a multi-line single statement. The "..." are MATLAB's line continuation syntax. 

Another thought is we could update org-babel-comint-with-output to have an option like:

     (BUFFER EOE-INDICATOR REMOVE-ECHO FULL-BODY EOL-FILTER-INDICATOR)

where optional EOL-FILTER-INDICATOR is text, typically a comment, that is appended to each line in FULL-BODY before it is evaluated. After evaluation, each line in the result containing EOL-FILTER-INDICATOR is removed from the result.

This way, if another language encounters this issue, they can leverage this service.

Thanks
John
[Quoted text hidden]

Ihor Radchenko <yantar92@posteo.net>Fri, Dec 27, 2024 at 10:26 AM
John C <john.ciolfi.32@gmail.com> writes:

> I'm not clear on how I'd update org-babel-comint-with-output to filter
> line-by-line. Could you give me a start at it?

For example, you can add an optional argument to
`org-babel-comint--echo-filter'. When it is passed, BODY argument is
split via `split-string' and then individual lines are removed in order
from STRING.

`org-babel-comint-with-output' is the caller of
`org-babel-comint--echo-filter'.

> ... Also, I'm not sure if
> line-by-line filtering would work. I suspect MATLAB echos when it sees a
> new line and a full statement. For example,
>
>     m1 = [4*pi, 3*pi; ...
>           2*pi, pi];
>
> is a multi-line single statement. The "..." are MATLAB's line continuation
> syntax.

AFAIU, removing line-by-line should still work in such case.

> Another thought is we could update org-babel-comint-with-output to have an
> option like:
>
>      (BUFFER EOE-INDICATOR REMOVE-ECHO FULL-BODY EOL-FILTER-INDICATOR)
>
> where optional EOL-FILTER-INDICATOR is text, typically a comment, that is
> appended to each line in FULL-BODY before it is evaluated. After
> evaluation, each line in the result containing EOL-FILTER-INDICATOR is
> removed from the result.

It may also work. Maybe even better.
Rather than just EOL-FILTER-INDICATOR, we may allow a regular expression
to be provided. Then, everything matching that regular expression will
be removed.

P.S. Is there any reason you are writing off-list? We generally try to
keep the discussion public to keep record of the decision-making.



On Mon, Dec 23, 2024 at 2:23 AM Ihor Radchenko <yantar92@posteo.net> wrote:
Hi John,

It has been a month since the last message in this thread.
Do you need any help working on the patch?

--
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>