* Re: org-babel shell in windows not finishing command
@ 2024-05-20 18:37 7% ` Ihor Radchenko
0 siblings, 0 replies; 17+ results
From: Ihor Radchenko @ 2024-05-20 18:37 UTC (permalink / raw)
To: Vlastimil Vondra, Matt; +Cc: emacs-orgmode
Ihor Radchenko <yantar92@posteo.net> writes:
> Vlastimil Vondra <vlastimil.vondra@gmail.com> writes:
>
>> after upgrading to 9.6. The output is exactly the same.
>
> It would be great if someone with access to Windows could debug this.
No takers, so I went ahead with installing virtual machine.
Confirmed.
The culprit is the fact that org-babel-eval uses process-file and on
Windows cmdproxy.exe provided by emacs has this weird behaviour with
outputting all the below when piped in script text:
In powershell,
echo 'echo foo' | cmdproxy.exe yields
: Microsoft Windows [Version 10.0.19045.2251]
: (c) Microsoft Corporation. All rights reserved.
:
: c:\Users\johndoe\Org>echo foo
: foo
In contrast, cmdproxy.exe -c 'echo foo' outputs
: foo
This looks like Emacs bug, because the same problem with all that
boilerplate text happens when using shell-command-on-region.
I reported it as
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=71081
Let's see what Emacs devs say.
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [relevance 7%]
* Re: [Pre-PATCH] Overhaul of the LaTeX preview system
@ 2024-04-18 21:49 6% Корякин Артём
0 siblings, 0 replies; 17+ results
From: Корякин Артём @ 2024-04-18 21:49 UTC (permalink / raw)
To: Max Nikulin; +Cc: emacs-orgmode
Max Nikulin <manikulin@gmail.com> writes:
> Please, evaluate the following expressions e.g. in the scratch buffer:
>
> shell-file-name
; "pwsh.exe"
> system-type
; windows-nt
> (w32-shell-dos-semantics)
; nil
> (w32-shell-name)
; "pwsh.exe"
For clarification:
I specifically configured[1] MS pwsh as default shell for emacs when I
was trying to get org-fc working[2]. I get some progress back then, but
in the end I just installed busybox using scoop[3] and removed
shell-quote-argument from org-fc functions[4].
> On Windows `shell-quote-argument' should use an escaping strategy
> quite different from POSIX systems. Disclaimer: I do not have a
> Windows box, so it is unlikely that I could help with further
> steps. Perhaps other functions should be used to deal with file names.
> Windows box
I need this laptop to run windows, because pirated CAD programms that I
need for studying would run only on Windows.
I also use emacs for studying and I plan to eventually switch to
KiCAD and freecad on the free operating system.
> Perhaps other functions should be used to deal with file names.
I will try to replace two functions that Karthik mentioned
org-latex-preview--tex-compile-async
org-latex-preview--image-extract-async
with their corresponding versions without shell-quote-argument.
But you also gave me another idea:
I could try to set default shell for emacs to sh that comes from
installed busybox package.
If there is an issue with shell-quote-argument behaving on windows like
on POSIX system, than it might be logical to try to launch these
commands using POSIX shell.
[1]:https://github.com/artsi0m/emacs-organizer/tree/1e1bac13b0f845cb622db08759b9e14fe35fcae7?tab=readme-ov-file#set-powershell-to-default-shell-on-windows
[2]: https://todo.sr.ht/~l3kn/org-fc/43#event-334320
[3]: https://scoop.sh
[4]: https://github.com/artsi0m/emacs-organizer/tree/1e1bac13b0f845cb622db08759b9e14fe35fcae7?tab=readme-ov-file#org-fc-awkfind-on-windows
Sorry for resending this message again.
I forgot to add CC to mailing list.
I don't know how to do this automatically from gnus unfortunately.
^ permalink raw reply [relevance 6%]
* Re: ob-shell: proposal to remove "posh"
2024-01-11 20:30 13% ob-shell: proposal to remove "posh" Matt
2024-01-12 10:04 0% ` Morgan Willcock
2024-01-13 10:43 6% ` Matt
@ 2024-01-13 18:02 10% ` Ihor Radchenko
2 siblings, 0 replies; 17+ results
From: Ihor Radchenko @ 2024-01-13 18:02 UTC (permalink / raw)
To: Matt; +Cc: emacs-orgmode
Matt <matt@excalamus.com> writes:
> There was discussion about different shells, Eric asked for others, "posh" was mentioned as "specially for POSIX compatibility checks", and then a "posh" was added to ob-shell.el by Eric (fb09863f).
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=fb09863fbb35bf15bcf78262b6e31b8b8b8617e7
>
> Around that time are a few stack exchange answers suggesting to use posh, the "Policy-compliant Ordinary SHell", to test for posix compliance. Debian distributed it by saying, "using posh as your /bin/sh may reveal breakage." It seems that posh was used to check for POSIX compliance. It's still available on Debian.
> https://unix.stackexchange.com/questions/48786/how-can-i-test-for-posix-compliance-of-shell-scripts
> https://web.archive.org/web/20130930231522/https://packages.debian.org/sid/posh
> https://packages.debian.org/sid/posh
It is also still available on Gentoo.
> ** No one uses the "Policy-compliant Ordinary SHell"
> The change on August 26, 2022 should have caused a breaking error for someone using the "Policy-compliant Ordinary SHell." The prompt for "posh" in "org-babel-shell-set-prompt-commands" is valid PowerShell syntax (AFAIKT) and invalid bash/dash syntax:
>
> function prompt { "org_babel_sh_prompt> " }
>
> It's not clear to me what this would do in the "Policy-compliant Ordinary SHell." In Bash, functions are defined using a form like:
>
> function fname [()] compound-command [ redirections ]
Simply because I struggled to find what is "posh" at the time I wrote that
code. I only found some obscure page (on the web!) saying the posh might
be PowerShell.
My commit a35d16368 did not intend to introduce anything new.
It turns out that we never supported PowerShell and never announced such
support. So we do not really need to support PowerShell.
> * My stance
> The unfortunate reality is we "support" PowerShell currently. We have code explicitly to handle PowerShell. Changing that could, technically, break someone's workflow.
We do not. It was just my mistake trying to figure out what is "posh".
Since posh is POSIX-compliant, generic PS1/PS2 variables should work, and we
do not need a special entry in `org-babel-shell-set-prompt-commands' -
we can remove "posh" record from there.
I'd rather keep the value in `org-babel-shell-names' though. It should
do no harm.
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [relevance 10%]
* Re: ob-shell: proposal to remove "posh"
2024-01-11 20:30 13% ob-shell: proposal to remove "posh" Matt
2024-01-12 10:04 0% ` Morgan Willcock
@ 2024-01-13 10:43 6% ` Matt
2024-01-13 18:02 10% ` Ihor Radchenko
2 siblings, 0 replies; 17+ results
From: Matt @ 2024-01-13 10:43 UTC (permalink / raw)
Cc: emacs-orgmode
---- On Thu, 11 Jan 2024 21:30:59 +0100 Matt wrote ---
> The change on August 26, 2022 should have caused a breaking error for someone using the "Policy-compliant Ordinary SHell." The prompt for "posh" in "org-babel-shell-set-prompt-commands" is valid PowerShell syntax (AFAIKT) and invalid bash/dash syntax:
>
> function prompt { "org_babel_sh_prompt> " }
>
> It's not clear to me what this would do in the "Policy-compliant Ordinary SHell."
It would cause an error.
I was able to build the "Policy-compliant Ordinary SHell" on Guix using:
#+begin_src guile
(use-modules (guix packages)
(guix download)
(guix build-system gnu)
(gnu packages autotools)
(gnu packages perl)
((guix licenses) #:prefix license:))
(define-public posh
(package
(name "posh")
(version "0.14.1")
(source (origin
(method url-fetch)
(uri (string-append
"https://salsa.debian.org/clint/posh/-/archive/debian/"
version "/posh-debian-" version ".tar.gz"))
(sha256
(base32
"070xnn996cjnc5yzp5819y36sgfikkrplhri4kx5r36h1fmp641d"))))
(native-inputs (list autoconf automake perl))
(build-system gnu-build-system)
(home-page "https://salsa.debian.org/clint/posh")
(synopsis "Policy-compliant Ordinary SHell")
(description
"Policy-compliant Ordinary SHell
posh is a stripped-down version of pdksh that aims for compliance with
Debian's policy, and few extra features.")
(license (list license:gpl2+))))
posh
#+end_src
The result is:
ahab@pequod /gnu/store/64wiqdp9lqjgsz0jg1v1sq2b3afincrb-posh-0.14.1/bin$ ./posh
$ function prompt { "org_babel_sh_prompt> " }
./posh: function: not found
This is expected because "function" is not a keyword in the "Policy-compliant Ordinary SHell." According to the man page:
#+begin_quote
name () command
Defines the function name. See Functions below. Note that redirections
specified after a function definition are performed whenever the
function is executed, not when the function definition is executed.
-- https://manpages.debian.org/bookworm/posh/posh.1.en.html
#+end_quote
So, yes, indeed, commit a35d1636 introduced a breaking change for the "Policy-compliant Ordinary SHell."
--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode
^ permalink raw reply [relevance 6%]
* Re: ob-shell: proposal to remove "posh"
2024-01-11 20:30 13% ob-shell: proposal to remove "posh" Matt
@ 2024-01-12 10:04 0% ` Morgan Willcock
2024-01-13 10:43 6% ` Matt
2024-01-13 18:02 10% ` Ihor Radchenko
2 siblings, 0 replies; 17+ results
From: Morgan Willcock @ 2024-01-12 10:04 UTC (permalink / raw)
To: Matt; +Cc: emacs-orgmode
Matt <matt@excalamus.com> writes:
> AFAIKT, both PowerShell and cmd are MIT licensed:
> - https://github.com/microsoft/terminal
> - https://github.com/PowerShell/PowerShell
>
> The thought of compiling either for a GNU system is...ugh. But maybe someone else has gotten them working? Otherwise, it looks like Microsoft distributes a developer VM image of Windows.
>
> All together, this means there's no *technical* barrier preventing us from running (and hence developing for) PowerShell or cmd.
As I understand it, the code for the cmd.exe shell is not released
anywhere.
What is in the "terminal" repository mentioned above is the code for the
console host process conhost.exe, which is the server process used to
host the shell process and present it (i.e. this is the terminal and not
the shell).
That said, I imagine Wine does contain its own implementation of
cmd.exe, although I wouldn't expect it to be 100% compatible with the
real thing.
--
Morgan Willcock
^ permalink raw reply [relevance 0%]
* ob-shell: proposal to remove "posh"
@ 2024-01-11 20:30 13% Matt
2024-01-12 10:04 0% ` Morgan Willcock
` (2 more replies)
0 siblings, 3 replies; 17+ results
From: Matt @ 2024-01-11 20:30 UTC (permalink / raw)
To: emacs-orgmode
Hi,
I would like people's thoughts on removing the "posh" language header.
ob-shell.el supports a "posh" shell. What is "posh?"
* Posh is not PowerShell
"posh" was added to =ob-shell= in fb09863f with no commit message on December 13, 2013 (Friday the 13th!).
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=fb09863fbb35bf15bcf78262b6e31b8b8b8617e7
A mail message the same day says,
> Eric Schulte writes:
> >> How about the following resolution? We rename ob-sh.el to ob-shell.el.
> >> New "shell" code blocks could use the value of the
> >> `org-babel-sh-command' environment variable. Then sh, bash, zsh, csh,
> >> ash, dash (am I missing any other common ones) use the specific shell
> >> specified.
>
> I've also seen ksh, mksh, posh (the latter specifically for POSIX
> compatibility checks).
https://list.orgmode.org/878uvychr1.fsf@pinto.chemeng.ucl.ac.uk/T/#mc20039f988519d409294dc604b5e0dc0f439307b
There was discussion about different shells, Eric asked for others, "posh" was mentioned as "specially for POSIX compatibility checks", and then a "posh" was added to ob-shell.el by Eric (fb09863f).
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=fb09863fbb35bf15bcf78262b6e31b8b8b8617e7
Around that time are a few stack exchange answers suggesting to use posh, the "Policy-compliant Ordinary SHell", to test for posix compliance. Debian distributed it by saying, "using posh as your /bin/sh may reveal breakage." It seems that posh was used to check for POSIX compliance. It's still available on Debian.
https://unix.stackexchange.com/questions/48786/how-can-i-test-for-posix-compliance-of-shell-scripts
https://web.archive.org/web/20130930231522/https://packages.debian.org/sid/posh
https://packages.debian.org/sid/posh
Wikipedia claims PowerShell is based on POSIX. However, it's not clear if it's "POSIX compliant" (and, if it were, I would be shocked if someone seriously suggested using it for POSIX compatibility checks).
https://en.wikipedia.org/wiki/PowerShell#Grammar
Checking the Microsoft documentation, I find no mention of "posh." PowerShell 7 introduced a "pwsh.exe" binary. However, the same page has a reference for Windows PowerShell 5.1, corresponding to "powershell.exe". This is also corroborated by Wikipedia which says "PowerShell 7 is the replacement for PowerShell Core 6.x products as well as Windows PowerShell 5.1, which is the last supported Windows PowerShell version." "posh" was added to =ob-shell= on December 13, 2013. This would correspond to Powershell 3 or 4 which, AFAIU, both have the binary "powershell.exe."
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_pwsh?view=powershell-7.4
https://en.wikipedia.org/wiki/PowerShell#PowerShell_7
It's possible that Eric meant to add PowerShell and abbreviated it to "posh." However, it appears much more likely that it refers to the "Policy-compliant Ordinary SHell." (https://salsa.debian.org/clint/posh)
* Now what?
So, if we're agreed that "posh" is not PowerShell, what should we do about this?
I propose dropping "posh" from =org-babel-shell-names= and removing it from =org-babel-shell-set-prompt-commands=. That is, let's not explicitly support the "Policy-compliant Ordinary SHell" or PowerShell.
We never supported PowerShell intentionally and only came to it by accident. I don't think anyone has used the "Policy-compliant Ordinary SHell" in at least 2 years. Die-hard adherents to either can still use them, even if we removed the "posh" language header.
** We didn't begin "supporting" PowerShell intentionally (AFAICT)
Up to 2020 and through most of 2022, Powershell wasn't considered supported.
https://list.orgmode.org/87pn707jdz.fsf@gnu.org/T/#u
August 26, 2022 changed that with commit a35d1636. When =org-babel-shell-set-prompt-commands= was added to set the shell prompt, PowerShell syntax was included for completeness because people thought "posh" was PowerShell. AFAIK, no one explicitly asked for PowerShell support as an end-user (then or after). The inclusion of PowerShell appears based on the misunderstanding of "posh" as PowerShell. Since then, the handful of list messages mentioning PowerShell assume it's supported. That "support" is all based on commit a35d1636.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=a35d163685908386833a3d549ed110931bf3915a
https://list.orgmode.org/?q=d%3A17%2FJune%2F2022..+AND+powershell
https://list.orgmode.org/?q=d%3A17%2FJune%2F2022..+AND+posh
AFAICT, PowerShell is mainly (only?) used for tests by people on the mailing list to check that ob-shell can run them. Occasionally, (once every few years) someone specifically asks about it or comments on how it's not (well) supported. Only after commit a35d1636 did people begin saying that PowerShell is supported. People do use PowerShell. However, there are only 36 hits for it on the list in (since Babel was made in 2009) and only a handful of side comments since the change.
** No one uses the "Policy-compliant Ordinary SHell"
The change on August 26, 2022 should have caused a breaking error for someone using the "Policy-compliant Ordinary SHell." The prompt for "posh" in "org-babel-shell-set-prompt-commands" is valid PowerShell syntax (AFAIKT) and invalid bash/dash syntax:
function prompt { "org_babel_sh_prompt> " }
It's not clear to me what this would do in the "Policy-compliant Ordinary SHell." In Bash, functions are defined using a form like:
function fname [()] compound-command [ redirections ]
It says, "Note that for historical reasons, in the most common usage the curly braces that surround the body of the function must be separated from the body by blanks or newlines. This is because the braces are reserved words and are only recognized as such when they are separated from the command list by whitespace or another shell metacharacter. Also, when using the braces, the list must be terminated by a semicolon, a ‘&’, or a newline."
https://www.gnu.org/software/bash/manual/bash.html#Shell-Functions
So, for 'bash --posix', the syntax is invalid. Dash simply errors out with "dash: 1: function: not found". Whatever it does on a POSIX-y shell, it's a show-stopper.
I don't have "posh" available, but I've run the following using dash and bash. It causes Emacs to hang until you hit C-g. Maybe the "Policy-compliant Ordinary SHell" is different and the command works? Since it fails for 'bash --posix' and dash, I doubt it. Here's what I ran:
(org-babel-do-load-languages 'org-babel-load-languages '((shell . t)))
(defconst org-babel-shell-set-prompt-commands
'((t . "function prompt { \"%s\" }")))
#+begin_src dash :session *dash*
echo "hi"
#+end_src
Canceling and switching to the shell that was created, the prompt is "> " (the PS2 prompt). This swallows all input that's not an error. =org-babel-execute:shell= calls =org-babel-sh-initiate-session= which calls =org-babel-comint-wait-for-output= which waits for the =comint-prompt-regexp= which is "org_babel_sh_prompt> ", not "> ", so Emacs hangs.
There have been no complaints from "Policy-compliant Ordinary SHell" users after August 26, 2022 about "posh" being broken. The talk of "posh" (and PowerShell, specifically) since August 26, 2022 is often as an aside.
I conclude that no one uses (or at least cares much about using) the "Policy-compliant Ordinary SHell."
** Both the "Policy-compliant Ordinary SHell" and PowerShell are usable without "posh"
Since the "Policy-compliant Ordinary SHell" is POSIX, it should be straight-forward to use with the "shell" language by replacing "shell-file-name." The "shell" language header will hit the default prompt command which is general POSIX syntax (that is, it changes PS1 and PS2). A shebang should also work.
I had access to a Windows 10 machine recently and verified the following makes PowerShell "work". I use quotes because the following outputs a banner and often fails to remove the prompt, just like the current "posh" behavior.
(org-babel-do-load-languages 'org-babel-load-languages '((shell . t)))
;;; Setup for non-sessions
(setq shell-file-name "C:/Windows/System32/WindowsPowerShell/v1.0/powershell.exe")
#+begin_src shell :results output
echo "hello"
#+end_src
;;; Setup for sessions
;; required by `shell.el' in `shell' command
(setq explicit-shell-file-name
"C:/Windows/System32/WindowsPowerShell/v1.0/powershell.exe")
(setq explicit-powershell.exe-args '("-NoExit"))
;; required by `ob-shell.el' in Org 9.6.6 (distributed with Emacs
;; 29.1 which is the latest build of Emacs for Windows)
(setq org-babel-prompt-command "function prompt { \"org_babel_sh_prompt> \" }")
#+begin_src shell :results output :session *powershell*
echo "world"
#+end_src
* My stance
The unfortunate reality is we "support" PowerShell currently. We have code explicitly to handle PowerShell. Changing that could, technically, break someone's workflow.
Here's how I reason it:
First, AFAIK no PowerShell user would refer to it as "posh". At best they may say "pwsh" and most likely "powershell".
We could keep "posh" for the "Policy-compliant Ordinary SHell" and add languages for "pwsh" or "powershell". This would need to be communicated PowerShell users.
I don't know what other keyword we could use for the "Policy-compliant Ordinary SHell". We could keep "posh" for PowerShell, drop all "support" for the "Policy-compliant Ordinary SHell", and commit to PowerShell only. This would need to be communicated to the "Policy-compliant Ordinary SHell" users.
In any case, we need to communicate our decision, in ORG-NEWS, through our mailing list communications, and definitely by updating the comment.
The cost of keeping PowerShell means maintaining it and whatever weirdo stuff comes with it.
I suggest we make the "t" in =org-babel-shell-set-prompt-commands= a catch-all for anything with PS1/PS2 (anything POSIX-y?) and remove the "posh" language header. The "t" case should handle bash, dash, and (I suspect) the "Policy-compliant Ordinary SHell". This has the potential to break someone using "posh" for PowerShell. However, I see no indication of anyone seriously using it. We've said for years we don't support it. The only places I'm aware of saying we support PowerShell is the code comment and a few people on the mailing list. Moreover, a "breakage" happens anyway, for either "Policy-compliant Ordinary SHell" users (as currently likely exists) or PowerShell users (if we give a new keyword). If we're going to have to put a work-around or message out there anyway, I figure let's see about removing any semblance of supporting PowerShell until we talk it over.
Fortunately, when I wrote the Worg page, I put "the "Policy-compliant Ordinary SHell" for "posh." So, the only place someone might think that PowerShell is supported is from the code comment or from the mailing list.
** PowerShell or cmd.exe?
I'm not claiming that people haven't wanted PowerShell support. It's been discussed a few times on the list. It always reduces to "it would be nice, however we lack a developer or the equipment, so sadly no."
As current ob-shell maintainer, here's how I see it:
I like the idea of supporting PowerShell. I like the idea of supporting cmd.exe much, much more.
Both are associated with a non-free system. Providing support (even partially) for non-free systems is good because it provides an opportunity to teach people about software freedom.
AFAIKT, both PowerShell and cmd are MIT licensed:
- https://github.com/microsoft/terminal
- https://github.com/PowerShell/PowerShell
The thought of compiling either for a GNU system is...ugh. But maybe someone else has gotten them working? Otherwise, it looks like Microsoft distributes a developer VM image of Windows.
All together, this means there's no *technical* barrier preventing us from running (and hence developing for) PowerShell or cmd.
It's important to note that Emacs defaults to cmdproxy.exe on Windows. Using "shell" in Babel on Windows returns from cmd.exe (through cmdproxy.exe). I believe it's best to support the defaults first.
Because it's the default, I've always used cmd when using Emacs on Windows. I would personally much prefer support for cmd.exe over PowerShell. That is, I'd be happy to work on PowerShell stuff after cmd stuff and both after cleaning up ob-shell, ob-comint, and ob-eval.
Thoughts overall?
--
Matt Trzcinski
Emacs Org maintainer (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode
^ permalink raw reply [relevance 13%]
* Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
2023-04-01 23:21 8% ` Matt
@ 2023-04-03 17:42 0% ` Derek Chen-Becker
0 siblings, 0 replies; 17+ results
From: Derek Chen-Becker @ 2023-04-03 17:42 UTC (permalink / raw)
To: Matt; +Cc: Ihor Radchenko, Emacs-orgmode
[-- Attachment #1: Type: text/plain, Size: 1719 bytes --]
I fiddled around a little bit this weekend and confirmed that this (sloppy)
code makes highlighting work for all shell types that sh-script supports:
;;A quick hack to try and support more shells syntax highlight in org babel
(require 'sh-script)
(require 'ob-shell)
(let ((shells (seq-filter (lambda (shell) (not (eq shell 'sh)))
(flatten-tree sh-ancestor-alist))))
(let ((toAppend (mapcar (lambda (shell) `(,(symbol-name shell) . sh))
shells)))
(setq org-src-lang-modes (-distinct (append toAppend
org-src-lang-modes)))))
I'm a relative newcomer to elisp, so comments and suggestions are welcome.
This is basically what I meant by "dynamically amend org-src-lang-modes
based on the contents of sh-ancestor-alist".
Thanks,
Derek
On Sat, Apr 1, 2023 at 5:22 PM Matt <matt@excalamus.com> wrote:
>
> ---- On Thu, 30 Mar 2023 04:55:32 -0400 Ihor Radchenko wrote ---
> > Matt matt@excalamus.com> writes:
> >
> > > I think this approach will work fine. I tried examples for each
> shell type and keywords like if/then/else and function names are
> highlighted.
> >
> > Even for posh (powershell)?
>
> Yes. It's not great since sh-mode looks for Korn-based keywords. It does
> string highlighting and common keywords like 'if', 'exit', and 'param'.
>
>
--
+---------------------------------------------------------------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7 7F42 AFC5 AFEE 96E4 6ACC |
+---------------------------------------------------------------+
[-- Attachment #2: Type: text/html, Size: 3793 bytes --]
^ permalink raw reply [relevance 0%]
* Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
2023-03-30 8:56 8% ` Ihor Radchenko
2023-03-30 23:10 0% ` Derek Chen-Becker
@ 2023-04-01 23:21 8% ` Matt
2023-04-03 17:42 0% ` Derek Chen-Becker
1 sibling, 1 reply; 17+ results
From: Matt @ 2023-04-01 23:21 UTC (permalink / raw)
To: Ihor Radchenko; +Cc: Derek Chen-Becker, Emacs-orgmode
---- On Thu, 30 Mar 2023 04:55:32 -0400 Ihor Radchenko wrote ---
> Matt matt@excalamus.com> writes:
>
> > I think this approach will work fine. I tried examples for each shell type and keywords like if/then/else and function names are highlighted.
>
> Even for posh (powershell)?
Yes. It's not great since sh-mode looks for Korn-based keywords. It does string highlighting and common keywords like 'if', 'exit', and 'param'.
^ permalink raw reply [relevance 8%]
* Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
2023-03-30 8:56 8% ` Ihor Radchenko
@ 2023-03-30 23:10 0% ` Derek Chen-Becker
2023-04-01 23:21 8% ` Matt
1 sibling, 0 replies; 17+ results
From: Derek Chen-Becker @ 2023-03-30 23:10 UTC (permalink / raw)
To: Ihor Radchenko; +Cc: Matt, Emacs-orgmode
[-- Attachment #1: Type: text/plain, Size: 1172 bytes --]
Would it be fair to use the sh-ancestor-alist as a basis for all of the
supported shell variants?
https://github.com/emacs-mirror/emacs/blob/master/lisp/progmodes/sh-script.el#L177
Cheers,
Derek
On Thu, Mar 30, 2023 at 2:54 AM Ihor Radchenko <yantar92@posteo.net> wrote:
> Matt <matt@excalamus.com> writes:
>
> > I think this approach will work fine. I tried examples for each shell
> type and keywords like if/then/else and function names are highlighted.
>
> Even for posh (powershell)?
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>
--
+---------------------------------------------------------------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7 7F42 AFC5 AFEE 96E4 6ACC |
+---------------------------------------------------------------+
[-- Attachment #2: Type: text/html, Size: 3327 bytes --]
^ permalink raw reply [relevance 0%]
* Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
@ 2023-03-30 8:56 8% ` Ihor Radchenko
2023-03-30 23:10 0% ` Derek Chen-Becker
2023-04-01 23:21 8% ` Matt
0 siblings, 2 replies; 17+ results
From: Ihor Radchenko @ 2023-03-30 8:56 UTC (permalink / raw)
To: Matt; +Cc: Derek Chen-Becker, Emacs-orgmode
Matt <matt@excalamus.com> writes:
> I think this approach will work fine. I tried examples for each shell type and keywords like if/then/else and function names are highlighted.
Even for posh (powershell)?
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [relevance 8%]
* Re: [BUG] conda doesn't work in ob-shell sessions
@ 2023-02-15 6:21 7% ` Jack Kamm
0 siblings, 0 replies; 17+ results
From: Jack Kamm @ 2023-02-15 6:21 UTC (permalink / raw)
To: Matt; +Cc: emacs-orgmode
Matt <matt@excalamus.com> writes:
> 1. M-x shell
When I follow your example, in particular starting the session with
M-x shell, then conda works as expected, and behavior is same as you
report.
But when I don't start with M-x shell, instead letting ob-shell create
the session, then conda gives the error about needing to run conda init
(despite the fact that conda init had already set up my zshrc).
Example when _NOT_ using the M-x shell:
#+begin_src emacs-lisp
(org-version nil t)
#+end_src
#+RESULTS:
: Org mode version 9.6.1 (release_9.6.1-250-ge6353d @ /home/jack/devel/org-mode/lisp/)
#+begin_src emacs-lisp
(org-babel-do-load-languages
'org-babel-load-languages
'((shell . t)))
#+end_src
#+begin_src sh :session *myshell* :results output
conda activate ledger
#+end_src
#+RESULTS:
#+begin_example
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
#+end_example
^ permalink raw reply [relevance 7%]
* Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
2023-01-19 16:28 0% ` Osher Jacob
@ 2023-01-20 9:24 7% ` Ihor Radchenko
0 siblings, 0 replies; 17+ results
From: Ihor Radchenko @ 2023-01-20 9:24 UTC (permalink / raw)
To: Osher Jacob; +Cc: Matt, emacs-orgmode
Osher Jacob <osherz5@gmail.com> writes:
> On Wed, Jan 18, 2023 at 11:05 AM Ihor Radchenko <yantar92@posteo.net> wrote:
>
>> If running cmdproxy.exe /c cmdproxy.exe /path/to/input is not wrong, the
>> problem is on Emacs side.
>
>
>> Osher, could you try putting your example script into a file and running
>> the command line directly? What will it output?
>>
>
> Unfortunately, it seems like cmdproxy.exe and cmd.exe cannot accept an
> input file as a command-line argument and execute it.
>
> In the case of running :
> cmdproxy.exe /c cmdproxy.exe /c C:/tmp/inp
>
> We get:
> cmdproxy.exe /c cmdproxy.exe /c C:/tmp/inp
> 'C:/tmp/inp' is not recognized as an internal or external command,
> operable program or batch file.
... which means that I did not dig deep enough.
Apparently, `process-file' is not passing the file to cmdproxy.exe. It
calls `call-process', which reads file and passes its contents as input.
So, we don't need to do awkward manoeuvring with "&" and making the src
block contents into a one-liner. It's about understanding how
cmdproxy.exe handles stdin.
> I think it could be enough to make sure the input ends with a newline, in
> which case it could be done the way I proposed in the first message, that
> is changing:
> (t (org-babel-eval shell-file-name (org-trim body))))))
> to
> (t (org-babel-eval shell-file-name (concat (org-trim body) "\n"))))))
>
> I think as long as this change doesn't break the code in non-Windows
> operating systems (How would we go about verifying this?), it would be
> preferable due to its simplicity and minimality.
This looks like the way to go. For non-Windows systems we have some test
coverage.
I am wondering if we should add tests for powershell and cmdproxy.
Though only users with Windows will actually be able to run those.
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [relevance 7%]
* Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
@ 2023-01-19 16:28 0% ` Osher Jacob
2023-01-20 9:24 7% ` Ihor Radchenko
0 siblings, 1 reply; 17+ results
From: Osher Jacob @ 2023-01-19 16:28 UTC (permalink / raw)
To: Ihor Radchenko, Matt; +Cc: emacs-orgmode
[-- Attachment #1: Type: text/plain, Size: 4038 bytes --]
Thanks for the suggestions!
On Wed, Jan 18, 2023 at 7:09 AM Matt <matt@excalamus.com> wrote:
>
> 1. Another naive work around attempt. Again, I'm going from memory,
> documentation, and what I have previously written.
>
> I have in my init a command to open a terminal when working on Windows
> that looks like:
>
> (start-process "cmd" nil "cmd.exe" "/C" "start" "cmd.exe" "/K" "cd"
> dir)
>
> This starts a cmd process with /c which terminates the prompt after the
> command that follows is run. The command That follows starts another cmd
> process with /k which changes the directory and leaves the terminal open.
>
> I have another command to open QGIS that does the same thing:
>
> (start-process "cmd" nil "cmd.exe" "/C" "start" "\"qgis\"" "cmd.exe"
> "/K" "C:\\Program Files\\QGIS 3.22.3\\bin\\qgis.bat")
>
> It's not clear to me why the extra call to a second cmd with /k is
> needed. Maybe it's copy-pasta or Windows being Windows.
>
> Anyway, I mention these because I wonder if we might be able to do
> something similar.
>
> Try changing `shell-file-name' to "cmd.exe" (ideally, the full path) and
> `shell-command-switch' to "/k" (or maybe "-k"). My hope would be that the
> final call would look something like,
>
> cmd.exe /k cmd.exe /k <executes-input-file-as-stdin-or-whatever>
>
> Given my analysis, I'm not sure if there's a way we could trick the call
> into being `cmd.exe /c cmd.exe /k input-file'.
>
On Wed, Jan 18, 2023 at 11:05 AM Ihor Radchenko <yantar92@posteo.net> wrote:
> If running cmdproxy.exe /c cmdproxy.exe /path/to/input is not wrong, the
> problem is on Emacs side.
> Osher, could you try putting your example script into a file and running
> the command line directly? What will it output?
>
Unfortunately, it seems like cmdproxy.exe and cmd.exe cannot accept an
input file as a command-line argument and execute it.
In the case of running :
cmdproxy.exe /c cmdproxy.exe /c C:/tmp/inp
We get:
cmdproxy.exe /c cmdproxy.exe /c C:/tmp/inp
'C:/tmp/inp' is not recognized as an internal or external command,
operable program or batch file.
If we're insistent on passing the input through the command line arguments,
I can think of two ways to go about this, but both seem undesirable:
- Dumping the content of the input buffer into a ".bat" file and then
passing it as an argument. That way we end up executing "cmdproxy.exe /c
cmdproxy.exe input.bat", but then batch files produce a slightly different
behaviour than running in the context of an interactive shell.
or
- Concating the lines together with the "&" in between them. This would
look like 'cmdproxy.exe /c cmdproxy.exe /c "echo 1 & echo 2 & echo 3",
which is also a pretty nasty solution in my opinion.
This leads me to your next point, which sounds like the most natural way to
go about it.
2. We could write some ob-shell code to explicitly handle Windows cmd and
> powershell. For example, call `process-file' similar to the stdin/cmdline
> branch of `org-babel-sh-evaluate'.
>
> I'm open to this, but, like I've said, I don't have a Windows machine to
> work on right now. I'd not be able to verify it. I, and I'm sure others,
> would be happy to guide you if that's something you'd like to help
> implement.
>
>
I think it could be enough to make sure the input ends with a newline, in
which case it could be done the way I proposed in the first message, that
is changing:
(t (org-babel-eval shell-file-name (org-trim body))))))
to
(t (org-babel-eval shell-file-name (concat (org-trim body) "\n"))))))
I think as long as this change doesn't break the code in non-Windows
operating systems (How would we go about verifying this?), it would be
preferable due to its simplicity and minimality.
If for any other reason you would rather not change
the org-babel-sh-evaluate, and would like to implement a separate function
for Windows, that's also a viable option.
I am willing to try and go down that path, although I'm not a very
experienced lisper.
Do any of these options sound like they could work well?
[-- Attachment #2: Type: text/html, Size: 5800 bytes --]
^ permalink raw reply [relevance 0%]
* Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
2023-01-17 19:53 8% ` Osher Jacob
@ 2023-01-18 5:09 3% ` Matt
0 siblings, 1 reply; 17+ results
From: Matt @ 2023-01-18 5:09 UTC (permalink / raw)
To: Osher Jacob; +Cc: emacs-orgmode
---- On Tue, 17 Jan 2023 14:53:39 -0500 Osher Jacob wrote ---
> changing shell-command-switch to "/k" or "-k", I get a similar output:
Thanks for checking that.
> You also mentioned the source code block is being passed through the "-c" flag as a command-line argument.I might be misunderstanding something here, but it seems like it is being passed through the stdin of the shell process when the calls process-file -> call-process are being made.
That the block source is passed through "-c" is an educated guess. I keep finding myself in C land where it's less clear to me what precisely happens. You're correct that `call-process' acts on the block source. It's not clear to me whether that's done through stdin or whether being passed through stdin would even help us. Here's what I'm seeing.
Evaluating the block goes through 4 lisp commands and then into C:
1. Calls `(org-babel-eval shell-file-name (org-trim body))'
This evaluates as...
(org-babel-eval
"cmdproxy.exe" ; command, whatever `shell-file-name' is
(org-trim body) ; query, the block body
)
The query parameter (the block body) gets inserted into a temp buffer on which `org-babel--shell-command-on-region' is called.
2. Calls `(org-babel--shell-command-on-region command error-buffer)' on a temp buffer containing block body.
This evaluates as...
(org-babel--shell-command-on-region
command ; "cmdproxy.exe"
error-buffer ; #<buffer *Org-Babel Error*>
)
This in turn calls `process-file' where INPUT-FILE is a temp file containing the contents of the buffer `org-babel--shell-command-on-region' was called on (that is, the temp file contains the block body).
3. Calls `(process-file shell-file-name input-file (if error-file (list t error-file) t) nil shell-command-switch command)'
This evaluates as...
(process-file
"cmdproxy.exe" ; shell-file-name
"/path/to/temp/containing/block/source" ; input-file
(t "/tmp/babel jTCHe/ob-error-yHOivA") ; output destination (like call-process's DESTINATION)
nil ; display
"-c" ; args
"cmdproxy.exe" ; args
)
Of course, I imagine the paths would be Windows paths when run on Windows.
According to the documentation, the args are passed to the process (cmdproxy.exe) "verbatim without filename handling, as `call-process' does." The `call-process' documentation says, "remaining arguments ARGS are strings passed as command arguments to PROGRAM."
To me, that sounds like cmdproxy.exe gets "passed into" cmdproxy.exe. This would explain the nested shell calls. If all this were written out, I imagine it would look something like:
cmdproxy.exe -c cmdproxy.exe
What's not clear to me is how the INPUT-FILE is handled.
The `process-file' documentation says "normally". I assume it's how `call-process' handles INPUT-FILE, but the documentation doesn't really address it.
Finally, `process-file' finally calls `call-process'.
4. Calls `(apply 'call-process program (or lc infile) (if stderr-file (list (car buffer) stderr-file) buffer) display args)'
This evaluates as...
(apply 'call-process
program ; cmdproxy
(or lc infile) ; local-copy of the INFILE, "/path/to/temp/containing/block/source"
(if stderr-file (list (car buffer) stderr-file) buffer) ; (t "/tmp/babel jTCHe/ob-error-yHOivA" )
display ; nil
args) ; ("-c" "cmdproxy.exe")
How this actually works is non-trivial (https://git.savannah.gnu.org/cgit/emacs.git/tree/src/callproc.c#n250) and not something I understand at the moment. We can see here that indeed a call like `cmdproxy.exe -c cmdproxy.exe' is being made. Still, I'm not sure how the INPUT-FILE gets processed. For example, is it passed in the second cmdproxy call? Or, maybe it gets called first and then the second call to cmdproxy happens and hangs? I don't know.
> Any ideas on how to proceed from here?
I have two ideas.
1. Another naive work around attempt. Again, I'm going from memory, documentation, and what I have previously written.
I have in my init a command to open a terminal when working on Windows that looks like:
(start-process "cmd" nil "cmd.exe" "/C" "start" "cmd.exe" "/K" "cd" dir)
This starts a cmd process with /c which terminates the prompt after the command that follows is run. The command That follows starts another cmd process with /k which changes the directory and leaves the terminal open.
I have another command to open QGIS that does the same thing:
(start-process "cmd" nil "cmd.exe" "/C" "start" "\"qgis\"" "cmd.exe" "/K" "C:\\Program Files\\QGIS 3.22.3\\bin\\qgis.bat")
It's not clear to me why the extra call to a second cmd with /k is needed. Maybe it's copy-pasta or Windows being Windows.
Anyway, I mention these because I wonder if we might be able to do something similar.
Try changing `shell-file-name' to "cmd.exe" (ideally, the full path) and `shell-command-switch' to "/k" (or maybe "-k"). My hope would be that the final call would look something like,
cmd.exe /k cmd.exe /k <executes-input-file-as-stdin-or-whatever>
Given my analysis, I'm not sure if there's a way we could trick the call into being `cmd.exe /c cmd.exe /k input-file'.
2. We could write some ob-shell code to explicitly handle Windows cmd and powershell. For example, call `process-file' similar to the stdin/cmdline branch of `org-babel-sh-evaluate'.
I'm open to this, but, like I've said, I don't have a Windows machine to work on right now. I'd not be able to verify it. I, and I'm sure others, would be happy to guide you if that's something you'd like to help implement.
Thoughts?
^ permalink raw reply [relevance 3%]
* Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
@ 2023-01-17 19:53 8% ` Osher Jacob
2023-01-18 5:09 3% ` Matt
0 siblings, 1 reply; 17+ results
From: Osher Jacob @ 2023-01-17 19:53 UTC (permalink / raw)
To: Matt; +Cc: emacs-orgmode
[-- Attachment #1.1: Type: text/plain, Size: 4124 bytes --]
Hey, thanks for the input!
My shell-file-name is indeed pointing to cmdproxy.exe, and after changing
shell-command-switch to "/k" or "-k", I get a similar output:
#+begin_src shell
echo 1
echo 2
echo 3
#+end_src
#+RESULTS:
| Microsoft | Windows | [Version | 10.0.14393] | | |
|
| (c) | 2016 | Microsoft | Corporation. | All | rights |
reserved. |
| | | | | | |
|
| c:\tmp>echo | 1 | | | | |
|
| 1 | | | | | |
|
| | | | | | |
|
| c:\tmp>echo | 2 | | | | |
|
| 2 | | | | | |
|
| | | | | | |
|
| c:\tmp>More? | | | | | |
|
Where only the last line is missing.
Another observation that might be relevant, is that the block execution
works well when I set shell-file-name to "powershell.exe"
#+begin_src shell
echo 1
echo 2
echo 3
#+end_src
#+RESULTS:
| Windows | PowerShell | | | | | |
|
| Copyright | (C) | 2016 | Microsoft | Corporation. | All | rights |
reserved. |
| | | | | | | |
|
| PS | C:\tmp> | echo | 1 | | | |
|
| 1 | | | | | | |
|
| PS | C:\tmp> | echo | 2 | | | |
|
| 2 | | | | | | |
|
| PS | C:\tmp> | echo | 33 | | | |
|
| PS | C:\tmp> | | | | | |
|
And here, it might be that the missing newline in the last command is
visible, hence the double '3' character (one is a part of the "echo 3"
input, the second is the output "3" echoed back).
You also mentioned the source code block is being passed through the "-c"
flag as a command-line argument.
I might be misunderstanding something here, but it seems like it is being
passed through the stdin of the shell process when the calls process-file
-> call-process are being made.
Just to confirm, I've made a simple utility that prints the contents of
argc and argv as hex strings, and got the following output when setting it
in place of the the shell-file-name:
#+begin_src shell
echo 1
echo 2
echo 3
#+end_src
#+RESULTS:
| argc: | 3 |
| 633a5c6d73797336345c686f6d655c4f736865725c6563686f5f6865782e657865 | |
| 2d63 | |
| 633a2f6d73797336342f686f6d652f4f736865722f6563686f5f6865782e657865 | |
Which translates to:
c:\msys64\home\Osher\echo_hex.exe -c c:/msys64/home/Osher/echo_hex.exe
Exposing that there are two nested shell-file-name running.
And now I was curious to see if babel executes cmdproxy.exe -c
cmdproxy.exe, so I fired up Procmon, and indeed that was the case (see
attached screenshot).
[image: image.png]
Not sure if this nesting is really relevant to the problem at hand, but
thought it could be useful information.
Any ideas on how to proceed from here?
On Tue, Jan 17, 2023 at 3:07 AM Matt <matt@excalamus.com> wrote:
>
> ---- On Mon, 16 Jan 2023 16:40:27 -0500 Matt wrote ---
> > That is, go through the steps to reproduce and before executing the
> block, run `M-: (setq shell-command-switch "/k")'.
>
> Whoa, just re-read this after stepping away and it sounds super
> demanding! Please excuse me, I've been trying to use active voice while
> writing documentation.
>
> Let me try that again:
>
> What happens if you go through the steps to reproduce and run `M-: (setq
> shell-command-switch "/k") before executing the block?
>
[-- Attachment #1.2: Type: text/html, Size: 5897 bytes --]
[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 13297 bytes --]
^ permalink raw reply [relevance 8%]
* Re: Org 9.6-pre and Bash sessions
@ 2022-10-21 5:29 6% ` Ihor Radchenko
0 siblings, 0 replies; 17+ results
From: Ihor Radchenko @ 2022-10-21 5:29 UTC (permalink / raw)
To: Rudolf Adamkovič; +Cc: Ihor Radchenko, emacs-orgmode
[-- Attachment #1: Type: text/plain, Size: 811 bytes --]
Ihor Radchenko <yantar92@posteo.net> writes:
> Rudolf Adamkovič <salutis@me.com> writes:
>
>> :
>> : > > hello
>>
>> on the subsequent runs.
>>
>> Better than the new version but still wrong. :)
>
> And this is what drove me crazy during debugging. I do not understand
> the logic of all that ob-comint code.
>
> I have identified that the hang happens because Org does not change PS2
> prompt. Just PS1. But fixing this would yield to
>
> : org_babel_sh_prompt> org_babel_sh_prompt> hello
>
> Then, I tried to see how the original code works. And it does not
> :facepalm:
>
> I asked emacs-devel
> https://yhetil.org/emacs-devel/87y1tgqhmc.fsf@localhost/T/#u
See the attached tentative patch.
I'd appreciate some testing. Hopefully, I did not break anything.
Comint is tricky.
[-- Attachment #2: 0001-ob-shell-Fix-multi-line-scripts-in-sessions.patch --]
[-- Type: text/x-patch, Size: 4766 bytes --]
From 4c6eead351cbdce1b9210a738c65b3a139d1cc0c Mon Sep 17 00:00:00 2001
Message-Id: <4c6eead351cbdce1b9210a738c65b3a139d1cc0c.1666330028.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Fri, 21 Oct 2022 13:21:57 +0800
Subject: [PATCH] ob-shell: Fix multi-line scripts in sessions
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
* lisp/ob-comint.el (org-babel-comint-with-output): Cleanup empty
output. Such output is spitted unnecessarily for multi-line scripts.
* lisp/ob-shell.el (org-babel-shell-set-prompt-commands): Disable
PS2 and equivalent prompts. Make sure that PROMPT_COMMAND does not
interfere with PS1 setting in POSIX shells.
(org-babel-sh-evaluate): Do not send input line-by-line. Instead, let
`org-babel-coming-with-output' handle waiting for the output as well
as recording it. Update to the new `org-babel-coming-with-output'
behavior of cleaning empty outputs.
* testing/lisp/test-ob-shell.el (test-ob-shell/session): Add test.
Reported-by: Rudolf Adamkovič <salutis@me.com>
Link: https://list.orgmode.org/orgmode/m2r0zboix1.fsf@me.com/
---
lisp/ob-comint.el | 2 +-
lisp/ob-shell.el | 29 ++++++++++++++---------------
testing/lisp/test-ob-shell.el | 6 +++++-
3 files changed, 20 insertions(+), 17 deletions(-)
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 36a55d36c..ec7d3f6c9 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -109,7 +109,7 @@ (defmacro org-babel-comint-with-output (meta &rest body)
"\n" "[\r\n]+" (regexp-quote (or ,full-body "")))
string-buffer))
(setq string-buffer (substring string-buffer (match-end 0))))
- (split-string string-buffer comint-prompt-regexp)))))
+ (delete "" (split-string string-buffer comint-prompt-regexp))))))
(defun org-babel-comint-input-command (buffer cmd)
"Pass CMD to BUFFER.
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 4d579ae87..d38d2d335 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -47,10 +47,15 @@ (defvar org-babel-default-header-args:shell '())
(defvar org-babel-shell-names)
(defconst org-babel-shell-set-prompt-commands
- '(("fish" . "function fish_prompt\n\techo \"%s\"\nend")
- ("csh" . "set prompt=\"%s\"")
+ '(;; Fish has no PS2 equivalent.
+ ("fish" . "function fish_prompt\n\techo \"%s\"\nend")
+ ;; prompt2 is like PS2 in POSIX shells.
+ ("csh" . "set prompt=\"%s\"\nset prompt2=\"\"")
+ ;; PowerShell, similar to fish, does not have PS2 equivalent.
("posh" . "function prompt { \"%s\" }")
- (t . "PS1=\"%s\""))
+ ;; PROMPT_COMMAND can override PS1 settings. Disable it.
+ ;; Disable PS2 to avoid garbage in multi-line inputs.
+ (t . "PROMPT_COMMAND=;PS1=\"%s\";PS2="))
"Alist assigning shells with their prompt setting command.
Each element of the alist associates a shell type from
@@ -299,20 +304,14 @@ (defun org-babel-sh-evaluate (session body &optional params stdin cmdline)
#'org-babel-sh-strip-weird-long-prompt
(mapcar
#'org-trim
- (butlast
+ (butlast ; Remove eoe indicator
(org-babel-comint-with-output
(session org-babel-sh-eoe-output t body)
- (dolist (line (append (split-string (org-trim body) "\n")
- (list org-babel-sh-eoe-indicator)))
- (insert line)
- (comint-send-input nil t)
- (while (save-excursion
- (goto-char comint-last-input-end)
- (not (re-search-forward
- comint-prompt-regexp nil t)))
- (accept-process-output
- (get-buffer-process (current-buffer))))))
- 2))
+ (insert (org-trim body) "\n"
+ org-babel-sh-eoe-indicator)
+ (comint-send-input nil t))
+ ;; Remove `org-babel-sh-eoe-indicator' output line.
+ 1))
"\n"))
;; External shell script, with or without a predefined
;; shebang.
diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el
index a0d5a8d22..05c369174 100644
--- a/testing/lisp/test-ob-shell.el
+++ b/testing/lisp/test-ob-shell.el
@@ -46,7 +46,11 @@ (ert-deftest test-ob-shell/session ()
ob-comint.el, which was not previously tested."
(let ((res (org-babel-execute:sh "echo 1; echo 2" '((:session . "yes")))))
(should res)
- (should (listp res))))
+ (should (listp res)))
+ ;; Test multi-line input.
+ (let ((res (org-babel-execute:sh "if true; then\necho \"yes\"\nfi" '((:session . "yes")))))
+ (should res)
+ (should (string= "yes" res))))
; A list of tests using the samples in ob-shell-test.org
(ert-deftest ob-shell/generic-uses-no-arrays ()
--
2.35.1
[-- Attachment #3: Type: text/plain, Size: 225 bytes --]
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply related [relevance 6%]
* [PATCH] babel output seems to drop anything before % (in session)
@ 2022-06-18 3:20 3% ` Ihor Radchenko
0 siblings, 0 replies; 17+ results
From: Ihor Radchenko @ 2022-06-18 3:20 UTC (permalink / raw)
To: Felix Freeman; +Cc: emacs-orgmode, kaushal.modi, nick
[-- Attachment #1: Type: text/plain, Size: 788 bytes --]
"Felix Freeman" <libsys@hacktivista.org> writes:
> Today I stumbed upon the same bug.
>
> Sadly I have no idea how to fix it, but thought it would be nice to post
> about it here so the bug doesn't get lost in the sands of time.
I finally managed to get some working fix.
At least it kind of works on my system using bash.
The idea is changing shell prompt to something unique when we initialize
the session.
See the attached patch.
Please test this on "sh" "zsh" "fish" "csh" "ash" "dash" "ksh" "mksh"
and "posh".
Especially on fish, csh, and posh (which I believe refers to
PowerShell).
Also, doing PS1="blabla > " in bash may not always work. At least it
does not work with https://github.com/nojhan/liquidprompt
So, more reliable ways to change prompt are welcome.
Best,
Ihor
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-disabled-test-for-in-babel-shell-output.patch --]
[-- Type: text/x-patch, Size: 2022 bytes --]
From 141f810a658d652e1cb3a147c5f71659f62df86f Mon Sep 17 00:00:00 2001
Message-Id: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com>
From: Daniele Pizzolli <dan+git@toel.it>
Date: Wed, 5 May 2021 17:00:28 +0200
Subject: [PATCH 1/2] Add disabled test for % in babel shell output
---
testing/lisp/test-ob-shell.el | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el
index 2f346f699..e81058ac7 100644
--- a/testing/lisp/test-ob-shell.el
+++ b/testing/lisp/test-ob-shell.el
@@ -106,6 +106,35 @@ (ert-deftest ob-shell/simple-list ()
"#+BEGIN_SRC sh :results output :var l='(1 2)\necho ${l}\n#+END_SRC"
(org-trim (org-babel-execute-src-block))))))
+(ert-deftest ob-shell/percent-simple ()
+ "Test percent in output."
+ (should
+ (equal "one 0% two\ntree 0% four"
+ (org-test-with-temp-text
+ "#+BEGIN_SRC bash :results output verbatim\necho one 0% two\necho tree 0% four\n#+END_SRC"
+ (org-trim (org-babel-execute-src-block)))))
+ (should
+ (equal "five 0% two\ntree 0% four"
+ (org-test-with-temp-text
+ "#+BEGIN_SRC bash :results output verbatim\necho five 0% two\necho tree 0% four\n#+END_SRC"
+ (org-trim (org-babel-execute-src-block))))))
+
+(ert-deftest ob-shell/percent-session ()
+ "Test percent in output for session.
+
+At the second iteration the string before % is dropped."
+ :expected-result :failed
+ (should
+ (equal "one 0% two\ntree 0% four"
+ (org-test-with-temp-text
+ "#+BEGIN_SRC bash :results output verbatim :session sess\necho one 0% two\necho tree 0% four\n#+END_SRC"
+ (org-trim (org-babel-execute-src-block)))))
+ (should
+ (equal "five 0% six\nseven 0% eight"
+ (org-test-with-temp-text
+ "#+BEGIN_SRC bash :results output verbatim :session sess\necho five 0% six\necho seven 0% eight\n#+END_SRC"
+ (org-trim (org-babel-execute-src-block))))))
+
(provide 'test-ob-shell)
;;; test-ob-shell.el ends here
--
2.35.1
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-ob-shell-Fix-output-containing-strings-matching-comi.patch --]
[-- Type: text/x-patch, Size: 6579 bytes --]
From 4f7a1369e72b22836cb74931c7c86e845155a729 Mon Sep 17 00:00:00 2001
Message-Id: <4f7a1369e72b22836cb74931c7c86e845155a729.1655522152.git.yantar92@gmail.com>
In-Reply-To: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com>
References: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sat, 18 Jun 2022 11:11:12 +0800
Subject: [PATCH 2/2] ob-shell: Fix output containing strings matching
`comint-prompt-regexp'
* lisp/ob-comint.el (org-babel-comint-wait-for-output): Do not rely on
`face-at-point' returning non-nil.
* lisp/ob-shell.el (org-babel-shell-set-prompt-commands): New constant
holding shell-specific commands to change prompt.
(org-babel-prompt-command): New variable holding command to be user to
set distinguishable prompt.
(org-babel-shell-initialize): Set `org-babel-prompt-command' according
to shell name.
(org-babel-sh-prompt): New variable holding default shell prompt.
(org-babel-sh-initiate-session): Change the default prompt to
`org-babel-sh-prompt' and alter `comint-prompt-regexp' to match it
tightly.
* testing/lisp/test-ob-shell.el (ob-shell/percent-session): Do not
expect failure in the fixed test.
Fixes https://list.orgmode.org/CKK9TULBP2BG.2UITT31YJV03J@laptop/T/#mc8e3ca2f5f1b9a94040a68b4c6201234b209041c
---
lisp/ob-comint.el | 4 +---
lisp/ob-shell.el | 42 ++++++++++++++++++++++++++++++-----
testing/lisp/test-ob-shell.el | 4 ++--
3 files changed, 39 insertions(+), 11 deletions(-)
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 427aba341..1d3e24607 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -124,9 +124,7 @@ (defun org-babel-comint-wait-for-output (buffer)
(while (progn
(goto-char comint-last-input-end)
(not (and (re-search-forward comint-prompt-regexp nil t)
- (goto-char (match-beginning 0))
- (string= (face-name (face-at-point))
- "comint-highlight-prompt"))))
+ (goto-char (match-beginning 0)))))
(accept-process-output (get-buffer-process buffer)))))
(defun org-babel-comint-eval-invisibly-and-wait-for-file
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index c25941a44..f80783653 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -42,6 +42,23 @@ (declare-function orgtbl-to-generic "org-table" (table params))
(defvar org-babel-default-header-args:shell '())
(defvar org-babel-shell-names)
+(defconst org-babel-shell-set-prompt-commands
+ '(("fish" . "function fish_prompt\n\techo \"%s\"\nend")
+ ("csh" . "set prompt=\"%s\"")
+ ("posh" . "function prompt { \"%s\" }")
+ (t . "PS1=\"%s\""))
+ "Alist assigning shells with their prompt setting command.
+
+Each element of the alist associates a shell type from
+`org-babel-shell-names' with a template used to create a command to
+change the default prompt. The template is an argument to `format'
+that will be called with a single additional argument: prompt string.
+
+The fallback association template is defined in (t . \"template\")
+alist element.")
+
+(defvar org-babel-prompt-command)
+
(defun org-babel-shell-initialize ()
"Define execution functions associated to shell names.
This function has to be called whenever `org-babel-shell-names'
@@ -51,7 +68,10 @@ (defun org-babel-shell-initialize ()
(eval `(defun ,(intern (concat "org-babel-execute:" name))
(body params)
,(format "Execute a block of %s commands with Babel." name)
- (let ((shell-file-name ,name))
+ (let ((shell-file-name ,name)
+ (org-babel-prompt-command
+ (or (alist-get ,name org-babel-shell-set-prompt-commands)
+ (alist-get t org-babel-shell-set-prompt-commands))))
(org-babel-execute:shell body params))))
(eval `(defalias ',(intern (concat "org-babel-variable-assignments:" name))
'org-babel-variable-assignments:shell
@@ -206,6 +226,13 @@ (defun org-babel-sh-var-to-string (var &optional sep hline)
(mapconcat echo-var var "\n"))
(t (funcall echo-var var)))))
+(defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'"
+ "String to indicate that evaluation has completed.")
+(defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
+ "String to indicate that evaluation has completed.")
+(defvar org-babel-sh-prompt "org_babel_sh_prompt> "
+ "String to set prompt in session shell.")
+
(defun org-babel-sh-initiate-session (&optional session _params)
"Initiate a session named SESSION according to PARAMS."
(when (and session (not (string= session "none")))
@@ -213,17 +240,20 @@ (defun org-babel-sh-initiate-session (&optional session _params)
(or (org-babel-comint-buffer-livep session)
(progn
(shell session)
+ ;; Set unique prompt for easier analysis of the output.
+ (org-babel-comint-wait-for-output (current-buffer))
+ (org-babel-comint-input-command
+ (current-buffer)
+ (format org-babel-prompt-command org-babel-sh-prompt))
+ (setq-local comint-prompt-regexp
+ (concat "^" (regexp-quote org-babel-sh-prompt)
+ " *"))
;; Needed for Emacs 23 since the marker is initially
;; undefined and the filter functions try to use it without
;; checking.
(set-marker comint-last-output-start (point))
(get-buffer (current-buffer)))))))
-(defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'"
- "String to indicate that evaluation has completed.")
-(defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
- "String to indicate that evaluation has completed.")
-
(defun org-babel-sh-evaluate (session body &optional params stdin cmdline)
"Pass BODY to the Shell process in BUFFER.
If RESULT-TYPE equals `output' then return a list of the outputs
diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el
index e81058ac7..2bfc29228 100644
--- a/testing/lisp/test-ob-shell.el
+++ b/testing/lisp/test-ob-shell.el
@@ -122,8 +122,8 @@ (ert-deftest ob-shell/percent-simple ()
(ert-deftest ob-shell/percent-session ()
"Test percent in output for session.
-At the second iteration the string before % is dropped."
- :expected-result :failed
+Percent is matching the default value of `comint-prompt-regexp', which
+can cause issues with analysis."
(should
(equal "one 0% two\ntree 0% four"
(org-test-with-temp-text
--
2.35.1
^ permalink raw reply related [relevance 3%]
Results 1-17 of 17 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2022-06-07 0:01 babel output seems to drop anything before % (in session) Felix Freeman via General discussions about Org-mode.
2022-06-18 3:20 3% ` [PATCH] " Ihor Radchenko
2022-10-13 15:14 Org 9.6-pre and Bash sessions Rudolf Adamkovič
2022-10-14 3:44 ` Ihor Radchenko
2022-10-15 20:56 ` Rudolf Adamkovič
2022-10-17 8:34 ` Ihor Radchenko
2022-10-21 5:29 6% ` Ihor Radchenko
2022-12-12 8:17 org-babel shell in windows not finishing command Vlastimil Vondra
2022-12-12 12:39 ` Ihor Radchenko
2022-12-12 13:07 ` Vlastimil Vondra
2022-12-12 13:10 ` Ihor Radchenko
2022-12-12 15:28 ` Vlastimil Vondra
2022-12-13 7:45 ` Ihor Radchenko
2024-05-20 18:37 7% ` Ihor Radchenko
2023-01-16 16:27 [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)] Osher Jacob
2023-01-16 21:40 ` Matt
2023-01-17 1:07 ` Matt
2023-01-17 19:53 8% ` Osher Jacob
2023-01-18 5:09 3% ` Matt
2023-01-18 9:05 ` Ihor Radchenko
2023-01-19 16:28 0% ` Osher Jacob
2023-01-20 9:24 7% ` Ihor Radchenko
2023-02-06 19:39 [PATCH] Async evaluation in ob-shell Matt
2023-02-11 20:56 ` jackkamm
2023-02-12 19:02 ` Matt
2023-02-13 3:16 ` Jack Kamm
2023-02-13 20:11 ` [BUG] conda doesn't work in ob-shell sessions Matt
2023-02-15 6:21 7% ` Jack Kamm
2023-03-28 17:24 Font lock for org-babel shell scripts? Derek Chen-Becker
2023-03-28 21:53 ` [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?) Matt
2023-03-29 9:35 ` Ihor Radchenko
2023-03-29 17:04 ` Derek Chen-Becker
2023-03-29 20:17 ` Matt
2023-03-30 8:56 8% ` Ihor Radchenko
2023-03-30 23:10 0% ` Derek Chen-Becker
2023-04-01 23:21 8% ` Matt
2023-04-03 17:42 0% ` Derek Chen-Becker
2024-01-11 20:30 13% ob-shell: proposal to remove "posh" Matt
2024-01-12 10:04 0% ` Morgan Willcock
2024-01-13 10:43 6% ` Matt
2024-01-13 18:02 10% ` Ihor Radchenko
2024-04-18 21:49 6% [Pre-PATCH] Overhaul of the LaTeX preview system Корякин Артём
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).