ob-shell: proposal to remove "posh"

ob-shell: proposal to remove "posh"

[Matt]

Hi,

I would like people's thoughts on removing the "posh" language header.

ob-shell.el supports a "posh" shell.  What is "posh?"

* Posh is not PowerShell
"posh" was added to =ob-shell= in fb09863f with no commit message on December 13, 2013 (Friday the 13th!).
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=fb09863fbb35bf15bcf78262b6e31b8b8b8617e7

A mail message the same day says,

> Eric Schulte writes:
> >> How about the following resolution?  We rename ob-sh.el to ob-shell.el.
> >> New "shell" code blocks could use the value of the
> >> `org-babel-sh-command' environment variable.  Then sh, bash, zsh, csh,
> >> ash, dash (am I missing any other common ones) use the specific shell
> >> specified.
>
> I've also seen ksh, mksh, posh (the latter specifically for POSIX
> compatibility checks).
https://list.orgmode.org/878uvychr1.fsf@pinto.chemeng.ucl.ac.uk/T/#mc20039f988519d409294dc604b5e0dc0f439307b

There was discussion about different shells, Eric asked for others, "posh" was mentioned as "specially for POSIX compatibility checks", and then a "posh" was added to ob-shell.el by Eric (fb09863f).
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=fb09863fbb35bf15bcf78262b6e31b8b8b8617e7

Around that time are a few stack exchange answers suggesting to use posh, the "Policy-compliant Ordinary SHell", to test for posix compliance.  Debian distributed it by saying, "using posh as your /bin/sh may reveal breakage."  It seems that posh was used to check for POSIX compliance.  It's still available on Debian.
https://unix.stackexchange.com/questions/48786/how-can-i-test-for-posix-compliance-of-shell-scripts
https://web.archive.org/web/20130930231522/https://packages.debian.org/sid/posh
https://packages.debian.org/sid/posh

Wikipedia claims PowerShell is based on POSIX.  However, it's not clear if it's "POSIX compliant" (and, if it were, I would be shocked if someone seriously suggested using it for POSIX compatibility checks).
https://en.wikipedia.org/wiki/PowerShell#Grammar

Checking the Microsoft documentation, I find no mention of "posh."  PowerShell 7 introduced a "pwsh.exe" binary.  However, the same page has a reference for Windows PowerShell 5.1, corresponding to "powershell.exe".  This is also corroborated by Wikipedia which says "PowerShell 7 is the replacement for PowerShell Core 6.x products as well as Windows PowerShell 5.1, which is the last supported Windows PowerShell version."  "posh" was added to =ob-shell= on December 13, 2013.  This would correspond to Powershell 3 or 4 which, AFAIU, both have the binary "powershell.exe."
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_pwsh?view=powershell-7.4
https://en.wikipedia.org/wiki/PowerShell#PowerShell_7

It's possible that Eric meant to add PowerShell and abbreviated it to "posh."  However, it appears much more likely that it refers to the "Policy-compliant Ordinary SHell." (https://salsa.debian.org/clint/posh)

* Now what?
So, if we're agreed that "posh" is not PowerShell, what should we do about this?

I propose dropping "posh" from =org-babel-shell-names= and removing it from =org-babel-shell-set-prompt-commands=.  That is, let's not explicitly support the "Policy-compliant Ordinary SHell" or PowerShell.

We never supported PowerShell intentionally and only came to it by accident.  I don't think anyone has used the "Policy-compliant Ordinary SHell" in at least 2 years.  Die-hard adherents to either can still use them, even if we removed the "posh" language header.

** We didn't begin "supporting" PowerShell intentionally (AFAICT)
Up to 2020 and through most of 2022, Powershell wasn't considered supported.
https://list.orgmode.org/87pn707jdz.fsf@gnu.org/T/#u

August 26, 2022 changed that with commit a35d1636.  When =org-babel-shell-set-prompt-commands= was added to set the shell prompt, PowerShell syntax was included for completeness because people thought "posh" was PowerShell.  AFAIK, no one explicitly asked for PowerShell support as an end-user (then or after).  The inclusion of PowerShell appears based on the misunderstanding of "posh" as PowerShell.  Since then, the handful of list messages mentioning PowerShell assume it's supported.  That "support" is all based on commit a35d1636.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=a35d163685908386833a3d549ed110931bf3915a
https://list.orgmode.org/?q=d%3A17%2FJune%2F2022..+AND+powershell
https://list.orgmode.org/?q=d%3A17%2FJune%2F2022..+AND+posh

AFAICT, PowerShell is mainly (only?) used for tests by people on the mailing list to check that ob-shell can run them.  Occasionally, (once every few years) someone specifically asks about it or comments on how it's not (well) supported.  Only after commit a35d1636 did people begin saying that PowerShell is supported.  People do use PowerShell.  However, there are only 36 hits for it on the list in (since Babel was made in 2009) and only a handful of side comments since the change.

** No one uses the "Policy-compliant Ordinary SHell"
The change on August 26, 2022 should have caused a breaking error for someone using the "Policy-compliant Ordinary SHell."  The prompt for "posh" in "org-babel-shell-set-prompt-commands" is valid PowerShell syntax (AFAIKT) and invalid bash/dash syntax:

function prompt { "org_babel_sh_prompt> " }

It's not clear to me what this would do in the "Policy-compliant Ordinary SHell."  In Bash, functions are defined using a form like:

function fname [()] compound-command [ redirections ]

It says, "Note that for historical reasons, in the most common usage the curly braces that surround the body of the function must be separated from the body by blanks or newlines. This is because the braces are reserved words and are only recognized as such when they are separated from the command list by whitespace or another shell metacharacter. Also, when using the braces, the list must be terminated by a semicolon, a ‘&’, or a newline."
https://www.gnu.org/software/bash/manual/bash.html#Shell-Functions

So, for 'bash --posix', the syntax is invalid.  Dash simply errors out with "dash: 1: function: not found".  Whatever it does on a POSIX-y shell, it's a show-stopper.

I don't have "posh" available, but I've run the following using dash and bash.  It causes Emacs to hang until you hit C-g.  Maybe the "Policy-compliant Ordinary SHell" is different and the command works?  Since it fails for 'bash --posix' and dash, I doubt it.  Here's what I ran:

(org-babel-do-load-languages 'org-babel-load-languages '((shell . t)))

(defconst org-babel-shell-set-prompt-commands
  '((t . "function prompt { \"%s\" }")))

#+begin_src dash :session *dash*
  echo "hi"
#+end_src

Canceling and switching to the shell that was created, the prompt is "> " (the PS2 prompt).  This swallows all input that's not an error.  =org-babel-execute:shell= calls =org-babel-sh-initiate-session= which calls =org-babel-comint-wait-for-output= which waits for the =comint-prompt-regexp= which is "org_babel_sh_prompt> ", not "> ", so Emacs hangs.

There have been no complaints from "Policy-compliant Ordinary SHell" users after August 26, 2022 about "posh" being broken.  The talk of "posh" (and PowerShell, specifically) since August 26, 2022 is often as an aside.

I conclude that no one uses (or at least cares much about using) the "Policy-compliant Ordinary SHell."

** Both the "Policy-compliant Ordinary SHell" and PowerShell are usable without "posh"
Since the "Policy-compliant Ordinary SHell" is POSIX, it should be straight-forward to use with the "shell" language by replacing "shell-file-name."  The "shell" language header will hit the default prompt command which is general POSIX syntax (that is, it changes PS1 and PS2).  A shebang should also work.

I had access to a Windows 10 machine recently and verified the following makes PowerShell "work".  I use quotes because the following outputs a banner and often fails to remove the prompt, just like the current "posh" behavior.

(org-babel-do-load-languages 'org-babel-load-languages '((shell . t)))

;;; Setup for non-sessions
(setq shell-file-name "C:/Windows/System32/WindowsPowerShell/v1.0/powershell.exe")

#+begin_src shell :results output
echo "hello"
#+end_src

;;; Setup for sessions
;; required by `shell.el' in `shell' command
(setq explicit-shell-file-name
      "C:/Windows/System32/WindowsPowerShell/v1.0/powershell.exe")
(setq explicit-powershell.exe-args '("-NoExit"))

;; required by `ob-shell.el' in Org 9.6.6 (distributed with Emacs
;; 29.1 which is the latest build of Emacs for Windows)
(setq org-babel-prompt-command "function prompt { \"org_babel_sh_prompt> \" }")

#+begin_src shell :results output :session *powershell*
echo "world"
#+end_src

* My stance
The unfortunate reality is we "support" PowerShell currently.  We have code explicitly to handle PowerShell.  Changing that could, technically, break someone's workflow.

Here's how I reason it:

First, AFAIK no PowerShell user would refer to it as "posh".  At best they may say "pwsh" and most likely "powershell".

We could keep "posh" for the "Policy-compliant Ordinary SHell" and add languages for "pwsh" or "powershell".  This would need to be communicated PowerShell users.

I don't know what other keyword we could use for the "Policy-compliant Ordinary SHell".  We could keep "posh" for PowerShell, drop all "support" for the "Policy-compliant Ordinary SHell", and commit to PowerShell only.  This would need to be communicated to the "Policy-compliant Ordinary SHell" users.

In any case, we need to communicate our decision, in ORG-NEWS, through our mailing list communications, and definitely by updating the comment.

The cost of keeping PowerShell means maintaining it and whatever weirdo stuff comes with it.

I suggest we make the "t" in =org-babel-shell-set-prompt-commands= a catch-all for anything with PS1/PS2 (anything POSIX-y?) and remove the "posh" language header.  The "t" case should handle bash, dash, and (I suspect) the "Policy-compliant Ordinary SHell".  This has the potential to break someone using "posh" for PowerShell.  However, I see no indication of anyone seriously using it.  We've said for years we don't support it.  The only places I'm aware of saying we support PowerShell is the code comment and a few people on the mailing list.  Moreover, a "breakage" happens anyway, for either "Policy-compliant Ordinary SHell" users (as currently likely exists) or PowerShell users (if we give a new keyword).  If we're going to have to put a work-around or message out there anyway, I figure let's see about removing any semblance of supporting PowerShell until we talk it over.

Fortunately, when I wrote the Worg page, I put "the "Policy-compliant Ordinary SHell" for "posh."  So, the only place someone might think that PowerShell is supported is from the code comment or from the mailing list.

** PowerShell or cmd.exe?
I'm not claiming that people haven't wanted PowerShell support.  It's been discussed a few times on the list.  It always reduces to "it would be nice, however we lack a developer or the equipment, so sadly no."

As current ob-shell maintainer, here's how I see it:

I like the idea of supporting PowerShell.  I like the idea of supporting cmd.exe much, much more.

Both are associated with a non-free system.  Providing support (even partially) for non-free systems is good because it provides an opportunity to teach people about software freedom.

AFAIKT, both PowerShell and cmd are MIT licensed:
- https://github.com/microsoft/terminal
- https://github.com/PowerShell/PowerShell

The thought of compiling either for a GNU system is...ugh.  But maybe someone else has gotten them working?  Otherwise, it looks like Microsoft distributes a developer VM image of Windows.

All together, this means there's no *technical* barrier preventing us from running (and hence developing for) PowerShell or cmd.

It's important to note that Emacs defaults to cmdproxy.exe on Windows.  Using "shell" in Babel on Windows returns from cmd.exe (through cmdproxy.exe).  I believe it's best to support the defaults first.

Because it's the default, I've always used cmd when using Emacs on Windows.  I would personally much prefer support for cmd.exe over PowerShell.  That is, I'd be happy to work on PowerShell stuff after cmd stuff and both after cleaning up ob-shell, ob-comint, and ob-eval.

Thoughts overall?

--
Matt Trzcinski
Emacs Org maintainer (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

Re: ob-shell: proposal to remove "posh"

[Morgan Willcock]

Matt <matt@excalamus.com> writes:

> AFAIKT, both PowerShell and cmd are MIT licensed:
> - https://github.com/microsoft/terminal
> - https://github.com/PowerShell/PowerShell
>
> The thought of compiling either for a GNU system is...ugh.  But maybe someone else has gotten them working?  Otherwise, it looks like Microsoft distributes a developer VM image of Windows.
>
> All together, this means there's no *technical* barrier preventing us from running (and hence developing for) PowerShell or cmd.

As I understand it, the code for the cmd.exe shell is not released
anywhere.

What is in the "terminal" repository mentioned above is the code for the
console host process conhost.exe, which is the server process used to
host the shell process and present it (i.e. this is the terminal and not
the shell).

That said, I imagine Wine does contain its own implementation of
cmd.exe, although I wouldn't expect it to be 100% compatible with the
real thing.

-- 
Morgan Willcock

Re: ob-shell: proposal to remove "posh"

[Matt]

 ---- On Fri, 12 Jan 2024 11:04:33 +0100  Morgan Willcock  wrote --- 

 > As I understand it, the code for the cmd.exe shell is not released
 > anywhere.

Ah, okay.  I clearly didn't search too hard for it.

 > That said, I imagine Wine does contain its own implementation of
 > cmd.exe, although I wouldn't expect it to be 100% compatible with the
 > real thing.

That's an interesting idea.  Might be a "good enough" work around.  The VM image I mentioned says, "The VM will require a minimum of 8GB of RAM and at least 70GB of disk space". XD

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

Re: ob-shell: proposal to remove "posh"

[Matt]

 ---- On Thu, 11 Jan 2024 21:30:59 +0100  Matt  wrote --- 

 > The change on August 26, 2022 should have caused a breaking error for someone using the "Policy-compliant Ordinary SHell."  The prompt for "posh" in "org-babel-shell-set-prompt-commands" is valid PowerShell syntax (AFAIKT) and invalid bash/dash syntax:
 > 
 > function prompt { "org_babel_sh_prompt> " }
 > 
 > It's not clear to me what this would do in the "Policy-compliant Ordinary SHell." 

It would cause an error.

I was able to build the "Policy-compliant Ordinary SHell" on Guix using:

#+begin_src guile
(use-modules (guix packages)
             (guix download)
             (guix build-system gnu)
             (gnu packages autotools)
             (gnu packages perl)
             ((guix licenses) #:prefix license:))

(define-public posh
  (package
    (name "posh")
    (version "0.14.1")
    (source (origin
              (method url-fetch)
              (uri (string-append
                    "https://salsa.debian.org/clint/posh/-/archive/debian/"
                    version "/posh-debian-" version ".tar.gz"))
              (sha256
               (base32
                "070xnn996cjnc5yzp5819y36sgfikkrplhri4kx5r36h1fmp641d"))))
    (native-inputs (list autoconf automake perl))
    (build-system gnu-build-system)
    (home-page "https://salsa.debian.org/clint/posh")
    (synopsis "Policy-compliant Ordinary SHell")
    (description
     "Policy-compliant Ordinary SHell
posh is a stripped-down version of pdksh that aims for compliance with
Debian's policy, and few extra features.")
    (license (list license:gpl2+))))

posh
#+end_src

The result is:

ahab@pequod /gnu/store/64wiqdp9lqjgsz0jg1v1sq2b3afincrb-posh-0.14.1/bin$ ./posh
$ function prompt { "org_babel_sh_prompt> " }
./posh: function: not found

This is expected because "function" is not a keyword in the "Policy-compliant Ordinary SHell."  According to the man page:

#+begin_quote
name () command

  Defines the function name. See Functions below. Note that redirections
  specified after a function definition are performed whenever the
  function is executed, not when the function definition is executed.

-- https://manpages.debian.org/bookworm/posh/posh.1.en.html
#+end_quote

So, yes, indeed, commit a35d1636 introduced a breaking change for the "Policy-compliant Ordinary SHell."

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

Re: ob-shell: proposal to remove "posh"

[Ihor Radchenko]

Matt <matt@excalamus.com> writes:

> There was discussion about different shells, Eric asked for others, "posh" was mentioned as "specially for POSIX compatibility checks", and then a "posh" was added to ob-shell.el by Eric (fb09863f).
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=fb09863fbb35bf15bcf78262b6e31b8b8b8617e7
>
> Around that time are a few stack exchange answers suggesting to use posh, the "Policy-compliant Ordinary SHell", to test for posix compliance.  Debian distributed it by saying, "using posh as your /bin/sh may reveal breakage."  It seems that posh was used to check for POSIX compliance.  It's still available on Debian.
> https://unix.stackexchange.com/questions/48786/how-can-i-test-for-posix-compliance-of-shell-scripts
> https://web.archive.org/web/20130930231522/https://packages.debian.org/sid/posh
> https://packages.debian.org/sid/posh

It is also still available on Gentoo.

> ** No one uses the "Policy-compliant Ordinary SHell"
> The change on August 26, 2022 should have caused a breaking error for someone using the "Policy-compliant Ordinary SHell."  The prompt for "posh" in "org-babel-shell-set-prompt-commands" is valid PowerShell syntax (AFAIKT) and invalid bash/dash syntax:
>
> function prompt { "org_babel_sh_prompt> " }
>
> It's not clear to me what this would do in the "Policy-compliant Ordinary SHell."  In Bash, functions are defined using a form like:
>
> function fname [()] compound-command [ redirections ]

Simply because I struggled to find what is "posh" at the time I wrote that
code. I only found some obscure page (on the web!) saying the posh might
be PowerShell.
My commit a35d16368 did not intend to introduce anything new.

It turns out that we never supported PowerShell and never announced such
support. So we do not really need to support PowerShell.

> * My stance
> The unfortunate reality is we "support" PowerShell currently.  We have code explicitly to handle PowerShell.  Changing that could, technically, break someone's workflow.

We do not. It was just my mistake trying to figure out what is "posh".
Since posh is POSIX-compliant, generic PS1/PS2 variables should work, and we
do not need a special entry in `org-babel-shell-set-prompt-commands' -
we can remove "posh" record from there.

I'd rather keep the value in `org-babel-shell-names' though. It should
do no harm.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: ob-shell: proposal to remove "posh"

[Matt]

 ---- On Sat, 13 Jan 2024 19:00:16 +0100  Ihor Radchenko  wrote --- 
 
 > Since posh is POSIX-compliant, generic PS1/PS2 variables should work, and we
 > do not need a special entry in `org-babel-shell-set-prompt-commands' -
 > we can remove "posh" record from there.
 > 
 > I'd rather keep the value in `org-babel-shell-names' though. It should
 > do no harm.

Done https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

Re: ob-shell: proposal to remove "posh"

[Ihor Radchenko]

Matt <matt@excalamus.com> writes:

>  ---- On Sat, 13 Jan 2024 19:00:16 +0100  Ihor Radchenko  wrote --- 
>  
>  > Since posh is POSIX-compliant, generic PS1/PS2 variables should work, and we
>  > do not need a special entry in `org-babel-shell-set-prompt-commands' -
>  > we can remove "posh" record from there.
>
> Done https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/

Thanks!

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>