How to get shell source blocks to read my profile?

How to get shell source blocks to read my profile?

[George Mauer]

[-- Attachment #1: Type: text/plain, Size: 1248 bytes --]

I am confused why no matter how I try to run shell commands they seem to be
missing variables exported in profiles.

I have added 3 variables to various startup scripts

  - ~./bash-profile~ :: ~export GIM_BASH_PROFILE="yes"~
  - ~./bashrc~ :: ~export GIM_BASHRC="yes"~
  - ~./zshrc~ :: ~export GIM_ZSHRC="yes"~

I am running emacs in GUI mode so I get why none of these are available
*directly* in emacs, so then I try

  #+begin_src shell :results list
    env | grep GIM
  #+end_src

  #+RESULTS:

Ok that's kinda surprising, but I suppose it could be running ~/bin/zsh~
(that's my ~shell-file-name~) directly

what about

  #+begin_src shell :results list
    bash -c env | grep GIM
  #+end_src

  #+RESULTS:

That's pretty surprising. I would have expected running it directly to
actually run my profile.

Shockingly

  #+begin_src shell :results list
    bash --rcfile ~/.bashrc -c env | grep GIM
  #+end_src

  #+RESULTS:

That is *still* nothing!

Sanity is restored slightly when I run

  #+begin_src shell :results list
    bash --login -c env | grep GIM
  #+end_src

which *does* indeed visit ~.bash_profile~ but only slightly.

What is going on, and is there a straightforward way in which I can get
shell block to read from a profile?

[-- Attachment #2: Type: text/html, Size: 1621 bytes --]

Re: How to get shell source blocks to read my profile?

[Tim Cross]

Is this perhaps on a Mac where Emacs is started from the dock? If so, it
could be because on the mac, applications started from the dock are not
executed inside a login shell. Do you get different results if you start
emacs from within a terminal? If so, this is almost certainly the source
of your issue/difference.

I am also a little confused by your examples. Some of them are
attempting to source .bash_profile, others .bashrc and others .zshrc.
Which shell are you using and which files do you have? I get the feeling
that in your frustration, you have taken a 'shotgun' approach and added
the variable everywhere. It might be time to go back and look at all
your shell environment files and rationalise them a bit. Things can
become a little confusing when running zsh as so many things
assume/expect a bash shell. While zsh is pretty compatible with bash, so
few problems arise, it can lead to confusion regarding where to set
variables and understand what is being sourced and when.

For some reason, there seems to be some confusion these days about the
difference between .profile, .bash_profile, .bashrc, .zsh_profile and
.zshrc. In particular, I frequently see incorrect/wrong advice regarding
the use of 'profile' and 'rc' files. Most of it can
be blamed on wrong advice in forums like stack overflow!

The 'profile' and the 'rc' files fulfil different purposes and are
sourced at different times. Have a careful read of the bash man page
(even if your running zsh as they all tend to follow the same pattern).
I would also experiment with just dumping out the output of 'env' and
not run it through grep so that you can see what is being defined. This
can help you understand what is being sourced. I also think it is a good
idea to be explicit about the shell so that you have more
control/understanding over what is being run e.g. put the line

#!/bin/bash

or

#!/bin/zsh

or

#!/bin/sh

as the first line.

In very broad terms (glossing over some of the subtleties, which is
likely the cause of some of your issues), the 'profile' files are only
sourced for login shells and the 'rc' files are sourced whenever a new
shell process is run. Note that these processes run in a hierarchy i.e.
shell processes have a parent process. Some things have to be set in
your 'profile' (or are better set there) while other things must be set
in your 'rc' file. For example, setting PATH in your 'profile' is normal
because you typically just want that set once and then 'exported' to all
child processes. Setting a dynamic prompt which changes depending on the
directory your in (or because it has some other dynamic data, like the
time) need to be set in your 'rc' file because these are sourced before
each shell process is executed. I the 'old days' it was important to get
these distinctions correct because sourcing the files could have a
performance impact - not as big an issue these days.

These days, things seem to have become even more confusing in an attempt
to make things easier. I often see people with a .profile, a
.bash_profile, a .bashrc, a .zsh_profile and a .zshrc with variables set
and exported mixed in across all these files. This is a source of
terrible confusion and unexpected results. Some 'canned' shell
configurations, like 'oh-my-zsh' also try to be extra helpful by
sourcing files which are not normally sourced by that shell (to make
smoother migration from bash shells for example). Sometimes, it can be
helpful to put a line like the following into each of these files

echo "Executing <filename> at `date`" >> ~/profile.log

just to see what is getting executed when (you won't want to leave this
there - just for diagnostic and experimentation to help understand when
things are being sourced. You could even dump out the output of 'env' so
that you can see how the environment is being changed as each of these
files executes.


George Mauer <gmauer@gmail.com> writes:

> I am confused why no matter how I try to run shell commands they seem to be missing variables exported in profiles.
>
> I have added 3 variables to various startup scripts
>
>   - ~./bash-profile~ :: ~export GIM_BASH_PROFILE="yes"~
>   - ~./bashrc~ :: ~export GIM_BASHRC="yes"~
>   - ~./zshrc~ :: ~export GIM_ZSHRC="yes"~
>
> I am running emacs in GUI mode so I get why none of these are available *directly* in emacs, so then I try
>
>   #+begin_src shell :results list
>     env | grep GIM
>   #+end_src
>
>   #+RESULTS:
>
> Ok that's kinda surprising, but I suppose it could be running ~/bin/zsh~ (that's my ~shell-file-name~) directly
>
> what about
>
>   #+begin_src shell :results list
>     bash -c env | grep GIM
>   #+end_src
>
>   #+RESULTS:
>
> That's pretty surprising. I would have expected running it directly to actually run my profile.
>
> Shockingly
>
>   #+begin_src shell :results list
>     bash --rcfile ~/.bashrc -c env | grep GIM
>   #+end_src
>
>   #+RESULTS:
>
> That is *still* nothing!
>
> Sanity is restored slightly when I run
>
>   #+begin_src shell :results list
>     bash --login -c env | grep GIM
>   #+end_src
>
> which *does* indeed visit ~.bash_profile~ but only slightly.
>
> What is going on, and is there a straightforward way in which I can get shell block to read from a profile?


--
Tim Cross

Re: How to get shell source blocks to read my profile?

[George Mauer]

[-- Attachment #1: Type: text/plain, Size: 7284 bytes --]

Hey Tim, thanks for helping out. I commented inline to your response below
but I'll sum up and ask the outstanding questions more directly here as
well.

I think you might have misread what I was doing - I had 3 different
variables set in 3 different places precisely because I want to dissect
things and figure out which profile/rc scripts were run.

Yes, this is emacs running in GUI mode so yes, it doesn't have a shell as a
parent. But that only explains why these environment variables aren't in
emacs itself, not the spawned bash or zsh processes

So to sum up the specific questions. My ~shell-file-name~ is ~/bin/zsh~
which seems to be what ~ob-shell~ is passing down to be run. So why on
earth does my ~GIM_ZSHRC~ variable not show up here?

  #+begin_src shell
    env
  #+end_src

When I actually directly call bash

 #+begin_src shell
    bash -c env
  #+end_src

Why would the output not include ~GIM_BASHRC~ - it should have been run,
right?

What about when I call this? Even with explicitly selecting the rc file to
run, it seems to not

  #+begin_src shell
    bash --rcfile ~/.bashrc -c env
  #+end_src

Finally, the outstanding question about ~ob-shell~ is if there is any way
to force it to run the shell processes' rc-script? I really would have
expected it to be run in the above situations already...

-----------

>
> Is this perhaps on a Mac where Emacs is started from the dock?


Yup like I said, its in GUI mode so I understand why those environment
variables aren't within emads itself


> If so, it
> could be because on the mac, applications started from the dock are not
> executed inside a login shell. Do you get different results if you start
> emacs from within a terminal?


Yup I do, but thats why I move on to the test where I run `bash` directly


> I am also a little confused by your examples. Some of them are
> attempting to source .bash_profile, others .bashrc and others .zshrc.
>

I created the three environment variables I indicated and placed one in
each file so that I could test which profile file ran


> Which shell are you using and which files do you have? I get the feeling
> that in your frustration, you have taken a 'shotgun' approach and added
> the variable everywhere.


That's exactly what I'm trying to avoid here. I *did* add variables
everywhere, but I added *different* variables everywhere so that I can see
exactly which profile ran.


> For some reason, there seems to be some confusion these days about the
> difference between .profile, .bash_profile, .bashrc, .zsh_profile and
> .zshrc. In particular, I frequently see incorrect/wrong advice regarding
> the use of 'profile' and 'rc' files. Most of it can
> be blamed on wrong advice in forums like stack overflow!
>

It is certainly confusing but I think I got a handle on it mostly. If
it's a login shell you'll run a profile, if it's not you'll run the default
rc file unless one of the options were specified. I think each is named
specific to each shell name but oftentimes you chain them together in
practice.


> I would also experiment with just dumping out the output of 'env' and
> not run it through grep so that you can see what is being defined.


I did, that just isn't conducive to posting on an email list ;)

iI also think it is a good dea to be explicit about the shell so that you
> have more
> control/understanding over what is being run e.g. put the line
>

Yeah, I suppose I could do a prologue with a hashbang, but I debugged into
the source and it's just ~shell-file-name~ which in my case is ~/bin/zsh~

In very broad terms (glossing over some of the subtleties, which is
> likely the cause of some of your issues), the 'profile' files are only
> sourced for login shells and the 'rc' files are sourced whenever a new
> shell process is run. Note that these processes run in a hierarchy i.e.
> shell processes have a parent process. Some things have to be set in
> your 'profile' (or are better set there) while other things must be set
> in your 'rc' file. For example, setting PATH in your 'profile' is normal
> because you typically just want that set once and then 'exported' to all
> child processes. Setting a dynamic prompt which changes depending on the
> directory your in (or because it has some other dynamic data, like the
> time) need to be set in your 'rc' file because these are sourced before
> each shell process is executed. I the 'old days' it was important to get
> these distinctions correct because sourcing the files could have a
> performance impact - not as big an issue these days.
>

Great history, thanks, that's mostly what I was thinking but gives me
useful background

These days, things seem to have become even more confusing in an attempt
> to make things easier. I often see people with a .profile, a
> .bash_profile, a .bashrc, a .zsh_profile and a .zshrc with variables set
> and exported mixed in across all these files.


Don't forget that many tools from homebrew to pyenv install scripts to
ansible will start modifying these for you


> This is a source of
> terrible confusion and unexpected results. Some 'canned' shell
> configurations, like 'oh-my-zsh' also try to be extra helpful by
> sourcing files which are not normally sourced by that shell (to make
> smoother migration from bash shells for example). Sometimes, it can be
> helpful to put a line like the following into each of these files
>
> echo "Executing <filename> at `date`" >> ~/profile.log
>
> just to see what is getting executed when (you won't want to leave this
> there - just for diagnostic and experimentation to help understand when
> things are being sourced. You could even dump out the output of 'env' so
> that you can see how the environment is being changed as each of these
> files executes.
>
>
Yup, thanks.


> George Mauer <gmauer@gmail.com> writes:
>
> > I am confused why no matter how I try to run shell commands they seem to
> be missing variables exported in profiles.
> >
> > I have added 3 variables to various startup scripts
> >
> >   - ~./bash-profile~ :: ~export GIM_BASH_PROFILE="yes"~
> >   - ~./bashrc~ :: ~export GIM_BASHRC="yes"~
> >   - ~./zshrc~ :: ~export GIM_ZSHRC="yes"~
> >
> > I am running emacs in GUI mode so I get why none of these are available
> *directly* in emacs, so then I try
> >
> >   #+begin_src shell :results list
> >     env | grep GIM
> >   #+end_src
> >
> >   #+RESULTS:
> >
> > Ok that's kinda surprising, but I suppose it could be running ~/bin/zsh~
> (that's my ~shell-file-name~) directly
> >
> > what about
> >
> >   #+begin_src shell :results list
> >     bash -c env | grep GIM
> >   #+end_src
> >
> >   #+RESULTS:
> >
> > That's pretty surprising. I would have expected running it directly to
> actually run my profile.
> >
> > Shockingly
> >
> >   #+begin_src shell :results list
> >     bash --rcfile ~/.bashrc -c env | grep GIM
> >   #+end_src
> >
> >   #+RESULTS:
> >
> > That is *still* nothing!
> >
> > Sanity is restored slightly when I run
> >
> >   #+begin_src shell :results list
> >     bash --login -c env | grep GIM
> >   #+end_src
> >
> > which *does* indeed visit ~.bash_profile~ but only slightly.
> >
> > What is going on, and is there a straightforward way in which I can get
> shell block to read from a profile?
>
>
> --
> Tim Cross
>
>

[-- Attachment #2: Type: text/html, Size: 10025 bytes --]

Re: How to get shell source blocks to read my profile?

[Tim Cross]

comments in-line ...

George Mauer <gmauer@gmail.com> writes:

> Hey Tim, thanks for helping out. I commented inline to your response below but I'll sum up and ask the outstanding questions more directly here as well.
>
> I think you might have misread what I was doing - I had 3 different variables set in 3 different places precisely because I want to dissect things and figure
> out which profile/rc scripts were run.
>
> Yes, this is emacs running in GUI mode so yes, it doesn't have a shell as a parent. But that only explains why these environment variables aren't in
> emacs itself, not the spawned bash or zsh processes
>

To clarify, the GUI mode is unrelated. You can run in GUI mode from a
terminal and the emacs process will inherit the environment in the
terminal. If that terminal is a login terminal (i.e. terminal shell was
run as a login shell), the 'profile' file will be sourced and the 'rc'
file will have been sourced. If not a login shell, only the rc file will
have been sourced.

If on the other hand, Emacs is started from the dock, it is started in a
completely different process chain and is not a child of a login
process. This is why, if you want your emacs to have /usr/local/bin in
the path, you have to add it to /etc/paths or /etc/path.d whereas if you
start emacs from within a terminal (GUI or text mode), you only need to
add /usr/local/bin to your PATH setting in .profile (just an example to
highlight the difference between running from in a terminal and running
from the dock).

> So to sum up the specific questions. My ~shell-file-name~ is ~/bin/zsh~ which seems to be what ~ob-shell~ is passing down to be run. So why on earth
> does my ~GIM_ZSHRC~ variable not show up here?
>
>   #+begin_src shell
>     env
>   #+end_src
>
> When I actually directly call bash
>
>  #+begin_src shell
>     bash -c env
>   #+end_src
>

Have a closer look on the section in the bash manual on the difference
between interactive and non-interactive shells. (also holds for zsh).
Basically, the 'rc' files are not sourced for non-interactive shells.

> Why would the output not include ~GIM_BASHRC~ - it should have been run, right?
>

No. Adding the -c means it is a non-interactive shell, so no .bashrc
sourcing.


> What about when I call this? Even with explicitly selecting the rc file to run, it seems to not
>
>   #+begin_src shell
>     bash --rcfile ~/.bashrc -c env
>   #+end_src
>

Same issue here. --rcfile only has effect in interactive shells.

> Finally, the outstanding question about ~ob-shell~ is if there is any way to force it to run the shell processes' rc-script? I really would have expected it to
> be run in the above situations already...
>

You could try sourcing it e.g.

#+begin_src shell
source ~/.bashrc
env
#+end_src

or use 'shorthand' dot

#+begin_src shell
. ~/.bashrc
env
#+end_src

there are also some options you can add at the 'shebang' line i.e.

#!/bin/bash -l

or

#!/bin/bash -i

which will change the behaviour.

There is a lot of 'meat' in the bash man page and there is a lot of
additional information in the bash info pages. However, both can be a
bit terse and because the info is very 'dense', it is very easy to miss
key points.

In order to have environment variables available inside your org source
blocks, you really need to

- have them in the environment inherited by emacs when it starts. This
  will depend how you start Emacs (i.e from within an interactive shell
  vs from the dock). Note that you typically don't see this issue under
  Linux because in most Linux setups, the X environment is started
  inside a login shell. So everything started as part of the X session,
  like a dock, is a child of the login process and therefore inherits
  the login environment. On a mac, the dock is not part of the login
  shell, so it only inherits what is in the system wide bash profile
  file.

- Ensure your source block run as interactive and/or login shells using
  shebang options like -i or -l

- explicitly source the .profile or .bashrc file using a source call

- pass the environment variable in on the command line e.g.

#+begin_src shell
FRED=barney env
#+end_src

will result in

FRED=barney

being in the output from 'env'.


> -----------
>
>  Is this perhaps on a Mac where Emacs is started from the dock?
>
> Yup like I said, its in GUI mode so I understand why those environment variables aren't within emads itself
>

It is actually a little more subtle. It isn't because your running in
GUI mode those variables are not there. It is because your running from
the dock. Run it in GUI mode from within a login terminal and all those
variables will be 'in' emacs.

<snip>

> It is certainly confusing but I think I got a handle on it mostly. If it's a login shell you'll run a profile, if it's not you'll run the default rc file unless one of the
> options were specified. I think each is named specific to each shell name but oftentimes you chain them together in practice.
>

There is also a difference between interactive and non-interactive
shells. I suspect this is the root cause of your issue. The source block
being run are non-interactive.

--
Tim Cross

Re: How to get shell source blocks to read my profile?

[George Mauer]

[-- Attachment #1: Type: text/plain, Size: 6041 bytes --]

Thanks a lot! The interactive/non-interactive was indeed the core issue.
Extra frustrating because it seems like supplying `--rcfile` does nothing
if you *do* use `-c` but *don't* use `-i`...ah ad-hoc cli design.

As a general solution, I've found that adding this block in my org buffer
will make shell (which is zsh in my case) run in interactive mode for all
of these

* Local Variables :noexport:
  :PROPERTIES:
  :VISIBILITY: folded
  :END:
  local variables:
  shell-file-name: "/bin/zsh -i"
  end:

On Sun, Mar 14, 2021 at 11:04 PM Tim Cross <theophilusx@gmail.com> wrote:

>
> comments in-line ...
>
> George Mauer <gmauer@gmail.com> writes:
>
> > Hey Tim, thanks for helping out. I commented inline to your response
> below but I'll sum up and ask the outstanding questions more directly here
> as well.
> >
> > I think you might have misread what I was doing - I had 3 different
> variables set in 3 different places precisely because I want to dissect
> things and figure
> > out which profile/rc scripts were run.
> >
> > Yes, this is emacs running in GUI mode so yes, it doesn't have a shell
> as a parent. But that only explains why these environment variables aren't
> in
> > emacs itself, not the spawned bash or zsh processes
> >
>
> To clarify, the GUI mode is unrelated. You can run in GUI mode from a
> terminal and the emacs process will inherit the environment in the
> terminal. If that terminal is a login terminal (i.e. terminal shell was
> run as a login shell), the 'profile' file will be sourced and the 'rc'
> file will have been sourced. If not a login shell, only the rc file will
> have been sourced.
>
> If on the other hand, Emacs is started from the dock, it is started in a
> completely different process chain and is not a child of a login
> process. This is why, if you want your emacs to have /usr/local/bin in
> the path, you have to add it to /etc/paths or /etc/path.d whereas if you
> start emacs from within a terminal (GUI or text mode), you only need to
> add /usr/local/bin to your PATH setting in .profile (just an example to
> highlight the difference between running from in a terminal and running
> from the dock).
>
> > So to sum up the specific questions. My ~shell-file-name~ is ~/bin/zsh~
> which seems to be what ~ob-shell~ is passing down to be run. So why on earth
> > does my ~GIM_ZSHRC~ variable not show up here?
> >
> >   #+begin_src shell
> >     env
> >   #+end_src
> >
> > When I actually directly call bash
> >
> >  #+begin_src shell
> >     bash -c env
> >   #+end_src
> >
>
> Have a closer look on the section in the bash manual on the difference
> between interactive and non-interactive shells. (also holds for zsh).
> Basically, the 'rc' files are not sourced for non-interactive shells.
>
> > Why would the output not include ~GIM_BASHRC~ - it should have been run,
> right?
> >
>
> No. Adding the -c means it is a non-interactive shell, so no .bashrc
> sourcing.
>
>
> > What about when I call this? Even with explicitly selecting the rc file
> to run, it seems to not
> >
> >   #+begin_src shell
> >     bash --rcfile ~/.bashrc -c env
> >   #+end_src
> >
>
> Same issue here. --rcfile only has effect in interactive shells.
>
> > Finally, the outstanding question about ~ob-shell~ is if there is any
> way to force it to run the shell processes' rc-script? I really would have
> expected it to
> > be run in the above situations already...
> >
>
> You could try sourcing it e.g.
>
> #+begin_src shell
> source ~/.bashrc
> env
> #+end_src
>
> or use 'shorthand' dot
>
> #+begin_src shell
> . ~/.bashrc
> env
> #+end_src
>
> there are also some options you can add at the 'shebang' line i.e.
>
> #!/bin/bash -l
>
> or
>
> #!/bin/bash -i
>
> which will change the behaviour.
>
> There is a lot of 'meat' in the bash man page and there is a lot of
> additional information in the bash info pages. However, both can be a
> bit terse and because the info is very 'dense', it is very easy to miss
> key points.
>
> In order to have environment variables available inside your org source
> blocks, you really need to
>
> - have them in the environment inherited by emacs when it starts. This
>   will depend how you start Emacs (i.e from within an interactive shell
>   vs from the dock). Note that you typically don't see this issue under
>   Linux because in most Linux setups, the X environment is started
>   inside a login shell. So everything started as part of the X session,
>   like a dock, is a child of the login process and therefore inherits
>   the login environment. On a mac, the dock is not part of the login
>   shell, so it only inherits what is in the system wide bash profile
>   file.
>
> - Ensure your source block run as interactive and/or login shells using
>   shebang options like -i or -l
>
> - explicitly source the .profile or .bashrc file using a source call
>
> - pass the environment variable in on the command line e.g.
>
> #+begin_src shell
> FRED=barney env
> #+end_src
>
> will result in
>
> FRED=barney
>
> being in the output from 'env'.
>
>
> > -----------
> >
> >  Is this perhaps on a Mac where Emacs is started from the dock?
> >
> > Yup like I said, its in GUI mode so I understand why those environment
> variables aren't within emads itself
> >
>
> It is actually a little more subtle. It isn't because your running in
> GUI mode those variables are not there. It is because your running from
> the dock. Run it in GUI mode from within a login terminal and all those
> variables will be 'in' emacs.
>
> <snip>
>
> > It is certainly confusing but I think I got a handle on it mostly. If
> it's a login shell you'll run a profile, if it's not you'll run the default
> rc file unless one of the
> > options were specified. I think each is named specific to each shell
> name but oftentimes you chain them together in practice.
> >
>
> There is also a difference between interactive and non-interactive
> shells. I suspect this is the root cause of your issue. The source block
> being run are non-interactive.
>
> --
> Tim Cross
>
>

[-- Attachment #2: Type: text/html, Size: 7218 bytes --]

Re: How to get shell source blocks to read my profile?

[Tim Cross]

George Mauer <gmauer@gmail.com> writes:

> Thanks a lot! The interactive/non-interactive was indeed the core issue. Extra frustrating because it seems like supplying `--rcfile` does nothing if you
> *do* use `-c` but *don't* use `-i`...ah ad-hoc cli design.
>

It certainly can seem rather ad hoc. However, it actually makes sense on
some levels. If you use -c your telling the shell to execute a command
and then exit. By definition, this is non-interactive. This is covered
in the manual.

Where it becomes confusing is when your mixing in different options as
some override others. So, provided you include -i with -c, it will be
forced into interactive i.e. -i overrides the non-interactive status
added with -c. If you add -s, telling the shell to read input from
stdin, you also override non-interactive status.

The other possible solution to your situation is to ensure
Emacs runs inside an environment which has all your exported variables
i.e. inside your login shell environment. There are a few ways to do
this, but probably the easiest is to create a script which opens a login
shell, then calls Emacs (may need to use open - not sure) and Emacs will
inherit your environment. Advantage is that processes you then spawn
from within Emacs will also inherit that environment. You then add this
script as the executable in the dock rather than calling Emacs directly.

One thing to watch out for is that if your also using oh-my-zsh, it
setups up some aliases with the name emacs which actually call
emacsclient. This can be confusing as it means running just 'emacs' in
the shell will run the alias and not actually run Emacs directly. Things
can be even more confusing as there are also multiple ways to install
Emacs on the mac and they are all slightly different with respect to how
they setup things.

What I find hardest with writing shell stuff is that I simply don't seem
to do it much anymore. My brain cache is just too small and when I find
it necessary to write a shell script again, all that knowledge has been
flushed! Once upon a time, many moons ago, I could write shell scripts
that used sed, awk, cut, uniq etc without even needing to look at the
man pages. These days, I have to check the bash man page just to
remember what the expr operators are!

--
Tim Cross

Re: How to get shell source blocks to read my profile?

[Steven Harris]

[-- Attachment #1: Type: text/plain, Size: 2702 bytes --]

Well on my mac Emacs is launched by Ruby.  I remember having to allow ruby
to do that two years ago when I set this up.

But thanks for sharing your expertise.  I've been a unixy admin for 20
years and didn't know a lot of that.

Cheers

Steven.

On Tue, 16 Mar 2021 at 07:34, Tim Cross <theophilusx@gmail.com> wrote:

>
> George Mauer <gmauer@gmail.com> writes:
>
> > Thanks a lot! The interactive/non-interactive was indeed the core issue.
> Extra frustrating because it seems like supplying `--rcfile` does nothing
> if you
> > *do* use `-c` but *don't* use `-i`...ah ad-hoc cli design.
> >
>
> It certainly can seem rather ad hoc. However, it actually makes sense on
> some levels. If you use -c your telling the shell to execute a command
> and then exit. By definition, this is non-interactive. This is covered
> in the manual.
>
> Where it becomes confusing is when your mixing in different options as
> some override others. So, provided you include -i with -c, it will be
> forced into interactive i.e. -i overrides the non-interactive status
> added with -c. If you add -s, telling the shell to read input from
> stdin, you also override non-interactive status.
>
> The other possible solution to your situation is to ensure
> Emacs runs inside an environment which has all your exported variables
> i.e. inside your login shell environment. There are a few ways to do
> this, but probably the easiest is to create a script which opens a login
> shell, then calls Emacs (may need to use open - not sure) and Emacs will
> inherit your environment. Advantage is that processes you then spawn
> from within Emacs will also inherit that environment. You then add this
> script as the executable in the dock rather than calling Emacs directly.
>
> One thing to watch out for is that if your also using oh-my-zsh, it
> setups up some aliases with the name emacs which actually call
> emacsclient. This can be confusing as it means running just 'emacs' in
> the shell will run the alias and not actually run Emacs directly. Things
> can be even more confusing as there are also multiple ways to install
> Emacs on the mac and they are all slightly different with respect to how
> they setup things.
>
> What I find hardest with writing shell stuff is that I simply don't seem
> to do it much anymore. My brain cache is just too small and when I find
> it necessary to write a shell script again, all that knowledge has been
> flushed! Once upon a time, many moons ago, I could write shell scripts
> that used sed, awk, cut, uniq etc without even needing to look at the
> man pages. These days, I have to check the bash man page just to
> remember what the expr operators are!
>
> --
> Tim Cross
>
>

[-- Attachment #2: Type: text/html, Size: 3306 bytes --]

Re: How to get shell source blocks to read my profile?

[Maxim Nikulin]

On 16/03/2021 00:49, George Mauer wrote:
>    shell-file-name: "/bin/zsh -i"

I am afraid, you should be prepared to face some problem accidentally. 
The value of this variable is used to execute the specified file
("zsh -i" in the "/bin" directory, I do not think, you have such file), 
not as a part of shell command. shell-file-name in namely file name to 
be executed with shell-command-switch as first argument (separate 
arguments, not merged into a string) to run shell commands (next argument).

Environment variables could be set on OS level, inside emacs 
https://www.gnu.org/software/emacs/manual/html_node/emacs/Environment.html 
, etc. It is hard to suggest something since you have not described the 
problem you are trying to solve. Tim in details explained why you 
attempt to solve it did not work, but the problem (or the goal) is still 
unknown.

Re: How to get shell source blocks to read my profile?

[George Mauer]

[-- Attachment #1: Type: text/plain, Size: 1907 bytes --]

I understand why you say that maxim but testing that this worked was the
first thing I did and it does work. I was surprised as well. I haven't
debugged all the way into the eval functions to see why this works but my
guess is that the authors were pretty smart about figuring out which
executable to invoke.

As for the problem I'm trying to solve, there isn't one exactly. This is
more me hacking on something I don't fully understand because it regularly
trips me up.

I do still wonder what would be the disadvantage of just configuring it to
do --login by default and doing all configuration in profile scripts. It
would be unconventional yes, but it would also make dynamic scoping of
environment variable effectively opt-in via --noprofile rather than opt-out
(which imo is how it should be). I would assume that uses extra resources
or risks improperly handling crashed processes, but I can't find anything
to that effect in the docs

On Tue, Mar 16, 2021, 07:32 Maxim Nikulin <manikulin@gmail.com> wrote:

> On 16/03/2021 00:49, George Mauer wrote:
> >    shell-file-name: "/bin/zsh -i"
>
> I am afraid, you should be prepared to face some problem accidentally.
> The value of this variable is used to execute the specified file
> ("zsh -i" in the "/bin" directory, I do not think, you have such file),
> not as a part of shell command. shell-file-name in namely file name to
> be executed with shell-command-switch as first argument (separate
> arguments, not merged into a string) to run shell commands (next argument).
>
> Environment variables could be set on OS level, inside emacs
> https://www.gnu.org/software/emacs/manual/html_node/emacs/Environment.html
> , etc. It is hard to suggest something since you have not described the
> problem you are trying to solve. Tim in details explained why you
> attempt to solve it did not work, but the problem (or the goal) is still
> unknown.
>
>
>

[-- Attachment #2: Type: text/html, Size: 2513 bytes --]

Re: How to get shell source blocks to read my profile?

[Nick Dokos]

George Mauer <gmauer@gmail.com> writes:

> I understand why you say that maxim but testing that this worked was the first thing I did
> and it does work. I was surprised as well. I haven't debugged all the way into the eval
> functions to see why this works but my guess is that the authors were pretty smart about
> figuring out which executable to invoke.
>

Starting with `emacs -q' I evaluate

    (setq shell-file-name "bash -i")

in the *scratch* buffer and then do `M-x shell'. I get

    apply: Searching for program: No such file or directory, bash -i


If I do

    (setq shell-file-name "/bin/bash -i")

I get

    emacs: /bin/bash -i: No such file or directory

    Process shell exited abnormally with code 127

The trouble is that `shell-file-name' may be used much more widely than you expect.

> As for the problem I'm trying to solve, there isn't one exactly. This is more me hacking on
> something I don't fully understand because it regularly trips me up.
>
> I do still wonder what would be the disadvantage of just configuring it to do --login by
> default and doing all configuration in profile scripts. It would be unconventional yes, but
> it would also make dynamic scoping of environment variable effectively opt-in via
> --noprofile rather than opt-out (which imo is how it should be). I would assume that uses
> extra resources or risks improperly handling crashed processes, but I can't find anything to
> that effect in the docs
>
> On Tue, Mar 16, 2021, 07:32 Maxim Nikulin <manikulin@gmail.com> wrote:
>
>     On 16/03/2021 00:49, George Mauer wrote:
>     >    shell-file-name: "/bin/zsh -i"
>    
>     I am afraid, you should be prepared to face some problem accidentally.
>     The value of this variable is used to execute the specified file
>     ("zsh -i" in the "/bin" directory, I do not think, you have such file),
>     not as a part of shell command. shell-file-name in namely file name to
>     be executed with shell-command-switch as first argument (separate
>     arguments, not merged into a string) to run shell commands (next argument).
>    
>     Environment variables could be set on OS level, inside emacs
>     https://www.gnu.org/software/emacs/manual/html_node/emacs/Environment.html
>     , etc. It is hard to suggest something since you have not described the
>     problem you are trying to solve. Tim in details explained why you
>     attempt to solve it did not work, but the problem (or the goal) is still
>     unknown.
>

-- 
Nick

"There are only two hard problems in computer science: cache
invalidation, naming things, and off-by-one errors." -Martin Fowler

Re: How to get shell source blocks to read my profile?

[Tim Cross]

George Mauer <gmauer@gmail.com> writes:

> I do still wonder what would be the disadvantage of just configuring it to do --login by default and doing all configuration in profile scripts. It would be
> unconventional yes, but it would also make dynamic scoping of environment variable effectively opt-in via --noprofile rather than opt-out (which imo is
> how it should be). I would assume that uses extra resources or risks improperly handling crashed processes, but I can't find anything to that effect in the
> docs

The following is long, but may provide some of the context you are
looking for to help understand all the moving parts involved here. Some
of ti is not 100% accurate, but trying to be 100% accurate would make it
longer and would not necessarily help with the mental model needed to
understand how this all fits together. Hopefully, this has a reasonable
balance between accuracy and simplicity!

While there is certainly aspects of shell architecture which are less
relevant in a modern computing environment, the way it all works is
actually pretty good. It can sometimes help to remember what early Unix
environments were like and what the resource constraints were to
understand some of the design decisions. A time when memory and disk
storage was extremely expensive and in short supply, where CPUs were
slower and less capable than those commonly found in a modern washing
machine. This was a time where every CPU cycle, every memory bit and
every byte of storage had to be justified and used efficiently.

The 'login' shell actually has a special purpose and a few subtleties
associated with it that may not be obvious at first. For example, the
login shell usually records a user's login in a few system files like
wtmp and utmp. The idea is that the login shell will be the 'root' of a
user's process tree. One of its key roles is to setup the environment
which will be inherited by all the other (child) processes which the
user initiates. This includes some things which you only want run once
and processes which must run as parents to all other processes. A good
example is setting up an ssh agent and adding ssh keys to that agent.
You only want this to occur once, so you might add this to your .profile
(or whatever the shell equivalent is, such as .bash_profile or
.zsh_profile). More critical is the requirement that the ssh agent must
be run inside a parent process for child processes to be able to use it.
This means it needs to be near the 'root' login process to ensure all
children are able to take advantage of the service it provides.
(Keep in mind that some platforms, like the mac, have created a whole
new architecture for managing keys etc and even on Linux, it is rarely
necessary to run ssh-agent like we did in the 'old days' as that
functionality has been subsumed into desktop environments like gnome)

It is also here that you will define static environment variables i.e.
those which will not change during the lifetime of the session and which
are exported, meaning they are made available in the environments of
child processes. Examples are setting up things like PATH, EMAIL etc.
Some common environment variables, like USER, HOME etc are done
automatically as part of the system wide profile in /etc/profile or the
system wide path defined in /etc/paths and /etc/paths.d etc.

In addition to these static environment variables are more dynamic ones
which change and therefore need to be updated with each new shell
process. This can include variables like PWD and CWD which track the
parent and current working directory or the PS1 and PS2 variables which
contain details about your shell prompt that may include dynamic
information such as the current working directory, current date/time or
status of the git repository associated with the current working
directory. This is essentially the role fulfilled by the 'rc' files,
like .bashrc. As these dynamic enironment variables tend to only be
relevant for interactive shells, the rc file is only 'sourced' for those
shells.

When you execute a command, it creates a new process which has an
execution environment associated with it (there are some exceptions,
such as built-in shell commands that run inside the current process and
command 'modifiers' like 'exec' which replace the current process with
the command being executed). Some of these processes will be very short
lived, executing a single command or possibly run for hours, days, weeks
(such as Emacs). This process is a child process of the process which
ran/executed the command. You can use the -f (forest) switch to ps to
see the relationship between processes. Those processes in turn may
execute more child processes (such as when Emacs creates a shell process
to perform some task, like run a compilation task, spawn a shell or
terminal, etc. The environment each of these processes inherits consists
of the exported variables defined in the parent process. This is
typically all the exported variables from the login process (as it is
the parent of all) and any variables exported by parent processes in the
process hierarchy (such as any exported as part of an interactive shell
which sourced .bashrc for example) or is defined and exported in a
parent (such as part of a shell script which then executes the command
which creates the child process.

So, in general, you don't want to run all or many of your processes as a
login shell. At the very least it is wasting resources and at the worst
it could actually cause confusion or break things because it is causing
things to be run multiple times which should only be run once.

I suspect some of the confusion here is being cause because of the
different model approach used by the Mac OS. On the mac, you actually
have at least two process trees (I'm not a mac expert, so it is possible
there are more than two - in fact I'm pretty sure there are, but for
explanation purposes....). On the mac you have your login process tree
and your 'dock' (launcher) process tree. The dock process tree is not
rooted as the child of a login shell, so there is no sourcing of
.profile and it is not an interactive shell, so no .bashrc either. The
execution environment associated with processes it spawns (such as when
you run a command) only consist of the global system environment setup
by the system prfile in /etc/profile, /etc/paths and /etc/paths.d and
any environment variables created and exported by the process itself
(i.e. using the setenv command in your emacs init.el). Whether emacs is
run in terminal mode or in GUI mode has nothing to do with what
environment context it has. The context is determined by the environment
context of the process parent.

So a common way to ensure your .profile environment variables are
available in Emacs when it is run from the dock is to use a shell
wrapper script which first sources .profile to create and export the
variables in .pofile and then run emacs. Often you won't just make it a
login shell, but simply source .profile. It is this script which is
added to the dock and is executed to start Emacs, ensuring it has your
.profile variables set and exported to the child process, Emacs. In
turn, any child processes created by Emacs will also include this
environment context (such as wehn you open an emacs terminal, run M-x
shell or execute org source blocks). 

The other approach is to just add the variables you need using 'setenv'
i your emacs init.el file. Often, there are only a few additional
variables you want inside of Emacs and child processes it creates, so
adding them to your init.el file is not hard. The one which is most
often needed is updates to the PATH variable to add additional
directories to search for executable files. You can use setenv to do
this or you can use the exec-path package or you can do what I do, which
is add these additional paths to /etc/paths.d. I don't bother with the
wrapper script on the Mac. There are only a couple of additional
variables I need defined, so I set them in my init.el using setenv. 

-- 
Tim Cross

Re: How to get shell source blocks to read my profile?

[Maxim Nikulin]

On 16/03/2021 20:21, George Mauer wrote:
> 
> I do still wonder what would be the disadvantage of just configuring it 
> to do --login by default and doing all configuration in profile scripts.

At the first glance it at least should not cause great trouble. Most 
shell stuff is written in fool-proof way to avoid loading interactive 
features in non-interactive shell or reinitialize login settings in 
nested shells. Local scripts could be written with less care.

I do not think you need completion functions or changing window title 
during execution of code snippets from emacs.

Self-contained org files have some value, you are trying to put some 
important settings outside. It might be reasonable for personal 
preferences as account names but hardly should be general practice.

On the other hand, I think, it is an equally valid question:
What is the profit?

For some reason your are avoiding over means to set environment 
variables. If you wish just exported environment variables, you could 
always create your own wrapper (untested)

#!/bin/bash

source ~/.config/wundershrc
exec "$0" "$@"

On 17/03/2021 03:32, Tim Cross wrote:
> It can sometimes help to remember what early Unix
> environments were like and what the resource constraints were to
> understand some of the design decisions. A time when memory and disk
> storage was extremely expensive and in short supply, where CPUs were
> slower and less capable than those commonly found in a modern washing
> machine.

Quite recent story is rewriting init scripts to be able to run them with 
dash instead of bash as /bin/sh. On the other hand unavailable features 
require more extensive usage of subprocesses like sed, awk, etc.