Babel, Python and UTF-8

Babel, Python and UTF-8

[Vincent Beffara]

Hi,

I am discovering org-mode and org-babel, and wanted to try it out with
python code blocks. My use case would involve text processing, encoded
as UTF-8. Usually a python file has to contain a line to that effect,
saying sth like   # -*- coding: utf-8 -*-    (without which an error is
raised).

Is there a way to manage that during org-babel block evaluation ?

Thanks for your help !

       /v

Re: Babel, Python and UTF-8

[Eric Schulte]

Hi Vincent,

I would suggest just trying it out first and seeing if you get an error
without such a line.  Also, you could try adding the line to the
beginning of your code block.

If there is definitely a problem please reply to this thread and we
could easily add a header argument for this case, possibly named
":prefix" which could be used to specify such a code-block-prefix for
code evaluation.  This header argument could then be set globally or on
the file, heading, or code block level.

Best -- Eric

Vincent Beffara <vbeffara@ens-lyon.fr> writes:

> Hi,
>
> I am discovering org-mode and org-babel, and wanted to try it out with
> python code blocks. My use case would involve text processing, encoded
> as UTF-8. Usually a python file has to contain a line to that effect,
> saying sth like   # -*- coding: utf-8 -*-    (without which an error is
> raised).
>
> Is there a way to manage that during org-babel block evaluation ?
>
> Thanks for your help !
>
>        /v
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Babel, Python and UTF-8

[Vincent Beffara]

Hi,

> I would suggest just trying it out first and seeing if you get an error
> without such a line.

Well, I do, that's why I'm asking ;-)

> Also, you could try adding the line to the beginning of your code
> block.

I tried like this :

--8<---------------cut here---------------start------------->8---
#+begin_src python
# -*- coding: utf-8 -*-
s = "é"
#+end_src
--8<---------------cut here---------------end--------------->8---

The error message tells me that the error happens on line 4, hence the
coding line ends up on line 3, and python only takes it into account if
it is one of the first two ... so it almost works, but it doesn't.

> If there is definitely a problem please reply to this thread and we
> could easily add a header argument for this case, possibly named
> ":prefix" which could be used to specify such a code-block-prefix for
> code evaluation.  This header argument could then be set globally or on
> the file, heading, or code block level.

That would be fantastic ! (And possibly useful in other cases as well
...)

Thanks,

        /vincent

Re: Babel, Python and UTF-8

[Sébastien Vauban]

Hi Vincent and Eric,

Vincent Beffara wrote:
>> I would suggest just trying it out first and seeing if you get an error
>> without such a line.
>
> Well, I do, that's why I'm asking ;-)
>
>> Also, you could try adding the line to the beginning of your code
>> block.
>
> I tried like this :
>
> #+begin_src python
> # -*- coding: utf-8 -*-
> s = "é"
> #+end_src

I'm not sure to understand your problem. In fact, the problem is not about
"inserting a prefix to the block", it's about the coding system itself, I
guess.

Your "é" in your Org buffer, how is it encoded?  Is your Org buffer an UTF-8
one, or a Latin-1?

In the first case, I don't see why:

- it would fail
- it would even need to be specified

In the second, you want an ISO Latin 1 block of text to be tangled or executed
as UTF-8. Is this right?


> The error message tells me that the error happens on line 4, hence the
> coding line ends up on line 3, and python only takes it into account if
> it is one of the first two ... so it almost works, but it doesn't.
>
>> If there is definitely a problem please reply to this thread and we
>> could easily add a header argument for this case, possibly named
>> ":prefix" which could be used to specify such a code-block-prefix for
>> code evaluation.  This header argument could then be set globally or on
>> the file, heading, or code block level.

Isn't such a prefix already available through the "shebang" option: can't it
be multiline?  Anyway, I don't think it's the problem here...

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Babel, Python and UTF-8

[Eric Schulte]

Hi,

I've just pushed up a new prefix header argument.  This header argument
only has meaning for python code blocks (since it is tied into
evaluation each language would have to handle it separately).  This is
only used during external evaluation (i.e. not when :session is
specified) and the value of the :prefix argument is prepended to the
temporary file evaluated by python.

The following demonstrate its usage:

** Python requires a utf-8 coding prefix
#+begin_src python :prefix # -*- coding: utf-8 -*- :return s
s = "é"
#+end_src

#+results:
: é

#+begin_src python :prefix # -*- coding: utf-8 -*- :results output
s = "é"
print(s)
#+end_src

#+results:
: é

Best -- Eric

Vincent Beffara <vbeffara@ens-lyon.fr> writes:

> Hi,
>
>> I would suggest just trying it out first and seeing if you get an error
>> without such a line.
>
> Well, I do, that's why I'm asking ;-)
>
>> Also, you could try adding the line to the beginning of your code
>> block.
>
> I tried like this :
>
> #+begin_src python
> # -*- coding: utf-8 -*-
> s = "é"
> #+end_src
>
> The error message tells me that the error happens on line 4, hence the
> coding line ends up on line 3, and python only takes it into account if
> it is one of the first two ... so it almost works, but it doesn't.
>
>> If there is definitely a problem please reply to this thread and we
>> could easily add a header argument for this case, possibly named
>> ":prefix" which could be used to specify such a code-block-prefix for
>> code evaluation.  This header argument could then be set globally or on
>> the file, heading, or code block level.
>
> That would be fantastic ! (And possibly useful in other cases as well
> ...)
>
> Thanks,
>
>         /vincent
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Babel, Python and UTF-8

[Sébastien Vauban]

Hi Vincent and Eric,

Vincent Beffara wrote:
>> I would suggest just trying it out first and seeing if you get an error
>> without such a line.
>
> Well, I do, that's why I'm asking ;-)
>
>> Also, you could try adding the line to the beginning of your code
>> block.
>
> I tried like this :
>
> #+begin_src python
> # -*- coding: utf-8 -*-
> s = "é"
> #+end_src

I'm not sure to understand your problem. In fact, the problem is not about
"inserting a prefix to the block", it's about the coding system itself, I
guess.

Your "é" in your Org buffer, how is it encoded?  Is your Org buffer an UTF-8
one, or a Latin-1?

In the first case, I don't see why:

- it would fail
- it would even need to be specified

In the second, you want an ISO Latin 1 block of text to be tangled or executed
as UTF-8. Is this right?


> The error message tells me that the error happens on line 4, hence the
> coding line ends up on line 3, and python only takes it into account if
> it is one of the first two ... so it almost works, but it doesn't.
>
>> If there is definitely a problem please reply to this thread and we
>> could easily add a header argument for this case, possibly named
>> ":prefix" which could be used to specify such a code-block-prefix for
>> code evaluation.  This header argument could then be set globally or on
>> the file, heading, or code block level.

Isn't such a prefix already available through the "shebang" option: can't it
be multiline?  Anyway, I don't think it's the problem here...

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Babel, Python and UTF-8

[Vincent Beffara]

Hi,

> I've just pushed up a new prefix header argument. 

Fantastic ! It works exactly as needed. Thanks a lot !

> This header argument only has meaning for python code blocks (since it
> is tied into evaluation each language would have to handle it
> separately).

That's fine, most cases can probably be handled using something like
<<prefix>> anyway. It is just Python insisting to find the coding
instruction in the first 2 lines that causes trouble, so a specific
implementation makes sense.

Thanks again,

       /v

Re: Babel, Python and UTF-8

[Vincent Beffara]

>> #+begin_src python
>> # -*- coding: utf-8 -*-
>> s = "é"
>> #+end_src
>
> I'm not sure to understand your problem. In fact, the problem is not
> about "inserting a prefix to the block", it's about the coding system
> itself, I guess.
>
> Your "é" in your Org buffer, how is it encoded?  Is your Org buffer an
> UTF-8 one, or a Latin-1?

Doesn't really matter, in fact ... Whenever a python file contains a
non-ascii character (utf-8 or latin-1) it insists on finding an
indication of the encoding within the file. If latin-1, it finds \xe9,
if utf-8, it finds \xc3, but in both cases it raises this:

# SyntaxError: Non-ASCII character '\xe9' in file /Users/vincent/toto.py
# on line 1, but no encoding declared; see
# http://www.python.org/peps/pep-0263.html for details

That's even if the character in question lies within a string. That
makes a lot of sense in terms of portability, but it makes on-the-fly
evaluation a bit more complicated.

> In the second, you want an ISO Latin 1 block of text to be tangled or
> executed as UTF-8. Is this right?

No, I want everything in utf-8, there is no converting to do anywhere,
just telling python that it is utf-8.

> Isn't such a prefix already available through the "shebang" option:
> can't it be multiline?  Anyway, I don't think it's the problem here...

Hmm, didn't know about :shebang, but it seems to be ignored (at least
for python) because this works without a syntax error:

--8<---------------cut here---------------start------------->8---
#+begin_src python :shebang sldfkj
return 1
#+end_src
--8<---------------cut here---------------end--------------->8---

Anyway, :prefix cannot hurt I guess ...

        /v

Re: Babel, Python and UTF-8

[Dan Davison]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 2893 bytes --]

"Eric Schulte" <schulte.eric@gmail.com> writes:

> Hi,
>
> I've just pushed up a new prefix header argument.

"preamble" might be a better name? or "header"? prefix makes me think of
adding a string onto the front of a string[1]; preamble and header are
used by e.g. latex to refer to prelimary lines at the top of a file. For
obscure options like this it might be OK to use a longer name
(e.g. ":header-line"), especially now we have multiline header args
with #+header.

>  This header argument
> only has meaning for python code blocks

but might in the future be used by others?

Dan

[1]: accurate i know, but many people won't think of a block as a
character string.


> (since it is tied into
> evaluation each language would have to handle it separately).  This is
> only used during external evaluation (i.e. not when :session is
> specified) and the value of the :prefix argument is prepended to the
> temporary file evaluated by python.
>
> The following demonstrate its usage:
>
> ** Python requires a utf-8 coding prefix
> #+begin_src python :prefix # -*- coding: utf-8 -*- :return s
> s = "é"
> #+end_src
>
> #+results:
> : é
>
> #+begin_src python :prefix # -*- coding: utf-8 -*- :results output
> s = "é"
> print(s)
> #+end_src
>
> #+results:
> : é
>
> Best -- Eric
>
> Vincent Beffara <vbeffara@ens-lyon.fr> writes:
>
>> Hi,
>>
>>> I would suggest just trying it out first and seeing if you get an error
>>> without such a line.
>>
>> Well, I do, that's why I'm asking ;-)
>>
>>> Also, you could try adding the line to the beginning of your code
>>> block.
>>
>> I tried like this :
>>
>> #+begin_src python
>> # -*- coding: utf-8 -*-
>> s = "é"
>> #+end_src
>>
>> The error message tells me that the error happens on line 4, hence the
>> coding line ends up on line 3, and python only takes it into account if
>> it is one of the first two ... so it almost works, but it doesn't.
>>
>>> If there is definitely a problem please reply to this thread and we
>>> could easily add a header argument for this case, possibly named
>>> ":prefix" which could be used to specify such a code-block-prefix for
>>> code evaluation.  This header argument could then be set globally or on
>>> the file, heading, or code block level.
>>
>> That would be fantastic ! (And possibly useful in other cases as well
>> ...)
>>
>> Thanks,
>>
>>         /vincent
>>
>>
>> _______________________________________________
>> Emacs-orgmode mailing list
>> Please use `Reply All' to send replies to the list.
>> Emacs-orgmode@gnu.org
>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Babel, Python and UTF-8

[Eric Schulte]

Sébastien Vauban <wxhgmqzgwmuf@spammotel.com> writes:

>
> Isn't such a prefix already available through the "shebang" option: can't it
> be multiline?  Anyway, I don't think it's the problem here...
>

The :shebang header argument is only used for tangling, not during
evaluation, however if you ever needed to tangle python code blocks with
non-ASCII encodings this would be the appropriate solution.

Re: Babel, Python and UTF-8

[Eric Schulte]

Dan Davison <dandavison7@gmail.com> writes:

> "Eric Schulte" <schulte.eric@gmail.com> writes:
>
>> Hi,
>>
>> I've just pushed up a new prefix header argument.
>
> "preamble" might be a better name? or "header"? prefix makes me think of
> adding a string onto the front of a string[1];

Absolutely, "prefix" was just the most obvious to me and I certainly
don't claim that makes it the most obvious choice globally :)

I would lean against "header" since we already talk about "header
arguments", but I think "preamble" is an improvement.  Does anyone have
a better suggestion?

Re: Babel, Python and UTF-8

[Sébastien Vauban]

Hi Eric and Dan,

"Eric Schulte" wrote:
> Dan Davison <dandavison7-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>> "Eric Schulte" <schulte.eric-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>>
>>> I've just pushed up a new prefix header argument.
>>
>> "preamble" might be a better name? or "header"? prefix makes me think of
>> adding a string onto the front of a string[1];
>
> Absolutely, "prefix" was just the most obvious to me and I certainly
> don't claim that makes it the most obvious choice globally :)
>
> I would lean against "header" since we already talk about "header
> arguments", but I think "preamble" is an improvement.  Does anyone have
> a better suggestion?

I second Dan's comment, and fully am in favor of preamble... Of course, then,
one might ask for postamble ;-)

What about shebang?  Isn't it a preamble as well?

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Babel, Python and UTF-8

[Sébastien Vauban]

Hi Eric,

"Eric Schulte" wrote:
> Sébastien Vauban <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes:
>> Isn't such a prefix already available through the "shebang" option: can't it
>> be multiline?  Anyway, I don't think it's the problem here...
>
> The :shebang header argument is only used for tangling, not during
> evaluation, however if you ever needed to tangle python code blocks with
> non-ASCII encodings this would be the appropriate solution.

Sorry, reading this after the other post...

But, does it still make sense to make that distinction:

- some sort of preamble only for tangling?
- a preamble for evaluation (and tangling)?

Why would the shebang not be inserted as well for evaluation?

Do you/we need:

- one only for eval,
- one only for tangling,
- and one for both?

Depending on that answer, I would maybe make it more explicit what's the role
of each, like:

- :eval-preamble
- :tangle-preambel
- :preamble

But I'm wondering if one sort of preamble wouldn't be enough (and be used both
for tangling and evaluating)?

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Babel, Python and UTF-8

[Vincent Beffara]

Hi,

"preamble" sounds fine (and it would be excellent to allow for a code
block as a preamble, instead of a string in the header or as an
alternative, because preambles once they are allowed tend to grow
uncontrollably ;->)

>> The :shebang header argument is only used for tangling, not during
>> evaluation, however if you ever needed to tangle python code blocks with
>> non-ASCII encodings this would be the appropriate solution.
>
> Sorry, reading this after the other post...
>
> But, does it still make sense to make that distinction:
>
> - some sort of preamble only for tangling?
> - a preamble for evaluation (and tangling)?

One naive question : why is the code path different for tangling and
evaluation ? One would think that a natural way for evaluation would be
to tangle the current block (plus included noweb stuff etc) into a
temporary file and eval that file ... and that would enable shebang for
evaluation as well. There must be something I am missing here.

           /v

Re: Babel, Python and UTF-8

[Sébastien Vauban]

Hi Vincent,

Vincent Beffara wrote:
> "preamble" sounds fine (and it would be excellent to allow for a code
> block as a preamble, instead of a string in the header or as an
> alternative, because preambles once they are allowed tend to grow
> uncontrollably ;->)

If you want a bigger preamble, just use noweb expansion:

#+begin_src python :noweb yes
<<preamble>>
...
my code
...
#+end_src

and have somewhere:

#+srcname: preamble
#+begin_src python
-*- coding-system: utf-8 -*-
...
rest of my preamble
...
#+end_src

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Babel, Python and UTF-8

[Eric Schulte]

Hi,

Vincent Beffara <vbeffara@ens-lyon.fr> writes:

> Hi,
>
> "preamble" sounds fine

As this seems to be the general consensus I've replaced "prefix" with
"preamble" as the header argument name

> (and it would be excellent to allow for a code block as a preamble,
> instead of a string in the header or as an alternative, because
> preambles once they are allowed tend to grow uncontrollably ;->)
>

This is currently possible using the `sbe' function.  Arbitrary emacs
lisp can be placed inside of header arguments, and the `sbe' take the
name of a code block and returns its result. e.g.

** Python requires a utf-8 coding prefix
#+source: my-preamble
#+begin_src python :return preamble
  preamble = "# -*- coding: utf-8 -*-"
#+end_src

#+begin_src python :preamble (sbe my-preamble) :return s
s = "é"
#+end_src

#+results:
: é

#+begin_src python :preamble # -*- coding: utf-8 -*- :results output
s = "é"
print(s)
#+end_src

#+results:
: é

>
>>> The :shebang header argument is only used for tangling, not during
>>> evaluation, however if you ever needed to tangle python code blocks with
>>> non-ASCII encodings this would be the appropriate solution.
>>
>> Sorry, reading this after the other post...
>>
>> But, does it still make sense to make that distinction:
>>
>> - some sort of preamble only for tangling?
>> - a preamble for evaluation (and tangling)?
>
> One naive question : why is the code path different for tangling and
> evaluation ? One would think that a natural way for evaluation would be
> to tangle the current block (plus included noweb stuff etc) into a
> temporary file and eval that file ... and that would enable shebang for
> evaluation as well. There must be something I am missing here.
>

Tangling works for *any* programming language, even those which have yet
to be created and have no explicit Emacs or Org-mode support, this is
because on tangling the code block is simply treated as text.

Evaluation requires language specific support for both performing the
evaluation and collecting the result.  Evaluation can (for example) be
performed using an external file or using a persistent session.  Even
when using an external file the contents of the file are often very
different when tangling and exporting, e.g.

#+source: my-preamble
#+begin_src python :return preamble
  preamble = "# -*- coding: utf-8 -*-"
#+end_src

tangles to the following (possibly with comments)

--8<---------------cut here---------------start------------->8---
preamble = "# -*- coding: utf-8 -*-"
--8<---------------cut here---------------end--------------->8---

and is evaluated as the following

--8<---------------cut here---------------start------------->8---
def main():
  preamble = "# -*- coding: utf-8 -*-"
  return preamble

open('temporary-result-collection-file', 'w').write( pprint.pformat(main()) )
--8<---------------cut here---------------end--------------->8---

Best -- Eric

>
>            /v
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Babel, Python and UTF-8

[Christopher Allan Webber]

(Also worth mentioning that python 3 allows you to use actual unicode
characters inside strings in the file, not \xff chars; not sure if
org-babel needs to change its habits between python2 and python3?)

Vincent Beffara <vbeffara@ens-lyon.fr> writes:

>>> #+begin_src python
>>> # -*- coding: utf-8 -*-
>>> s = "é"
>>> #+end_src
>>
>> I'm not sure to understand your problem. In fact, the problem is not
>> about "inserting a prefix to the block", it's about the coding system
>> itself, I guess.
>>
>> Your "é" in your Org buffer, how is it encoded?  Is your Org buffer an
>> UTF-8 one, or a Latin-1?
>
> Doesn't really matter, in fact ... Whenever a python file contains a
> non-ascii character (utf-8 or latin-1) it insists on finding an
> indication of the encoding within the file. If latin-1, it finds \xe9,
> if utf-8, it finds \xc3, but in both cases it raises this:
>
> # SyntaxError: Non-ASCII character '\xe9' in file /Users/vincent/toto.py
> # on line 1, but no encoding declared; see
> # http://www.python.org/peps/pep-0263.html for details
>
> That's even if the character in question lies within a string. That
> makes a lot of sense in terms of portability, but it makes on-the-fly
> evaluation a bit more complicated.
>
>> In the second, you want an ISO Latin 1 block of text to be tangled or
>> executed as UTF-8. Is this right?
>
> No, I want everything in utf-8, there is no converting to do anywhere,
> just telling python that it is utf-8.
>
>> Isn't such a prefix already available through the "shebang" option:
>> can't it be multiline?  Anyway, I don't think it's the problem here...
>
> Hmm, didn't know about :shebang, but it seems to be ignored (at least
> for python) because this works without a syntax error:
>
> #+begin_src python :shebang sldfkj
> return 1
> #+end_src
>
> Anyway, :prefix cannot hurt I guess ...
>
>         /v
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Babel, Python and UTF-8

[Vincent Beffara]

Hi,

>> (and it would be excellent to allow for a code block as a preamble,
>> instead of a string in the header or as an alternative, because
>> preambles once they are allowed tend to grow uncontrollably ;->)
>
> This is currently possible using the `sbe' function.  Arbitrary emacs
> lisp can be placed inside of header arguments, and the `sbe' take the
> name of a code block and returns its result.

Very cool ! That does all I want, thanks for the info. For multi-line it
is a bit heavy to write, with lots of \n and preamble .= "lskjd", but I
can live with that. Unless there is a way already to write something
like this ?

--8<---------------cut here---------------start------------->8---
#+source: my-preamble
#+begin_src python :return preamble
  # -*- coding: utf-8 -*-"
  import os,sys,whatever
#+end_src

#+begin_src python :preamble (org-babel-get-and-expand-source-code-body my-preamble) :return s
  s = "é"
#+end_src
--8<---------------cut here---------------end--------------->8---

There is org-babel-get-src-block-info but it looks at the block around
(point), not by name ... so I guess it would not be too hard to write
the extraction method, but it might be somewhere in the code already.

>> One naive question : why is the code path different for tangling and
>> evaluation ? One would think that a natural way for evaluation would be
>> to tangle the current block (plus included noweb stuff etc) into a
>> temporary file and eval that file ... and that would enable shebang for
>> evaluation as well. There must be something I am missing here.
>
> Tangling works for *any* programming language, even those which have yet
> to be created and have no explicit Emacs or Org-mode support, this is
> because on tangling the code block is simply treated as text.

As far as I understood from testing, tangling does adapt to the language
(at least to implement :var in a suitable way), so I was under the
impression that evaluating could be implemented as some kind of wrapping
around the tangled output - and obviously the wrapping would have to be
language-specific even if for the most part the tanglong is not.

I am just discovering all of this, sorry if I have horrible
misconceptions about the thing ...

Regards,

        /v

Re: Re: Babel, Python and UTF-8

[Eric Schulte]

Vincent Beffara <vbeffara@ens-lyon.fr> writes:

> Hi,
>
>>> (and it would be excellent to allow for a code block as a preamble,
>>> instead of a string in the header or as an alternative, because
>>> preambles once they are allowed tend to grow uncontrollably ;->)
>>
>> This is currently possible using the `sbe' function.  Arbitrary emacs
>> lisp can be placed inside of header arguments, and the `sbe' take the
>> name of a code block and returns its result.
>
> Very cool ! That does all I want, thanks for the info. For multi-line it
> is a bit heavy to write, with lots of \n and preamble .= "lskjd", but I
> can live with that. Unless there is a way already to write something
> like this ?
>
> #+source: my-preamble
> #+begin_src python :return preamble
>   # -*- coding: utf-8 -*-"
>   import os,sys,whatever
> #+end_src
>
> #+begin_src python :preamble (org-babel-get-and-expand-source-code-body my-preamble) :return s
>   s = "é"
> #+end_src
>
> There is org-babel-get-src-block-info but it looks at the block around
> (point), not by name ... so I guess it would not be too hard to write
> the extraction method, but it might be somewhere in the code already.
>

Yes, the following uses an internal Babel function, but is probably much
simpler

#+results: my-preamble
: # -*- coding: utf-8 -*-
: import os,sys,whatever

#+begin_src python :preamble (org-babel-ref-resolve "my-preamble") :return s
s = "é"
#+end_src

Note that as written this will return the following python error

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ImportError: No module named whatever

>
>>> One naive question : why is the code path different for tangling and
>>> evaluation ? One would think that a natural way for evaluation would be
>>> to tangle the current block (plus included noweb stuff etc) into a
>>> temporary file and eval that file ... and that would enable shebang for
>>> evaluation as well. There must be something I am missing here.
>>
>> Tangling works for *any* programming language, even those which have yet
>> to be created and have no explicit Emacs or Org-mode support, this is
>> because on tangling the code block is simply treated as text.
>
> As far as I understood from testing, tangling does adapt to the language
> (at least to implement :var in a suitable way), so I was under the
> impression that evaluating could be implemented as some kind of wrapping
> around the tangled output - and obviously the wrapping would have to be
> language-specific even if for the most part the tanglong is not.
>

Yes, some language specific features (e.g. variable expansion) can be
used by the tangling mechanisms if such features are defined for the
language in question, however tangling can be done in the absence of any
language specific features and thus works for any arbitrary language.

That shebang and preamble should remain separate for the other reasons
mentioned in my previous email.

>
> I am just discovering all of this, sorry if I have horrible
> misconceptions about the thing ...
>

No problem, it is a fairly (but I don't think overly) complex system.

>
> Regards,
>
>         /v
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Babel, Python and UTF-8

[Dan Davison]

"Eric Schulte" <schulte.eric@gmail.com> writes:

> Vincent Beffara <vbeffara@ens-lyon.fr> writes:
>
>> Hi,
>>
>>>> (and it would be excellent to allow for a code block as a preamble,
>>>> instead of a string in the header or as an alternative, because
>>>> preambles once they are allowed tend to grow uncontrollably ;->)
>>>
>>> This is currently possible using the `sbe' function.  Arbitrary emacs
>>> lisp can be placed inside of header arguments, and the `sbe' take the
>>> name of a code block and returns its result.

This makes me think of another good use of the sbe ("src block eval")
function. I'm often seeing Org documents with a src block like this,

#+source: essential-document-config
#+begin_src emacs-lisp
 ;; some essential document-specific configuration
#+end_src

and some instructions saying something like "To use this document, first
evaluate this code block".

This can be automated by using sbe in a local variables line at the end
of the Org file:

# Local variables:
# eval:(sbe "essential-document-config")
# End:

When the file is first opened, Emacs will evaluate the set-up blocks
(after asking for confirmation).

This isn't restricted to configuration of Emacs variables with
emacs-lisp blocks; eval lines could reference blocks in any language,
for example to start an ESS session and run some preparatory code, etc,
e.g.

#+source: document-config
#+begin_src emacs-lisp
(set (make-local-variable 'org-edit-src-content-indentation) 0)
#+end_src

#+source: start-ess
#+begin_src R :session *R session*
  a <- 1
#+end_src

# Local variables:
# eval:(sbe "document-config")
# eval:(sbe "start-ess")
# End:


Dan

>>
>> Very cool ! That does all I want, thanks for the info. For multi-line it
>> is a bit heavy to write, with lots of \n and preamble .= "lskjd", but I
>> can live with that. Unless there is a way already to write something
>> like this ?
>>
>> #+source: my-preamble
>> #+begin_src python :return preamble
>>   # -*- coding: utf-8 -*-"
>>   import os,sys,whatever
>> #+end_src
>>
>> #+begin_src python :preamble (org-babel-get-and-expand-source-code-body my-preamble) :return s
>>   s = "é"
>> #+end_src
>>
>> There is org-babel-get-src-block-info but it looks at the block around
>> (point), not by name ... so I guess it would not be too hard to write
>> the extraction method, but it might be somewhere in the code already.
>>
>
> Yes, the following uses an internal Babel function, but is probably much
> simpler
>
> #+results: my-preamble
> : # -*- coding: utf-8 -*-
> : import os,sys,whatever
>
> #+begin_src python :preamble (org-babel-ref-resolve "my-preamble") :return s
> s = ""
> #+end_src
>
> Note that as written this will return the following python error
>
> Traceback (most recent call last):
>   File "<stdin>", line 2, in <module>
> ImportError: No module named whatever
>
>>
>>>> One naive question : why is the code path different for tangling and
>>>> evaluation ? One would think that a natural way for evaluation would be
>>>> to tangle the current block (plus included noweb stuff etc) into a
>>>> temporary file and eval that file ... and that would enable shebang for
>>>> evaluation as well. There must be something I am missing here.
>>>
>>> Tangling works for *any* programming language, even those which have yet
>>> to be created and have no explicit Emacs or Org-mode support, this is
>>> because on tangling the code block is simply treated as text.
>>
>> As far as I understood from testing, tangling does adapt to the language
>> (at least to implement :var in a suitable way), so I was under the
>> impression that evaluating could be implemented as some kind of wrapping
>> around the tangled output - and obviously the wrapping would have to be
>> language-specific even if for the most part the tanglong is not.
>>
>
> Yes, some language specific features (e.g. variable expansion) can be
> used by the tangling mechanisms if such features are defined for the
> language in question, however tangling can be done in the absence of any
> language specific features and thus works for any arbitrary language.
>
> That shebang and preamble should remain separate for the other reasons
> mentioned in my previous email.
>
>>
>> I am just discovering all of this, sorry if I have horrible
>> misconceptions about the thing ...
>>
>
> No problem, it is a fairly (but I don't think overly) complex system.
>
>>
>> Regards,
>>
>>         /v
>>
>>
>> _______________________________________________
>> Emacs-orgmode mailing list
>> Please use `Reply All' to send replies to the list.
>> Emacs-orgmode@gnu.org
>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Babel, Python and UTF-8

[Eric S Fraga]

Dan Davison <dandavison7@gmail.com> writes:
> This makes me think of another good use of the sbe ("src block eval")
> function. I'm often seeing Org documents with a src block like this,
>
> #+source: essential-document-config
> #+begin_src emacs-lisp
>  ;; some essential document-specific configuration
> #+end_src
>
> and some instructions saying something like "To use this document, first
> evaluate this code block".
>
> This can be automated by using sbe in a local variables line at the end
> of the Org file:
>
> # Local variables:
> # eval:(sbe "essential-document-config")
> # End:

This is *really* nice.  Thanks!

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 23.2.1
: using Org-mode version 7.3 (release_7.3.211.g6388.dirty)

Re: Re: Babel, Python and UTF-8

[Thomas S. Dye]

Hi Dan,

Emacs configuration is one of the highest barriers to entry for  
potential adopters of Org-mode, IMO.  The idea of context-sensitive  
configuration is potentially terrific.  It gets the user to work more  
quickly than would otherwise be the case.  The problem I've run into  
is that exiting a buffer doesn't change the configuration back to some  
initial, or base, state.  I'm on to the next task but still configured  
to do the last thing.

I'm not a software engineer, so this idea might be wacky, but it seems  
to me that some code to restore the user's start-up settings would be  
a useful, standard part of an Org-mode file, something like the export  
template that many (most?) org-mode files use.   I think it would be  
handy to be able to stick something simple in my Org-mode file so that  
I was confident I knew exactly how Emacs was configured while I was  
using that file.

Tom

On Dec 5, 2010, at 11:42 PM, Dan Davison wrote:

> "Eric Schulte" <schulte.eric@gmail.com> writes:
>
>> Vincent Beffara <vbeffara@ens-lyon.fr> writes:
>>
>>> Hi,
>>>
>>>>> (and it would be excellent to allow for a code block as a  
>>>>> preamble,
>>>>> instead of a string in the header or as an alternative, because
>>>>> preambles once they are allowed tend to grow uncontrollably ;->)
>>>>
>>>> This is currently possible using the `sbe' function.  Arbitrary  
>>>> emacs
>>>> lisp can be placed inside of header arguments, and the `sbe' take  
>>>> the
>>>> name of a code block and returns its result.
>
> This makes me think of another good use of the sbe ("src block eval")
> function. I'm often seeing Org documents with a src block like this,
>
> #+source: essential-document-config
> #+begin_src emacs-lisp
> ;; some essential document-specific configuration
> #+end_src
>
> and some instructions saying something like "To use this document,  
> first
> evaluate this code block".
>
> This can be automated by using sbe in a local variables line at the  
> end
> of the Org file:
>
> # Local variables:
> # eval:(sbe "essential-document-config")
> # End:
>
> When the file is first opened, Emacs will evaluate the set-up blocks
> (after asking for confirmation).
>
> This isn't restricted to configuration of Emacs variables with
> emacs-lisp blocks; eval lines could reference blocks in any language,
> for example to start an ESS session and run some preparatory code,  
> etc,
> e.g.
>
> #+source: document-config
> #+begin_src emacs-lisp
> (set (make-local-variable 'org-edit-src-content-indentation) 0)
> #+end_src
>
> #+source: start-ess
> #+begin_src R :session *R session*
>  a <- 1
> #+end_src
>
> # Local variables:
> # eval:(sbe "document-config")
> # eval:(sbe "start-ess")
> # End:
>
>
> Dan
>
>>>
>>> Very cool ! That does all I want, thanks for the info. For multi- 
>>> line it
>>> is a bit heavy to write, with lots of \n and preamble .= "lskjd",  
>>> but I
>>> can live with that. Unless there is a way already to write something
>>> like this ?
>>>
>>> #+source: my-preamble
>>> #+begin_src python :return preamble
>>>  # -*- coding: utf-8 -*-"
>>>  import os,sys,whatever
>>> #+end_src
>>>
>>> #+begin_src python :preamble (org-babel-get-and-expand-source-code- 
>>> body my-preamble) :return s
>>>  s = "é"
>>> #+end_src
>>>
>>> There is org-babel-get-src-block-info but it looks at the block  
>>> around
>>> (point), not by name ... so I guess it would not be too hard to  
>>> write
>>> the extraction method, but it might be somewhere in the code  
>>> already.
>>>
>>
>> Yes, the following uses an internal Babel function, but is probably  
>> much
>> simpler
>>
>> #+results: my-preamble
>> : # -*- coding: utf-8 -*-
>> : import os,sys,whatever
>>
>> #+begin_src python :preamble (org-babel-ref-resolve "my- 
>> preamble") :return s
>> s = ""
>> #+end_src
>>
>> Note that as written this will return the following python error
>>
>> Traceback (most recent call last):
>>  File "<stdin>", line 2, in <module>
>> ImportError: No module named whatever
>>
>>>
>>>>> One naive question : why is the code path different for tangling  
>>>>> and
>>>>> evaluation ? One would think that a natural way for evaluation  
>>>>> would be
>>>>> to tangle the current block (plus included noweb stuff etc) into a
>>>>> temporary file and eval that file ... and that would enable  
>>>>> shebang for
>>>>> evaluation as well. There must be something I am missing here.
>>>>
>>>> Tangling works for *any* programming language, even those which  
>>>> have yet
>>>> to be created and have no explicit Emacs or Org-mode support,  
>>>> this is
>>>> because on tangling the code block is simply treated as text.
>>>
>>> As far as I understood from testing, tangling does adapt to the  
>>> language
>>> (at least to implement :var in a suitable way), so I was under the
>>> impression that evaluating could be implemented as some kind of  
>>> wrapping
>>> around the tangled output - and obviously the wrapping would have  
>>> to be
>>> language-specific even if for the most part the tanglong is not.
>>>
>>
>> Yes, some language specific features (e.g. variable expansion) can be
>> used by the tangling mechanisms if such features are defined for the
>> language in question, however tangling can be done in the absence  
>> of any
>> language specific features and thus works for any arbitrary language.
>>
>> That shebang and preamble should remain separate for the other  
>> reasons
>> mentioned in my previous email.
>>
>>>
>>> I am just discovering all of this, sorry if I have horrible
>>> misconceptions about the thing ...
>>>
>>
>> No problem, it is a fairly (but I don't think overly) complex system.
>>
>>>
>>> Regards,
>>>
>>>        /v
>>>
>>>
>>> _______________________________________________
>>> Emacs-orgmode mailing list
>>> Please use `Reply All' to send replies to the list.
>>> Emacs-orgmode@gnu.org
>>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>>
>> _______________________________________________
>> Emacs-orgmode mailing list
>> Please use `Reply All' to send replies to the list.
>> Emacs-orgmode@gnu.org
>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Babel, Python and UTF-8

[Achim Gratz]

"Thomas S. Dye" <tsd@tsdye.com> writes:
> Emacs configuration is one of the highest barriers to entry for
> potential adopters of Org-mode, IMO.  The idea of context-sensitive
> configuration is potentially terrific.  It gets the user to work more
> quickly than would otherwise be the case.  The problem I've run into
> is that exiting a buffer doesn't change the configuration back to some
> initial, or base, state.  I'm on to the next task but still configured
> to do the last thing.

Isn't that what buffer-local variables are for?


Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: Babel, Python and UTF-8

[Dan Davison]

"Thomas S. Dye" <tsd@tsdye.com> writes:

> Hi Dan,
>
> Emacs configuration is one of the highest barriers to entry for
> potential adopters of Org-mode, IMO.  The idea of context-sensitive
> configuration is potentially terrific.  It gets the user to work more
> quickly than would otherwise be the case.  The problem I've run into
> is that exiting a buffer doesn't change the configuration back to some
> initial, or base, state.  I'm on to the next task but still configured
> to do the last thing.

Hi Tom,

I think someone else suggested using buffer-local variables to address
this problem, and I think that's the correct suggestion. One of the
examples below shows how to set a buffer-local value:

[...]

> #+source: document-config
>> #+begin_src emacs-lisp
>> (set (make-local-variable 'org-edit-src-content-indentation) 0)
>> #+end_src

As it happens, for that particular setting to be useful requires a patch
that is pending review, but, in general, I think this is the way to do
what you're describing.

Dan


>>
>> #+source: start-ess
>> #+begin_src R :session *R session*
>>  a <- 1
>> #+end_src
>>
>> # Local variables:
>> # eval:(sbe "document-config")
>> # eval:(sbe "start-ess")
>> # End:
>>
>>
>> Dan
>>
>>>>
>>>> Very cool ! That does all I want, thanks for the info. For multi-
>>>> line it
>>>> is a bit heavy to write, with lots of \n and preamble .= "lskjd",
>>>> but I
>>>> can live with that. Unless there is a way already to write something
>>>> like this ?
>>>>
>>>> #+source: my-preamble
>>>> #+begin_src python :return preamble
>>>>  # -*- coding: utf-8 -*-"
>>>>  import os,sys,whatever
>>>> #+end_src
>>>>
>>>> #+begin_src python :preamble
>>>> (org-babel-get-and-expand-source-code-
>>>> body my-preamble) :return s
>>>>  s = "é"
>>>> #+end_src
>>>>
>>>> There is org-babel-get-src-block-info but it looks at the block
>>>> around
>>>> (point), not by name ... so I guess it would not be too hard to
>>>> write
>>>> the extraction method, but it might be somewhere in the code
>>>> already.
>>>>
>>>
>>> Yes, the following uses an internal Babel function, but is probably
>>> much
>>> simpler
>>>
>>> #+results: my-preamble
>>> : # -*- coding: utf-8 -*-
>>> : import os,sys,whatever
>>>
>>> #+begin_src python :preamble (org-babel-ref-resolve "my-
>>> preamble") :return s
>>> s = ""
>>> #+end_src
>>>
>>> Note that as written this will return the following python error
>>>
>>> Traceback (most recent call last):
>>>  File "<stdin>", line 2, in <module>
>>> ImportError: No module named whatever
>>>
>>>>
>>>>>> One naive question : why is the code path different for tangling
>>>>>> and
>>>>>> evaluation ? One would think that a natural way for evaluation
>>>>>> would be
>>>>>> to tangle the current block (plus included noweb stuff etc) into a
>>>>>> temporary file and eval that file ... and that would enable
>>>>>> shebang for
>>>>>> evaluation as well. There must be something I am missing here.
>>>>>
>>>>> Tangling works for *any* programming language, even those which
>>>>> have yet
>>>>> to be created and have no explicit Emacs or Org-mode support,
>>>>> this is
>>>>> because on tangling the code block is simply treated as text.
>>>>
>>>> As far as I understood from testing, tangling does adapt to the
>>>> language
>>>> (at least to implement :var in a suitable way), so I was under the
>>>> impression that evaluating could be implemented as some kind of
>>>> wrapping
>>>> around the tangled output - and obviously the wrapping would have
>>>> to be
>>>> language-specific even if for the most part the tanglong is not.
>>>>
>>>
>>> Yes, some language specific features (e.g. variable expansion) can be
>>> used by the tangling mechanisms if such features are defined for the
>>> language in question, however tangling can be done in the absence
>>> of any
>>> language specific features and thus works for any arbitrary language.
>>>
>>> That shebang and preamble should remain separate for the other
>>> reasons
>>> mentioned in my previous email.
>>>
>>>>
>>>> I am just discovering all of this, sorry if I have horrible
>>>> misconceptions about the thing ...
>>>>
>>>
>>> No problem, it is a fairly (but I don't think overly) complex system.
>>>
>>>>
>>>> Regards,
>>>>
>>>>        /v
>>>>
>>>>
>>>> _______________________________________________
>>>> Emacs-orgmode mailing list
>>>> Please use `Reply All' to send replies to the list.
>>>> Emacs-orgmode@gnu.org
>>>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>>>
>>> _______________________________________________
>>> Emacs-orgmode mailing list
>>> Please use `Reply All' to send replies to the list.
>>> Emacs-orgmode@gnu.org
>>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>>
>> _______________________________________________
>> Emacs-orgmode mailing list
>> Please use `Reply All' to send replies to the list.
>> Emacs-orgmode@gnu.org
>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Babel, Python and UTF-8

[Thomas S. Dye]

On Dec 6, 2010, at 8:07 AM, Achim Gratz wrote:

> "Thomas S. Dye" <tsd@tsdye.com> writes:
>> Emacs configuration is one of the highest barriers to entry for
>> potential adopters of Org-mode, IMO.  The idea of context-sensitive
>> configuration is potentially terrific.  It gets the user to work more
>> quickly than would otherwise be the case.  The problem I've run into
>> is that exiting a buffer doesn't change the configuration back to  
>> some
>> initial, or base, state.  I'm on to the next task but still  
>> configured
>> to do the last thing.
>
> Isn't that what buffer-local variables are for?
>
>
> Achim.
> -- 
> +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
>
> SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
> http://Synth.Stromeko.net/Downloads.html#KorgSDada
>
>

Aloha Achim,

I suppose so, but AFAIK I haven't seen them in the configuration code  
blocks that Dan mentioned.  I might be missing something.

I experimented with buffer-local configuration variables myself a  
while back, but my Lisp skills are sub-minimal and I couldn't make  
them work to my satisfaction.

All the best,
Tom


> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode