org-babel, python, encoding and table

org-babel, python, encoding and table

[Roland Donat]

[-- Attachment #1: Type: text/plain, Size: 945 bytes --]

Hello,

My problem is about python code evaluation with org-babel that should give
a table containing accented characters.

Here is an example :
#+NAME: test1
#+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return
b
a = ( ( "é", "a" ), ( "a", "à" ) )
b = "é"
#+end_src

#+RESULTS: test1
: é

It's ok, no problem!

But :
#+NAME: test2
#+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return
a
a = ( ( "é", "a" ), ( "a", "à" ) )
b = "é"
#+end_src

#+RESULTS: test2
| \303\251 | a        |
| a        | \303\240 |

I don't understand why the accented characters are replaced by some codes
when the results is interpreted as org-table...

Any idea, workaround to solve my problem would be much appreciated!

I use Org-mode version 8.0.2 (8.0.2-2-g93da18-elpaplus @
/home/roland/.emacs.d/elpa/org-plus-contrib-20130429/).

Thanks in advance.

Best regards,

Roland.

[-- Attachment #2: Type: text/html, Size: 1646 bytes --]

Re: org-babel, python, encoding and table

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

> #+NAME: test2
> #+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return
> a
> a = ( ( "é", "a" ), ( "a", "à" ) )
> b = "é"
> #+end_src
>
> #+RESULTS: test2
> | \303\251 | a        |
> | a        | \303\240 |
>

Maybe this isn't an execution problem, but is rather a buffer encoding
problem.  I executed your example above in a small buffer (attached).  I
then saved this buffer and was forced to specify an encoding, I selected
utf8.  If I cat the resulting file from disk, the accented characters
appear correctly.


[-- Attachment #2: encodings.org --]
[-- Type: text/plain, Size: 192 bytes --]

#+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return a
a = ( ( "é", "a" ), ( "a", "à" ) )
b = "é"
#+end_src

#+RESULTS:
| é | a        |
| a        | à |


[-- Attachment #3: Type: text/plain, Size: 144 bytes --]


So maybe the problem is simply Emacs not displaying utf8 characters
correctly.

Hope this helps,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: org-babel, python, encoding and table

[Andreas Röhler]

Am 07.05.2013 18:41, schrieb Eric Schulte:
>> #+NAME: test2
>> #+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return
>> a
>> a = ( ( "é", "a" ), ( "a", "à" ) )
>> b = "é"
>> #+end_src
>>
>> #+RESULTS: test2
>> | \303\251 | a        |
>> | a        | \303\240 |
>>
>
> Maybe this isn't an execution problem, but is rather a buffer encoding
> problem.  I executed your example above in a small buffer (attached).  I
> then saved this buffer and was forced to specify an encoding, I selected
> utf8.  If I cat the resulting file from disk, the accented characters
> appear correctly.
>
>
>
>
> So maybe the problem is simply Emacs not displaying utf8 characters
> correctly.
>
> Hope this helps,
>

Here error occurs depending if a or b is specified for return, really strange.

Andreas

Re: org-babel, python, encoding and table

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 1032 bytes --]

Andreas Röhler <andreas.roehler@easy-emacs.de> writes:

> Am 07.05.2013 18:41, schrieb Eric Schulte:
>>> #+NAME: test2
>>> #+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return
>>> a
>>> a = ( ( "é", "a" ), ( "a", "à" ) )
>>> b = "é"
>>> #+end_src
>>>
>>> #+RESULTS: test2
>>> | \303\251 | a        |
>>> | a        | \303\240 |
>>>
>>
>> Maybe this isn't an execution problem, but is rather a buffer encoding
>> problem.  I executed your example above in a small buffer (attached).  I
>> then saved this buffer and was forced to specify an encoding, I selected
>> utf8.  If I cat the resulting file from disk, the accented characters
>> appear correctly.
>>
>>
>>
>>
>> So maybe the problem is simply Emacs not displaying utf8 characters
>> correctly.
>>
>> Hope this helps,
>>
>
> Here error occurs depending if a or b is specified for return, really strange.
>

I think this may be a problem with the Python code.  The snippet above
expands to the following python.


[-- Attachment #2: print.py --]
[-- Type: text/x-python, Size: 150 bytes --]

# -*- coding: utf-8 -*-

def main():
	a = ( ( "é", "a" ), ( "a", "à" ) )
	b = "é"
	return a

open('/tmp/output', 'w').write( str(main()) )

[-- Attachment #3: Type: text/plain, Size: 97 bytes --]


which, when executed returns the following raw string (which is then
interpreted by Org-mode).


[-- Attachment #4: output --]
[-- Type: text/plain, Size: 38 bytes --]

(('\xc3\xa9', 'a'), ('a', '\xc3\xa0'))

[-- Attachment #5: Type: text/plain, Size: 132 bytes --]


Maybe Python simply needs to be convinced to print in utf-8 format?

>
> Andreas
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: org-babel, python, encoding and table

[Andreas Röhler]

[-- Attachment #1: Type: text/plain, Size: 1399 bytes --]

Am 07.05.2013 20:18, schrieb Eric Schulte:
> Andreas Röhler <andreas.roehler@easy-emacs.de> writes:
>
>> Am 07.05.2013 18:41, schrieb Eric Schulte:
>>>> #+NAME: test2
>>>> #+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return
>>>> a
>>>> a = ( ( "é", "a" ), ( "a", "à" ) )
>>>> b = "é"
>>>> #+end_src
>>>>
>>>> #+RESULTS: test2
>>>> | \303\251 | a        |
>>>> | a        | \303\240 |
>>>>
>>>
>>> Maybe this isn't an execution problem, but is rather a buffer encoding
>>> problem.  I executed your example above in a small buffer (attached).  I
>>> then saved this buffer and was forced to specify an encoding, I selected
>>> utf8.  If I cat the resulting file from disk, the accented characters
>>> appear correctly.
>>>
>>>
>>>
>>>
>>> So maybe the problem is simply Emacs not displaying utf8 characters
>>> correctly.
>>>
>>> Hope this helps,
>>>
>>
>> Here error occurs depending if a or b is specified for return, really strange.
>>
>
> I think this may be a problem with the Python code.  The snippet above
> expands to the following python.
>
>
>
>
> which, when executed returns the following raw string (which is then
> interpreted by Org-mode).
>
>
>
>
> Maybe Python simply needs to be convinced to print in utf-8 format?
>
>>
>> Andreas
>>
>>
>

Get the wrong results with a Ipython0.12, but correct with Python3.2.3 and Python3.3 - all called from Emacs24.3

[-- Attachment #2: python.png --]
[-- Type: image/png, Size: 35303 bytes --]

Re: org-babel, python, encoding and table

[Nick Dokos]

Andreas Röhler <andreas.roehler@easy-emacs.de> writes:

> Am 07.05.2013 20:18, schrieb Eric Schulte:
>> Andreas Röhler <andreas.roehler@easy-emacs.de> writes:
>> ...
>> Maybe Python simply needs to be convinced to print in utf-8 format?
>
> Get the wrong results with a Ipython0.12, but correct with Python3.2.3 and Python3.3 - all called from Emacs24.3

unicode handling is one of the big changes between Python 2.x and Python
3.x. It's good to know that Python 3.x seems to make it trivial to
handle unicode correctly (although there might still be dragons there).

Here are some links:

  http://docs.python.org/2/howto/unicode.html
  http://docs.python.org/3/howto/unicode.html

that might shed some light.

-- 
Nick

Re: org-babel, python, encoding and table

[Andreas Röhler]

Am 07.05.2013 21:20, schrieb Nick Dokos:
> Andreas Röhler <andreas.roehler@easy-emacs.de> writes:
>
>> Am 07.05.2013 20:18, schrieb Eric Schulte:
>>> Andreas Röhler <andreas.roehler@easy-emacs.de> writes:
>>> ...
>>> Maybe Python simply needs to be convinced to print in utf-8 format?
>>
>> Get the wrong results with a Ipython0.12, but correct with Python3.2.3 and Python3.3 - all called from Emacs24.3
>
> unicode handling is one of the big changes between Python 2.x and Python
> 3.x. It's good to know that Python 3.x seems to make it trivial to
> handle unicode correctly (although there might still be dragons there).
>
> Here are some links:
>
>    http://docs.python.org/2/howto/unicode.html
>    http://docs.python.org/3/howto/unicode.html
>
> that might shed some light.
>

So it's up to choose the Python default interpreter.

Unfortunatly

(org-babel-do-load-languages
  'org-babel-load-languages
  '((python3 . t)))

seems not working.

while

type python3
python3 ist /usr/bin/python3

Re: org-babel, python, encoding and table

[Nick Dokos]

Andreas Röhler <andreas.roehler@easy-emacs.de> writes:

> So it's up to choose the Python default interpreter.
>
> Unfortunatly
>
> (org-babel-do-load-languages
>  'org-babel-load-languages
>  '((python3 . t)))
>
> seems not working.
>

The language is still python (more or less).  What you need to do is
customize org-babel-python-command.

> while
>
> type python3
> python3 ist /usr/bin/python3
>

-- 
Nick

Re: org-babel, python, encoding and table

[Andreas Röhler]

Am 07.05.2013 22:05, schrieb Nick Dokos:
> Andreas Röhler <andreas.roehler@easy-emacs.de> writes:
>
>> So it's up to choose the Python default interpreter.
>>
>> Unfortunatly
>>
>> (org-babel-do-load-languages
>>   'org-babel-load-languages
>>   '((python3 . t)))
>>
>> seems not working.
>>
>
> The language is still python (more or less).  What you need to do is
> customize org-babel-python-command.
>

Thanks. This reveals a bug in ob-python then. Will open a separate thread.

Andreas

Re: org-babel, python, encoding and table

[Roland Donat]

Nick Dokos <ndokos <at> gmail.com> writes:

> 
> Andreas Röhler <andreas.roehler <at> easy-emacs.de> writes:
> 
> > Am 07.05.2013 20:18, schrieb Eric Schulte:
> >> Andreas Röhler <andreas.roehler <at> easy-emacs.de> writes:
> >> ...
> >> Maybe Python simply needs to be convinced to print in utf-8 format?
> >
> > Get the wrong results with a Ipython0.12, but correct with Python3.2.3 
and Python3.3 - all called from Emacs24.3
> 
> unicode handling is one of the big changes between Python 2.x and Python
> 3.x. It's good to know that Python 3.x seems to make it trivial to
> handle unicode correctly (although there might still be dragons there).
> 
> Here are some links:
> 
>   http://docs.python.org/2/howto/unicode.html
>   http://docs.python.org/3/howto/unicode.html
> 
> that might shed some light.
> 

Thank you very much for your answers!!!

Well, I read that Python 3.x has improved the encoding handling but for now 
I have to keep using 2.7.x Python version.

Using Python outside of org-mode is not a problem to manage encoding 
correctly. In general, you can use encode/decode <str> methods to get what 
you want. Then, using the print function turns the coded character into 
human readable ones. 

But, what is hurting my poor little neurone is the fact that I don't 
understand why I get the correct result when the evaluation returns a single 
value and a wrong answer with a table.

I think I have to understand how org-babel builds a table from a Python list 
of lists but I have to confess that I'm afraid of reading Lisp programs, 
though Emacs is my best friend ;)... 

Well, I'll keep on investigating on that subject

Thanks again for your help!

Roland.
 

Re: org-babel, python, encoding and table

[Andreas Röhler]

[ ... ]
> But, what is hurting my poor little neurone is the fact that I don't
> understand why I get the correct result when the evaluation returns a single
> value and a wrong answer with a table.
>

that's not the case.
One from returns a, the other b, which are bound differently.
The tuple fails with Python2 that way.

> I think I have to understand how org-babel builds a table from a Python list
> of lists but I have to confess that I'm afraid of reading Lisp programs,
> though Emacs is my best friend ;)...
>
> Well, I'll keep on investigating on that subject
>
> Thanks again for your help!
>
> Roland.
>
>
>
>
>
>

Re: org-babel, python, encoding and table

[Andreas Röhler]

Am 07.05.2013 18:41, schrieb Eric Schulte:
>> #+NAME: test2
>> #+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return
>> a
>> a = ( ( "é", "a" ), ( "a", "à" ) )
>> b = "é"
>> #+end_src
>>
>> #+RESULTS: test2
>> | \303\251 | a        |
>> | a        | \303\240 |
>>
>
> Maybe this isn't an execution problem, but is rather a buffer encoding
> problem.  I executed your example above in a small buffer (attached).  I
> then saved this buffer and was forced to specify an encoding, I selected
> utf8.  If I cat the resulting file from disk, the accented characters
> appear correctly.
>

Confirming this.

BTW also return a[0][0] displays correct so far.

Cheers,

Andreas

Re: org-babel, python, encoding and table

[Roland DONAT]

Andreas Röhler <andreas.roehler <at> easy-emacs.de> writes:

> 
> Am 07.05.2013 18:41, schrieb Eric Schulte:
> >> #+NAME: test2
> >> #+begin_src python :results value :preamble # -*- coding: utf-8 -*- 
:return
> >> a
> >> a = ( ( "é", "a" ), ( "a", "à" ) )
> >> b = "é"
> >> #+end_src
> >>
> >> #+RESULTS: test2
> >> | \303\251 | a        |
> >> | a        | \303\240 |
> >>
> >
> > Maybe this isn't an execution problem, but is rather a buffer encoding
> > problem.  I executed your example above in a small buffer (attached).  I
> > then saved this buffer and was forced to specify an encoding, I selected
> > utf8.  If I cat the resulting file from disk, the accented characters
> > appear correctly.
> >
> 
> Confirming this.
> 
> BTW also return a[0][0] displays correct so far.
> 
> Cheers,
> 
> Andreas
> 
> 

Hello,

Just an update about this post.

I've kept on digging on the problem of org-babel python results that 
produces encoding problems in
the emacs buffer when the requested results is turned into a org table.

To remind and illustrate the problem, here is an example :
#+name: pytab-test
#+begin_src python :results value :session :preamble # -*- coding: utf-8 -*- 
:return a
a = ( ( "é", "a" ), ( "a", "à" ) )
a
#+end_src

#+TBLNAME: pytab-test
| \303\251 | a        |
| a        | \303\240 |


I have then two problems :
1. The characters are not well displayed in the buffer
2. If I try to save the buffer, emacs doesn't recognize the encoding and 
tells me that "utf-8-unix cannot encode these: \303 \251 [...]

So I decided to inspect what happened during the Python session...
Basically, Org-babel just write the str conversion of my tuple ( ( "é", "a" 
), ( "a", "à" ) ) (that appears (('\xc3\xa9',
'a'), ('a', '\xc3\xa0')) in the python interpreter) in a temporary file.

Then looking in this temporary file, I see that the strange characters are 
written directly \xc3, \xa9, etc.

Consequently, my guess is that org-babel has maybe some difficulties to deal 
with these characters
while reading the temporary file before displaying the results in the 
buffer. 

Unfortunately, this is just a guess and even less a solution... But am I on 
relevant lead???

Thanks in advance for any help...

Roland.

Re: org-babel, python, encoding and table

[Eric Schulte]

>
> To remind and illustrate the problem, here is an example :
> #+name: pytab-test
> #+begin_src python :results value :session :preamble # -*- coding: utf-8 -*- :return a
> a = ( ( "é", "a" ), ( "a", "à" ) )
> a
> #+end_src
>
> #+TBLNAME: pytab-test
> | \303\251 | a        |
> | a        | \303\240 |
>
>
> I have then two problems :
> 1. The characters are not well displayed in the buffer
> 2. If I try to save the buffer, emacs doesn't recognize the encoding and 
> tells me that "utf-8-unix cannot encode these: \303 \251 [...]
>
> So I decided to inspect what happened during the Python session...
> Basically, Org-babel just write the str conversion of my tuple ( ( "é", "a" 
> ), ( "a", "à" ) ) (that appears (('\xc3\xa9',
> 'a'), ('a', '\xc3\xa0')) in the python interpreter) in a temporary file.
>
> Then looking in this temporary file, I see that the strange characters
> are written directly \xc3, \xa9, etc.
>
> Consequently, my guess is that org-babel has maybe some difficulties
> to deal with these characters while reading the temporary file before
> displaying the results in the buffer.
>
> Unfortunately, this is just a guess and even less a solution... But am I on 
> relevant lead???
>

This does seem to be in the right direction.  This is similar to my
diagnosis of the non-session case [1].  If the python process prints
weird hex character values (e.g., '\xc3\xa0') instead of utf8 characters
(e.g., "à"), then Babel will insert the hex values into the buffer.

As I recall switching to python3 solved this problem?

For the session case, it may be possible to fix this by changing the
python code in the `org-babel-python-evaluate-session' function around
line 300 in ob-python.el.

Cheers,

>
> Thanks in advance for any help...
>
> Roland.
>
>
>
>


Footnotes: 
[1]  http://article.gmane.org/gmane.emacs.orgmode/71866

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: org-babel, python, encoding and table

[Christian Wittern]

On 2013-05-29 22:00, Eric Schulte wrote:
> This does seem to be in the right direction.  This is similar to my
> diagnosis of the non-session case [1].  If the python process prints
> weird hex character values (e.g., '\xc3\xa0') instead of utf8 characters
> (e.g., "à"), then Babel will insert the hex values into the buffer.
>
> As I recall switching to python3 solved this problem?
What I do regularily in my python code to force python2 into emitting utf8 
characters is the following:
#+BEGIN_SRC python
import codecs
sys.stdout = codecs.getwriter('utf8')(sys.stdout)
#+END_SRC

There are other methods, for example resetting the default encoding in the 
site-wide file site.py, but that is not very portable.

Hope this helps,

Christian

-- 
Christian Wittern, Kyoto