[-- Attachment #1: Type: text/plain, Size: 1639 bytes --] The attached patch adds ob-python value results handling for the following types of results: - Dictionaries - Numpy arrays - Pandas dataframes - Matplotlib figures This is a bigger commit than I'm used to, so I thought I better send it out before merging, in case someone notices obvious problems I missed. Overview of changes: Dictionaries are now transformed into alists before being converted to lisp. Previously, they had been getting mangled, like so: #+begin_src python return {"a": 1, "b": 2} #+end_src #+RESULTS: | a | : | 1 | b | : | 2 | But now they appear like so: #+begin_src python return {"a": 1, "b": 2} #+end_src #+RESULTS: | a | 1 | | b | 2 | Numpy arrays and pandas dataframes are also converted to tables automatically now. Tables converted from Pandas dataframes have row and column names. To avoid conversion, you can specify "raw", "verbatim", "scalar", or "output" in the ":results" header argument. For plotting, you can specify "graphics" in the ":results" header. You'll also need to provide a ":file" argument. The behavior depends on whether using output or value results. For output results, the current figure (pyplot.gcf) is cleared before evaluating, then the result saved. For value results, the block is expected to return a matplotlib Figure, which is saved. To set the figure size, do it from within Python. Here is an example of how to plot: #+begin_src python :results output graphics file :file boxplot.svg import matplotlib.pyplot as plt import seaborn as sns plt.figure(figsize=(5, 5)) tips = sns.load_dataset("tips") sns.boxplot(x="day", y="tip", data=tips) #+end_src [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-ob-python-Add-results-handling-for-dicts-dataframes-.patch --] [-- Type: text/x-patch, Size: 11849 bytes --] From 09f9c42bb629a356e1c36f04f69c8baf795b411b Mon Sep 17 00:00:00 2001 From: Jack Kamm <jackkamm@gmail.com> Date: Tue, 25 Aug 2020 21:57:24 -0700 Subject: [PATCH] ob-python: Add results handling for dicts, dataframes, arrays, plots * lisp/ob-python.el (org-babel-execute:python): Parse graphics-file from params. (org-babel-python--def-format-value): Python code for formatting value results before returning. (org-babel-python--output-graphics-wrapper): Python code for handling output graphics results. (org-babel-python--nonsession-value-wrapper): Replaces org-babel-python-wrapper-method, org-babel-python-pp-wrapper-method. (org-babel-python--session-output-wrapper): Renamed from org-babel-python--exec-tmpfile. (org-babel-python--session-value-wrapper): Renamed and modified from org-babel-python--eval-ast. (org-babel-python-evaluate-external-process): New parameter for graphics file. (org-babel-python-evaluate-session): New parameter for graphics file. Added results handling for dictionaries, Pandas and numpy tables, and matplotlib plots. --- etc/ORG-NEWS | 16 +++++- lisp/ob-python.el | 122 +++++++++++++++++++++++++++++++--------------- 2 files changed, 96 insertions(+), 42 deletions(-) diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS index 10658a970..4f9863a5b 100644 --- a/etc/ORG-NEWS +++ b/etc/ORG-NEWS @@ -66,8 +66,8 @@ to switch to the new signature. *** Python session return values must be top-level expression statements Python blocks with ~:session :results value~ header arguments now only -return a value if the last line is a top-level expression statement. -Also, when a None value is returned, "None" will be printed under +return a value if the last line is a top-level expression statement, +otherwise the result is None. Also, None will now show up under "#+RESULTS:", as it already did with ~:results value~ for non-session blocks. @@ -235,6 +235,18 @@ Screen blocks now recognize the =:screenrc= header argument and pass its value to the screen command via the "-c" option. The default remains =/dev/null= (i.e. a clean screen session) +*** =ob-python.el=: Support for more result types and plotting + +=ob-python= now recognizes dictionaries, numpy arrays, and pandas +dataframes, and will convert them to org-mode tables when appropriate. + +When the header argument =:results graphic= is set, =ob-python= will +use matplotlib to save graphics. The behavior depends on whether value +or output results are used. For value results, the last line should +return a matplotlib Figure object to plot. For output results, the +current figure (as returned by =pyplot.gcf()=) is cleared and then +plotted. + *** =RET= and =C-j= now obey ~electric-indent-mode~ Since Emacs 24.4, ~electric-indent-mode~ is enabled by default. In diff --git a/lisp/ob-python.el b/lisp/ob-python.el index 44e1b63e0..92ca82625 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -79,6 +79,8 @@ (defun org-babel-execute:python (body params) org-babel-python-command)) (session (org-babel-python-initiate-session (cdr (assq :session params)))) + (graphics-file (and (member "graphics" (assq :result-params params)) + (org-babel-graphical-output-file params))) (result-params (cdr (assq :result-params params))) (result-type (cdr (assq :result-type params))) (return-val (when (and (eq result-type 'value) (not session)) @@ -89,7 +91,8 @@ (defun org-babel-execute:python (body params) (concat body (if return-val (format "\nreturn %s" return-val) "")) params (org-babel-variable-assignments:python params))) (result (org-babel-python-evaluate - session full-body result-type result-params preamble))) + session full-body result-type result-params preamble + graphics-file))) (org-babel-reassemble-table result (org-babel-pick-name (cdr (assq :colname-names params)) @@ -225,67 +228,102 @@ (defun org-babel-python-initiate-session (&optional session _params) (org-babel-python-session-buffer (org-babel-python-initiate-session-by-key session)))) -(defconst org-babel-python-wrapper-method - " -def main(): +(defconst org-babel-python--def-format-value "\ +def __org_babel_python_format_value(result, result_file, result_params): + with open(result_file, 'w') as f: + if 'graphics' in result_params: + result.savefig(result_file) + elif 'pp' in result_params: + import pprint + f.write(pprint.pformat(result)) + else: + if not set(result_params).intersection(\ +['scalar', 'verbatim', 'raw']): + def dict2alist(res): + if isinstance(res, dict): + return [(k, dict2alist(v)) for k, v in res.items()] + elif isinstance(res, list) or isinstance(res, tuple): + return [dict2alist(x) for x in res] + else: + return res + result = dict2alist(result) + try: + import pandas as pd + except ModuleNotFoundError: + pass + else: + if isinstance(result, pd.DataFrame): + result = [[''] + list(result.columns), None] + \ +[[i] + list(row) for i, row in result.iterrows()] + try: + import numpy as np + except ModuleNotFoundError: + pass + else: + if isinstance(result, np.ndarray): + result = result.tolist() + f.write(str(result))") + +(defun org-babel-python--output-graphics-wrapper + (body graphics-file) + "Wrap BODY to plot to GRAPHICS-FILE if it is non-nil." + (if graphics-file + (format "\ +import matplotlib.pyplot as __org_babel_python_plt +__org_babel_python_plt.gcf().clear() %s +__org_babel_python_plt.savefig('%s')" body graphics-file) + body)) -open('%s', 'w').write( str(main()) )") -(defconst org-babel-python-pp-wrapper-method - " -import pprint +(defconst org-babel-python--nonsession-value-wrapper + (concat org-babel-python--def-format-value " def main(): %s -open('%s', 'w').write( pprint.pformat(main()) )") +__org_babel_python_format_value(main(), '%s', %s)") + "TODO") -(defconst org-babel-python--exec-tmpfile "\ +(defconst org-babel-python--session-output-wrapper "\ with open('%s') as f: exec(compile(f.read(), f.name, 'exec'))" - "Template for Python session command with output results. + "Wrapper for session block with output results. Has a single %s escape, the tempfile containing the source code to evaluate.") -(defconst org-babel-python--eval-ast "\ +(defconst org-babel-python--session-value-wrapper + (concat org-babel-python--def-format-value " import ast - with open('%s') as f: __org_babel_python_ast = ast.parse(f.read()) __org_babel_python_final = __org_babel_python_ast.body[-1] - if isinstance(__org_babel_python_final, ast.Expr): __org_babel_python_ast.body = __org_babel_python_ast.body[:-1] exec(compile(__org_babel_python_ast, '<string>', 'exec')) __org_babel_python_final = eval(compile(ast.Expression( __org_babel_python_final.value), '<string>', 'eval')) - with open('%s', 'w') as f: - if %s: - import pprint - f.write(pprint.pformat(__org_babel_python_final)) - else: - f.write(str(__org_babel_python_final)) else: exec(compile(__org_babel_python_ast, '<string>', 'exec')) - __org_babel_python_final = None" - "Template for Python session command with value results. + __org_babel_python_final = None +__org_babel_python_format_value(__org_babel_python_final, '%s', %s)") + "Wrapper for session block with value results. Has three %s escapes to be filled in: 1. Tempfile containing source to evaluate. 2. Tempfile to write results to. -3. Whether to pretty print, \"True\" or \"False\".") +3. result-params, converted from lisp to Python list.") (defun org-babel-python-evaluate - (session body &optional result-type result-params preamble) + (session body &optional result-type result-params preamble graphics-file) "Evaluate BODY as Python code." (if session (org-babel-python-evaluate-session - session body result-type result-params) + session body result-type result-params graphics-file) (org-babel-python-evaluate-external-process - body result-type result-params preamble))) + body result-type result-params preamble graphics-file))) (defun org-babel-python-evaluate-external-process - (body &optional result-type result-params preamble) + (body &optional result-type result-params preamble graphics-file) "Evaluate BODY in external python process. If RESULT-TYPE equals `output' then return standard output as a string. If RESULT-TYPE equals `value' then return the value of the @@ -294,16 +332,16 @@ (defun org-babel-python-evaluate-external-process (pcase result-type (`output (org-babel-eval org-babel-python-command (concat preamble (and preamble "\n") - body))) - (`value (let ((tmp-file (org-babel-temp-file "python-"))) + (org-babel-python--output-graphics-wrapper + body graphics-file)))) + (`value (let ((results-file (or graphics-file + (org-babel-temp-file "python-")))) (org-babel-eval org-babel-python-command (concat preamble (and preamble "\n") (format - (if (member "pp" result-params) - org-babel-python-pp-wrapper-method - org-babel-python-wrapper-method) + org-babel-python--nonsession-value-wrapper (with-temp-buffer (python-mode) (insert body) @@ -314,14 +352,15 @@ (defun org-babel-python-evaluate-external-process (line-end-position))) (forward-line 1)) (buffer-string)) - (org-babel-process-file-name tmp-file 'noquote)))) - (org-babel-eval-read-file tmp-file)))))) + (org-babel-process-file-name results-file 'noquote) + (org-babel-python-var-to-python result-params)))) + (org-babel-eval-read-file results-file)))))) (org-babel-result-cond result-params raw (org-babel-python-table-or-string (org-trim raw))))) (defun org-babel-python-evaluate-session - (session body &optional result-type result-params) + (session body &optional result-type result-params graphics-file) "Pass BODY to the Python process in SESSION. If RESULT-TYPE equals `output' then return standard output as a string. If RESULT-TYPE equals `value' then return the value of the @@ -334,17 +373,20 @@ (defun org-babel-python-evaluate-session (with-temp-file tmp-src-file (insert body)) (pcase result-type (`output - (let ((src-str (format org-babel-python--exec-tmpfile - (org-babel-process-file-name - tmp-src-file 'noquote)))) + (let ((src-str (org-babel-python--output-graphics-wrapper + (format org-babel-python--session-output-wrapper + (org-babel-process-file-name + tmp-src-file 'noquote)) + graphics-file))) (if (eq 'python-mode org-babel-python-mode) (py-send-string-no-output src-str (get-buffer-process session) session) (python-shell-send-string-no-output src-str)))) (`value - (let* ((results-file (org-babel-temp-file "python-")) + (let* ((results-file (or graphics-file + (org-babel-temp-file "python-"))) (src-str (format - org-babel-python--eval-ast + org-babel-python--session-value-wrapper (org-babel-process-file-name tmp-src-file 'noquote) (org-babel-process-file-name results-file 'noquote) (org-babel-python-var-to-python result-params)))) -- 2.28.0
Jack Kamm writes: > The attached patch adds ob-python value results handling for the > following types of results: > > - Dictionaries > - Numpy arrays > - Pandas dataframes > - Matplotlib figures Thanks. Just a couple of drive-by comments... > + try: > + import pandas as pd > + except ModuleNotFoundError: ModuleNotFoundError wasn't added until Python 3.6, so I think it'd be better to use its parent class, ImportError. > + pass > + else: > + if isinstance(result, pd.DataFrame): > + result = [[''] + list(result.columns), None] + \ > +[[i] + list(row) for i, row in result.iterrows()] Should handling of Series also be added?
[-- Attachment #1: Type: text/plain, Size: 514 bytes --] Hi Kyle, Thanks for the comments, I'm attaching an updated patch. Kyle Meyer <kyle@kyleam.com> writes: > ModuleNotFoundError wasn't added until Python 3.6, so I think it'd be > better to use its parent class, ImportError. I did not know this, thanks for the tip. > Should handling of Series also be added? Yes, I've done so now. I'm not sure whether it's better to treat it like a row or column vector, but since it has an "index", which are the row names in a DataFrame, I decided to treat it as a column. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-ob-python-Add-results-handling-for-dicts-dataframes-.patch --] [-- Type: text/x-patch, Size: 12292 bytes --] From 40db6b5497de78a9e69de219f4686b405db10c81 Mon Sep 17 00:00:00 2001 From: Jack Kamm <jackkamm@gmail.com> Date: Tue, 25 Aug 2020 21:57:24 -0700 Subject: [PATCH] ob-python: Add results handling for dicts, dataframes, arrays, plots * lisp/ob-python.el (org-babel-execute:python): Parse graphics-file from params. (org-babel-python--def-format-value): Python code for formatting value results before returning. (org-babel-python--output-graphics-wrapper): Python code for handling output graphics results. (org-babel-python--nonsession-value-wrapper): Replaces org-babel-python-wrapper-method, org-babel-python-pp-wrapper-method. (org-babel-python--session-output-wrapper): Renamed from org-babel-python--exec-tmpfile. (org-babel-python--session-value-wrapper): Renamed and modified from org-babel-python--eval-ast. (org-babel-python-evaluate-external-process): New parameter for graphics file. (org-babel-python-evaluate-session): New parameter for graphics file. Added results handling for dictionaries, Pandas and numpy tables, and matplotlib plots. --- etc/ORG-NEWS | 17 ++++++- lisp/ob-python.el | 126 +++++++++++++++++++++++++++++++--------------- 2 files changed, 100 insertions(+), 43 deletions(-) diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS index 10658a970..75c945572 100644 --- a/etc/ORG-NEWS +++ b/etc/ORG-NEWS @@ -66,8 +66,8 @@ to switch to the new signature. *** Python session return values must be top-level expression statements Python blocks with ~:session :results value~ header arguments now only -return a value if the last line is a top-level expression statement. -Also, when a None value is returned, "None" will be printed under +return a value if the last line is a top-level expression statement, +otherwise the result is None. Also, None will now show up under "#+RESULTS:", as it already did with ~:results value~ for non-session blocks. @@ -235,6 +235,19 @@ Screen blocks now recognize the =:screenrc= header argument and pass its value to the screen command via the "-c" option. The default remains =/dev/null= (i.e. a clean screen session) +*** =ob-python.el=: Support for more result types and plotting + +=ob-python= now recognizes dictionaries, numpy arrays, and pandas +dataframes/series, and will convert them to org-mode tables when +appropriate. + +When the header argument =:results graphics= is set, =ob-python= will +use matplotlib to save graphics. The behavior depends on whether value +or output results are used. For value results, the last line should +return a matplotlib Figure object to plot. For output results, the +current figure (as returned by =pyplot.gcf()=) is cleared before +evaluation, and then plotted afterwards. + *** =RET= and =C-j= now obey ~electric-indent-mode~ Since Emacs 24.4, ~electric-indent-mode~ is enabled by default. In diff --git a/lisp/ob-python.el b/lisp/ob-python.el index 44e1b63e0..fb8fe380e 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -79,6 +79,8 @@ (defun org-babel-execute:python (body params) org-babel-python-command)) (session (org-babel-python-initiate-session (cdr (assq :session params)))) + (graphics-file (and (member "graphics" (assq :result-params params)) + (org-babel-graphical-output-file params))) (result-params (cdr (assq :result-params params))) (result-type (cdr (assq :result-type params))) (return-val (when (and (eq result-type 'value) (not session)) @@ -89,7 +91,8 @@ (defun org-babel-execute:python (body params) (concat body (if return-val (format "\nreturn %s" return-val) "")) params (org-babel-variable-assignments:python params))) (result (org-babel-python-evaluate - session full-body result-type result-params preamble))) + session full-body result-type result-params preamble + graphics-file))) (org-babel-reassemble-table result (org-babel-pick-name (cdr (assq :colname-names params)) @@ -225,67 +228,104 @@ (defun org-babel-python-initiate-session (&optional session _params) (org-babel-python-session-buffer (org-babel-python-initiate-session-by-key session)))) -(defconst org-babel-python-wrapper-method - " -def main(): +(defconst org-babel-python--def-format-value "\ +def __org_babel_python_format_value(result, result_file, result_params): + with open(result_file, 'w') as f: + if 'graphics' in result_params: + result.savefig(result_file) + elif 'pp' in result_params: + import pprint + f.write(pprint.pformat(result)) + else: + if not set(result_params).intersection(\ +['scalar', 'verbatim', 'raw']): + def dict2alist(res): + if isinstance(res, dict): + return [(k, dict2alist(v)) for k, v in res.items()] + elif isinstance(res, list) or isinstance(res, tuple): + return [dict2alist(x) for x in res] + else: + return res + result = dict2alist(result) + try: + import pandas as pd + except ImportError: + pass + else: + if isinstance(result, pd.DataFrame): + result = [[''] + list(result.columns), None] + \ +[[i] + list(row) for i, row in result.iterrows()] + elif isinstance(result, pd.Series): + result = list(result.items()) + try: + import numpy as np + except ImportError: + pass + else: + if isinstance(result, np.ndarray): + result = result.tolist() + f.write(str(result))") + +(defun org-babel-python--output-graphics-wrapper + (body graphics-file) + "Wrap BODY to plot to GRAPHICS-FILE if it is non-nil." + (if graphics-file + (format "\ +import matplotlib.pyplot as __org_babel_python_plt +__org_babel_python_plt.gcf().clear() %s +__org_babel_python_plt.savefig('%s')" body graphics-file) + body)) -open('%s', 'w').write( str(main()) )") -(defconst org-babel-python-pp-wrapper-method - " -import pprint +(defconst org-babel-python--nonsession-value-wrapper + (concat org-babel-python--def-format-value " def main(): %s -open('%s', 'w').write( pprint.pformat(main()) )") +__org_babel_python_format_value(main(), '%s', %s)") + "TODO") -(defconst org-babel-python--exec-tmpfile "\ +(defconst org-babel-python--session-output-wrapper "\ with open('%s') as f: exec(compile(f.read(), f.name, 'exec'))" - "Template for Python session command with output results. + "Wrapper for session block with output results. Has a single %s escape, the tempfile containing the source code to evaluate.") -(defconst org-babel-python--eval-ast "\ +(defconst org-babel-python--session-value-wrapper + (concat org-babel-python--def-format-value " import ast - with open('%s') as f: __org_babel_python_ast = ast.parse(f.read()) __org_babel_python_final = __org_babel_python_ast.body[-1] - if isinstance(__org_babel_python_final, ast.Expr): __org_babel_python_ast.body = __org_babel_python_ast.body[:-1] exec(compile(__org_babel_python_ast, '<string>', 'exec')) __org_babel_python_final = eval(compile(ast.Expression( __org_babel_python_final.value), '<string>', 'eval')) - with open('%s', 'w') as f: - if %s: - import pprint - f.write(pprint.pformat(__org_babel_python_final)) - else: - f.write(str(__org_babel_python_final)) else: exec(compile(__org_babel_python_ast, '<string>', 'exec')) - __org_babel_python_final = None" - "Template for Python session command with value results. + __org_babel_python_final = None +__org_babel_python_format_value(__org_babel_python_final, '%s', %s)") + "Wrapper for session block with value results. Has three %s escapes to be filled in: 1. Tempfile containing source to evaluate. 2. Tempfile to write results to. -3. Whether to pretty print, \"True\" or \"False\".") +3. result-params, converted from lisp to Python list.") (defun org-babel-python-evaluate - (session body &optional result-type result-params preamble) + (session body &optional result-type result-params preamble graphics-file) "Evaluate BODY as Python code." (if session (org-babel-python-evaluate-session - session body result-type result-params) + session body result-type result-params graphics-file) (org-babel-python-evaluate-external-process - body result-type result-params preamble))) + body result-type result-params preamble graphics-file))) (defun org-babel-python-evaluate-external-process - (body &optional result-type result-params preamble) + (body &optional result-type result-params preamble graphics-file) "Evaluate BODY in external python process. If RESULT-TYPE equals `output' then return standard output as a string. If RESULT-TYPE equals `value' then return the value of the @@ -294,16 +334,16 @@ (defun org-babel-python-evaluate-external-process (pcase result-type (`output (org-babel-eval org-babel-python-command (concat preamble (and preamble "\n") - body))) - (`value (let ((tmp-file (org-babel-temp-file "python-"))) + (org-babel-python--output-graphics-wrapper + body graphics-file)))) + (`value (let ((results-file (or graphics-file + (org-babel-temp-file "python-")))) (org-babel-eval org-babel-python-command (concat preamble (and preamble "\n") (format - (if (member "pp" result-params) - org-babel-python-pp-wrapper-method - org-babel-python-wrapper-method) + org-babel-python--nonsession-value-wrapper (with-temp-buffer (python-mode) (insert body) @@ -314,14 +354,15 @@ (defun org-babel-python-evaluate-external-process (line-end-position))) (forward-line 1)) (buffer-string)) - (org-babel-process-file-name tmp-file 'noquote)))) - (org-babel-eval-read-file tmp-file)))))) + (org-babel-process-file-name results-file 'noquote) + (org-babel-python-var-to-python result-params)))) + (org-babel-eval-read-file results-file)))))) (org-babel-result-cond result-params raw (org-babel-python-table-or-string (org-trim raw))))) (defun org-babel-python-evaluate-session - (session body &optional result-type result-params) + (session body &optional result-type result-params graphics-file) "Pass BODY to the Python process in SESSION. If RESULT-TYPE equals `output' then return standard output as a string. If RESULT-TYPE equals `value' then return the value of the @@ -334,24 +375,27 @@ (defun org-babel-python-evaluate-session (with-temp-file tmp-src-file (insert body)) (pcase result-type (`output - (let ((src-str (format org-babel-python--exec-tmpfile - (org-babel-process-file-name - tmp-src-file 'noquote)))) + (let ((src-str (org-babel-python--output-graphics-wrapper + (format org-babel-python--session-output-wrapper + (org-babel-process-file-name + tmp-src-file 'noquote)) + graphics-file))) (if (eq 'python-mode org-babel-python-mode) (py-send-string-no-output src-str (get-buffer-process session) session) (python-shell-send-string-no-output src-str)))) (`value - (let* ((results-file (org-babel-temp-file "python-")) + (let* ((results-file (or graphics-file + (org-babel-temp-file "python-"))) (src-str (format - org-babel-python--eval-ast + org-babel-python--session-value-wrapper (org-babel-process-file-name tmp-src-file 'noquote) (org-babel-process-file-name results-file 'noquote) (org-babel-python-var-to-python result-params)))) (if (eq 'python-mode org-babel-python-mode) (py-shell-send-string src-str (get-buffer-process session)) (python-shell-send-string src-str)) - (sleep-for 0 5) + (sleep-for 0 10) (org-babel-eval-read-file results-file))))))) (org-babel-result-cond result-params results -- 2.28.0
[-- Attachment #1: Type: text/plain, Size: 1036 bytes --] After taking another look at my patch, I realized that I was not quite converting dictionaries to proper alists. Attached is a tweak to do this properly. The printing of dictionaries is not quite as pretty, in particular it's not a table anymore: #+begin_src python return {"a": 1, "b": 2} #+end_src #+RESULTS: : ((a . 1) (b . 2)) But, it feels like the right thing to do, since the result handling code works by converting the result to an elisp value, before passing it to org-mode to decide how to render it. And the proper elisp conversion of a dict should be an alist or a plist. Ideally I wouldn't have to do this from the Python code, and could let org-babel-script-escape convert the dict objects. It would also be useful for other languages with similar dictionaries, like javascript. But it seems fairly complex to implement this from the elisp side, and I'm not sure I'm up for it. I also noticed that I had left a couple docstrings as TODOs -- I'll fix those before finalizing the patch over the next couple weeks. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0002-Convert-dictionary-output-to-a-proper-alist.patch --] [-- Type: text/x-patch, Size: 1364 bytes --] From 76a1ad4d50e6638244d9aa17e45895b8b38b3cd0 Mon Sep 17 00:00:00 2001 From: Jack Kamm <jackkamm@gmail.com> Date: Sun, 30 Aug 2020 08:51:04 -0700 Subject: [PATCH 2/2] Convert dictionary output to a proper alist Note: to be squashed with the previous patch before merging --- lisp/ob-python.el | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lisp/ob-python.el b/lisp/ob-python.el index fb8fe380e..08c1c48e9 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -239,9 +239,14 @@ (defconst org-babel-python--def-format-value "\ else: if not set(result_params).intersection(\ ['scalar', 'verbatim', 'raw']): + class alist(dict): + def __str__(self): + return '({})'.format(' '.join(['({} . {})'.format(repr(k), repr(v)) for k, v in self.items()])) + def __repr__(self): + return self.__str__() def dict2alist(res): if isinstance(res, dict): - return [(k, dict2alist(v)) for k, v in res.items()] + return alist({k: dict2alist(v) for k, v in res.items()}) elif isinstance(res, list) or isinstance(res, tuple): return [dict2alist(x) for x in res] else: -- 2.28.0
[-- Attachment #1: Type: text/plain, Size: 937 bytes --] After letting it sit, I'm not sure that my patch above is a good idea anymore. While it would be useful, it also adds substantial complexity to ob-python. For now, I think I prefer to keep ob-python leaner, so am going to hold off on this. An alternative approach is to have the user handle graphics and dataframes via noweb or header arguments. I've added a couple examples on worg, demonstrating how to use noweb to insert boilerplate code for handling matplotlib figures and pandas dataframes [0,1]. Additionally, I'm attaching a small patch to make it easier to handle graphics/dataframes via the :return header argument, as an alternative to noweb. The patch includes a couple examples in ORG-NEWS illustrating this. I'll wait a week or so for comments before merging this new, more limited patch into master. [0] https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-python.html [1] worg commit 59e320ad Cheers, Jack [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-ob-python-Improvements-to-return-header-argument.patch --] [-- Type: text/x-patch, Size: 3643 bytes --] From 118d8b5eb817e9a21e9d84f2f942fcc841ddc51f Mon Sep 17 00:00:00 2001 From: Jack Kamm <jackkamm@gmail.com> Date: Sat, 19 Sep 2020 08:44:30 -0700 Subject: [PATCH] ob-python: Improvements to :return header argument * lisp/ob-python.el (org-babel-execute:python): Allow return-val to be non-nil in sessions, and concatenate it after the expanded body. --- etc/ORG-NEWS | 53 +++++++++++++++++++++++++++++++++++++++++++++++ lisp/ob-python.el | 11 ++++++---- 2 files changed, 60 insertions(+), 4 deletions(-) diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS index 0ed626fb7..50a455ad5 100644 --- a/etc/ORG-NEWS +++ b/etc/ORG-NEWS @@ -11,6 +11,59 @@ See the end of the file for license conditions. Please send Org bug reports to mailto:emacs-orgmode@gnu.org. * Version 9.5 (not yet released) +** New features +*** =ob-python= improvements to =:return= header argument + +The =:return= header argument in =ob-python= now works for session +blocks as well as non-session blocks. Also, it now works with the +=:epilogue= header argument -- previously, setting the =:return= +header would cause the =:epilogue= to be ignored. + +This change allows more easily moving boilerplate out of the main code +block and into the header. For example, for plotting, we need to add +boilerplate to save the figure to a file and return the +filename. Instead of doing this within the code block, we can now +handle it through the header arguments as follows: + +#+BEGIN_SRC org +,#+header: :var fname="/home/jack/tmp/plot.svg" +,#+header: :epilogue plt.savefig(fname) +,#+header: :return fname +,#+begin_src python :results value file + import matplotlib, numpy + import matplotlib.pyplot as plt + fig=plt.figure(figsize=(4,2)) + x=numpy.linspace(-15,15) + plt.plot(numpy.sin(x)/x) + fig.tight_layout() +,#+end_src + +,#+RESULTS: +[[file:/home/jack/tmp/plot.svg]] +#+END_SRC + +As another example, we can use =:return= with the external [[https://pypi.org/project/tabulate/][tabulate]] +package, to convert pandas Dataframes into orgmode tables: + +#+begin_src org +,#+header: :prologue from tabulate import tabulate +,#+header: :return tabulate(table, headers=table.columns, tablefmt="orgtbl") +,#+begin_src python :results value raw :session + import pandas as pd + table = pd.DataFrame({ + "a": [1,2,3], + "b": [4,5,6] + }) +,#+end_src + +,#+RESULTS: +| | a | b | +|---+---+---| +| 0 | 1 | 4 | +| 1 | 2 | 5 | +| 2 | 3 | 6 | +#+end_src + * Version 9.4 ** Incompatible changes *** Possibly broken internal file links: please check and fix diff --git a/lisp/ob-python.el b/lisp/ob-python.el index 00a7c1a2d..785b9191b 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -81,13 +81,16 @@ (defun org-babel-execute:python (body params) (cdr (assq :session params)))) (result-params (cdr (assq :result-params params))) (result-type (cdr (assq :result-type params))) - (return-val (when (and (eq result-type 'value) (not session)) + (return-val (when (eq result-type 'value) (cdr (assq :return params)))) (preamble (cdr (assq :preamble params))) (full-body - (org-babel-expand-body:generic - (concat body (if return-val (format "\nreturn %s" return-val) "")) - params (org-babel-variable-assignments:python params))) + (concat + (org-babel-expand-body:generic + body params + (org-babel-variable-assignments:python params)) + (when return-val + (format (if session "\n%s" "\nreturn %s") return-val)))) (result (org-babel-python-evaluate session full-body result-type result-params preamble))) (org-babel-reassemble-table -- 2.28.0
Hi Jack, Jack Kamm <jackkamm@gmail.com> writes: > For now, I think I prefer to keep ob-python leaner, so am going to hold > off on this. The leaner the less maintainance ahead :) > I'll wait a week or so for comments before merging this new, more > limited patch into master. LGTM, thanks! -- Bastien
Thanks -- I've pushed this to master now.
Jack
Bastien <bzg@gnu.org> writes:
> Hi Jack,
>
> Jack Kamm <jackkamm@gmail.com> writes:
>
>> For now, I think I prefer to keep ob-python leaner, so am going to hold
>> off on this.
>
> The leaner the less maintainance ahead :)
>
>> I'll wait a week or so for comments before merging this new, more
>> limited patch into master.
>
> LGTM, thanks!
>
> --
> Bastien