Enabling another language in org-babel

Enabling another language in org-babel

[Brendan Halpin]

Does there exist any documentation on extending org-babel to another
language?

Relatedly, is anyone working on adding Stata coverage to org-babel?


Regards,

Brendan
-- 
Brendan Halpin, Head, Department of Sociology, University of Limerick, Ireland
Tel: w +353-61-213147  f +353-61-202569  h +353-61-338562;  Room F1-002 x 3147
mailto:brendan.halpin@ul.ie    ULSociology on Facebook: http://on.fb.me/fjIK9t
http://teaching.sociology.ul.ie/bhalpin/wordpress         twitter:@ULSociology

Re: Enabling another language in org-babel

[Eric Schulte]

Brendan Halpin <brendan.halpin@ul.ie> writes:

> Does there exist any documentation on extending org-babel to another
> language?
>

Hi Brendan,

There is a template [1] which may be filled out to add support for a new
language.  More generally simply defining a function named
`org-babel-execute:foo' will cause babel to try to execute foo code
blocks using that function.  See the template for existing
`org-babel-execute:*' functions.

>
> Relatedly, is anyone working on adding Stata coverage to org-babel?
>

I'm not but who knows.  If you do add support for Stata please consider
contributing it to Org-mode (see [2]).

Best,
Eric

>
>
> Regards,
>
> Brendan


Footnotes: 
[1]  http://orgmode.org/w/worg.git/blob/HEAD:/org-contrib/babel/ob-template.el

[2]  http://orgmode.org/worg/org-contribute.html

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D (see https://u.fsf.org/yw)

Re: Enabling another language in org-babel

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 1524 bytes --]

Eric Schulte <schulte.eric@gmail.com> writes:

> Brendan Halpin <brendan.halpin@ul.ie> writes:
>
>> Does there exist any documentation on extending org-babel to another
>> language?
>>
>
> Hi Brendan,
>
> There is a template [1] which may be filled out to add support for a new
> language.  More generally simply defining a function named
> `org-babel-execute:foo' will cause babel to try to execute foo code
> blocks using that function.  See the template for existing
> `org-babel-execute:*' functions.
>
>>
>> Relatedly, is anyone working on adding Stata coverage to org-babel?
>>
>
> I'm not but who knows.  If you do add support for Stata please consider
> contributing it to Org-mode (see [2]).

stata should be not to difficult to add, as ESS supports stata and I
would guess that most functions would be the same (or very similar)

Cheers,

Rainer

>
> Best,
> Eric
>
>>
>>
>> Regards,
>>
>> Brendan
>
>
> Footnotes: 
> [1]  http://orgmode.org/w/worg.git/blob/HEAD:/org-contrib/babel/ob-template.el
>
> [2]  http://orgmode.org/worg/org-contribute.html

-- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer@krugs.de

Skype:      RMkrug

PGP: 0x0F52F982

[-- Attachment #2: Type: application/pgp-signature, Size: 494 bytes --]

Re: Enabling another language in org-babel

[Ista Zahn]

I just hacked up a rudimentary ob-stata.el by replacing "julia" with
"stata" in ob-julia.el by G. Jay Kerns (and making a few other minor
tweaks). Only ":results output" works (no graphics, no :results
value). In short it's in pretty rough shape, but it does work for some
basic things. Perhaps someone with better emacs-lisp can polish it up.
The code is available at
https://gist.github.com/izahn/5745dcf6dd07a6b05084

Best,
Ista

On Tue, Jun 24, 2014 at 8:05 AM, Rainer M Krug <Rainer@krugs.de> wrote:
> Eric Schulte <schulte.eric@gmail.com> writes:
>
>> Brendan Halpin <brendan.halpin@ul.ie> writes:
>>
>>> Does there exist any documentation on extending org-babel to another
>>> language?
>>>
>>
>> Hi Brendan,
>>
>> There is a template [1] which may be filled out to add support for a new
>> language.  More generally simply defining a function named
>> `org-babel-execute:foo' will cause babel to try to execute foo code
>> blocks using that function.  See the template for existing
>> `org-babel-execute:*' functions.
>>
>>>
>>> Relatedly, is anyone working on adding Stata coverage to org-babel?
>>>
>>
>> I'm not but who knows.  If you do add support for Stata please consider
>> contributing it to Org-mode (see [2]).
>
> stata should be not to difficult to add, as ESS supports stata and I
> would guess that most functions would be the same (or very similar)
>
> Cheers,
>
> Rainer
>
>>
>> Best,
>> Eric
>>
>>>
>>>
>>> Regards,
>>>
>>> Brendan
>>
>>
>> Footnotes:
>> [1]  http://orgmode.org/w/worg.git/blob/HEAD:/org-contrib/babel/ob-template.el
>>
>> [2]  http://orgmode.org/worg/org-contribute.html
>
> --
> Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)
>
> Centre of Excellence for Invasion Biology
> Stellenbosch University
> South Africa
>
> Tel :       +33 - (0)9 53 10 27 44
> Cell:       +33 - (0)6 85 62 59 98
> Fax :       +33 - (0)9 58 10 27 44
>
> Fax (D):    +49 - (0)3 21 21 25 22 44
>
> email:      Rainer@krugs.de
>
> Skype:      RMkrug
>
> PGP: 0x0F52F982

Re: Enabling another language in org-babel

[G. Jay Kerns]

Great work, Ista!

Cheers,
Jay

-- 
G. Jay Kerns, Ph.D.
Youngstown State University
http://gkerns.people.ysu.edu/


On 06/24/14 15:52, Ista Zahn wrote:
> I just hacked up a rudimentary ob-stata.el by replacing "julia" with
> "stata" in ob-julia.el by G. Jay Kerns (and making a few other minor
> tweaks). Only ":results output" works (no graphics, no :results
> value). In short it's in pretty rough shape, but it does work for some
> basic things. Perhaps someone with better emacs-lisp can polish it up.
> The code is available at
> https://gist.github.com/izahn/5745dcf6dd07a6b05084
> 
> Best,
> Ista
> 
> On Tue, Jun 24, 2014 at 8:05 AM, Rainer M Krug <Rainer@krugs.de> wrote:
>> Eric Schulte <schulte.eric@gmail.com> writes:
>>
>>> Brendan Halpin <brendan.halpin@ul.ie> writes:
>>>
>>>> Does there exist any documentation on extending org-babel to another
>>>> language?
>>>>
>>>
>>> Hi Brendan,
>>>
>>> There is a template [1] which may be filled out to add support for a new
>>> language.  More generally simply defining a function named
>>> `org-babel-execute:foo' will cause babel to try to execute foo code
>>> blocks using that function.  See the template for existing
>>> `org-babel-execute:*' functions.
>>>
>>>>
>>>> Relatedly, is anyone working on adding Stata coverage to org-babel?
>>>>
>>>
>>> I'm not but who knows.  If you do add support for Stata please consider
>>> contributing it to Org-mode (see [2]).
>>
>> stata should be not to difficult to add, as ESS supports stata and I
>> would guess that most functions would be the same (or very similar)
>>
>> Cheers,
>>
>> Rainer
>>
>>>
>>> Best,
>>> Eric
>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Brendan
>>>
>>>
>>> Footnotes:
>>> [1]  http://orgmode.org/w/worg.git/blob/HEAD:/org-contrib/babel/ob-template.el
>>>
>>> [2]  http://orgmode.org/worg/org-contribute.html
>>
>> --
>> Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)
>>
>> Centre of Excellence for Invasion Biology
>> Stellenbosch University
>> South Africa
>>
>> Tel :       +33 - (0)9 53 10 27 44
>> Cell:       +33 - (0)6 85 62 59 98
>> Fax :       +33 - (0)9 58 10 27 44
>>
>> Fax (D):    +49 - (0)3 21 21 25 22 44
>>
>> email:      Rainer@krugs.de
>>
>> Skype:      RMkrug
>>
>> PGP: 0x0F52F982

Re: Enabling another language in org-babel

[Bastien]

Hi Ista,

Ista Zahn <istazahn@gmail.com> writes:

> The code is available at
> https://gist.github.com/izahn/5745dcf6dd07a6b05084

Looks good -- would you like to add it to the contrib/ directory
in Org's repository?

As for the original question, I think it would be a nice addition,
let's put it somewhere on our collective TODO list :)

-- 
 Bastien

Re: Enabling another language in org-babel

[Ista Zahn]

Hi Bastien,

I don't know if this is ready for contrib yet (though I don't object
if you think it is). I'm not sure what quality standards exist for
contrib, but this is really quick and dirty; it would be nice to at
least implement graphics support. On the other hand its better than
nothing, so maybe it does make sense to add it to contrib.

Also I'm not sure what the development process is once things go in
contrib (sorry for my ignorance!). Does contrib live in the same code
repository as the rest of org, or are things in contrib developed
separately and pulled in later?

In short feel free to add it to contrib if you want, but be aware that
it is  in pretty rough shape.

Best,
Ista

On Wed, Jun 25, 2014 at 4:33 AM, Bastien <bzg@gnu.org> wrote:
> Hi Ista,
>
> Ista Zahn <istazahn@gmail.com> writes:
>
>> The code is available at
>> https://gist.github.com/izahn/5745dcf6dd07a6b05084
>
> Looks good -- would you like to add it to the contrib/ directory
> in Org's repository?
>
> As for the original question, I think it would be a nice addition,
> let's put it somewhere on our collective TODO list :)
>
> --
>  Bastien

Re: Enabling another language in org-babel

[Bastien]

Hi Ista,

Ista Zahn <istazahn@gmail.com> writes:

> I don't know if this is ready for contrib yet (though I don't object
> if you think it is). I'm not sure what quality standards exist for
> contrib, but this is really quick and dirty; it would be nice to at
> least implement graphics support. On the other hand its better than
> nothing, so maybe it does make sense to add it to contrib.

Yes, it makes sense to add it to contrib, it will help others step up
and improve the code if needed.

> Also I'm not sure what the development process is once things go in
> contrib (sorry for my ignorance!). Does contrib live in the same code
> repository as the rest of org, or are things in contrib developed
> separately and pulled in later?

contrib/ lives in the same Git repository than the rest of Org (which
is arguably a problem).  So everything is developed at the same place.

Once you send me your public key, I'll grant you write access to the
repository and you can push commits.

> In short feel free to add it to contrib if you want, but be aware that
> it is  in pretty rough shape.

Please provide a separate commit on the list for this piece of code,
so that we can review it and commit it.  And if you want to get push
access, please send me your public key.

Thanks!

-- 
 Bastien

Re: Enabling another language in org-babel

[Ista Zahn]

[-- Attachment #1: Type: text/plain, Size: 2030 bytes --]

Hi all,

Please find code attached that implements stata support in org-babel.
The attached ob-stata.el is a modified version of ob-julia.el. It
currently supports evaluation of stata code from babel code blocks
with ':results output' (and this has been set as the default). Neither
':results value' nor ':results graphics' work. I suspect that
':results value' doen't make sense for stata, in which case all the
value-specific stuff should be removed. I would like to eventually get
':results graphics' working, but have not yet implemented it.

If there are no objections I would like to add this to contrib.

Best,
Ista

On Mon, Jul 28, 2014 at 9:20 AM, Bastien <bzg@gnu.org> wrote:
> Hi Ista,
>
> Ista Zahn <istazahn@gmail.com> writes:
>
>> I don't know if this is ready for contrib yet (though I don't object
>> if you think it is). I'm not sure what quality standards exist for
>> contrib, but this is really quick and dirty; it would be nice to at
>> least implement graphics support. On the other hand its better than
>> nothing, so maybe it does make sense to add it to contrib.
>
> Yes, it makes sense to add it to contrib, it will help others step up
> and improve the code if needed.
>
>> Also I'm not sure what the development process is once things go in
>> contrib (sorry for my ignorance!). Does contrib live in the same code
>> repository as the rest of org, or are things in contrib developed
>> separately and pulled in later?
>
> contrib/ lives in the same Git repository than the rest of Org (which
> is arguably a problem).  So everything is developed at the same place.
>
> Once you send me your public key, I'll grant you write access to the
> repository and you can push commits.
>
>> In short feel free to add it to contrib if you want, but be aware that
>> it is  in pretty rough shape.
>
> Please provide a separate commit on the list for this piece of code,
> so that we can review it and commit it.  And if you want to get push
> access, please send me your public key.
>
> Thanks!
>
> --
>  Bastien

[-- Attachment #2: ob-stata.el --]
[-- Type: text/x-emacs-lisp, Size: 12428 bytes --]

;;; ob-stata.el --- org-babel functions for stata code evaluation

;; Copyright (C) 2014 Ista Zahn
;; Author: Ista Zahn, based on ob-julia.el by G. Jay Kerns, and ob-R.el
;; by Eric Schulte and Dan Davison


;; This file is not part of GNU Emacs.

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation; either version 3, or (at your option)
;; any later version.
;;
;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs; see the file COPYING.  If not, write to the
;; Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
;; Boston, MA 02110-1301, USA.

;;; Commentary:

;; The file provides Org-Babel support for evaluating stata code.
;; It is basically result of find-and-replace "stata" for "julia"
;; in ob-julia.el by G. Jay Kerns. Only ":results output" works: the
;; header args must include ":results output" (this is the default).
;; Note that I'm not sure ':results value' makes sense or is useful 
;; but I have left all the value-processing stuff inherited from 
;; ob-julia and ob-R. ':results graphics' would be nice, but I have
;; not tried to implement it.

;;; Requirements:

;; Stata and ESS are required.

;;; Code:
(require 'ob)
(eval-when-compile (require 'cl))

(declare-function orgtbl-to-csv "org-table" (table params))
(declare-function stata "ext:ess-stata" (&optional start-args))
(declare-function inferior-ess-send-input "ext:ess-inf" ())
(declare-function ess-make-buffer-current "ext:ess-inf" ())
(declare-function ess-eval-buffer "ext:ess-inf" (vis))
(declare-function org-number-sequence "org-compat" (from &optional to inc))
(declare-function org-remove-if-not "org" (predicate seq))

(defconst org-babel-header-args:stata
  '((width		 . :any)
    (horizontal		 . :any)
    (results             . ((file list vector table scalar verbatim)
			    (raw org html latex code pp wrap)
			    (replace silent append prepend)
                            ;; NOTE: not sure 'value' makes sense in stata
                            ;; we may want to remove it from the list
			    (output value graphics))))
  "stata-specific header arguments.")

(add-to-list 'org-babel-tangle-lang-exts '("stata" . "do"))

;; only ':results output' currently works, so make that the default
(defvar org-babel-default-header-args:stata '((:results . "output")))

(defcustom org-babel-stata-command inferior-STA-program-name
  "Name of command to use for executing stata code."
  :group 'org-babel
  :version "24.4"
  :package-version '(Org . "8.0")
  :type 'string)

(defvar ess-local-process-name) ; dynamically scoped
(defun org-babel-edit-prep:stata (info)
  (let ((session (cdr (assoc :session (nth 2 info)))))
    (when (and session (string-match "^\\*\\(.+?\\)\\*$" session))
      (save-match-data (org-babel-stata-initiate-session session nil)))))

(defun org-babel-expand-body:stata (body params &optional graphics-file)
  "Expand BODY according to PARAMS, return the expanded body."
  (let ((graphics-file
	 (or graphics-file (org-babel-stata-graphical-output-file params))))
    (mapconcat
     #'identity
     ((lambda (inside)
	(if graphics-file
            inside
	  inside))
      (append (org-babel-variable-assignments:stata params)
	      (list body))) "\n")))

(defun org-babel-execute:stata (body params)
  "Execute a block of stata code.
This function is called by `org-babel-execute-src-block'."
  (save-excursion
    (let* ((result-params (cdr (assoc :result-params params)))
	   (result-type (cdr (assoc :result-type params)))
           (session (org-babel-stata-initiate-session
		     (cdr (assoc :session params)) params))
	   (colnames-p (cdr (assoc :colnames params)))
	   (rownames-p (cdr (assoc :rownames params)))
	   (graphics-file (org-babel-stata-graphical-output-file params))
	   (full-body (org-babel-expand-body:stata body params graphics-file))
	   (result
	    (org-babel-stata-evaluate
	     session full-body result-type result-params
	     (or (equal "yes" colnames-p)
		 (org-babel-pick-name
		  (cdr (assoc :colname-names params)) colnames-p))
	     (or (equal "yes" rownames-p)
		 (org-babel-pick-name
		  (cdr (assoc :rowname-names params)) rownames-p)))))
      (if graphics-file nil result))))

(defun org-babel-prep-session:stata (session params)
  "Prepare SESSION according to the header arguments specified in PARAMS."
  (let* ((session (org-babel-stata-initiate-session session params))
	 (var-lines (org-babel-variable-assignments:stata params)))
    (org-babel-comint-in-buffer session
      (mapc (lambda (var)
              (end-of-line 1) (insert var) (comint-send-input nil t)
              (org-babel-comint-wait-for-output session)) var-lines))
    session))

(defun org-babel-load-session:stata (session body params)
  "Load BODY into SESSION."
  (save-window-excursion
    (let ((buffer (org-babel-prep-session:stata session params)))
      (with-current-buffer buffer
        (goto-char (process-mark (get-buffer-process (current-buffer))))
        (insert (org-babel-chomp body)))
      buffer)))

;; helper functions

(defun org-babel-variable-assignments:stata (params)
  "Return list of stata statements assigning the block's variables."
  (let ((vars (mapcar #'cdr (org-babel-get-header params :var))))
    (mapcar
     (lambda (pair)
       (org-babel-stata-assign-elisp
	(car pair) (cdr pair)
	(equal "yes" (cdr (assoc :colnames params)))
	(equal "yes" (cdr (assoc :rownames params)))))
     (mapcar
      (lambda (i)
	(cons (car (nth i vars))
	      (org-babel-reassemble-table
	       (cdr (nth i vars))
	       (cdr (nth i (cdr (assoc :colname-names params))))
	       (cdr (nth i (cdr (assoc :rowname-names params)))))))
      (org-number-sequence 0 (1- (length vars)))))))

(defun org-babel-stata-quote-csv-field (s)
  "Quote field S for export to stata."
  (if (stringp s)
      (concat "\"" (mapconcat 'identity (split-string s "\"") "\"\"") "\"")
    (format "%S" s)))

(defun org-babel-stata-assign-elisp (name value colnames-p rownames-p)
  "Construct stata code assigning the elisp VALUE to a variable named NAME."
  (if (listp value)
      (let ((max (apply #'max (mapcar #'length (org-remove-if-not
						#'sequencep value))))
	    (min (apply #'min (mapcar #'length (org-remove-if-not
						#'sequencep value))))
	    (transition-file (org-babel-temp-file "stata-import-")))
        ;; ensure VALUE has an orgtbl structure (depth of at least 2)
        (unless (listp (car value)) (setq value (list value)))
        (with-temp-file transition-file
          (insert
	   (orgtbl-to-csv value '(:fmt org-babel-stata-quote-csv-field))
	   "\n"))
	(let ((file (org-babel-process-file-name transition-file 'noquote))
	      (header (if (or (eq (nth 1 value) 'hline) colnames-p)
			  "TRUE" "FALSE"))
	      (row-names (if rownames-p "1" "NULL")))
	  (if (= max min)
	      (format "%s = insheet using \"%s\"" name file)
	    (format "%s = insheet using \"%s\""
		    name file))))
    (format "%s = %s" name (org-babel-stata-quote-csv-field value))))

(defvar ess-ask-for-ess-directory) ; dynamically scoped

(defun org-babel-stata-initiate-session (session params)
  "If there is not a current stata process then create one."
  (unless (string= session "none")
    (let ((session (or session "*stata*"))
	  (ess-ask-for-ess-directory
	   (and (and (boundp 'ess-ask-for-ess-directory) ess-ask-for-ess-directory)
		(not (cdr (assoc :dir params))))))
      (if (org-babel-comint-buffer-livep session)
	  session
	(save-window-excursion
	  (require 'ess) (stata)
	  (rename-buffer
	   (if (bufferp session)
	       (buffer-name session)
	     (if (stringp session)
		 session
	       (buffer-name))))
	  (current-buffer))))))

(defun org-babel-stata-associate-session (session)
  "Associate stata code buffer with a stata session.
Make SESSION be the inferior ESS process associated with the
current code buffer."
  (setq ess-local-process-name
	(process-name (get-buffer-process session)))
  (ess-make-buffer-current))

(defun org-babel-stata-graphical-output-file (params)
  "Name of file to which stata should send graphical output."
  (and (member "graphics" (cdr (assq :result-params params)))
       (cdr (assq :file params))))

(defvar org-babel-stata-eoe-indicator "display \"org_babel_stata_eoe\"")
(defvar org-babel-stata-eoe-output "org_babel_stata_eoe")

(defvar org-babel-stata-write-object-command "outsheet using \"%s\"")

;; The following was a very complicated write object command
;; The replacement needs to add error catching
;(defvar org-babel-stata-write-object-command "{function(object,transfer.file){object;invisible(if(inherits(try({tfile<-tempfile();write.table(object,file=tfile,sep=\"\\t\",na=\"nil\",row.names=%s,col.names=%s,quote=FALSE);file.rename(tfile,transfer.file)},silent=TRUE),\"try-error\")){if(!file.exists(transfer.file))file.create(transfer.file)})}}(object=%s,transfer.file=\"%s\")")

(defun org-babel-stata-evaluate
  (session body result-type result-params column-names-p row-names-p)
  "Evaluate stata code in BODY."
  (if session
      (org-babel-stata-evaluate-session
       session body result-type result-params column-names-p row-names-p)
    (org-babel-stata-evaluate-external-process
     body result-type result-params column-names-p row-names-p)))

(defun org-babel-stata-evaluate-external-process
  (body result-type result-params column-names-p row-names-p)
  "Evaluate BODY in external stata process.
If RESULT-TYPE equals 'output then return standard output as a
string.  If RESULT-TYPE equals 'value then return the value of the
last statement in BODY, as elisp."
  (case result-type
    (value
     (let ((tmp-file (org-babel-temp-file "stata-")))
       (org-babel-eval org-babel-stata-command
		       (format org-babel-stata-write-object-command
			       (org-babel-process-file-name tmp-file 'noquote)
			       (format "begin\n%s\nend" body)))
       (org-babel-stata-process-value-result
	(org-babel-result-cond result-params
	  (with-temp-buffer
	    (insert-file-contents tmp-file)
	    (buffer-string))
	  (org-babel-import-elisp-from-file tmp-file '(4)))
	column-names-p)))
    (output (org-babel-eval org-babel-stata-command body))))

(defun org-babel-stata-evaluate-session
  (session body result-type result-params column-names-p row-names-p)
  "Evaluate BODY in SESSION.
If RESULT-TYPE equals 'output then return standard output as a
string.  If RESULT-TYPE equals 'value then return the value of the
last statement in BODY, as elisp."
  (case result-type
    (value
     (with-temp-buffer
       (insert (org-babel-chomp body))
       (let ((ess-local-process-name
	      (process-name (get-buffer-process session)))
	     (ess-eval-visibly-p nil))
	 (ess-eval-buffer nil)))
     (let ((tmp-file (org-babel-temp-file "stata-")))
       (org-babel-comint-eval-invisibly-and-wait-for-file
	session tmp-file
	(format org-babel-stata-write-object-command
		(org-babel-process-file-name tmp-file 'noquote) "ans"))
       (org-babel-stata-process-value-result
	(org-babel-result-cond result-params
	  (with-temp-buffer
	    (insert-file-contents tmp-file)
	    (buffer-string))
	  (org-babel-import-elisp-from-file tmp-file '(4)))
	column-names-p)))
    (output
     (mapconcat
      #'org-babel-chomp
      (butlast
       (delq nil
	     (mapcar
	      (lambda (line) (when (> (length line) 0) line))
	      (mapcar
	       (lambda (line) ;; cleanup extra prompts left in output
		 (if (string-match
		      "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)" line)
		     (substring line (match-end 1))
		   line))
	       (org-babel-comint-with-output (session org-babel-stata-eoe-output)
		 (insert (mapconcat #'org-babel-chomp
				    (list body org-babel-stata-eoe-indicator)
				    "\n"))
		 (inferior-ess-send-input)))))) "\n"))))

(defun org-babel-stata-process-value-result (result column-names-p)
  "stata-specific processing of return value.
Insert hline if column names in output have been requested."
  (if column-names-p
      (cons (car result) (cons 'hline (cdr result)))
    result))

(provide 'ob-stata)

;;; ob-stata.el ends here

Re: Enabling another language in org-babel

[Bastien]

Hi Ista,

some comments on the code below.

Ista Zahn <istazahn@gmail.com> writes:

> ;; Copyright (C) 2014 Ista Zahn
> ;; Author: Ista Zahn, based on ob-julia.el by G. Jay Kerns, and ob-R.el
> ;; by Eric Schulte and Dan Davison

You need to reformat this -- see files with multiple authors in Org or
in Emacs.

> ;; This file is not part of GNU Emacs.

We plan to add this to Org's core, so this will be part of Emacs.

> ;; Stata and ESS are required.

Maybe add some links for the requirements.

> (defcustom org-babel-stata-command inferior-STA-program-name
>   "Name of command to use for executing stata code."
>   :group 'org-babel
>   :version "24.4"
>   :package-version '(Org . "8.0")
                              ^^^

This needs to be 8.3

> ;; The following was a very complicated write object command
> ;; The replacement needs to add error catching
> ;(defvar org-babel-stata-write-object-command "{function(object,transfer.file){object;invisible(if(inherits(try({tfile<-tempfile();write.table(object,file=tfile,sep=\"\\t\",na=\"nil\",row.names=%s,col.names=%s,quote=FALSE);file.rename(tfile,transfer.file)},silent=TRUE),\"try-error\")){if(!file.exists(transfer.file))file.create(transfer.file)})}}(object=%s,transfer.file=\"%s\")")

Maybe simply remove this, or make it more readable.

Thanks in advance for this contribution!

-- 
 Bastien

Re: Enabling another language in org-babel

[Ista Zahn]

[-- Attachment #1: Type: text/plain, Size: 2000 bytes --]

I've made the suggested changes, with the exception of the "part of
Emacs bit",  as this should go in contrib not core.

I asked Bastien privately about getting access to the git repository,
but will ask here as well: When I run 'git clone
orgmode@orgmode.org:org-mode.git' I get a response saying

"Permission denied (publickey).
fatal: Could not read from remote repository"

Does that mean my key wasn't properly added, or did I miss some required set-up?

Best,
Ista

On Tue, Jul 29, 2014 at 10:36 AM, Bastien <bzg@gnu.org> wrote:
> Hi Ista,
>
> some comments on the code below.
>
> Ista Zahn <istazahn@gmail.com> writes:
>
>> ;; Copyright (C) 2014 Ista Zahn
>> ;; Author: Ista Zahn, based on ob-julia.el by G. Jay Kerns, and ob-R.el
>> ;; by Eric Schulte and Dan Davison
>
> You need to reformat this -- see files with multiple authors in Org or
> in Emacs.
>
>> ;; This file is not part of GNU Emacs.
>
> We plan to add this to Org's core, so this will be part of Emacs.
>
>> ;; Stata and ESS are required.
>
> Maybe add some links for the requirements.
>
>> (defcustom org-babel-stata-command inferior-STA-program-name
>>   "Name of command to use for executing stata code."
>>   :group 'org-babel
>>   :version "24.4"
>>   :package-version '(Org . "8.0")
>                               ^^^
>
> This needs to be 8.3
>
>> ;; The following was a very complicated write object command
>> ;; The replacement needs to add error catching
>> ;(defvar org-babel-stata-write-object-command "{function(object,transfer.file){object;invisible(if(inherits(try({tfile<-tempfile();write.table(object,file=tfile,sep=\"\\t\",na=\"nil\",row.names=%s,col.names=%s,quote=FALSE);file.rename(tfile,transfer.file)},silent=TRUE),\"try-error\")){if(!file.exists(transfer.file))file.create(transfer.file)})}}(object=%s,transfer.file=\"%s\")")
>
> Maybe simply remove this, or make it more readable.
>
> Thanks in advance for this contribution!
>
> --
>  Bastien

[-- Attachment #2: ob-stata.el --]
[-- Type: text/x-emacs-lisp, Size: 11983 bytes --]

;;; ob-stata.el --- org-babel functions for stata code evaluation

;; Copyright (C) 2014 Ista Zahn
;; Author: Ista Zahn istazahn@gmail.com
;;      G. Jay Kerns
;;      Eric Schulte
;;      Dan Davison


;; This file is not part of GNU Emacs.

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation; either version 3, or (at your option)
;; any later version.
;;
;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs; see the file COPYING.  If not, write to the
;; Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
;; Boston, MA 02110-1301, USA.

;;; Commentary:

;; The file provides Org-Babel support for evaluating stata code.
;; It is basically result of find-and-replace "stata" for "julia"
;; in ob-julia.el by G. Jay Kerns. Only ":results output" works: the
;; header args must include ":results output" (this is the default).
;; Note that I'm not sure ':results value' makes sense or is useful 
;; but I have left all the value-processing stuff inherited from 
;; ob-julia and ob-R. ':results graphics' would be nice, but I have
;; not tried to implement it. 
;; --Ista, 07/30/2014

;;; Requirements:
;; Stata: http://stata.com
;; ESS: http://ess.r-project.org

;;; Code:
(require 'ob)
(eval-when-compile (require 'cl))

(declare-function orgtbl-to-csv "org-table" (table params))
(declare-function stata "ext:ess-stata" (&optional start-args))
(declare-function inferior-ess-send-input "ext:ess-inf" ())
(declare-function ess-make-buffer-current "ext:ess-inf" ())
(declare-function ess-eval-buffer "ext:ess-inf" (vis))
(declare-function org-number-sequence "org-compat" (from &optional to inc))
(declare-function org-remove-if-not "org" (predicate seq))

(defconst org-babel-header-args:stata
  '((width		 . :any)
    (horizontal		 . :any)
    (results             . ((file list vector table scalar verbatim)
			    (raw org html latex code pp wrap)
			    (replace silent append prepend)
                            ;; NOTE: not sure 'value' makes sense in stata
                            ;; we may want to remove it from the list
			    (output value graphics))))
  "stata-specific header arguments.")

(add-to-list 'org-babel-tangle-lang-exts '("stata" . "do"))

;; only ':results output' currently works, so make that the default
(defvar org-babel-default-header-args:stata '((:results . "output")))

(defcustom org-babel-stata-command inferior-STA-program-name
  "Name of command to use for executing stata code."
  :group 'org-babel
  :version "24.4"
  :package-version '(Org . "8.3")
  :type 'string)

(defvar ess-local-process-name) ; dynamically scoped
(defun org-babel-edit-prep:stata (info)
  (let ((session (cdr (assoc :session (nth 2 info)))))
    (when (and session (string-match "^\\*\\(.+?\\)\\*$" session))
      (save-match-data (org-babel-stata-initiate-session session nil)))))

(defun org-babel-expand-body:stata (body params &optional graphics-file)
  "Expand BODY according to PARAMS, return the expanded body."
  (let ((graphics-file
	 (or graphics-file (org-babel-stata-graphical-output-file params))))
    (mapconcat
     #'identity
     ((lambda (inside)
	(if graphics-file
            inside
	  inside))
      (append (org-babel-variable-assignments:stata params)
	      (list body))) "\n")))

(defun org-babel-execute:stata (body params)
  "Execute a block of stata code.
This function is called by `org-babel-execute-src-block'."
  (save-excursion
    (let* ((result-params (cdr (assoc :result-params params)))
	   (result-type (cdr (assoc :result-type params)))
           (session (org-babel-stata-initiate-session
		     (cdr (assoc :session params)) params))
	   (colnames-p (cdr (assoc :colnames params)))
	   (rownames-p (cdr (assoc :rownames params)))
	   (graphics-file (org-babel-stata-graphical-output-file params))
	   (full-body (org-babel-expand-body:stata body params graphics-file))
	   (result
	    (org-babel-stata-evaluate
	     session full-body result-type result-params
	     (or (equal "yes" colnames-p)
		 (org-babel-pick-name
		  (cdr (assoc :colname-names params)) colnames-p))
	     (or (equal "yes" rownames-p)
		 (org-babel-pick-name
		  (cdr (assoc :rowname-names params)) rownames-p)))))
      (if graphics-file nil result))))

(defun org-babel-prep-session:stata (session params)
  "Prepare SESSION according to the header arguments specified in PARAMS."
  (let* ((session (org-babel-stata-initiate-session session params))
	 (var-lines (org-babel-variable-assignments:stata params)))
    (org-babel-comint-in-buffer session
      (mapc (lambda (var)
              (end-of-line 1) (insert var) (comint-send-input nil t)
              (org-babel-comint-wait-for-output session)) var-lines))
    session))

(defun org-babel-load-session:stata (session body params)
  "Load BODY into SESSION."
  (save-window-excursion
    (let ((buffer (org-babel-prep-session:stata session params)))
      (with-current-buffer buffer
        (goto-char (process-mark (get-buffer-process (current-buffer))))
        (insert (org-babel-chomp body)))
      buffer)))

;; helper functions

(defun org-babel-variable-assignments:stata (params)
  "Return list of stata statements assigning the block's variables."
  (let ((vars (mapcar #'cdr (org-babel-get-header params :var))))
    (mapcar
     (lambda (pair)
       (org-babel-stata-assign-elisp
	(car pair) (cdr pair)
	(equal "yes" (cdr (assoc :colnames params)))
	(equal "yes" (cdr (assoc :rownames params)))))
     (mapcar
      (lambda (i)
	(cons (car (nth i vars))
	      (org-babel-reassemble-table
	       (cdr (nth i vars))
	       (cdr (nth i (cdr (assoc :colname-names params))))
	       (cdr (nth i (cdr (assoc :rowname-names params)))))))
      (org-number-sequence 0 (1- (length vars)))))))

(defun org-babel-stata-quote-csv-field (s)
  "Quote field S for export to stata."
  (if (stringp s)
      (concat "\"" (mapconcat 'identity (split-string s "\"") "\"\"") "\"")
    (format "%S" s)))

(defun org-babel-stata-assign-elisp (name value colnames-p rownames-p)
  "Construct stata code assigning the elisp VALUE to a variable named NAME."
  (if (listp value)
      (let ((max (apply #'max (mapcar #'length (org-remove-if-not
						#'sequencep value))))
	    (min (apply #'min (mapcar #'length (org-remove-if-not
						#'sequencep value))))
	    (transition-file (org-babel-temp-file "stata-import-")))
        ;; ensure VALUE has an orgtbl structure (depth of at least 2)
        (unless (listp (car value)) (setq value (list value)))
        (with-temp-file transition-file
          (insert
	   (orgtbl-to-csv value '(:fmt org-babel-stata-quote-csv-field))
	   "\n"))
	(let ((file (org-babel-process-file-name transition-file 'noquote))
	      (header (if (or (eq (nth 1 value) 'hline) colnames-p)
			  "TRUE" "FALSE"))
	      (row-names (if rownames-p "1" "NULL")))
	  (if (= max min)
	      (format "%s = insheet using \"%s\"" name file)
	    (format "%s = insheet using \"%s\""
		    name file))))
    (format "%s = %s" name (org-babel-stata-quote-csv-field value))))

(defvar ess-ask-for-ess-directory) ; dynamically scoped

(defun org-babel-stata-initiate-session (session params)
  "If there is not a current stata process then create one."
  (unless (string= session "none")
    (let ((session (or session "*stata*"))
	  (ess-ask-for-ess-directory
	   (and (and (boundp 'ess-ask-for-ess-directory) ess-ask-for-ess-directory)
		(not (cdr (assoc :dir params))))))
      (if (org-babel-comint-buffer-livep session)
	  session
	(save-window-excursion
	  (require 'ess) (stata)
	  (rename-buffer
	   (if (bufferp session)
	       (buffer-name session)
	     (if (stringp session)
		 session
	       (buffer-name))))
	  (current-buffer))))))

(defun org-babel-stata-associate-session (session)
  "Associate stata code buffer with a stata session.
Make SESSION be the inferior ESS process associated with the
current code buffer."
  (setq ess-local-process-name
	(process-name (get-buffer-process session)))
  (ess-make-buffer-current))

(defun org-babel-stata-graphical-output-file (params)
  "Name of file to which stata should send graphical output."
  (and (member "graphics" (cdr (assq :result-params params)))
       (cdr (assq :file params))))

(defvar org-babel-stata-eoe-indicator "display \"org_babel_stata_eoe\"")
(defvar org-babel-stata-eoe-output "org_babel_stata_eoe")

(defvar org-babel-stata-write-object-command "outsheet using \"%s\"")

(defun org-babel-stata-evaluate
  (session body result-type result-params column-names-p row-names-p)
  "Evaluate stata code in BODY."
  (if session
      (org-babel-stata-evaluate-session
       session body result-type result-params column-names-p row-names-p)
    (org-babel-stata-evaluate-external-process
     body result-type result-params column-names-p row-names-p)))

(defun org-babel-stata-evaluate-external-process
  (body result-type result-params column-names-p row-names-p)
  "Evaluate BODY in external stata process.
If RESULT-TYPE equals 'output then return standard output as a
string.  If RESULT-TYPE equals 'value then return the value of the
last statement in BODY, as elisp."
  (case result-type
    (value
     (let ((tmp-file (org-babel-temp-file "stata-")))
       (org-babel-eval org-babel-stata-command
		       (format org-babel-stata-write-object-command
			       (org-babel-process-file-name tmp-file 'noquote)
			       (format "begin\n%s\nend" body)))
       (org-babel-stata-process-value-result
	(org-babel-result-cond result-params
	  (with-temp-buffer
	    (insert-file-contents tmp-file)
	    (buffer-string))
	  (org-babel-import-elisp-from-file tmp-file '(4)))
	column-names-p)))
    (output (org-babel-eval org-babel-stata-command body))))

(defun org-babel-stata-evaluate-session
  (session body result-type result-params column-names-p row-names-p)
  "Evaluate BODY in SESSION.
If RESULT-TYPE equals 'output then return standard output as a
string.  If RESULT-TYPE equals 'value then return the value of the
last statement in BODY, as elisp."
  (case result-type
    (value
     (with-temp-buffer
       (insert (org-babel-chomp body))
       (let ((ess-local-process-name
	      (process-name (get-buffer-process session)))
	     (ess-eval-visibly-p nil))
	 (ess-eval-buffer nil)))
     (let ((tmp-file (org-babel-temp-file "stata-")))
       (org-babel-comint-eval-invisibly-and-wait-for-file
	session tmp-file
	(format org-babel-stata-write-object-command
		(org-babel-process-file-name tmp-file 'noquote) "ans"))
       (org-babel-stata-process-value-result
	(org-babel-result-cond result-params
	  (with-temp-buffer
	    (insert-file-contents tmp-file)
	    (buffer-string))
	  (org-babel-import-elisp-from-file tmp-file '(4)))
	column-names-p)))
    (output
     (mapconcat
      #'org-babel-chomp
      (butlast
       (delq nil
	     (mapcar
	      (lambda (line) (when (> (length line) 0) line))
	      (mapcar
	       (lambda (line) ;; cleanup extra prompts left in output
		 (if (string-match
		      "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)" line)
		     (substring line (match-end 1))
		   line))
	       (org-babel-comint-with-output (session org-babel-stata-eoe-output)
		 (insert (mapconcat #'org-babel-chomp
				    (list body org-babel-stata-eoe-indicator)
				    "\n"))
		 (inferior-ess-send-input)))))) "\n"))))

(defun org-babel-stata-process-value-result (result column-names-p)
  "stata-specific processing of return value.
Insert hline if column names in output have been requested."
  (if column-names-p
      (cons (car result) (cons 'hline (cdr result)))
    result))

(provide 'ob-stata)

;;; ob-stata.el ends here

Re: Enabling another language in org-babel

[Nick Dokos]

Ista Zahn <istazahn@gmail.com> writes:

> I asked Bastien privately about getting access to the git repository,
> but will ask here as well: When I run 'git clone
> orgmode@orgmode.org:org-mode.git' I get a response saying
>
> "Permission denied (publickey).
> fatal: Could not read from remote repository"
>
> Does that mean my key wasn't properly added, or did I miss some required set-up?
>

I don't think there is any set-up you need to do (and the command works
for me, so there does not seem to be anything wrong with the server);
I think you are right in suspecting that your key was not properly
added. Unfortunately, Bastien is going to be away for a while and I'm
not sure who has access in his absence.

Nick

Re: Enabling another language in org-babel

[Achim Gratz]

Ista Zahn writes:
> I've made the suggested changes, with the exception of the "part of
> Emacs bit",  as this should go in contrib not core.
>
> I asked Bastien privately about getting access to the git repository,
> but will ask here as well: When I run 'git clone
> orgmode@orgmode.org:org-mode.git' I get a response saying

Unrelated to your problem: you should not use the SSH connection for
pulling.  Configure the orgmode repo to use the normal git for reading,
then configure (in your global user configuration) to use the SSH
connection just for pushing:

--8<---------------cut here---------------start------------->8---
[url "git+ssh:orgmode@orgmode.org:org-mode.git"]
        pushInsteadOf = git://orgmode.org/org-mode.git
--8<---------------cut here---------------end--------------->8---

> "Permission denied (publickey).
> fatal: Could not read from remote repository"
>
> Does that mean my key wasn't properly added, or did I miss some required set-up?

Most likely, but it could also be that you haven't entered the correct
passphrase or are using the wrong key.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: Enabling another language in org-babel

[Jay Kerns]

Dear Ista,

I don't know if Bastien has already responded to your private message
(if so, please disregard this), but for what it's worth, it looks like
you are doing everything right to me.  I am not an org-mode developer
so I can't speak with 100% certainty, but it looks like a permissions
issue on the org-mode.org side (read: your key hasn't been added just
quite yet) and nothing on your side.

Good luck, and best wishes.
Jay



On 07/30/14 12:47, Ista Zahn wrote:
> I've made the suggested changes, with the exception of the "part of
> Emacs bit",  as this should go in contrib not core.
> 
> I asked Bastien privately about getting access to the git repository,
> but will ask here as well: When I run 'git clone
> orgmode@orgmode.org:org-mode.git' I get a response saying
> 
> "Permission denied (publickey).
> fatal: Could not read from remote repository"
> 
> Does that mean my key wasn't properly added, or did I miss some required set-up?
> 
> Best,
> Ista
> 
> On Tue, Jul 29, 2014 at 10:36 AM, Bastien <bzg@gnu.org> wrote:
>> Hi Ista,
>>
>> some comments on the code below.
>>
>> Ista Zahn <istazahn@gmail.com> writes:
>>
>>> ;; Copyright (C) 2014 Ista Zahn
>>> ;; Author: Ista Zahn, based on ob-julia.el by G. Jay Kerns, and ob-R.el
>>> ;; by Eric Schulte and Dan Davison
>>
>> You need to reformat this -- see files with multiple authors in Org or
>> in Emacs.
>>
>>> ;; This file is not part of GNU Emacs.
>>
>> We plan to add this to Org's core, so this will be part of Emacs.
>>
>>> ;; Stata and ESS are required.
>>
>> Maybe add some links for the requirements.
>>
>>> (defcustom org-babel-stata-command inferior-STA-program-name
>>>   "Name of command to use for executing stata code."
>>>   :group 'org-babel
>>>   :version "24.4"
>>>   :package-version '(Org . "8.0")
>>                               ^^^
>>
>> This needs to be 8.3
>>
>>> ;; The following was a very complicated write object command
>>> ;; The replacement needs to add error catching
>>> ;(defvar org-babel-stata-write-object-command "{function(object,transfer.file){object;invisible(if(inherits(try({tfile<-tempfile();write.table(object,file=tfile,sep=\"\\t\",na=\"nil\",row.names=%s,col.names=%s,quote=FALSE);file.rename(tfile,transfer.file)},silent=TRUE),\"try-error\")){if(!file.exists(transfer.file))file.create(transfer.file)})}}(object=%s,transfer.file=\"%s\")")
>>
>> Maybe simply remove this, or make it more readable.
>>
>> Thanks in advance for this contribution!
>>
>> --
>>  Bastien

Re: Enabling another language in org-babel

[Bastien]

Hi Ista,

Ista Zahn <istazahn@gmail.com> writes:

> I asked Bastien privately about getting access to the git repository,
> but will ask here as well: When I run 'git clone
> orgmode@orgmode.org:org-mode.git' I get a response saying
>
> "Permission denied (publickey).
> fatal: Could not read from remote repository"

This should be fixed now, let me know,

-- 
 Bastien

Re: Enabling another language in org-babel

[Ista Zahn]

On Thu, Jul 31, 2014 at 6:52 AM, Bastien <bzg@gnu.org> wrote:
> Hi Ista,
>
> Ista Zahn <istazahn@gmail.com> writes:
>
>> I asked Bastien privately about getting access to the git repository,
>> but will ask here as well: When I run 'git clone
>> orgmode@orgmode.org:org-mode.git' I get a response saying
>>
>> "Permission denied (publickey).
>> fatal: Could not read from remote repository"
>
> This should be fixed now, let me know,

Yes working now, thank you!

>
> --
>  Bastien

Re: Enabling another language in org-babel

[Esa Palosaari]

Ista Zahn <istazahn <at> gmail.com> writes:

> 
> I've made the suggested changes, with the exception of the "part of
> Emacs bit",  as this should go in contrib not core.


Hi,

thank you for the code! It would be very cool to have this working. 
For some reason, the it returns only the command (for example 
".display 2 + 2" and the stata shell start screen) instead 
of the results ("4"). I use Stata-SE 12.1. Any guesses what might 
be causing this?

Thanks!
Esa 

Re: Enabling another language in org-babel

[Esa Palosaari]

> Hi,
> 
> thank you for the code! It would be very cool to have this working. 
> For some reason, the it returns only the command (for example 
> ".display 2 + 2" and the stata shell start screen) instead 
> of the results ("4"). I use Stata-SE 12.1. Any guesses what might 
> be causing this?
> 
> Thanks!
> Esa 
> 


Sorry about bothering you. It's to be working ok. I just didn't
send a return. Great to have this!

Esa