[babel] Painless integration of source blocks with language

[babel] Painless integration of source blocks with language

[Seth Burleigh]

[-- Attachment #1.1: Type: text/plain, Size: 3070 bytes --]

Preface: I hope attachments show up, i dont know if they are allowed ....

Im currently interested in using babel for a medium size clojure project. I
think the below propositions would greatly benefit babel in accomplishing
literate programming.

First part of the proposal to make this painless:

In a literate document, you might very well have small chunks of code for
one ns scattered around and then finally combined using noweb into one file
and tangle the output. Currently, keys like
compile-file (C-c C-k), goto function definition (M-.), do not work at all
in the source blocks. This is unfortunate, and makes life very painful. So,
first we need to get these keys to work.
Heres how it can possibly be done.

Lets say we want to compile the code in block A. Block B uses noweb syntax
to import block A, and then tangles to file src/B.clj. So we need to search
for  the presence of <<A>> in a tangled code chunk , tangle the chunk to a
file, go to the file buffer and position the cursor at the correct relative
position, and then invoke the appropriate function (in this case,
slime-compile-and-load-file).

Ive attached two files which do this, the first one is an example org file
in which you can use the compile keystrokes on (after opening a slime server
for clojure). However, it doesnt position the cursor in the tangled file
correctly, but this is not needed for compile/load file. Its not a very good
implementation, but it works for this case. Also, it doesn't work when the
indirect buffer created by C-c '  is open. The advantage of this method is
that it can easily be generalized to any language, since the compiler gets
what it expects - a file of code to operate on.

Second part of proposal:

Literate documents are good for documentation, but if you're the author, you
dont need the documentation and it will certainly get in the way of you
writing code to keep having to type C-c ' to open various chunks of code.
Ideally, we would like to have our file of code (that is, the tangled file
output for one ns in clojure) as the top buffer, and our documentation at
the bottom. Changes to the code will automatically reflect itself in the
documentation (after a save). In order to accomplish this, there has to be a
method to map from tangled file line number, to the correct chunk name and
line number in the .org file. This might be accomplished by tangling code
like this:
;;#chunk-name
..code
;;#

I think this would be much more natural than editing each chunk separately.
Of course, the ;;# might become an annoyance, but im sure there might  be a
better way.
With this mapping , we can develop the code until we get it right. Then we
can hit a keystroke and have our documentation jump to the correct line
number in the .org file, and then do all the documentation for that.

With the line mapping ability also comes the ability to map errors in the
line number to numbers in the org file. Although, this probably wouldn't be
necessary, since we would be looking at the combined chunk file.


Any thoughts/ideas how to implement this?

[-- Attachment #1.2: Type: text/html, Size: 3598 bytes --]

[-- Attachment #2: test.org --]
[-- Type: application/octet-stream, Size: 730 bytes --]

#+results:silent
#+noweb:yes 

* Functions
** Add
This is a function which adds numbers together 
#+srcname:add
#+begin_src clojure
  (defn add [a b]
    (+ a b))
#+end_src
*** TODO make work with complex numbers
** Subtract
This subtracts numbers
#+srcname:subtract
#+begin_src clojure
  (defn subtract [a b]
    (- a b))
#+end_src
*** TODO complex numbers
** Multiply
#+srcname:multiply
#+begin_src clojure
  (defn multiply [a b]
    (* a b))
#+end_src
*** TODO complex numbers
** Divide
#+srcname:divide
#+begin_src clojure
  (defn divide [a b]   
    (/ a b))
#+end_src
** All 
#+srcname:all
#+begin_src clojure :tangle src/functions.clj
  (ns functions) 
  <<add>> 
  <<subtract>>
  <<multiply>>  
  <<divide>>
#+end_src
 



[-- Attachment #3: seth-obj-clj.el --]
[-- Type: application/octet-stream, Size: 5760 bytes --]

(require 'cl)

;;UTILS
(defun char (i str)
  "get char at index"
  (substring str i (+ i 1)))
;;cant get multiline regex to work for some reason. grr
(defun string-trim-right (test-str)
  "trim all right hand whitespace,newlines"
  (let ((place (- (length test-str) 1)) (continue t))
    (while continue
      (let ((c (char place test-str))) 
	(if (or (equalp c "\n")
		(equalp c "\r")
		(equalp c " "))
	    (decf place)
	  (progn (setq continue nil) (incf place))) 
	(if (= place 0) (setq continue nil))))
    (substring test-str 0 place)))
(defmacro with-file (file &rest body)
  "open up file in a buffer, and set current buffer to it"
  `(with-current-buffer (find-file-noselect ,file t) ,@body))
(defun info (&rest args)
  "message which returns nil"
  (apply #'message args)
  nil)
(defmacro comment (&rest body) nil)
(defun mkdir (file)
  "given file name, make a directory if needed"
   (let ((dir (file-name-directory file-name)))
     (if dir (make-directory dir t))))
;;;

(defun ob-expand (&optional arg info params)
  "Expand the current source code block.
Expand according to the source code block's header
arguments and pop open the results in a preview buffer.
Trims whitespaces at the very right of text"
  (interactive) 
  (let* ((info (or info (org-babel-get-src-block-info)))
         (lang (nth 0 info))
	 (params (setf (nth 2 info)
                       (sort (org-babel-merge-params (nth 2 info) params)
                             (lambda (el1 el2) (string< (symbol-name (car el1))
                                                   (symbol-name (car el2)))))))
         (body (setf (nth 1 info)
		     (if (and (cdr (assoc :noweb params))
                              (string= "yes" (cdr (assoc :noweb params))))
			 (org-babel-expand-noweb-references info) (nth 1 info))))
         (cmd (intern (concat "org-babel-expand-body:" lang)))
         (expanded (funcall (if (fboundp cmd) cmd 'org-babel-expand-body:generic)
                            body params)))
    (string-trim-right expanded)))

;;TODO: more efficient
(defun ob-name ()
  (nth 4 (ob-info)))
(defun ob-info ()
  "return info, making sure to trim right hand whitespaces. The reason to trim is so that , in the indirect macro, we can search for the beginning and end
of the chunk. i was getting some problems with newlines at the very end messing things up - so i trimmed them away. im sure theres a better way."
  (let ((info (org-babel-get-src-block-info)))
    (if info `(,(first info) ,(string-trim-right (second info)) ,@(cddr info)))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defun tangle-chunk (tag)
  "find code block with the tag: if found, tangle it to file"
  (let* ((chunk (find-chunk tag))
	 (file-name (first chunk)))
    (when chunk
      (if (file-exists-p file-name) 
	  (delete-file file-name))
      (mkdir file-name)
      (with-temp-buffer
	(insert (second chunk))
	;; We avoid append-to-file as it does not work with tramp.
	(let ((content (buffer-string)))
	  (with-temp-buffer
	    (goto-char (point-max))
	    (insert content)
	    (write-region nil nil file-name))))
      chunk)))

(defun find-chunk (tag)
  "find the src block with the appropriate tag (i.e. srcname)"
  (when tag
    (flet ((chunk! 
	    (tag prev)
	    (save-excursion
	      (goto-char (point-min))
	      (let (chunks)
		(if (re-search-forward (format "<<[ \t]*%s[ \t]*>>" tag) nil t)
		    (let ((info (ob-info)))
		      (if (member info prev)
			  (message "when finding chunk, infinite loop detected")
			(if (and (equalp (first info) "clojure")
				 info (not (equalp "no"
						   (cdr (assoc :tangle (third info))))))
			    (list (cdr (assoc :tangle (third info)))
				  (ob-expand (second info))
				  (point))
			  (chunk! (ob-name) (append prev (list info)))))))))))
      (let ((info (ob-info)))
	(if (not (equalp "no" (cdr (assoc :tangle (third info)))))
	    (list (cdr (assoc :tangle (third info)))
		  (ob-expand (second info))
		  (point))	  
	  (chunk! tag nil))))))



(defun indirect-helper (operation type)
  "Given the operation and language type, return a list of the info for the tangled chunk, the tangled chunk, and the original untangled source"
  (let ((info (ob-info)))
    (when (equalp (first info) type)
      (let ((srcname (ob-name))
	    (src-untangle (ob-expand)))
	(if srcname
	    (let ((chunk (tangle-chunk srcname))) 
	      (if chunk
		  (list info chunk src-untangle)
		(info "cant %s - chunk not associated with file" operation) nil))
	  (info "cant %s - no srcname" operation))))))

(defmacro* indirect ((type operation chunk  start end) &rest body)
  (let ((lst-gen (gensym)) (orig-str (gensym)))
    `(let* ((,lst-gen (indirect-helper ,operation ,type))
	    (,orig-str (second (cdr ,lst-gen)))
	    (,chunk (second ,lst-gen)))
       (if ,chunk
	   (with-file (first ,chunk)
		      (save-excursion (goto-char (point-min)) (search-forward  ,orig-str))
		      (let ((,start (match-beginning 0)) (,end (match-end 0)))
			(progn ,@body)))))))
(defun ob-clj-compile ()
  (interactive) 
  (indirect ("clojure" "compile" chunk start end)
	    (slime-compile-and-load-file)))

(defun ob-clj-load ()
  (interactive)
  (indirect ("clojure" "compile" chunk start end)
	    (slime-load-file (buffer-file-name))))

(comment
 (defun obj-clj-slime-edit-definition () 
   (interactive)
   (indirect ("clojure" "edit definition" chunk start end offset) ;;once i can get the offset of the cursor within the original src chunk
	     (goto-char offset)
	     (slime-edit-definition))))

;;TODO: if block is in language x, use the bindings for that language automatically
(define-key org-mode-map (kbd "C-c C-k") 'ob-clj-compile)
(define-key org-mode-map (kbd "C-c C-l") 'ob-clj-load)

[-- Attachment #4: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] Painless integration of source blocks with language

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 1302 bytes --]

Hi Seth,

Thanks for the thoughtful comments.  I especially like the method of
literate programming described in your second proposal.  Over the last
months I have switched from working mainly in code blocks to working
mainly in pure source files due to issues along the lines of those
mentioned in your first proposal.

It seems to me that working in two frames as you suggest -- e.g. writing
code in the pure code buffer, documentation in the Org-mode buffer, and
maintaining a constant mapping between the two -- would resolve the
issues mentioned in your first, and should be sufficient.  With some
elisp code it should be possible to support no-web in such a scheme
allowing the code buffer to be automatically re-organized based on
changes to the .org buffer.

A crude version of the above is already possible using the
`org-babel-detangle' function.  For example, follow the instructions in
the attached org-mode file (which uses elisp rather than clojure code
blocks simply for wider portability to non-clojure users).

Ultimately I think that more code support (possibly implemented using a
minor mode) allowing things like easy navigation, interactive
tangling/arrangement, noweb support, and (possibly) removing the need
for comment markers would be the preferred solution.

Cheers -- Eric


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: detangle.org --]
[-- Type: text/org, Size: 1514 bytes --]

#+Title: Detangle Example
#+Babel: :comments link :tangle detangle.el

This file demonstrates detangling of source files with Org-mode.

1. Evaluate this elisp code block to customize the behavior of link
   creation on your system -- in the future this could be folded into
   ob-tangle.
   #+begin_src emacs-lisp :results silent :tangle no
     (setq org-link-to-org-use-id nil)
   #+end_src
2. Tangle out this file by calling `[[elisp:(org-babel-tangle)]]' bound to
   =C-c C-v t=, notice the buffer-wide =:comments= and =:tangle=
   header arguments at the top of the file.
3. Open up the [[file:detangle.el][detangle.el]] file to find the tangled source code.
   Notice the comments which are used to associate parts of the source
   file with this org file.
4. Call the `org-babel-tangle-jump-to-org' function from one of the
   =message= lines in the elisp file to jump back to the related
   portion of this org file.
5. Navigate back to [[file:detangle.el][detangle.el]] and edit part of the elisp code
   (e.g. change the text of one of the messages), then call the
   `org-babel-detangle' function from inside of the elisp code
   buffer.  Notice that your edits have now been propagated back to
   the original Org-mode buffer.

* first
A first section.

#+source: a-named-block
#+begin_src emacs-lisp
  (message "this is the contents of the first code block.")
#+end_src

* second
A second section.

#+begin_src emacs-lisp
  (message "this block has no name -- it is the second code block")
#+end_src

[-- Attachment #3: Type: text/plain, Size: 3480 bytes --]


Seth Burleigh <wburle@gmail.com> writes:

> Preface: I hope attachments show up, i dont know if they are allowed ....
>

The attachments did make it through.

>
> Im currently interested in using babel for a medium size clojure project. I
> think the below propositions would greatly benefit babel in accomplishing
> literate programming.
>
> First part of the proposal to make this painless:
>
> In a literate document, you might very well have small chunks of code for
> one ns scattered around and then finally combined using noweb into one file
> and tangle the output. Currently, keys like
> compile-file (C-c C-k), goto function definition (M-.), do not work at all
> in the source blocks. This is unfortunate, and makes life very painful. So,
> first we need to get these keys to work.
> Heres how it can possibly be done.
>
> Lets say we want to compile the code in block A. Block B uses noweb syntax
> to import block A, and then tangles to file src/B.clj. So we need to search
> for  the presence of <<A>> in a tangled code chunk , tangle the chunk to a
> file, go to the file buffer and position the cursor at the correct relative
> position, and then invoke the appropriate function (in this case,
> slime-compile-and-load-file).
>
> Ive attached two files which do this, the first one is an example org file
> in which you can use the compile keystrokes on (after opening a slime server
> for clojure). However, it doesnt position the cursor in the tangled file
> correctly, but this is not needed for compile/load file. Its not a very good
> implementation, but it works for this case. Also, it doesn't work when the
> indirect buffer created by C-c '  is open. The advantage of this method is
> that it can easily be generalized to any language, since the compiler gets
> what it expects - a file of code to operate on.
>
> Second part of proposal:
>
> Literate documents are good for documentation, but if you're the author, you
> dont need the documentation and it will certainly get in the way of you
> writing code to keep having to type C-c ' to open various chunks of code.
> Ideally, we would like to have our file of code (that is, the tangled file
> output for one ns in clojure) as the top buffer, and our documentation at
> the bottom. Changes to the code will automatically reflect itself in the
> documentation (after a save). In order to accomplish this, there has to be a
> method to map from tangled file line number, to the correct chunk name and
> line number in the .org file. This might be accomplished by tangling code
> like this:
> ;;#chunk-name
> ..code
> ;;#
>
> I think this would be much more natural than editing each chunk separately.
> Of course, the ;;# might become an annoyance, but im sure there might  be a
> better way.
> With this mapping , we can develop the code until we get it right. Then we
> can hit a keystroke and have our documentation jump to the correct line
> number in the .org file, and then do all the documentation for that.
>
> With the line mapping ability also comes the ability to map errors in the
> line number to numbers in the org file. Although, this probably wouldn't be
> necessary, since we would be looking at the combined chunk file.
>
>
> Any thoughts/ideas how to implement this?
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

[-- Attachment #4: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] Painless integration of source blocks with language

[Štěpán Němec]

"Eric Schulte" <schulte.eric@gmail.com> writes:

> Hi Seth,
>
> Thanks for the thoughtful comments.  I especially like the method of
> literate programming described in your second proposal.  Over the last
> months I have switched from working mainly in code blocks to working
> mainly in pure source files due to issues along the lines of those
> mentioned in your first proposal.

BTW, why is there not an option that would enable all mode-specific key
bindings inside source blocks? It could be implemented e.g. by putting
the mode keymap into the `keymap' text property. Seems like a simple
solution.

  Štěpán

Re: [babel] Painless integration of source blocks with language

[Eric Schulte]

Štěpán Němec <stepnem@gmail.com> writes:

> "Eric Schulte" <schulte.eric@gmail.com> writes:
>
>> Hi Seth,
>>
>> Thanks for the thoughtful comments.  I especially like the method of
>> literate programming described in your second proposal.  Over the last
>> months I have switched from working mainly in code blocks to working
>> mainly in pure source files due to issues along the lines of those
>> mentioned in your first proposal.
>
> BTW, why is there not an option that would enable all mode-specific key
> bindings inside source blocks? It could be implemented e.g. by putting
> the mode keymap into the `keymap' text property. Seems like a simple
> solution.
>

That's a very good idea, I didn't realize that keymaps where handled
through text properties, if/when I find some time I'll take a look at
making this change (although if anyone else feels like taking this on
please don't let me discourage you :)).  I imagine this could be done in
org-src, and tucked behind a `org-src-keybind-natively' (or somesuch)
configuration variable.

Thanks for the suggestion! -- Eric

>
>   Štěpán

Re: Re: [babel] Painless integration of source blocks with language

[Seth Burleigh]

[-- Attachment #1.1: Type: text/plain, Size: 786 bytes --]

As an update, ive been working on something i call chunks.
Basically, they are blocks of code (i.e. emacs overlays) that are linked
together.
So far, i have each ns of my clojure code in one source block which is
then tangled to one file.
So, i would like to open the tangled file and then make changes, and finally
'push' those changes to the org file.

Theres some bugs (pushing when mark is next to a parantheses), but i think
it is going in the correct way to also include noweb tangling. In the
attached code, you hit f8 in a source block to link the block to its file
and then f8 if you want to unlink it, and you hit ctrl-alt-p to push changes
from source file to org file.

just execute lp.el in an ielm buffer. and try it out with the previous test
org file that was attached.

[-- Attachment #1.2: Type: text/html, Size: 890 bytes --]

[-- Attachment #2: lp.el --]
[-- Type: application/octet-stream, Size: 8796 bytes --]

(require 'cl)


;;UTILS
(defun inform (operation arg)
  (if arg
      (progn (message "%s success!" operation) arg)
    (info "%s failed!" operation)))
(defun char (i str)
  "get char at index"
  (substring str i (+ i 1)))
;;cant get multiline regex to work for some reason. grr
(defun string-trim-right (test-str)
  "trim all right hand whitespace,newlines"
  (let ((place (- (length test-str) 1)) (continue t))
    (while continue
      (let ((c (char place test-str)))
	(if (or (equalp c "\n")
		(equalp c "\r")
		(equalp c " "))
	    (decf place)
	  (progn (setq continue nil) (incf place))) 
	(if (= place 0) (setq continue nil))))
    (substring test-str 0 place)))
(defmacro with-file (file &rest body)
  "open up file in a buffer, and set current buffer to it"
  `(with-current-buffer (find-file-noselect ,file t) ,@body))
(defun info (&rest args)
  "message which returns nil"
  (apply #'message args)
  nil)
(defmacro comment (&rest body) nil)
(defun mkdir (file)
  "given file name, make a directory if needed"
   (let ((dir (file-name-directory file-name)))
     (if dir (make-directory dir t))))
;;;

(defun ob-expand (&optional arg info params)
  "Expand the current source code block.
Expand according to the source code block's header
arguments and pop open the results in a preview buffer.
Trims whitespaces at the very right of text"
  (interactive) 
  (let* ((info (or info (org-babel-get-src-block-info)))
         (lang (nth 0 info))
	 (params (setf (nth 2 info)
                       (sort (org-babel-merge-params (nth 2 info) params)
                             (lambda (el1 el2) (string< (symbol-name (car el1))
                                                   (symbol-name (car el2)))))))
         (body (setf (nth 1 info)
		     (if (and (cdr (assoc :noweb params))
                              (string= "yes" (cdr (assoc :noweb params))))
			 (org-babel-expand-noweb-references info) (nth 1 info))))
         (cmd (intern (concat "org-babel-expand-body:" lang)))
         (expanded (funcall (if (fboundp cmd) cmd 'org-babel-expand-body:generic)
                            body params)))
    (string-trim-right expanded)))

;;TODO: more efficient
(defun ob-name ()
  (nth 4 (ob-info)))
(defun ob-info ()
  "return info, making sure to trim right hand whitespaces. The reason to trim is so that , in the indirect macro, we can search for the beginning and end
of the chunk. i was getting some problems with newlines at the very end messing things up - so i trimmed them away. im sure theres a better way."
  (let ((info (org-babel-get-src-block-info)))
    (if info `(,(first info) ,(string-trim-right (second info)) ,@(cddr info)))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defun tangle-chunk (tag)
  "find code block with the tag: if found, tangle it to file"
  (let* ((chunk (find-chunk tag))
	 (file-name (first chunk)))
    (when chunk
      (if (file-exists-p file-name) 
	  (delete-file file-name))
      (mkdir file-name)
      (with-temp-buffer
	(insert (second chunk))
	;; We avoid append-to-file as it does not work with tramp.
	(let ((content (buffer-string)))
	  (with-temp-buffer
	    (goto-char (point-max))
	    (insert content)
	    (write-region nil nil file-name))))
      chunk)))

(defun find-chunk (tag)
  "find the src block with the appropriate tag (i.e. srcname)"
  (when tag
    (flet ((chunk! 
	    (tag prev)
	    (save-excursion
	      (goto-char (point-min))
	      (let (chunks)
		(if (re-search-forward (format "<<[ \t]*%s[ \t]*>>" tag) nil t)
		    (let ((info (ob-info)))
		      (if (member info prev)
			  (message "when finding chunk, infinite loop detected")
			(if (and (equalp (first info) "clojure")
				 info (not (equalp "no"
						   (cdr (assoc :tangle (third info))))))
			    (list (cdr (assoc :tangle (third info)))
				  (ob-expand (second info))
				  (point))
			  (chunk! (ob-name) (append prev (list info)))))))))))
      (let ((info (ob-info)))
	(if (not (equalp "no" (cdr (assoc :tangle (third info)))))
	    (list (cdr (assoc :tangle (third info)))
		  (ob-expand (second info))
		  (point))	  
	  (chunk! tag nil))))))

;;TODO: if youre at end of buffer and the chunk is at the end of the buffer, the chunk will grow - how to inserta  new line?
;;TODO: cursor is set to beginning of chunk in synced buffer. fix?
(defmacro comment (&rest body))
(defun set-color (chunk color)
  (overlay-put chunk 'face (list :background color)))
(defvar lp-default-color "gray94")
(defun* new-chunk (start end buffer &optional (color lp-default-color))
  (interactive)
  (let ((chunk (make-overlay start end buffer t t)))
    (set-color chunk (if color color lp-default-color))
    ;;(overlay-put chunk 'modification-hooks '(mod-hook))
    chunk))

(defun link (from to) 
  (pushnew to (overlay-get from 'lp-link)))

(defun bi-link (node1 node2)
  (link node1 node2)
  (link node2 node1))

(defun chunks (o) (overlay-get o 'lp-link))
(defun overlay-string (overlay)
  (save-excursion
    (set-buffer (overlay-buffer overlay))
    (buffer-substring-no-properties
     (overlay-start overlay)
     (overlay-end overlay))))

;;TODO: instead of replacing entire region, replace only part that has changed
;;TODO: remove orphoned chunks?
(defun chunk? (chunk) (and (overlayp chunk) (overlay-buffer chunk)))
(defun push-string (str to &optional length)
  (when (chunk? to)
    (let ((start (overlay-start to))
	  (end (overlay-end to)))
      (save-excursion
	(set-buffer (overlay-buffer to))
	(delete-region start end)
	(goto-char start)
	(insert str)
	(move-overlay to start (+ start (length str)))))))

(defun push-chunk (from to &optional length)
  (push-string (overlay-string from) to))

(defun delete-chunks (c)
  (if (listp c)
      (mapcar #'delete-chunk c)
    (when (chunk? c) (delete-overlay c))))

;;after modification hook
(defun mod-hook (from type start end &optional length)
  (when type
    (dolist (to (chunks from))
      (push-chunk from to))))
(defun get-chunk ()
  (first (overlays-in (point-min) (point-max))))
(defun get-chunks ()
  (overlays-in (point-min) (point-max)))

;;src block chunks
(defun new-src-block-chunk (&optional color)
  (save-excursion
    (when (org-babel-mark-block)
      (new-chunk (point) (mark) (current-buffer)))))
(defun get-src-block-chunk ()
  (save-excursion
    (when (org-babel-mark-block) 
      (first (overlays-in (point) (mark))))))
(defun get-src-block-chunk-create (&optional color)
  (let ((chunk (get-src-block-chunk)))
    (if chunk chunk (new-src-block-chunk color))))
(defun delete-src-block-chunk ()
  (delete-chunks (get-src-block-chunk)))

;;file chunks
(defun new-src-block-file-chunk (&optional color)
  "get tangled file associated with chunk, open it, and create a chunk out of the whole file"
  (let ((file-name (first (find-chunk (ob-name)))))
    (if file-name
      (with-file
       file-name
       (save-excursion
	 (new-chunk (point-min) (point-max) (current-buffer) color)))
      (info "couldn't do it"))))
(defun get-src-block-file-chunk ()
  (let ((file-name (first (find-chunk (ob-name)))))
    (when file-name 
      (with-file
       file-name
       (get-chunk)))))
(defun get-src-block-file-chunk-create (&optional color)
  (let ((chunk (get-src-block-file-chunk)))
    (if chunk chunk (new-src-block-file-chunk color))))
(defun delete-src-block-file-chunk ()
  (delete-chunks (get-src-block-file-chunk)))

;;linking
(defun unlink-src-block ()
  (interactive)
  (delete-src-block-file-chunk)
  (delete-src-block-chunk))

(defun pull-src-block ()
  (interactive)
  (let ((chunk (get-src-block-chunk))
	(file-chunk (get-src-block-file-chunk)))
    (if (and chunk file-chunk)
      (inform "pull" (push-chunk file-chunk chunk))
      (info "only found chunk %s and file-chunk %s" chunk file-chunk))))



(defun bi-link-src-block (&optional color)
  (interactive)
  (let ((chunk (get-src-block-chunk-create)))
    (if chunk
      (let ((file-chunk (get-src-block-file-chunk-create (or color "white"))))
	(if file-chunk
	  (bi-link chunk file-chunk)
	  (progn (info "couldn't chunkify file") (unlink-src-block))))
      (info "couldn't chunkify src block"))))

(defun linked? ()
  (get-src-block-chunk))

(defun push-src-block ()
  (interactive)
  (let ((source (first (chunks (get-chunk)))))
    (if source
	(inform "push"
		(push-string
		 (buffer-substring-no-properties (point-min) (point-max))
		 source))
      (message "chunks are only %s" source))))

;;TODO; for some reason, when f8 is by a paren, dont work
(define-key org-mode-map (kbd "<f8>")
  (lambda ()
    (interactive)
    (if (linked?)
	(unlink-src-block)
      (bi-link-src-block))))
;;TODO: minor mode
(global-set-key
 (kbd "M-P")
 (lambda ()
   (interactive) 
   (if (linked?) (pull-src-block) (push-src-block))))
 

   

[-- Attachment #3: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: [babel] Painless integration of source blocks with language

[Eric Schulte]

Hi Seth,

I looked briefly at your code and much of it re-implements functionality
already provided by org-mode using a slight variation of Org-mode
syntax.  While I fully understand that it is often easier to write your
own functions rather than look up and parse existing functions, I think
that your code will stand a much better chance of re-use if it is works
with existing org-mode syntax rather than against it.  A couple of
examples below [1].  It would probably be worthwhile to read the "Source
Code" section of the Org-mode manual at [2].

I look forward to understanding the concepts behind your example, but
the above issues make this difficult.

Best -- Eric

Seth Burleigh <wburle4@gmail.com> writes:

> As an update, ive been working on something i call chunks.  Basically,
> they are blocks of code (i.e. emacs overlays) that are linked
> together.  So far, i have each ns of my clojure code in one source
> block which is then tangled to one file.  So, i would like to open the
> tangled file and then make changes, and finally 'push' those changes
> to the org file.
>
> Theres some bugs (pushing when mark is next to a parantheses), but i think
> it is going in the correct way to also include noweb tangling. In the
> attached code, you hit f8 in a source block to link the block to its file
> and then f8 if you want to unlink it, and you hit ctrl-alt-p to push changes
> from source file to org file.
>
> just execute lp.el in an ielm buffer. and try it out with the previous test
> org file that was attached.

Footnotes: 
[1]  a couple of examples...

       #+results:silent
       #+noweb:yes 

     should be

       #+Babel: :noweb yes :results silent

     also,

       #+srcname:add

     should be

       #+srcname: add

     once that syntax is fixed, then `org-babel-find-named-block' can be
     used instead of `find-chunk'.  Also, your `tangle-chunk' function
     duplicates the functionality of `org-babel-tangle'.

[2]  http://orgmode.org/manual/Working-With-Source-Code.html

Re: Re: [babel] Painless integration of source blocks with language

[Seth Burleigh]

[-- Attachment #1.1: Type: text/plain, Size: 2726 bytes --]

My bad. I believe i did look up the functions, but they didn't do what i
needed (or so i thought).
The code doesn't matter too much, let me explain the idea.

A file may contain many blocks of code. Lets look at a arbitrary block A. In
the end, block A will somehow become noweb embedded into a source block
which is actually tangled to a file. Lets call this source block C.
find-chunk finds this source block, given the position of block A, checking
also that the source has a type of "clojure". So, basically it searches for
the string <<A>>, if it finds it, checks if the source block (with an
arbitrary name, lets say my-block) is tangled, if not, it searches for
<<my-block>> until it actually finds a tangled source block.

tangle-chunk is used to tangle block A by finding block C (using find-chunk)
and tangling that. It is basically copied from org-babel except that it uses
mkdir to create any parent directories that do not exist (i think this
should be included as an option in tangle file, btw, if it isn't)

So, conceptually we have many blocks which are noweb embedded into various
source files, however indirectly. We want to find the source files that
these blocks are embedded in so that we can (in the future) navigate from
the block to the file number.
 '
Thats all that does, from line 110 up. I copied it from my previous posting.
From below that, i implement an emacs overlay, created by new-chunk.
Basically all it does is associate a chunk of code in the buffer to a chunk
of code in another chunk. We can 'add a link', i.e. append a chunk to its
'lp-link property.

Next, we add the ability to replace the code of one chunk with another. This
is what push-chunk does. Now, all that we have left is the code to create
chunks from source blocks.

Pushing code from one chunk to another code work with noweb embedding, but
it was just a proof of concept anyways.

What i was imagining when i did this is that i would have a source code file
which has various 'chunks' highlighted in different colors (various
background colors), each having a 'link' to the appropriate source code
chunk in the babel file. Thus, we could push changes from the source code
files to the org-mode file and navigate between. This would eliminate the
code markers. I was imagining that the tangled source code files could be
viewed simply as an indirect buffer created from the org file, so we
wouldn't need code markers to save the chunk info for future sessions - we
would just generate it all from the org-file.

 What is needed now is to be able to push from one chunk to another, except
leaving any <<srcname>> blocks as they are.  With that, i believe the ideas
mentioned in the first post could be fully implemented.

[-- Attachment #1.2: Type: text/html, Size: 3103 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: [babel] Painless integration of source blocks with language

[Seth Burleigh]

[-- Attachment #1.1: Type: text/plain, Size: 108 bytes --]

and i misspoke also, the previous .org file wouldnt work since that uses
noweb. Heres a simple new example.

[-- Attachment #1.2: Type: text/html, Size: 123 bytes --]

[-- Attachment #2: test.org --]
[-- Type: application/octet-stream, Size: 555 bytes --]

#+Babel: :noweb yes :results silent
* Instructions
evaluate prevous .el file in ielm buffer. tangle output. hit f8 somewhere in the
source code below. open up out.clj and then type something and hit alt-ctrl-p to push
the buffer string to the blue chunk. This is a proof of concept that we can link two chunks
together and push contents from one to another, in the future, allowing hopefully
navigation. Noweb support is needed. 
* Functions
this is documentation
#+srcname:add
#+begin_src clojure :tangle out.clj

(defn add [a b]
  (+ a b))

#+end_src



[-- Attachment #3: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] Painless integration of source blocks with language

[Eric S Fraga]

"Eric Schulte" <schulte.eric@gmail.com> writes:

[...]

> A crude version of the above is already possible using the
> `org-babel-detangle' function.  For example, follow the instructions in
> the attached org-mode file (which uses elisp rather than clojure code
> blocks simply for wider portability to non-clojure users).

Detangling, as currently implemented, doesn't do the job for me as it
doesn't understand noweb.  My current mode of operation with org and
babel is to have various snippets of code throughout a file and then
combine these in different ways using noweb syntax which I then tangle
to create different source files (each bringing together different
pieces in different configurations).

In any case, and please excuse me for hijacking this thread a little,
the increasing use of babel (a good thing!) especially with noweb syntax
and tangling (as this thread is about) is bringing up a document
management issue: I find it difficult (a) to remember what all my source
code snippets are called and (b) to navigate to any given snippet.  I
would love to see a babel table of contents popup (a la the table of
contents popup with reftex implements for latex files).  Is something
like this already available?  If not, how difficult would it be to
implement (I'm happy to try given a pointer in the right
direction(s)...).

Thanks,
eric

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 23.2.1
: using Org-mode version 7.4 (release_7.4.153.ga0b80)

Re: Re: [babel] Painless integration of source blocks with language

[Eric Schulte]

Seth Burleigh <wburle4@gmail.com> writes:

> My bad. I believe i did look up the functions, but they didn't do what i
> needed (or so i thought).
> The code doesn't matter too much, let me explain the idea.
>
> A file may contain many blocks of code. Lets look at a arbitrary block A. In
> the end, block A will somehow become noweb embedded into a source block
> which is actually tangled to a file. Lets call this source block C.
> find-chunk finds this source block, given the position of block A, checking
> also that the source has a type of "clojure". So, basically it searches for
> the string <<A>>, if it finds it, checks if the source block (with an
> arbitrary name, lets say my-block) is tangled, if not, it searches for
> <<my-block>> until it actually finds a tangled source block.
>

This is functional which is currently missing in Babel (e.g. finding
where code block A is referenced), I can see the appeal for such a
function.

>
> tangle-chunk is used to tangle block A by finding block C (using find-chunk)
> and tangling that. It is basically copied from org-babel except that it uses
> mkdir to create any parent directories that do not exist (i think this
> should be included as an option in tangle file, btw, if it isn't)
>

Does tangle chunk tangle just block A, or just block C, or does it
re-tangle the entire file.  If just selected code blocks, how does it
know where to insert them into the file?

WRT: creating missing directories, maybe this would be better as an
option rather than the default, as I often mistype a tangle path and
it's useful for me when an error is thrown upon trying to tangle to a
non-existent directory.  Also, for packaging up repositories, there is
an `org-babel-pre-tangle-hook' which could be used to prepare a
directory structure in which to tangle source files.

>
> So, conceptually we have many blocks which are noweb embedded into various
> source files, however indirectly. We want to find the source files that
> these blocks are embedded in so that we can (in the future) navigate from
> the block to the file number.
>  '

Agreed, the lack of the ability to trace noweb references through to raw
source files (and more importantly to trace back from source files
through to the original Babel files) is becoming a pain-point in the
current code base.

It is currently possible to both jump back from source code to the
embedded code block using `org-babel-tangle-jump-to-org', it should be
fairly straightforward to write a function for jumping from a code block
to source using the same comment lines as anchors.

As for how to trace back through noweb links, the best option seem to be
using the existing jump function to navigate from raw source to the
embedded block, keeping track of the point's offset form the beginning
of the block, then using `org-babel-expand-src-block' to expand the body
of the embedded code block marking noweb references with text properties
as they are inserted into the expanding body, and then using the point
offset to place the point into the appropriate noweb reference.  This
process could then recurse on the embedded noweb code block until it
ends up in a non-noweb portion of an expanded code block body.

> 
> Thats all that does, from line 110 up. I copied it from my previous posting.
> From below that, i implement an emacs overlay, created by new-chunk.
> Basically all it does is associate a chunk of code in the buffer to a chunk
> of code in another chunk. We can 'add a link', i.e. append a chunk to its
> 'lp-link property.
>
> Next, we add the ability to replace the code of one chunk with another. This
> is what push-chunk does. Now, all that we have left is the code to create
> chunks from source blocks.
>

Meaning you can update a portion of the raw source file and push the
changes back to the org file?  This sounds similar to what is done by
`org-babel-detangle' only it uses marks and overlays to maintain the
source->org mapping rather than comments (which is a cleaner interactive
solution, but lacks persistence beyond a single Emacs session, or the
ability to say fold in changes to the raw source made by someone else on
a group project).

>
> Pushing code from one chunk to another code work with noweb embedding, but
> it was just a proof of concept anyways.
>

That's great, what was the logic used to push back through noweb
references?  Did this rely solely on the overlays you created during
tangling?  The analog of that approach would be to insert noweb
references wrapped in comments, which may be the simplest solution, as
long as there are not cases where the additional comments would be
unwanted in the raw source code file.  Perhaps there is a way for Babel
to use marks rather than comments and then save the mark information in
the org file, this would be great in that it wouldn't require any Babel
comments in the source file, although I'm not sure how robust marks are
to offline editing.

>
> What i was imagining when i did this is that i would have a source code file
> which has various 'chunks' highlighted in different colors (various
> background colors), each having a 'link' to the appropriate source code
> chunk in the babel file. Thus, we could push changes from the source code
> files to the org-mode file and navigate between. This would eliminate the
> code markers. I was imagining that the tangled source code files could be
> viewed simply as an indirect buffer created from the org file, so we
> wouldn't need code markers to save the chunk info for future sessions - we
> would just generate it all from the org-file.
>

Yes, this is similar to what I imagined after reading your first post,
and I think it is a great goal to shoot for.  Unfortunately I personally
will be very busy through nearly the end of January...

Thanks for the explanation, Cheers -- Eric

>
>  What is needed now is to be able to push from one chunk to another, except
> leaving any <<srcname>> blocks as they are.  With that, i believe the ideas
> mentioned in the first post could be fully implemented.

Re: [babel] Painless integration of source blocks with language

[Eric Schulte]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> "Eric Schulte" <schulte.eric@gmail.com> writes:
>
> [...]
>
>> A crude version of the above is already possible using the
>> `org-babel-detangle' function.  For example, follow the instructions in
>> the attached org-mode file (which uses elisp rather than clojure code
>> blocks simply for wider portability to non-clojure users).
>
> Detangling, as currently implemented, doesn't do the job for me as it
> doesn't understand noweb.  My current mode of operation with org and
> babel is to have various snippets of code throughout a file and then
> combine these in different ways using noweb syntax which I then tangle
> to create different source files (each bringing together different
> pieces in different configurations).
>

I agree, this is an area ripe for improvement.

>
> In any case, and please excuse me for hijacking this thread a little,
> the increasing use of babel (a good thing!) especially with noweb syntax
> and tangling (as this thread is about) is bringing up a document
> management issue: I find it difficult (a) to remember what all my source
> code snippets are called and (b) to navigate to any given snippet.  I
> would love to see a babel table of contents popup (a la the table of
> contents popup with reftex implements for latex files).  Is something
> like this already available?  If not, how difficult would it be to
> implement (I'm happy to try given a pointer in the right
> direction(s)...).
>

There was some talk of merging imenu with Babel which would provide the
functionality you describe, I don't believe this ever resulting in
working code however.  There are a couple of options...

If you know the name of the code block you want to find you can use
`org-babel-goto-named-src-block' (bound to C-c C-v g) to jump to a named
code block (∃ a similar function for finding named results).  This
function provides completion on the block names, the function
`org-babel-src-block-names' returns a list of all named blocks in the
current buffer, so it could be used to built up such a table.  In fact
the following code block will insert a table of such names in the
current buffer.  Note: you will need to pull the latest as I had to fix
a small bug in `org-babel-src-block-names'.

#+begin_src emacs-lisp :results list
  (mapcar #'list (reverse (org-babel-src-block-names)))
#+end_src

Hope that helps, Best -- Eric

>
> Thanks,
> eric

[babel] Painless integration of source blocks with language

[Seth Burleigh]

[-- Attachment #1.1: Type: text/plain, Size: 574 bytes --]

woops, what i actually meant is that noweb doesnt work. I was thinking about
it, and it might be possible to
(a) automatically create overlays around begin_src blocks using
auto-overlays
(b) have a custom syntax parser that parses noweb syntax in those blocks.
 Sort of like what is done for syntax highlighting.

Im gradually converting my code to literate programming and am just on the
verge of splitting up my one clojure ns to  one-block org file into multiple
blocks per ns, so i will soon need the noweb functionality. When i get it
ill make sure to post and update!

[-- Attachment #1.2: Type: text/html, Size: 681 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] Painless integration of source blocks with language

[Eric S Fraga]

"Eric Schulte" <schulte.eric@gmail.com> writes:

> Eric S Fraga <e.fraga@ucl.ac.uk> writes:

[...]

>> management issue: I find it difficult (a) to remember what all my source
>> code snippets are called and (b) to navigate to any given snippet.  I
>> would love to see a babel table of contents popup (a la the table of
>> contents popup with reftex implements for latex files).  Is something
>> like this already available?  If not, how difficult would it be to
>> implement (I'm happy to try given a pointer in the right
>> direction(s)...).
>>
>
> There was some talk of merging imenu with Babel which would provide the
> functionality you describe, I don't believe this ever resulting in
> working code however.  There are a couple of options...
>
> If you know the name of the code block you want to find you can use
> `org-babel-goto-named-src-block' (bound to C-c C-v g) to jump to a named
> code block (∃ a similar function for finding named results).  This
> function provides completion on the block names, the function
> `org-babel-src-block-names' returns a list of all named blocks in the
> current buffer, so it could be used to built up such a table.  In fact
> the following code block will insert a table of such names in the
> current buffer.  Note: you will need to pull the latest as I had to fix
> a small bug in `org-babel-src-block-names'.
>
> #+begin_src emacs-lisp :results list
>   (mapcar #'list (reverse (org-babel-src-block-names)))
> #+end_src

This is brilliant.  Thanks.  Having the table of contents, together with
being able to jump to any block, is half the battle won!  Using it already.

eric

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 23.2.1
: using Org-mode version 7.4 (release_7.4.166.gf7a7)

Re: [babel] Painless integration of source blocks with language

[Seth Burleigh]

[-- Attachment #1.1: Type: text/plain, Size: 1044 bytes --]

I would just like to throw in a quick idea.
The easiest way to support noweb tangling is to get org-babel-tangle to
create nested tags and change detangle to take these into account.

for example, i have the forex_user source block that is tangled.

;; [[file:~/Dropbox/.rep/clj-forex/clj-forex.org::*User][forex_user]]
my code...
;; [[file:~/Dropbox/.rep/clj-forex/clj-forex.org::*User][embedded_block]]
my embedded code ...
;; embedded_block ends here
;; forex_user ends here

corresponding to a forex_user block of
#+begin_src
  my code ...
  <<embedded_block>>
#+end_src

This would only work for noweb on separate lines and preferably only when
one block is noweb embedded into one other source block, but i believe that
this is the most common case anyways. This could be enforced in the
detangle/tangle code.

Then, a separate minor mode could make these file: markers invisible and
create a colored overlay pattern per source block code. Saving the file
would still save the markers, you just wouldnt see them if the minor mode
was on.

[-- Attachment #1.2: Type: text/html, Size: 1308 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] Painless integration of source blocks with language

[Eric Schulte]

I briefly mentioned the approach you describe below earlier in this
thread [1], and while I see the appeal of this approach, I think that
the second approach I described in that same message (using offsets
instead of comments) is preferable -- although I am of course open to
being persuaded otherwise.

Best -- Eric

Seth Burleigh <wburle@gmail.com> writes:

> I would just like to throw in a quick idea.
> The easiest way to support noweb tangling is to get org-babel-tangle to
> create nested tags and change detangle to take these into account.
>
> for example, i have the forex_user source block that is tangled.
>
> ;; [[file:~/Dropbox/.rep/clj-forex/clj-forex.org::*User][forex_user]]
> my code...
> ;; [[file:~/Dropbox/.rep/clj-forex/clj-forex.org::*User][embedded_block]]
> my embedded code ...
> ;; embedded_block ends here
> ;; forex_user ends here
>
> corresponding to a forex_user block of
> #+begin_src
>   my code ...
>   <<embedded_block>>
> #+end_src
>
> This would only work for noweb on separate lines and preferably only when
> one block is noweb embedded into one other source block, but i believe that
> this is the most common case anyways. This could be enforced in the
> detangle/tangle code.
>
> Then, a separate minor mode could make these file: markers invisible and
> create a colored overlay pattern per source block code. Saving the file
> would still save the markers, you just wouldnt see them if the minor mode
> was on.

Footnotes: 
[1]  http://thread.gmane.org/gmane.emacs.orgmode/35863/focus=35971

Re: [babel] Painless integration of source blocks with language

[Seth Burleigh]

[-- Attachment #1.1: Type: text/plain, Size: 1401 bytes --]

>As for how to trace back through noweb links, the best option seem to be
>using the existing jump function to navigate from raw source to the
>embedded block, keeping track of the point's offset form the beginning
>of the block, then using `org-babel-expand-src-block' to expand the body
>of the embedded code block marking noweb references with text properties
>as they are inserted into the expanding body, and then using the point
>offset to place the point into the appropriate noweb reference.  This
>process could then recurse on the embedded noweb code block until it
>ends up in a non-noweb portion of an expanded code block body.

If you dont have any comment anchors in the text, how would you know which
 noweb block you are in? If your org file and raw files are synched, then
you could, but if they become unsynched, which is the way it should be
(change raw code until done, then detangle to org), i dont exactly see how
you can do it.

The only option which allows someone to change the raw text file is to use
comment anchors - or you could save marks in a separate file, but this
wouldn't  allow for offline editing unless you open it with a program that
understands this format. With the comment method, you get the raw file
advantage, plus emacs can hide these comments if one makes a minor mode to
do so - a win win situation, i think, except for the aforementioned
shortcomings.

[-- Attachment #1.2: Type: text/html, Size: 1683 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] Painless integration of source blocks with language

[Eric Schulte]

Seth Burleigh <wburle@gmail.com> writes:

>>As for how to trace back through noweb links, the best option seem to be
>>using the existing jump function to navigate from raw source to the
>>embedded block, keeping track of the point's offset form the beginning
>>of the block, then using `org-babel-expand-src-block' to expand the body
>>of the embedded code block marking noweb references with text properties
>>as they are inserted into the expanding body, and then using the point
>>offset to place the point into the appropriate noweb reference.  This
>>process could then recurse on the embedded noweb code block until it
>>ends up in a non-noweb portion of an expanded code block body.
>
> If you dont have any comment anchors in the text, how would you know which
>  noweb block you are in? If your org file and raw files are synched, then
> you could, but if they become unsynched, which is the way it should be
> (change raw code until done, then detangle to org), i dont exactly see how
> you can do it.
>

Yea, I think you're right.  I can't come up with any way of marking or
remembering the boundaries of noweb reference sections without inserting
comment wrappers around such sections.

I've just pushed up a new header argument combination ":comments noweb"
which will wrap all embedded noweb sections in link comments, as
demonstrated in the following example.  Hopefully this should be
sufficient for a complete mapping from a pure code file back to the
original org-mode file.

#+source: wrappable
#+begin_src emacs-lisp
  (setq x (+ 4 x))
#+end_src

#+begin_src emacs-lisp :comments noweb :noweb yes :tangle yes
  (let ((x 1))
    (message "x=%s" x)
    <<wrappable>>
    (message "x=%s" x))
#+end_src

which tangles out the following emacs-lisp
#+begin_src emacs-lisp
  ;; [[file:~/src/babel-dev/scraps.org::*wrap%20noweb%20references%20in%20comments][wrap-noweb-references-in-comments:2]]
  (let ((x 1))
    (message "x=%s" x)
    ;; [[file:~/src/babel-dev/scraps.org::*wrap%20noweb%20references%20in%20comments][wrappable]]
    (setq x (+ 4 x))
    ;; wrappable ends here
    (message "x=%s" x))
  ;; wrap-noweb-references-in-comments:2 ends here
#+end_src

Cheers -- Eric

Re: [babel] Painless integration of source blocks with language

[Sébastien Vauban]

Hi Eric,

"Eric Schulte" wrote:
> I've just pushed up a new header argument combination ":comments noweb"
> which will wrap all embedded noweb sections in link comments, as
> demonstrated in the following example.  Hopefully this should be
> sufficient for a complete mapping from a pure code file back to the
> original org-mode file.

If I understand correctly, this is some sort of enhancement to the ":comments
yes" option.

If yes, is there a reason to have both values, with subtle differences?
Shouldn't we port the behavior of ":comments noweb" to the ":comments yes"
value?

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: [babel] Painless integration of source blocks with language

[Eric Schulte]

Sébastien Vauban <wxhgmqzgwmuf@spammotel.com> writes:

> Hi Eric,
>
> "Eric Schulte" wrote:
>> I've just pushed up a new header argument combination ":comments noweb"
>> which will wrap all embedded noweb sections in link comments, as
>> demonstrated in the following example.  Hopefully this should be
>> sufficient for a complete mapping from a pure code file back to the
>> original org-mode file.
>
> If I understand correctly, this is some sort of enhancement to the ":comments
> yes" option.
>
> If yes, is there a reason to have both values, with subtle differences?
> Shouldn't we port the behavior of ":comments noweb" to the ":comments yes"
> value?
>

Yes, the reason to retain both variants is that in the case of short
noweb references embedded in a single line, e.g.

#+source: first
#+begin_src ruby
  5
#+end_src

#+begin_src ruby :noweb yes
  <<first>> + 6
#+end_src

the yes option would be preferable, as the insertion of comments around
the noweb reference would result in adding line breaks around the
reference and possibly breaking new-line sensitive code.

Cheers -- Eric

>
> Best regards,
>   Seb

Re: [babel] Painless integration of source blocks with language

[Sébastien Vauban]

Hi Eric,

"Eric Schulte" wrote:
> Sébastien Vauban <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes:
>> "Eric Schulte" wrote:
>>> I've just pushed up a new header argument combination ":comments noweb"
>>> which will wrap all embedded noweb sections in link comments, as
>>> demonstrated in the following example.  Hopefully this should be
>>> sufficient for a complete mapping from a pure code file back to the
>>> original org-mode file.
>>
>> If I understand correctly, this is some sort of enhancement to the ":comments
>> yes" option.
>>
>> If yes, is there a reason to have both values, with subtle differences?
>> Shouldn't we port the behavior of ":comments noweb" to the ":comments yes"
>> value?
>
> Yes, the reason to retain both variants is that in the case of short
> noweb references embedded in a single line, e.g.
>
> #+source: first
> #+begin_src ruby
>   5
> #+end_src
>
> #+begin_src ruby :noweb yes
>   <<first>> + 6
> #+end_src
>
> the yes option would be preferable, as the insertion of comments around
> the noweb reference would result in adding line breaks around the
> reference and possibly breaking new-line sensitive code.

I see. I do second your choice...

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] Painless integration of source blocks with language

[Seth Burleigh]

[-- Attachment #1.1: Type: text/plain, Size: 914 bytes --]

On Sun, Jan 16, 2011 at 9:31 AM, Eric Schulte <schulte.eric@gmail.com>wrote:

>
> #+source: wrappable
> #+begin_src emacs-lisp
>  (setq x (+ 4 x))
> #+end_src
>
> #+begin_src emacs-lisp :comments noweb :noweb yes :tangle yes
>  (let ((x 1))
>    (message "x=%s" x)
>    <<wrappable>>
>    (message "x=%s" x))
> #+end_src
>
> which tangles out the following emacs-lisp
> #+begin_src emacs-lisp
>  ;; [[file:~/src/babel-dev/scraps.org:
> :*wrap%20noweb%20references%20in%20comments][wrap-noweb-references-in-comments:2]]
>  (let ((x 1))
>    (message "x=%s" x)
>    ;; [[file:~/src/babel-dev/scraps.org:
> :*wrap%20noweb%20references%20in%20comments][wrappable]]
>    (setq x (+ 4 x))
>    ;; wrappable ends here
>    (message "x=%s" x))
>  ;; wrap-noweb-references-in-comments:2 ends here
> #+end_src
>
> Cheers -- Eric
>


Cool! Does it yet support detangling? I tried it out and it said i wasnt in
a tangle code.

[-- Attachment #1.2: Type: text/html, Size: 1352 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] Painless integration of source blocks with language

[Sébastien Vauban]

Hi Seth,

Seth Burleigh wrote:
> On Sun, Jan 16, 2011 at 9:31 AM, Eric Schulte <span dir="ltr"><mailto:schulte.eric-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org></span> wrote:

Please try using a decent sender, or at least no HTML. I know this cannot be
obvious, though, but look at the results:

> #+source: wrappable<br>
> #+begin_src emacs-lisp<br>
>  .(setq x (+ 4 x))
> #+end_src
>
> #+begin_src emacs-lisp :comments noweb :noweb yes :tangle yes<br>
>  .(let ((x 1))<br>
>  . .(message &quot;x=%s&quot; x)<br>
>  . .&lt;&lt;wrappable&gt;&gt;<br>
>  . .(message &quot;x=%s&quot; x))<br>
> #+end_src<br>
>
> which tangles out the following emacs-lisp
> #+begin_src emacs-lisp<br>
>  .;; [[file:~/src/babel-dev/scraps.org::*wrap%20noweb%20references%20in%20comments][wrap-noweb-references-in-comments:2]]<br>
>  .(let ((x 1))<br>
>  . .(message &quot;x=%s&quot; x)<br>
>  . .;; [[file:~/src/babel-dev/scraps.org::*wrap%20noweb%20references%20in%20comments][wrappable]]<br>
>  . .(setq x (+ 4 x))<br>
>  . .;; wrappable ends here<br>
>  . .(message &quot;x=%s&quot; x))<br>
>  .;; wrap-noweb-references-in-comments:2 ends here<br>
> #+end_src<br>
>
> Cheers -- Eric
> </blockquote>Cool! Does it yet support detangling? I tried it out and it said i wasnt in a tangle code.

Your cursor must be on a code line, not on a special "comments" line. I've had
the same behavior once. Just move down your cursor one or two lines away...

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: [babel] Painless integration of source blocks with language

[Seth Burleigh]

[-- Attachment #1.1: Type: text/plain, Size: 707 bytes --]

> Your cursor must be on a code line, not on a special "comments" line. I've
> had

the same behavior once. Just move down your cursor one or two lines away...
>
>
 Are you saying you have succesfully detangled noweb embedded code? If so,
ill have to take a look at the code to see where the bug is. It doesn't work
no matter where i place my cursor. My tangled output is also different -

;; [[file:~/Desktop/test.org][/home/seth/Desktop/test\.org:2]]

(let ((x 1))
  (message "x=%s" x)
  ;; [[file:~/Desktop/test.org][wrappable]]
  (setq x (+ 4 x))
  ;; wrappable ends here
  (message "x=%s" x))

;; /home/seth/Desktop/test\.org:2 ends here


Notice that detangling works if there is no noweb being used.

[-- Attachment #1.2: Type: text/html, Size: 1302 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: [babel] Painless integration of source blocks with language

[Seth Burleigh]

[-- Attachment #1.1: Type: text/plain, Size: 371 bytes --]

>
>
> Please try using a decent sender, or at least no HTML. I know this cannot
> be
> obvious, though, but look at the results:
>
>
>
Not to get off on a tanget, but im using gmail. Looks perfect in my gmail
account - i guess your email doesnt support html markup? I just reply to
all, and it automatically does the markup. I guess ill have to learn how to
disable it:)

[-- Attachment #1.2: Type: text/html, Size: 581 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: [babel] Painless integration of source blocks with language

[Seth Burleigh]

ok, so im not sure how you got it working, but heres the problem in
the code (i think).

We start with the tangled output
--------------------------------------
;; [[file:~/Desktop/test.org][/home/seth/Desktop/test\.org:2]]

(let ((x 1))
  (message "x=%s" x)
  ;; [[file:~/Desktop/test.org][wrappable]]
  (setq x (+ 4 x))
  ;; wrappable ends here
  (message "x=%s" x))
;;>> before org-babel-detangle calls org-babel-tangle-jump-to-org, it
is at this point
;; /home/seth/Desktop/test\.org:2 ends here
--------------------------------------

The code jumps to the point marked above (notice any comment with ;;>>
is just my markup)
So, lets go to the relevant portion of org-babel-tangle-jump-to-org.

--------------------------------------
(defun org-babel-tangle-jump-to-org ()
  "Jump from a tangled code file to the related Org-mode file."
  (interactive)
  (let ((mid (point))
	start end done
        target-buffer target-char link path block-name body)
    (save-window-excursion
      (save-excursion
	(while (and (re-search-backward org-bracket-link-analytic-regexp nil t)
		    (not ; ever wider searches until matching block comments
		     (and (setq start (point-at-eol))
			  (setq link (match-string 0))
			  (setq path (match-string 3))
			  (setq block-name (match-string 5))
			  (save-excursion
			    (save-match-data
			      (re-search-forward
			       (concat " " (regexp-quote block-name)
				       " ends here") nil t)
			      (setq end (point-at-bol))))))))
	;;(message "start %s mid %s end %s" start mid end)
	(unless (and start (< start mid) (< mid end))
	  (error (format "not in tangled code %s %s %s " start mid end)))
---------------------------------------


notice that mid is set to the point commented previously. Then it
searches backward for org-bracket-link-analytic-regexp , and it finds
;; [[file:~/Desktop/test.org][wrappable]]. Then it searches forward
and matches ;; wrappable ends here. So, start and end encompass the
inner block, but the mid point is in the outside block. Therefore, it
fails with the error message

(unless (and start (< start mid) (< mid end))
	  (error (format "not in tangled code %s %s %s " start mid end)))

Re: Re: [babel] Painless integration of source blocks with language

[Bastien]

Hi Seth,

Seth Burleigh <wburle4@gmail.com> writes:

> Not to get off on a tanget, but im using gmail. Looks perfect in my
> gmail account - i guess your email doesnt support html markup? I just
> reply to all, and it automatically does the markup. I guess ill have
> to learn how to disable it:)

Easy enough: there is an option in gmail to send mail in plain text.

Especially useful if you want to send patches!

Thanks,

-- 
 Bastien

Re: Re: [babel] Painless integration of source blocks with language

[Eric Schulte]

Hi Seth,

You are correct, while the tangling works, the detangling still needs to
be updated to take into account the fact that there may now be nested
sections of tangled code -- which it doesn't currently.  Hopefully this
wont be too large of a code change...

Seth Burleigh <wburle4@gmail.com> writes:

> ok, so im not sure how you got it working, but heres the problem in
> the code (i think).
>
> We start with the tangled output
> --------------------------------------
> ;; [[file:~/Desktop/test.org][/home/seth/Desktop/test\.org:2]]
>
> (let ((x 1))
>   (message "x=%s" x)
>   ;; [[file:~/Desktop/test.org][wrappable]]
>   (setq x (+ 4 x))
>   ;; wrappable ends here
>   (message "x=%s" x))
> ;;>> before org-babel-detangle calls org-babel-tangle-jump-to-org, it
> is at this point
> ;; /home/seth/Desktop/test\.org:2 ends here
> --------------------------------------
>
> The code jumps to the point marked above (notice any comment with ;;>>
> is just my markup)
> So, lets go to the relevant portion of org-babel-tangle-jump-to-org.
>
> --------------------------------------
> (defun org-babel-tangle-jump-to-org ()
>   "Jump from a tangled code file to the related Org-mode file."
>   (interactive)
>   (let ((mid (point))
> 	start end done
>         target-buffer target-char link path block-name body)
>     (save-window-excursion
>       (save-excursion
> 	(while (and (re-search-backward org-bracket-link-analytic-regexp nil t)
> 		    (not ; ever wider searches until matching block comments
> 		     (and (setq start (point-at-eol))
> 			  (setq link (match-string 0))
> 			  (setq path (match-string 3))
> 			  (setq block-name (match-string 5))
> 			  (save-excursion
> 			    (save-match-data
> 			      (re-search-forward
> 			       (concat " " (regexp-quote block-name)
> 				       " ends here") nil t)
> 			      (setq end (point-at-bol))))))))
> 	;;(message "start %s mid %s end %s" start mid end)
> 	(unless (and start (< start mid) (< mid end))
> 	  (error (format "not in tangled code %s %s %s " start mid end)))
> ---------------------------------------
>
>
> notice that mid is set to the point commented previously. Then it
> searches backward for org-bracket-link-analytic-regexp , and it finds
> ;; [[file:~/Desktop/test.org][wrappable]]. Then it searches forward
> and matches ;; wrappable ends here. So, start and end encompass the
> inner block, but the mid point is in the outside block. Therefore, it
> fails with the error message
>
> (unless (and start (< start mid) (< mid end))
> 	  (error (format "not in tangled code %s %s %s " start mid end)))
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] Painless integration of source blocks with language

[Dan Davison]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> "Eric Schulte" <schulte.eric@gmail.com> writes:
>
>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
> [...]
>
>>> management issue: I find it difficult (a) to remember what all my source
>>> code snippets are called and (b) to navigate to any given snippet.  I
>>> would love to see a babel table of contents popup (a la the table of
>>> contents popup with reftex implements for latex files).  Is something
>>> like this already available?  If not, how difficult would it be to
>>> implement (I'm happy to try given a pointer in the right
>>> direction(s)...).
>>>
>>
>> There was some talk of merging imenu with Babel which would provide the
>> functionality you describe, I don't believe this ever resulting in
>> working code however.  There are a couple of options...
>>
>> If you know the name of the code block you want to find you can use
>> `org-babel-goto-named-src-block' (bound to C-c C-v g) to jump to a named
>> code block (∃ a similar function for finding named results).

Would anyone object if I change that function slightly so that it
doesn't strip text properties, so that the search hits come up in the
minibuffer with the in-context fontification?

>> This function provides completion on the block names, the function
>> `org-babel-src-block-names' returns a list of all named blocks in the
>> current buffer, so it could be used to built up such a table.  In fact
>> the following code block will insert a table of such names in the
>> current buffer.  Note: you will need to pull the latest as I had to fix
>> a small bug in `org-babel-src-block-names'.
>>
>> #+begin_src emacs-lisp :results list
>>   (mapcar #'list (reverse (org-babel-src-block-names)))
>> #+end_src

As a different approach, `occur' is good for doing this (results listed
in a separate buffer with jump-back links). The function below can be
extended in the obvious way.

#+begin_src emacs-lisp
(defun dan/find-in-buffer ()
  (interactive)
  (let ((targets
         `(("<named src blocks>" . ,org-babel-src-name-regexp)
           ("<src block results>" . ,org-babel-result-regexp))))
    (occur
     (cdr
      (assoc
       (ido-completing-read "Find: " (mapcar #'car targets))
       targets)))
    (other-window 1)))
#+end_src

Dan

>
> This is brilliant.  Thanks.  Having the table of contents, together with
> being able to jump to any block, is half the battle won!  Using it already.
>
> eric

Re: Re: [babel] Painless integration of source blocks with language

[Seth Burleigh]

> You are correct, while the tangling works, the detangling still needs to
> be updated to take into account the fact that there may now be nested
> sections of tangled code -- which it doesn't currently.  Hopefully this
> wont be too large of a code change...
>

It probably wouldnt involve much code change, since there is very few
functions which operate on tangled files - detangle and jump to source
block?
It would be a good time to implement some functions, maybe like this,
which did operate on tangled files -

org-babel-detangle-src-block-info
which, given a point on the line of a link, would produced something like
((end . '(0 100)) (middle . (10 90)) (children . (((end . (20 30))
...) ... more children))
where end is the position in the buffer from the beginning of the link
to the end of the matching end link, and middle is the body portion.

. Then define a function which maps over
all of the detangled toplevel src blocks of a file(like what is done
for tangled src blocks).

And then one could call org-babel-detangle-src-block which would do
some recursive detangling


if has children, detangle children
get body string
replace the children in body string (using end) with appropriate noweb syntax
replace appropriate org block with code

and then you could call org-babel-detangle which would detangle all of
a file, using the map function mentioned previously to collect all
source blocks.

(with-error
  detangle all toplevel blocks of file
)

where with-error will catch any errors, replace org file with original
content if an error is thrown, and then reraise the error.

Re: Re: [babel] Painless integration of source blocks with language

[Eric S Fraga]

Dan Davison <dandavison7@gmail.com> writes:

[...]

>>> If you know the name of the code block you want to find you can use
>>> `org-babel-goto-named-src-block' (bound to C-c C-v g) to jump to a named
>>> code block (∃ a similar function for finding named results).
>
> Would anyone object if I change that function slightly so that it
> doesn't strip text properties, so that the search hits come up in the
> minibuffer with the in-context fontification?

I won't object but I'm not entirely sure what I would not be objecting
to ... ;-) 

Basically, what exactly are you proposing to display?  How much
in-context fontification are you talking about?  I can see the benefit
of having the actual #+begin_src line displayed so that I can see
whether the code snippet is R or octave or ...

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1
: using Org-mode version 7.4 (release_7.4.232.g8d003)

Re: [babel] Painless integration of source blocks with language

[Sébastien Vauban]

Hi Dan,

Dan Davison wrote:
> Eric S Fraga <e.fraga-hclig2XLE9Zaa/9Udqfwiw@public.gmane.org> writes:
>> "Eric Schulte" <schulte.eric-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>> Eric S Fraga <e.fraga-hclig2XLE9Zaa/9Udqfwiw@public.gmane.org> writes:
>>
>>>> management issue: I find it difficult (a) to remember what all my source
>>>> code snippets are called and (b) to navigate to any given snippet. I
>>>> would love to see a babel table of contents popup (a la the table of
>>>> contents popup with reftex implements for latex files). Is something like
>>>> this already available? If not, how difficult would it be to implement
>>>> (I'm happy to try given a pointer in the right direction(s)...).
>>>
>>> There was some talk of merging imenu with Babel which would provide the
>>> functionality you describe, I don't believe this ever resulting in working
>>> code however. There are a couple of options...
>>>
>>> If you know the name of the code block you want to find you can use
>>> `org-babel-goto-named-src-block' (bound to C-c C-v g) to jump to a named
>>> code block (∃ a similar function for finding named results).
>
> Would anyone object if I change that function slightly so that it
> doesn't strip text properties, so that the search hits come up in the
> minibuffer with the in-context fontification?

Not sure either to understand what you mean...


>>> This function provides completion on the block names, the function
>>> `org-babel-src-block-names' returns a list of all named blocks in the
>>> current buffer, so it could be used to built up such a table.  In fact
>>> the following code block will insert a table of such names in the
>>> current buffer.  Note: you will need to pull the latest as I had to fix
>>> a small bug in `org-babel-src-block-names'.
>>>
>>> #+begin_src emacs-lisp :results list
>>>   (mapcar #'list (reverse (org-babel-src-block-names)))
>>> #+end_src
>
> As a different approach, `occur' is good for doing this (results listed
> in a separate buffer with jump-back links). The function below can be
> extended in the obvious way.
>
> #+begin_src emacs-lisp
> (defun dan/find-in-buffer ()
>   (interactive)
>   (let ((targets
>          `(("<named src blocks>" . ,org-babel-src-name-regexp)
>            ("<src block results>" . ,org-babel-result-regexp))))
>     (occur
>      (cdr
>       (assoc
>        (ido-completing-read "Find: " (mapcar #'car targets))
>        targets)))
>     (other-window 1)))
> #+end_src

Thanks for sharing this.

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode