org-html link building diff

org-html link building diff

[Tom Breton (Tehom)]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

I've refactored `org-export-as-html', factored code to build links
into `org-html-make-link'.

This does two things that I needed:

 * It allows custom link types to build anchors.
   * How: Call org-html-make-link.  Many parameters, see the function
     docstring.  It returns a string containing an HTML link.
 * It adds the capability to convert links when exporting.
   * How: Around the export call, bind org-html-cvt-link-fn to a
     function that takes 1 parameter (filename) and returns a url as a
     string.

I think it also makes the code cleaner.

There are more things that could be done - it's only used by some of
the cond branches, the others are unchanged.  But "publish early and
often", so here it is.

I will append the changes as a diff, since I can't push to the org
repository ("fatal: The remote end hung up unexpectedly")

	Tom Breton (Tehom)

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: org-html.el.diff --]
[-- Type: text/x-patch; name="org-html.el.diff", Size: 9145 bytes --]

diff --git a/lisp/org-html.el b/lisp/org-html.el
index 74f3a55..9aaadec 100644
--- a/lisp/org-html.el
+++ b/lisp/org-html.el
@@ -533,6 +533,106 @@ in a window.  A non-interactive call will only return the buffer."
 
 (defvar html-table-tag nil) ; dynamically scoped into this.
 (defvar org-par-open nil)
+(defconst org-html-cvt-link-fn 
+   ;;In the future this might change to take more args: type + path +
+   ;;fragment
+   #'identity
+   "Function to convert link URLs to exportable URLs.
+Takes one argument, PATH.
+Returns exportable URL.
+Intended to be locally bound around a call to `org-export-as-html'." )
+
+;;; org-html-cvt-link-fn
+(defconst org-html-cvt-link-fn 
+   ;;In the future this might change to take more args: type + path +
+   ;;fragment
+   #'identity
+   "Function to convert link URLs to exportable URLs.
+Takes one argument, PATH.
+Returns exportable URL.
+Intended for remote exporting." )
+
+
+;;; org-html-make-link
+;;Special variables seen:
+;;`html-extension' -- From plist
+;;`org-par-open' is a special variable so it's not in the arglist.
+(defun org-html-make-link (type path fragment desc descp attr
+				   may-inline-p) 
+   "Make an HTML link
+TYPE is the device-type of the link (And isn't used yet) (THIS://foo.html)
+PATH is the path of the link (http://THIS)
+FRAGMENT is the fragment part of the link, if any (The foo.html#THIS part)
+DESC is the link description, if any.
+DESCP is whether there originally was a description.
+ATTR is a string of other attributes of the a element.
+MAY-INLINE-P allows inlining it as an image."
+
+   (declare (special html-extension org-par-open))
+   (let ((filename path)
+	    thefile)
+      (save-match-data
+	 ;;First pass.  Mostly deals with treating local files.  TYPE
+	 ;;may still change.
+	 (cond
+	    ((string= type "file")
+	       ;;Substitute just if original path was absolute.
+	       ;;(Otherwise path must remain relative)
+	       (setq thefile 
+		  (if (file-name-absolute-p filename) 
+		     (expand-file-name filename) 
+		     filename))
+		  
+	       (when (and org-export-html-link-org-files-as-html
+			(string-match "\\.org$" thefile))
+		  (setq type "http")
+		  (setq thefile (concat (substring thefile 0
+					   (match-beginning 0))
+				   "." html-extension))))
+	    (t (setq thefile filename)))
+	    
+	 ;;If applicable, convert local path to remote URL
+	 (setq thefile
+	    (or
+	       (funcall org-html-cvt-link-fn thefile)
+	       thefile))
+
+	 ;;Second pass.  Build final link except for leading type
+	 ;;spec.  Now TYPE is final.
+	 (cond
+	    ((or
+		(string= type "http")
+		(string= type "https"))
+	       (if fragment
+		  (setq thefile (concat thefile "#" fragment))))
+	       
+	    (t))
+	    
+	 ;;Final URL-build, for all types.
+	 (setq thefile 
+	    (concat type ":" (org-export-html-format-href thefile)))
+
+	 (if (and 
+		may-inline-p
+		;;Can't inline a URL with a fragment.
+		(not fragment)
+		(or 
+		   (eq t org-export-html-inline-images)
+		   (and 
+		      org-export-html-inline-images
+		      (not descp)))
+		(org-file-image-p
+		   filename org-export-html-inline-image-extensions))
+
+	    (progn
+	       (message "image %s %s" thefile org-par-open)
+	       (org-export-html-format-image thefile org-par-open))
+	    (concat 
+	       "<a href=\"" thefile "\"" attr ">"
+	       (org-export-html-format-desc desc)
+	       "</a>")))))
+
+;;; org-export-as-html
 ;;;###autoload
 (defun org-export-as-html (arg &optional hidden ext-plist
 			       to-buffer body-only pub-dir)
@@ -1014,7 +1114,7 @@ lang=\"%s\" xml:lang=\"%s\">
 				  "\" class=\"target\">" (match-string 1 line)
 				  "@</a> ")
 			  t t line)))))
-
+	    
 	  (setq line (org-html-handle-time-stamps line))
 
 	  ;; replace "&" by "&amp;", "<" and ">" by "&lt;" and "&gt;"
@@ -1070,28 +1170,25 @@ lang=\"%s\" xml:lang=\"%s\">
 	      (save-match-data
 		(setq id-file (file-relative-name
 			       id-file (file-name-directory org-current-export-file)))
-		(setq id-file (concat (file-name-sans-extension id-file)
-				      "." html-extension))
-		(setq rpl (concat "<a href=\"" id-file "#"
-				  (if (org-uuidgen-p path) "ID-")
-				  path "\""
-				  attr ">"
-				  (org-export-html-format-desc desc)
-				  "</a>"))))
+		(setq rpl 
+		   (org-html-make-link
+		      "file" id-file 
+		      (concat (if (org-uuidgen-p path) "ID-") path)
+		      (org-export-html-format-desc desc)
+		      descp
+		      attr
+		      nil))))
 	     ((member type '("http" "https"))
-	      ;; standard URL, just check if we need to inline an image
-	      (if (and (or (eq t org-export-html-inline-images)
-			   (and org-export-html-inline-images (not descp)))
-		       (org-file-image-p
-			path org-export-html-inline-image-extensions))
-		  (setq rpl (org-export-html-format-image
-			     (concat type ":" path) org-par-open))
-		(setq link (concat type ":" path))
-		(setq rpl (concat "<a href=\""
-				  (org-export-html-format-href link)
-				  "\"" attr ">"
-				  (org-export-html-format-desc desc)
-				  "</a>"))))
+	      ;; standard URL, just check if we need to inline an
+	      ;; image
+		(setq rpl
+		   (org-html-make-link
+		      type path nil
+		      (org-export-html-format-desc desc)
+		      descp
+		      attr
+		      ;;But desc already becomes image.
+		      t)))
 	     ((member type '("ftp" "mailto" "news"))
 	      ;; standard URL
 	      (setq link (concat type ":" path))
@@ -1115,52 +1212,49 @@ lang=\"%s\" xml:lang=\"%s\">
 
 	     ((string= type "file")
 	      ;; FILE link
-	      (let* ((filename path)
-		     (abs-p (file-name-absolute-p filename))
-		     thefile file-is-image-p search)
 		(save-match-data
-		  (if (string-match "::\\(.*\\)" filename)
-		      (setq search (match-string 1 filename)
-			    filename (replace-match "" t nil filename)))
-		  (setq valid
-			(if (functionp link-validate)
-			    (funcall link-validate filename current-dir)
-			  t))
-		  (setq file-is-image-p
-			(org-file-image-p
-			 filename org-export-html-inline-image-extensions))
-		  (setq thefile (if abs-p (expand-file-name filename) filename))
-		  (when (and org-export-html-link-org-files-as-html
-			     (string-match "\\.org$" thefile))
-		    (setq thefile (concat (substring thefile 0
-						     (match-beginning 0))
-					  "." html-extension))
-		    (if (and search
-			     ;; make sure this is can be used as target search
-			     (not (string-match "^[0-9]*$" search))
-			     (not (string-match "^\\*" search))
-			     (not (string-match "^/.*/$" search)))
-			(setq thefile
-			      (concat thefile
-				      (if (= (string-to-char search) ?#) "" "#")
-				      (org-solidify-link-text
-				       (org-link-unescape search)))))
-		    (when (string-match "^file:" desc)
-		      (setq desc (replace-match "" t t desc))
-		      (if (string-match "\\.org$" desc)
-			  (setq desc (replace-match "" t t desc))))))
-		(setq rpl (if (and file-is-image-p
-				   (or (eq t org-export-html-inline-images)
-				       (and org-export-html-inline-images
-					    (not descp))))
-			      (progn
-				(message "image %s %s" thefile org-par-open)
-				(org-export-html-format-image thefile org-par-open))
-			    (concat "<a href=\"" thefile "\"" attr ">"
-				    (org-export-html-format-desc desc)
-				    "</a>")))
-		(if (not valid) (setq rpl desc))))
-
+		   (let*
+		      ((frag-p
+			  (string-match "::\\(.*\\)" path))
+			 ;;Get the proper path
+			 (path-1
+			    (if frag-p
+			       (replace-match "" t nil path)
+			       path))
+			 ;;Get the raw fragment
+			 (fragment-0
+			    (match-string 1 filename))
+			 ;;Check the fragment.  If it can't be used as
+			 ;;target fragment we'll use nil instead.
+			 (fragment-1
+			    (if
+			       (and frag-p
+				  (not (string-match "^[0-9]*$" fragment-0))
+				  (not (string-match "^\\*" fragment-0))
+				  (not (string-match "^/.*/$" fragment-0)))
+		     
+			       (org-solidify-link-text
+				  (org-link-unescape fragment-0))
+			       nil))
+			 (desc-2
+			    (if (string-match "^file:" desc)
+			       (let
+				  ((desc-1 (replace-match "" t t desc)))
+				  (if (string-match "\\.org$" desc-1)
+				     (replace-match "" t t desc-1)
+				     desc-1))
+			       desc)))
+		      
+		      (setq rpl
+			 (if
+			    (and
+			       (functionp link-validate)
+			       (not (funcall link-validate path-1 current-dir)))
+			    desc
+			    (org-html-make-link
+			       "file" path-1 fragment-1 desc-2 descp
+			       attr t))))))
+	       
 	     (t
 	      ;; just publish the path, as default
 	      (setq rpl (concat "<i>&lt;" type ":"
@@ -1502,6 +1596,7 @@ lang=\"%s\" xml:lang=\"%s\">
 	    (kill-buffer (current-buffer)))
 	(current-buffer)))))
 
+
 (defun org-export-html-insert-plist-item (plist key &rest args)
   (let ((item (plist-get plist key)))
     (cond ((functionp item)

[-- Attachment #3: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: org-html link building diff

[Carsten Dominik]

On Apr 18, 2010, at 4:13 AM, Tom Breton (Tehom) wrote:

> I've refactored `org-export-as-html', factored code to build links
> into `org-html-make-link'.
>
> This does two things that I needed:
>
> * It allows custom link types to build anchors.
>   * How: Call org-html-make-link.  Many parameters, see the function
>     docstring.  It returns a string containing an HTML link.
> * It adds the capability to convert links when exporting.
>   * How: Around the export call, bind org-html-cvt-link-fn to a
>     function that takes 1 parameter (filename) and returns a url as a
>     string.
>
> I think it also makes the code cleaner.

I am sure it does - the export function has grown like cancer in the  
run of years.
Unfortunately, I right now do not have the time to study this  
carefully enough
to make sure this does not break anything.

Tom, maybe you can update the patch to the current master.

Are there any volunteers who can put this patch through the mill?  Or  
Tom, maybe
you first want to implement the other stuff you are thinking about?

Thanks!

- Carsten

>
> There are more things that could be done - it's only used by some of
> the cond branches, the others are unchanged.  But "publish early and
> often", so here it is.
>
> I will append the changes as a diff, since I can't push to the org
> repository ("fatal: The remote end hung up unexpectedly")
>
> 	Tom Breton (Tehom)
> <org-html.el.diff>_______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

- Carsten

Re: org-html link building diff

[Sebastian Rose]

Carsten Dominik <carsten.dominik@gmail.com> writes:
> On Apr 18, 2010, at 4:13 AM, Tom Breton (Tehom) wrote:
>
>> I've refactored `org-export-as-html', factored code to build links
>> into `org-html-make-link'.
>>
>> This does two things that I needed:
>>
>> * It allows custom link types to build anchors.
>>   * How: Call org-html-make-link.  Many parameters, see the function
>>     docstring.  It returns a string containing an HTML link.
>> * It adds the capability to convert links when exporting.
>>   * How: Around the export call, bind org-html-cvt-link-fn to a
>>     function that takes 1 parameter (filename) and returns a url as a
>>     string.
>>
>> I think it also makes the code cleaner.
>
> I am sure it does - the export function has grown like cancer in the run of
> years.
> Unfortunately, I right now do not have the time to study this carefully enough
> to make sure this does not break anything.
>
> Tom, maybe you can update the patch to the current master.
>
> Are there any volunteers who can put this patch through the mill?  Or Tom, maybe
> you first want to implement the other stuff you are thinking about?


Tom and Carsten,


here's a volunteer for testing it. I've been looking in this export
and publishing stuff frequently lately, and a clean-up would be great.


  Sebastian

Re: org-html link building diff

[Tom Breton (Tehom)]

> I am sure it does - the export function has grown like cancer in the
> run of years.
> Unfortunately, I right now do not have the time to study this
> carefully enough
> to make sure this does not break anything.

> Tom, maybe you can update the patch to the current master.

Actually, I don't seem to be able to push to the repository.
Unfortch, my machine isn't that great for hosting a public repository,
as the instructions suggest.  Is there something I need to do?

Or did you mean, base it on the current version of org?  I thought I
had because I had just pulled it.

> Are there any volunteers who can put this patch through the mill?  Or
> Tom, maybe
> you first want to implement the other stuff you are thinking about?

That's half-done already;

 * Mostly, I made the other branches of `cond' use
   `org-html-make-link' too.  That's done.

 * I'd like to cleanup the test whether to inline images.  My first
   patch had copied it more or less unchanged from what it was.

   Now the bulk of the test is encapsulated in
   `org-html-should-inline-p', but I've yet to make the code use it.

 * And I'd like to make the two sources of name conversion respect
   each other.

   * There are now two sources of name conversion:

     * The existing code to convert org files which operates if
       `org-export-html-link-org-files-as-html' is non-nil.

       * This changes url type as well as url path

     * org-html-cvt-link-fn, which supports converting via a lookup
       table or similar.  I need that for org2blog, which posts to a
       Blogger-style blog directly from org.

       * Presently this does not treat url type.

     * (For completeness) And it can not convert the name at all.

   * So `org-html-cvt-link-fn' signature may change in the near
     future, and I'll probably encapsulate the other conversion, and
     then the logic will just be choose one by precedence.

A couple of notes:

 * I am taking "to make sure this does not break anything" to heart
   and I will write some semblance of a test suite for org-html.  I'll
   post that soon.

   Everybody, please feel free to contribute any tests of the proper
   current behavior to it.

 * Potential snippage:

   * There is a FIXME comment in `org-export-as-html' that's been there
     for a long time.  I suspect it's been fixed, because I haven't
     encountered a problem about not escaping something.

     * "FIXME: do we need to unescape here somewhere?"

   * I don't believe the code just before the `cond' still adds
     anything useful.  It seems redundant to other code.  But I'd like
     to hear from others before I dare snip it out.

     That is, the code that begins ";; Make an image out of the
     description if that is so wanted"

	Tom Breton (Tehom)

Re: org-html link building diff

[Carsten Dominik]

On Apr 26, 2010, at 9:45 PM, Tom Breton (Tehom) wrote:

>
>
>> I am sure it does - the export function has grown like cancer in the
>> run of years.
>> Unfortunately, I right now do not have the time to study this
>> carefully enough
>> to make sure this does not break anything.
>
>> Tom, maybe you can update the patch to the current master.

What I meant here is that you rebase your patch to the current origin/ 
master
and republish it.  I have made some small changes to the HTML exporter  
(should not have conflicts with your changes, I hope...)

>
> Actually, I don't seem to be able to push to the repository.
> Unfortch, my machine isn't that great for hosting a public repository,
> as the instructions suggest.  Is there something I need to do?

Yes.  Send me your name on repo.or.cz and I'll add push for you.
Please create your own branch and stay on it.

>
> Or did you mean, base it on the current version of org?  I thought I
> had because I had just pulled it.
>
>> Are there any volunteers who can put this patch through the mill?  Or
>> Tom, maybe
>> you first want to implement the other stuff you are thinking about?
>
> That's half-done already;
>
> * Mostly, I made the other branches of `cond' use
>   `org-html-make-link' too.  That's done.
>
> * I'd like to cleanup the test whether to inline images.  My first
>   patch had copied it more or less unchanged from what it was.
>
>   Now the bulk of the test is encapsulated in
>   `org-html-should-inline-p', but I've yet to make the code use it.
>
> * And I'd like to make the two sources of name conversion respect
>   each other.
>
>   * There are now two sources of name conversion:
>
>     * The existing code to convert org files which operates if
>       `org-export-html-link-org-files-as-html' is non-nil.
>
>       * This changes url type as well as url path
>
>     * org-html-cvt-link-fn, which supports converting via a lookup
>       table or similar.  I need that for org2blog, which posts to a
>       Blogger-style blog directly from org.
>
>       * Presently this does not treat url type.
>
>     * (For completeness) And it can not convert the name at all.
>
>   * So `org-html-cvt-link-fn' signature may change in the near
>     future, and I'll probably encapsulate the other conversion, and
>     then the logic will just be choose one by precedence.
>
> A couple of notes:
>
> * I am taking "to make sure this does not break anything" to heart
>   and I will write some semblance of a test suite for org-html.  I'll
>   post that soon.

Awesome.  As you have noticed, image inclusion has a number of special  
cases
which I have added over the years and I don't even remember everyting  
in there.
Important things:

- Inlining is an issue
- attribute handling
- figure captions and references to figures



>
>   Everybody, please feel free to contribute any tests of the proper
>   current behavior to it.
>
> * Potential snippage:
>
>   * There is a FIXME comment in `org-export-as-html' that's been there
>     for a long time.  I suspect it's been fixed, because I haven't
>     encountered a problem about not escaping something.
>
>     * "FIXME: do we need to unescape here somewhere?"

Yes, this can be removed - we have not had a problem here for quite a  
while.

>
>   * I don't believe the code just before the `cond' still adds
>     anything useful.  It seems redundant to other code.  But I'd like
>     to hear from others before I dare snip it out.
>
>     That is, the code that begins ";; Make an image out of the
>     description if that is so wanted"

This is for having a clickable Thumbnail - I am not sure if this is  
also handled elsewhere.

Thanks Tom, for taking a look at this mess.


- Carsten

Re: org-html link building diff

[Tom Breton (Tehom)]

The changes are essentially made and pass my tests now, there's mostly
housekeeping now: pull, merge, push.

> Yes.  Send me your name on repo.or.cz and I'll add push for you.
> Please create your own branch and stay on it.

It is "Tehom".

> This is for having a clickable Thumbnail - I am not sure if this is
> also handled elsewhere.

I believe it is.  The only difference seems to be that the first
builds:

 : <a href="foo/target.html"><img src="some.jpg"></a>

all by itself and the second builds:

 : <img src="some.jpg" href="foo/target.html">

thru `org-export-html-format-image'.  Only the second handles
captions.  If the captions etc are the issue, then it should all go
thru the second.  Plus, `org-export-html-format-image' seems to be the
right place for image code.  It would be bad if changes added to
`org-export-html-format-image' didn't take because this other code
handled it instead.

A few questions:

 * Encountered while writing tests: When type is file and path is an
   absolute filename, we do substitutions.  Like "/foo/unfoo/.././baz"
   becomes "/foo/baz".  But we don't do them when path is relative.
   Why not?

   Is that just because we'd then need to make it relative again which
   is more code, or is there some other reason?

 * Also found in the course of testing: "id:" links cause errors when
   buffer is not associated with a file.  This can happen when the arg
   body-only is passed.

   * Punt id links in that case?

   * Do them but avoid the filename relativizing step?

 * How do you feel about url-parse?  It's bundled with emacs, builds
   and destructures urls.  IMO we're not at the stage where it
   provides more help than the extra work it requires yet.

	Tom Breton (Tehom)

Re: org-html link building diff

[Carsten Dominik]

Hi Tom,

On Apr 28, 2010, at 5:01 AM, Tom Breton (Tehom) wrote:

>
> The changes are essentially made and pass my tests now, there's mostly
> housekeeping now: pull, merge, push.
>
>> Yes.  Send me your name on repo.or.cz and I'll add push for you.
>> Please create your own branch and stay on it.
>
> It is "Tehom".

I have added you.

>
>> This is for having a clickable Thumbnail - I am not sure if this is
>> also handled elsewhere.
>
> I believe it is.  The only difference seems to be that the first
> builds:
>
> : <a href="foo/target.html"><img src="some.jpg"></a>
>
> all by itself and the second builds:
>
> : <img src="some.jpg" href="foo/target.html">
>
> thru `org-export-html-format-image'.

Are these equivalent?

My brain is a black hole as to why I might have made two ways.

When you have made you branch, be sure to get Sebastian Rose try it  
out - I think he has lots of image links in his setup.

>  Only the second handles
> captions.  If the captions etc are the issue, then it should all go
> thru the second.  Plus, `org-export-html-format-image' seems to be the
> right place for image code.  It would be bad if changes added to
> `org-export-html-format-image' didn't take because this other code
> handled it instead.
>
> A few questions:
>
> * Encountered while writing tests: When type is file and path is an
>   absolute filename, we do substitutions.  Like "/foo/unfoo/.././baz"
>   becomes "/foo/baz".  But we don't do them when path is relative.
>   Why not?
>
>   Is that just because we'd then need to make it relative again which
>   is more code, or is there some other reason?

Maybe the reason is that the exporten/published result will live  
somewhere else, and a relative path need to remain relative in order  
to make things work correctly.

>
> * Also found in the course of testing: "id:" links cause errors when
>   buffer is not associated with a file.  This can happen when the arg
>   body-only is passed.
>
>   * Punt id links in that case?
>
>   * Do them but avoid the filename relativizing step?


I guess either one i OK with me.  How about the people whi use this  
for jekyl bloggin engines, what would be the right behavior for them?
>
> * How do you feel about url-parse?  It's bundled with emacs, builds
>   and destructures urls.  IMO we're not at the stage where it
>   provides more help than the extra work it requires yet.

If it (our code) works, don't fix it.


- Carsten

Re: org-html link building diff

[Tom Breton (Tehom)]

> Hi Tom,
>
> On Apr 28, 2010, at 5:01 AM, Tom Breton (Tehom) wrote:
>
>>
>> The changes are essentially made and pass my tests now, there's mostly
housekeeping now: pull, merge, push.
>>
>>> Yes.  Send me your name on repo.or.cz and I'll add push for you.
Please create your own branch and stay on it.
>>
>> It is "Tehom".
>
> I have added you.

Oops, when I went to push, I realized that I had capitalized that but it's
apparently not capitalized on repo.or.cz.  It's "tehom".

My branch is called "tehom-master" and the branch that treats link export
based on it is called html-export-refactor-build-link

Tom Breton (Tehom)

Re: org-html link building diff

[Carsten Dominik]

On Apr 30, 2010, at 12:24 AM, Tom Breton (Tehom) wrote:

>> Hi Tom,
>>
>> On Apr 28, 2010, at 5:01 AM, Tom Breton (Tehom) wrote:
>>
>>>
>>> The changes are essentially made and pass my tests now, there's  
>>> mostly
> housekeeping now: pull, merge, push.
>>>
>>>> Yes.  Send me your name on repo.or.cz and I'll add push for you.
> Please create your own branch and stay on it.
>>>
>>> It is "Tehom".
>>
>> I have added you.
>
> Oops, when I went to push, I realized that I had capitalized that  
> but it's
> apparently not capitalized on repo.or.cz.  It's "tehom".

OK, I changed that.

>
> My branch is called "tehom-master" and the branch that treats link  
> export
> based on it is called html-export-refactor-build-link

Will check it out later next week.  Thanks!

- Carsten

Re: org-html link building diff

[Carsten Dominik]

Hi Tom,

what is the status of this development?  Ready for prime time?  If  
yes, can you please rebase to master and send me the pointer to the  
branch again?

Or is there still stuff unclear?  Can I help?

Thanks!

- Carsten

On Apr 30, 2010, at 12:24 AM, Tom Breton (Tehom) wrote:

>> Hi Tom,
>>
>> On Apr 28, 2010, at 5:01 AM, Tom Breton (Tehom) wrote:
>>
>>>
>>> The changes are essentially made and pass my tests now, there's  
>>> mostly
> housekeeping now: pull, merge, push.
>>>
>>>> Yes.  Send me your name on repo.or.cz and I'll add push for you.
> Please create your own branch and stay on it.
>>>
>>> It is "Tehom".
>>
>> I have added you.
>
> Oops, when I went to push, I realized that I had capitalized that  
> but it's
> apparently not capitalized on repo.or.cz.  It's "tehom".
>
> My branch is called "tehom-master" and the branch that treats link  
> export
> based on it is called html-export-refactor-build-link
>
> Tom Breton (Tehom)
>
>
>
>
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

- Carsten

Re: org-html link building diff

[Tom Breton (Tehom)]

> Hi Tom,
>
> what is the status of this development?  Ready for prime time?  If
> yes, can you please rebase to master and send me the pointer to the
> branch again?

I believe it is ready, though I haven't heard any feedback.  I pushed the
changes on about the 4th of may.

> Or is there still stuff unclear?  Can I help?

Maybe.  I hadn't done this before, so some feedback and (if needed)
correction would be appreciated.

I had to change the url to git+ssh so it would authenticate me, but when I
pushed, it was visible in the public (non-ssh) git repo, so I thought it
was visible to everybody.  Isn't it?

I thought I had done what you wanted by creating a branch for all my
changes ever (tehom-master) and rebasing the org-html link changes on that
(html-export-refactor-build-link).  Is that working for you?  If not, how
can I fix it?

Tom Breton (Tehom)

Re: org-html link building diff

[Carsten Dominik]

Hi Tom,

I have now taken a brief look at the html-export-refactor-build-link  
patch.  I see that in addition to changing org-html, it create a new  
file, with tests.  While it is great that you have defined tests, this  
meant that I cannot simply apply he patch to the master without making  
unwanted changes.

If we build a test strucure, it should be in a completely separate  
directory, not in, for example, the lisp directory.

Could you please make me a branch which contains only the changes that  
would go into org?

And if you want to provide a test framework for Org, pleas, by all  
means, go ahead, build it and document it so that we can all use it.

Maybe we could make a "tests" subdirectory in the git repo that would  
contain it...

Thanks

- Carsten

On May 15, 2010, at 11:37 PM, Tom Breton (Tehom) wrote:

>> Hi Tom,
>>
>> what is the status of this development?  Ready for prime time?  If
>> yes, can you please rebase to master and send me the pointer to the
>> branch again?
>
> I believe it is ready, though I haven't heard any feedback.  I  
> pushed the
> changes on about the 4th of may.
>
>> Or is there still stuff unclear?  Can I help?
>
> Maybe.  I hadn't done this before, so some feedback and (if needed)
> correction would be appreciated.
>
> I had to change the url to git+ssh so it would authenticate me, but  
> when I
> pushed, it was visible in the public (non-ssh) git repo, so I  
> thought it
> was visible to everybody.  Isn't it?
>
> I thought I had done what you wanted by creating a branch for all my
> changes ever (tehom-master) and rebasing the org-html link changes  
> on that
> (html-export-refactor-build-link).  Is that working for you?  If  
> not, how
> can I fix it?
>
> Tom Breton (Tehom)
>
>

- Carsten

Re: org-html link building diff

[Carsten Dominik]

On May 15, 2010, at 11:37 PM, Tom Breton (Tehom) wrote:

>> Hi Tom,
>>
>> what is the status of this development?  Ready for prime time?  If
>> yes, can you please rebase to master and send me the pointer to the
>> branch again?
>
> I believe it is ready, though I haven't heard any feedback.  I  
> pushed the
> changes on about the 4th of may.
>
>> Or is there still stuff unclear?  Can I help?
>
> Maybe.  I hadn't done this before, so some feedback and (if needed)
> correction would be appreciated.
>
> I had to change the url to git+ssh so it would authenticate me, but  
> when I
> pushed, it was visible in the public (non-ssh) git repo, so I  
> thought it
> was visible to everybody.  Isn't it?


Yes.  It is the same repo - ssh access allows you to push, that is all.
>
> I thought I had done what you wanted by creating a branch for all my
> changes ever (tehom-master) and rebasing the org-html link changes  
> on that
> (html-export-refactor-build-link).  Is that working for you?  If  
> not, how
> can I fix it?

I think the best would be to have individual topic branches that you  
devellpp for yourself, on your machine at home, and that you only push  
them to the repo when you want others or me to look at them.

- Carsten

Re: org-html link building diff

[Tom Breton (Tehom)]

Hi, Carsten.  The new changes are pushed as
tehom-html-export-refactor-build-link

I couldn't undo the basing onto tehom-master, try though I might.
Something to do with intermediate changes that I couldn't fast-forward
and that I probably edited wrong.  So I made a separate branch,
tehom-html-export-refactor-build-link, which is now pushed to the
repo.  So tehom-master and html-export-refactor-build-link are already
obsolete.

I moved the tests into top-level directory testing, as you asked.  I'm
going to write a separate post describing the test conventions I use.

Now a question: IIUC you want a branch that has no tests, that
"contains only the changes that would go into org".  Since I develop
tests and code at about the same time (tests slightly before code),
I'm not sure how to arrange the branching in a maintainable way.

I could of course make a one-time branch that just removes the testing
directory.  But then what happens for any future fixes?  Seems like
each time I'd have to rebase that branch and pick thru changes and
make it discard each change that deals with testing/.  It seems hard
to maintain.

Tom Breton (Tehom)

Test-file naming conventions - mine and suggested for org

[Tom Breton (Tehom)]

In developing emtest, I've given some thought to conventions for test
files.  My convention is that:

   * tests of foo.el go into foo/tests.el

     * With feature name foo/tests

   * Any shared testhelp goes into foo/testhelp.el

     * With feature name foo/testhelp

   * Example files go in foo/examples/

   * Alternative, they can all be placed in another directory
     hierarchy, like "org/testing/foo/tests.el" as long as
     "org/testing" is in the load-path (emacs doesn't know the
     difference).

I welcome any comments on this convention.  I considered it carefully;
I wrote a small document considering the alternatives and chose this
as best.  But it's young enough that it could be changed, were strong
arguments made towards some alternative.

        Tom Breton (Tehom)

Re: org-html link building diff

[Carsten Dominik]

On May 18, 2010, at 2:59 AM, Tom Breton (Tehom) wrote:

>
> Hi, Carsten.  The new changes are pushed as
> tehom-html-export-refactor-build-link
>
> I couldn't undo the basing onto tehom-master, try though I might.
> Something to do with intermediate changes that I couldn't fast-forward
> and that I probably edited wrong.  So I made a separate branch,
> tehom-html-export-refactor-build-link, which is now pushed to the
> repo.  So tehom-master and html-export-refactor-build-link are already
> obsolete.
>
> I moved the tests into top-level directory testing, as you asked.  I'm
> going to write a separate post describing the test conventions I use.
>
> Now a question: IIUC you want a branch that has no tests, that
> "contains only the changes that would go into org".  Since I develop
> tests and code at about the same time (tests slightly before code),
> I'm not sure how to arrange the branching in a maintainable way.

Hi Tom,

If we decide to use your emtests framework for Org on a brader basis,  
then there would be no reason to have a branch free of tests.  This is  
only necessary because I am supposed to apply your patches, but  
without the testing framework at the moment.

Maybe we should just go ahead and start using emtest for Org-mode.   
Why don't you go ahead and propose this in a mail to emacs-orgmode.   
Lets see if there is any resistance by people who understand more  
about testing than I do.  If not, we go ahead and do it.

I would still prefer to have the tests in a separate directory if your  
package allows to do so.  Hope that this is no problem?

I guess all the emtest code itself would then have to go into the org- 
mode git repo as well, so that we can use it?

And we should develop a make target that will run all tests.

- Carsten


>
> I could of course make a one-time branch that just removes the testing
> directory.  But then what happens for any future fixes?  Seems like
> each time I'd have to rebase that branch and pick thru changes and
> make it discard each change that deals with testing/.  It seems hard
> to maintain.
>
> Tom Breton (Tehom)
>
>

- Carsten

Re: org-html link building diff

[Sebastian Rose]

Carsten Dominik <carsten.dominik@gmail.com> writes:
> On May 18, 2010, at 2:59 AM, Tom Breton (Tehom) wrote:
>
>>
>> Hi, Carsten.  The new changes are pushed as
>> tehom-html-export-refactor-build-link
>>
>> I couldn't undo the basing onto tehom-master, try though I might.
>> Something to do with intermediate changes that I couldn't fast-forward
>> and that I probably edited wrong.  So I made a separate branch,
>> tehom-html-export-refactor-build-link, which is now pushed to the
>> repo.  So tehom-master and html-export-refactor-build-link are already
>> obsolete.
>>
>> I moved the tests into top-level directory testing, as you asked.  I'm
>> going to write a separate post describing the test conventions I use.
>>
>> Now a question: IIUC you want a branch that has no tests, that
>> "contains only the changes that would go into org".  Since I develop
>> tests and code at about the same time (tests slightly before code),
>> I'm not sure how to arrange the branching in a maintainable way.
>
> Hi Tom,
>
> If we decide to use your emtests framework for Org on a brader basis, then there
> would be no reason to have a branch free of tests.  This is  only necessary
> because I am supposed to apply your patches, but  without the testing framework
> at the moment.
>
> Maybe we should just go ahead and start using emtest for Org-mode.  Why don't
> you go ahead and propose this in a mail to emacs-orgmode.   Lets see if there is
> any resistance by people who understand more  about testing than I do.  If not,
> we go ahead and do it.


We had that discussion here aleady, and the result is: we have no test
framework to work with.

Better just do it ;)


  Sebastian

Re: Test-file naming conventions - mine and suggested for org

[Carsten Dominik]

Hi Tom,

as you may or may not have noticed, by merging your patch I actually  
have created the new directory "testing" in the org-mode git repository.

Can I invite you to build this out into a testing system for Org-mode?

Basically, I think, this would mean adding your testing package (is it  
called emt?)
and documenting for other how to create and run tests like the ones  
which you have in
testing/org-html/tests.el

Cheers

- Carsten

On May 18, 2010, at 3:01 AM, Tom Breton (Tehom) wrote:

> In developing emtest, I've given some thought to conventions for test
> files.  My convention is that:
>
>   * tests of foo.el go into foo/tests.el
>
>     * With feature name foo/tests
>
>   * Any shared testhelp goes into foo/testhelp.el
>
>     * With feature name foo/testhelp
>
>   * Example files go in foo/examples/
>
>   * Alternative, they can all be placed in another directory
>     hierarchy, like "org/testing/foo/tests.el" as long as
>     "org/testing" is in the load-path (emacs doesn't know the
>     difference).
>
> I welcome any comments on this convention.  I considered it carefully;
> I wrote a small document considering the alternatives and chose this
> as best.  But it's young enough that it could be changed, were strong
> arguments made towards some alternative.
>
>        Tom Breton (Tehom)
>
>

- Carsten

Proposal: Emtest as tester

[Tom Breton (Tehom)]

At Carsten's request, I am proposing emtest as the tester for
org-mode.  I would like to hear if there are any objections or
questions.

****** About Emtest

Emtest is an emacs-based test framework.  It reads tests, runs them on
command and presents their results.  Test suites can be run by suite,
by clause, or by library.

It is extensible and modular.  Nearly everything about it can be
replaced or extended.

One important feature is its testhelp libraries:

 * mocks/filebuf - for making mock files and buffers to run tests in.
 * mocks/dirtree - for making mock directory trees.
 * deep-type-checker - for testing that objects, especially
   structures, are type-correct right down to their leaves.
 * match - for pattern-matching.  When you want to test return values
   or similar, but some fields or elements don't have stable values
   (say, a timestamp or a UUID).
 * tagnames - extremely useful for defining test data and iterating
   over examples.
 * testpoint - useful for:
   1. testing functionality that is called deep inside something else,
      where writing a viable test would mean nearly cloning the
      something else to get the calling conditions right.
   2. Testing functionality that uses other functionality that can't
      be easily controlled by passing arguments.
   3. Testing that under given circumstances a certain point is
      reached, not reached, or reached the right number of times.

Also, in less than perfect shape right now:

 * mocks/keystuffer - work in progress, for capturing canned user input
 * misc and standard - standard testhelp functions.  Works but
   undergoing reorganization.
 * types - type specifications, extending what cl provides.  Right
   now, just a few that I needed.
 * persist - useful for tests of inspected output.  Not working right
   now due to redesign of an underlying package.

****** Some questions

  * Where to include it:

    * I'm proposing to put it under org-mode/testing/ So the directory
      structure would look like:

      * org-mode

       	* lisp

       	* (etc)

       	* testing

	  * emtest

	    * Many files

	  * Some support
	  * packages emtest
	  * uses.

	  * org-agenda

	    * tests.el

	    * (And other test files)

	  * org-archive

	    * tests.el

	  * org-ascii

	  * etc (the other org files' directories of test files)

       	* (other existing org directories)

  * Should testing of contrib files be in a separate directory?  It's
    not clear to me that it needs to be or should be.

  * Loading.

    Of course this shouldn't require much extra work to build and
    install.  Yet there's a case to be made for not building or
    installing it by default, "them that don't use it doesn't pay a
    cost".

    So I'm thinking I should add another target to the makefile to
    install it, as well as (of course) a test target.

  * How to include it, git-wise.

    What git wants to do with included external projects is to make
    them submodules.  However, I'm told that's a pain to deal with,
    moreso from the other end than from mine.  And it does seem like it
    would be.  Basically git treats a submodule as a single thing, but
    still "signs" it version-wise with a hex ID, and wants it to be the
    correct version.  So git insulates you just a little bit, at the
    cost of having to deal with an additional repository.

    So I'm thinking I'd just include it literally and if that proves
    hard to maintain then we still have the other option.


	 Tom Breton (Tehom)

Re: Proposal: Emtest as tester

[Dan Davison]

"Tom Breton (Tehom)" <tehom@panix.com> writes:

> At Carsten's request, I am proposing emtest as the tester for
> org-mode.  I would like to hear if there are any objections or
> questions.

Hi Tom,

My googling didn't manage to find emtest -- where does the code live at
the moment? Is there an Org repo out there demonstrating how it would
integrate, and/or some documentation and examples of usage?

Dan

>
> ****** About Emtest
>
> Emtest is an emacs-based test framework.  It reads tests, runs them on
> command and presents their results.  Test suites can be run by suite,
> by clause, or by library.
>
> It is extensible and modular.  Nearly everything about it can be
> replaced or extended.
>
> One important feature is its testhelp libraries:
>
>  * mocks/filebuf - for making mock files and buffers to run tests in.
>  * mocks/dirtree - for making mock directory trees.
>  * deep-type-checker - for testing that objects, especially
>    structures, are type-correct right down to their leaves.
>  * match - for pattern-matching.  When you want to test return values
>    or similar, but some fields or elements don't have stable values
>    (say, a timestamp or a UUID).
>  * tagnames - extremely useful for defining test data and iterating
>    over examples.
>  * testpoint - useful for:
>    1. testing functionality that is called deep inside something else,
>       where writing a viable test would mean nearly cloning the
>       something else to get the calling conditions right.
>    2. Testing functionality that uses other functionality that can't
>       be easily controlled by passing arguments.
>    3. Testing that under given circumstances a certain point is
>       reached, not reached, or reached the right number of times.
>
> Also, in less than perfect shape right now:
>
>  * mocks/keystuffer - work in progress, for capturing canned user input
>  * misc and standard - standard testhelp functions.  Works but
>    undergoing reorganization.
>  * types - type specifications, extending what cl provides.  Right
>    now, just a few that I needed.
>  * persist - useful for tests of inspected output.  Not working right
>    now due to redesign of an underlying package.
>
> ****** Some questions
>
>   * Where to include it:
>
>     * I'm proposing to put it under org-mode/testing/ So the directory
>       structure would look like:
>
>       * org-mode
>
>        	* lisp
>
>        	* (etc)
>
>        	* testing
>
> 	  * emtest
>
> 	    * Many files
>
> 	  * Some support
> 	  * packages emtest
> 	  * uses.
>
> 	  * org-agenda
>
> 	    * tests.el
>
> 	    * (And other test files)
>
> 	  * org-archive
>
> 	    * tests.el
>
> 	  * org-ascii
>
> 	  * etc (the other org files' directories of test files)
>
>        	* (other existing org directories)
>
>   * Should testing of contrib files be in a separate directory?  It's
>     not clear to me that it needs to be or should be.
>
>   * Loading.
>
>     Of course this shouldn't require much extra work to build and
>     install.  Yet there's a case to be made for not building or
>     installing it by default, "them that don't use it doesn't pay a
>     cost".
>
>     So I'm thinking I should add another target to the makefile to
>     install it, as well as (of course) a test target.
>
>   * How to include it, git-wise.
>
>     What git wants to do with included external projects is to make
>     them submodules.  However, I'm told that's a pain to deal with,
>     moreso from the other end than from mine.  And it does seem like it
>     would be.  Basically git treats a submodule as a single thing, but
>     still "signs" it version-wise with a hex ID, and wants it to be the
>     correct version.  So git insulates you just a little bit, at the
>     cost of having to deal with an additional repository.
>
>     So I'm thinking I'd just include it literally and if that proves
>     hard to maintain then we still have the other option.
>
>
> 	 Tom Breton (Tehom)
>
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Proposal: Emtest as tester

[Carsten Dominik]

On May 24, 2010, at 11:26 PM, Tom Breton (Tehom) wrote:

> At Carsten's request, I am proposing emtest as the tester for
> org-mode.  I would like to hear if there are any objections or
> questions.
>
> ****** About Emtest
>
> Emtest is an emacs-based test framework.  It reads tests, runs them on
> command and presents their results.  Test suites can be run by suite,
> by clause, or by library.
>
> It is extensible and modular.  Nearly everything about it can be
> replaced or extended.
>
> One important feature is its testhelp libraries:
>
> * mocks/filebuf - for making mock files and buffers to run tests in.
> * mocks/dirtree - for making mock directory trees.
> * deep-type-checker - for testing that objects, especially
>   structures, are type-correct right down to their leaves.
> * match - for pattern-matching.  When you want to test return values
>   or similar, but some fields or elements don't have stable values
>   (say, a timestamp or a UUID).
> * tagnames - extremely useful for defining test data and iterating
>   over examples.
> * testpoint - useful for:
>   1. testing functionality that is called deep inside something else,
>      where writing a viable test would mean nearly cloning the
>      something else to get the calling conditions right.
>   2. Testing functionality that uses other functionality that can't
>      be easily controlled by passing arguments.
>   3. Testing that under given circumstances a certain point is
>      reached, not reached, or reached the right number of times.
>
> Also, in less than perfect shape right now:
>
> * mocks/keystuffer - work in progress, for capturing canned user input
> * misc and standard - standard testhelp functions.  Works but
>   undergoing reorganization.
> * types - type specifications, extending what cl provides.  Right
>   now, just a few that I needed.
> * persist - useful for tests of inspected output.  Not working right
>   now due to redesign of an underlying package.
>
> ****** Some questions
>
>  * Where to include it:
>
>    * I'm proposing to put it under org-mode/testing/ So the directory
>      structure would look like:
>
>      * org-mode
>
>       	* lisp
>
>       	* (etc)
>
>       	* testing
>
> 	  * emtest
>
> 	    * Many files
>
> 	  * Some support
> 	  * packages emtest
> 	  * uses.
>
> 	  * org-agenda
>
> 	    * tests.el
>
> 	    * (And other test files)
>
> 	  * org-archive
>
> 	    * tests.el
>
> 	  * org-ascii
>
> 	  * etc (the other org files' directories of test files)
>
>       	* (other existing org directories)
>
>  * Should testing of contrib files be in a separate directory?  It's
>    not clear to me that it needs to be or should be.
>
>  * Loading.
>
>    Of course this shouldn't require much extra work to build and
>    install.  Yet there's a case to be made for not building or
>    installing it by default, "them that don't use it doesn't pay a
>    cost".
>
>    So I'm thinking I should add another target to the makefile to
>    install it, as well as (of course) a test target.

Yes, I agree.

>
>  * How to include it, git-wise.
>
>    What git wants to do with included external projects is to make
>    them submodules.  However, I'm told that's a pain to deal with,
>    moreso from the other end than from mine.  And it does seem like it
>    would be.  Basically git treats a submodule as a single thing, but
>    still "signs" it version-wise with a hex ID, and wants it to be the
>    correct version.  So git insulates you just a little bit, at the
>    cost of having to deal with an additional repository.
>
>    So I'm thinking I'd just include it literally and if that proves
>    hard to maintain then we still have the other option.

I agree with you.

- Carsten

Re: Proposal: Emtest as tester

[Martin Pohlack]

Hi Tom,

On 24.05.2010 23:26, Tom Breton (Tehom) wrote:
> At Carsten's request, I am proposing emtest as the tester for
> org-mode.  I would like to hear if there are any objections or
> questions.
>
> ****** About Emtest
>
[...]

I made an attempt earlier to propose a testing framework and wrote some 
thoughts in an email thread:

http://www.mail-archive.com/emacs-orgmode@gnu.org/msg22558.html

Would your proposal cover my example-based approach?  How hard would it 
be to turn a typical bug report into a test case?

Cheers,
Martin

Re: Proposal: Emtest as tester

[Tom Breton (Tehom)]

> "Tom Breton (Tehom)" <tehom@panix.com> writes:
>
>> At Carsten's request, I am proposing emtest as the tester for
>> org-mode.  I would like to hear if there are any objections or
>> questions.
>
> Hi Tom,
>
> My googling didn't manage to find emtest -- where does the code live at
> the moment?

At the moment it lives in a tarball.  I will put up an updated version at
panix.com/~tehom/my-code/emtest-bundle.tar.gz  That's emtest plus all
supporting packages.  (Should be all, I may have missed some dependencies
I have locally)

I'm looking to have it hosted on Savannah. You've given me the impetus to
go and do that, so thanks.

> Is there an Org repo out there demonstrating how it would
> integrate,

Right now the only git repo lives on my machine.

> and/or some documentation and examples of usage?

Right now the documentation consists of my design notes, which are
probably confusing since they include as-built, unrealized plans,
self-debates over the right approach, discarded and/or obsolete plans,
bugfixes, and progress notes.  I will try to write up something more
accessible.

Every feature has examples of usage in the tests.el files, one of the
virtues of tests.

Tom Breton (Tehom)

Re: Proposal: Emtest as tester

[Tom Breton (Tehom)]

> Hi Tom,
>
>>
> [...]
>
> I made an attempt earlier to propose a testing framework and wrote some
> thoughts in an email thread:
>
> http://www.mail-archive.com/emacs-orgmode@gnu.org/msg22558.html

I read that.  Interesting.

> Would your proposal cover my example-based approach?  How hard would it
> be to turn a typical bug report into a test case?

Not hard at all.  Most of what you propose is already supported.  You
would put both the input and expected output in FOO/examples/UNIQUE-NAMES

The sequence of commands can easily be made into a test case. 
`emt:insert' inserts the template for you (choose `emt:insert-test'), then
paste it in.

Rather than saving the buffer, you probably want to do it all inside a
mock buffer, like (excuse the online-editor formatting)

(emtb:with-buf (:file INPUT-FILE-NAME)
(operations)
(emtb:buf-contents-matches (:file OUTPUT-FILE-NAME))

For testing only visible output, you'd presumably use org-export-visible
and compare that buffer's contents to what's expected.

Wildcard comparisons in files are not yet supported.  I will take that as
a feature request.

Tom Breton (Tehom)

Re: Proposal: Emtest as tester

[Benjamin Andresen]

Hey Tom,

I was wondering what the status was of the emtest inclusion in org-mode?

Looks like the tehom-master in the org-mode repo is not the right one
(last commit Wed Apr 28 16:39:59 2010 -0500)

br,
benny