* Re: Discussion request: 15m tangle time, details follow
2014-06-18 0:16 Discussion request: 15m tangle time, details follow Grant Rettke
@ 2014-06-18 2:41 ` Aaron Ecay
2014-06-18 8:13 ` Sebastien Vauban
` (2 more replies)
2014-06-18 2:54 ` Nick Dokos
` (2 subsequent siblings)
3 siblings, 3 replies; 17+ messages in thread
From: Aaron Ecay @ 2014-06-18 2:41 UTC (permalink / raw)
To: Grant Rettke, emacs-orgmode@gnu.org
[-- Attachment #1: Type: text/plain, Size: 964 bytes --]
Hi Grant,
2014ko ekainak 17an, Grant Rettke-ek idatzi zuen:
>
> Good evening,
>
> Over the past few months I've been working on the same literate
> document. It has been a learning
> experience for me, trial and error has abounded. The key tenet that
> I've adhered too though is to truly
> embrace literate programming, and the more I learn the more it makes
> sense. The document has
> grown quite organically and it has been and continues to be a
> wonderful experience. What I need
> help, feedback, discussion, and more on is the build time.
>
> The average build takes 15m.
Here you mean time to tangle, correct? (As opposed to exporting to
HTML/LaTeX/etc.)
I can confirm very long times to tangle a document with a structure like
yours. I ran the emacs profiler
<https://www.gnu.org/software/emacs/manual/html_node/elisp/Profiling.html>
while tangling the document for 30 secs, then interrupted with C-g and
generated a report. That is attached.
[-- Attachment #2: profile.txt --]
[-- Type: text/plain, Size: 11000 bytes --]
- ... 29140 96%
- org-babel-expand-noweb-references 27746 92%
- let* 27746 92%
- let 27746 92%
- save-current-buffer 27746 92%
- unwind-protect 27746 92%
- progn 27746 92%
- while 27746 92%
- funcall 27746 92%
- save-current-buffer 27746 92%
- save-restriction 27746 92%
- mapconcat 27746 92%
- split-string 27746 92%
- if 27746 92%
- or 27746 92%
- let 27743 92%
- save-excursion 27743 92%
- if 27743 92%
- let* 27743 92%
- let 27743 92%
- unwind-protect 27743 92%
- progn 27743 92%
- while 27743 92%
- if 27696 92%
- progn 27475 91%
- let 27457 91%
- let 27411 91%
- org-babel-get-src-block-info 27218 90%
- org-babel-parse-src-block-match 27093 90%
- org-babel-params-from-properties 26595 88%
- let 26595 88%
- unwind-protect 26592 88%
- progn 26585 88%
- list 26585 88%
- let 26218 87%
- org-babel-parse-multiple-vars 26218 87%
- delq 26212 87%
- mapcar 26205 87%
- org-babel-params-from-properties-inner1 26184 86%
- let 26178 86%
- and 26175 86%
- setq 26152 86%
- org-entry-get 26135 86%
- let 26119 86%
- save-excursion 26095 86%
- save-excursion 26051 86%
- if 26041 86%
- org-entry-get-with-inheritance 25257 83%
- byte-code 25179 83%
- org-entry-get 20968 69%
- let 20951 69%
- save-excursion 20898 69%
- save-excursion 20859 69%
- if 20834 69%
- if 20821 69%
- save-excursion 20699 68%
- save-restriction 20674 68%
- let 20640 68%
- org-get-property-block 18232 60%
- byte-code 15555 51%
+ org-back-to-heading 2511 8%
outline-next-heading 2105 6%
org-before-first-heading-p 1764 5%
+ if 2325 7%
+ goto-char 3 0%
+ org-up-heading-safe 4078 13%
+ org-back-to-heading 8 0%
+ and 14 0%
goto-char 3 0%
if 6 0%
cons 11 0%
+ mapcar 12 0%
mapc 6 0%
+ if 198 0%
+ org-babel-parse-header-arguments 169 0%
+ apply 188 0%
+ byte-code 163 0%
generate-new-buffer 61 0%
+ org-do-remove-indentation 29 0%
+ org-unescape-code-in-string 22 0%
org-babel-where-is-src-block-head 15 0%
last 3 0%
+ if 73 0%
+ org-babel-active-location-p 221 0%
+ save-excursion 3 0%
Automatic GC 1394 4%
+ command-execute 900 2%
+ timer-event-handler 63 0%
[-- Attachment #3: Type: text/plain, Size: 3448 bytes --]
I did two non-standard things to this profile. The first was:
(setq profiler-report-cpu-line-format
'((100 left)
;; The 100 above is increased from the default of 50
;; to allow the deeply nested call tree to be seen
(24 right ((19 right)
(5 right)))))
The second was to convert an anonymous lambda found in
org-babel-params-from-properties into a named function, so that it would
show up in the profiling results on its own line:
(defun org-babel-params-from-properties-inner1 (header-arg)
(let (val)
(and (setq val (org-entry-get (point) header-arg t))
(cons (intern (concat ":" header-arg))
(org-babel-read val)))))
The profile shows that most of the slowdown is in org-entry-get. Indeed,
org-babel-params-from-properties calls this function ~30 times per source
block. When called with the inherit arg set to t (as here), this function
takes time (at least) proportional to the number of headings dominating
the source block, which in your document can be up to 5.
I think there are two problems here. The first is the situation where
babel needs to fetch 30 properties per source block. Indeed, this is
marked “deprecated” in the source, in favor of a system where there is
only one header arg. This has been marked deprecated for almost exactly
a year in the code (Achim’s commit 90b16870 of 2013-06-23), but I don’t
know of any prominent announcement of the deprecation. So I doubt the
old slow code could be removed without breaking many people’s setups,
although possibly a customization variable could be introduced to allow
users to opt in to the new, faster system. You’d then have to update
your file:
:PROPERTIES:
:exports: none
:tangle: no
:END:
becomes
:PROPERTIES:
:header-args: :exports none :tangle no
:END:
The new system is also a bit inferior, in that it doesn’t allow header
arg inheritance as easily. So with the one-prop-per-arg system the
following works as expected:
* foo
:PROPERTIES:
:exports: none
:END:
** bar
:PROPERTIES:
:tangle: no
:END:
(src block here)
On the other hand, in the new system there’s no way to specify some
header args at foo and some at bar; the lowest header-args property
wins. (At least as far as I can see)
The second issue is that it’s desirable to memoize calls to
org-entry-get. Probably the easiest way to do this is to use the
org-element cache. Indeed, a quick and hacky test that I did seemed to
confirm that this yields some speedup. There are conceptual issues
though – org-element forces all property keys to be uppercase, whereas
org-entry-get (as near as I can tell...) follows the user’s
customization of case-fold-search to determine its case sensitivity. So
one has to think carefully about how a rewrite to use org-element might
affect the case-sensitivity of the property API (although code relying
on the API to be sensitive to case of property keys might be rare in
practice).
TL;DR:
1. I see the same slowness you report
2. It seems like an architectural issue rather than one of
(mis)configuration
3. There are broad fixes available, but they require potentially
compatibility-breaking changes to Org
4. (maybe with this analysis someone can come up with a more targeted
fix for your use case)
Hope this is helpful,
--
Aaron Ecay
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: Discussion request: 15m tangle time, details follow
2014-06-18 2:41 ` Aaron Ecay
@ 2014-06-18 8:13 ` Sebastien Vauban
2014-06-18 17:47 ` Aaron Ecay
2014-06-18 20:59 ` Eric Schulte
2014-06-19 0:56 ` Grant Rettke
2 siblings, 1 reply; 17+ messages in thread
From: Sebastien Vauban @ 2014-06-18 8:13 UTC (permalink / raw)
To: emacs-orgmode-mXXj517/zsQ
Hi Aaron,
Aaron Ecay wrote:
> [...]
> babel needs to fetch 30 properties per source block. Indeed, this is
> marked “deprecated” in the source, in favor of a system where there is
> only one header arg. This has been marked deprecated for almost exactly
> a year in the code (Achim’s commit 90b16870 of 2013-06-23), but I don’t
> know of any prominent announcement of the deprecation.
I neither was aware of such a deprecation.
Are you talking of the comment in function
`org-babel-params-from-properties' (in ob-core.el)?
Thought, I can't parse it yet the way you do -- without understanding
much more of that code, as the comments differ in "at point of
definition" vs "at point of call":
;; DEPRECATED header arguments specified as separate property at
;; point of definition
;; header arguments specified with the header-args property at
;; point of call
What you're talking about is for specifying header arguments in
a subtree, anyway always at the same point:
> [...] You’d then have to update your file:
>
> :PROPERTIES:
> :exports: none
> :tangle: no
> :END:
>
> becomes
>
> :PROPERTIES:
> :header-args: :exports none :tangle no
> :END:
>
> The new system is also a bit inferior, in that it doesn’t allow header
> arg inheritance as easily. So with the one-prop-per-arg system the
> following works as expected:
>
> * foo
> :PROPERTIES:
> :exports: none
> :END:
> ** bar
> :PROPERTIES:
> :tangle: no
> :END:
>
> (src block here)
>
> On the other hand, in the new system there’s no way to specify some
> header args at foo and some at bar; the lowest header-args property
> wins. (At least as far as I can see)
Maybe the "+" mechanism for concatenating property strings would help
here?
Best regards,
Seb
--
Sebastien Vauban
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: Discussion request: 15m tangle time, details follow
2014-06-18 8:13 ` Sebastien Vauban
@ 2014-06-18 17:47 ` Aaron Ecay
0 siblings, 0 replies; 17+ messages in thread
From: Aaron Ecay @ 2014-06-18 17:47 UTC (permalink / raw)
To: Sebastien Vauban, emacs-orgmode
Hi Sebastien,
2014ko ekainak 18an, Sebastien Vauban-ek idatzi zuen:
>
> Hi Aaron,
>
> Aaron Ecay wrote:
>> [...]
>> babel needs to fetch 30 properties per source block. Indeed, this is
>> marked “deprecated” in the source, in favor of a system where there is
>> only one header arg. This has been marked deprecated for almost exactly
>> a year in the code (Achim’s commit 90b16870 of 2013-06-23), but I don’t
>> know of any prominent announcement of the deprecation.
>
> I neither was aware of such a deprecation.
>
> Are you talking of the comment in function
> `org-babel-params-from-properties' (in ob-core.el)?
>
> Thought, I can't parse it yet the way you do -- without understanding
> much more of that code, as the comments differ in "at point of
> definition" vs "at point of call":
>
> ;; DEPRECATED header arguments specified as separate property at
> ;; point of definition
>
> ;; header arguments specified with the header-args property at
> ;; point of call
That also differs between the two methods.
,----
|
| * foo
| :PROPERTIES:...
|
| #+name: xyz
| #+begin_src
| ...
| #+end_src
|
| * bar
| :PROPERTIES:...
|
| #+call: xyz()
`----
The current code looks for individual header arg properties at foo, but
for the property header-args at bar (for the #+call line)
>
> What you're talking about is for specifying header arguments in
> a subtree, anyway always at the same point:
>
>> [...] You’d then have to update your file:
>>
>> :PROPERTIES:
>> :exports: none
>> :tangle: no
>> :END:
>>
>> becomes
>>
>> :PROPERTIES:
>> :header-args: :exports none :tangle no
>> :END:
>>
>> The new system is also a bit inferior, in that it doesn’t allow header
>> arg inheritance as easily. So with the one-prop-per-arg system the
>> following works as expected:
>>
>> * foo
>> :PROPERTIES:
>> :exports: none
>> :END:
>> ** bar
>> :PROPERTIES:
>> :tangle: no
>> :END:
>>
>> (src block here)
>>
>> On the other hand, in the new system there’s no way to specify some
>> header args at foo and some at bar; the lowest header-args property
>> wins. (At least as far as I can see)
>
> Maybe the "+" mechanism for concatenating property strings would help
> here?
No, org-entry-get-with-inheritance will not continue climbing the tree
once it finds one instance of the property in question, as demonstrated
here:
,----
| * foo
| :PROPERTIES:
| :quux+: abc
| :END:
| ** bar
| :PROPERTIES:
| :quux+: foo
| :quux+: bar
| :END:
|
| #+BEGIN_SRC emacs-lisp
| (org-entry-get nil "quux" t)
| #+END_SRC
|
| #+RESULTS:
| : foo bar
`----
--
Aaron Ecay
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Discussion request: 15m tangle time, details follow
2014-06-18 2:41 ` Aaron Ecay
2014-06-18 8:13 ` Sebastien Vauban
@ 2014-06-18 20:59 ` Eric Schulte
2014-06-19 1:05 ` Grant Rettke
2014-06-19 0:56 ` Grant Rettke
2 siblings, 1 reply; 17+ messages in thread
From: Eric Schulte @ 2014-06-18 20:59 UTC (permalink / raw)
To: Grant Rettke; +Cc: emacs-orgmode@gnu.org
Aaron Ecay <aaronecay@gmail.com> writes:
> Hi Grant,
>
> 2014ko ekainak 17an, Grant Rettke-ek idatzi zuen:
>>
>> Good evening,
>>
>> Over the past few months I've been working on the same literate
>> document. It has been a learning
>> experience for me, trial and error has abounded. The key tenet that
>> I've adhered too though is to truly
>> embrace literate programming, and the more I learn the more it makes
>> sense. The document has
>> grown quite organically and it has been and continues to be a
>> wonderful experience. What I need
>> help, feedback, discussion, and more on is the build time.
>>
>> The average build takes 15m.
>
> Here you mean time to tangle, correct? (As opposed to exporting to
> HTML/LaTeX/etc.)
>
> I can confirm very long times to tangle a document with a structure like
> yours. I ran the emacs profiler
> <https://www.gnu.org/software/emacs/manual/html_node/elisp/Profiling.html>
> while tangling the document for 30 secs, then interrupted with C-g and
> generated a report. That is attached.
>
>
>
> I did two non-standard things to this profile. The first was:
>
> (setq profiler-report-cpu-line-format
> '((100 left)
> ;; The 100 above is increased from the default of 50
> ;; to allow the deeply nested call tree to be seen
> (24 right ((19 right)
> (5 right)))))
>
> The second was to convert an anonymous lambda found in
> org-babel-params-from-properties into a named function, so that it would
> show up in the profiling results on its own line:
>
> (defun org-babel-params-from-properties-inner1 (header-arg)
> (let (val)
> (and (setq val (org-entry-get (point) header-arg t))
> (cons (intern (concat ":" header-arg))
> (org-babel-read val)))))
>
> The profile shows that most of the slowdown is in org-entry-get. Indeed,
> org-babel-params-from-properties calls this function ~30 times per source
> block. When called with the inherit arg set to t (as here), this function
> takes time (at least) proportional to the number of headings dominating
> the source block, which in your document can be up to 5.
>
Thanks for taking the time to profile this. It's nice to have more
evidence that the use of properties is definitely the culprit here.
>
> I think there are two problems here. The first is the situation where
> babel needs to fetch 30 properties per source block. Indeed, this is
> marked “deprecated” in the source, in favor of a system where there is
> only one header arg. This has been marked deprecated for almost exactly
> a year in the code (Achim’s commit 90b16870 of 2013-06-23), but I don’t
> know of any prominent announcement of the deprecation. So I doubt the
> old slow code could be removed without breaking many people’s setups,
> although possibly a customization variable could be introduced to allow
> users to opt in to the new, faster system. You’d then have to update
> your file:
>
> :PROPERTIES:
> :exports: none
> :tangle: no
> :END:
>
> becomes
>
> :PROPERTIES:
> :header-args: :exports none :tangle no
> :END:
>
> The new system is also a bit inferior, in that it doesn’t allow header
> arg inheritance as easily. So with the one-prop-per-arg system the
> following works as expected:
>
> * foo
> :PROPERTIES:
> :exports: none
> :END:
> ** bar
> :PROPERTIES:
> :tangle: no
> :END:
>
> (src block here)
>
> On the other hand, in the new system there’s no way to specify some
> header args at foo and some at bar; the lowest header-args property
> wins. (At least as far as I can see)
>
As I recall this inheritance issue is the wall that we ran up against.
The deprecation comment in the code was premature.
>
> The second issue is that it’s desirable to memoize calls to
> org-entry-get. Probably the easiest way to do this is to use the
> org-element cache. Indeed, a quick and hacky test that I did seemed to
> confirm that this yields some speedup. There are conceptual issues
> though – org-element forces all property keys to be uppercase, whereas
> org-entry-get (as near as I can tell...) follows the user’s
> customization of case-fold-search to determine its case sensitivity. So
> one has to think carefully about how a rewrite to use org-element might
> affect the case-sensitivity of the property API (although code relying
> on the API to be sensitive to case of property keys might be rare in
> practice).
>
Thanks, it does sound like org-element cache could be useful here, I
don't believe this existed last time we wrestled with this performance
issue.
The only other options I can think of are;
- introduce a customization variable to eliminate or limit the use of
property lookup for code blocks to either perform none or to only
search for a limited set of properties
- possibly extend org-element-get (or provide an alternative) which
takes multiple keys (which may be more efficient depending on the
implementation)
>
> TL;DR:
>
> 1. I see the same slowness you report
> 2. It seems like an architectural issue rather than one of
> (mis)configuration
> 3. There are broad fixes available, but they require potentially
> compatibility-breaking changes to Org
> 4. (maybe with this analysis someone can come up with a more targeted
> fix for your use case)
>
> Hope this is helpful,
Very helpful, thanks for providing both empirical data and useful
analysis.
Best,
Eric
--
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D (see https://u.fsf.org/yw)
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Discussion request: 15m tangle time, details follow
2014-06-18 20:59 ` Eric Schulte
@ 2014-06-19 1:05 ` Grant Rettke
2014-06-19 13:44 ` Eric Schulte
0 siblings, 1 reply; 17+ messages in thread
From: Grant Rettke @ 2014-06-19 1:05 UTC (permalink / raw)
To: Eric Schulte; +Cc: emacs-orgmode@gnu.org
I still want to be using org,, so my plan for now then is to write a
pre-processing script for before org-babel-tangle is run that:
1. Checks each headline
2. Checks if there is a source block
3. If that source block doesn't have a noweb-ref name, and there is
another source block under that
headline, then "merge" it with the next source block of the same
nature in that heading.
That is off the top of my head, and I will try it by hand, first and
see if it tangles the same result.
Thoughts?
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: Discussion request: 15m tangle time, details follow
2014-06-19 1:05 ` Grant Rettke
@ 2014-06-19 13:44 ` Eric Schulte
2014-06-20 0:01 ` Grant Rettke
0 siblings, 1 reply; 17+ messages in thread
From: Eric Schulte @ 2014-06-19 13:44 UTC (permalink / raw)
To: Grant Rettke; +Cc: emacs-orgmode@gnu.org
Grant Rettke <gcr@wisdomandwonder.com> writes:
> I still want to be using org,, so my plan for now then is to write a
> pre-processing script for before org-babel-tangle is run that:
>
> 1. Checks each headline
> 2. Checks if there is a source block
> 3. If that source block doesn't have a noweb-ref name, and there is
> another source block under that
> headline, then "merge" it with the next source block of the same
> nature in that heading.
>
> That is off the top of my head, and I will try it by hand, first and
> see if it tangles the same result.
>
> Thoughts?
Perhaps an easier workaround would be to customize the value of
`org-babel-common-header-args-w-values'. This variable determines which
header arguments are checked for in properties. If you remove all
header arguments from this variable which you know are not used in a
property, it may dramatically speed up tangle time.
This may reduce the utility of some convenience functions, but should
not have too large of an impact on overall functionality.
Best,
Eric
--
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D (see https://u.fsf.org/yw)
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Discussion request: 15m tangle time, details follow
2014-06-19 13:44 ` Eric Schulte
@ 2014-06-20 0:01 ` Grant Rettke
2014-06-20 1:26 ` Grant Rettke
0 siblings, 1 reply; 17+ messages in thread
From: Grant Rettke @ 2014-06-20 0:01 UTC (permalink / raw)
To: Eric Schulte; +Cc: emacs-orgmode@gnu.org
Understood. I will start testing.
Just out of curiosity, knowing full well that it will break the
tangle, and also since I hadn't tried
the quick and dirty flag, I ran it with this to see how long it took:
(setq org-babel-common-header-args-w-values nil)
It went from 936 seconds (15 minutes)
Down to 279 (4 minutes).
Now I am wondering if it is worth converting my document *not* to use
property inheritance...
What would you folks do?
Grant Rettke | ACM, ASA, FSF, IEEE, SIAM
gcr@wisdomandwonder.com | http://www.wisdomandwonder.com/
“Wisdom begins in wonder.” --Socrates
((λ (x) (x x)) (λ (x) (x x)))
“Life has become immeasurably better since I have been forced to stop
taking it seriously.” --Thompson
On Thu, Jun 19, 2014 at 8:44 AM, Eric Schulte <schulte.eric@gmail.com> wrote:
> Grant Rettke <gcr@wisdomandwonder.com> writes:
>
>> I still want to be using org,, so my plan for now then is to write a
>> pre-processing script for before org-babel-tangle is run that:
>>
>> 1. Checks each headline
>> 2. Checks if there is a source block
>> 3. If that source block doesn't have a noweb-ref name, and there is
>> another source block under that
>> headline, then "merge" it with the next source block of the same
>> nature in that heading.
>>
>> That is off the top of my head, and I will try it by hand, first and
>> see if it tangles the same result.
>>
>> Thoughts?
>
> Perhaps an easier workaround would be to customize the value of
> `org-babel-common-header-args-w-values'. This variable determines which
> header arguments are checked for in properties. If you remove all
> header arguments from this variable which you know are not used in a
> property, it may dramatically speed up tangle time.
>
> This may reduce the utility of some convenience functions, but should
> not have too large of an impact on overall functionality.
>
> Best,
> Eric
>
> --
> Eric Schulte
> https://cs.unm.edu/~eschulte
> PGP: 0x614CA05D (see https://u.fsf.org/yw)
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Discussion request: 15m tangle time, details follow
2014-06-20 0:01 ` Grant Rettke
@ 2014-06-20 1:26 ` Grant Rettke
2014-06-20 14:50 ` Grant Rettke
0 siblings, 1 reply; 17+ messages in thread
From: Grant Rettke @ 2014-06-20 1:26 UTC (permalink / raw)
To: Eric Schulte; +Cc: emacs-orgmode@gnu.org
Just excluded what seemed excludable and got build time down to 8
minutes from 15 minutes.
Kept the following as it seems that it would allow the most important things:
(session . :any)
(noweb-ref . :any)
(noweb . ((yes no tangle no-export strip-export)))
(file . :any)
(tangle . ((tangle yes no :any)))
(results . ((file list vector table scalar verbatim)
(raw html latex org code pp drawer)
(replace silent none append prepend)
(output value)))
Here is what I excluded:
(setq org-babel-common-header-args-w-values
(assq-delete-all 'cache org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'cmdline org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'colnames org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'comments org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'dir org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'eval org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'exports org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'epilogue org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'file-desc org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'hlines org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'mkdirp org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'no-expand org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'noeval org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'noweb-sep org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'padline org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'post org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'prologue org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'rownames org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'sep org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'shebang org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'tangle-mode org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'var org-babel-common-header-args-w-values))
(setq org-babel-common-header-args-w-values
(assq-delete-all 'wrap org-babel-common-header-args-w-values))
Grant Rettke | ACM, ASA, FSF, IEEE, SIAM
gcr@wisdomandwonder.com | http://www.wisdomandwonder.com/
“Wisdom begins in wonder.” --Socrates
((λ (x) (x x)) (λ (x) (x x)))
“Life has become immeasurably better since I have been forced to stop
taking it seriously.” --Thompson
On Thu, Jun 19, 2014 at 7:01 PM, Grant Rettke <gcr@wisdomandwonder.com> wrote:
> Understood. I will start testing.
>
> Just out of curiosity, knowing full well that it will break the
> tangle, and also since I hadn't tried
> the quick and dirty flag, I ran it with this to see how long it took:
>
> (setq org-babel-common-header-args-w-values nil)
>
> It went from 936 seconds (15 minutes)
>
> Down to 279 (4 minutes).
>
> Now I am wondering if it is worth converting my document *not* to use
> property inheritance...
>
> What would you folks do?
> Grant Rettke | ACM, ASA, FSF, IEEE, SIAM
> gcr@wisdomandwonder.com | http://www.wisdomandwonder.com/
> “Wisdom begins in wonder.” --Socrates
> ((λ (x) (x x)) (λ (x) (x x)))
> “Life has become immeasurably better since I have been forced to stop
> taking it seriously.” --Thompson
>
>
> On Thu, Jun 19, 2014 at 8:44 AM, Eric Schulte <schulte.eric@gmail.com> wrote:
>> Grant Rettke <gcr@wisdomandwonder.com> writes:
>>
>>> I still want to be using org,, so my plan for now then is to write a
>>> pre-processing script for before org-babel-tangle is run that:
>>>
>>> 1. Checks each headline
>>> 2. Checks if there is a source block
>>> 3. If that source block doesn't have a noweb-ref name, and there is
>>> another source block under that
>>> headline, then "merge" it with the next source block of the same
>>> nature in that heading.
>>>
>>> That is off the top of my head, and I will try it by hand, first and
>>> see if it tangles the same result.
>>>
>>> Thoughts?
>>
>> Perhaps an easier workaround would be to customize the value of
>> `org-babel-common-header-args-w-values'. This variable determines which
>> header arguments are checked for in properties. If you remove all
>> header arguments from this variable which you know are not used in a
>> property, it may dramatically speed up tangle time.
>>
>> This may reduce the utility of some convenience functions, but should
>> not have too large of an impact on overall functionality.
>>
>> Best,
>> Eric
>>
>> --
>> Eric Schulte
>> https://cs.unm.edu/~eschulte
>> PGP: 0x614CA05D (see https://u.fsf.org/yw)
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: Discussion request: 15m tangle time, details follow
2014-06-20 1:26 ` Grant Rettke
@ 2014-06-20 14:50 ` Grant Rettke
0 siblings, 0 replies; 17+ messages in thread
From: Grant Rettke @ 2014-06-20 14:50 UTC (permalink / raw)
To: Eric Schulte; +Cc: emacs-orgmode@gnu.org
This is a nicer way, makes it obvious what *is* being used; uses dash.el:
(let* ((allowed '(exports
file
noweb
noweb-ref
session
tangle))
(new-ls
(--filter (member (car it) allowed)
org-babel-common-header-args-w-values)))
(setq org-babel-common-header-args-w-values new-ls))
Grant Rettke | ACM, ASA, FSF, IEEE, SIAM
gcr@wisdomandwonder.com | http://www.wisdomandwonder.com/
“Wisdom begins in wonder.” --Socrates
((λ (x) (x x)) (λ (x) (x x)))
“Life has become immeasurably better since I have been forced to stop
taking it seriously.” --Thompson
On Thu, Jun 19, 2014 at 8:26 PM, Grant Rettke <gcr@wisdomandwonder.com> wrote:
> Just excluded what seemed excludable and got build time down to 8
> minutes from 15 minutes.
>
> Kept the following as it seems that it would allow the most important things:
>
> (session . :any)
> (noweb-ref . :any)
> (noweb . ((yes no tangle no-export strip-export)))
> (file . :any)
> (tangle . ((tangle yes no :any)))
> (results . ((file list vector table scalar verbatim)
> (raw html latex org code pp drawer)
> (replace silent none append prepend)
> (output value)))
>
> Here is what I excluded:
>
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'cache org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'cmdline org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'colnames org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'comments org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'dir org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'eval org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'exports org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'epilogue org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'file-desc org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'hlines org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'mkdirp org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'no-expand org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'noeval org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'noweb-sep org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'padline org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'post org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'prologue org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'rownames org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'sep org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'shebang org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'tangle-mode org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'var org-babel-common-header-args-w-values))
> (setq org-babel-common-header-args-w-values
> (assq-delete-all 'wrap org-babel-common-header-args-w-values))
> Grant Rettke | ACM, ASA, FSF, IEEE, SIAM
> gcr@wisdomandwonder.com | http://www.wisdomandwonder.com/
> “Wisdom begins in wonder.” --Socrates
> ((λ (x) (x x)) (λ (x) (x x)))
> “Life has become immeasurably better since I have been forced to stop
> taking it seriously.” --Thompson
>
>
> On Thu, Jun 19, 2014 at 7:01 PM, Grant Rettke <gcr@wisdomandwonder.com> wrote:
>> Understood. I will start testing.
>>
>> Just out of curiosity, knowing full well that it will break the
>> tangle, and also since I hadn't tried
>> the quick and dirty flag, I ran it with this to see how long it took:
>>
>> (setq org-babel-common-header-args-w-values nil)
>>
>> It went from 936 seconds (15 minutes)
>>
>> Down to 279 (4 minutes).
>>
>> Now I am wondering if it is worth converting my document *not* to use
>> property inheritance...
>>
>> What would you folks do?
>> Grant Rettke | ACM, ASA, FSF, IEEE, SIAM
>> gcr@wisdomandwonder.com | http://www.wisdomandwonder.com/
>> “Wisdom begins in wonder.” --Socrates
>> ((λ (x) (x x)) (λ (x) (x x)))
>> “Life has become immeasurably better since I have been forced to stop
>> taking it seriously.” --Thompson
>>
>>
>> On Thu, Jun 19, 2014 at 8:44 AM, Eric Schulte <schulte.eric@gmail.com> wrote:
>>> Grant Rettke <gcr@wisdomandwonder.com> writes:
>>>
>>>> I still want to be using org,, so my plan for now then is to write a
>>>> pre-processing script for before org-babel-tangle is run that:
>>>>
>>>> 1. Checks each headline
>>>> 2. Checks if there is a source block
>>>> 3. If that source block doesn't have a noweb-ref name, and there is
>>>> another source block under that
>>>> headline, then "merge" it with the next source block of the same
>>>> nature in that heading.
>>>>
>>>> That is off the top of my head, and I will try it by hand, first and
>>>> see if it tangles the same result.
>>>>
>>>> Thoughts?
>>>
>>> Perhaps an easier workaround would be to customize the value of
>>> `org-babel-common-header-args-w-values'. This variable determines which
>>> header arguments are checked for in properties. If you remove all
>>> header arguments from this variable which you know are not used in a
>>> property, it may dramatically speed up tangle time.
>>>
>>> This may reduce the utility of some convenience functions, but should
>>> not have too large of an impact on overall functionality.
>>>
>>> Best,
>>> Eric
>>>
>>> --
>>> Eric Schulte
>>> https://cs.unm.edu/~eschulte
>>> PGP: 0x614CA05D (see https://u.fsf.org/yw)
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Discussion request: 15m tangle time, details follow
2014-06-18 2:41 ` Aaron Ecay
2014-06-18 8:13 ` Sebastien Vauban
2014-06-18 20:59 ` Eric Schulte
@ 2014-06-19 0:56 ` Grant Rettke
2 siblings, 0 replies; 17+ messages in thread
From: Grant Rettke @ 2014-06-19 0:56 UTC (permalink / raw)
To: Grant Rettke, emacs-orgmode@gnu.org
On Tue, Jun 17, 2014 at 9:41 PM, Aaron Ecay <aaronecay@gmail.com> wrote:
> Here you mean time to tangle, correct?
Yes, thank you for digging in.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Discussion request: 15m tangle time, details follow
2014-06-18 0:16 Discussion request: 15m tangle time, details follow Grant Rettke
2014-06-18 2:41 ` Aaron Ecay
@ 2014-06-18 2:54 ` Nick Dokos
2014-06-18 8:18 ` Thorsten Jolitz
2014-06-18 8:20 ` Andreas Leha
3 siblings, 0 replies; 17+ messages in thread
From: Nick Dokos @ 2014-06-18 2:54 UTC (permalink / raw)
To: emacs-orgmode
Grant Rettke <gcr@wisdomandwonder.com> writes:
>
> This build is documented in the github project; it only loads the
> absolute minimum required to do the build.
>
> Ideas: Separate the documents. Hack on org directly.
>
> Non-ideas: Faster hardware. More ram. Newer software.
>
> Details:
>
> Emacs 24.3.1.
> Org 8.2.6
> OSX 10.9 (software updated)
> Darwin orion 13.2.0 Darwin Kernel Version 13.2.0: Thu Apr 17 23:03:13
> PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
> 8core 2.x GHz, 16GB ram
> Anti-virus turned off (compliance)
>
> Please let me know any comments, questions, or concerns; looking
> forward to all and every thought
> and idea.
>
> Where I may contribute is with time, effort, patience, cheerfulness,
> and experience.
>
Have you profiled the build? Do you know where the time is spent?
If not, that's where I'd start:
(info "(elisp) Profiling")
Nick
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Discussion request: 15m tangle time, details follow
2014-06-18 0:16 Discussion request: 15m tangle time, details follow Grant Rettke
2014-06-18 2:41 ` Aaron Ecay
2014-06-18 2:54 ` Nick Dokos
@ 2014-06-18 8:18 ` Thorsten Jolitz
2014-06-18 9:34 ` Thorsten Jolitz
2014-06-19 1:00 ` Grant Rettke
2014-06-18 8:20 ` Andreas Leha
3 siblings, 2 replies; 17+ messages in thread
From: Thorsten Jolitz @ 2014-06-18 8:18 UTC (permalink / raw)
To: emacs-orgmode
Grant Rettke <gcr@wisdomandwonder.com> writes:
> The average build takes 15m.
Which file do you mean - TC3F.org?
Thats some 4400 lines, thus not _that_ big really. I once used a giant
init.el copied from the web that had some 8000 lines (now I'm back to
2500 again ...).
I converted that file to outshine, since its mainly one programming
language (emacs-lisp in this case):
[with-current-buffer "TC3F.org]
,-------------------------------------------------
| (benchmark-run (outorg-convert-org-to-outshine))
| (1.7926752110000002 8 1.042145478000009)
`-------------------------------------------------
After this is done *once*, you can always switch between emacs-lisp and
org-mode with outorg, It takes 0.4 sec to convert the whole file to org
again
,------------------------------------------
| (benchmark-run (outorg-edit-as-org '(4)))
| (0.365756325 1 0.13800897400000167)
`------------------------------------------
and 1.6 sec to convert it back to outshine (I have to fix this speed
difference ;)
,---------------------------------------------
| (benchmark-run (outorg-copy-edits-and-exit))
| (1.616835235 8 1.106696710999998)
`---------------------------------------------
But normally you do not convert the whole buffer to Org with outorg,
just the subtree at hand, and thats instantly.
Then productivity means that your init file *is* in a programming
language mode (TC3F.el) and you can modify and eval your code
on-the-fly. Whenever you need to edit the comment text, you do
M-# M-#
,-----------------------
| M-x outorg-edit-as-org
`-----------------------
and when you are done, M-#
,-------------------------------
| M-x outorg-copy-edits-and-exit
`-------------------------------
Its just the reverse of Org-mode with souce-blocks, and in cases like an
Emacs init-file, when its mostly one programming language and the
source-code is more important than the text (and frequently modified),
this reverse approach might be more productive.
PS
I just figured that I ran the benchmarks on an outorg testing branch,
which is faster than master but not yet ready. So things might be a bit
slower with master branch, but in terms of seconds (maybe 2 or 3 sec to
convert the whole file?)
--
cheers,
Thorsten
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Discussion request: 15m tangle time, details follow
2014-06-18 8:18 ` Thorsten Jolitz
@ 2014-06-18 9:34 ` Thorsten Jolitz
2014-06-19 1:00 ` Grant Rettke
1 sibling, 0 replies; 17+ messages in thread
From: Thorsten Jolitz @ 2014-06-18 9:34 UTC (permalink / raw)
To: emacs-orgmode
Thorsten Jolitz <tjolitz@gmail.com> writes:
> Grant Rettke <gcr@wisdomandwonder.com> writes:
>
>> The average build takes 15m.
> [with-current-buffer "TC3F.org]
> After this is done *once*, you can always switch between emacs-lisp and
> org-mode with outorg, It takes 0.4 sec to convert the whole file to org
> again
>
> ,------------------------------------------
> | (benchmark-run (outorg-edit-as-org '(4)))
> | (0.365756325 1 0.13800897400000167)
> `------------------------------------------
>
> and 1.6 sec to convert it back to outshine (I have to fix this speed
> difference ;)
>
> ,---------------------------------------------
> | (benchmark-run (outorg-copy-edits-and-exit))
> | (1.616835235 8 1.106696710999998)
> `---------------------------------------------
This thread inspired me to profile these two commands - they do roughly
the same thing, only in the opposite direction, and I found it strange
why converting from Source to Org should be 4x faster than converting
from Org to Source.
It turned out that
,------------------------------------------------------------
| kill-whole-line is an interactive compiled Lisp function in
| `simple.el'.
`------------------------------------------------------------
was the sole culprit (used for killing the source-block
ddelimiters). Its a lisp function and does too many things besides
killing the line. Replacing it with C function
,------------------------------------------------------------------
| delete-region is an interactive built-in function in `editfns.c'.
`------------------------------------------------------------------
improves performance to (on whole file TC3):
,---------------------------------------------
| (benchmark-run (outorg-copy-edits-and-exit))
| (0.66580701 2 0.28894333600000266)
`---------------------------------------------
The remaining speed difference is partly because I undo the indendation
introduced in the Org source-blocks before the conversion back to
source, and thus need to process the file twice.
--
cheers,
Thorsten
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Discussion request: 15m tangle time, details follow
2014-06-18 8:18 ` Thorsten Jolitz
2014-06-18 9:34 ` Thorsten Jolitz
@ 2014-06-19 1:00 ` Grant Rettke
1 sibling, 0 replies; 17+ messages in thread
From: Grant Rettke @ 2014-06-19 1:00 UTC (permalink / raw)
To: Thorsten Jolitz; +Cc: emacs-orgmode@gnu.org
On Wed, Jun 18, 2014 at 3:18 AM, Thorsten Jolitz <tjolitz@gmail.com> wrote:
> Grant Rettke <gcr@wisdomandwonder.com> writes:
>
>> The average build takes 15m.
>
> Which file do you mean - TC3F.org?
> Thats some 4400 lines, thus not _that_ big really. I once used a giant
> init.el copied from the web that had some 8000 lines (now I'm back to
> 2500 again ...).
Yes. Thank you for digging in. Thanks for the tip, I had not learned
out-org yet.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Discussion request: 15m tangle time, details follow
2014-06-18 0:16 Discussion request: 15m tangle time, details follow Grant Rettke
` (2 preceding siblings ...)
2014-06-18 8:18 ` Thorsten Jolitz
@ 2014-06-18 8:20 ` Andreas Leha
2014-06-19 0:58 ` Grant Rettke
3 siblings, 1 reply; 17+ messages in thread
From: Andreas Leha @ 2014-06-18 8:20 UTC (permalink / raw)
To: emacs-orgmode
Hi Grant,
Grant Rettke <gcr@wisdomandwonder.com> writes:
> Good evening,
>
> Over the past few months I've been working on the same literate
> document. It has been a learning
> experience for me, trial and error has abounded. The key tenet that
> I've adhered too though is to truly
> embrace literate programming, and the more I learn the more it makes
> sense. The document has
> grown quite organically and it has been and continues to be a
> wonderful experience. What I need
> help, feedback, discussion, and more on is the build time.
>
> The average build takes 15m. It didn't start this way; it was about 3
> minutes way back when. The last time it
> got kind of big was 9m and I didn't think too much of it. After
> literally a day of additions, it shot up to 15m.
> I tried upgrading to the latest org release with no change. I also
> removed all of the non-tangleable text with no change there, either.To
> give a fair picture, I did publish the system here:
>
> https://github.com/grettke/home
>
> My specific request: I need help with pointers on where I should start
> looking to speed things up. My goal is to have a full powered literate
> programming system in org mode that is blazing fast. This is from a
> user perspective, I use it every chance I get now and have barely
> scratched the surface. Right now though I'm sort of hobbled by the
> build time. That is actually understating it, I can't really be
> productive anymore at all. Little
> changes take 15m each and if I test it the "right way", 30m. Usually I
> would make little changes and every
> so often make sure that it can rebuild itself; usually it may :).
>
> This build is documented in the github project; it only loads the
> absolute minimum required to do the build.
>
> Ideas: Separate the documents. Hack on org directly.
>
> Non-ideas: Faster hardware. More ram. Newer software.
>
> Details:
>
> Emacs 24.3.1.
> Org 8.2.6
> OSX 10.9 (software updated)
> Darwin orion 13.2.0 Darwin Kernel Version 13.2.0: Thu Apr 17 23:03:13
> PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
> 8core 2.x GHz, 16GB ram
> Anti-virus turned off (compliance)
>
> Please let me know any comments, questions, or concerns; looking
> forward to all and every thought
> and idea.
I do not have the time to look into this, but fortunately, other
have done so already ;-)
Just one quick idea: Have you tried [fn:1]
(setq org-babel-use-quick-and-dirty-noweb-expansion t)
This can lead to dramatic speedups in my experience.
Best,
Andreas
>
> Where I may contribute is with time, effort, patience, cheerfulness,
> and experience.
>
> Kind regards,
>
> Grant Rettke | ACM, ASA, FSF, IEEE, SIAM
> gcr@wisdomandwonder.com | http://www.wisdomandwonder.com/
> “Wisdom begins in wonder.” --Socrates
> ((λ (x) (x x)) (λ (x) (x x)))
> “Life has become immeasurably better since I have been forced to stop
> taking it seriously.” --Thompson
Footnotes:
[fn:1]
(see article.gmane.org/gmane.emacs.orgmode/50625/match=problem+noweb+ref+property)
^ permalink raw reply [flat|nested] 17+ messages in thread