A file with 'org-mode rot'?

A file with 'org-mode rot'?

[Sharon Kimble]

[-- Attachment #1: Type: text/plain, Size: 1616 bytes --]

I have been keeping my diary in an org-mode file since the beginning of the
year, with this format -

--8<---------------cut here---------------start------------->8---
* April
** 01 Tue
blah de blah
--8<---------------cut here---------------end--------------->8---

Today I have had to create a new file, even though the old one was only 194kb
in size and of 2673 lines. However, it was very slow in opening, navigating to
any desired place, and appeared to be getting confused as to whether it was in
org-mode or not! Its set up for continuous org-mode, but recently to correct
formatting I've had to turn on org-mode in the middle of a diary posting, or
even close the file and then reopen it again.

On opening the master file for the year which will contain *all* my postings
for the year, it is still very slow and ponderous. But it is called
'2014.org', which is a renaming of my working file '2014a.org'.

Has '2014.org' got "org-mode rot", so many hidden commands of change of
org-mode functionality still within its meta-post that it makes the whole file
difficult to work with? I know that org-mode files are just plain text
formatted in a certain way that emacs recognises and then performs certain
predestined actions on, but somehow this file is slowing down in its
interactions with the rest of emacs.

So how can I keep a file 'healthy' and working well over many months please?

Sharon.
-- 
A taste of linux = http://www.sharons.org.uk
my git repo = https://bitbucket.org/boudiccas/dots
TGmeds = http://www.tgmeds.org.uk
Debian testing, Fluxbox 1.3.5, emacs 24.3.1

[-- Attachment #2: Type: application/pgp-signature, Size: 818 bytes --]

Re: A file with 'org-mode rot'?

[Karl Voit]

Hello Sharon!

* Sharon Kimble <boudiccas@talktalk.net> wrote:
>
> Has '2014.org' got "org-mode rot", so many hidden commands of change of
> org-mode functionality still within its meta-post that it makes the whole f=
> ile
> difficult to work with? I know that org-mode files are just plain text
> formatted in a certain way that emacs recognises and then performs certain
> predestined actions on, but somehow this file is slowing down in its
> interactions with the rest of emacs.
>
> So how can I keep a file 'healthy' and working well over many months please?

I am facing "weird" things as well. Mainly performance drop when
doing (very) simple things like adding | in front of a table line,
changing to the next table column, marking tasks as DONE, archiving
tasks, and so on.

I have no idea how to debug those issues since the profiler-report
does not seem to look strange.

This is why I asked here for an Org-mode syntax checker a couple of
days ago. There does not seem to exist such a thing.

I am afraid, that I have to do the painful task of commenting out
separate sections of my Org-mode configuration (2000 lines) and
check whether or not the behavior changes.

-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github

Re: A file with 'org-mode rot'?

[Eric S Fraga]

Hello,

My experience is that there is some correlation between the number of
src blocks and/or tables and performance.  I haven't done any analysis
but I do find that emacs lags when attempting to navigate some of my
large org files with many tables and src blocks.

However, I see no performance hits when working in large org files with
many headings and levels but little in the way tables or src blocks.

You may wish to try changing the values of some of the fontification
variables, such as

,----[ C-h v org-src-fontify-natively RET ]
| org-src-fontify-natively is a variable defined in `org.el'.
| Its value is t
| 
| Documentation:
| When non-nil, fontify code in code blocks.
| 
| You can customize this variable.
| 
| This variable was introduced, or its default value was changed, in
| version 24.4 of Emacs.
`----

if you do have src blocks in your large files.
-- 
: Eric S Fraga (0xFFFCF67D), Emacs 24.4.50.2, Org release_8.2.5h-842-gad16ef.dirty-git

Re: A file with 'org-mode rot'?

[Eric S Fraga]

On Friday,  4 Apr 2014 at 10:11, Eric S Fraga wrote:

[...]

> However, I see no performance hits when working in large org files with
> many headings and levels but little in the way tables or src blocks.

Actually, I need to take this back.  I upgraded org to the most recent
git version this morning and everything slowed down dramatically.  I
reset to the version

:  git checkout f57b2d71414fe195ee98ffb2531fa1423e8d70bf

which was from Monday (31 March) and things are fast again.

Looking at the log, I notice some changes to the caching mechanism in
org and changes to org-element.el.

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 24.4.50.2, Org release_8.2.5h-876-gf57b2d

Re: A file with 'org-mode rot'?

[Bastien]

Hi Eric and Karl,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Actually, I need to take this back.  I upgraded org to the most recent
> git version this morning and everything slowed down dramatically.  I
> reset to the version
>
> :  git checkout f57b2d71414fe195ee98ffb2531fa1423e8d70bf
>
> which was from Monday (31 March) and things are fast again.
>
> Looking at the log, I notice some changes to the caching mechanism in
> org and changes to org-element.el.

Just checking -- are things better now?

Nicolas, do you see what change in org-element.el may cause this?

-- 
 Bastien

Re: A file with 'org-mode rot'?

[Nicolas Goaziou]

Hello,

Bastien <bzg@gnu.org> writes:

> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> Actually, I need to take this back.  I upgraded org to the most recent
>> git version this morning and everything slowed down dramatically.  I
>> reset to the version
>>
>> :  git checkout f57b2d71414fe195ee98ffb2531fa1423e8d70bf
>>
>> which was from Monday (31 March) and things are fast again.
>>
>> Looking at the log, I notice some changes to the caching mechanism in
>> org and changes to org-element.el.
>
> Just checking -- are things better now?
>
> Nicolas, do you see what change in org-element.el may cause this?

Big changes in cache happened before f57b2d, so I'm not even sure this
is related. A profile report would help, so would a test file.


Regards,

-- 
Nicolas Goaziou

Re: A file with 'org-mode rot'?

[Karl Voit]

* Bastien <bzg@gnu.org> wrote:
> Hi Eric and Karl,

Hi Bastien!

> Just checking -- are things better now?

I am still facing strange behavior and I still do think that we
need an org-mode-syntax-checker for Org-mode files that cause
problems.

Example:

| foo | bar | baz |
|  42 |     |     |
|  23 |     |     |

It takes seven seconds with 100% CPU core load to move the
"foo"-column from above from the second column to the first by
applying ALT-<left>. This is in my main Org-mode file which has got
29000 lines.

#+BEGIN_EXAMPLE
- command-execute                                               29550  99%
 - call-interactively                                           29550  99%
  - org-metaleft                                                29469  99%
   - call-interactively                                         29469  99%
    - org-table-move-column                                     24559  83%
       org-table-align                                           9832  33%
  + byte-code                                                      74   0%
  + execute-extended-command                                        7   0%
+ yas--post-command-handler                                         6   0%
+ timer-event-handler                                               6   0%
+ ...                                                               0   0%
#+END_EXAMPLE

The very same operation in a small test file works fine.

Another example:

moving from @2$1 to @2$2 with TAB: 4 seconds
#+BEGIN_EXAMPLE
- command-execute                                                3847  80%
 - call-interactively                                            3845  80%
  - yas-expand                                                   3509  73%
   - yas--fallback                                               3509  73%
    - call-interactively                                         3509  73%
     - org-cycle                                                 3509  73%
      - call-interactively                                       3503  73%
       - org-table-next-field                                    3503  73%
          org-table-align                                        1760  36%
      + org-try-structure-completion                                6   0%
  + org-self-insert-command                                        91   1%
  + byte-code                                                      85   1%
  + next-line                                                      49   1%
  + previous-line                                                  42   0%
  + minibuffer-complete                                            22   0%
  + execute-extended-command                                       17   0%
  + profiler-report-toggle-entry                                   12   0%
  + org-yank                                                        7   0%
  + kill-ring-save                                                  3   0%
  + org-return                                                      3   0%
    org-kill-line                                                   3   0%
    scroll-up-command                                               2   0%
+ yas--post-command-handler                                       648  13%
+ redisplay_internal (C function)                                 103   2%
+ ...                                                             100   2%
+ timer-event-handler                                              73   1%
+ clm/log-command                                                  12   0%
+ xselect-convert-to-string                                        10   0%
  internal-timer-start-idle                                         2   0%
#+END_EXAMPLE

Again: in a small test file, it works in a fast way.

Still not possible: org-agenda-list & org-agenda-write to ICS file.
Org-mode is working hard but is in some kind of endless loop. I
assume that this also would work when I would change to a small
Org-mode file. I can test this if it helps.

-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github

Re: A file with 'org-mode rot'?

[Nicolas Goaziou]

Hello,

Karl Voit <devnull@Karl-Voit.at> writes:

> I am still facing strange behavior and I still do think that we
> need an org-mode-syntax-checker for Org-mode files that cause
> problems.
>
> Example:
>
> | foo | bar | baz |
> |  42 |     |     |
> |  23 |     |     |

I don't understand what a "syntax checker" has to do with it.
Notwithstanding the fact that such a checker assumes all Org obeys to
a single syntax definition, which is what I'm struggling for, but is not
the case actually, it would report, in this case, that syntax is valid.
How would that help?

> It takes seven seconds with 100% CPU core load to move the
> "foo"-column from above from the second column to the first by
> applying ALT-<left>. This is in my main Org-mode file which has got
> 29000 lines.
>
> #+BEGIN_EXAMPLE
> - command-execute                                               29550  99%
>  - call-interactively                                           29550  99%
>   - org-metaleft                                                29469  99%
>    - call-interactively                                         29469  99%
>     - org-table-move-column                                     24559  83%
>        org-table-align                                           9832  33%
>   + byte-code                                                      74   0%
>   + execute-extended-command                                        7   0%
> + yas--post-command-handler                                         6   0%
> + timer-event-handler                                               6   0%
> + ...                                                               0   0%
> #+END_EXAMPLE

I generated a 32000 lines long file and couldn't reproduce the problem.
What Org version are you using?

> Still not possible: org-agenda-list & org-agenda-write to ICS file.
> Org-mode is working hard but is in some kind of endless loop. I
> assume that this also would work when I would change to a small
> Org-mode file. I can test this if it helps.

It would help. So would an ECM for the reported infloop.

Also you can try to set `toggle-debug-on-quit', C-g amid the infloop,
and report the backtrace.


Regards,

-- 
Nicolas Goaziou

Re: A file with 'org-mode rot'?

[Karl Voit]

* Nicolas Goaziou <n.goaziou@gmail.com> wrote:
> Hello,
>
> Karl Voit <devnull@Karl-Voit.at> writes:
>
>> I am still facing strange behavior and I still do think that we
>> need an org-mode-syntax-checker for Org-mode files that cause
>> problems.
>>
>> Example:
>>
>> | foo | bar | baz |
>> |  42 |     |     |
>> |  23 |     |     |
>
> I don't understand what a "syntax checker" has to do with it.
> Notwithstanding the fact that such a checker assumes all Org obeys to
> a single syntax definition, which is what I'm struggling for, but is not
> the case actually, it would report, in this case, that syntax is valid.
> How would that help?

I had some issues where time-stamps of deadlines and logbook were
written in the wrong position, messing up drawers.

A syntax checker is not a 100% sure thing, yes. However, it would
help to get information on unknown drawers, missing :END: lines,
mismatching blocks, and so forth. I am not completely sure how to
achieve this. 

However, my (long) Org-mode files with the performance issues is not
a situation I am satisfied. Last week I presented Org-mode at a
Linuxdays-event here in Graz and I had to apologize for all the
weird behavior. This was somewhat embarrassing :-(

> I generated a 32000 lines long file and couldn't reproduce the problem.
> What Org version are you using?

d5484ae 2014-04-12 from the repos.

Probably it has to do with my Emacs binary as well: GNU Emacs
24.3.50.1 (x86_64-pc-linux-gnu, GTK+ Version 3.4.2)
 of 2014-01-01 on gkar, modified by Debian

The Emacs on the Windows machine in the office does not show many
issues. However, my configuration is only 95% the same for both
machines.

>> Still not possible: org-agenda-list & org-agenda-write to ICS file.
>> Org-mode is working hard but is in some kind of endless loop. I
>> assume that this also would work when I would change to a small
>> Org-mode file. I can test this if it helps.
>
> It would help. So would an ECM for the reported infloop.

Pfuh, I totally understand this :-)

Wow, I am afraid this will be hard work to generate but I will
schedule it. Please be patient because I have lots of stuff to do
these days.

> Also you can try to set `toggle-debug-on-quit', C-g amid the infloop,
> and report the backtrace.

Sure. After a couple of minutes, I C-g and this is the output:
http://paste.grml.org/1898/

-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github

Re: A file with 'org-mode rot'?

[Nicolas Goaziou]

[-- Attachment #1: Type: text/plain, Size: 512 bytes --]

Karl Voit <devnull@Karl-Voit.at> writes:

> However, my (long) Org-mode files with the performance issues is not
> a situation I am satisfied. Last week I presented Org-mode at a
> Linuxdays-event here in Graz and I had to apologize for all the
> weird behavior. This was somewhat embarrassing :-(

Does this happen on stable branch?

> Sure. After a couple of minutes, I C-g and this is the output:
> http://paste.grml.org/1898/

Would the following patch improve the situation?


Regards,

-- 
Nicolas Goaziou

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-exp-Improve-speed.patch --]
[-- Type: text/x-diff, Size: 2308 bytes --]

From 3b772bab6d99298844d9d05f79a12e1a9fb87a3a Mon Sep 17 00:00:00 2001
From: Nicolas Goaziou <n.goaziou@gmail.com>
Date: Sun, 13 Apr 2014 19:28:20 +0200
Subject: [PATCH] ob-exp: Improve speed

---
 lisp/ob-exp.el | 25 +++++++++++++------------
 1 file changed, 13 insertions(+), 12 deletions(-)

diff --git a/lisp/ob-exp.el b/lisp/ob-exp.el
index 220a3c3..dc2bc94 100644
--- a/lisp/ob-exp.el
+++ b/lisp/ob-exp.el
@@ -153,20 +153,12 @@ may make them unreachable."
   (save-window-excursion
     (save-excursion
       (let ((case-fold-search t)
-	    (org-babel-exp-reference-buffer reference-buffer)
-	    (regexp (concat org-babel-inline-src-block-regexp "\\|"
-			    org-babel-lob-one-liner-regexp "\\|"
-			    "^[ \t]*#\\+BEGIN_SRC")))
+	    (org-babel-exp-reference-buffer reference-buffer))
 	(goto-char (point-min))
-	(while (re-search-forward regexp nil t)
+	(while (re-search-forward
+		"src_\\|^[ \t]*#\\+\\(call:\\|begin_src\\)\\|call_" nil t)
 	  (unless (save-match-data (org-in-commented-heading-p))
-	    (let* ((element (save-excursion
-			      ;; If match is inline, point is at its
-			      ;; end.  Move backward so
-			      ;; `org-element-context' can get the
-			      ;; object, not the following one.
-			      (backward-char)
-			      (save-match-data (org-element-context))))
+	    (let* ((element (save-match-data (org-element-context)))
 		   (type (org-element-type element))
 		   (begin (copy-marker (org-element-property :begin element)))
 		   (end (copy-marker
@@ -176,6 +168,10 @@ may make them unreachable."
 			   (point)))))
 	      (case type
 		(inline-src-block
+		 (goto-char begin)
+		 (unless (bolp) (backward-char))
+		 (looking-at org-babel-inline-src-block-regexp)
+		 (goto-char (match-end 0))
 		 (let* ((info (org-babel-parse-inline-src-block-match))
 			(params (nth 2 info)))
 		   (setf (nth 1 info)
@@ -201,6 +197,11 @@ may make them unreachable."
 		       (delete-region begin end)
 		       (insert replacement)))))
 		((babel-call inline-babel-call)
+		 (goto-char (or (org-element-property :post-affiliated element)
+				begin))
+		 (unless (bolp) (backward-char))
+		 (looking-at org-babel-lob-one-liner-regexp)
+		 (goto-char (match-end 0))
 		 (let* ((lob-info (org-babel-lob-get-info))
 			(results
 			 (org-babel-exp-do-export
-- 
1.9.2

Re: A file with 'org-mode rot'?

[Erik Iverson]

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

Remote possibility: I've had issues with org-mode being slow when using
linum-mode in the past:

E.g.,
http://stackoverflow.com/questions/5229705/emacs-org-mode-turn-off-line-numbers

So you might want to make sure that's not enabled.


On Sun, Apr 13, 2014 at 12:49 PM, Nicolas Goaziou <n.goaziou@gmail.com>wrote:

> Karl Voit <devnull@Karl-Voit.at> writes:
>
> > However, my (long) Org-mode files with the performance issues is not
> > a situation I am satisfied. Last week I presented Org-mode at a
> > Linuxdays-event here in Graz and I had to apologize for all the
> > weird behavior. This was somewhat embarrassing :-(
>
> Does this happen on stable branch?
>
> > Sure. After a couple of minutes, I C-g and this is the output:
> > http://paste.grml.org/1898/
>
> Would the following patch improve the situation?
>
>
> Regards,
>
> --
> Nicolas Goaziou
>

[-- Attachment #2: Type: text/html, Size: 1562 bytes --]

Re: A file with 'org-mode rot'?

[Eric S Fraga]

On Friday, 11 Apr 2014 at 10:46, Bastien wrote:

[...]

> Just checking -- are things better now?

For me, they seem to be but I haven't consciously been thinking about
this problem... too busy with meetings at work!  I know that I have
recently upgraded org several times on at least one of my systems and I
didn't notice any performance hits.

thanks,
eric

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 24.3.1, Org release_8.2.5h-660-gef207f

Re: A file with 'org-mode rot'?

[Karl Voit]

* Erik Iverson <erikriverson@gmail.com> wrote:
>
> Remote possibility: I've had issues with org-mode being slow when using
> linum-mode in the past:
>
> E.g.,
> http://stackoverflow.com/questions/5229705/emacs-org-mode-turn-off-line-numbers
>
> So you might want to make sure that's not enabled.

I haven't used it. When I started it in order to try it out, my
Emacs instantly froze. Thanks for the warning :-)

-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github

Re: A file with 'org-mode rot'?

[Karl Voit]

Hi!

* Karl Voit <devnull@Karl-Voit.at> wrote:
> Hello Sharon!
>
> * Sharon Kimble <boudiccas@talktalk.net> wrote:
>>
>> So how can I keep a file 'healthy' and working well over many months please?
>
> I am facing "weird" things as well. Mainly performance drop when
> doing (very) simple things like adding | in front of a table line,
> changing to the next table column, marking tasks as DONE, archiving
> tasks, and so on.

Fun fact: this is still an issue but only on my Linux machine. My
Windows Emacs performs fine with tables.

> This is why I asked here for an Org-mode syntax checker a couple of
> days ago. There does not seem to exist such a thing.

Another example:

In the recent weeks I had the issue that LOGBOOK drawers were
moved before PROPERTIES drawers ending up in double PROPERTIES
drawers where the second one is ignored (habit, ...) or just one
LOGBOOK drawer followed by one ignored PROPERTIES drawer:

* example heading with two PROPERTIES drawers
:PROPERTIES:
:CREATED:  [2014-08-02 Sat 11:37]
:END:
:LOGBOOK:...
:PROPERTIES:
:CREATED:  [2014-07-16 Mon 14:42]
:STYLE: habit
:END:

* example heading with LOGBOOK before PROPERTIES and thus being ignored
:LOGBOOK:...
:PROPERTIES:
:CREATED:  [2014-07-16 Mon 14:42]
:STYLE: habit
:END:

(I am using (org-expiry-insinuate) to automatically create the
PROPERTIES drawer with the CREATED line if this matters here.)

This results in a big mess on my agenda and I had to fix those
headings semi-manually. Very annoying!

I don't have a clue, what causes this behavior since there is only a
(large) sub-set of my headings that got this mix-up.

Also: I recognized that many older headings were changed that way
that all empty lines were removed.

I got the suspicion that generating some Agendas or the org-mobile
export might correlate with the time-stamps of the modifications.


Besides my personal issues above: is there some kind of
automatically regression testing with real-world Org-mode files and
non-default settings?

I love this community! However, I get the impression that with every
"git pull", I end up being tester of a couple of new issues. :-(

Thinking about getting rid of the git repository and (only) using
"stable" Org releases ...

Re: A file with 'org-mode rot'?

[Marco Wahl]

Hi Karl,

Karl Voit <devnull@Karl-Voit.at> writes:

> In the recent weeks I had the issue that LOGBOOK drawers were
> moved before PROPERTIES drawers ending up in double PROPERTIES
> drawers where the second one is ignored (habit, ...) or just one
> LOGBOOK drawer followed by one ignored PROPERTIES drawer:
>
> * example heading with two PROPERTIES drawers
> :PROPERTIES:
> :CREATED:  [2014-08-02 Sat 11:37]
> :END:
> :LOGBOOK:...
> :PROPERTIES:
> :CREATED:  [2014-07-16 Mon 14:42]
> :STYLE: habit
> :END:
>
> * example heading with LOGBOOK before PROPERTIES and thus being ignored
> :LOGBOOK:...
> :PROPERTIES:
> :CREATED:  [2014-07-16 Mon 14:42]
> :STYLE: habit
> :END:
>
> (I am using (org-expiry-insinuate) to automatically create the
> PROPERTIES drawer with the CREATED line if this matters here.)
>
> This results in a big mess on my agenda and I had to fix those
> headings semi-manually. Very annoying!
>
> I don't have a clue, what causes this behavior since there is only a
> (large) sub-set of my headings that got this mix-up.

There has been an incompatible change for 8.3 about a month ago.  Since
then property drawers need "to be located right after a headline and its
planning line, when applicable."  See file etc/ORG-NEWS and commit
1aad53b.  Note also the org-repair-property-drawers function there.


HTH,  Marco
-- 
http://www.wahlzone.de
GPG: 0x0A3AE6F2