Clock-in in agenda makes some headings with links disappear

Clock-in in agenda makes some headings with links disappear

[Thomas Morgan]

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]

Dear Org mode hackers,

I came across a bug in Org-mode 8.2.6.  I had an overdue
TODO item in the agenda with a link at the end of the heading
text.  I clocked it in with `I' and the heading text
disappeared; only the link remained visible on that line in
the agenda.

This bug was reproducible for a while and I made the attached
patch which fixed it, but I didn't succeed in making a minimal
test case, and now I'm not able to reproduce it at all.

The problem seemed to be that `(beginning-of-line 0)' was
not moving to the beginning of the line, but between closing
brackets in the link.  I thought this might be related to
`inhibit-field-text-motion', but enabling that didn't help,
nor did using `(forward-line -1)'.  What seemed to solve it
was adding `(beginning-of-line)' after `(forward-line -1)',
as in the patch.

I know this is not a very helpful bug report but thought I'd
send it anyway in case someone else has seen the same problem
and understands what's going on.

Best regards,
Thomas


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Fix-bug-mading-agenda-line-disappear-on-clock-in.patch --]
[-- Type: text/x-diff, Size: 1337 bytes --]

From 06cde5027e8ca93d82d14c5c0b46deb3e1c80b25 Mon Sep 17 00:00:00 2001
From: Thomas Morgan <tlm@ziiuu.com>
Date: Tue, 13 May 2014 23:52:48 -0400
Subject: [PATCH] Fix bug mading agenda line disappear on clock-in.

In some cases with a link at the end of the line,
`(beginning-of-line 0)' moves to a position within
the link (between the closing bracket of the link
description and the following closing bracket) and
not to the beginning of the line.

Then the properties at that position in the link
are made the properties of the whole line, and the
entire line disappears.

* lisp/org-agenda.el (org-agenda-change-all-lines): Move backward one
  line and to the beginning of that line as an explicit second step.

TINYCHANGE
---
 lisp/org-agenda.el |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index e2a0629..428d31f 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -8895,7 +8895,8 @@ If FORCE-TAGS is non nil, the car of it returns the new tags."
 	  (save-restriction
 	    (narrow-to-region (point-at-bol) (point-at-eol))
 	    (org-agenda-finalize)))
-	(beginning-of-line 0)))))
+	(forward-line -1)
+	(beginning-of-line)))))
 
 (defun org-agenda-align-tags (&optional line)
   "Align all tags in agenda items to `org-agenda-tags-column'."
-- 
1.7.10.4

Re: Clock-in in agenda makes some headings with links disappear

[Bastien]

Hi Thomas,

thanks for the bug report.  This bug has been showing up and
disappearing as this used to be quite an unstable area.  But
I think I fixed this for good in maint.

One use that still needs careful check is when the agenda is
filtered, so perhaps you can hunt things from there.

Thanks,

-- 
 Bastien

Re: Clock-in in agenda makes some headings with links disappear

[Thomas Morgan]

[-- Attachment #1: Type: text/plain, Size: 1304 bytes --]

Hi, Bastien, 

Here is a recipe for what might be another manifestation
of this bug.

I'm using the maint branch: Org-mode version 8.2.6
(release_8.2.6-21-g3b9419 @ /src/org-mode/lisp/).

Files `setup.el' and `test-case.org' are attached.
The directory added to the load path in `setup.el'
will probably need to be customized.

Steps:

1. Run Emacs: `emacs -Q -l setup.el'.

2. Open the agenda: `M-x org-agenda RET a'.

3. Add a tag filter: `/ TAB foo RET'.

4. Move point to the agenda line containing the TODO item:
`C-s TODO RET'.

5. Move point to the end of the line with `C-e'.
It does apparently move to the end of the line.
However, `C-x =' reports that it is in column 0.
I expected it to be in column 108 or 109.

6. Try to move point to the beginning of the line with `C-a'.
It appears to move one column to the left.  I expected it
to move to the left edge of the window.  `C-x =' says it is
in column 108.

7. Try again to move to the beginning of line with `C-a'.
This works.

8. Clock in with `I'.  Parts of the line change color
but in column 0 `C-u C-x =' does not list any face.
I expected the whole line to be highlighted with face
`org-agenda-clocking'.

9. Now try to go to the item in the source Org file: `TAB'.
I see this error: `Command not allowed in this line'.

Best,
Thomas


[-- Attachment #2: setup.el --]
[-- Type: application/emacs-lisp, Size: 166 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: test-case.org --]
[-- Type: text/x-org, Size: 209 bytes --]

* TODO Buckle my shoe 							:foo:
  SCHEDULED: <2014-05-15 Thu +1d>
  :PROPERTIES:
  :STYLE:    habit
  :END:

* TODO Open the door
  SCHEDULED: <2014-05-15 Thu +1d>
  :PROPERTIES:
  :STYLE:    habit
  :END:

Re: Clock-in in agenda makes some headings with links disappear

[Thomas Morgan]

Bisecting shows that the bug (or at least the one involving
habits, if that's distinct from the one where items with links
disappear except for the links) was introduced by this commit:

eb0f8afdf42d696c1dd53d38a10ceaec3ae4c4f3
Author: Bastien Guerry <bzg@altern.org>
Date:   Thu Apr 17 02:31:23 2014 +0200

    org-agenda.el: Fix bug when displaying the changed date

Thomas Morgan <tlm@ziiuu.com> writes:

> Hi, Bastien, 
>
> Here is a recipe for what might be another manifestation
> of this bug.
>
> I'm using the maint branch: Org-mode version 8.2.6
> (release_8.2.6-21-g3b9419 @ /src/org-mode/lisp/).
>
> Files `setup.el' and `test-case.org' are attached.
> The directory added to the load path in `setup.el'
> will probably need to be customized.
>
> Steps:
>
> 1. Run Emacs: `emacs -Q -l setup.el'.
>
> 2. Open the agenda: `M-x org-agenda RET a'.
>
> 3. Add a tag filter: `/ TAB foo RET'.
>
> 4. Move point to the agenda line containing the TODO item:
> `C-s TODO RET'.
>
> 5. Move point to the end of the line with `C-e'.
> It does apparently move to the end of the line.
> However, `C-x =' reports that it is in column 0.
> I expected it to be in column 108 or 109.
>
> 6. Try to move point to the beginning of the line with `C-a'.
> It appears to move one column to the left.  I expected it
> to move to the left edge of the window.  `C-x =' says it is
> in column 108.
>
> 7. Try again to move to the beginning of line with `C-a'.
> This works.
>
> 8. Clock in with `I'.  Parts of the line change color
> but in column 0 `C-u C-x =' does not list any face.
> I expected the whole line to be highlighted with face
> `org-agenda-clocking'.
>
> 9. Now try to go to the item in the source Org file: `TAB'.
> I see this error: `Command not allowed in this line'.
>
> Best,
> Thomas

Re: Clock-in in agenda makes some headings with links disappear

[Bastien]

Hi Thomas,

Thomas Morgan <tlm@ziiuu.com> writes:

> Here is a recipe for what might be another manifestation
> of this bug.

Yes -- something tricky, but I really want to fix this for the
next maintainance release.  Thanks for the heads up,

-- 
 Bastien

Re: Clock-in in agenda makes some headings with links disappear

[Bastien]

[-- Attachment #1: Type: text/plain, Size: 499 bytes --]

Hi Thomas,

Thomas Morgan <tlm@ziiuu.com> writes:

> Here is a recipe for what might be another manifestation
> of this bug.

Thanks for this new report -- can you apply this patch against maint
and see if it works correctly?  Not only for the bug at stake, but all
kind of agenda filtering, rescheduling, clocking, etc.

The tests are pretty weak for the agenda (my bad), and this is quite
a significant rewrite, using text properties instead of overlays for
agenda filtering.

Thanks in advance!


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: org-use-properties-for-agenda-filtering.patch --]
[-- Type: text/x-diff, Size: 6594 bytes --]

diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index fe32202..c1a2e1b 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -2092,11 +2092,8 @@ When nil, `q' will kill the single agenda buffer."
     org-agenda-info
     org-agenda-pre-window-conf
     org-agenda-columns-active
-    org-agenda-tag-filter-overlays
     org-agenda-tag-filter
-    org-agenda-cat-filter-overlays
     org-agenda-category-filter
-    org-agenda-re-filter-overlays
     org-agenda-regexp-filter
     org-agenda-markers
     org-agenda-last-search-view-search-was-boolean
@@ -3324,19 +3321,12 @@ If AGENDA-BUFFER-NAME, use this as the buffer name for the agenda to write."
   (org-let (if nosettings nil org-agenda-exporter-settings)
     '(save-excursion
        (save-window-excursion
-	 (org-agenda-mark-filtered-text)
 	 (let ((bs (copy-sequence (buffer-string))) beg content)
-	   (org-agenda-unmark-filtered-text)
 	   (with-temp-buffer
 	     (rename-buffer org-agenda-write-buffer-name t)
 	     (set-buffer-modified-p nil)
 	     (insert bs)
 	     (org-agenda-remove-marked-text 'org-filtered)
-	     (while (setq beg (text-property-any (point-min) (point-max)
-						 'org-filtered t))
-	       (delete-region
-		beg (or (next-single-property-change beg 'org-filtered)
-			(point-max))))
 	     (run-hooks 'org-agenda-before-write-hook)
 	     (cond
 	      ((org-bound-and-true-p org-mobile-creating-agendas)
@@ -3402,28 +3392,6 @@ If AGENDA-BUFFER-NAME, use this as the buffer name for the agenda to write."
 		    org-agenda-buffer-name)))
   (when open (org-open-file file)))
 
-(defvar org-agenda-tag-filter-overlays nil)
-(defvar org-agenda-cat-filter-overlays nil)
-(defvar org-agenda-re-filter-overlays nil)
-
-(defun org-agenda-mark-filtered-text ()
-  "Mark all text hidden by filtering with a text property."
-  (let ((inhibit-read-only t))
-    (mapc
-     (lambda (o)
-       (when (equal (overlay-buffer o) (current-buffer))
-	 (put-text-property
-	  (overlay-start o) (overlay-end o)
-	  'org-filtered t)))
-     (append org-agenda-tag-filter-overlays
-	     org-agenda-cat-filter-overlays
-	     org-agenda-re-filter-overlays))))
-
-(defun org-agenda-unmark-filtered-text ()
-  "Remove the filtering text property."
-  (let ((inhibit-read-only t))
-    (remove-text-properties (point-min) (point-max) '(org-filtered t))))
-
 (defun org-agenda-remove-marked-text (property &optional value)
   "Delete all text marked with VALUE of PROPERTY.
 VALUE defaults to t."
@@ -3432,7 +3400,7 @@ VALUE defaults to t."
     (while (setq beg (text-property-any (point-min) (point-max)
 					property value))
       (delete-region
-       beg (or (next-single-property-change beg 'org-filtered)
+       beg (or (next-single-property-change beg property)
 	       (point-max))))))
 
 (defun org-agenda-add-entry-text ()
@@ -7400,7 +7368,8 @@ With two prefix arguments, remove the regexp filters."
   (when org-agenda-category-filter
     (org-agenda-filter-show-all-cat))
   (when org-agenda-regexp-filter
-    (org-agenda-filter-show-all-re)))
+    (org-agenda-filter-show-all-re))
+  (org-agenda-finalize))
 
 (defun org-agenda-filter-by-tag (strip &optional char narrow)
   "Keep only those lines in the agenda buffer that have a specific tag.
@@ -7672,50 +7641,31 @@ When NO-OPERATOR is non-nil, do not add the + operator to returned tags."
 (defun org-agenda-filter-hide-line (type)
   "Hide lines with TYPE in the agenda buffer."
   (let* ((b (max (point-min) (1- (point-at-bol))))
-	 (e (point-at-eol))
-	 (ov (make-overlay b e)))
-    (overlay-put ov 'invisible t)
-    (overlay-put ov 'intangible t)
-    (overlay-put ov 'type type)
-    (cond ((eq type 'tag) (push ov org-agenda-tag-filter-overlays))
-	  ((eq type 'category) (push ov org-agenda-cat-filter-overlays))
-	  ((eq type 'regexp) (push ov org-agenda-re-filter-overlays)))))
-
-(defun org-agenda-fix-tags-filter-overlays-at (&optional pos)
-  (setq pos (or pos (point)))
-  (save-excursion
-    (dolist (ov (overlays-at pos))
-      (when (and (overlay-get ov 'invisible)
-		 (eq (overlay-get ov 'type) 'tag))
-	(goto-char pos)
-	(if (< (overlay-start ov) (point-at-eol))
-	    (move-overlay ov (point-at-eol)
-			  (overlay-end ov)))))))
+	 (e (point-at-eol)))
+    (let ((inhibit-read-only t))
+      (add-text-properties
+       b e `(invisible t org-filtered t org-filter-type ,type)))))
 
-(defun org-agenda-filter-show-all-tag nil
-  "Remove tag filter overlays from the agenda buffer."
-  (mapc 'delete-overlay org-agenda-tag-filter-overlays)
-  (setq org-agenda-tag-filter-overlays nil
-	org-agenda-tag-filter nil
-	org-agenda-filter-form nil)
+(defun org-agenda-remove-filter (type)
+  (interactive)
+  "Remove filter of type TYPE from the agenda buffer."
+  (goto-char (point-min))
+  (let ((inhibit-read-only t) pos)
+    (while (setq pos (text-property-any (point) (point-max) 'org-filter-type type))
+      (goto-char pos)
+      (remove-text-properties
+       (point) (next-single-property-change (point) 'org-filter-type)
+       `(invisible t org-filter-type ,type))))
+  (set (intern (format "org-agenda-%s-filter" (intern-soft type))) nil)
+  (setq org-agenda-filter-form nil)
   (org-agenda-set-mode-name))
 
+(defun org-agenda-filter-show-all-tag nil
+  (org-agenda-remove-filter 'tag))
 (defun org-agenda-filter-show-all-re nil
-  "Remove regexp filter overlays from the agenda buffer."
-  (mapc 'delete-overlay org-agenda-re-filter-overlays)
-  (setq org-agenda-re-filter-overlays nil
-	org-agenda-regexp-filter nil
-	org-agenda-filter-form nil)
-  (org-agenda-set-mode-name))
-
+  (org-agenda-remove-filter 'regexp))
 (defun org-agenda-filter-show-all-cat nil
-  "Remove category filter overlays from the agenda buffer."
-  (mapc 'delete-overlay org-agenda-cat-filter-overlays)
-  (setq org-agenda-cat-filter-overlays nil
-	org-agenda-filtered-by-category nil
-	org-agenda-category-filter nil
-	org-agenda-filter-form nil)
-  (org-agenda-set-mode-name))
+  (org-agenda-remove-filter 'category))
 
 (defun org-agenda-manipulate-query-add ()
   "Manipulate the query by adding a search term with positive selection.
@@ -9180,8 +9130,6 @@ Called with a universal prefix arg, show the priority instead of setting it."
 	(when (equal marker (org-get-at-bol 'org-marker))
 	  (remove-text-properties (point-at-bol) (point-at-eol) '(display))
 	  (org-move-to-column (- (window-width) (length stamp)) t)
-
-	  (org-agenda-fix-tags-filter-overlays-at (point))
           (if (featurep 'xemacs)
 	      ;; Use `duplicable' property to trigger undo recording
               (let ((ex (make-extent nil nil))

[-- Attachment #3: Type: text/plain, Size: 14 bytes --]


-- 
 Bastien

Re: Clock-in in agenda makes some headings with links disappear

[Bastien]

Bastien <bzg@gnu.org> writes:

> Thanks for this new report -- can you apply this patch against maint
> and see if it works correctly?  Not only for the bug at stake, but all
> kind of agenda filtering, rescheduling, clocking, etc.

I've now applied this patch in maint.

Let's continue testing this *heavily*.

-- 
 Bastien

Re: Clock-in in agenda makes some headings with links disappear

[Thomas Morgan]

Hi, Bastien, 

I've noticed some new issues on maint.

Sometimes refreshing with `r' can empty an agenda of all its contents.
I'm using sticky agendas and the only way to recover seems to be to
kill the agenda buffer and create a new one.

Sometimes scheduling in the agenda with `C-c C-s' or `S-right' hangs.
There is a message saying the item is scheduled but the popup calendar
window remains until I type `C-g'.  I had been seeing this bug for a
while but not immediately before the change from overlays to text
properties.

Sometimes changing the tag filter puts point at the end of a line,
which is inconvenient if it happens to fall on a long line.

Adding a tag filter is sometimes slow.

I'll try to investigate and send proper reports.

Thanks,

Thomas

Re: Clock-in in agenda makes some headings with links disappear

[Thomas Morgan]

[-- Attachment #1: Type: text/plain, Size: 527 bytes --]

Hi, Bastien, 

Here is a recipe for the bug where scheduling hangs.

1. Change the load path entry in `setup.el'.

2. Start Emacs with `emacs -Q -l setup.el'.

3. Open a TODO list with `M-x org-todo-list'.

4. Filter out tag with `C-u / TAB foo RET'.

5. Move point to first item with `C-s todo RET'.

6. Schedule it for today with `C-c C-s RET'.

An indication that it was scheduled appears in the echo area:

,----
| Scheduled to <2014-05-25 Sun>
`----

But the calendar window remains until you press `C-g'.

Best,

Thomas


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: test-case.org --]
[-- Type: text/x-org, Size: 123 bytes --]

* TODO Buckle my shoe

* TODO Open the door                                                   :foo:

* TODO Pick up sticks

[-- Attachment #3: setup.el --]
[-- Type: application/emacs-lisp, Size: 130 bytes --]

Re: Clock-in in agenda makes some headings with links disappear

[Thomas Morgan]

[-- Attachment #1: Type: text/plain, Size: 158 bytes --]

Here is a small patch for this bug.  It works as far as I can tell
though it's possible that it introduces other problems or that there's
a better solution.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Fix-bug-where-scheduling-in-agenda-hangs.patch --]
[-- Type: text/x-diff, Size: 1338 bytes --]

From 3181ee87106275bf947324b75054c6e2acb18ed1 Mon Sep 17 00:00:00 2001
From: Thomas Morgan <tlm@ziiuu.com>
Date: Sun, 25 May 2014 13:40:35 -0400
Subject: [PATCH] Fix bug where scheduling in agenda hangs.

* lisp/org-agenda.el (org-agenda-show-new-time): Disable invisibility
  before moving to the column where the new time will be shown.
  Otherwise org-move-to-column may skip past several filtered lines,
  and though the cursor appears at the end of the current line, point
  is actually on the newline before the next visible line.

TINYFIX
---
 lisp/org-agenda.el |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index 1be2bdb..8e2a627 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -9176,7 +9176,8 @@ Called with a universal prefix arg, show the priority instead of setting it."
       (while (not (bobp))
 	(when (equal marker (org-get-at-bol 'org-marker))
 	  (remove-text-properties (point-at-bol) (point-at-eol) '(display))
-	  (org-move-to-column (- (window-width) (length stamp)) t)
+	  (let ((buffer-invisibility-spec nil))
+	    (org-move-to-column (- (window-width) (length stamp)) t))
           (if (featurep 'xemacs)
 	      ;; Use `duplicable' property to trigger undo recording
               (let ((ex (make-extent nil nil))
-- 
1.7.10.4


[-- Attachment #3: Type: text/plain, Size: 608 bytes --]


Thomas Morgan <tlm@ziiuu.com> writes:

> Hi, Bastien, 
>
> Here is a recipe for the bug where scheduling hangs.
>
> 1. Change the load path entry in `setup.el'.
>
> 2. Start Emacs with `emacs -Q -l setup.el'.
>
> 3. Open a TODO list with `M-x org-todo-list'.
>
> 4. Filter out tag with `C-u / TAB foo RET'.
>
> 5. Move point to first item with `C-s todo RET'.
>
> 6. Schedule it for today with `C-c C-s RET'.
>
> An indication that it was scheduled appears in the echo area:
>
> ,----
> | Scheduled to <2014-05-25 Sun>
> `----
>
> But the calendar window remains until you press `C-g'.
>
> Best,
>
> Thomas

Re: Clock-in in agenda makes some headings with links disappear

[Bastien]

Hi Thomas,

Thomas Morgan <tlm@ziiuu.com> writes:

> Here is a small patch for this bug.  It works as far as I can tell
> though it's possible that it introduces other problems or that there's
> a better solution.

The solution was to clean-up `org-move-to-column' directly.

It used to do complex stuff when agenda filters were using overlays,
now it's more simple and things should be all right also in other
places than the agenda where `org-move-to-column' is in use (e.g.
tables, headlines when aligning the tags, etc.)

Thanks again for the patch,

-- 
 Bastien

Re: Clock-in in agenda makes some headings with links disappear

[Bastien]

Hi Thomas,

Thomas Morgan <tlm@ziiuu.com> writes:

> An indication that it was scheduled appears in the echo area:
>
> ,----
> | Scheduled to <2014-05-25 Sun>
> `----
>
> But the calendar window remains until you press `C-g'.

this is now fixed, thanks!

-- 
 Bastien