emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* Merging .org files
@ 2011-01-30 11:39 Pere Quintana Seguí
  2011-01-30 17:39 ` Juan Pechiar
                   ` (2 more replies)
  0 siblings, 3 replies; 13+ messages in thread
From: Pere Quintana Seguí @ 2011-01-30 11:39 UTC (permalink / raw)
  To: Org Mode


[-- Attachment #1.1: Type: text/plain, Size: 831 bytes --]

Hello,

In my current system, I have about 200 .org files. With this number of
files, building the agenda is very slow in my home computer and, also,
makes the performance of MobileOrg quite poor.

So, I would like to join files, to increase the system's performance.

Is there any script to cleanly join files, by transforming the title of
the file in a first level heading (*) and adding an star to all other
headings of the file?

I think this is the cleaner way to join files.

Thank You,

Pere

-- 
Dr. Pere Quintana Seguí
Observatori de l'Ebre (Universitat Ramon Llull - CSIC)
Horta Alta, 38. 43520 Roquetes (Tarragona), Spain.
T. (+34) 977 500 511 - F. (+34) 977 504 660
http://www.obsebre.es - http://pere.quintanasegui.com
GnuPG:2CAB4330
http://pere.quintanasegui.com/coses/key-transition.txt


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 888 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Merging .org files
  2011-01-30 11:39 Merging .org files Pere Quintana Seguí
@ 2011-01-30 17:39 ` Juan Pechiar
  2011-01-30 22:09   ` Pere Quintana Seguí
  2011-01-30 19:22 ` Michael Welle
  2011-01-30 21:51 ` Matt Lundin
  2 siblings, 1 reply; 13+ messages in thread
From: Juan Pechiar @ 2011-01-30 17:39 UTC (permalink / raw)
  To: Pere Quintana Seguí; +Cc: Org Mode

Hola Pere,

I'd suggest creating a master document with #includes of all your
Orgmode files, and then export this master document as org-mode.

http://orgmode.org/manual/Include-files.html#Include-files

There is a :minlevel attribute to automatically demote included
org-mode files.

Saludos,
.j.

On Sun, Jan 30, 2011 at 12:39:35PM +0100, Pere Quintana Seguí wrote:
> In my current system, I have about 200 .org files. With this number of
> files, building the agenda is very slow in my home computer and, also,
> makes the performance of MobileOrg quite poor.
>
> So, I would like to join files, to increase the system's performance.
>
> Is there any script to cleanly join files, by transforming the title of
> the file in a first level heading (*) and adding an star to all other
> headings of the file?
>
> I think this is the cleaner way to join files.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Merging .org files
  2011-01-30 11:39 Merging .org files Pere Quintana Seguí
  2011-01-30 17:39 ` Juan Pechiar
@ 2011-01-30 19:22 ` Michael Welle
  2011-01-30 22:20   ` Pere Quintana Seguí
  2011-01-30 21:51 ` Matt Lundin
  2 siblings, 1 reply; 13+ messages in thread
From: Michael Welle @ 2011-01-30 19:22 UTC (permalink / raw)
  To: emacs-orgmode

Hello,

Pere Quintana Seguí <pquintana@obsebre.es> writes:

> Hello,
>
> In my current system, I have about 200 .org files. With this number of
> files, building the agenda is very slow in my home computer and, also,
> makes the performance of MobileOrg quite poor.
>
> So, I would like to join files, to increase the system's performance.
don't the org files reflect another structure, for instance projects
you are working on, customers you are dealing with etc? 


> Is there any script to cleanly join files, by transforming the title of
> the file in a first level heading (*) and adding an star to all other
> headings of the file?
If you don't follow the advice of Juan, maybe you want only a few
files merged into one file each, you can hack a Perl script or something
similar. Finding regexp that match the headings isn't that complicated.

VG
hmw

-- 
biff4emacsen - A biff-like tool for (X)Emacs
http://www.c0t0d0s0.de/biff4emacsen/biff4emacsen.html
Flood - Your friendly network packet generator
http://www.c0t0d0s0.de/flood/flood.html

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Merging .org files
  2011-01-30 11:39 Merging .org files Pere Quintana Seguí
  2011-01-30 17:39 ` Juan Pechiar
  2011-01-30 19:22 ` Michael Welle
@ 2011-01-30 21:51 ` Matt Lundin
  2011-01-31 15:18   ` Pere Quintana Seguí
  2 siblings, 1 reply; 13+ messages in thread
From: Matt Lundin @ 2011-01-30 21:51 UTC (permalink / raw)
  To: Pere Quintana Seguí; +Cc: Org Mode

Pere Quintana Seguí <pquintana@obsebre.es> writes:

> In my current system, I have about 200 .org files. With this number of
> files, building the agenda is very slow in my home computer and, also,
> makes the performance of MobileOrg quite poor.
>
> So, I would like to join files, to increase the system's performance.

FWIW, I've found the opposite to be true. Splitting up larger files and
reducing the depth of the hierarchy marginally improves agenda
performance (~ 0.2 seconds acc. to elp). But in my case, I split 10
files into 40, so I have no idea what 200 files would do. Perhaps
there's a sweet spot somewhere? I also haven't used mobile org.

In my experience, archiving old items and reducing the number of active
todos is the best way to keep the agenda snappy.

> Is there any script to cleanly join files, by transforming the title of
> the file in a first level heading (*) and adding an star to all other
> headings of the file?

Perl is always handy for this type of thing:

--8<---------------cut here---------------start------------->8---
#!/usr/bin/perl
use strict;
use warnings;

my $mergefile = "/tmp/merged.org";

open NEWFILE, ">", $mergefile 
  or die "Can't open $mergefile: $!";

while (<>) {
  s/^(\*+)/*$1/;
  s/^#\+TITLE:\s+(.*)$/* $1/;
  s/^(#\+\w+:.*)$/: $1/;
  print NEWFILE "$_";
}

close NEWFILE;
--8<---------------cut here---------------end--------------->8---

Note: This is a quick proof of concept. Use at your own risk. ;)

Best,
Matt

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Merging .org files
  2011-01-30 17:39 ` Juan Pechiar
@ 2011-01-30 22:09   ` Pere Quintana Seguí
  0 siblings, 0 replies; 13+ messages in thread
From: Pere Quintana Seguí @ 2011-01-30 22:09 UTC (permalink / raw)
  To: Juan Pechiar; +Cc: Org Mode


[-- Attachment #1.1: Type: text/plain, Size: 779 bytes --]

Al 30/01/11 18:39, En/na Juan Pechiar ha escrit:
> Hola Pere,
> 
> I'd suggest creating a master document with #includes of all your
> Orgmode files, and then export this master document as org-mode.
> 
> http://orgmode.org/manual/Include-files.html#Include-files
> 
> There is a :minlevel attribute to automatically demote included
> org-mode files.
> 
> Saludos,
> .j.

I'll try this. It looks like a very clean way to do this.

Gracias,

Pere

-- 
Dr. Pere Quintana Seguí
Observatori de l'Ebre (Universitat Ramon Llull - CSIC)
Horta Alta, 38. 43520 Roquetes (Tarragona), Spain.
T. (+34) 977 500 511 - F. (+34) 977 504 660
http://www.obsebre.es - http://pere.quintanasegui.com
GnuPG:2CAB4330
http://pere.quintanasegui.com/coses/key-transition.txt


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 888 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Re: Merging .org files
  2011-01-30 19:22 ` Michael Welle
@ 2011-01-30 22:20   ` Pere Quintana Seguí
  0 siblings, 0 replies; 13+ messages in thread
From: Pere Quintana Seguí @ 2011-01-30 22:20 UTC (permalink / raw)
  To: emacs-orgmode


[-- Attachment #1.1: Type: text/plain, Size: 1846 bytes --]

Al 30/01/11 20:22, En/na Michael Welle ha escrit:
>> In my current system, I have about 200 .org files. With this number of
>> files, building the agenda is very slow in my home computer and, also,
>> makes the performance of MobileOrg quite poor.
>>
>> So, I would like to join files, to increase the system's performance.
> don't the org files reflect another structure, for instance projects
> you are working on, customers you are dealing with etc? 

Yes, they are organised in subjects. But, they could be organised in
another way. There is more than one valid way to efficiently organise my
information and I should choose the one that performs better. My current
system is not good enough.

> 
>> Is there any script to cleanly join files, by transforming the title of
>> the file in a first level heading (*) and adding an star to all other
>> headings of the file?
> If you don't follow the advice of Juan, maybe you want only a few
> files merged into one file each, you can hack a Perl script or something
> similar. Finding regexp that match the headings isn't that complicated.

I'll probably follow Juan's advice. Which is very clean.

I know that hacking a script is easy, but it is even more efficient to
use an already written script if it was already written by someone. My
belief is that one of the points of free software is to reuse code as
much as possible. And I think that there is much more wisdom in this
list than in my humble brain, as Juan has proved.

Thank you,

Pere

-- 
Dr. Pere Quintana Seguí
Observatori de l'Ebre (Universitat Ramon Llull - CSIC)
Horta Alta, 38. 43520 Roquetes (Tarragona), Spain.
T. (+34) 977 500 511 - F. (+34) 977 504 660
http://www.obsebre.es - http://pere.quintanasegui.com
GnuPG:2CAB4330
http://pere.quintanasegui.com/coses/key-transition.txt


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 888 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Merging .org files
  2011-01-30 21:51 ` Matt Lundin
@ 2011-01-31 15:18   ` Pere Quintana Seguí
  2011-01-31 20:45     ` Michael Welle
  0 siblings, 1 reply; 13+ messages in thread
From: Pere Quintana Seguí @ 2011-01-31 15:18 UTC (permalink / raw)
  To: Matt Lundin; +Cc: Org Mode


[-- Attachment #1.1: Type: text/plain, Size: 2151 bytes --]

Al 30/01/11 22:51, En/na Matt Lundin ha escrit:
> Pere Quintana Seguí <pquintana@obsebre.es> writes:
> 
>> In my current system, I have about 200 .org files. With this number of
>> files, building the agenda is very slow in my home computer and, also,
>> makes the performance of MobileOrg quite poor.
>>
>> So, I would like to join files, to increase the system's performance.
> 
> FWIW, I've found the opposite to be true. Splitting up larger files and
> reducing the depth of the hierarchy marginally improves agenda
> performance (~ 0.2 seconds acc. to elp). But in my case, I split 10
> files into 40, so I have no idea what 200 files would do. Perhaps
> there's a sweet spot somewhere? I also haven't used mobile org.

So, maybe my initial assumption is wrong.

Has anybody run some experiments to check if it is better to have a
small number of big files or many small files?


> In my experience, archiving old items and reducing the number of active
> todos is the best way to keep the agenda snappy.
> 
>> Is there any script to cleanly join files, by transforming the title of
>> the file in a first level heading (*) and adding an star to all other
>> headings of the file?
> 
> Perl is always handy for this type of thing:
> 
> --8<---------------cut here---------------start------------->8---
> #!/usr/bin/perl
> use strict;
> use warnings;
> 
> my $mergefile = "/tmp/merged.org";
> 
> open NEWFILE, ">", $mergefile 
>   or die "Can't open $mergefile: $!";
> 
> while (<>) {
>   s/^(\*+)/*$1/;
>   s/^#\+TITLE:\s+(.*)$/* $1/;
>   s/^(#\+\w+:.*)$/: $1/;
>   print NEWFILE "$_";
> }
> 
> close NEWFILE;
> --8<---------------cut here---------------end--------------->8---
> 
> Note: This is a quick proof of concept. Use at your own risk. ;)

Great, thank you!

Pere


-- 
Dr. Pere Quintana Seguí
Observatori de l'Ebre (Universitat Ramon Llull - CSIC)
Horta Alta, 38. 43520 Roquetes (Tarragona), Spain.
T. (+34) 977 500 511 - F. (+34) 977 504 660
http://www.obsebre.es - http://pere.quintanasegui.com
GnuPG:2CAB4330
http://pere.quintanasegui.com/coses/key-transition.txt


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Merging .org files
  2011-01-31 15:18   ` Pere Quintana Seguí
@ 2011-01-31 20:45     ` Michael Welle
  2011-02-07  8:28       ` Pere Quintana Seguí
  0 siblings, 1 reply; 13+ messages in thread
From: Michael Welle @ 2011-01-31 20:45 UTC (permalink / raw)
  To: emacs-orgmode

Hello,

Pere Quintana Seguí <pquintana@obsebre.es> writes:
[...]
> Has anybody run some experiments to check if it is better to have a
> small number of big files or many small files?
it depends on so many factors, disk speed vs. cpu speed for
instance. Maybe you could experiment with your data set and post some
numbers ;).

Regards
hmw

-- 
biff4emacsen - A biff-like tool for (X)Emacs
http://www.c0t0d0s0.de/biff4emacsen/biff4emacsen.html
Flood - Your friendly network packet generator
http://www.c0t0d0s0.de/flood/flood.html

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Re: Merging .org files
  2011-01-31 20:45     ` Michael Welle
@ 2011-02-07  8:28       ` Pere Quintana Seguí
  2011-03-18 20:38         ` Matt Lundin
  0 siblings, 1 reply; 13+ messages in thread
From: Pere Quintana Seguí @ 2011-02-07  8:28 UTC (permalink / raw)
  To: Michael Welle; +Cc: emacs-orgmode


[-- Attachment #1.1: Type: text/plain, Size: 1441 bytes --]

Al 31/01/11 21:45, En/na Michael Welle ha escrit:
> it depends on so many factors, disk speed vs. cpu speed for
> instance. Maybe you could experiment with your data set and post some
> numbers ;).

At the end, I did my experiment, but not in a scientific manner. I did
no measurements. Nevertheless, qualitatively, I can now say that I
reduced the number of org files from ~200 to 8 and the agenda builds
much faster, specially on my netbook which was very slow and now it is
just slow. On my Desktop computer, building the agenda was fast and now
it is very fast. But, what is better, now MobileOrg works fine on my
iPhone, before it was useless. Sorry for not being able to provide
quantitative details.

At the end, I did the merging of the files manually. Inspecting the
files, one by one, was very useful, as it allowed me to delete some
useless information and better organise the rest.

Now I have to learn to better navigate within my much longer org files.
Before, I used ido-mode to jump from buffer to buffer, now I guess I
have to practise more sparse trees to jump from headline to headline.

-- 
Dr. Pere Quintana Seguí
Observatori de l'Ebre (Universitat Ramon Llull - CSIC)
Horta Alta, 38. 43520 Roquetes (Tarragona), Spain.
T. (+34) 977 500 511 - F. (+34) 977 504 660
http://www.obsebre.es - http://pere.quintanasegui.com
GnuPG:2CAB4330
http://pere.quintanasegui.com/coses/key-transition.txt


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Merging .org files
  2011-02-07  8:28       ` Pere Quintana Seguí
@ 2011-03-18 20:38         ` Matt Lundin
  2011-03-19  8:51           ` Aankhen
  0 siblings, 1 reply; 13+ messages in thread
From: Matt Lundin @ 2011-03-18 20:38 UTC (permalink / raw)
  To: pquintana; +Cc: emacs-orgmode, Michael Welle

Pere Quintana Seguí <pquintana@obsebre.es> writes:

> Now I have to learn to better navigate within my much longer org files.
> Before, I used ido-mode to jump from buffer to buffer, now I guess I
> have to practise more sparse trees to jump from headline to headline.

I use this function to jump quickly (via ido) to a first level headline
in my org files:

--8<---------------cut here---------------start------------->8---
(defun my-org-global-goto ()
  (interactive)
  (let ((org-completion-use-ido t)
	(org-outline-path-complete-in-steps nil)
	(org-refile-use-outline-path nil)
	(org-refile-targets '((my-org-refile-targets :maxlevel . 1))))
    (org-refile t)))
--8<---------------cut here---------------end--------------->8---

Best,
Matt

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Re: Merging .org files
  2011-03-18 20:38         ` Matt Lundin
@ 2011-03-19  8:51           ` Aankhen
  2011-03-19 12:53             ` Matt Lundin
  0 siblings, 1 reply; 13+ messages in thread
From: Aankhen @ 2011-03-19  8:51 UTC (permalink / raw)
  To: Matt Lundin, pquintana, emacs-orgmode, Michael Welle

On Sat, Mar 19, 2011 at 02:08, Matt Lundin <mdl@imapmail.org> wrote:
> Pere Quintana Seguí <pquintana@obsebre.es> writes:
>
>> Now I have to learn to better navigate within my much longer org files.
>> Before, I used ido-mode to jump from buffer to buffer, now I guess I
>> have to practise more sparse trees to jump from headline to headline.
>
> I use this function to jump quickly (via ido) to a first level headline
> in my org files:
>
> [snip]

Do you normally have ‘org-completion-use-ido’ turned off or something?
(Just wondering why you couldn’t use ‘org-refile’ directly.)

Aankhen

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Merging .org files
  2011-03-19  8:51           ` Aankhen
@ 2011-03-19 12:53             ` Matt Lundin
  2011-03-19 17:42               ` Aankhen
  0 siblings, 1 reply; 13+ messages in thread
From: Matt Lundin @ 2011-03-19 12:53 UTC (permalink / raw)
  To: Aankhen; +Cc: Michael Welle, emacs-orgmode, pquintana

Aankhen <aankhen@gmail.com> writes:

> On Sat, Mar 19, 2011 at 02:08, Matt Lundin <mdl@imapmail.org> wrote:
>> Pere Quintana Seguí <pquintana@obsebre.es> writes:
>>
>>> Now I have to learn to better navigate within my much longer org files.
>>> Before, I used ido-mode to jump from buffer to buffer, now I guess I
>>> have to practise more sparse trees to jump from headline to headline.
>>
>> I use this function to jump quickly (via ido) to a first level headline
>> in my org files:
>>
>> [snip]
>
> Do you normally have ‘org-completion-use-ido’ turned off or something?
> (Just wondering why you couldn’t use ‘org-refile’ directly.)

Yes, that is correct. I normally have org-completion-use-ido turned off.

You could easily call org-refile with a prefix argument directly from
within an org-buffer. However, I find it more convenient to bind
"(org-refile t)" to one of the function keys than to type C-u C-c C-w.
The latter works only on org buffers, while the former is global.
Moreover, when navigating org files in this way, I only want to see
first level headlines, whereas my default refile binding uses deeper
levels.

Best,
Matt

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Merging .org files
  2011-03-19 12:53             ` Matt Lundin
@ 2011-03-19 17:42               ` Aankhen
  0 siblings, 0 replies; 13+ messages in thread
From: Aankhen @ 2011-03-19 17:42 UTC (permalink / raw)
  To: Matt Lundin, pquintana, emacs-orgmode, Michael Welle

On Sat, Mar 19, 2011 at 18:23, Matt Lundin <mdl@imapmail.org> wrote:
> Aankhen <aankhen@gmail.com> writes:
>> [snip]
>>
>> Do you normally have ‘org-completion-use-ido’ turned off or something?
>> (Just wondering why you couldn’t use ‘org-refile’ directly.)
>
> Yes, that is correct. I normally have org-completion-use-ido turned off.
>
> You could easily call org-refile with a prefix argument directly from
> within an org-buffer. However, I find it more convenient to bind
> "(org-refile t)" to one of the function keys than to type C-u C-c C-w.
> The latter works only on org buffers, while the former is global.
> Moreover, when navigating org files in this way, I only want to see
> first level headlines, whereas my default refile binding uses deeper
> levels.

A’right, makes sense.  I appreciate the detailed explanation.

Aankhen

^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2011-03-19 17:43 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-01-30 11:39 Merging .org files Pere Quintana Seguí
2011-01-30 17:39 ` Juan Pechiar
2011-01-30 22:09   ` Pere Quintana Seguí
2011-01-30 19:22 ` Michael Welle
2011-01-30 22:20   ` Pere Quintana Seguí
2011-01-30 21:51 ` Matt Lundin
2011-01-31 15:18   ` Pere Quintana Seguí
2011-01-31 20:45     ` Michael Welle
2011-02-07  8:28       ` Pere Quintana Seguí
2011-03-18 20:38         ` Matt Lundin
2011-03-19  8:51           ` Aankhen
2011-03-19 12:53             ` Matt Lundin
2011-03-19 17:42               ` Aankhen

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).