Re: One vs many directories

[Texas Cyberthal <texas.cyberthal@gmail.com>]

Hi Dr. Arne,

> The only part that hits performance limits is the agenda.

Well, IIRC your Org Textmind is much smaller than mine.

> My current guess is that the agenta is slow because it has to parse all my 7500 clock entries, and it has to check the Todo states of around 1200 headings.

Ouch.  I'd rather keep a "ramble log" so I can reconstruct an exactly
honest time accounting, with discounts for partial attention, without
worrying about fiddly clockin/outs.  At least when working from home.
If clocking into a work site, that's different, because one can
reasonably bill for the entire time, with minimal clock toggling.

> Did you check against filesystem limits? At 10k entries in a directory typical filesystems start becoming slow. That’s the main reason I see for adding hierarchies.

10k entries in a directory sounds inhumanely unergonomic.  I guess my
biggest flat name directory might eventually reach that size?  In
which case I could just split it in the middle of the alphabet, or
similar solution.

Right now it's only 600.  If I guess a generous growth rate of 2 per
day, times 30 years, that would be an additional 22k.  Sounds
manageable.

Remember there are ways to consolidate entries even in flat "solid
names" directory.  It's advantageous to do so to facilitate isearch
matching.  For example, everyone with the same last name is one
directory.  Ditto everything that starts with the same word or even
prefix.  For example I have a directory called ~Wiki-~ and another
called ~Tru-~ which contains truth, Trudeau and Trump.

Most adults know 20-35k words.  That's not the same as "solid names"
known, but gives a ballpark on human memory size for a similar name
type.  I suspect computers will advance faster than anyone's Textmind
reaches the Dired lag limit.

No, if we are talking about scaling limits, then limits such as buffer
size and Agenda search speed are orders of magnitude more relevant.
Which problems deep tree nesting fixes.

A 10k entry directory is getting into enterprise territory, and I'm
sure enterprise has tech tricks that become worthwhile at that scale.

> There are scaling problems in every direction: Too many files per directory, too large files, too much content per heading, too many headings.

There are scaling problems from too much deep tree nesting, namely too
much fiddly ambiguous manual refiling.  Solution is flat "solid name"
directories just below feasible 10 Bins.  Work fine.

> I would have to build lots of additional tooling to make that work as well. Many of the tools in Emacs work better on large files than on many files — I will switch to more files when performance on large files reaches its limits.

Nah, my 100 mb (non archived) Textmind works fine.  I just separated
Agenda metadata from bulk prose.

I am curious how many headings I have, how would I count that recursively?

On Sun, Nov 22, 2020 at 8:04 PM Dr. Arne Babenhauserheide
<arne_bab@web.de> wrote:
>
>
> Texas Cyberthal <texas.cyberthal@gmail.com> writes:
>
> >> I need instant search in the knowledge database and quick filing of tasks. Also I need the agenda to create a clocktable (that’s on the limit of being too slow) and the calendar and tasks of the week.
> >
> >> Also I need quick filing of notes and quotes (in specific files, not part of the agenda) and of long-form articles, one file per article (using journal.el, also outside the agenda, searched using M-x deft), and quick creation of website entries for a given category within the site (i.e. M-x draketo-software).
> >
> > So your Org usage style quickly hits critical performance problems at scale.
>
> The only part that hits performance limits is the agenda. All the rest
> scales nicely. My current guess is that the agenta is slow because it
> has to parse all my 7500 clock entries, and it has to check the TODO
> states of around 1200 headings. Having multiple files would only add to
> that.
>
> > I don't have these problems.  Treefactor refiling is immune to scale.
>
> Did you check against filesystem limits? At 10k entries in a directory
> typical filesystems start becoming slow. That’s the main reason I see
> for adding hierarchies.
>
> > Org's many tools and tricks are still handy in niche cases, but they
> > don't cause scaling problems because they don't handle bulk info
> > management.  For example Org's refile tools are useful when writing
> > advanced documentation with large single-file outlines.  Most info
> > doesn't require that much organization.  It works fine as flat lists
> > of headings in a detailed directory tree.
>
> Or as sub-headings in a large outline.
>
> There are scaling problems in every direction: Too many files per
> directory, too large files, too much content per heading, too many
> headings.
>
> I would have to build lots of additional tooling to make that work as
> well. Many of the tools in Emacs work better on large files than on many
> files — I will switch to more files when performance on large files
> reaches its limits.
>
> I have one file where I’m reaching the limit. That’ my 7.3 MiB
> emacs-remember-mode.org file where I throw long-form articles for
> full-text search. I am considering to switch to a multi-file approach
> for that and then to use deft to retrieve articles.
>
> Best wishes,
> Arne
> --
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken