Thoughts on the standardization of Org

Thoughts on the standardization of Org

[Asa Zeren]

Hi,

Even though I am new to the org-mode community, I would like to share
some thoughts on the specification of org-mode, especially since I
have seen some recent discussion of it in relation to registering org
as a MIME type.

First, I would like to repeat the importance of developing standards
for org-mode. If we want to expand the influence of org, tooling must
expand beyond Emacs. While Emacs is an amazing tool, (a) we cannot
convince the entire world to use Emacs and (b) org-mode should be
integrated into tooling unrelated to text editing, and is outside of
the Emacs-Lisp environment. Without additional org implementations,
this is impossible. If org catches on before it is standardized, we
end up in the situation of Markdown, with many competing standards and
non-standards. Hence, standardization is essential.

Standardizing org is much harder than standardizing something like
Markdown, but I think by breaking it down as follows will maximize the
portability of org while not compromising on development of org.

I see three areas of standardization, which I think should be
standardized separately:
 - Org DOM
 - Org Syntax
 - Org Standard Environments

Before we get to that, a brief note on /how/ I think that org should
be specified. I think that org should be specified in terms of an
/environment/ that defines the properties, etc. that can be used in a
document. For instance, the org standard would say something to the
effect of "An environment may specify block bounding keywords that may
be used like #+<kwd_0>\n...#+<kwd_1>. and the environment would specify
"begin_src and end_src are a pair of block bounding keyword that
indicates a source code block." This is for two reasons. First, this
allows for development of org tool features independent of the
standard. Second, this separates the individual features of org mode
from the overall structure.

Org DOM:
The first thing to specify is the org DOM. (Maybe a different name
should be used to avoid confusion with the HTML DOM) This is the
structure of an org-mode document, without the textual
representation. Many org-related tools operate on org documents
without needing to use the textual representation. Specifying the DOM
separately would (a) create a separation of concerns and (b) allow for
better libraries built around org mode.

Org Syntax:
This would be specifying the mapping between the DOM and the textual
representation, specified in terms of an environment.

Org Standard Environments:
This is how I would specify elements such as #+begin_src..#+end_src
would be specified, as standardized elements of the environment. This
would be structured as a number of individual standard environments,
such as "Source Blocks" or "Standard Header Properties" (specifying
#+title, #+author, etc.)

I would appreciate thoughts on these ideas about how to develop and
org specification.

Thanks for reading,
Asa Zeren

Re: Thoughts on the standardization of Org

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]


Asa Zeren <asaizeren@gmail.com> writes:
> I would appreciate thoughts on these ideas about how to develop and
> org specification.

The most important point I see here is to avoid hindering the
development of org-mode within Emacs.

So the most important part of the standard would be areas it doesn’t
standardize: Reserved for future use in org-mode.

These would then be sections that external tools must handle as opaque
text so their processing does not break usage within org-mode.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Thoughts on the standardization of Org

[Tim Cross]

I think there are a couple of important points to consider in
discussions of this type. I should state up-front, I am somewhat
sceptical regarding an org-mode which is separate or independent of
Emacs. Much of what makes org-mode so powerful and useful is due to
features of Emacs. While most, if not all, of these features could be
implemented in other solutions, the amount of work and level of
maintenance should not be under-estimated.

While standards and formal standardisation of something can be
important, it is often a dual edge sword. Over the last 30+ years
involved in technology, I have seen many good ideas come undone as the
result of a standardisation effort. consider for example, the results of
the lisp standardisation that resulted in common Lisp, what happened
with CORBA, xml-rpc and the move to REST based APIs. XHTML and the
breakdown of standardisation processes within the W3C and the
development of HTML5. Sometimes, we can emerge from a standardisation
process with a clearer, consistent standard that is easy to implement
and use. Other times, we can emerge with a complex, difficult to
implement and confusing standard which can kill or stifle further
development. The trick with standardisation seem to be getting the
balance right between clarity and complexity and focusing on the key
requirements, avoiding the trap of trying to cover everything.

One of the things I like a lot about org-mode is that it is not terribly
prescriptive. It provides a collection of features and functions which
you are able to assemble according to your own needs and preferences.
While I think it is important to have a clear idea of basic syntax for
each of these elements and how they relate to each other, I'm less
convinced we would want to prescribe an overly formal specification for
how an org document should be structured i.e. the org DOM idea. The
existing draft syntax document is probably sufficiently prescriptive
here already.

The four areas which I think would provide the greatest benefit would
be to

- Finalise the draft org syntax document on Worg, possibly adding it to
  the manual once complete. A considerable amount of work has already
  been put into this document and I think it is a good start.

- Define a specification for a property API which compliant org-mode
  implementation should support. This could be based on the existing
  ELisp mapping API.

- Define a specification for an element mapping API which compliant
  org-mode implementation should support. Again, this could be based on
  the existing ELisp element mapping API.

- Define a set of org reference documents. These would be documents that
  all compliant parsers should be able to process successfully. It might
  also be worthwhile including some documents with common errors which
  parses should be able to handle and recover from in a graceful manner.
  Those developing external tools can then use these documents as a
  guide and for testing their implementations.

Tim

Asa Zeren <asaizeren@gmail.com> writes:

> Hi,
>
> Even though I am new to the org-mode community, I would like to share
> some thoughts on the specification of org-mode, especially since I
> have seen some recent discussion of it in relation to registering org
> as a MIME type.
>
> First, I would like to repeat the importance of developing standards
> for org-mode. If we want to expand the influence of org, tooling must
> expand beyond Emacs. While Emacs is an amazing tool, (a) we cannot
> convince the entire world to use Emacs and (b) org-mode should be
> integrated into tooling unrelated to text editing, and is outside of
> the Emacs-Lisp environment. Without additional org implementations,
> this is impossible. If org catches on before it is standardized, we
> end up in the situation of Markdown, with many competing standards and
> non-standards. Hence, standardization is essential.
>
> Standardizing org is much harder than standardizing something like
> Markdown, but I think by breaking it down as follows will maximize the
> portability of org while not compromising on development of org.
>
> I see three areas of standardization, which I think should be
> standardized separately:
>  - Org DOM
>  - Org Syntax
>  - Org Standard Environments
>
> Before we get to that, a brief note on /how/ I think that org should
> be specified. I think that org should be specified in terms of an
> /environment/ that defines the properties, etc. that can be used in a
> document. For instance, the org standard would say something to the
> effect of "An environment may specify block bounding keywords that may
> be used like #+<kwd_0>\n...#+<kwd_1>. and the environment would specify
> "begin_src and end_src are a pair of block bounding keyword that
> indicates a source code block." This is for two reasons. First, this
> allows for development of org tool features independent of the
> standard. Second, this separates the individual features of org mode
> from the overall structure.
>
> Org DOM:
> The first thing to specify is the org DOM. (Maybe a different name
> should be used to avoid confusion with the HTML DOM) This is the
> structure of an org-mode document, without the textual
> representation. Many org-related tools operate on org documents
> without needing to use the textual representation. Specifying the DOM
> separately would (a) create a separation of concerns and (b) allow for
> better libraries built around org mode.
>
> Org Syntax:
> This would be specifying the mapping between the DOM and the textual
> representation, specified in terms of an environment.
>
> Org Standard Environments:
> This is how I would specify elements such as #+begin_src..#+end_src
> would be specified, as standardized elements of the environment. This
> would be structured as a number of individual standard environments,
> such as "Source Blocks" or "Standard Header Properties" (specifying
> #+title, #+author, etc.)
>
> I would appreciate thoughts on these ideas about how to develop and
> org specification.
>
> Thanks for reading,
> Asa Zeren


--
Tim Cross

Re: Thoughts on the standardization of Org

[Asa Zeren]

On Sat, Oct 31, 2020 at 8:40 PM Dr. Arne Babenhauserheide
<arne_bab@web.de> wrote:
> The most important point I see here is to avoid hindering the
> development of org-mode within Emacs.

While I definitely support enabling the further development of org-mode, and not
restricting it via a standard, I do see some problems.

> So the most important part of the standard would be areas it doesn’t
> standardize: Reserved for future use in org-mode.

The issue with this is that by picking what areas to reserve, one has
effectively limited the syntactic space that new features can use. This is not a
problem in and of itself, but does make the notion of leaving arbitrary syntax
space reserved impossible, particularly, since in org-mode and similar markup
languages, unadorned text is part of the content, rather than being ill formed,
as in programming languages. This also does not mean that tools can interpret
part of what org-mode considers content as having some domain or implementation
specific meaning. For example, latex blocks. In my opinion, the translation of
these are a language extension by the org-export tool. Even within parts of the
Emacs org implementation, latex blocks should not be considered part of the org
language. For example, the line "* Headline?" in the example below is still
identified as a headline, even though, if the area inside the \begin and \end
commands were supposed to be latex, not org, it should not be.

#+begin_example org
\begin{equation}
* Headline?
\end{equation}
#+end_example

> The most important point I see here is to avoid hindering the development of
> org-mode within Emacs.

> These would then be sections that external tools must handle as
> opaque text so their processing does not break usage within
> org-mode.

In these concerns I see one major flaw. The way they are worded at present
implies that the Emacs implementation of org is the "one true implementation,"
and that all tools in other environments are auxiliary. I believe that if we
want org to grow, then it needs to become unbound from Emacs. It should become a
universal markup format, which just happens to have had many tools first
implemented for Emacs (even if Emacs still will probably remain the best way to
edit org files).

Best,

Asa

On Sat, Oct 31, 2020 at 8:40 PM Dr. Arne Babenhauserheide
<arne_bab@web.de> wrote:
>
>
> Asa Zeren <asaizeren@gmail.com> writes:
> > I would appreciate thoughts on these ideas about how to develop and
> > org specification.
>
> The most important point I see here is to avoid hindering the
> development of org-mode within Emacs.
>
> So the most important part of the standard would be areas it doesn’t
> standardize: Reserved for future use in org-mode.
>
> These would then be sections that external tools must handle as opaque
> text so their processing does not break usage within org-mode.
>
> Best wishes,
> Arne
> --
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken

Re: Thoughts on the standardization of Org

[Pankaj Jangid]

Tim Cross <theophilusx@gmail.com> writes:

> I should state up-front, I am somewhat sceptical regarding an org-mode
> which is separate or independent of Emacs. Much of what makes org-mode
> so powerful and useful is due to features of Emacs. While most, if not
> all, of these features could be implemented in other solutions, the
> amount of work and level of maintenance should not be under-estimated.

Not only that. This will really slowdown the evolution of org-mode if
the standardization is outside the control of Emacs Developers.

> Over the last 30+ years involved in technology, I have seen many good
> ideas come undone as the result of a standardisation effort. consider
> for example, the results of the lisp standardisation that resulted in
> common Lisp, what happened with CORBA, xml-rpc and the move to REST
> based APIs. XHTML and the breakdown of standardisation processes
> within the W3C and the development of HTML5.

I am not only skeptical, I totally believe that this sort of
standardization where some other party is giving org-mode a certificate,
will be harmful for the development of org-mode.

> The four areas which I think would provide the greatest benefit would
> be to
>
> - Finalise the draft org syntax document on Worg, possibly adding it
> to
>   the manual once complete. A considerable amount of work has already
>   been put into this document and I think it is a good start.

And this should be the only source of truth. If MIME type registration
changes this then better to avoid that.

> - Define a specification for a property API which compliant org-mode
>   implementation should support. This could be based on the existing
>   ELisp mapping API.
>
> - Define a specification for an element mapping API which compliant
>   org-mode implementation should support. Again, this could be based
>   on the existing ELisp element mapping API.
>
> - Define a set of org reference documents. These would be documents
> that
>   all compliant parsers should be able to process successfully. It
>   might also be worthwhile including some documents with common errors
>   which parses should be able to handle and recover from in a graceful
>   manner.  Those developing external tools can then use these
>   documents as a guide and for testing their implementations.

Agree. And again... the standard place should be Worg only.

-- Pankaj

Re: Thoughts on the standardization of Org

[Pankaj Jangid]

Asa Zeren <asaizeren@gmail.com> writes:

> In these concerns I see one major flaw. The way they are worded at
> present implies that the Emacs implementation of org is the "one true
> implementation," and that all tools in other environments are
> auxiliary.

At present, that is the truth. Where are other implementations that we
can call true implementations? May be I am not aware of them. Could you
please list a few?

> I believe that if we want org to grow, then it needs to become unbound
> from Emacs. It should become a universal markup format, which just
> happens to have had many tools first implemented for Emacs (even if
> Emacs still will probably remain the best way to edit org files).

I would love to see org becoming a universal markup format. My wish is
that the standard should be under control of Worg. This should be the
only source of truth and implementors may contribute to the standard by
participating in the community or may be some steering committee here.

Re: Thoughts on the standardization of Org

[Tom Gillespie]

Hi Asa,
    My general take is that any active work toward standardization
would be premature. At the very least a full implementation outside
of Emacs would need to exist. In the absence of that there is little
point to standardization. There is ample existing documentation to
build a compliant parser (pandoc exists as well ...) and any effort
toward standardization right now would be better spent improving
the existing implementation or fixing broken ones (e.g. org-ruby).

From your comments, I would suggest reading through
https://orgmode.org/worg/dev/org-syntax.html if you have not
done so already. Much of what you mention is already there.

If something like standardization is still desired, I would suggest that
the proper framing for any such activities would be as improvement and
clarification in the documentation, and potentially as formalization of some
of the existing behaviors of the system. Org is a fairly stable system,
and as others have said, explicitly leaving things open an unspecified
would be vital.

There are also parts of org (e.g. babel) where the behavior needs to be
regularized and made consistent. At the moment those areas need
contributors, not standardization.

A few more thoughts in line. Best!
Tom

On Sat, Oct 31, 2020 at 8:22 PM Asa Zeren <asaizeren@gmail.com> wrote:

> this is impossible. If org catches on before it is standardized, we
> end up in the situation of Markdown, with many competing standards and
> non-standards. Hence, standardization is essential.

The situation for Org is not comparable to markdown. There is a single reference
implementation for org at the moment. The codebase is massive. There are many
existing parsers for org files. Many are obviously broken since they
do not match the reference
implementation's behavior. The obviousness is a sign that there is not a need
for standardization at this time. Further, there is little risk that
another impl will
be created without interoperating with the elisp implementation. For example,
consider Mauro's use case: being able to get colleagues who do not use Emacs
to use Org. I suspect most of the people who would be working on other
implementations
would be starting from Emacs and would be unlikely to leave. Also
unlike markdown,
html export is just one tiny part of Org, whereas markdown was
implemented repeatedly
to allow text input on web pages where people needed to implement
parts of html that
had not already been specified in markdown.

> Standardizing org is much harder than standardizing something like
> Markdown, but I think by breaking it down as follows will maximize the
> portability of org while not compromising on development of org.

See some of my other recent emails. In the short term this is impossible
due to the deep dependence on Emacs Lisp. Any outside implementation
that is created today would have to implement elisp. Few have been able
to do this in over 30 years. Moving beyond elisp requires additional machinery
to be added to org to be able to specify other top level langauges. This is
not something that is remotely ready for standardization because no one
even has a single working implementation yet!

> I see three areas of standardization, which I think should be
> standardized separately:
>  - Org DOM
No. This is an implementation detail (see below for more).

>  - Org Syntax
This is pretty much done, there are some outstanding points for discussion,
but they are about implementation details, not about the contents of the
syntax. Also extension of the syntax needs to be open and defined entirely
by the elisp implementation, as mentioned by others.

>  - Org Standard Environments
Read https://orgmode.org/worg/dev/org-syntax.html. It will get you up to speed
with the existing terminology that is used in the community.

>
> Org DOM:
> The first thing to specify is the org DOM. (Maybe a different name
> should be used to avoid confusion with the HTML DOM) This is the
> structure of an org-mode document, without the textual
> representation. Many org-related tools operate on org documents
> without needing to use the textual representation. Specifying the DOM
> separately would (a) create a separation of concerns and (b) allow for
> better libraries built around org mode.

Depending on exactly what you mean by DOM this does not need to be standardized.
There are a couple of points that need to be clarified regarding how
to treeify the flat
list of elements that come out of a parse in order to tie things like
associated keywords
to the correct elements, but these are quite minimal. The potential
rats nest that is
trying to standardize a DOM when it is an implementation detail means
that I would
strongly discourage even thinking about Org in that way. I would even
discourage putting
too much emphasis on the org-element api which, while extremely useful
inside Emacs,
is not something that should be standardized because it is a detail
peculiar to the elisp
implementation.

There are cases where certain behaviors, such as how to parse and format
footnotes, could be specified, but such behaviors don't require a dom in order
to be specified, and adding a DOM to the picture does nothing but complicate
the format. Org is a text format. The semantics for interaction with the text
format are defined entirely by the text representation (In Emacs
there.is.only.buffer).
Other semantics, such as export to html and latex, are not something that you
would want to try to standardize, you would likely lose friends, enemies, and
whatever sanity you had left at the end (see discussion on Mauro's thread about
the fact that it is probably just easier to use Emacs directly if you
need to export
to a certain format in a specific way. It is free software after all.)

To the extent that an element tree could be useful, I think it would
be as a concept
in an implementation guide, not as something formally specified.

> Org Syntax:
> This would be specifying the mapping between the DOM and the textual
> representation, specified in terms of an environment.

There is no DOM. Modification to an org document must be made on the
text representation otherwise it is meaningless. This isn't html where there
is no canonical representation outside the DOM. The text representation of
an org document IS the canonical representation (modulo a normalization
pass).

> Org Standard Environments:
> This is how I would specify elements such as #+begin_src..#+end_src
> would be specified, as standardized elements of the environment. This
> would be structured as a number of individual standard environments,
> such as "Source Blocks" or "Standard Header Properties" (specifying
> #+title, #+author, etc.)

These are well specified already in the worg syntax draft. There are a couple
of special cases such as src and example blocks that could be included
explicitly in the syntax to facilitate interoperability with parsers
for org babel
languages. Beyond that, the community already has vocabulary that covers
what you describe here, as mentioned above.

Re: Thoughts on the standardization of Org

[TEC]

Hi all,

Following what I've read on the list I've developed thoughts on 
what the
best approach might be. My current thinking is that it may be 
possible
to have Org registered as a standard in such a way that it does 
not
constrain our development efforts.

How?

We forgo locking down the precise format and behaviour of every 
Org
element. Instead we only submit something like
https://orgmode.org/worg/dev/org-syntax.html which /just/ 
describes the
overall syntax. I don't imagine that 'locking' ourself into the 
current
syntax described in https://orgmode.org/worg/dev/org-syntax.html 
would
actually hurt development, but might be enough for a MIME type 
etc.

Then perhaps just say for description of how specific/special 
instances
of those structural elements are supposed to work see the 
reference
implementation.

Just a few thoughts from me.

All the best,
Timothy.

Asa Zeren <asaizeren@gmail.com> writes:

> Hi,
>
> Even though I am new to the org-mode community, I would like to 
> share
> some thoughts on the specification of org-mode, especially since 
> I
> have seen some recent discussion of it in relation to 
> registering org
> as a MIME type.
>
> First, I would like to repeat the importance of developing 
> standards
> for org-mode. If we want to expand the influence of org, tooling 
> must
> expand beyond Emacs. While Emacs is an amazing tool, (a) we 
> cannot
> convince the entire world to use Emacs and (b) org-mode should 
> be
> integrated into tooling unrelated to text editing, and is 
> outside of
> the Emacs-Lisp environment. Without additional org 
> implementations,
> this is impossible. If org catches on before it is standardized, 
> we
> end up in the situation of Markdown, with many competing 
> standards and
> non-standards. Hence, standardization is essential.
>
> Standardizing org is much harder than standardizing something 
> like
> Markdown, but I think by breaking it down as follows will 
> maximize the
> portability of org while not compromising on development of org.
>
> I see three areas of standardization, which I think should be
> standardized separately:
>  - Org DOM
>  - Org Syntax
>  - Org Standard Environments
>
> Before we get to that, a brief note on /how/ I think that org 
> should
> be specified. I think that org should be specified in terms of 
> an
> /environment/ that defines the properties, etc. that can be used 
> in a
> document. For instance, the org standard would say something to 
> the
> effect of "An environment may specify block bounding keywords 
> that may
> be used like #+<kwd_0>\n...#+<kwd_1>. and the environment would 
> specify
> "begin_src and end_src are a pair of block bounding keyword that
> indicates a source code block." This is for two reasons. First, 
> this
> allows for development of org tool features independent of the
> standard. Second, this separates the individual features of org 
> mode
> from the overall structure.
>
> Org DOM:
> The first thing to specify is the org DOM. (Maybe a different 
> name
> should be used to avoid confusion with the HTML DOM) This is the
> structure of an org-mode document, without the textual
> representation. Many org-related tools operate on org documents
> without needing to use the textual representation. Specifying 
> the DOM
> separately would (a) create a separation of concerns and (b) 
> allow for
> better libraries built around org mode.
>
> Org Syntax:
> This would be specifying the mapping between the DOM and the 
> textual
> representation, specified in terms of an environment.
>
> Org Standard Environments:
> This is how I would specify elements such as 
> #+begin_src..#+end_src
> would be specified, as standardized elements of the environment. 
> This
> would be structured as a number of individual standard 
> environments,
> such as "Source Blocks" or "Standard Header Properties" 
> (specifying
> #+title, #+author, etc.)
>
> I would appreciate thoughts on these ideas about how to develop 
> and
> org specification.
>
> Thanks for reading,
> Asa Zeren

Re: Thoughts on the standardization of Org

[Tim Cross]

Asa Zeren <asaizeren@gmail.com> writes:

>
> In these concerns I see one major flaw. The way they are worded at present
> implies that the Emacs implementation of org is the "one true implementation,"
> and that all tools in other environments are auxiliary. I believe that if we
> want org to grow, then it needs to become unbound from Emacs. It should become a
> universal markup format, which just happens to have had many tools first
> implemented for Emacs (even if Emacs still will probably remain the best way to
> edit org files).
>

I think the above is likely the biggest point of disagreement amongst
many org users. Org is not simply a different markup language. Org-mode
is primarily and Emacs mode to help organise *things*. The fact
non-Emacs users have seen it, like it and wish it was available outside
of emacs is great, but for me, not a significant objective. Anything
which took org-mode out of Emacs or made non-Emacs requirements a higher
priority than the on-going development of the mode in Emacs would be
misplaced.

For me and I suspect many others, the ability for non-Emacs users to
work with org files is a 'nice to have', but not a key objective. The
main objective is to make org-mode an even more useful and powerful
Emacs package. I do over 90% of my work from within Emacs - writing this
email, browsing documentation, writing code in various languages,
developing software requirement documentation, writing user manuals,
interacting with version control, compiling, testing software, working
with databases, tracking tasks and time, invoicing clients, planning and
managing projects, appointments etc. About the only things I don't do
with Emacs are general web browsing (because I need Javascript support),
playing music (though I can do this from Emacs as well) and playing the
occasional game.

The old cliche about open source is that it is about scratching your own
itch. For many org-mode users, that means configuring and extending
org-mode as an Emacs package. I suspect this is part of the reason there
are few org-mode tools outside of Emacs. Many of those which do exist
seem to be more oriented towards minimal org-mode support on platforms
where Emacs cannot run (or run easily), like mobile devices. There are
other efforts to bring org-mode into other editors, like vscode, but the
lack of a powerful elisp equivalent (particularly one with all the
editor oriented support of elisp) makes the effort prohibitive.

In many respects, the real 'genius' of org-mode was that it took many of
the separate features and existing functions of Emacs and elisp, wrapped
them all up in a consistent interface called org-mode. Many of the
features and much of the functionality which makes up org-mode existed
in Emacs before org-mode was developed (folding, overlays, font-lock,
inferior process buffers, latex fragment previews, hyperlinks,
encryption, networking, etc.). To implement all of this on another
platform is a large job. Even editors which support many of these
individual features can't easily achieve the same level of integration
because those features are implemented as distinct modules and lack the
consistent extensible power of elisp.

I'm sorry if all of this and other responses seem too negative. The
desire to bring org-mode to non-Emacs users is a worthy goal, but I fear
the size of the task is too large. What may be feasible are stand-alone
org-mode parses which can make up parts of tool chains for limited
processing of org files and perhaps basic org-mode modules for other
editors which enable basic editing of org files.

Tim

Re: Thoughts on the standardization of Org

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 4424 bytes --]

> see discussion on Mauro's thread about
> the fact that it is probably just easier to use Emacs directly if you
> need to export
> to a certain format in a specific way. It is free software after all.

I would like to add, that this is pretty easy to do, and also to make
independent of the users emacs environment. Here is an example that
uses the whole orgmode-babel-latex-html machinery to create derived
documents from source-of-truth org-mode files which get exported to a
book:
https://hg.sr.ht/~arnebab/ews/browse/Hauptdokument/ews30/Makefile.am?rev=b8e3899c6d8b#L121


chargen.tex: chargen.org $(ewstables_SOURCES) kasten-alter-groesse-gewicht.org ews30setup.el
	echo yes > $$(tty); Xvfb :3 -screen 0 1024x768x16 & time DISPLAY=:3 HOME=@abs_top_srcdir@ @EMACS@ -l "@abs_top_srcdir@/ews30setup.el" --eval '(setq vc-follow-symlinks nil)' --eval '(setq org-id-locations-file "@abs_top_builddir@/.org-id-locations")' "$<" -e org-latex-export-to-latex -e kill-emacs  < $$(tty) >> build.log && rm -f "@abs_top_builddir@/.org-id-locations"

Note how this sets the HOME to the sourcedir (so a project-specific
.emacs.d setup is used) and loads ews30setup.el at startup for
additional customization. Also note the call to Xvfb which avoids
showing a graphical Emacs during build.


This uses an org-mode file that pulls data from tables in other org-mode
files by setting variables for code based on autotools-included
datafiles. Here’s an example of pulling the tables into variables:
https://hg.sr.ht/~arnebab/ews/browse/Hauptdokument/ews30/chargen.org.in?rev=b8e3899c6d8b#L153

#+begin_src scheme :exports none :results output raw :prologue "(import (srfi srfi-1)(ice-9 match)(ice-9 receive))(set! *random-state*  (random-state-from-platform))\n" :tangle chargen.scm :noweb yes :var kernantriebe=tabelle-kernantriebe :var hautfarbe=tabelle-hautfarbe :var haarfarbe=tabelle-haarfarbe :var augenfarbe=tabelle-augenfarbe :var darstellung1=tabelle-darstellung1 :var darstellung2=tabelle-darstellung2 :var kleidung_oben_maenner=tabelle-kleidung-fantasy-oben-maenner  :var kleidung_unten_maenner=tabelle-kleidung-fantasy-unten-maenner :var kleidung_oben_frauen=tabelle-kleidung-fantasy-oben-frauen  :var kleidung_unten_frauen=tabelle-kleidung-fantasy-unten-frauen :var kleidung_oben_frauen=tabelle-kleidung-fantasy-oben-frauen  :var kleidung_unten_frauen=tabelle-kleidung-fantasy-unten-frauen :var namen=tabelle-namen-fantasy-jetzt :var sex=tabelle-sexualitaet :var stichwort=tabelle-stichwort-fantasy
  (let ()
    {{{chargen-setup}}}
    {{{chargen-generic}}}
    {{{chargen-colors}}}
    {{{chargen-specifics-fantasy}}}
    {{{chargen-print-char}}}
    (chargen-print-char)
  )
#+end_src


Note the {{{…}}} blocks. Those use literate programming to include
blocks defined below, with customized separators:
chargen-setup block: https://hg.sr.ht/~arnebab/ews/browse/Hauptdokument/ews30/chargen.org.in?rev=b8e3899c6d8b#L360
customization of separators: https://hg.sr.ht/~arnebab/ews/browse/Hauptdokument/ews30/chargen.org.in?rev=b8e3899c6d8b#L638
# Local Variables:
# org-confirm-babel-evaluate: nil
# org-export-allow-bind-keywords: t
# org-babel-noweb-wrap-start: "{{{"
# org-babel-noweb-wrap-end: "}}}"
# End:


Here’s how it pulls tables:
https://hg.sr.ht/~arnebab/ews/browse/Hauptdokument/ews30/chargen.org.in?rev=b8e3899c6d8b#L578
@tabelle_aussehen@


And this is an example of the datafiles that are used as source-of-truth
and also directly inluded in the main book as tables:
https://hg.sr.ht/~arnebab/ews/browse/Hauptdokument/ews30/tabelle-aussehen.org?rev=b8e3899c6d8b#L578

#+tblname: tabelle-hautfarbe
|      | -5     | direkt      | 6              |
|------+--------+-------------+----------------|
|   -3 | blass  | rosig       | sommersprossig |
|   -1 | grau   | gelblich    | elfenbein      |
|    2 | kupfer | rotbraun    | bronze         |
|    4 | oliv   | dunkelbraun | schwarz        |
| -5/6 | albino | -           | fleckig        |


All this machinery can be invoked without ever seeing Emacs.


So yes, the Emacs implementation is the source of truth, and yes, this
can be used without requiring people to operate Emacs by simply using
Emacs as utility with project-specific setup — just as you would do it
with a compiler.


Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Thoughts on the standardization of Org

[TEC]

I feel that this also ties into my earlier idea of putting Emacs
as/inside an LSP server for Org.  I suspect there may be a a lot 
of
potential in making it dead easy to use Emacs as a tool.

I'm rather busy over the next few weeks, but I'd be happy to 
spearhead a
project in this direction.

Timothy.

Dr. Arne Babenhauserheide <arne_bab@web.de> writes:

>> see discussion on Mauro's thread about
>> the fact that it is probably just easier to use Emacs directly 
>> if you
>> need to export
>> to a certain format in a specific way. It is free software 
>> after all.
>
> I would like to add, that this is pretty easy to do, and also to 
> make
> independent of the users emacs environment. Here is an example 
> that
> uses the whole orgmode-babel-latex-html machinery to create 
> derived
> documents from source-of-truth org-mode files which get exported 
> to a
> book:
> https://hg.sr.ht/~arnebab/ews/browse/Hauptdokument/ews30/Makefile.am?rev=b8e3899c6d8b#L121
>
>
> chargen.tex: chargen.org $(ewstables_SOURCES) 
> kasten-alter-groesse-gewicht.org ews30setup.el
> 	echo yes > $$(tty); Xvfb :3 -screen 0 1024x768x16 & time 
> DISPLAY=:3 HOME=@abs_top_srcdir@ @EMACS@ -l 
> "@abs_top_srcdir@/ews30setup.el" --eval '(setq 
> vc-follow-symlinks nil)' --eval '(setq org-id-locations-file 
> "@abs_top_builddir@/.org-id-locations")' "$<" -e 
> org-latex-export-to-latex -e kill-emacs  < $$(tty) >> build.log 
> && rm -f "@abs_top_builddir@/.org-id-locations"
>
> Note how this sets the HOME to the sourcedir (so a 
> project-specific
> .emacs.d setup is used) and loads ews30setup.el at startup for
> additional customization. Also note the call to Xvfb which 
> avoids
> showing a graphical Emacs during build.
>
>
> This uses an org-mode file that pulls data from tables in other 
> org-mode
> files by setting variables for code based on autotools-included
> datafiles. Here’s an example of pulling the tables into 
> variables:
> https://hg.sr.ht/~arnebab/ews/browse/Hauptdokument/ews30/chargen.org.in?rev=b8e3899c6d8b#L153
>
> #+begin_src scheme :exports none :results output raw :prologue 
> "(import (srfi srfi-1)(ice-9 match)(ice-9 receive))(set! 
> *random-state*  (random-state-from-platform))\n" :tangle 
> chargen.scm :noweb yes :var kernantriebe=tabelle-kernantriebe 
> :var hautfarbe=tabelle-hautfarbe :var 
> haarfarbe=tabelle-haarfarbe :var augenfarbe=tabelle-augenfarbe 
> :var darstellung1=tabelle-darstellung1 :var 
> darstellung2=tabelle-darstellung2 :var 
> kleidung_oben_maenner=tabelle-kleidung-fantasy-oben-maenner 
> :var 
> kleidung_unten_maenner=tabelle-kleidung-fantasy-unten-maenner 
> :var kleidung_oben_frauen=tabelle-kleidung-fantasy-oben-frauen 
> :var kleidung_unten_frauen=tabelle-kleidung-fantasy-unten-frauen 
> :var kleidung_oben_frauen=tabelle-kleidung-fantasy-oben-frauen 
> :var kleidung_unten_frauen=tabelle-kleidung-fantasy-unten-frauen 
> :var namen=tabelle-namen-fantasy-jetzt :var 
> sex=tabelle-sexualitaet :var stichwort=tabelle-stichwort-fantasy
>   (let ()
>     {{{chargen-setup}}}
>     {{{chargen-generic}}}
>     {{{chargen-colors}}}
>     {{{chargen-specifics-fantasy}}}
>     {{{chargen-print-char}}}
>     (chargen-print-char)
>   )
> #+end_src
>
>
> Note the {{{…}}} blocks. Those use literate programming to 
> include
> blocks defined below, with customized separators:
> chargen-setup block: 
> https://hg.sr.ht/~arnebab/ews/browse/Hauptdokument/ews30/chargen.org.in?rev=b8e3899c6d8b#L360
> customization of separators: 
> https://hg.sr.ht/~arnebab/ews/browse/Hauptdokument/ews30/chargen.org.in?rev=b8e3899c6d8b#L638
> # Local Variables:
> # org-confirm-babel-evaluate: nil
> # org-export-allow-bind-keywords: t
> # org-babel-noweb-wrap-start: "{{{"
> # org-babel-noweb-wrap-end: "}}}"
> # End:
>
>
> Here’s how it pulls tables:
> https://hg.sr.ht/~arnebab/ews/browse/Hauptdokument/ews30/chargen.org.in?rev=b8e3899c6d8b#L578
> @tabelle_aussehen@
>
>
> And this is an example of the datafiles that are used as 
> source-of-truth
> and also directly inluded in the main book as tables:
> https://hg.sr.ht/~arnebab/ews/browse/Hauptdokument/ews30/tabelle-aussehen.org?rev=b8e3899c6d8b#L578
>
> #+tblname: tabelle-hautfarbe
> |      | -5     | direkt      | 6              |
> |------+--------+-------------+----------------|
> |   -3 | blass  | rosig       | sommersprossig |
> |   -1 | grau   | gelblich    | elfenbein      |
> |    2 | kupfer | rotbraun    | bronze         |
> |    4 | oliv   | dunkelbraun | schwarz        |
> | -5/6 | albino | -           | fleckig        |
>
>
> All this machinery can be invoked without ever seeing Emacs.
>
>
> So yes, the Emacs implementation is the source of truth, and 
> yes, this
> can be used without requiring people to operate Emacs by simply 
> using
> Emacs as utility with project-specific setup — just as you would 
> do it
> with a compiler.
>
>
> Best wishes,
> Arne

Re: Thoughts on the standardization of Org

[Gustav Wikström]

Hi,

I agree with your sentiment Asa. It would indeed be good to "standardize" Org. It's worth spending a few words here reasoning about what this standardization would mean. The text below are not specifically to you, Asa. But to the list. As food for thought on this topic. FWIW.

It's easy to be hesitant to standardization, thinking it will slow down and limit development. Personally I don't think Org mode is at risk of that. The issues come first with coordination between multiple parties. Especially if the visions, goals and perspectives of the parties differ. For Org mode this coordination should not be an issue, since no one as of now could dispute that Emacs Org mode implementation is the standard implementation. Few would also dispute which party has the leading role in defining the standard. (I.e. this community, with the maintainers as the "signing" bodies, define the standard. And it can be done either in the manual or in worg).

Issues with a standard hindering evolution of Emacs Org mode can be dealt with in the same way as the evolution of Org mode itself is handled. By versioning. Right now the Org mode version effectively declares what the DOM and syntax is. If we can extract that information from the source code. And assign a version to the DOM and syntax so they can be managed separate from the Emacs implementation, it's quite easy to evolve them as well. Although I think the syntax will evolve much less than the tool itself, since much of the changes aren't about changing the syntax but rather about changing the way Emacs augments and works with that syntax!

What it really boils down to, I think, is that there is a benefit of a clear document describing what an org mode document can consist of (DOM in your terminology I suppose?) and what the textual representation of that is (the syntax). Put a version number on that/those things that can be incremented as the community see fit. And it's done. Standard is defined.  No third party should need to sign it. It would be the "Emacs Org mode" community standardization of the Org mode object model and textual syntax and document format. And that in itself should be more than enough to get the ".org" file extension globally approved. And help parser developers and other tool developers to support the format. AND help further develop Emacs implementation. Discussions regarding composing these documents are already started in the MIME-type threads. In my humble opinion there is not much else needed to get this "standardization" done.

Nicolas has started the journey in an excellent way with the Org element documentation and source code library in my opinion. Hats off to him! Anyone willing of following in his footstep gets another hat off. I'm sure it will be of great benefit to all Org mode users out there!

Kind regards
Gustav

> -----Ursprungligt meddelande-----
> Från: Emacs-orgmode <emacs-orgmode-bounces+gustav=whil.se@gnu.org> För Asa
> Zeren
> Skickat: den 1 november 2020 01:22
> Till: emacs-orgmode@gnu.org
> Ämne: Thoughts on the standardization of Org
> 
> Hi,
> 
> Even though I am new to the org-mode community, I would like to share
> some thoughts on the specification of org-mode, especially since I
> have seen some recent discussion of it in relation to registering org
> as a MIME type.
> 
> First, I would like to repeat the importance of developing standards
> for org-mode. If we want to expand the influence of org, tooling must
> expand beyond Emacs. While Emacs is an amazing tool, (a) we cannot
> convince the entire world to use Emacs and (b) org-mode should be
> integrated into tooling unrelated to text editing, and is outside of
> the Emacs-Lisp environment. Without additional org implementations,
> this is impossible. If org catches on before it is standardized, we
> end up in the situation of Markdown, with many competing standards and
> non-standards. Hence, standardization is essential.
> 
> Standardizing org is much harder than standardizing something like
> Markdown, but I think by breaking it down as follows will maximize the
> portability of org while not compromising on development of org.
> 
> I see three areas of standardization, which I think should be
> standardized separately:
>  - Org DOM
>  - Org Syntax
>  - Org Standard Environments
> 
> Before we get to that, a brief note on /how/ I think that org should
> be specified. I think that org should be specified in terms of an
> /environment/ that defines the properties, etc. that can be used in a
> document. For instance, the org standard would say something to the
> effect of "An environment may specify block bounding keywords that may
> be used like #+<kwd_0>\n...#+<kwd_1>. and the environment would specify
> "begin_src and end_src are a pair of block bounding keyword that
> indicates a source code block." This is for two reasons. First, this
> allows for development of org tool features independent of the
> standard. Second, this separates the individual features of org mode
> from the overall structure.
> 
> Org DOM:
> The first thing to specify is the org DOM. (Maybe a different name
> should be used to avoid confusion with the HTML DOM) This is the
> structure of an org-mode document, without the textual
> representation. Many org-related tools operate on org documents
> without needing to use the textual representation. Specifying the DOM
> separately would (a) create a separation of concerns and (b) allow for
> better libraries built around org mode.
> 
> Org Syntax:
> This would be specifying the mapping between the DOM and the textual
> representation, specified in terms of an environment.
> 
> Org Standard Environments:
> This is how I would specify elements such as #+begin_src..#+end_src
> would be specified, as standardized elements of the environment. This
> would be structured as a number of individual standard environments,
> such as "Source Blocks" or "Standard Header Properties" (specifying
> #+title, #+author, etc.)
> 
> I would appreciate thoughts on these ideas about how to develop and
> org specification.
> 
> Thanks for reading,
> Asa Zeren

Re: Thoughts on the standardization of Org

[Asa Zeren]

Thanks for the comments.

Both of you have raised some very good points, but I think that there has been
some confusion as to a number of my arguments. I hope to clarify some things
below.

On Sun Nov. 1, 2020, at 1:20AM Tom Gillespie<tgbugs@gmail.com> wrote:
> My general take is that any active work toward standardization
> would be premature. At the very least a full implementation outside
> of Emacs would need to exist. In the absence of that there is little
> point to standardization. There is ample existing documentation to
> build a compliant parser (pandoc exists as well ...) and any effort
> toward standardization right now would be better spent improving
> the existing implementation or fixing broken ones (e.g. org-ruby).

This could very well be the case. When to create a formal standard is a very
hard question, and there are lots of reasons for it to be too early.

One point I do think needs to be clarified is the extent of a "full
implementation". I don't think that a full editing environment like the one that
exists in Emacs today needs to exist, only a fully functional export
framework. This would require it to understand the full org syntax and
semantics.

Also, part of the reason I wrote my original thoughts is because I observed
some motivation towards standardization, as part of the MIME type effort.

> From your comments, I would suggest reading through
> https://orgmode.org/worg/dev/org-syntax.html if you have not done so
> already. Much of what you mention is already there.

I did give it a read, and I have just given it another read. While I do confess
I did make some terminology mistakes, most of my points still stand after giving
it another read through.

> If something like standardization is still desired, I would suggest that the
> proper framing for any such activities would be as improvement and
> clarification in the documentation, and potentially as formalization of some
> of the existing behaviors of the system. Org is a fairly stable system, and as
> others have said, explicitly leaving things open an unspecified would be
> vital.  There are also parts of org (e.g. babel) where the behavior needs to
> be regularized and made consistent. At the moment those areas need
> contributors, not standardization.

I do agree that this is the right method of creating the standard. Org-mode is a
very large beast to standardize, and it can only be done incrementally, or it is
doomed to fail.

> On Sat, Oct 31, 2020 at 8:22 PM Asa Zeren <asaizeren@gmail.com> wrote:
> > this is impossible. If org catches on before it is standardized, we end up
> > in the situation of Markdown, with many competing standards and
> > non-standards. Hence, standardization is essential.
> The situation for Org is not comparable to markdown. There is a single
> reference implementation for org at the moment. The codebase is massive. There
> are many existing parsers for org files. Many are obviously broken since they
> do not match the reference implementation's behavior. The obviousness is a
> sign that there is not a need for standardization at this time. Further, there
> is little risk that another impl will be created without interoperating with
> the elisp implementation. For example, consider Mauro's use case: being able
> to get colleagues who do not use Emacs to use Org. I suspect most of the
> people who would be working on other implementations would be starting from
> Emacs and would be unlikely to leave. Also unlike markdown, html export is
> just one tiny part of Org, whereas markdown was implemented repeatedly to
> allow text input on web pages where people needed to implement parts of html
> that had not already been specified in markdown.

I agree that this is the current situation. However, there is a real danger
here. People are continually trying to create org implementations (myself
included), and if one of these is successful before an org standard is created,
and it differs from the original elisp implementation in non-trivial to fix
ways, then we have an issue. Perhaps this will not come to pass, and other
implementations should strive for parity, but it is still a danger.

> Standardizing org is much harder than standardizing something like Markdown,
> but I think by breaking it down as follows will maximize the portability of
> org while not compromising on development of org. See some of my other
> recent emails. In the short term this is impossible due to the deep
> dependence on Emacs Lisp. Any outside implementation that is created today
> would have to implement elisp. Few have been able to do this in over 30
> years. Moving beyond elisp requires additional machinery to be added to
> org to be able to specify other top level languages. This is not something
> that is remotely ready for standardization because no one even has a single
> working implementation yet!

I definitely agree that a deep dependence on Emacs Lisp should not be
standardized, and thus there are certain parts of the current org-mode
implementation that cannot be currently specified. However, there still are
areas of org that /can/ be specified without elisp, and we should not stop
standardization of anything because of some things.

> > I see three areas of standardization, which I think should be standardized
> > separately:
> > - Org DOM
> No. This is an implementation detail (see below for more).
> ...
> Depending on exactly what you mean by DOM this does not need to be
> standardized.  There are a couple of points that need to be clarified
> regarding how to treeify the flat list of elements that come out of a parse in
> order to tie things like associated keywords to the correct elements, but
> these are quite minimal. The potential rats nest that is trying to standardize
> a DOM when it is an implementation detail means that I would strongly
> discourage even thinking about Org in that way. I would even discourage
> putting too much emphasis on the org-element api which, while extremely useful
> inside Emacs, is not something that should be standardized because it is a
> detail peculiar to the elisp implementation.

I think that my use of the word DOM has been very confusing here. I definitely
agree that we should not standardize the org-element API, nor the particular way
syntax nodes are represented in elisp. However, what I do think we should
standardize is the abstract tree representation of an org document. For
example, elements vs objects, the idea of nested headlines, etc. would be
specified in the DOM, separate from how to write them. For clarification, this
is what I mean should be specified for HTML. (Though org would necessarily be a
bit more complicated)

    A document is a node. Nodes are either text nodes, which contain only text,
    or they are normal nodes (what's the real name for them?). If they are
    normal nodes, they have a tag, which is text, and a number of attributes
    that have a text key and optionally a text value. Each normal node contains
    an ordered list of children nodes.

Also another note is that the worg syntax document does begin to specify
this. My point is to bring this out into a separate document.

> To the extent that an element tree could be useful, I think it would be as a
> concept in an implementation guide, not as something formally specified.

You may be right about this. Perhaps a formal standard is unnecessary for this.

> > - Org Standard Environments
> Read https://orgmode.org/worg/dev/org-syntax.html. It will get you up to speed with
> the existing terminology that is used in the community.
> ...
> > Org Standard Environments:
> >This is how I would specify elements such as #+begin_src..#+end_src would be
> > specified, as standardized elements of the environment. This would be
> > structured as a number of individual standard environments, such as
> > "Source Blocks" or "Standard Header Properties" (specifying #+title, #+author, etc.)
> These are well specified already in the
> worg syntax draft. There are a couple of special cases such as src and example
> blocks that could be included explicitly in the syntax to facilitate
> interoperability with parsers for org babel languages. Beyond that, the
> community already has vocabulary that covers what you describe here, as
> mentioned above.

I think I was unclear. I am discussing /how/ they are specified, not their
specification, which is, as you say, currently specified in the worg
document. Perhaps the best way to illustrate my idea is with an example.

Worg said:
> Affiliated Keywords
>
> With the exception of comment, clocks, headlines, inlinetasks, items, node
> properties, planning, property drawers, sections, and table rows, every other
> element type can be assigned attributes.
>
> This is done by adding specific keywords, named “affiliated keywords”, just
> above the element considered, no blank line allowed.
>
> Affiliated keywords
are built upon one of the following patterns:
>
> #+KEY: VALUE
> #+KEY[OPTIONAL]: VALUE
> #+ATTR_BACKEND: VALUE
>
> KEY is either “CAPTION”, “HEADER”, “NAME”, “PLOT” or “RESULTS” string.
>
> BACKEND is a string constituted of alpha-numeric characters, hyphens or
> underscores.
>
> OPTIONAL and VALUE can contain any character but a new line. Only “CAPTION”
> and “RESULTS” keywords can have an optional value.
>
> An affiliated keyword can appear more than once if KEY is either “CAPTION” or
> “HEADER” or if its pattern is “#+ATTR_BACKEND: VALUE”.
>
> “CAPTION”, “AUTHOR”, “DATE” and “TITLE” keywords can contain objects in their
> value and their optional value, if applicable.

The way I envision this standardized is the following:
> Affiliated Keywords
>
> With the exception of comment, clocks, headlines, inlinetasks, items, node
> properties, planning, property drawers, sections, and table rows, every other
> element type can be assigned attributes.
>
> This is done by adding specific keywords, named “affiliated keywords”, just
> above the element considered, no blank line allowed.
>
> Affiliated keywords are built upon one of the following patterns:
>
> #+KEY: VALUE
> #+KEY[OPTIONAL]: VALUE
>
> OPTIONAL and VALUE can contain any character but a new line.
>
> An environment specifies a number of legal KEYs, and for each one must
> specify the following:
> - the structure of VALUE and OPTIONAL
> - whether OPTIONAL is permitted
> - whether the keyword can be repeated multiple times on a single element
>
> ...
>
> Org Standard Environment #42: Backend Attributes
>
> Affiliated keywords where key begins with =ATTR_=, followed by a string
> BACKEND, which must consist of alphanumeric characters, hyphens, or
> underscores, are defined. OPTIONAL is not permitted. Multiple occurrences of
> the keyword are permitted. The structure of VALUE is determined by the export
> backend specified by BACKEND.
>
> These should be used to give additional information to an export backend
> identified by BACKEND.
>
> Org Standard Environment #314: Captioning
>
> The affiliated keywords "CAPTION," "AUTHOR," "DATE," and "TITLE," are defined.
> OPTIONAL is permitted in CAPTION. CAPTION may appear multiple times on a
> single element.
> OPTIONAL is not permitted in AUTHOR, DATE, or TITLE. These may not appear
> multiple times on a single element.
>
> For CAPTION, AUTHOR, DATE, and TITLE, objects may appear in VALUE and
> OPTIONAL (if applicable).

I hope that this example explains what I mean better.

Dr. Arne Babenhauserheide said:
> I would like to add, that this is pretty easy to do, and also to make
> independent of the users emacs environment. Here is an example that
> uses the whole orgmode-babel-latex-html machinery to create derived
> documents from source-of-truth org-mode files which get exported to a
> book:

Yes. Emacs can definitely be used in this way. However, I do not believe that it
should be the only tool that can be used in this way, even if no other tool
exists as present.



I hope I have clarified some of the confusions surrounding my argument.

Thanks, Asa

Re: Thoughts on the standardization of Org

[Russell Adams]

On Sat, Oct 31, 2020 at 08:22:01PM -0400, Asa Zeren wrote:
> First, I would like to repeat the importance of developing standards
> for org-mode. If we want to expand the influence of org, tooling must
> expand beyond Emacs.

I disagree. There are other open text based formats outside of
Emacs. That Org is so compelling is because it's tightly integrated
into an Emacs mode which makes using Org data so easy.

Like many open source projects no one is paid to work on and develop
Org. Despite the best volunteer efforts, Org has burnt out many
volunteer maintainers along the way. To ask for more of Org outside of
Emacs when the main Org project inside Emacs is understaffed seems a
backward notion to me.

> While Emacs is an amazing tool, (a) we cannot convince the entire
> world to use Emacs

We also cannot expect our maintainers to spend their time on the world
outside of Emacs.

> (b) org-mode should be integrated into tooling unrelated to text
> editing, and is outside of the Emacs-Lisp environment.

Isn't that the opposite goal? Org mode is all about your data being
text only and easy to edit inside Emacs, so why would "unrelated to
text editing and outside Emacs" be of any interest or worth any time?

> Without additional org implementations, this is impossible. If org
> catches on before it is standardized, we end up in the situation of
> Markdown, with many competing standards and non-standards. Hence,
> standardization is essential.

Org has caught on. Org has a reference implementation, which you could
consider the standard parser. Org isn't just the format, it's the
editing experience. That's only realized inside of Emacs.

Could it benefit from better documentation, and finishing up some
rough edges? Of course. That would be an excellent place to spend
volunteer time.

I can't recommend asking for volunteer time on tools outside of Emacs
though.

I've recommended a few times now the idea that if you want to work on
Org data outside of Emacs, use Org to do an export your data into a
machine readable format (xml, json, etc). Don't parse it directly from
text. I believe there are already exporters which can help with that,
and they may be worthy of additional effort.

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Thoughts on the standardization of Org

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 410 bytes --]


Asa Zeren <asaizeren@gmail.com> writes:
> Also another note is that the worg syntax document does begin to specify
> this. My point is to bring this out into a separate document.

Why should this be in a separate document? The obvious place for a
standard is worg, and the way forward is to improve what’s there.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Thoughts on the standardization of Org

[TEC]

Dr. Arne Babenhauserheide <arne_bab@web.de> writes:

> Asa Zeren <asaizeren@gmail.com> writes:
>> Also another note is that the worg syntax document does begin 
>> to specify
>> this. My point is to bring this out into a separate document.
>
> Why should this be in a separate document? The obvious place for 
> a
> standard is worg, and the way forward is to improve what’s 
> there.

I'll try to avoid interpreting others words for them :P but my 
take is
that it while documents like the worg syntax document are 
fantastic, we
may want to add a bit more into a single more comprehensive 
document.

I have a suggestion in this regard. #+include

Why not have the best of both worlds :P

--
Timothy

Re: Thoughts on the standardization of Org

[Jack Kamm]

Hi Timothy,

TEC <tecosaur@gmail.com> writes:

> I feel that this also ties into my earlier idea of putting Emacs
> as/inside an LSP server for Org.  I suspect there may be a a lot 
> of
> potential in making it dead easy to use Emacs as a tool.

I'm too busy to help on this, but I think it's a very good idea and hope
you can pursue it.

In terms of making org-mode more accessible outside emacs, there are 2
main benefits I'd personally be excited about:

1. Being able to collaborate on org-babel computational notebooks with
non-emacs users.
2. Improved mobile apps for org-mode.

Jack

Re: Thoughts on the standardization of Org

[Asa Zeren]

Hi all,

Thank you for your comments on my post "Thoughts on the Standardization of Org."
I appreciate all the feedback you have given me, I feel that, based off of the
responses, there have been a number of miscommunications as to my intention.

First, I did not mean the post to be primarily an argument for whether/when org
should be standardized, but rather a discussion on how a standard should be
structured. I realize now that including my position on whether org should be
standardized in the preamble was a mistake. Also, I want to note that I was not
intending to discuss by whom the standard should be governed. (Though I do
believe it should be by the org community, not an external standards body.)

Second, there is the matter of principle and practice. I am not arguing for the
org community to direct volunteer effort to a second editing environment, as
some are concerned. I am also personally not planning on creating one. However,
I want to make sure that standardization effort does not prevent another
first-class editing environment from being created, should there at some point
in the future emerge a group of people motivated to do that. In summary, I think
that it is important to think about the Emacs implementation as one of many /in
principle/, even though it is the sole implementation /in practice/, and may
remain so.

I hope we can have a productive discussion on how an org standard should be
structured, separate from, though perhaps in addition to, the discussion of
whether org should be standardized.

Thanks,
Asa

Re: Thoughts on the standardization of Org

[Asa Zeren]

Dr. Arne Babenhauserheide <arne_bab@web.de> wrote:

> Why should this be in a separate document? The obvious place for a
> standard is worg, and the way forward is to improve what’s there.

It does not necessarily need to be in a separate file, and if it is it should be
in worg or another communally owned place, and could just be another top level
heading. What I really mean, however, is that the three parts I outline should
be logically separate entities.

Re: Thoughts on the standardization of Org

[Daniele Nicolodi]

On 01/11/2020 17:13, Russell Adams wrote:
> On Sat, Oct 31, 2020 at 08:22:01PM -0400, Asa Zeren wrote:
>> First, I would like to repeat the importance of developing standards
>> for org-mode. If we want to expand the influence of org, tooling must
>> expand beyond Emacs.
> 
> I disagree. There are other open text based formats outside of
> Emacs. That Org is so compelling is because it's tightly integrated
> into an Emacs mode which makes using Org data so easy.

I cannot agree more with this statement and a similar statement (that I
am too lazy to go search to provide correct authorship) that stated that
one of the advantages of Org is that, being implemented in Emacs, it has
infinite potential for customization, thus we would need to agree about
what org mode is before standardizing it: my Org is not your org, and,
thank to the features offered be the Emacs environment, I use different
flavors of Org in different buffers.

Reading the thread I have the impression that what wants to be
standardized is the syntax of Org, but that is not very different from
the syntax of other text based markup languages such at
reStructuredText, asciidoc, Markdown, or others. What makes Org stand
out, for some applications, is "org-mode" (as the implementation of Org
in Emacs) and the tools built on top of the syntax.

I don't think it is reasonable to expect much of org-mode being
implemented in another environment, because that quickly becomes a task
as complex as implementing Emacs. For example, Org has org-tables, and
the formula syntax allows for Emacs Calc expressions or Elisp functions
to be called: should an hypothetical implementation of Org allow the
same formulas to be executed? Wouldn't that mean that this
implementation needs to re-implement a good fraction of Emacs (or use
Emacs itself for interpreting the formulas?

Maybe the standardization should cover only the "static" parts of Org
(ie no table formulas, no babel, no agenda, no exporters, etc). However,
in this case, what is left is little more of a markup language with an
editor that allows sections folding. You can have this on top of pretty
much any markup language using Emacs' outline-minor-mode.

If this is what you are after, I think a much more interesting goal
(from the point of view of Emacs users and Org developers) would be to
fold back some of the improvements org-mode builds on top of
outline-mode back into outline-mode itself. This would immediately be
beneficial as it would probably make Org simpler and the improvements
could be actually be used by many.

I also think that, if you are in the position to choose a format to use
for collaborative projects, reStructuredText may be a better choice as
the syntax is simple, well defined, and extensible. Although, without
trying I don't know if outline-minor-mode can be made to work with
reST-style sections headers. But, if it does not, it should not be hard
to adjust it (see previous paragraph).

Cheers,
Dan

Re: Thoughts on the standardization of Org

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]


Daniele Nicolodi <daniele@grinta.net> writes:
> Maybe the standardization should cover only the "static" parts of Org
> (ie no table formulas, no babel, no agenda, no exporters, etc). However,
> in this case, what is left is little more of a markup language with an
> editor that allows sections folding. You can have this on top of pretty
> much any markup language using Emacs' outline-minor-mode.

It could become stronger competition for asciidoc by being available in
more places.

Having an acceptance criterion for “supports basic org-mode
presentation” and “can edit org-files without breaking editing in
org-mode” could help adoption.

That would be the only part I’d really expect from standardization:
There would be a clear-cut point when a tool could claim compatibility
with org level N or by components (i.e. basic presentation, code-blocks,
…).

Having org-files parsed as html on a VCS-infopage is pretty nice.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Thoughts on the standardization of Org

[Ken Mankoff]

To all who argue that Org is too tightly coupled to Emacs to consider working with it outside of Emacs, I point to GitHub. The fact that GitHub natively renders Org files "well enough" is a huge benefit to those of us who use Org. It is also useful for gaining new users (assuming more users is a good thing).

Therefore, if other tools have the ability to do *something* with an Org file (display most of it well enough, allow editing without breaking things, maybe implementing a simple Babel interpreter for a few popular languages, whatever), this would be A Good Thing.

For example, I'd contribute more to Stack Exchange sites if I could answer using Org syntax rather than Markdown syntax. None of these 3rd party implementations need to be comprehensive or perfect, just as the GitHub renderer is neither complete nor perfect. But we all benefit from the GitHub rendering.

  -k.

Re: Thoughts on the standardization of Org

[Russell Adams]

On Sun, Nov 01, 2020 at 05:17:19PM -0800, Ken Mankoff wrote:
>
> To all who argue that Org is too tightly coupled to Emacs to
> consider working with it outside of Emacs, I point to GitHub. The
> fact that GitHub natively renders Org files "well enough" is a huge
> benefit to those of us who use Org. It is also useful for gaining
> new users (assuming more users is a good thing).

I don't mind that at all. Consider though how few of Org's abilities
you can use in that context. Headlines, basic text formatting, etc is
fine. It's easy to interpret, and well standardized by now.

The issue with implementing additional parsers outside of Emacs is the
overhead of exciting high level features like Babel, Exports,
spreadsheet, agenda, etc. Those would be very difficult to
reimplement.

Using Org as a replacement for basic Markdown on sites looks like it's
already been done successfully.


------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Thoughts on the standardization of Org

[Daniele Nicolodi]

On 02/11/2020 00:10, Dr. Arne Babenhauserheide wrote:
> 
> Daniele Nicolodi <daniele@grinta.net> writes:
>> Maybe the standardization should cover only the "static" parts of Org
>> (ie no table formulas, no babel, no agenda, no exporters, etc). However,
>> in this case, what is left is little more of a markup language with an
>> editor that allows sections folding. You can have this on top of pretty
>> much any markup language using Emacs' outline-minor-mode.
> 
> It could become stronger competition for asciidoc by being available in
> more places.

Why does Org need to compete with asciidoc? I don't see any advantage in
fighting with anyone for market share.

> Having an acceptance criterion for “supports basic org-mode
> presentation” and “can edit org-files without breaking editing in
> org-mode” could help adoption.

Acceptance criterion for what? Adoption of what?

It seems to me that some see a the adoption of a simplified version of
the Org markup language outside Emacs and the org-mode implementation as
something desirable. However, I don't see what the Org community would
gain from that.

> That would be the only part I’d really expect from standardization:
> There would be a clear-cut point when a tool could claim compatibility
> with org level N or by components (i.e. basic presentation, code-blocks,
> …).
> 
> Having org-files parsed as html on a VCS-infopage is pretty nice.

As explained many times now, you don't a formal specification for this:
the specification is the org-mode implementation itself.

However, I will not discourage anyone from working on some form of
standardization, other than pointing out that IMO it is an exercise with
very limited usefulness, impact and future.

Cheers,
Dan

Re: Thoughts on the standardization of Org

[TEC]

[-- Attachment #1: Type: text/plain, Size: 1015 bytes --]


Daniele Nicolodi <daniele@grinta.net> writes:

> Acceptance criterion for what? Adoption of what?
>
> It seems to me that some see a the adoption of a simplified 
> version of
> the Org markup language outside Emacs and the org-mode 
> implementation as
> something desirable. However, I don't see what the Org community 
> would
> gain from that.
> 
> [...]
> 
> As explained many times now, you don't a formal specification 
> for this:
> the specification is the org-mode implementation itself.
>
> However, I will not discourage anyone from working on some form 
> of
> standardization, other than pointing out that IMO it is an 
> exercise with
> very limited usefulness, impact and future.

I think there are absolutely some benefits for Org users. I am
personally interested in registering Org as an IANA MIME type.

What will this do? Well, for starters I'd like to be able to 
attach org
files without the type being recognised as
"application/vnd.lotus-organizer" 🤢.


[-- Attachment #2: test.org --]
[-- Type: application/vnd.lotus-organizer, Size: 58 bytes --]

[-- Attachment #3: Type: text/plain, Size: 582 bytes --]


I also think it's to our benefit that non-Emacsers become more
comfortable with seeing an org file --- as I see it that improves 
our
chances that we can directly share Org files with them, which they 
might
be comfortable editing and sending back for example, or that a 
generic
tool might think to support Org files.

So I'd like to assure you that my interest in improving 
recognition and
support for Org is motivated by selfish reasons 😛 which just so 
happen
to potentially benefit non-Emacsers.

I hope that clarifies my view of the proposal,

Timothy

Re: Thoughts on the standardization of Org

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2931 bytes --]


Daniele Nicolodi <daniele@grinta.net> writes:

> On 02/11/2020 00:10, Dr. Arne Babenhauserheide wrote:
>> 
>> Daniele Nicolodi <daniele@grinta.net> writes:
>>> Maybe the standardization should cover only the "static" parts of Org
>>> (ie no table formulas, no babel, no agenda, no exporters, etc). However,
>>> in this case, what is left is little more of a markup language with an
>>> editor that allows sections folding. You can have this on top of pretty
>>> much any markup language using Emacs' outline-minor-mode.
>> 
>> It could become stronger competition for asciidoc by being available in
>> more places.
>
> Why does Org need to compete with asciidoc? I don't see any advantage in
> fighting with anyone for market share.

That’s something really personal: I would prefer to use org at work :-)

>> Having an acceptance criterion for “supports basic org-mode
>> presentation” and “can edit org-files without breaking editing in
>> org-mode” could help adoption.
>
> Acceptance criterion for what? Adoption of what?

Acceptance criterion for „this is good enough to show an org-mode
document at gitlab/sr.ht/…“.

A minimum requirement so people who write a new library have something
to target.

Adoption: For example for editing in the browser. It would be nice to
have basic editing support in gitlab — with sufficient support for users
to not break org.

There are many places where people cannot actually run full Emacs,
because Emacs can do far too much.

If you have an online editor and anyone can file a pull-request, you
really won’t want to have support for running arbitrary code. But in the
actual export you might want it (after checking what you’ll run).

#+begin_src bash :exports results
echo rm -rf /
echo 0day-exploit-against-docker
#+end_src

> It seems to me that some see a the adoption of a simplified version of
> the Org markup language outside Emacs and the org-mode implementation as
> something desirable. However, I don't see what the Org community would
> gain from that.

I am missing ways to allow people to do small changes. For example to
enable people to change text in my RPG in an online-editor (i.e. gitlab)
and file a pull-request without ever leaving the browser.

Building a full org-mode parser is a daunting task for all but those who
don’t know enough to do it well. So people who start will most likely
not know enough to get a basic parser right.

Defining a subset as basic-org-target would give people a target they
can actually aim for.

And then defer to full Emacs, if they want to do more.

> As explained many times now, you don't a formal specification for this:
> the specification is the org-mode implementation itself.

I won’t argue against that because I see it the same way :-)

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Thoughts on the standardization of Org

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1246 bytes --]


Russell Adams <RLAdams@AdamsInfoServ.Com> writes:

> On Sun, Nov 01, 2020 at 05:17:19PM -0800, Ken Mankoff wrote:
>>
>> To all who argue that Org is too tightly coupled to Emacs to
>> consider working with it outside of Emacs, I point to GitHub. The
>> fact that GitHub natively renders Org files "well enough" is a huge
>> benefit to those of us who use Org. It is also useful for gaining
>> new users (assuming more users is a good thing).
>
> I don't mind that at all. Consider though how few of Org's abilities
> you can use in that context. Headlines, basic text formatting, etc is
> fine. It's easy to interpret, and well standardized by now.
>
> The issue with implementing additional parsers outside of Emacs is the
> overhead of exciting high level features like Babel, Exports,
> spreadsheet, agenda, etc. Those would be very difficult to
> reimplement.

For these it would suffice for me if the parsers did not break the
features. For example if a table cannot be rendered as table, I want to
see it as plain text. Maybe the parser could hide @@latex:@@ blocks in
html presentation, but they must be shown when editing …

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Thoughts on the standardization of Org

[Daniele Nicolodi]

On 02/11/2020 10:02, TEC wrote:
> I think there are absolutely some benefits for Org users. I am
> personally interested in registering Org as an IANA MIME type.

I don't think that registering Org as IANA MIME type will have the
consequences you hope it has.

> What will this do? Well, for starters I'd like to be able to 
> attach org
> files without the type being recognised as
> "application/vnd.lotus-organizer" 🤢.

This is not governed by the presence of an MIME type for Org, but by the
mechanism that assigns a MIME type to the file when it is attached tp a
message, and this is independent of the IANA MIME type database.

You (most probably) already can configure your email client (or
operative system) to use "text/x-org" as the mime type for files with
extension ".org". Having a IANA MIME type, would only make using
"text/org" a bit more "right" (there is nothing that stops you from
using the "text/org" mime type without it being registered).

If you want the association to be automatic, you need to convince the
maintainers of the MIME type association databases to change their
defaults. And this is a rather long list of people you have to interact
with and convince that "text/org" is a better choice than
"application/vnd.lotus-organizer" as the default MIME type for ".org" files.

Finally, even if you get your attachment automatically tagged as
"text/org", the receiving side needs to have a mime type handler
configured to display it. As far as I know, not even Emacs (on the
platforms that allow it) registers itself as an handler for any MIME
type. Therefore, what you get, assuming that the mail client on the
other side behaves correctly and uses "text/*" as a fallback for
"text/org" is that your attachment will be displayed in a generic text
editor.

> I also think it's to our benefit that non-Emacsers become more
> comfortable with seeing an org file --- as I see it that improves 
> our
> chances that we can directly share Org files with them, which they 
> might
> be comfortable editing and sending back for example, or that a 
> generic
> tool might think to support Org files.

I send Org files as "text/plain" (often even using ".txt" extension to
avoid confusion on the receiving side) and I think this is the best
choice as it puts the least burden on the receiving side to consume the
content and it is displayed inline by most email clients.

> So I'd like to assure you that my interest in improving 
> recognition and
> support for Org is motivated by selfish reasons 😛 which just so 
> happen
> to potentially benefit non-Emacsers.

I don't think that registering "text/org" with the IANA will have the
consequences that you hope it has.

Cheers,
Dan

Re: Thoughts on the standardization of Org

[Eric S Fraga]

+1 for everything that both Pankaj and Tim have said.

I've said this elsewhere: for me, the power of org mode is that it is
Emacs.  Org allows me to leverage the power of Emacs more easily for
what I want to do (everything from project management to dissemination
of various sorts).

Anything that curtails what I can do in Emacs (with org mode) would have
significantly negative repercussions.

I understand the desire to see org mode's capabilities implemented
elsewhere but I think this misses the point: it's Emacs that people
actually want implemented elsewhere without realising it.

A small example: in teaching, I often use embedded Calc in my org
documents (primarily for export via beamer) for inline calculations
illustrating concepts.  This is *part* of my use of org mode and feels
like it's org mode but it's really Emacs that allows me to do this.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-61-ga88806.dirty

Re: Thoughts on the standardization of Org

[TEC]

Daniele Nicolodi <daniele@grinta.net> writes:

> On 02/11/2020 10:02, TEC wrote:
>> I think there are absolutely some benefits for Org users. I am
>> personally interested in registering Org as an IANA MIME type.
>
> I don't think that registering Org as IANA MIME type will have 
> the
> consequences you hope it has.

Hmmm. I'm glad you've brought this up. I wouldn't want to put in 
hours
of effort for a unlikely benefit.
My hope is that registering org as an IANA MIME type will cause it 
to
trickle down into individual MIME registries.

The benefit I see here is Org being able to be treated more as a 
'first
class' plaintext format --- like I see markdown being treated 
today.
Open a ".md" file in Ocular and you see a nice rendered version (I 
use
this as a general, not specific example).

Then, if we can package Emacs into an LSP client, we can provide a 
very
well-featured Org experience to dozens of editors, and the 
"text/*"
association will be important to ensure that Org files are opened 
in
those text editors.

For people who use editors like VSCode, being able to send an Org 
file
and have it open in VSCode will likely prompt them to install an
extension that provides support for .org files...

This may be a bit unrealistic, and I'm hugely appreciative of any 
effort
to inform me of any aspects that I may be overlooking.

> Finally, even if you get your attachment automatically tagged as
> "text/org", the receiving side needs to have a mime type handler
> configured to display it. As far as I know, not even Emacs (on 
> the
> platforms that allow it) registers itself as an handler for any 
> MIME
> type. Therefore, what you get, assuming that the mail client on 
> the
> other side behaves correctly and uses "text/*" as a fallback for
> "text/org" is that your attachment will be displayed in a 
> generic text
> editor.

1. Could Emacs change to register itself as a text/org handler?
2. See above for why I don't think that opening it in a generic 
text
editor would necessarily be a bad idea.

> I send Org files as "text/plain" (often even using ".txt" 
> extension to
> avoid confusion on the receiving side) and I think this is the 
> best
> choice as it puts the least burden on the receiving side to 
> consume the
> content and it is displayed inline by most email clients.

This seems like a good idea, thanks for sharing it!

> I don't think that registering "text/org" with the IANA will 
> have the
> consequences that you hope it has.

Thanks for highlighting some potential pitfalls that I haden't
considered. As outlined above, in light of your comments I still 
see
this being potentially beneficial/worth the effort, but you've 
opened my
eyes to some complications that I was previously unaware of.

Thank you,

Timothy.

Re: Thoughts on the standardization of Org

[Greg Minshall]

Eric,

i was thinking of replying to your earlier post on the power of emacs.
now i guess i'll ask my question or make my vague point or whatever.

i wonder if it's possible (ignoring the possible utiltiy) to divide org
mode into two (maybe three?) things.

first is "org mode as a document structuring [hierarchy, tables, lists]
and markup [links, ??? -- maybe it's all part of "document structure"]
technique.

(versus, possible "third": agenda, task manager. [that i'm not all that
familiar with.])

versus org mode as a computing environment (== "org mode as life", which
is to a large extent true for me, as well).

the border between these divisions is, of necessity (i suspect) fairly
fluid.

but, i wonder if one could draw a useful boundary at "babel execute".
i.e., on one side, one is definitely in "computing environment".

(and, in my notes on your earlier e-mail: "what about calc, used in
table formulae?".  which you also mention.)

i'm neutral on the issue of standardization (though i agree with Daniele
Nicolodi that standardization might not bring what everyone wants).  i
will note that for many years the C language was not a formal
"standard", but people figured out how to write "portable" code that
worked with the major compilers and runtime environments.  (it's nicer
today that, e.g., C is an international standard, but that wasn't the
first step towards "interoperability".)

my bias is i'd love to see "everybody" able to export *and* tangle an
org document, possibly within a limited subset, such as those that don't
require babel-execute for that purpose.

cheers, Greg

Re: Thoughts on the standardization of Org

[Eric S Fraga]

On Monday,  2 Nov 2020 at 17:22, Greg Minshall wrote:
> i wonder if it's possible (ignoring the possible utiltiy) to divide org
> mode into two (maybe three?) things.

Everything is possible!  Whether it's desirable or not is a different
question. :-)

Although at first glance, it would seem straightforward to separate
syntax from use (what you call structure versus environment), the two
are so intimately tied together that fixing the syntax could make the
latter harder to evolve.

For instance, in my recent org documents, I have added a #+calc: keyword
which I use for embedded calc lines.  This allows me to have a clearly
labelled line that Calc will recognise and that I can process using a
filter before export while also ensuring that other tools, e.g. ones
which will ignore lines starting with #, do not fail.  If the standard
did not allow for arbitrary keywords, would this limit my use?

A more subtle issue, and one that I raised earlier, is the underlying
infinite customization provided by Emacs.  Some of my macros are elisp
code.  A standard for the structure of org mode documents could exist
but using such standard-compliant documents would be shackled by not
having elisp available to process the macros.  They would really only be
usable within Emacs and hence my suggestion that what people really
want, without knowing it, is Emacs everywhere. ;-) [1]

(as an aside, Emacs as an LSP could be interesting, especially if
network based)


Footnotes:
[1] for my sins, I've recently had to use various tools for writing
     (text & data) including Word, Excel and Teams chats: how *do*
     people work efficiently in some of these environments?  No
     abbreviations, no registers, no embedded Calc, no line highlighting
     (I keep losing the cursor), ...

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-61-ga88806.dirty

Re: Thoughts on the standardization of Org

[Russell Adams]

On Mon, Nov 02, 2020 at 02:56:58PM +0000, Eric S Fraga wrote:
> (as an aside, Emacs as an LSP could be interesting, especially if
> network based)

#+BEGIN_RANT

LSP is a standard from Microsoft:
https://github.com/Microsoft/language-server-protocol/

It allows networked JSON and telemetry, as well as all the other
problems that come from trusting the remote server:
https://microsoft.github.io/language-server-protocol/specifications/specification-current/#telemetry_event

Either of these items would provide justification for me to exclude
LSP from any personal consideration, and for me to strongly recommend
against it's use in any capacity.

Microsoft and related technologies have no place in my Emacs. I don't
care to see Org made easy or functional in Visual Studio.

Why contribute to layers which allow an illegal monopoly more
market share? Why code for them for free?

#+END_RANT

On a milder tone, I think it's silly to send JSON packets every time I
type a key, even to a local service.

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Thoughts on the standardization of Org

[TEC]

Russell Adams <RLAdams@AdamsInfoServ.Com> writes:

> On Mon, Nov 02, 2020 at 02:56:58PM +0000, Eric S Fraga wrote:
>> (as an aside, Emacs as an LSP could be interesting, especially 
>> if
>> network based)
>
> LSP is a standard from Microsoft:
> https://github.com/Microsoft/language-server-protocol/
>
> It allows networked JSON and telemetry, as well as all the other
> problems that come from trusting the remote server:
> https://microsoft.github.io/language-server-protocol/specifications/specification-current/#telemetry_event
>
> Either of these items would provide justification for me to 
> exclude
> LSP from any personal consideration, and for me to strongly 
> recommend
> against it's use in any capacity.
>
> Microsoft and related technologies have no place in my Emacs. I 
> don't
> care to see Org made easy or functional in Visual Studio.
>
> Why contribute to layers which allow an illegal monopoly more
> market share? Why code for them for free?

Because it's supported by: Atom, Brackets, Delphi, Eclipse, Emacs, 
Kakoune, Kate,
VSCode, NeoVim, Sublime Text 3, JupyterLab, and more 🙃.

--
Timothy

Re: Thoughts on the standardization of Org

[Eric S Fraga]

On Monday,  2 Nov 2020 at 16:23, Russell Adams wrote:
> #+BEGIN_RANT
> [...]
> #+END_RANT

Apologies for my comment then!  :-( I am fully sympathetic to the views
you have expressed.

Let me rephrase, therefore: it could be interesting to see Emacs as a
SaaS which processes org mode documents.  But, note, I only say it might
be interesting.  I am not particularly keen on cloud services for all
kinds of reasons including ownership, security, etc.  

My solution is to have Emacs with me at all times so I have org & gnus &
erc & emms & ... :-) I have a Planet Computers Gemini and an
OpenPandora, for this reason, and am awaiting the soon to be available
Pyra with impatience to have a fully open system (open software in the
form of Debian & Emacs and open hardware, as much as is possible at the
moment).

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-61-ga88806.dirty

Re: Thoughts on the standardization of Org

[Carsten Dominik]

[-- Attachment #1: Type: text/plain, Size: 2761 bytes --]

Dear all,

this is an interesting discussion to read, and I think lots of  clever
people have made this an interesting discussion. So I hesitated to even
join the discussion, because I am quite removed from current development
and no longer feel qualified to guide it.  Still, my 5c.

For me, it seems unrealistic to standardize Org in a way that make it
desirable or even feasible to have *full implementations* in other tools.
What makes sense it to have tools that can

- *display* org files in a pleasant and useful way
- *convert* org files into other formats, with some accepted loss of
functionality
- *write* org files that then will function properly in Emacs.

Orgzly is a fantastic example.  It reads and displays Org files,
understands enough syntax to provide very useful functionality, and is
decent enough to not change stuff that is does not understand and use, so
that the files it writes are again fully functional in Emacs.

It seems to me that this covers most of what we can hope for, as a basic
formula.  No definition of Org syntax can fully know what I have done in my
personal environment, and therefore will not be able to reproduce that
functionality elsewhere.  This is intrinsic in Org and Emacs, I think.

The efforts to clean up the markup syntax have been fantastic (thank you,
in particular, Nicolas), and they have made it possible to have meaningful
parsers like the one on github.  And they provide a certain guarantee that
the three items I list above will work, also going forward.

Now, do I want that an arbitrary web browser or email client understands if
a file is org syntax, and that clicking on it should open Emacs.  Yes, I
would like that.  So in that sense, a mime type would be useful, for sure.

Greetings

Carsten

On Mon, Nov 2, 2020 at 4:50 PM Eric S Fraga <e.fraga@ucl.ac.uk> wrote:

> On Monday,  2 Nov 2020 at 16:23, Russell Adams wrote:
> > #+BEGIN_RANT
> > [...]
> > #+END_RANT
>
> Apologies for my comment then!  :-( I am fully sympathetic to the views
> you have expressed.
>
> Let me rephrase, therefore: it could be interesting to see Emacs as a
> SaaS which processes org mode documents.  But, note, I only say it might
> be interesting.  I am not particularly keen on cloud services for all
> kinds of reasons including ownership, security, etc.
>
> My solution is to have Emacs with me at all times so I have org & gnus &
> erc & emms & ... :-) I have a Planet Computers Gemini and an
> OpenPandora, for this reason, and am awaiting the soon to be available
> Pyra with impatience to have a fully open system (open software in the
> form of Debian & Emacs and open hardware, as much as is possible at the
> moment).
>
> --
> : Eric S Fraga via Emacs 28.0.50, Org release_9.4-61-ga88806.dirty
>
>

[-- Attachment #2: Type: text/html, Size: 3425 bytes --]

Re: Thoughts on the standardization of Org

[Tim Cross]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

>
> A more subtle issue, and one that I raised earlier, is the underlying
> infinite customization provided by Emacs.  Some of my macros are elisp
> code.  A standard for the structure of org mode documents could exist
> but using such standard-compliant documents would be shackled by not
> having elisp available to process the macros.  They would really only be
> usable within Emacs and hence my suggestion that what people really
> want, without knowing it, is Emacs everywhere. ;-) [1]
>

I think the above is perhaps the most critical point. Much of the talk
on standardisation seems to be focusing on org's markup layer. This is
only a small part of org. Many of my org documents include macros,
results from code block evaluation and rely on elisp code execution when
certain content changes (such as changing a todo status). Without any of
this, the document is not an accurate representation compared to how it
is when viewed/used within Emacs. Even the goal of collaborative editing
won't work because changing the data outside Emacs won't trigger the
macros, functions, code blocks etc to update dependent parts (e.g.
changing the TODO status won't result in the timestamp I record in the
draw from being updated or updating that value in a table wo't result in
re-calculation on formulas etc).

One suggested benefit for standardisation was in being able to add a
MIME type. Like others have mentioned, I'm unclear on how exactly adding
a mime type really helps anyone. Having a MIME type only gives value if
you can also have a MIME handler. Nobody except Emacs users have a MIME
handler for *.org files and if you are an Emacs user, then Emacs will
handle that type based on the built-in type handlers. Even if you had a
very basic org parser, about all you might get is some syntax
highlighting and maybe some folding support. You are unlikely to have
much editing support (for example, you won't know from just the file
what the defined keywords are, only the keywords used, same with tags,
priorities, etc).

People have mentioned that github and other systems support org mode
files. I think this is a little misleading - they support a subset of
the org-mode markup and can add minimal text highlighting based on that
markup. These implementations all seem only partially complete and tend
to only handle highlighting for more common languages. While it is great
they have this support, to view this as proof of a the viability of an
external non-Emacs org-mode is perhaps a little optimistic.

The term 'standardisation' might have been a little misleading for this
discussion. Much of what people seem to be talking about could be
satisfied with a completed syntax reference like the one on worg. This
would be sufficient to allow development of systems that are able to
parse an org file and render it in specific formats (like github does).
Perhaps focusing on how to make this document as clear and complete as
possible would be a better effort than trying to define some formal
specification of the whole org-mode system.

Re: Thoughts on the standardization of Org

[Greg Minshall]

Eric,

> For instance, in my recent org documents, I have added a #+calc: keyword
> which I use for embedded calc lines.  This allows me to have a clearly
> labelled line that Calc will recognise and that I can process using a
> filter before export while also ensuring that other tools, e.g. ones
> which will ignore lines starting with #, do not fail.  If the standard
> did not allow for arbitrary keywords, would this limit my use?

perhaps the standard for e-mail headers (originally, RFC822) might be a
useful way of thinking about this issue.  it standardizes what it
standardizes, and then says, "and, by the way, you can put in almost
anything else [X-Mailer, ...], but you can't count on any other node
understanding it".  over time, new things are standardized (and, so,
moved to, e.g., Mailer, and other things aren't.

it seems to me this has worked fairly well, and partly this works
because of the late Jon Postel's admonition for designing internet
protocols: be conservative in what you send, and liberal in what you
accept. [1]

cheers, Greg


[1] i.e., be extremely picky about sticking exactly to the letter of the
standard in generating, e.g., documents, but allow for some sloppiness
in the format of incoming documents.

Re: Thoughts on the standardization of Org

[David Rogers]

Asa Zeren <asaizeren@gmail.com> writes:

> Hi,
>
> Even though I am new to the org-mode community, I would like to 
> share
> some thoughts on the specification of org-mode, especially since 
> I
> have seen some recent discussion of it in relation to 
> registering org
> as a MIME type.
>
> First, I would like to repeat the importance of developing 
> standards
> for org-mode. If we want to expand the influence of org, tooling 
> must
> expand beyond Emacs. While Emacs is an amazing tool, ...

I disagree (in principle, not just because it would be difficult) 
with the idea of “expanding beyond Emacs”. Org-mode benefits 
greatly from current and future Emacs development, and asking to 
standardize “just the parts that are not Emacs” would cause 
Org-mode to lose that huge advantage. Org-mode relies heavily on 
the editor it’s built on, and if it ceased to rely on Emacs, it 
would be forced to rely on “nothing at all” instead. Not only 
that, but for Org-mode users being able to count on all of Emacs 
is a big part of why it works. This means separating Org-mode from 
Emacs is a “lose-lose” idea.

As far as the level of difficulty, any developer thinking “I want 
my application to be able to run Org-mode” really just has one 
item on their to-do list: Duplicate every detail of Emacs, 
including all the obscure capabilities and all the quirks, 
especially that everything user-modifiable is in Emacs Lisp, and 
maintain all future updates in lockstep with whatever the future 
Emacs developers decide. Sounds like fun. :)

If you admire a certain car maker’s engines, you can probably get 
one somehow to install in a unique car you’re building. But this 
situation isn’t really like that. This idea is more like wanting 
to buy just the speed indicator from a fast car and install that, 
expecting it to improve your car’s acceleration.

-- 
David Rogers

Re: Thoughts on the standardization of Org

[Ken Mankoff]

On 2020-11-03 at 00:24 -08, David Rogers <davidandrewrogers@gmail.com> wrote...
> I disagree (in principle, not just because it would be difficult) with
> the idea of “expanding beyond Emacs”. Org-mode benefits greatly from
> current and future Emacs development, and asking to standardize “just
> the parts that are not Emacs” would cause Org-mode to lose that huge
> advantage. Org-mode relies heavily on the editor it’s built on, and if
> it ceased to rely on Emacs, it would be forced to rely on “nothing at
> all” instead. Not only that, but for Org-mode users being able to
> count on all of Emacs is a big part of why it works. This means
> separating Org-mode from Emacs is a “lose-lose” idea.

It seems like you have never used Orgzly or read on Org file on GitHub. Those are not ideas, but are actual current real-world win-win implementations of parts of Org outside of Emacs.

More of these would be better.

Everyone on this thread who says you can't separate Org from Emacs is correct that it is unreasonable to expect a 100 % bit-compatible and keystroke-compatible experience outside of Emacs. I don't think that level of re-implementation was what the OP was suggesting.

Again: GitHub. Orgzly. The conversation should move from "it can't be done" or "it isn't helpful" (why so much negativity on this thread?) to

+ What parts can be standardized and re-implemented outside of Emacs.
+ How do we define graceful failure for the other parts.
+ How do we support 3rd-party implementation in a way that benefits all of us.

  -k.

Re: Thoughts on the standardization of Org

[Russell Adams]

On Tue, Nov 03, 2020 at 04:14:41AM -0800, Ken Mankoff wrote:
> It seems like you have never used Orgzly or read on Org file on
> GitHub. Those are not ideas, but are actual current real-world
> win-win implementations of parts of Org outside of Emacs.

Supporting a subset of trivial formatting options from Org should be
that, trivial. Those items are not changing and fairly standardized
already. Could an existing Markdown importer be trivially altered to
support basic Org formats? I suspect that's what Github did.

If that's the case, why are we even arguing? What is missing that so
desperately needs standardization?

> Again: GitHub. Orgzly. The conversation should move from "it can't
> be done" or "it isn't helpful" (why so much negativity on this
> thread?) to

Not negative for negative sake or dismissing new ideas. However
understand there is a lack of resources working on Org as it stands
today. Org has grown exponentially in use cases for years, and there's
no way to expect our devs to keep up with that, much less tooling for
users outside of Emacs. I'm barely involved in the periphery, but I've
watched since 2006. It's a thankless job maintaining any popular OSS
project. I'm grateful for what has been done already, I have no
standing to ask for more or demand time spent on outside software.

We are already having to argue over what is core, and what is an addon
because every advanced user has extensions they want to make. Org
ballooned dramatically and is having to be trimmed.

> + What parts can be standardized and re-implemented outside of
> Emacs.

I'm negative on this because it's outside of Emacs. I like having
clear documentation and a good reference for the benefit of Emacs
users.

Why ask for developer time for external environments? Why contribute
to centralized non-free services who exploit our integration for more
marketshare?

> + How do we support 3rd-party implementation in a way that benefits
> all of us.

I'd rather invest our time on improving our internal implementation,
documentation, and support.

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Thoughts on the standardization of Org

[Eric S Fraga]

On Tuesday,  3 Nov 2020 at 04:14, Ken Mankoff wrote:
> Again: GitHub. Orgzly. The conversation should move from "it can't be
> done" or "it isn't helpful" (why so much negativity on this thread?)
> to

Hi Ken,

I'm sorry if I came across as being negative.  I am not.  I just think
that there is a clear difference between what some people think org mode
is and what others think.  I think it's primarily Emacs plus some
specific markup, including org babel and some understanding of LaTeX
constructs, whereas others maybe think it's markup that happens to be
interpreted by Emacs.  The two are very different animals.

The benefits of org mode for me are that it is Emacs.  I could continue
to work (maybe not quite as efficiently) without org mode but I would
really suffer without Emacs.  I've been using Emacs since 1984 and org
mode since 2008.  And gnus for what feels like forever. ;-) So maybe I'm
old skool and fossilised in my views. :-(  Apologies if so.

I have no problem with a clear syntax definition for org mode documents
and Nikolas has gone a long ways in providing that.  Beyond that, I find
it difficult to see any further standardization that would provide any
real benefits *to me*.  If others see those benefits, excellent!  All
power to them and I hope there is success in greater use of org
documents outside Emacs which would possibly trickle back benefits into
org mode in Emacs.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-61-ga88806.dirty

Re: Thoughts on the standardization of Org

[Ken Mankoff]

Hi Eric,

On 2020-11-03 at 05:00 -08, Eric S Fraga <e.fraga@ucl.ac.uk> wrote...
> The benefits of org mode for me are that it is Emacs. [...] I find it
> difficult to see any further standardization that would provide any
> real benefits *to me*. If others see those benefits, excellent! All
> power to them and I hope there is success in greater use of org
> documents outside Emacs which would possibly trickle back benefits
> into org mode in Emacs.

No need to apologize!

I'm similar to you re emacs usage and cannot imagine I'd leave it even if 3rd-party implementations existed. There is no way that 3rd party tool would also read my email, have magit, implement all of Org that I use, edit TeX, etc.

But I'm weary of seeing all my colleagues say "Jupyter" and not "Org" when it comes to sharing their code/work. I have a fantasy that in 5 years I tell someone who doesn't use Emacs but uses Atom or some other editor to "pip install org-lsp" and then double-click on this Org file I just emailed you, and then right-click on the code block and select "evaluate" from the popup menu. That sounds horrific to me as a user, except the part where I can interact more powerfully with more colleagues.

In this fantasy this adds no additional effort or stress to the current devs, because we're a big diverse community, and if TEC doesn't want to work on Org core code, but does want to make an emacs-lsp or org-lsp, more power to them.

  -k.

Re: Thoughts on the standardization of Org

[Devin Prater]

[-- Attachment #1: Type: text/plain, Size: 5198 bytes --]

I'm coming at this from the viewpoint of accessibility. I am a blind
person, who is pretty technical, but not technical enough to bend Emacs to
my will as easily as some of you do. But I have begun bending Org-mode to
fit what I use it for, and love heading folding and having all things
pertaining to work, for example, in one document, and being able to easily
find things from there and navigate using folding. I've since moved from
Mac, where I used Emacs and Emacspeak, to Windows, where most blind people
use VS Code. And I love how streamlined VS Code is. No linter is
inaccessible because of using parts of the interface that the TTS extension
author hasn't made accessible yet, because the whole UI works with the
screen reader. Sure, it's not perfect yet, but Language Tool, Grammarly,
all Markdown extensions, all work great with my screen reader in VS Code,
and I love using those tools. But no Markdown extension comes close to the
power and flexibility of Org-mode in Emacs.

Yes, this could be me just wishing Emacs worked better with accessibility
tools in Windows and we didn't have to rely on Emacspeak and such in other
operating systems, because ultimately Emacspeak pretty much relies on the
author or other contributors working around Emacs, to make Emacs' modes
work well in an auditory environment. And that works great, until you roam
outside of the use cases set forth by the developers. And not all
Emacs/Emacspeak users are developers, so cannot then make these use cases,
like Hugo-mode and Jekyll-mode, before I asked the maintainer of those
modes if there was anything they could do to make their modes work better
with Emacspeak, work better. And I'm not saying that we should dumb down
Emacs for people like me, but I do think that VS Code and other editors
that try to help out the writer or developer and such, have their place,
and so Org-mode should have a more prominent place within these editors.
Besides, I think bringing Org-mode to VS Code would help bring Org-mode to
the web, which would be pretty cool, since VS Code is pretty much built on
web technologies. And there is an Emacspeak package for Windows, but it is
no longer maintained, and I think most Emacspeak users use Linux or Mac
anyways. And one can sort of use Emacs in a Terminal with a screen reader,
but you can't use C-N, C-P, or other Emacs-specific keys, defeating one
purpose of even having Emacs in the first place. I might just get a virtual
machine up and going and just use that, just for Emacs and Org-mode. :)

And yes, there are things that Emacspeak can do that current screen readers
do not. Emacspeak shows syntax highlighting through speech effects and
parameter changes, and has many sounds for common actions, which helps a
lot. But the non-standardization of many Emacs packages, and the popularity
of VS Code among blind people who want to code, or learn to code, or are
more technically inclined like me, make it far easier for me to recommend
VS Code with a current screen reader than to recommend blind people switch
to Linux or Mac, and use Emacs with Emacspeak.

Of course, I'm only one blind person. There are blind people who have
mastered Emacs and use it for everything, but I just want things to work
for me nowadays, and find myself much more productive on Windows and VS
Code, but really miss Org-mode and its simple power that I could actually
grasp.
Devin Prater



On Tue, Nov 3, 2020 at 6:15 AM Ken Mankoff <mankoff@gmail.com> wrote:

>
> On 2020-11-03 at 00:24 -08, David Rogers <davidandrewrogers@gmail.com>
> wrote...
> > I disagree (in principle, not just because it would be difficult) with
> > the idea of “expanding beyond Emacs”. Org-mode benefits greatly from
> > current and future Emacs development, and asking to standardize “just
> > the parts that are not Emacs” would cause Org-mode to lose that huge
> > advantage. Org-mode relies heavily on the editor it’s built on, and if
> > it ceased to rely on Emacs, it would be forced to rely on “nothing at
> > all” instead. Not only that, but for Org-mode users being able to
> > count on all of Emacs is a big part of why it works. This means
> > separating Org-mode from Emacs is a “lose-lose” idea.
>
> It seems like you have never used Orgzly or read on Org file on GitHub.
> Those are not ideas, but are actual current real-world win-win
> implementations of parts of Org outside of Emacs.
>
> More of these would be better.
>
> Everyone on this thread who says you can't separate Org from Emacs is
> correct that it is unreasonable to expect a 100 % bit-compatible and
> keystroke-compatible experience outside of Emacs. I don't think that level
> of re-implementation was what the OP was suggesting.
>
> Again: GitHub. Orgzly. The conversation should move from "it can't be
> done" or "it isn't helpful" (why so much negativity on this thread?) to
>
> + What parts can be standardized and re-implemented outside of Emacs.
> + How do we define graceful failure for the other parts.
> + How do we support 3rd-party implementation in a way that benefits all of
> us.
>
>   -k.
>
>

[-- Attachment #2: Type: text/html, Size: 5888 bytes --]

Re: Thoughts on the standardization of Org

[Eric S Fraga]

On Tuesday,  3 Nov 2020 at 05:31, Ken Mankoff wrote:
> But I'm weary of seeing all my colleagues say "Jupyter" and not "Org"

+1 (not to mention the case of MSWord instead of Jupyter)

So, yes, if TEC or others can get us there, I'm all for it but they'll
have to prise emacs out of hands when I expire...

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-61-ga88806.dirty

Re: Thoughts on the standardization of Org

[TEC]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Tuesday,  3 Nov 2020 at 05:31, Ken Mankoff wrote:
>> But I'm weary of seeing all my colleagues say "Jupyter" and not 
>> "Org"
>
> +1 (not to mention the case of MSWord instead of Jupyter)
🤢

> So, yes, if TEC or others can get us there, I'm all for it but 
> they'll
> have to prise emacs out of hands when I expire...

Hehe. I'll have a look and see how hard it seems. I'm hoping at 
least
someone else finds it interesting enough to reply to my call for 
help
and give me some company 😃.

No need to pry Emacs out of anything ... after all, his idea is 
all
about sneakily inserting it into people's workflow 😉.

--
Timothy

Re: Thoughts on the standardization of Org

[David Rogers]

Ken Mankoff <mankoff@gmail.com> writes:

> On 2020-11-03 at 00:24 -08, David Rogers 
> <davidandrewrogers@gmail.com> wrote...
>> I disagree (in principle, not just because it would be 
>> difficult) with
>> the idea of “expanding beyond Emacs”. Org-mode benefits greatly 
>> from
>> current and future Emacs development, and asking to standardize 
>> “just
>> the parts that are not Emacs” would cause Org-mode to lose that 
>> huge
>> advantage. Org-mode relies heavily on the editor it’s built on, 
>> and if
>> it ceased to rely on Emacs, it would be forced to rely on 
>> “nothing at
>> all” instead. Not only that, but for Org-mode users being able 
>> to
>> count on all of Emacs is a big part of why it works. This means
>> separating Org-mode from Emacs is a “lose-lose” idea.
>
> It seems like you have never used Orgzly or read on Org file on 
> GitHub. Those are not ideas, but are actual current real-world 
> win-win implementations of parts of Org outside of Emacs.
>
> More of these would be better.
>
> Everyone on this thread who says you can't separate Org from 
> Emacs is correct that it is unreasonable to expect a 100 % 
> bit-compatible and keystroke-compatible experience outside of 
> Emacs. I don't think that level of re-implementation was what 
> the OP was suggesting.
>
> Again: GitHub. Orgzly. The conversation should move from "it 
> can't be done" or "it isn't helpful" (why so much negativity on 
> this thread?) to
>
> + What parts can be standardized and re-implemented outside of 
> Emacs.
> + How do we define graceful failure for the other parts.
> + How do we support 3rd-party implementation in a way that 
> benefits all of us.

I have used most of what you’re describing. None of what you’re 
describing does what the OP was discussing, namely to create a 
clean separation between Emacs and Org-mode in the interest of 
enabling what might be termed “the full complete official 
Org-mode” on non-Emacs editors. Orgzly is a very nice 
implementation of a partial viewer and partial editor for a 
certain subset of .org files, but it doesn’t aim to be an Emacs 
replacement or an authoritative standard. More and better partial 
viewers/partial editors, knowing that “the Org-mode standard” 
equals “precisely how Emacs Org-mode works at the time the 
question is asked”, is a great idea IMO.

Cleaning up a separation between Org-mode and Emacs would 
necessarily mean Org-mode would lose the ability to take advantage 
of future Emacs development - a de facto permanent feature-freeze. 
One of the main advantages of Org-mode is its extensibility, and 
leaving that extensibility behind would suddenly remove one of the 
major actual reasons that it’s attractive (i.e. “Org-mode can’t do 
that yet? Give me half an hour and maybe it will”). Enabling “the 
full complete official Org-mode” to run anywhere, would have the 
eventual effect of it running nowhere, because no one actually 
needs or wants the “It’s Just Markdown On Steroids” list of how to 
properly arrange asterisks and pound-signs that would be the 
result. 

As far as I’m aware, people are currently free to develop 
applications that use their personal interpretation of the .org 
format, and they should do so. The only thing standardization 
would accomplish would be to halt development of the real thing, 
or at least impede it.

(That’s unless the standardization includes so much detail, and 
the bar is set so high, that the standard becomes impractical to 
try to meet anyway.)

-- 
David Rogers

Re: Thoughts on the standardization of Org

[Asa Zeren]

I have collected below some quotations to try to represent the general
sentiments expressed so far in the conversation, upon which I would like to
express some thoughts. Most of these have been repeated several times in spirit,
so they are not just the opinions of the listed authors.

Daniele Nicolodi <daniele@grinta.net> said:
> I don't think it is reasonable to expect much of org-mode being implemented in
> another environment, because that quickly becomes a task as complex as
> implementing Emacs. For example, Org has org-tables, and the formula syntax
> allows for Emacs Calc expressions or Elisp functions to be called: should an
> hypothetical implementation of Org allow the same formulas to be executed?
> Wouldn't that mean that this implementation needs to re-implement a good
> fraction of Emacs (or use Emacs itself for interpreting the formulas?
>
> Maybe the standardization should cover only the "static" parts of Org (IE no
> table formulas, no babel, no agenda, no exporters, etc). However, in this
> case, what is left is little more of a markup language with an editor that
> allows sections folding. You can have this on top of pretty much any markup
> language using Emacs' outline-minor-mode.

Daniele Nicolodi <daniele@grinta.net> said:
> one of the advantages of Org is that, being implemented in Emacs, it has
> infinite potential for customization, thus we would need to agree about what
> org mode is before standardizing it: my Org is not your org, and, thank to the
> features offered be the Emacs environment, I use different flavors of Org in
> different buffers.

I 100% agree that much of the power of org mode comes from the infinite
customization. However, I do not agree that this fact makes standardization
impossible (see below). Additionally, Emacs is not the only customizable tool,
though it is a very good one. The org standard would have to be flexible to
accommodate that "my org is not your org," but this is not impossible.

Ken Mankoff <mankoff@gmail.com> said:
> Therefore, if other tools have the ability to do *something* with an Org file
> (display most of it well enough, allow editing without breaking things, maybe
> implementing a simple Babel interpreter for a few popular languages,
> whatever), this would be A Good Thing.

TEC <tecosaur@gmail.com> said:
> I also think it's to our benefit that non-Emacsers become more comfortable
> with seeing an org file --- as I see it that improves our chances that we can
> directly share Org files with them, which they might be comfortable editing
> and sending back for example, or that a generic tool might think to support
> Org files.

These points are especially important, and while a standard is not the only way
to progress these goals, it is a good step. Also, re the MIME type discussion,
while there was a long list of steps that need to occur for a MIME registration
to be useful, I would like to point out that a standard is the first step
towards that goal, and even though there are more steps, many of them difficult,
that does not mean that we should not make progress.

Greg Minshall <minshall@umich.edu>
> perhaps the standard for e-mail headers (originally, RFC822) might be a useful
> way of thinking about this issue.  it standardizes what it standardizes, and
> then says, "and, by the way, you can put in almost anything else [X-Mailer,
> ...], but you can't count on any other node understanding it".  over time, new
> things are standardized (and, so, moved to, e.g., Mailer, and other things
> aren't.
>
> it seems to me this has worked fairly well, and partly this works because of
> the late Jon Postel's admonition for designing internet protocols: be
> conservative in what you send, and liberal in what you accept.

This is a good precedent for how I would like to standardize org. First,
standardize general structure, without the specifics. Then, as ideas around a
particular feature stabilize, they can be separately standardized, and
incrementally adopted. (Though likely many major implementations (aka Emacs)
would have already implemented it, even though it is non standard.)

For more details on how I think this should be accomplished, see my original
post, and also this reply for clarification:
https://lists.gnu.org/archive/html/emacs-orgmode/2020-11/msg00009.html

Ken Mankoff <mankoff@gmail.com> said:
> The conversation should move from "it can't be done" or "it isn't helpful"
> (why so much negativity on this thread?) to
>
> + What parts can be standardized and re-implemented outside of Emacs.
> + How do we define graceful failure for the other parts.
> + How do we support 3rd-party implementation in a way that benefits all of us.

This post I also find especially important. There seem to be the ideas that (a)
an incomplete implementation is not useful and (b) a standard would necessarily
either require another implementation to implement all of Emacs or would
eliminate the customizability of org.

Thanks,
Asa

Re: Thoughts on the standardization of Org

[Jean Louis]

* Daniele Nicolodi <daniele@grinta.net> [2020-11-02 14:10]:
> On 02/11/2020 10:02, TEC wrote:
> > I think there are absolutely some benefits for Org users. I am
> > personally interested in registering Org as an IANA MIME type.
> 
> I don't think that registering Org as IANA MIME type will have the
> consequences you hope it has.
> 
> > What will this do? Well, for starters I'd like to be able to 
> > attach org
> > files without the type being recognised as
> > "application/vnd.lotus-organizer" 🤢.
> 
> This is not governed by the presence of an MIME type for Org, but by the
> mechanism that assigns a MIME type to the file when it is attached tp a
> message, and this is independent of the IANA MIME type database.
> 
> You (most probably) already can configure your email client (or
> operative system) to use "text/x-org" as the mime type for files with
> extension ".org". Having a IANA MIME type, would only make using
> "text/org" a bit more "right" (there is nothing that stops you from
> using the "text/org" mime type without it being registered).

That is right, I am using it since years in ~/.mailcap that works well
for mutt email client.

text/org;	emacsclient %s; nametemplate=%s.org;
text/x-org;	emacsclient %s; nametemplate=%s.org;

Jean

Re: Thoughts on the standardization of Org

[Maxim Nikulin]

2020-11-08 Jean Louis wrote:
> That is right, I am using it since years in ~/.mailcap that works well
> for mutt email client.
> 
> text/org;	emacsclient %s; nametemplate=%s.org;
> text/x-org;	emacsclient %s; nametemplate=%s.org;

Just for curiosity, couldn't it lead to execution of arbitrary code 
placed into elisp table expressions, some macro, etc.? I have not 
convinced myself that just opening of a file (without executing of src 
blocks) is safe enough and there no dangerous #+startup options or other 
tricks. Emacs is too powerful and too flexible...

Re: Thoughts on the standardization of Org

[Daniele Nicolodi]

On 09/11/2020 15:04, Maxim Nikulin wrote:
> 2020-11-08 Jean Louis wrote:
>> That is right, I am using it since years in ~/.mailcap that works well
>> for mutt email client.
>>
>> text/org;	emacsclient %s; nametemplate=%s.org;
>> text/x-org;	emacsclient %s; nametemplate=%s.org;
> 
> Just for curiosity, couldn't it lead to execution of arbitrary code 
> placed into elisp table expressions, some macro, etc.? I have not 
> convinced myself that just opening of a file (without executing of src 
> blocks) is safe enough and there no dangerous #+startup options or other 
> tricks. Emacs is too powerful and too flexible...

As far as I know it is not any more dangerous than opening any other
file with Emacs. Why do you think the fact that the file extension is
".org" instead of something else makes a relevant difference?

Cheers,
Dan

Re: Thoughts on the standardization of Org

[Jean Louis]

* Maxim Nikulin <manikulin@gmail.com> [2020-11-09 17:06]:
> 2020-11-08 Jean Louis wrote:
> > That is right, I am using it since years in ~/.mailcap that works well
> > for mutt email client.
> > 
> > text/org;	emacsclient %s; nametemplate=%s.org;
> > text/x-org;	emacsclient %s; nametemplate=%s.org;
> 
> Just for curiosity, couldn't it lead to execution of arbitrary code
> placed into elisp table expressions, some macro, etc.?

The file name is created on the fly like temporarily file name. Email
does not carry file name.

But it is true that file names can be used maliciously. Only not in
the case when I am opening Org file from Mutt email client or others.

But if I would be opening Org file with some malicious file name from
other software, I guess there could be problems. Quoting '%s' is
recommended. Mailcap has security issues just as file system has.

When file is opened there is Org file. There is no automatic execution
unless user has set his system to maybe automatically execute stuff.

> I have not convinced myself that just opening of a file (without
> executing of src blocks) is safe enough and there no dangerous
> #+startup options or other tricks.

That is why on GNU/Linux and BSD systems and other systems we have
login with username and passwords and locking screensavers. Those are
for use. Computers should be protected from malicious access.

By all means you are right to be cautious with Emacs that executes
here and there all kinds of things.

For the same reason one shall be cautious of any packages coming from
various popular package repositories as such are not verified for
safety issues.

For any Emacs package never allow local file variables to be executed
unless you are sure what you are doing. Just say no if unsure.

For any package offered by some not common communication line, such as
XMPP chat, or IRC like "Hey there, look what this theme does", do not
trust without being very sure that package is verified or at least
downloaded by many people without complaints.

Any programming language is unsecure if people just execute programs
without verifying background of such programs, people behind it and
fact if many users appreciate programs.

When receiving Org file by email you should know who is person behind
it.

Only Org files I am receiving currently is from Sacha Chua, the Emacs
News as I am subscribed to it. You may subscribe too:
https://sachachua.com/blog/#text-3

-- 
Thanks,
Jean Louis
⎔ λ 🄯 𝍄 𝌡 𝌚

Re: Thoughts on the standardization of Org

[Tim Cross]

Maxim Nikulin <manikulin@gmail.com> writes:

> 2020-11-08 Jean Louis wrote:
>> That is right, I am using it since years in ~/.mailcap that works well
>> for mutt email client.
>>
>> text/org;	emacsclient %s; nametemplate=%s.org;
>> text/x-org;	emacsclient %s; nametemplate=%s.org;
>
> Just for curiosity, couldn't it lead to execution of arbitrary code
> placed into elisp table expressions, some macro, etc.? I have not
> convinced myself that just opening of a file (without executing of src
> blocks) is safe enough and there no dangerous #+startup options or other
> tricks. Emacs is too powerful and too flexible...

By default, it is pretty safe. While you can customize things in such a
way as to expose you to additional danger, you have to explicitly do
that.

There is a risk with many MIME types, for example images, word and excel
documents etc. Even HTML can be a threat, especially if your mail reader
supports JS and is not well engineered with security checks.

No email can be considered 100% safe. However, in addition to the
possible security consequences, you also have to consider the
likelihood. The effort it takes to craft a malicious payload needs some
sort of reward and while that reward might be as trivial as just causing
mayhem, the relatively small user base for org compared to other MIME
types is unlikely to make it an attractive mechanism. You are more
likely to choose something more popular to put your efforts into.

--
Tim Cross

Re: Thoughts on the standardization of Org

[Greg Minshall]

Tim,

> No email can be considered 100% safe.

"e-mail doesn't kill people; e-mail readers kill people".  i would guess
using 'cat -v' to read e-mail is 100% safe.  even throwing in
uudecode(1), or whatever is needed to decode base64, (and then piping
through 'cat -v', of course ), it's probably still safe.

cheers, Greg

Re: Thoughts on the standardization of Org

[Tim Cross]

Greg Minshall <minshall@umich.edu> writes:

> Tim,
>
>> No email can be considered 100% safe.
>
> "e-mail doesn't kill people; e-mail readers kill people".  i would guess
> using 'cat -v' to read e-mail is 100% safe.  even throwing in
> uudecode(1), or whatever is needed to decode base64, (and then piping
> through 'cat -v', of course ), it's probably still safe.
>
> cheers, Greg

There always has to be at least one! ;-)

--
Tim Cross

Re: Thoughts on the standardization of Org

[Greg Minshall]

:)

Re: Thoughts on the standardization of Org

[Maxim Nikulin]

> * Maxim Nikulin [2020-11-09 17:06]:
>> 2020-11-08 Jean Louis wrote:
>>> That is right, I am using it since years in ~/.mailcap that works well
>>> for mutt email client.
>>>
>>> text/org;	emacsclient %s; nametemplate=%s.org;
>>> text/x-org;	emacsclient %s; nametemplate=%s.org;
>>
>> Just for curiosity, couldn't it lead to execution of arbitrary code
>> placed into elisp table expressions, some macro, etc.?

My question is solely concerning content of an org file. Let's assume 
default emacs and org mode settings or customization that does not bring 
more weakness.

I consider an attack through content of an org file obtained from 
network. It may be a message from a person even from usual contact list 
whose computer was infected by some malware. Imagine that botnet 
developers would notice new RFC on org mode and would add a plugin 
capable to add specific payload (fetching and launching its agent using 
elisp) if org files noticed in the host owner mailbox. I would not like 
to see org next to office files in security warnings.

The reason why I am afraid to add emacs as a MIME handler for org files 
is the following. Nowadays vim is shipped with disabled (at least by 
default) modeline that was used to specify e.g. tab width for the 
particular file, but it allows to change too much settings. After 
skimming through the org manual, my impression is that org mode allows 
to override a lot of settings through "#+setup:" and other directives. I 
do not have a solid notion related to all possibilities to inject elisp 
code so I am not sure that no elisp code embedded into received file is 
executed during opening of the file without any user action.

Viewing received file I would prefer a restricted mode at least to avoid 
obviously dangerous actions:
- C-c C-c for src blocks
- recalculation of a table field containing elisp expression 
accidentally fired by Tab or C-c
- export (however it would require more keystrokes, so a chance to 
activate it is not so significant)

I could miss some possibilities to activate arbitrary code. Just 
speculations, maybe such options are safe without modification of 
init.el: custom link handlers, dynamic blocks, column view.

Non-emacs viewer might be safer as a MIME handler despite limited 
functionality.

There was a thread concerning "security considerations" section of RFC 
discussing if the similar section of MarkDown document is suitable for 
org. My impression that org is much more complex.

My worries if arbitrary code could be executed during just opening of a 
file are not directly related to standardization. However I do not think 
that argument on low attack probability due to negligible popularity is 
appropriate in the thread with discussion that standardization could 
make org more wide spread.

2020-11-09 22:59, Jean Louis wrote:
> Quoting '%s' is
> recommended. Mailcap has security issues just as file system has.

I was not going to raise such issues. However I agree that in shell it 
is quite easy to use quotes in a wrong way.

> That is why on GNU/Linux and BSD systems and other systems we have
> login with username and passwords and locking screensavers. Those are
> for use. Computers should be protected from malicious access.

I do not see why it is relevant. Joking colleagues and angry students is 
another attack vector. Mail reader and emacs almost certainly have 
privileges to put something to user's autostart. Passwords are not 
involved. What could help is running a dedicated emacs used as MIME 
handler inside a container with restrictive mount and network namespaces.

> For the same reason one shall be cautious of any packages coming from
> various popular package repositories as such are not verified for
> safety issues.

I would prefer to not touch the subject of degree of trust in respect to 
external packages. Let's limit the scope to org "core", maybe even as a 
part of emacs distribution.

Re: Thoughts on the standardization of Org

[Maxim Nikulin]

2020-11-10 Greg Minshall wrote:
> 
> i would guess
> using 'cat -v' to read e-mail is 100% safe.  even throwing in
> uudecode(1), or whatever is needed to decode base64, (and then piping
> through 'cat -v', of course ), it's probably still safe.

Please, check that you have at least updated tmux before applying such 
"safe" handler: https://www.openwall.com/lists/oss-security/2020/11/05/3 
The news are too recent to not mention the link in such context.

The sour story is that it is unsafe to feed non-trusted files directly 
to terminal. A filter against control sequences is required.

Re: Thoughts on the standardization of Org

[Jean Louis]

* Maxim Nikulin <manikulin@gmail.com> [2020-11-10 19:22]:
> > * Maxim Nikulin [2020-11-09 17:06]:
> > > 2020-11-08 Jean Louis wrote:
> > > > That is right, I am using it since years in ~/.mailcap that works well
> > > > for mutt email client.
> > > > 
> > > > text/org;	emacsclient %s; nametemplate=%s.org;
> > > > text/x-org;	emacsclient %s; nametemplate=%s.org;
> > > 
> > > Just for curiosity, couldn't it lead to execution of arbitrary code
> > > placed into elisp table expressions, some macro, etc.?
> 
> My question is solely concerning content of an org file. Let's assume
> default emacs and org mode settings or customization that does not bring
> more weakness.
> 
> I consider an attack through content of an org file obtained from network.
> It may be a message from a person even from usual contact list whose
> computer was infected by some malware. Imagine that botnet developers would
> notice new RFC on org mode and would add a plugin capable to add specific
> payload (fetching and launching its agent using elisp) if org files noticed
> in the host owner mailbox. I would not like to see org next to office files
> in security warnings.
> 
> The reason why I am afraid to add emacs as a MIME handler for org files is
> the following. Nowadays vim is shipped with disabled (at least by default)
> modeline that was used to specify e.g. tab width for the particular file,
> but it allows to change too much settings. After skimming through the org
> manual, my impression is that org mode allows to override a lot of settings
> through "#+setup:" and other directives. I do not have a solid notion
> related to all possibilities to inject elisp code so I am not sure that no
> elisp code embedded into received file is executed during opening of the
> file without any user action.

Quite understandable and I appreciate your awareness on safe
computing.

That question is solved when there is clear answer if anything gets
executed automatically from #+SETUP line when opening an Org file.

‘#+SETUPFILE: file’
     The setup file or a URL pointing to such file is for additional
     in-buffer settings.  Org loads this file and parses it for any
     settings in it only when Org opens the main file.  If URL is
     specified, the contents are downloaded and stored in a temporary
     file cache.  ‘C-c C-c’ on the settings line parses and loads the
     file, and also resets the temporary file cache.  Org also parses
     and loads the document during normal exporting process.  Org parses
     the contents of this document as if it was included in the buffer.
     It can be another Org file.  To visit the file—not a URL—use ‘C-c
     '’ while point is on the line with the file name.

I have tried to execute file by using:

#+SETUPFILE: ~/tmp/message-me.el in Org file and:

message-me.el:
(message "I have been executed.")

But I did not see message executing. 

> Viewing received file I would prefer a restricted mode at least to avoid
> obviously dangerous actions:
> - C-c C-c for src blocks

See file local variables in manual. They can also be dangerous and
unsafe. Emacs is environment with programming language and executing
many things is possible when user desires so.

Safety of files should be decided from viewpoint of what Emacs does by
default

Question is if anything would be executed by default when Org file is
opening?

> I could miss some possibilities to activate arbitrary code. Just
> speculations, maybe such options are safe without modification of init.el:
> custom link handlers, dynamic blocks, column view.

As activating any code is not by default and is left to user, there is
no need to prevent that generally.

You are aware that it can be executed if you invoke some key presses,
so you can control it by not invoking it or by inspecting it before
invoking it.

> There was a thread concerning "security considerations" section of RFC
> discussing if the similar section of MarkDown document is suitable for org.
> My impression that org is much more complex.

Org is much more complex.

Of course there are ways how some markdown implementations could
execute external programs. Markdown is meant to create HTML. It was
not meant originally to share information directly in Markdown and yet
people do it.

We just need to make sure that nothing by default executes when any
Org file is opened and that is it.

> My worries if arbitrary code could be executed during just opening
> of a file are not directly related to standardization. However I do
> not think that argument on low attack probability due to negligible
> popularity is appropriate in the thread with discussion that
> standardization could make org more wide spread.

Emacs allows any file-local-variables in any type of a document. Then
it asks user if user would like to accept such variables or not:

Hypothetical and untested example:

> Local Variables:
> compile-command: "rm -rf ~/*"
> End:

Many users may not know what "rm -rf ~/*" mean and would say
Y. Automated or hidden attack on many users could be initiated by
implementing a shiny theme, making it popular and having few banal
variables to be accepted, and later introducing change that: captures
users' private information, passwords, contact address book for
spamming or similar. rm -rf would be too obvious. It could download
backdoor shell and have attacker browse files.

> I do not see why it is relevant. Joking colleagues and angry students is
> another attack vector. Mail reader and emacs almost certainly have
> privileges to put something to user's autostart. Passwords are not involved.
> What could help is running a dedicated emacs used as MIME handler inside a
> container with restrictive mount and network namespaces.

chroot, jails (BSD), or running program over shell in other user space
could be possibilites. See example how DragonflyBSD author recommends
how to run browser safely:

How to Run a More Secure Browser
https://www.dragonflybsd.org/docs/handbook/RunSecureBrowser/

> > For the same reason one shall be cautious of any packages coming from
> > various popular package repositories as such are not verified for
> > safety issues.
> 
> I would prefer to not touch the subject of degree of trust in respect to
> external packages. Let's limit the scope to org "core", maybe even as a part
> of emacs distribution.

I am getting you fully.

Re: Thoughts on the standardization of Org

[Jean Louis]

* Maxim Nikulin <manikulin@gmail.com> [2020-11-10 19:31]:
> 2020-11-10 Greg Minshall wrote:
> > 
> > i would guess
> > using 'cat -v' to read e-mail is 100% safe.  even throwing in
> > uudecode(1), or whatever is needed to decode base64, (and then piping
> > through 'cat -v', of course ), it's probably still safe.
> 
> Please, check that you have at least updated tmux before applying such
> "safe" handler: https://www.openwall.com/lists/oss-security/2020/11/05/3 The
> news are too recent to not mention the link in such context.
> 
> The sour story is that it is unsafe to feed non-trusted files directly to
> terminal. A filter against control sequences is required.

Is there anyway to disable control sequences? Than cat can be aliased.

Re: Thoughts on the standardization of Org

[Tim Cross]

Jean Louis <bugs@gnu.support> writes:

> * Maxim Nikulin <manikulin@gmail.com> [2020-11-10 19:31]:
>> 2020-11-10 Greg Minshall wrote:
>> >
>> > i would guess
>> > using 'cat -v' to read e-mail is 100% safe.  even throwing in
>> > uudecode(1), or whatever is needed to decode base64, (and then piping
>> > through 'cat -v', of course ), it's probably still safe.
>>
>> Please, check that you have at least updated tmux before applying such
>> "safe" handler: https://www.openwall.com/lists/oss-security/2020/11/05/3 The
>> news are too recent to not mention the link in such context.
>>
>> The sour story is that it is unsafe to feed non-trusted files directly to
>> terminal. A filter against control sequences is required.
>
> Is there anyway to disable control sequences? Than cat can be aliased.


It should be noted that this vulnerability is a buffer overflow exploit
which ASLR effectively mitigates. This doesn't mean that it isn't a
serious bug in tmux, but it does mean that unless you have disabled
ASLR, there is no known exploit (i.e. it is only theoretical). Given the
popularity of tmux, I suspect it will be patched and a new version
released fairly quickly.

I guess this does highlight the point that *any* data from an external
source can potentially be a threat. You cannot eliminate the risks, only
manage them down to an acceptable level. What is acceptable will vary
for each user.

--
Tim Cross

Re: Thoughts on the standardization of Org

[Tom Gillespie]

This is a great sub-thread that should probably be its own top level
thread on org security.

Org files are mostly benign unless the user does something extremely
dangerous like
setting enable-local-eval t. However, there are some areas where
arbitrary code can be
executed (as intended) that some users may not be aware of. Consider
the following.

#+begin_src elisp :var lol=(message "If I were evil I could use
call-process here to give you nasal demons")
#+end_src

When is the closure for the variable lol evaluated? The most obvious time is if
you run the block. The less obvious time is if you run org export!

More subtly.

#+name: some-block
#+begin_src elisp
#+end_src

#+call: some-block() :var pwnd=(message "oops!")

When will the closure for pwnd be called? When exporting it happens before
you receive the prompt asking whether you want to execute some-block!
Very easy to fool someone who doesn't know how top level closures work
into goofing on that one. Furthermore even when a file also sets

#+property: header-args :eval no-export

then the closure will still be evaluated!

Or how about

Running org export can execute arbitrary code even if you decline to run blocks
or set default header-args in your config. This means that it is not safe to,
for example, run a public server that transforms arbitrary org files into pdfs.

How about some more fun?

#+name: some-block
#+begin_src elisp :tangle ./itsatrap.el :dir (message "OH NO")
#+end_src

Yep. That does what you think it does. If you tangle the file, arbitrary code
execution. No tangling files from strangers either.

The examples I present here do require user input to fire off the process,
it won't happen without them doing something but it is much easier to
C-c C-e h h or C-c C-v t when you are an expert user, so you have to
be careful. Sharp tools.

Despite these examples, the ability to define values using arbitrary closures
is one of Org's most powerful features.

In the context of standardization, this suggests that it might be nice to
have a dynamic variable that controls whether bare closures will be
evaluated (one might already exist, I have no idea), along with a spec
that says what the failure modes should be if one is encountered that
cannot be evaluated.

In the context of org generally, maybe that variable could also be used like
org-confirm-babel-evaluate and take a function as an argument, ask if t, don't
ask if nil, or ask only if result is t (or was it nil ... regardless match
org-confirm-babel-evaluate). org-confirm-closure-evaluate maybe? (Again if
this already exists, then woo!).

Best!
Tom

Re: Thoughts on the standardization of Org

[Tim Cross]

Tom Gillespie <tgbugs@gmail.com> writes:

> This is a great sub-thread that should probably be its own top level
> thread on org security.
>
> Org files are mostly benign unless the user does something extremely
> dangerous like
> setting enable-local-eval t. However, there are some areas where
> arbitrary code can be
> executed (as intended) that some users may not be aware of. Consider
> the following.
>
> #+begin_src elisp :var lol=(message "If I were evil I could use
> call-process here to give you nasal demons")
> #+end_src
>
> When is the closure for the variable lol evaluated? The most obvious time is if
> you run the block. The less obvious time is if you run org export!
>
> More subtly.
>
> #+name: some-block
> #+begin_src elisp
> #+end_src
>
> #+call: some-block() :var pwnd=(message "oops!")
>
> When will the closure for pwnd be called? When exporting it happens before
> you receive the prompt asking whether you want to execute some-block!
> Very easy to fool someone who doesn't know how top level closures work
> into goofing on that one. Furthermore even when a file also sets
>
> #+property: header-args :eval no-export
>
> then the closure will still be evaluated!
>
> Or how about
>
> Running org export can execute arbitrary code even if you decline to run blocks
> or set default header-args in your config. This means that it is not safe to,
> for example, run a public server that transforms arbitrary org files into pdfs.
>
> How about some more fun?
>
> #+name: some-block
> #+begin_src elisp :tangle ./itsatrap.el :dir (message "OH NO")
> #+end_src
>
> Yep. That does what you think it does. If you tangle the file, arbitrary code
> execution. No tangling files from strangers either.
>
> The examples I present here do require user input to fire off the process,
> it won't happen without them doing something but it is much easier to
> C-c C-e h h or C-c C-v t when you are an expert user, so you have to
> be careful. Sharp tools.
>
> Despite these examples, the ability to define values using arbitrary closures
> is one of Org's most powerful features.
>
> In the context of standardization, this suggests that it might be nice to
> have a dynamic variable that controls whether bare closures will be
> evaluated (one might already exist, I have no idea), along with a spec
> that says what the failure modes should be if one is encountered that
> cannot be evaluated.
>
> In the context of org generally, maybe that variable could also be used like
> org-confirm-babel-evaluate and take a function as an argument, ask if t, don't
> ask if nil, or ask only if result is t (or was it nil ... regardless match
> org-confirm-babel-evaluate). org-confirm-closure-evaluate maybe? (Again if
> this already exists, then woo!).
>

Another one, which I think is even a little more subtle, is table
formula evaluation. Using the advanced features of tables, you can
define a table that will automatically run formulas (which could be a
calc expression OR an elisp function) when you navigate through a table,
such as by hitting tab to move to the next column. I imagine this is
something you could easily do as tabbing through a table is a
convenient way to browse the contents.

While by default, org is pretty good, the real risk is that I suspect
many will have tweaked the evaluation settings - I know I have and while
I would not evaluate or export anything in an org file I was sent until
I had assessed it, mistakes do happen and I could easily hit C-c C-c in
the wrong context and cause a block to be evaluated without any
prompting for confirmation.

Given the complexities of dynamic content evaluation in org, perhaps a
simple solution would be to add a special MIME handler function which
would open an org file in a restricted mode, ignoring user settings
affecting code evaluation and preventing any form of evaluation, even if
the user asks for it. Basically, allow viewing, folding, navigation of
the org file, but no evaluation. If the user wants to export or tangle
or update tables etc, they would be forced to save the file and then
re-open it as a normal org file. It would be this handler function you
would configure in your MIME types as the function to run to view an org
attachment etc.

You could take it one step further an allow the definition of a 'tursted
senders' list. When opening an org attachment, this list is checked and
if the sender is in the list, normal org open process is applied,
otherwise the restricted MIME open function is applied. This would be
similar to the Gmail approach for handling images.

--
Tim Cross

Re: Thoughts on the standardization of Org

[Greg Minshall]

Maxim,

thanks.  small note.

> The sour story is that it is unsafe to feed non-trusted files directly
> to terminal. A filter against control sequences is required.

thus, the '-v' argument to cat(1) (which Rob Pike famously considered
harmful. :)

cheers.

Re: Thoughts on the standardization of Org

[Jean Louis]

* Tim Cross <theophilusx@gmail.com> [2020-11-11 01:30]:
> 
> Jean Louis <bugs@gnu.support> writes:
> 
> > * Maxim Nikulin <manikulin@gmail.com> [2020-11-10 19:31]:
> >> 2020-11-10 Greg Minshall wrote:
> >> >
> >> > i would guess
> >> > using 'cat -v' to read e-mail is 100% safe.  even throwing in
> >> > uudecode(1), or whatever is needed to decode base64, (and then piping
> >> > through 'cat -v', of course ), it's probably still safe.
> >>
> >> Please, check that you have at least updated tmux before applying such
> >> "safe" handler: https://www.openwall.com/lists/oss-security/2020/11/05/3 The
> >> news are too recent to not mention the link in such context.
> >>
> >> The sour story is that it is unsafe to feed non-trusted files directly to
> >> terminal. A filter against control sequences is required.
> >
> > Is there anyway to disable control sequences? Than cat can be aliased.
> 
> 
> It should be noted that this vulnerability is a buffer overflow exploit
> which ASLR effectively mitigates. This doesn't mean that it isn't a
> serious bug in tmux, but it does mean that unless you have disabled
> ASLR, there is no known exploit (i.e. it is only theoretical). Given the
> popularity of tmux, I suspect it will be patched and a new version

Do you know how to disable control sequences?

Re: Thoughts on the standardization of Org

[Tim Cross]

Jean Louis <bugs@gnu.support> writes:

> * Tim Cross <theophilusx@gmail.com> [2020-11-11 01:30]:
>>
>> Jean Louis <bugs@gnu.support> writes:
>>
>> > * Maxim Nikulin <manikulin@gmail.com> [2020-11-10 19:31]:
>> >> 2020-11-10 Greg Minshall wrote:
>> >> >
>> >> > i would guess
>> >> > using 'cat -v' to read e-mail is 100% safe.  even throwing in
>> >> > uudecode(1), or whatever is needed to decode base64, (and then piping
>> >> > through 'cat -v', of course ), it's probably still safe.
>> >>
>> >> Please, check that you have at least updated tmux before applying such
>> >> "safe" handler: https://www.openwall.com/lists/oss-security/2020/11/05/3 The
>> >> news are too recent to not mention the link in such context.
>> >>
>> >> The sour story is that it is unsafe to feed non-trusted files directly to
>> >> terminal. A filter against control sequences is required.
>> >
>> > Is there anyway to disable control sequences? Than cat can be aliased.
>>
>>
>> It should be noted that this vulnerability is a buffer overflow exploit
>> which ASLR effectively mitigates. This doesn't mean that it isn't a
>> serious bug in tmux, but it does mean that unless you have disabled
>> ASLR, there is no known exploit (i.e. it is only theoretical). Given the
>> popularity of tmux, I suspect it will be patched and a new version
>
> Do you know how to disable control sequences?

No, I doubt you can as they are fairly fundamental to tmux operation.

--
Tim Cross

Re: Thoughts on the standardization of Org

[Maxim Nikulin]

2020-11-11 Jean Louis wrote:
> * Maxim Nikulin [2020-11-10 19:31]:
>> 2020-11-10 Greg Minshall wrote:
>>>
>>> i would guess
>>> using 'cat -v' to read e-mail is 100% safe.  even throwing in
>>> uudecode(1), or whatever is needed to decode base64, (and then piping
>>> through 'cat -v', of course ), it's probably still safe.
>>
>> Please, check that you have at least updated tmux before applying such
>> "safe" handler: https://www.openwall.com/lists/oss-security/2020/11/05/3 The
>> news are too recent to not mention the link in such context.
>>
>> The sour story is that it is unsafe to feed non-trusted files directly to
>> terminal. A filter against control sequences is required.
> 
> Is there anyway to disable control sequences? Than cat can be aliased.

We were kidding.

You do not need a terminal if you do not need control sequences. They 
plays the role of interface to allow line (or full screen) editing that 
is why control sequences is the essence of terminals. I suppose you 
would get tired almost immediately having to type everything strictly 
sequential without ability to remove even the last character. Some 
terminals allows to disable particular features, e.g. setting of title 
in xterm. But there are still a lot of rather basic capabilities.

Likely pasting a command from a web page is a more real threat. E.g. zsh 
could be more restrictive than bash during copy a peace of text into 
terminal. If you have to work in non-trusted environments, some general 
recommendations (e.g. keep you system up to date) and isolation 
techniques (virtual machines or at least separate system users) could be 
applied.

My point was that MIME handlers have to be carefully chosen. Even well 
known applications could have special options. And sorry, I somehow 
missed "-v" option of cat in Greg's message. It is exactly the case of a 
tool that everyone knows and a significantly more rare option.

Re: Thoughts on the standardization of Org

[Jean Louis]

* Maxim Nikulin <manikulin@gmail.com> [2020-11-11 20:17]:
> 2020-11-11 Jean Louis wrote:
> > * Maxim Nikulin [2020-11-10 19:31]:
> > > 2020-11-10 Greg Minshall wrote:
> > > > 
> > > > i would guess
> > > > using 'cat -v' to read e-mail is 100% safe.  even throwing in
> > > > uudecode(1), or whatever is needed to decode base64, (and then piping
> > > > through 'cat -v', of course ), it's probably still safe.
> > > 
> > > Please, check that you have at least updated tmux before applying such
> > > "safe" handler: https://www.openwall.com/lists/oss-security/2020/11/05/3 The
> > > news are too recent to not mention the link in such context.
> > > 
> > > The sour story is that it is unsafe to feed non-trusted files directly to
> > > terminal. A filter against control sequences is required.
> > 
> > Is there anyway to disable control sequences? Than cat can be aliased.
> 
> We were kidding.
> 
> You do not need a terminal if you do not need control sequences. They plays
> the role of interface to allow line (or full screen) editing that is why
> control sequences is the essence of terminals. I suppose you would get tired
> almost immediately having to type everything strictly sequential without
> ability to remove even the last character. Some terminals allows to disable
> particular features, e.g. setting of title in xterm. But there are still a
> lot of rather basic capabilities.

I know what you mean. I did not express myself very specific. What I
meant is to alias cat to something else that specifically turns off
control sequences.

Like alias cat='sequence off; cat' something like that

Somebody already mentioned there is cat -v to show nonprinting
characters with notation ^- and M- so that may be the solution and I
may be wrong there.

I am often using cat to view files and for some remote files I have to
put attention.

Re: Thoughts on the standardization of Org

[Greg Minshall]

Jean Louis,

> Like alias cat='sequence off; cat' something like that
> 
> Somebody already mentioned there is cat -v to show nonprinting
> characters with notation ^- and M- so that may be the solution and I
> may be wrong there.

yes, 'cat -v' will do it for you.  (or, i'd like to know if i've been
wrong all these years! :)

: cat `which cat` | cat -v | less

cheers.

Re: Thoughts on the standardization of Org

[Maxim Nikulin]

2020-11-11 Jean Louis wrote:
> 
> Do you know how to disable control sequences?

It is neither directly related to `cat -v` nor specific to org, but 
still a notable demonstration that inaccurate treatment of text to be 
inserted into a terminal could do some dangerous things due to hidden 
control characters (e.g. escape from emacs to shell and execute 
something weird):

https://security.stackexchange.com/questions/39118/how-can-i-protect-myself-from-this-kind-of-clipboard-abuse

Re: Thoughts on the standardization of Org

[Jean Louis]

* Maxim Nikulin <manikulin@gmail.com> [2020-11-27 19:51]:
> 2020-11-11 Jean Louis wrote:
> > 
> > Do you know how to disable control sequences?
> 
> It is neither directly related to `cat -v` nor specific to org, but still a
> notable demonstration that inaccurate treatment of text to be inserted into
> a terminal could do some dangerous things due to hidden control characters
> (e.g. escape from emacs to shell and execute something weird):
> 
> https://security.stackexchange.com/questions/39118/how-can-i-protect-myself-from-this-kind-of-clipboard-abuse

I have read the article, thank you. Valid points.