Binary Table & Babel Data Passing

Binary Table & Babel Data Passing

[Psionic K]

Buffer text is not a good data transfer protocol for moving return
values from babel block to block.  It is also not a good storage
medium for table data.  I propose we implement more binary data
passing strategies.

Elisp memory is it's own source of problems, so what would likely work
better is a dynamic module to store values, like a binary cache for
data.  Don't quote me.  I didn't do a three year design analysis on
this.

Within the org document, such a cache would be implemented by
returning truncated results to Elisp for display in an overlay.  Table
display can be handled much the same way.  The user sees some data.
Everything runs basically the same and with a similar user experience,
but wildly faster.

In fact, going the dynamic module route paves the way for displaying
tables outside of Emacs display, getting around the limitations that
make multi-line cells etc problematic.  Such a table viewer could
operate its own frames, support tabs etc, and bring Org much, much
closer to a traditional spreadsheet application in terms of
capability.

But first things first, binary data passing between babel blocks and
avoiding serialization for languages where the runtime can be
convinced to do it should have been done two decades ago on POSIX.
Next best time is now, using 2025 tools.

Re: Binary Table & Babel Data Passing

[Ihor Radchenko]

Psionic K <psionik@positron.solutions> writes:

> But first things first, binary data passing between babel blocks and
> avoiding serialization for languages where the runtime can be
> convinced to do it should have been done two decades ago on POSIX.
> Next best time is now, using 2025 tools.

That might be possible in some cases, if you pass some kind of reference
to data between the code blocks, in addition to the printed
representation of that data (if it needs to be inserted into Org
document). For example, it can be a link to file dump or reference
inside running babel session.

I'd say that sessions is one way to pass the data around without having
to serialize it into Emacs.

AFAIK, dumping to a file is something we currently do not have. If you
want to work on such feature, do let me know.

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>