Python code for producing Org tables

Python code for producing Org tables

[François Pinard]

Hi, Org people.

I recently needed to produce Org tables from within Python, splitting
them as needed to fit within a preset width.  I append the code after my
signature, in case it would be useful to others (or even, if you have
ideas to improve it).

One thing I only realized after writing this, however, is that I wrongly
thought Org mode was aligning floating numbers on the decimal period,
while it merely right align floating numbers regardless of the position
of the period.  So, I turn my mistake into a suggestion, as I think it
would be more convenient if Org mode was aligning floating numbers more
appropriately.

I even thought trying to contribute some Emacs Lisp code to do so, but
seeing that I'm short on free time in these days (like too often), I now
find more fruitful to merely share the idea (and the Python code) now.

François



def to_org(titles, rows, write, hide_empty=False, margin=0, easy=4, span=1,
           fillto=None, limit=None):
    """\
Given a list of column TITLES, and a list of ROWS, each containing a list
of columns, use WRITE to produce a Org formatted table with the text of
columns. If HIDE_EMPTY is not False, then omit columns containing nothing but
empty strings.  The formatted table is shifted right by MARGIN columns.

To accomodate for titles, the width of a column will easily extend to EASY,
or to whatever is needed so the title is not split on more than SPAN lines.
FILLTO may be used to force last column to extend until that position.  LIMIT
may be used to impose a limit to the number of characters in produced lines.

If TITLES is None, the titles are not produced.

Columns containing only numbers (integer or floating) align them properly.
"""

    # Exit if nothing to display.
    if not rows:
        return

    # Compute widths from data.
    rows = [[safe_unicode(column, 30).replace('\\', '\\\\')
             .replace('|', '\\vert{}') for column in row]
             for row in rows]
    # Each WIDTH is the column width as strings.  Each LEFT is either
    # False when there is a no-number in the column or the maximum width
    # of the integral part of all numbers.  When LEFT is not None, each
    # RIGHT is either the maximum width of the fraction part including
    # the decimal point of all numbers, or 0 if all pure integers.
    widths = [0] * len(rows[0])
    lefts = [0] * len(rows[0])
    rights = [0] * len(rows[0])
    for row in rows:
        for counter, cell in enumerate(row):
            widths[counter] = max(widths[counter], len(cell))
            if lefts[counter] is not False:
                match = re.match('([0-9]*)(\\.[0-9]*)$', cell)
                if match is None:
                    lefts[counter] = False
                else:
                    lefts[counter] = max(lefts[counter],
                                         len(match.group(1)))
                    if match.group(2):
                        rights[counter] = max(rights[counter],
                                              len(match.group(2)))
    for counter, (left, right) in enumerate(zip(lefts, rights)):
        if left == 0 and right == 0:
            lefts[counter] = False
        elif left is not False:
            widths[counter] = left + right

    # Extend widths as needed to make room for titles.
    if titles is not None:
        for counter, (width, title) in enumerate(zip(widths, titles)):
            if (not hide_empty or width) and len(title) > width:
                if len(title) <= easy:
                    widths[counter] = len(title)
                else:
                    for nlines in range(2, span):
                        if len(title) <= easy * nlines:
                            widths[counter] = max(
                                width, (len(title) + nlines - 1) // nlines)
                            break
                    else:
                        widths[counter] = max(
                            width, (len(title) + span - 1) // span)
    if fillto:
        extend = fillto - margin - sum(widths) - 3 * len(widths) - 1
        if extend > 0:
            widths[-1] += extend

    # Horizontally split the display so each part fits within LIMIT columns.
    end = 0
    while end < len(widths):
        start = end
        if limit is None:
            end = len(widths)
        else:
            remaining = limit - margin - widths[start] - 4
            end = start + 1
            while end < len(widths) and remaining >= widths[end] + 3:
                remaining -= widths[end] + 3
                end += 1
        # Now ready to output columns from START to END (excluded).

        # Skip this part if nothing to display.
        if hide_empty:
            for width in widths[start:end]:
                if width:
                    break
            else:
                continue
        if start > 0:
            write('\n')

        if titles is not None:
            # Write title lines, splitting titles as needed.
            pairs = zip(widths[start:end], titles[start:end])
            for counter in range(span):
                fragments = []
                inked = False
                for width, title in pairs:
                    if not hide_empty or width:
                        fragment = title[counter * width:(counter + 1) * width]
                        if fragment:
                            inked = True
                        fragments.append(
                            fragment.replace('|', ' ').lstrip().ljust(width))
                if not inked:
                    break
                write('%s| %s |\n' % (' ' * margin,
                                      ' | '.join(fragments)))

            # Write separator line.
            fragments = []
            for width in widths[start:end]:
                if not hide_empty or width:
                    fragments.append('-' * width)
            write('%s|-%s-|\n' % (' ' * margin, '-+-'.join(fragments)))

        # Write body lines.
        for row in rows:
            fragments = []
            for width, left, cell in zip(
                    widths[start:end], lefts[start:end], row[start:end]):
                if not hide_empty or width:
                    if left is False:
                        text = cell.replace('|', ' ').lstrip()
                    else:
                        position = cell.find('.')
                        if position < 0:
                            position = len(cell)
                        text = ' ' * (left - position) + cell
                    fragments.append(text.ljust(width))
            write('%s| %s |\n' % (' ' * margin,
                                  ' | '.join(fragments)))


unprintable_regexp = re.compile(
    '[%s]' % re.escape(''.join(map(unichr, range(0, 32) + range(127, 160)))))


def safe_unicode(value, limit=None):
    if value is None:
        return ''
    if isinstance(value, str):
        try:
            value = unicode(value, encoding)
        except UnicodeDecodeError:
            # FIXME: Too fishy!
            value = unicode(value, 'iso-8859-1')
    elif not isinstance(value, unicode):
        # FIXME: Il semble que la sortie de Rpy2 ne souffre pas ", encoding"?
        value = unicode(value)
    if re.search(unprintable_regexp, value):
        value = repr(value)
        if value.startswith('u'):
            value = value[1:]
    if limit is not None and len(value) > limit:
        left_cut = limit * 2 // 3
        right_cut = limit - left_cut
        return value[:left_cut - 1] + u'…' + value[-right_cut:]
    return value

Re: Python code for producing Org tables

[Bastien]

Hi François,

François Pinard <pinard@iro.umontreal.ca> writes:

> I recently needed to produce Org tables from within Python, splitting
> them as needed to fit within a preset width.  I append the code after my
> signature, in case it would be useful to others (or even, if you have
> ideas to improve it).

Thanks for the code, if some can test it and put it on worg, all the
better (if you agree of course.)

> One thing I only realized after writing this, however, is that I wrongly
> thought Org mode was aligning floating numbers on the decimal period,
> while it merely right align floating numbers regardless of the position
> of the period.  So, I turn my mistake into a suggestion, as I think it
> would be more convenient if Org mode was aligning floating numbers more
> appropriately.

Is it a standard practise when displaying float numbers in
spreadsheets/tables?

If so, this is a nice Elisp challenge for anyone willing to
implement it :)  I won't try it myself for now.

-- 
 Bastien

Re: Python code for producing Org tables

[François Pinard]

Bastien <bzg@altern.org> writes:

> Hi François,

Salut, et Joyeuses Festivités! :-)

>> I recently needed to produce Org tables from within Python

> Thanks for the code, if some can test it and put it on worg, all the
> better (if you agree of course.)

Of course!

>> I wrongly thought Org mode was aligning floating numbers on the
>> decimal period, while it merely right align floating numbers
>> regardless of the position of the period.

> Is it a standard practise when displaying float numbers in
> spreadsheets/tables?

I do not know, and should check indeed.  Yet, even then, it does not
mean Org tables could not do better than standard practice! :-)

> If so, this is a nice Elisp challenge for anyone willing to
> implement it :)

I'd tackle it myself, yet if I do it before a few weeks, it'd mean I'm
toying rather than doing what I should rather be doing...  Sigh!

Happy Times to everybody!

François

Re: Python code for producing Org tables

[Bastien]

Hi François,

François Pinard <pinard@iro.umontreal.ca> writes:

> I do not know, and should check indeed.  Yet, even then, it does not
> mean Org tables could not do better than standard practice! :-)

Would you like something like

| 3.1 |
| 3.1415 |

be aligned/modified as  

| 3.1000 |
| 3.1415 |

?

I personally wouldn't like this, but I'm curious.

>> If so, this is a nice Elisp challenge for anyone willing to
>> implement it :)
>
> I'd tackle it myself, yet if I do it before a few weeks, it'd mean I'm
> toying rather than doing what I should rather be doing...  Sigh!

Don't worry, once you'll dive into `org-table-align' chances are that
you will feel like "working" instead of "toying" :)

Bonnes fêtes !

-- 
 Bastien

Re: Python code for producing Org tables

[Dov Grobgeld]

Nice! I've needed that often. But please add licensing information to
the code to make clear in what contexts it may be used.

Regards,
Dov

On Mon, Dec 24, 2012 at 3:12 PM, Bastien <bzg@altern.org> wrote:
>
> Hi François,
>
> François Pinard <pinard@iro.umontreal.ca> writes:
>
> > I do not know, and should check indeed.  Yet, even then, it does not
> > mean Org tables could not do better than standard practice! :-)
>
> Would you like something like
>
> | 3.1 |
> | 3.1415 |
>
> be aligned/modified as
>
> | 3.1000 |
> | 3.1415 |
>
> ?
>
> I personally wouldn't like this, but I'm curious.
>
> >> If so, this is a nice Elisp challenge for anyone willing to
> >> implement it :)
> >
> > I'd tackle it myself, yet if I do it before a few weeks, it'd mean I'm
> > toying rather than doing what I should rather be doing...  Sigh!
>
> Don't worry, once you'll dive into `org-table-align' chances are that
> you will feel like "working" instead of "toying" :)
>
> Bonnes fêtes !
>
> --
>  Bastien
>

Re: Python code for producing Org tables

[François Pinard]

Bastien <bzg@altern.org> writes:

> Would you like something like

> | 3.1 |
> | 3.1415 |

> be aligned/modified as  

> | 3.1000 |
> | 3.1415 |

> ?

More precisely as:

> | 3.1    |
> | 3.1415 |

which is aligning on dots, even if it looks like left-justifying.

There is a problem in that I have tables using a decimal period, and
others using a decimal comma, and the Python code only handles periods.
Instead of diving into complex locales things, which would not do
anyway, I would merely recognize both as a reasonable compromise.

> Don't worry, once you'll dive into `org-table-align' chances are that
> you will feel like "working" instead of "toying" :)

Grrr! :-)

> Bonnes fêtes !

Itou, itou!

François (who is soon going, to play the office this morning).

Re: Python code for producing Org tables

[François Pinard]

Dov Grobgeld <dov.grobgeld@gmail.com> writes:

> Nice!  I've needed that often.  But please add licensing information
> to the code to make clear in what contexts it may be used.

You mean, for the Python bits I sent to the list?  For such a short
amount of code, do whatever you feel like it.  If you really want an
explicit license, then consider either the GPLv2, or whatever Org Lisp
code is using.  I'll merely let you add it on your side, rather than
polluting the list with a re-invoice. :-)

François (who is not a fussy guy).