Re: packages submissions : ob-sql-session and org-blog

Re: packages submissions : ob-sql-session and org-blog

[Phil Estival]

* [2024-09-11 05:01] Adam Porter:
> Hi Phil,
> 
> A quick, initial suggestion: it would be good to reformat the Lisp code 
> in both projects, as I see several cases of incorrect indentation, 
> hanging parens, extra blank lines, etc, and some of them would make the 
> code hard to read.  For my Lisp editing, I use aggressive-indent-mode, 
> which ensures that the code is always indented correctly.  For some 
> other reformatting, Oleh Krehel's Lispy package has commands to help.

Hi Adam,

> Also, for org-sql-session (which sounds very useful), it would be good
> to compare and contrast it with the package it's meant to supersede.

The following comparison was added to the readme file.

ob-sql-mode :
- is very simple : forward the sql source through `sql-redirect'
- has a test suite
- but gives clunky output
- no :results table
- sql client shell commands messes up output
- prompt again for connection parameters when restarting a session

ob-sql-session :
- handle large results
- has :results tables
- accept header variables (:var)
- accept sql client shell commands
- keep login parameters
- prompt only for blank connection parameters
- can use `with-environment-variables'
- provide some more tests

> since it's meant to integrate with org-babel, it might be good to 
> discuss its inclusion on the Org mailing list also.

CC-ing the Org mailing list.

Phil

Re: packages submissions : ob-sql-session and org-blog

[Adam Porter]

Hi Phil,

On 9/17/24 05:00, Phil Estival wrote:

>> Also, for org-sql-session (which sounds very useful), it would be good
>> to compare and contrast it with the package it's meant to supersede.
> 
> The following comparison was added to the readme file.
> 
> ob-sql-mode :
> - is very simple : forward the sql source through `sql-redirect'
> - has a test suite
> - but gives clunky output
> - no :results table
> - sql client shell commands messes up output
> - prompt again for connection parameters when restarting a session
> 
> ob-sql-session :
> - handle large results
> - has :results tables
> - accept header variables (:var)
> - accept sql client shell commands
> - keep login parameters
> - prompt only for blank connection parameters
> - can use `with-environment-variables'
> - provide some more tests
> 
>> since it's meant to integrate with org-babel, it might be good to 
>> discuss its inclusion on the Org mailing list also.
To be clear, this makes 3 implementations of SQL support for Org Babel 
that I know of:

1. The built-in Org Babel SQL support, aka ob-sql, documented at 
<https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-sql.html>

2. ob-sql-mode, hosted at <https://github.com/nikclayton/ob-sql-mode>, 
which advertises itself as "an alternative backend for Org-Babel SQL SRC 
blocks that uses sql-mode to evaluate the query instead of Org-Babel's 
built-in SQL backends."

3. Your new ob-sql-session library.

You've mentioned how your library compares to ob-sql-mode, but how does 
it compare to the built-in ob-sql library?

Thanks,
Adam

Re: package submission : ob-sql-session

[Phil Estival]

Hi Adam,

* [2024-09-18 00:25] Adam Porter:
>
> Hi Phil,
> 
> On 9/17/24 05:00, Phil Estival wrote:
> 
>>> Also, for org-sql-session (which sounds very useful), it would be good
>>> to compare and contrast it with the package it's meant to supersede.
>>
>> The following comparison was added to the readme file.
>>
>> ob-sql-mode :
>> - is very simple : forward the sql source through `sql-redirect'
>> - has a test suite
>> - but gives clunky output
>> - no :results table
>> - sql client shell commands messes up output
>> - prompt again for connection parameters when restarting a session
>>
>> ob-sql-session :
>> - handle large results
>> - has :results tables
>> - accept header variables (:var)
>> - accept sql client shell commands
>> - keep login parameters
>> - prompt only for blank connection parameters
>> - can use `with-environment-variables'
>> - provide some more tests
>>
>>> since it's meant to integrate with org-babel, it might be good to 
>>> discuss its inclusion on the Org mailing list also.
> To be clear, this makes 3 implementations of SQL support for Org Babel 
> that I know of:
> 
> 1. The built-in Org Babel SQL support, aka ob-sql, documented at 
> <https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-sql.html>
> 
> 2. ob-sql-mode, hosted at <https://github.com/nikclayton/ob-sql-mode>, 
> which advertises itself as "an alternative backend for Org-Babel SQL SRC 
> blocks that uses sql-mode to evaluate the query instead of Org-Babel's 
> built-in SQL backends."
> 
> 3. Your new ob-sql-session library.
> 
> You've mentioned how your library compares to ob-sql-mode, but how does 
> it compare to the built-in ob-sql library?
> 

Here is a comparison with ob-sql-mode

ob-sql-session exists is for session support, which is in the TODOs
of ob-sql.

- ob-sql command execution relies on org-babel-eval
   (→ process-file → call-process).

- ob-sql-session runs an inferior process (in which
   sqli-interactive-mode can be activated when needed).
   The process output is filtered (e.g. results and prompts).
   When a session is demanded, this shell stays open for further commands
   and can keep a state (typically, when given special SQL commands).

|-----------+----------------------------+----------------------------|
|           | ob-sql                     | ob-sql-session             |
|-----------+----------------------------+----------------------------|
| Feat.     | - cmdline                  | - support for sessions     |
|           | - colnames as header arg   | - optionnal colnames       |
|-----------+----------------------------+----------------------------|
| TODO      |                            |                            |
|-----------+----------------------------+----------------------------|
|           | - support for sessions     | - colnames as header arg   |
|           | - support for more engines | - support for more engines |
|-----------+----------------------------+----------------------------|
| engines   |                            |                            |
| supported |                            |                            |
|-----------+----------------------------+----------------------------|
|           | - mysql                    | - Postgresql               |
|           | - dbi                      | - sqlite                   |
|           | - mssql                    |                            |
|           | - sqsh                     |                            |
|           | - postgresql               |                            |
|           | - oracle                   |                            |
|           | - vertica                  |                            |
|           | - saphana                  |                            |
|-----------+----------------------------+----------------------------|

ob-sql-session adapts sql-connect of sql.el by declaring
ob-sql-connect, in order to prompt only for missing connection
parameters.

- ob-sql defines org-babel-sql-dbstring-[engine]
    to be provided on a shell command line.

- likewise, ob-sql-session has to define
    - a connection string,
    - the prompt,
    - and the terminal command prefix
      for a every supported SQL client shell (or "engines")

- ob-sql-session requires sql.el.
   With the above defined, it should be compatible
   with most database of the sql.el's zoo. maybe.

Phil

Re: package submission : ob-sql-session

[Ihor Radchenko]

Phil Estival <pe@7d.nz> writes:

>>> ob-sql-session :
>>> - handle large results
>>> - has :results tables
>>> - accept header variables (:var)
>>> - accept sql client shell commands
>>> - keep login parameters
>>> - prompt only for blank connection parameters
>>> - can use `with-environment-variables'
>>> - provide some more tests

Have you considered integrating it into ob-sql itself?
If not, what are the obstacles?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: package submission : ob-sql-session

[Phil Estival]

* [2024-10-20 14:34] Ihor Radchenko:
> Phil Estival <pe@7d.nz> writes:
> 
>>>> ob-sql-session :
>>>> - handle large results
>>>> - has :results tables
>>>> - accept header variables (:var)
>>>> - accept sql client shell commands
>>>> - keep login parameters
>>>> - prompt only for blank connection parameters
>>>> - can use `with-environment-variables'
>>>> - provide some more tests
> 
> Have you considered integrating it into ob-sql itself?
> If not, what are the obstacles?
> 

Hello Ihor,

yes I have. It grows ob-sql by approximately 300 lines.
There are rooms for improvements.
https://github.com/flintforge/ob-sql-session/blob/main/ob-sql.el

Current status :
- added support of :sessions to ob-sql.el, by merging ob-sql-session.el
- restored support for sqlite (even if there is ob-sqlite.el)
- added composite types for options
- moved options from org-babel to a new group org-babel-sql

TODO :
- use `org-babel-prep-session'
- verify headers and columns options
- check use and storage of :dbconnection
- minimize writings of temporary files
- Tests

Cheers,

Phil

Re: package submission : ob-sql-session

[Ihor Radchenko]

Phil Estival <pe@7d.nz> writes:

>> Have you considered integrating it into ob-sql itself?
>> If not, what are the obstacles?
>> 
>
> Hello Ihor,
>
> yes I have. It grows ob-sql by approximately 300 lines.
> There are rooms for improvements.
> https://github.com/flintforge/ob-sql-session/blob/main/ob-sql.el
>
> Current status :
> - added support of :sessions to ob-sql.el, by merging ob-sql-session.el
> - restored support for sqlite (even if there is ob-sqlite.el)
> - added composite types for options
> - moved options from org-babel to a new group org-babel-sql
>
> TODO :
> - use `org-babel-prep-session'
> - verify headers and columns options
> - check use and storage of :dbconnection
> - minimize writings of temporary files
> - Tests

Thanks for the update!
I will be looking towards your improvements to be merged upstream.
Feel free to ask me anything if you need any help.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: patch : ob-sql :session

[Phil Estival]

* [2024-11-09 15:43] Ihor Radchenko:
> Phil Estival <pe@7d.nz> writes:
> 
>>> Have you considered integrating it into ob-sql itself?
>>> If not, what are the obstacles?
>>>
>>
>> Hello Ihor,
>>
>> yes I have. It grows ob-sql by approximately 300 lines.
>> There are rooms for improvements.
>> https://github.com/flintforge/ob-sql-session/blob/main/ob-sql.el
>>
> Thanks for the update!
> I will be looking towards your improvements to be merged upstream.
> Feel free to ask me anything if you need any help.
> 
I will keep running ob-sql with :session in the next days.
This work configuration should help to detect if there is an anomaly.

Shall I submit a patch?

Since many lines are changed due to proper indentation, docstrings,
untabification and a few lateral improvements, I was wondering if
such a patch has to be gradual and minimize sudden changes.

Also I'm trying to run the tests with

: make BTEST_RE='ob-sql.*' test-dirty

but they get skipped due to a missing dependency,
and I don't know in which direction to look for from there.
Below is what the command returns. Any idea ?

27 files scanned, 13 files contains IDs, and 54 IDs found.
0 source block added to Library of Babel
Local variables list is not properly terminated
selected tests: ob-sql.*
Running 2 tests (2024-11-11 11:17:40+0100, selector ‘"ob-sql.*"’)
   skipped  1/2  *org-missing-dependency*/test-ob-sql (0.000139 sec)
   skipped  2/2  *org-missing-dependency*/test-ob-sqlite (0.000098 sec)

Ran 2 tests, 0 results as expected, 0 unexpected, 2 skipped (2024-11-11 
11:17:40+0100, 0.000470 sec) ?

--
Phil Estival

Re: patch : ob-sql :session

[Ihor Radchenko]

Phil Estival <pe@7d.nz> writes:

> I will keep running ob-sql with :session in the next days.
> This work configuration should help to detect if there is an anomaly.
>
> Shall I submit a patch?

Sure!

> Since many lines are changed due to proper indentation, docstrings,
> untabification and a few lateral improvements, I was wondering if
> such a patch has to be gradual and minimize sudden changes.

If you can make a series of patches introducing individual changes, it
would be best. A single patch will also be ok.

> Also I'm trying to run the tests with
>
> : make BTEST_RE='ob-sql.*' test-dirty
>
> but they get skipped due to a missing dependency,
> and I don't know in which direction to look for from there.
> Below is what the command returns. Any idea ?

Check your local.mk
ob-sql tests are disabled by default. You need to modify BTEST_OB_LANGUAGES

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>