Passing table to Ruby session

Passing table to Ruby session

[Mike Gauland]

I'm trying to pass a table to a ruby code block. The table is 211 lines, 
of four columns.

I want to pass it to a session, so I can use the data in subsequent 
blocks.  I've done this successfully on my work laptop, running Windows 
emacs (28.2), with ruby 3.2.

On my linux box (NixOS), however, the code block never completes.  If I 
cut the table down to 65 lines or so, it works just fine.

If I sent the full table to a stand-alone block (no :session), the block 
executes successfully.

Both machines are using ob-ruby.el version 2.4.4 and inf-ruby.el 2.7.0.

If anyone has any thoughts on what might be going on, I'd love to hear 
them. I'm not familiar enough with the code (not yet, anyway) to know 
where to start.

I'll work on putting together a minimal example, if anyone is interested 
in testing it on their system.

Kind regards,

Mike

Re: Passing table to Ruby session

[Ihor Radchenko]

Mike Gauland <mikelygee@gmail.com> writes:

> I'll work on putting together a minimal example, if anyone is interested 
> in testing it on their system.

A minimal example would certainly help.
See https://orgmode.org/manual/Feedback.html#Feedback

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Passing table to Ruby session

[Mike Gauland]

On 9/08/23 22:14, Ihor Radchenko wrote:
> Mike Gauland <mikelygee@gmail.com> writes:
>
>> I'll work on putting together a minimal example, if anyone is interested
>> in testing it on their system.
> A minimal example would certainly help.
> See https://orgmode.org/manual/Feedback.html#Feedback
>

Here is an example that demonstrates the problem on my system. I've run 
with my init files disabled, and that makes no difference. Also, I've 
taken the data as it is sent to the session, and pasted into irb in  a 
terminal window, and it ran just fine (which I think rules out any 
problems with line length, or with the session code generating a bad 
script for the longer variable).  I'm keen to find out if anyone else 
can reproduce this, or if there's something quirky about my system.

# Without a session, this works even when the table isn't shortened:
#
#+begin_src ruby :var raw_data=raw-data :colnames no :exports code 
:results output
   puts(raw_data.count)
#+end_src

#+RESULTS:
: 75

# The next two blocks *only* work when the table is cut off at 73. If the
# last two lines are added back in to the table, the ruby session does not
# return.
# Also, even with a truncated table, I don't get any output without the
# extra call to puts() to print a blank line.
#
#+begin_src ruby :var raw_data=raw-data :session :colnames no :exports 
code :results output
   puts(raw_data.count)
   puts()
#+end_src

#+RESULTS:
: 75


#+begin_src ruby :var raw_data=raw-data :session :colnames no :exports 
code :results value
   raw_data.count
#+end_src

#+RESULTS:
: 75


#+NAME: raw-data
| Column 1 | Column 2 | Notes |
|----------+----------+-------------------------------------------|
|        0 |        0 | This is note #0. It just takes up space. |
|        1 |        1 | This is note #1. It just takes up space. |
|        2 |        2 | This is note #2. It just takes up space. |
|        3 |        3 | This is note #3. It just takes up space. |
|        4 |        4 | This is note #4. It just takes up space. |
|        5 |        5 | This is note #5. It just takes up space. |
|        6 |        6 | This is note #6. It just takes up space. |
|        7 |        7 | This is note #7. It just takes up space. |
|        8 |        8 | This is note #8. It just takes up space. |
|        9 |        9 | This is note #9. It just takes up space. |
|       10 |       10 | This is note #10. It just takes up space. |
|       11 |       11 | This is note #11. It just takes up space. |
|       12 |       12 | This is note #12. It just takes up space. |
|       13 |       13 | This is note #13. It just takes up space. |
|       14 |       14 | This is note #14. It just takes up space. |
|       15 |       15 | This is note #15. It just takes up space. |
|       16 |       16 | This is note #16. It just takes up space. |
|       17 |       17 | This is note #17. It just takes up space. |
|       18 |       18 | This is note #18. It just takes up space. |
|       19 |       19 | This is note #19. It just takes up space. |
|       20 |       20 | This is note #20. It just takes up space. |
|       21 |       21 | This is note #21. It just takes up space. |
|       22 |       22 | This is note #22. It just takes up space. |
|       23 |       23 | This is note #23. It just takes up space. |
|       24 |       24 | This is note #24. It just takes up space. |
|       25 |       25 | This is note #25. It just takes up space. |
|       26 |       26 | This is note #26. It just takes up space. |
|       27 |       27 | This is note #27. It just takes up space. |
|       28 |       28 | This is note #28. It just takes up space. |
|       29 |       29 | This is note #29. It just takes up space. |
|       30 |       30 | This is note #30. It just takes up space. |
|       31 |       31 | This is note #31. It just takes up space. |
|       32 |       32 | This is note #32. It just takes up space. |
|       33 |       33 | This is note #33. It just takes up space. |
|       34 |       34 | This is note #34. It just takes up space. |
|       35 |       35 | This is note #35. It just takes up space. |
|       36 |       36 | This is note #36. It just takes up space. |
|       37 |       37 | This is note #37. It just takes up space. |
|       38 |       38 | This is note #38. It just takes up space. |
|       39 |       39 | This is note #39. It just takes up space. |
|       40 |       40 | This is note #40. It just takes up space. |
|       41 |       41 | This is note #41. It just takes up space. |
|       42 |       42 | This is note #42. It just takes up space. |
|       43 |       43 | This is note #43. It just takes up space. |
|       44 |       44 | This is note #44. It just takes up space. |
|       45 |       45 | This is note #45. It just takes up space. |
|       46 |       46 | This is note #46. It just takes up space. |
|       47 |       47 | This is note #47. It just takes up space. |
|       48 |       48 | This is note #48. It just takes up space. |
|       49 |       49 | This is note #49. It just takes up space. |
|       50 |       50 | This is note #50. It just takes up space. |
|       51 |       51 | This is note #51. It just takes up space. |
|       52 |       52 | This is note #52. It just takes up space. |
|       53 |       53 | This is note #53. It just takes up space. |
|       54 |       54 | This is note #54. It just takes up space. |
|       55 |       55 | This is note #55. It just takes up space. |
|       56 |       56 | This is note #56. It just takes up space. |
|       57 |       57 | This is note #57. It just takes up space. |
|       58 |       58 | This is note #58. It just takes up space. |
|       59 |       59 | This is note #59. It just takes up space. |
|       60 |       60 | This is note #60. It just takes up space. |
|       61 |       61 | This is note #61. It just takes up space. |
|       62 |       62 | This is note #62. It just takes up space. |
|       63 |       63 | This is note #63. It just takes up space. |
|       64 |       64 | This is note #64. It just takes up space. |
|       65 |       65 | This is note #65. It just takes up space. |
|       66 |       66 | This is note #66. It just takes up space. |
|       67 |       67 | This is note #67. It just takes up space. |
|       68 |       68 | This is note #68. It just takes up space. |
|       69 |       69 | This is note #69. It just takes up space. |
|       70 |       70 | This is note #70. It just takes up space. |
|       71 |       71 | This is note #71. It just takes up space. |
|       72 |       72 | This is note #72. It just takes up space. |
|       73 |       73 | This is note #73. It just takes up space. |

|       74 |       74 | This is note #74. It just takes up space. |
|       75 |       75 | This is note #75. It just takes up space. |

Re: Passing table to Ruby session

[Mike Gauland]

Just to follow up on my problem sending a table to ruby. It doesn't' 
seem to be an org problem, but with inf-ruby.

I copied the ruby code that org generates to set the variable, and 
pasted it in to an inf-ruby session, and that works fine.

However, when I select it as a region, and do "ruby-send-region", 
inf-ruby complains about an "unexpected end-of-input".

As long as the code to set the variable is under 4k, it works. When it's 
longer than that,  process-send-region sends it in pieces, and inf-ruby 
is apparently prematurely interpreting the incomplete code.

I'll take this up elsewhere, though if anyone here has suggestions for 
me, or wants further updates, let me know.

Kind regards,

Mike

Re: Passing table to Ruby session

[Mike Gauland]

On 10/08/23 09:24, Mike Gauland wrote:
> Just to follow up on my problem sending a table to ruby. It doesn't' 
> seem to be an org problem, but with inf-ruby.
>
> I copied the ruby code that org generates to set the variable, and 
> pasted it in to an inf-ruby session, and that works fine.
>
> However, when I select it as a region, and do "ruby-send-region", 
> inf-ruby complains about an "unexpected end-of-input".
>
> As long as the code to set the variable is under 4k, it works. When 
> it's longer than that,  process-send-region sends it in pieces, and 
> inf-ruby is apparently prematurely interpreting the incomplete code.
>
> I'll take this up elsewhere, though if anyone here has suggestions for 
> me, or wants further updates, let me know.
>
> Kind regards,
>
> Mike
>
Just a final note in case any one else has runs into the same problem. I 
got the same results with jruby, but "pry" handles my file without 
complaint.

Re: Passing table to Ruby session

[Ihor Radchenko]

Mike Gauland <mikelygee@gmail.com> writes:

> Just to follow up on my problem sending a table to ruby. It doesn't' 
> seem to be an org problem, but with inf-ruby.
> ...
> As long as the code to set the variable is under 4k, it works. When it's 
> longer than that,  process-send-region sends it in pieces, and inf-ruby 
> is apparently prematurely interpreting the incomplete code.
>
> I'll take this up elsewhere, though if anyone here has suggestions for 
> me, or wants further updates, let me know.

4k sounds like a known quirk with pty connection type in Emacs
processes. It may have this 4k limitation at least on some systems.

If you posted a bug report elsewhere, may you please share the link to
that report?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Passing table to Ruby session

[Mike Gauland]

On 10/08/23 21:11, Ihor Radchenko wrote:
> Mike Gauland <mikelygee@gmail.com> writes:
>
>> Just to follow up on my problem sending a table to ruby. It doesn't'
>> seem to be an org problem, but with inf-ruby.
>> ...
>> As long as the code to set the variable is under 4k, it works. When it's
>> longer than that,  process-send-region sends it in pieces, and inf-ruby
>> is apparently prematurely interpreting the incomplete code.
>>
>> I'll take this up elsewhere, though if anyone here has suggestions for
>> me, or wants further updates, let me know.
> 4k sounds like a known quirk with pty connection type in Emacs
> processes. It may have this 4k limitation at least on some systems.
>
> If you posted a bug report elsewhere, may you please share the link to
> that report?
>
I've reported it to the inf-ruby project: 
https://github.com/nonsequitur/inf-ruby/issues/172

I wasn't aware that Emacs would insert EOF in a stream, but that doesn't 
seem to be the problem (if it is even occurring here).

I've found that both irb and pry will exit if I type ^D at the start of 
a line, but anywhere else it seems to be ignored.

Re: Passing table to Ruby session

[Ihor Radchenko]

Mike Gauland <mikelygee@gmail.com> writes:

> I've reported it to the inf-ruby project: 
> https://github.com/nonsequitur/inf-ruby/issues/172
>
> I wasn't aware that Emacs would insert EOF in a stream, but that doesn't 
> seem to be the problem (if it is even occurring here).

AFAIK, it has nothing to do with Emacs itself. Rather Linux PTY quirks.
See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=6149

I posted the same link in the github issue.

On Org side, the best we might do is splitting the long command into
multiline (if ruby REPL supports line continuation like \
this or similar). Of course, it will be a workaround.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Passing table to Ruby session

[Mike Gauland]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

On 11/08/23 19:47, Ihor Radchenko wrote:
> On Org side, the best we might do is splitting the long command into 
> multiline (if ruby REPL supports line continuation like \ this or 
> similar). Of course, it will be a workaround.

I've redefined org-babel-ruby-var-to-ruby in my init.el to add a newline 
after each element in an array, which I think will work for anything 
less than a 4k-long string. Ideally, the hard-coded \n would follow the 
EOL convention of the buffer, but I haven't gotten that far yet.


(defun org-babel-ruby-var-to-ruby (var)
   "Convert VAR into a ruby variable.
Convert an elisp value into a string of ruby source code
specifying a variable of the same value."
   (if (listp var)
       (concat "[" (mapconcat #'org-babel-ruby-var-to-ruby var ", \n") "]")
     (if (eq var 'hline)
     org-babel-ruby-hline-to
       (format "%S" var))))

[-- Attachment #2: Type: text/html, Size: 1574 bytes --]

Re: Passing table to Ruby session

[Ihor Radchenko]

Mike Gauland <mikelygee@gmail.com> writes:

> I've redefined org-babel-ruby-var-to-ruby in my init.el to add a newline 
> after each element in an array, which I think will work for anything 
> less than a 4k-long string. Ideally, the hard-coded \n would follow the 
> EOL convention of the buffer, but I haven't gotten that far yet.
>
>
> (defun org-babel-ruby-var-to-ruby (var)
>    "Convert VAR into a ruby variable.
> Convert an elisp value into a string of ruby source code
> specifying a variable of the same value."
>    (if (listp var)
>        (concat "[" (mapconcat #'org-babel-ruby-var-to-ruby var ", \n") "]")

I think Emacs should handle it automatically.

Also, it looks like ruby sessions are broken now:

#+begin_src ruby :session org-test-ruby :results output
s = "1"
s = "2"
s = "3"
puts s
s = "4"
#+end_src

yields nothing.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Passing table to Ruby session

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> Also, it looks like ruby sessions are broken now:

At least, this one is fixed.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=430c72924

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Passing table to Ruby session

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 1369 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> On Org side, the best we might do is splitting the long command into
> multiline (if ruby REPL supports line continuation like \
> this or similar). Of course, it will be a workaround.

Fixed, on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=deb5ea0fc

There are still some issues with inf-ruby I cannot understand though.
When I tried to evaluate the src blocks multiple times, inf-ruby (AFAIU)
interprets some of the puts output as input for some reason, which is
bizarre and does not look like Org's fault.

May you please check if you can reproduce the same problem on your side?

Instructions:

1. Download the attached bug.org file
2. Clone the latest Org: git clone https://git.savannah.gnu.org/cgit/emacs/org-mode.git
3. cd org-mode
4. make repro REPRO_ARGS="-L /path/to/inf-ruby -l ob-ruby /path/to/bug.org"
5. Go to the second code block (the one with :session)
6. Evaluate it several times
7. At some point, you will see either empty result or an error.
   In the *ruby* buffer, you will see
   _org_babel_ruby_prompt (irb):481:in `<main>': undefined method `_org_babel_ruby_prompt' for main:Object (NoMethodError)
	from /usr/lib64/ruby/gems/3.1.0/gems/irb-1.4.1/exe/irb:11:in `<top (required)>'
	from /usr/bin/irb:25:in `load'
	from /usr/bin/irb:25:in `<main>'
_org_babel_ruby_prompt


[-- Attachment #2: bug.org --]
[-- Type: application/vnd.lotus-organizer, Size: 6246 bytes --]

# Without a session, this works even when the table isn't shortened:
#
#+begin_src ruby :var raw_data=raw-data :colnames no :exports code :results output
puts(raw_data.count)
#+end_src

#+RESULTS:
: 77


# The next two blocks *only* work when the table is cut off at 73. If the
# last two lines are added back in to the table, the ruby session does not
# return.
# Also, even with a truncated table, I don't get any output without the
# extra call to puts() to print a blank line.
#
#+begin_src ruby :var raw_data=raw-data :session :colnames no :exports code :results output
puts(raw_data.count)
puts()
#+end_src

#+RESULTS:
: 77


#+begin_src ruby :var raw_data=raw-data :session :colnames no :exports code :results value
raw_data.count
#+end_src

#+RESULTS:
: 77


  #+NAME: raw-data
  | Column 1 | Column 2 | Notes                                     |
  |----------+----------+-------------------------------------------|
  |        0 |        0 | This is note #0. It just takes up space.  |
  |        1 |        1 | This is note #1. It just takes up space.  |
  |        2 |        2 | This is note #2. It just takes up space.  |
  |        3 |        3 | This is note #3. It just takes up space.  |
  |        4 |        4 | This is note #4. It just takes up space.  |
  |        5 |        5 | This is note #5. It just takes up space.  |
  |        6 |        6 | This is note #6. It just takes up space.  |
  |        7 |        7 | This is note #7. It just takes up space.  |
  |        8 |        8 | This is note #8. It just takes up space.  |
  |        9 |        9 | This is note #9. It just takes up space.  |
  |       10 |       10 | This is note #10. It just takes up space. |
  |       11 |       11 | This is note #11. It just takes up space. |
  |       12 |       12 | This is note #12. It just takes up space. |
  |       13 |       13 | This is note #13. It just takes up space. |
  |       14 |       14 | This is note #14. It just takes up space. |
  |       15 |       15 | This is note #15. It just takes up space. |
  |       16 |       16 | This is note #16. It just takes up space. |
  |       17 |       17 | This is note #17. It just takes up space. |
  |       18 |       18 | This is note #18. It just takes up space. |
  |       19 |       19 | This is note #19. It just takes up space. |
  |       20 |       20 | This is note #20. It just takes up space. |
  |       21 |       21 | This is note #21. It just takes up space. |
  |       22 |       22 | This is note #22. It just takes up space. |
  |       23 |       23 | This is note #23. It just takes up space. |
  |       24 |       24 | This is note #24. It just takes up space. |
  |       25 |       25 | This is note #25. It just takes up space. |
  |       26 |       26 | This is note #26. It just takes up space. |
  |       27 |       27 | This is note #27. It just takes up space. |
  |       28 |       28 | This is note #28. It just takes up space. |
  |       29 |       29 | This is note #29. It just takes up space. |
  |       30 |       30 | This is note #30. It just takes up space. |
  |       31 |       31 | This is note #31. It just takes up space. |
  |       32 |       32 | This is note #32. It just takes up space. |
  |       33 |       33 | This is note #33. It just takes up space. |
  |       34 |       34 | This is note #34. It just takes up space. |
  |       35 |       35 | This is note #35. It just takes up space. |
  |       36 |       36 | This is note #36. It just takes up space. |
  |       37 |       37 | This is note #37. It just takes up space. |
  |       38 |       38 | This is note #38. It just takes up space. |
  |       39 |       39 | This is note #39. It just takes up space. |
  |       40 |       40 | This is note #40. It just takes up space. |
  |       41 |       41 | This is note #41. It just takes up space. |
  |       42 |       42 | This is note #42. It just takes up space. |
  |       43 |       43 | This is note #43. It just takes up space. |
  |       44 |       44 | This is note #44. It just takes up space. |
  |       45 |       45 | This is note #45. It just takes up space. |
  |       46 |       46 | This is note #46. It just takes up space. |
  |       47 |       47 | This is note #47. It just takes up space. |
  |       48 |       48 | This is note #48. It just takes up space. |
  |       49 |       49 | This is note #49. It just takes up space. |
  |       50 |       50 | This is note #50. It just takes up space. |
  |       51 |       51 | This is note #51. It just takes up space. |
  |       52 |       52 | This is note #52. It just takes up space. |
  |       53 |       53 | This is note #53. It just takes up space. |
  |       54 |       54 | This is note #54. It just takes up space. |
  |       55 |       55 | This is note #55. It just takes up space. |
  |       56 |       56 | This is note #56. It just takes up space. |
  |       57 |       57 | This is note #57. It just takes up space. |
  |       58 |       58 | This is note #58. It just takes up space. |
  |       59 |       59 | This is note #59. It just takes up space. |
  |       60 |       60 | This is note #60. It just takes up space. |
  |       61 |       61 | This is note #61. It just takes up space. |
  |       62 |       62 | This is note #62. It just takes up space. |
  |       63 |       63 | This is note #63. It just takes up space. |
  |       64 |       64 | This is note #64. It just takes up space. |
  |       65 |       65 | This is note #65. It just takes up space. |
  |       66 |       66 | This is note #66. It just takes up space. |
  |       67 |       67 | This is note #67. It just takes up space. |
  |       68 |       68 | This is note #68. It just takes up space. |
  |       69 |       69 | This is note #69. It just takes up space. |
  |       70 |       70 | This is note #70. It just takes up space. |
  |       71 |       71 | This is note #71. It just takes up space. |
  |       72 |       72 | This is note #72. It just takes up space. |
  |       73 |       73 | This is note #73. It just takes up space. |
  |       74 |       74 | This is note #74. It just takes up space. |
  |       75 |       75 | This is note #75. It just takes up space. |

[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>