Re: Feedback on Emacs-Jupyter

[David Dynerman <emperordali@block-party.net>]

Dear Nathaniel,

Let me echo others’ comments by saying how much I appreciate your work on emacs-jupyter. I use the package daily in my work and it’s been really fantastic. Thank you very, very much for your huge efforts and time investment in creating and maintaining this package.

emacs-jupyter has proved so useful that I’ve been collecting a list of suggested improvements - I was actually intending on trying to contribute some of these myself, but I think it would be good to discuss first with you and other users of emacs-jupyter (is there an emacs-jupyter mailing list or preferred venue?). For now I’ll just share what I think would be the biggest improvement for my use case.

I use emacs-jupyter to connect from my laptop to remote jupyter kernels and notebook servers running on powerful machines provided by my employer. I work in a notebook locally on my laptop, but execute code blocks remotely. I specify a single remote jupyter notebook server+kernel to connect to in a #+PROPERTY line at the top of my org file which globally sets header-args for jupyter-python to include the correct :session argument.

For me, the biggest improvements would be addressing some pain points with disconnecting/reconnecting to the jupyter notebook server or kernel. I don’t have technical suggestions yet on how these could be addressed, so I’ll just describe the user experience side of things.

1) Reconnecting to the notebook server after disconnecting. Commonly I’ll work in my notebook, then take a break and close my laptop or disconnect from my work VPN. When I re-open my laptop/reconnect to VPN it seems like the connections to the remote notebook server are in a bad state (this may be a tramp thing, also). Currently my work around is to kill the associated emacs jupyter REPL buffer, then navigate to one of the source blocks in the notebook and run org-babel-pop-to-session which has the side-effect of re-connecting to the jupyter notebook server and recreating the jupyter REPL buffer. 

The main downside to this workaround is that the scroll back history of the REPL buffer is lost, since the buffer was killed. It would be great to be able to re-establish the connection to the REPL without discarding the buffer.

2) Getting results from a running block after disconnecting/reconnecting. Currently it’s difficult to manage long-running code blocks using emacs-jupyter if I disconnect from the remote server while the code block is running. The work around in #1 above to re-connect to a remote kernel doesn’t work if the kernel is still executing a block. If the kernel has completed executing the block, then the results are not populated back into the notebook (the execution UUID populated when the code block began executing remains).

I don’t know much about jupyter, but I saw some jupyter logs about buffered messages. Perhaps jupyter buffers output when a remote client disconnects? If so, maybe this buffered output could be replayed in a notebook when the emacs jupyter REPL connection is re-established?

3) Similarly, improving feedback from a long-running block would be very helpful. Currently you can use print statements in a long running block to report progress, but these are lost after disconnecting (maybe they are buffered on the jupyter server side and could be replayed on reconnect?). Further, I don’t know if this is feasible, but it would be amazing to have support for progress bars for long-running tasks (e.g., have a tqdm progress bar render in the org notebook).

Thanks again for the wonderful package - I hope we can talk a bit more about some of these friction points. I’d be happy to give a crack at contributing a PR, if that would be welcome.

Thank you,
David

> On Jan 4, 2022, at 15:24, Nathaniel Nicandro <nathanielnicandro@gmail.com> wrote:
> 
> 
> Hello all,
> 
> I'm the maintainer of the emacs-jupyter project
> (https://github.com/nnicandro/emacs-jupyter) which essentially
> integrates Jupyter kernels (https://jupyter.org) with Org mode source
> blocks.
> 
> I wanted to make an introduction to the Org community.  So...hello!  And
> thanks for promoting the project on https://orgmode.org/features.html.
> 
> I believe a lot of users of the project use it mainly for the Org
> integration.  I thought it would be a good idea to get some feedback
> from the community on how their experience using emacs-jupyter has been.
> I'm getting back into active maintenance of the project and am looking
> for feedback to get a better idea of what the future of the project
> could look like.  What features of standard Org source blocks do you
> find Jupyter source blocks are lacking?  What potential features do you
> think would be useful for Jupyter source blocks to support, given the
> capabilities of Jupyter?  What would it mean to see Emacs-Jupyter and
> Org more integrated?  Of course, any other thoughts are welcome.
> 
> -- 
> Nathaniel
>