Hi Ihor,
The hook that is going wrong is pdf-tools-enable-minor-modes.
After digging into it, there seems to be a whole bunch of problematic minor modes. My methodology wasn't super rigorous, but what I found was
> pdf-misc-context-menu-minor-mode
> pdf-misc-size-indication-minor-mode
> pdf-misc-menu-bar-minor-mode
> pdf-misc-minor-mode
cause problems every time the pdf is loaded, and
> pdf-links-minor-mode
> pdf-outline-minor-mode
> pdf-annot-minor-mode
cause problems every other time the pdf is loaded.
I found a couple of workarounds: adding (org-reload) to my init.el (actually config.el as I'm running doom emacs) solves the problem, though it does add another second to my load time. Alternatively, I was able to load the first set of minor modes without issue by using doom's (after! pdf-tools ...). This didn't seem to work for the latter minor modes though.
Cheers,
Sam