Perhaps the other variables it binds should be buffer local as well?

This bug is new in Emacs 25.1, but changing with-silent-modifications is
a bit risky.  Therefore, I propose the following for the emacs-25
branch: