Ah, thanks! So it was configuration then...
I initially figured it was language-dependant, which was why I tried it in Ruby, since "$" is a valid character in that language... What I didn't anticipate was that the boxes came from intelligent-syntax checking--I figured that it was just some sort of fancy character I was unfamiliar with that needed to be escaped.
So, the way I fixed the issue in my Org file was to add the following to override the syntax error checking:
#+LATEX_HEADER: \expandafter\def\csname PY@tok@err\endcsname{}
It was recommended on that Stack Exchange answer not to do this, but we aren't writing code directly in LaTeX, we're using Org mode..! That means we don't really need error highlighting in our exporting, since that sort of thing is already covered by existing language support.