org-mode 7.8.03
emacs 23.3, 23.4, 24.0.94

steps to recreate:

emacs -q

copy the following code and paste it into the *scratch* buffer:
(add-to-list 'auto-mode-alist '("\\.txt\\'" . org-mode))
(add-hook 'org-mode-hook '(lambda () (text-scale-set 4)))

M-x eval-buffer

then C-x C-f to open a .txt file, and then C-x C-f to open another .txt
file.
Every time a .txt is opened, the font size become larger and larger.
However, the indicator in the mode line remains +4.