Ah, now the problem appears. It says "No such language mode: mathematica-mode".

So there is a mismatch between the name mathematica (which I used as name in org-babel extension) and the name mma-mode (which is provided by mma.el). 

However, as auto-mode-alist suggests (where there is ("\\.m\\'" . mma-mode), and no entry with mathematica-mode), I still have no idea why org-edit-special looks for mathematica-mode instead of mma-mode.

On Thu, Jan 9, 2014 at 11:15 AM, Bastien <bzg@altern.org> wrote:

Yi Wang <tririverwangyi@gmail.com> writes:

> (1) If I open a .m file, there is font lock. Thus the problem
> shouldn't be binding between mma.el and .m files.
> (2) If I insert a src block of C or python code in org mode, there is
> font lock. Thus the problem shouldn't be org-src-fontify-natively
> (and I checked it is indeed t).

What happen if you C-c ' on a .m block?

--
 Bastien