>  Since this slowness is caused by vc-mode, I wonder if it's possible
  >for Org to open the agenda files without enabling vc-mode, for vc-mode
  >is not necessary for org-mode.

I suspect part of the problem is that vc-mode is slower than it should
be.  Probably almost all of this time is 'git status foo.org' to find
out if the buffer gets a * or not.  Perhaps a variable to vc-mode to
omit that part would help, but I don't really know.