I don't think so. I just restarted emacs and compiled the agenda about 15 times in a row and it was fast. But - if I leave it to do something else for a while, often the next time I compile agenda it is slow, even if I haven't changed the org mode agenda files in the meantime.
Does it slow down depending on the number of times you run agenda?