Primary examples would be adding a note (CTRL-z) or changing a tag on a person and then having org-agenda update that. I am assuming it is because the entire file needs to be parsed rather than say, some index of entries.
(so perhaps I mischaracterized org-contacts as being slow versus its interaction with other programs.)
(for search I use swiper which is very efficient for searching the file whenI need it.).
tho quite interested in seeing what perf enhancements you've done on large org files would be interesting.
Daryl.
PS> As an outside feature though, interoperability of the org-contact formats with other operating system address books, most notable gnome contacts/evolution, goog contacts, and OSX address book would be high on my list in terms of improving org-contacts though. (eg, raw, structued info in all address books, and say perhaps notes or similar maintained and synced in ome manner.