Our AvatarBuilder caches were cleared in their entirety any time a
profile changes. If a profile update raced with a avatar fetch in
ConversationAvatarView, they'd blink briefly while the cached items were
removed.
This change ensures that we're only clearing out avatars that are
associated with an updated profile.
