Its relatively confusing.

When theres someone around that you generally dislike, you wish for that day to end quickly; swiftly.

And when that same person isn't there, you begin to see what they truly bring to the table, despite you disliking him/her, either only relatively or with a passion.