Side channel cache: Prime & Probe attack

I have a hard time completely understanding the Prime & Probe attack:
My current understanding is:

  1. Boot phase: The attacker occupies all cache sets containing attacker data.

  2. Probe phase: The attacker measures the access time to determine which dataset was accessed by the victim.

But the set is only part of the address. How can an attacker deduce which address was accessed? I know it has something to do with Spatiotemporal locality, but I can not understand exactly how an attacker can abuse it to get the full address a victim has accessed?