I hear you asking for a simple answer. The best I can give you:
Feel faster than your competition – 20%
All basic research on system perception and responsiveness remains valid (for example, the basic scale of 0.1 / 1/10 seconds).
Check out this great summary of perception of performance, with ideas that could guide you. Like optimization for "just noticeable difference" (jnd).
It makes no sense to optimize for a number (load speed of Google Analytics) or a measure (stay away from the ranking versus the speed of a page). These are specific development tools that have virtually no connection to the big picture of user perceived speed.
If you are not able to put more gear in (and it will come), just remember the first heurestic of usability:
Visibility of the state of the system
and just ask for patience.
PS: Kissmetrics infographics are catchy, but offer very little real data. And coincidentally, comes first in the search for Google images for "How fast should a website load?" 😉