Google has recently announced that site speed is now one of the components in their Search Engine algorithm. This has been on the cards for a while, when they flagged it by including Site Performance in the labs section of Google Webmaster Tools. Matt Cutts of Google discussing page speed on his blog.
Page Speed has been concerning me for a while as my sites on a shared server environment started getting slower and slower. I have just upgraded to a cloud hosting environment with VPS.NET and the difference in the speed of my WordPress sites has improved massively.
If you want to see how quickly your site loads try: http://www.webpagetest.org/
Finally, Matt Cutts himself interviewed by WebPro News.