You should try following things, I guarantee that using even a few will help to speed up your WordPress site:
- Use an effective caching plugin like W3 Total Cache
- Use a Conent Delivery Network [CDN]
- Optimize images, there is one plugin called WP-SmushIt
- Optimize your database, there is one plugin called WP-Optimize
- Try to reduce number of plugins.
- And last but not the list, keep your Wordpress installation updated.