There are few things which can be done in php configuration to help speed up Magento site.

Realpath Cache:

A lot of file I/O is done to run Magento site. This file I/O can be improved with faster hard disk. There is more can be done to improve this. PHP configuration allows to configure size of memory which can be used to cache realpaths, which help improve file I/O. The configuration is “realpath_cache_size”. Increase its value give more cache space.
Also increase the value for “realpath_cache_ttl” to 3600 to 7200 to increase the time of cache.

Memory Limit

Also its good idea to increase page memory limit to be used by PHP. You can increase it by increasing value of “memory_limit” configuration.

Remove unused extentions

See if you disable any unused php extension to save some more memory. There few extension which are used by Magento in default installations. These include PDO_MYSQL, simplexml, mcrypt, hash, gd, dom, iconv2, SOAP (list may increase in case of more Magento extensions and features.). If server is running other site please check required extensions for those as well.

More on Magento Performance

Disclaimer: Above recommendations are extracted based on “Magento Enterprise Edition Whitepaper” and my prior knowledge so no liabilities are accepted. Be sure the benchmark and adjust values according to your environment.