We had a fantastic time in Sri Lanka. The only downside was that it took for ever to get anywhere (approx 2hrs to travel 60km). If we had known this beforehand we might have trimmed back some of the culture visits - in particular visiting the ruins of the ancient capital which took up most of the day. We stayed in some beautiful hotels but were only in them from 6-7pm onwards. However, the locals were extremely friendly and overall we had a fantastic time.
The driver was brilliant, always cheerful and with some local fact or anecdote at regular intervals. He was very punctual, happy to fit in with us and suggested some excellent alternatives when time/weather required it
Perhaps being more realistic with journey times in the so-called "cultural triangle" - we were spending up to 6hrs a day in the minibus
Be realistic in what is achievable each day unless you don't mind spending hours each day plodding along the local roads
Dinner at the Amangalla Hotel, Galle and our 4 days on the beach at South Point Villa