By having a well defined core API you have at least one part of your system (hopefully the important bit) that is well documented, well considered and well written. It's something akin to maintaining a tidy formal lounge whilst the rest of your house is being subjected to an ill-considered conga line.
When it eventually comes time to make your API public, it's already tested and known to work. You know what problems your users are likely to encounter, because you've already encountered them yourself. Best of all, you're not trying to shoehorn a heap of public points of access where they were never intended. Writing an API as you go means you solve a lot of problems before they ever happen...and all without having to think about it too much. The API is as flexible as you want it to be until the day you make it public.
The Angry Monkeys talk about an interesting way to solve two common problems in software development: documentation and testing. He advocates writing an API for accessing your code and following it, even internally. By structuring your code with an API, you can maintain some semblance of order by adhering to the same contracts you would expect anyone else's code to follow.
Developing that way from Day One allows you to not take stupid shortcuts in the name of expediency since you'll be violating that API. Preferably your API will be fluent and easily readable. Also by using it yourself, you've worked out the code in the same way the maintainers and extenders will.
I recently worked on a infrastructure application that employed an API since it will be used by our entire organization (hopefully). It made a lot of choices easier by restricting all interfaces through this API. When it came time to extend it, we had a starting point already since the API was a known quantity. Overall, it's good idea to develop with a target API in mind. Do it from the beginning to realize your biggest gains.