Culture does move slow but once you get a new planet and you start upgrading it goes not bad. Once you get 4 or 5 planets and people start attacking every day you won't worry about your culture.
If you can go in with extra ships to attack planets you get more xp's and you never know how many ships he has circling his planet. I know I always have some in the sky all the time.