I am assuming that you know this but you can attack someone and not take over there planet, so you don't really need to wait for your culture to get to 4 just to attack but if you know the guy has been working then you are better off to take over his planet as you will get all his stuff.
I guess what I am saying is that you could attack him if you know you will win and then just wait till your culture is higher to take over the planet. I think that would work.
PS sorry for the long message my brain does not thing right when I am at work
