Ok, so you've had quite a bit of constructive advice here, but I've been stuck in a room with no windows learning about this sort of thing for the last couple of months. I had to suffer, now I'm going to bore you with it too.
Part of your 'problem' is that you have a pretty new house.
In old houses there were enough leaks that the warm moist air would rise to the top of the building and leak out, probably not doing the fabric of the building or whatever insulation there was much good in the process. Cold, dry air would then be sucked in (through a wall cavity containing who knows what in the way of formaldehyde loaded insulation etc.) to replace what leaked out the top, this is called the stack effect. In an older home you may not even notice any condensation issues because the air leaks are ventilating it for you (and adding to your heating bill).
Newer houses are much more airtight than they used to be, so this stack effect is reduced or eliminated and all that humid air is stuck in there. Enter the HRV...
Where I live you aren't allowed to build a house without one. Unlike an exhaust fan, a HRV brings air in in balance with what it exhausts and recovers some heat from the exhaust air to warm the incoming air (which is very useful in a Manitoba winter). The HRV wins over the exhaust fan in two ways. First that it saves energy. Second, in a newer tighter home there are not (hopefully) enough leaks to allow new air to come in and replace what the exhaust fan is removing, so the house ends up at a negative pressure, and the flow through the fan and therefore its effectiveness decreases - the HRV is bringing air in as well as exhausting, so this isn't an issue.
I know an HRV isn't cheap, but you may find it paying for itself in heating costs, especially if you live in an area where hydro/gas is expensive.
As swill noted these mould issues may well manifest themselves in the upper levels of a house since that's where the moisture will be taken by the rising hot air, and also the windows since the R value of a window is pretty low compared to a well insulated wall. An upstairs bathroom is a prime candidate, which is why our HRV takes air from all the bathrooms one of which is upstairs and has a switch so it can be used instead of a bathroom fan. It copes pretty well with the 5 gallons a well that my tank pushes into the air, to the point I only really run it intermittently.
I have also seen them just attached in the cool air return duct of the forced air system (exhaust air upstream incoming air) and wired to the furnace blower so that it operates when the HRV is running.
Hope that's of some help.