This decision depends on water temperature, air temperature, your paddling skill, your experience on and in cold water, water and weather conditions (wind strength, direction), proximity to shore and who you are paddling with. Don't dress less carefully because you put any responsibility on others for your rescue (not a good mindset). Paddling with groups that have a strong “watch out for each other” ethic you should still be at least as prepared as you would be paddling solo. Otherwise you are putting unfair responsibility and risk on your fellow paddlers and at the very least reducing their freedom to enjoy their own outing. And If you are responsible for other paddlers, particularly less experienced ones then you may even need to dress and prepare more carefully for more potential water immersion in order to assist in someone else's rescue.