I don't think the surf--mine is the Maine coast, with smaller waves than yours--is any more dangerous than the rivers I wade all the time. Given a choice between a swim in the surf and one in a Class 4 or 5 rapid, I'm not sure which I'd prefer.
I've spent a ton of time in canoes and drift boats with my waders on during float and wade trips.
The key is to be smart about it. Wear a wading belt and keep it tight. Wear a lifejacket when wading or paddling in any kind of risky spot. Practice a wet exit with big clunky wading boots on. (As a guy with size 15 feet who paddles a decked kayak, this is not academic to me.

Don't stack your gear in the cockpit around you where it will impede your ability to get out of the boat. Don't wear wader that don't let you move well.
It's not waders that are dangerous; it's the surf, or the river, or the cold.