God vs. The Nephilim - The Curse of Canaan

Puzzled as to why Noah curses Canaan, Hams son and not Ham himself? Or for that matter any of Hams other sons. Why specifically Ham's son?