The German-born star, who was raised in Ireland, scooped the prize for his role as sex addict Brandon in Turner Prize-winning film-maker Steve McQueen's latest release.
Oscar-winning British director Andrea Arnold's presentation of the Emily Bronte classic Wuthering Heights was also recognised, with Robbie Ryan winning the gong for best cinematography.
But it was a disappointing night for the much-fancied Tinker Tailor Soldier Spy as the re-make, inspired by John le Carre's novel, failed to pick up any awards.
Film4, which backed both Shame and Wuthering Heights, said the films were "outstanding quality".
Tessa Ross, controller of film and drama at Channel 4, said: "It is absolutely wonderful to see Michael and Robbie's talent recognised in this way - both have pushed their craft to the limits of excellence to produce work of truly outstanding quality."
Meanwhile the Golden Lion prize for best film went to Russian film Faust, directed by Aleksander Sokurov, and Deanie Yip took the best actress award for her role in Chinese film Tao Jie (A Simple Life).
Suggested For You
SUBSCRIBE AND FOLLOW
Get top stories and blog posts emailed to me each day. Newsletters may offer personalized content or advertisements.Learn more