They're not the worst. I just hate the audience segment fillers in almost every sketch show, because they ruin some jokes by setting them up. "Any of you ever have one of those bosses that..." and all.
A good exception would be Mr. Show, for example, where whenever they did an audience bit it was with plants and they would parody something by doing it.