Ah, yes...
I see this is becoming more of a common occurance. The last game I was at (due to time constraints) was the 2005 region semi-final at LA where the entire JC student section (30+ people, myself included, but I graduated a year before) was thrown out of the gym. I'm sure there are those familiar with the incident. For those that aren't, we had been doing the typical student section thing when someone called a timeout with less than two minutes in the game. The ref turned around and stared down the crowd, which I believe is a major no-no. One student said something they should not have (and yes, it was uncalled for) so the ref kicked out the entire section. Mind you it was probably in the 30's to 40's outside, and instead of being allowed to stand in the commons area/lunchroom/whatever you call it, we were escorted outside the school and were not allowed to go back in for anything. Mind you there were probably kids whose parents were inside, and I would have been INFURIATED if I were a parent. I'm sure TSSAA got many many complaints about it (I know I filed one). I have heard the ref in particular was removed, but I do not know for sure.
Basically, referees are paid to ref a game, not to be the police of the gym.