I don't know exactly which side the rules are on in this situation. But I can answer for myself and give my opinion.
So, we are all here to have fun. The goal is to finish maps, and try to have a good time doing so. Just because you all finished, doesn't invalidate his existence. He is also trying to finish a map, maybe even getting a REALLY good time. You aldo said that it was a short main map and he'd already gotten halfways.
Do you REALLY not have the patience to wait that extra 1-2 minutes to let him finish his run?
And just because you are the majority, I don't think the server belongs to you. You need to respect others.
Imagine you are in a group of 7, you are in a team and you have gotten halfways on a map. There is also this HUGE group in team 0 that had just finished the map. They could be 20 or more. Now the big group decides to vote for a new map. Obviously the big team 0 group will have the majority of the votes. So your team of 7 has no power. To me this scenario is the same as the one you described
Also you said that you didn't know he was far before you made the vote, if there are 13 of you and only 1 of him and NO ONE cheks if that one other person has gotten far. You are not being considerate of others at all. It' all about respect, consideration and not being egoistic. (edited)
1