I agree, after reading the 2nd post, that you should have called him out.
I definitely think it depends on the size/area. I've been on both sides of this several times. On an elk hunt in NW Oregon we parked behind another vehicle at the trailhead. The 2 guys in front were right at the gate, and they were ready to fight to keep us out. It was a very large area, they told us the water hole they were hunting, and we wanted to go down lower to a clearcut. I was furious, but we ended up moving to keep from having a physical confrontation.
I've also driven past hunting places in SE KS because I knew that 2 guys could not hunt the area effectively. I went on to find something else.
Last fall I was the first one to a good spot for whitetail hunting (I've had good success out of a particular tree over the years). I packed my treestand in and was nearly finished putting it up in the dark when a guy came walking up the trail with a headlamp. He proceeded to walk to a tree 10 yards from where I was without ever seeing me! Come to find out he had a climber stand already on that tree and had been hunting there a few days. I had not seen his stand. We had come in from different directions and he never saw my vehicle. I pulled my stand and moved back a ways, but it ended up ruining my hunt for the morning.
It's extremely frustrating when you put a plan together and someone else has the same idea. I think you just have to look at every situation and discuss it like a mature adult. Nobody likes to be in that situation.