I don't worry about it. I find hunters where I hunt but I know where the deer are. Most of these people don't have a clue. I went after a 3x4 after 5 quads went by him on the mountain. I knew where he was but he ended up getting spooked. I also scout and most people don't so scouting key. Also not all of us archery guys are going to take a buck if any are like me.
Ya you've had a tough one but you have the drive. You will get the chance and get the bomber your looking for. Keep up the hard work.