Walk-Ins Welcome?

I have to say it here because I can't say it on Facebook, even though I would really like to. Ok, does anyone else find it odd that my brother-in-law, Adam's oldest brother, just walks into our house without knocking? I mean, I understand that we're living at his grandma's house and that he has never had to knock before but we're living here now and I find it really rude when he just walks in. One time he came over and it was just me at home, deathly sick on the couch and my mom had taken the kids for the day. I thought it was Adam but when I looked up it was my BIL looking down at me with his little daughter and it was really uncomfortable. I mean, I was sick and had he knocked, I would not have even answered the door. Instead, I was forced to make small talk with him for a good 20-30 minutes while he hung around for awhile. I honestly would not mind at all and don't mind when women-folk in the family stop by and they don't knock but it feels totally different when a male member of the family comes barging in unannounced. I mean, we go to the pool a lot and maybe I would prefer to be a little more covered up if company is coming over or whatever. Not that I walk around the house naked or anything but I suppose it's my right to do so. He came over the other day without knocking and was surprised to find Adam home, like he was just going to hang out with me and the kids, which is fine but I would prefer a knock at the door. Or am I being too uptight?