"Why do people keep complaining about my privilege?"
Because if you have some sort of privilege (most people do, even those who are marginalized on some axis!), and you don't recognize it, you are a danger to marginalized folks.
From their perspective, you are a "bull in a china shop"; you're going to say, do, or assume things that end up harming people, and you will not show accountability for it if you don't even recognize that you're doing it!
This doesn't mean you're expected to be perfect. It just means you're expected to recognize your imperfections and relative advantages, and live your life in a way that minimizes their harm.