Is caffeine bad for people with anxiety disorder?

I keep hearing this. I have this kind of disorder but I drink a LOT of coffee and on top of that take caffeine pills sometimes. Is ti making my anxiety worse you think? Also, I have trouble sleeping at night so I get around 4-5 hours average a night.