As a fairly new Atheist I have a lot of religious stuff around the house that I want to get rid of?

Contact the Mormon Church, they'll take it. No on the other hand, maybe you shouldn't. They'll just try to convert you and they are freakin annoying and won't give up. Just did a hole and bury it in the backyard. Then plant a nice tree on top of it with some canna lillies. White of course.