I'd say leave the badges. There's not that much badging to begin with. It's not really unsightly and doesn't attract that much attention.
I'd venture to say that most people don't really notice the badges on cars unless they are specifically looking to identify the model and/or trim level (which is seldom). Conversely, I don't think most people notice or care whether the car "looks clean" without the badges either.