Am I crazy for thinking this?

Ya, Descartes first wondered about that in 1500 but he phrased it a bit differently - but in the 500 years since, we haven't been able to refute it, either.