Early before dawn

They start to move into the Prophet's Mosque in Madinah to make sure that they get a place to pray. Late means you will have to pray outside, and in the mornings of Madinah, it could be chilly and uncomfortable.