At the end of the summer in the year 636 CE, following the Battle of Yarmuk, the outcome of which signaled the end of the Byzantine period in Palestine, the Muslims made for Jerusalem and laid siege to the city for two years. After eighteen months the besieging army was joined by a second force, led by ben-Jarah, and finally the city surrendered.

According to Muslim tradition, Ben-Jarah called on Omar to assume the crown of victory over the important Christian (and, of course, Jewish) city. Another tradition holds that the city's inhabitants, who had heroically withstood a series of attacks, declared that they would surrender to no one but the leader of the Muslim world himself and Omar indeed came to the city to accept the surrender. Omar is said to have entered the city riding a camel and wearing a simple camel-hair cloak. This modest entry embarrassed his army commanders, who had been in the area for three years and were accustomed to Byzantine splendor. To the people of Jerusalem Omar proposed a convenient treaty of surrender, the contents of which were related by the Muslim historian Tabari: "In the name of Allah the merciful and compassionate, this is the writ of protection [aman] which the servant of God, Omar, commander of the faithful, gives to the people of Aelia [the usual name for Jerusalem in the early Muslim period]... for their souls, their property, their churches, their crosses, the sick and the healthy, and their whole community..." The treaty contained several clauses: (1) security for life and property; (2) security for the churches and for ritual worship; (3) a ban on Jewish residence in the city; (4) obligation to pay tax (jizah); (5) freedom to choose whether to remain in the city and pay the tax as stipulated, or to leave in safety. There are several versions of this text, some omitting or emphasizing certain clauses.

According to Muslim tradition, Omar built a modest mosque on the Temple Mount. He refrained from establishing the mosque north of the Foundation Stone, as his adviser, the Jewish convert Kha ab al-Akhbar, had recommended, so that the faithful could, during prayer, face both the sacred Jewish site and Mecca. Instead, he built on the southern part of the mount, where El-Aksa Mosque now stands, so that the worshippers faced only Mecca.