He was accompanied by the King of Oman, Sultan Qaboos bin Said al Said and other officials.
After visiting the mosque, Prime Minister Modi signed the visitors' book and met supporters at the mosque.
He then boarded the Air India One plane back to New Delhi. He is scheduled to arrive in the evening.
Earlier, Prime Minister Modi on Monday visited the 125-year-old Shiva Temple in the Matrah area of Old Muscat. He also met with a group of the Indian community outside the temple.
This completes the four-nation tour of the Prime Minister, which took him to Jordan, Palestine, the United Arab Emirates (UAE) and Oman between February 9 to 12. (ANI)