They’re big on forts in Oman. There are dozens of them, often situated on a strategic high point in the  rugged, mountainous terrain that spans much of the country: the fort at Nizwa, with its single round tower 43 metres in diameter, its top circled with firing ports for cannon, is a striking example. The sultanate’s capital, Muscat, is situated on a strip of coastal plain, but even this is strewn with rocky hillsides, many of them topped with forts – some old and ruined, some very much in use, as evidenced by the serious-looking fatigue-clad sentry outside the Al Mirani fort, one of a pair of forts which flank the Al Alam palace by the old harbour. The Al Mirani and its sister the Al Jalali were built by the Portuguese during a century-long occupation: it’s an episode which features strongly in Omani history books, even if, as our Omani guide pointed out to us somewhat ruefully, it has been largely forgotten by the Portuguese.

Unsurprisingly, the other most prominent landmarks are the mosques, which dominate the landscape. Omani mosque architecture seems to be never less than elegant and beautifully crafted. At its best, it’s jaw-dropping, nowhere more so than in the Sultan Qaboos Grand Mosque in Muscat, inaugurated in 2001 to celebrate 30 years of the present Sultan’s reign. The quality and quantity of the craftsmanship leave you gasping for breath, as do the elegance and proportions of the many buildings that make up the complex. The calligraphy in shining white stone around the outside of the main building, the elaborate carving on the wooden doors of the women’s prayer hall, the single piece Persian carpet that spans the whole of the men’s prayer hall (many times the size, because Omani women are permitted to pray at home, while the men are required, broadly speaking, to come to the mosque), the lighting and tiles: all are beautifully conceived and immaculately executed.

As a visitor leaving the Grand Mosque, you are likely to meet one or more Omani women wearing the standard black robes and a broad smile of welcome, to be taken to the Islamic Information Centre, where you are encouraged to ask questions about the mosque and Omani life in general while being plied with dates, Omani coffee and ginger/lemon grass tea. It’s all done with grace and charm, supporting the received view of Oman as a moderate Islamic nation welcoming to non-Muslims. They’re hard-line on some things, particularly within the mosque – my wife’s honest attempt at dressing respectfully flunked due to her cotton shawl being deemed to reveal her skin colour through white cloth – but people here have been invariably charming, and at no point have I felt that I would have caused outrage by admitting to being Jewish (or perhaps even to having Israeli parents). I must confess, though, that I wasn’t brave enough to put this feeling to the test.

Oman has nearly 8,000,000 date palms and more than 250 indigenous varieties of dates. The locals miss few opportunities to offer them to you: as a greeting when you enter a shop or a hotel, as a petit-four equivalent (with Omani coffee) after a meal, or pretty much any other time when anyone thinks you might be about to sit down somewhere. They’re mainly eaten dried (though sometimes fresh) and they’re very, very good. The love affair is deep and goes back a long time: in Fort Nizwa in the mountains above Muscat, you can see the room where sacks of dates were piled on top of each other as provision against siege: the weight of the dates at the top expelled juice that was gathered in special ducts carved into the stone floor of the store room.

The other notable agricultural product is frankincense, known as “luban”. As well as burning it for incense or turning it into a perfume, they swear by “luban water” as a medicinal item: you drink water into which a small piece of the resin has been soaked overnight (only the higher quality luban should be used – the lower grade stuff is for burning). A visit to any of the souks will yield ample opportunities to buy some, as well as many perfume shops which will sell you many oils and perfumes very different in character from the typical big brands in the West: the scent is therefore usually less blended (and therefore less complex) but considerably longer lasting.

Oman is a sparsely populated country (27% bigger than the United Kingdom, with one fifteenth of the population).  It therefore shouldn’t surprise one that the city planning and general road layout in Muscat has a very American feel, with wide roads, cars parked perpendicular to the houses, the main shopping concentrated into large malls, the smaller units in strip and corner malls. The roads are excellent and driving around is straightforward, apart from the occasional bear trap for the unwary (a street sign facing you could either denote the name of the cross street, as in the US, or the street you’re on). Signage, like Omani driving, is generally good but not always: it works really well most of the time, but will occasionally send you in the wrong direction or (in Italian style) abandon you altogether at critical moments.

Like all the oil states in the Gulf, Oman needs to plan for the day when the oil will run out. The first part of this has consisted of building infrastructure: apart from the roads, there has clearly been a giant construction boom. The next part, currently very much under way, is to build up Omani businesses equivalent to the best of the multinationals so that there is a proper local economy offering jobs and profits to Omani nationals. From what we heard, progress is being made, but there are obstacles which are taking time to overcome. The first is a paternalistic, deferential culture in which it’s not a good idea to tell a superior that what he’s asking for is a seriously bad idea: one expatriate described to me his discomfiture at discovering that his entire (extremely busy) team had downed tools at his Omani superior’s command and were patiently waiting in a conference room for this man to address them. Another described the difficulties of turf wars in situations where you need more than one department to implement a decision for the project to succeed, but where one department simply decides that they don’t agree. The will to make decisions is there, but the ability to grasp the implication of a decision and do all the things needed to turn into reality is sometimes missing. This is by no means a uniquely Omani trait – plenty of Western businesses suffer from similar types of sclerosis – so let’s hope that education and experience will alleviate the problems and enable the new economy to thrive.

We met several of the many Western expatriates working in Oman, all of whom attested to the Omanis’ charm, family-mindedness and hospitality: they had regularly been invited to visit colleagues’ homes to meet the family and it had invariably been a delightful experience. Perhaps that innate feel for hospitality contributes to the fact the hotels are superb, and indeed, one area where the new economy seems to be thriving already is tourism. There’s good diving (which was what initially attracted us) and the country is becoming popular as a destination for outdoors types: there’s is masses of rugged territory for hiking and off-road driving or biking: we did some great hikes up at 2,000 metres in Jabal Akhdar, the “green mountain”, with spectacular limestone canyons as the scenery.

We stayed in two of the luxury hotels, one in Muscat and one in Jabal Akhdar: both were architecturally amazing and both gave us a genuinely warm welcome: perhaps the long-standing Omani reputation for hospitality extends to being good hoteliers (and yes, I realise that most of the staff are expatriates, as is around half the population of the country). A slight warning, though: food in the big hotels can be very expensive. 

Just as Oman is around half way between, Egypt/Syria/Lebanon and India, Omani food seems to be at a mid-point between Arab and Indian cooking, with lots of curries and lots of spices (unsurprising once you realise that Oman used to own Zanzibar and control its spice trade). But they seem to use their spices more for fragrance and aroma than most Indians: cardamom, saffron and ginger are popular, fiery chili-induced heat is not. Arab influence is seen in dishes like kibbeh (fried meat and lentil balls), moutabel (aubergine dip), a penchant for a soft curd cheese called labneh and the ubiquity of flatbreads and hummus.

One aspect of Oman’s strict Islam is that they don’t drink alcohol (although hotels serving westerners generally serve alcoholic drinks, as do some restaurants). This has the side effect that they serve the most wonderful selection of soft drinks. My personal top tip is to order “lemon mint”: fresh lemon juice with fresh mint blitzed to a powder, with crushed ice, water and sugar (you can ask them to adjust the sugar level). Your dates will often be served with Omani coffee, which is mild and lightly laced with cardamom and sometimes other spices, poured out of a characteristically shaped pot which looks like it takes years of training to learn to use without spillage. If you don’t like coffee, lemon grass and ginger tea is a winner. But there are tons of other inventive fruit and spice-based drinks to enjoy: being the designated driver has never felt less of a chore.

We originally came to Oman as a place to go diving with Indo-Pacific marine life and less jet lag than the Maldives or Indonesia. In the event, our diving was curtailed by bad weather – apparently, storms in the Gulf of Oman can be whipped up out of nothing – which gave us extra time to enjoy the country. Whisper it not, but I’m thankful for the bad weather at sea: the country is fascinating and well worth the visit.