At 5 am and 6 pm, government loudspeakers crackle to life all over Vietnam. Relics of an era before homes had televisions and radios, these public address systems – broadcasting news, propaganda and weather reports – are usually barely audible above the din of this modern nation: a mix of motorcycle engines, truck horns and construction.

But on Con Son, the news carries loud and clear over the tiny township of just 5,000 residents, two sets of traffic lights and one seaside promenade. Visiting the largest island in the Con Dao archipelago – a group of 16 mostly uninhabited picturesque islands in the South China Sea – is like stepping back in time. 

Despite being a short 45-minute turboprop flight from Ho Chi Minh City, the island paradise of Con Son is a world away from Vietnam’s well-beaten tourist trail. It is a throwback that has somehow flown under the radar, escaping the overdevelopment of mainland resort towns like Nha Trang and the party beaches of Phu Quoc. There are no touts, only peaceful empty beaches, and peak season means being just one of a dozen Westerners.

It won't stay this way forever, though. The island's first resort, the ultra-luxe Six Senses Con Dao, opened at Dat Doc Bay on the island's east coast in 2010, and a megaresort is currently under construction in the south. There is also talk of an Italian-backed spiritual retreat, as well as rumors about extending the airport's runway so that bigger planes can land.

But for now, most of the visitors are domestic Vietnamese paying homage to the island’s dark past. Known as Southeast Asia's Devil's Island, Con Son was once a penal colony used to brutal and cruel effect first by French colonists and later during the Vietnam War. The French worked 914 men to death building the island’s jetty, while Vietnam War prisoners were kept in infamous "tiger cages", where captives – actual or suspected Communists – were shackled to the floor of deep concrete pits with steel bars for a roof. The main prison walls still dominate the town as a constant reminder, and the jails and cemeteries have become pilgrimage sites to the thousands of Vietnamese who suffered and died on the island between 1862 and 1975.

Though the horrific memories hang heavy still, modern island life is languid and laid back. Con Son's steep green interior is fringed by warm turquoise water and coral reefs. Flame trees and bougainvillea give licks of color to the jungle, and frangipani and magnolia trees line the wide, quiet boulevards. A single main road wraps about halfway around the island; a motorbike ride along the shoreline takes in ponds filled with lotus flowers, spectacular reddish-orange cliffs and one empty white-sand beach after another. The sea is calm, clean and perfect for swimming year-round.

The island's daily routine starts at the bustling marketplace, where squid, crabs, clams, rambutans, plantains, mangos, dragon fruit and lotus flowers are piled for sale outside. Inside, young soldiers in jungle greens sit at the food stalls on low plastic chairs, gulping down breakfasts of bun rieu (crab noodle soup) or bun thit nuong (char-grilled pork with vermicelli noodles) while the morning sun floods the doorway. By 9 am, the food is sold out and by noon the market is deserted.

Then, nothing happens until 2 pm, when new afternoon stallholders will arrive, selling roast pork sandwiches, sugarcane juice and rice paper rolls. For two hours, the postmistress goes home, the market is empty, the island naps and the sun beats down on the blue sea. There is nothing to do but swim at one of several beautiful beaches: it doesn't matter where you go, you'll have it to yourself.

If this were the mainland, you might find hourly booze-cruises around the harbor. If you want to see the archipelago from Con Dao, ask around in the evening and you'll be able to hitch a ride with a fisherman early the next day. Slowly putt-putting out of the harbor in a brightly painted blue-and-orange ramshackle boat, the fisherman will likely take you towards the bays and reefs of the tiny outer islands. A snorkel and goggles will be enough to catch a glimpse of a turtle, and the diving is renowned as some of Vietnam's best. As the heat of the day passes, many travelers choose to explore the island via motorbike.

At An Hai bay, 1km south of Con Son town, pearlers and fishermen moor their boats, keeping coracles –  circular bamboo vessels, waterproofed with coconut-palm resin and propelled by paddle – on the shore as dinghies. For the next 6km, climb the road along the bay until it reaches the island's southmost tip, with views in every direction. The archipelago unfolds to the east; the harbor and Con Son town to the north; the island's rugged and rocky interior to the west.

Hugging the cliffs, the road winds past steep-walled crystal-clear bays and culminates at the island's jewel – the wide flats of Nhat Beach. Here mountains give way to an expanse of deserted white sand that runs hundreds of meters from shore at low tide, with the warm water waist-deep for hundred of meters more. Stay until twilight when the horizon twinkles with the lights of container ships plying the busy South China Sea.

Come evening, Con Son town’s seaside promenade takes its turn as the island's social hub. The sky turns pink and barbecue carts roll up roasting corn, chicken skewers and pork. Swimmers – mostly Vietnamese tourists who shun the sand at daytime – arrive for a dusk dip. The waterfront swells with people.

Across the road from the beach, colonial French villas crumble in various stages of disrepair; their jungle gardens slowly taking over. French composer Camille Saint-Saens stayed in one while he finished work on his opera Brunhilda in 1895; today, that same building is the popular Con Son cafe, which provides only the essentials: beer, ice cream and Vietnamese coffee.

By night, a different market opens on Tran Huy Lieu, two blocks to the east. Half the street is taken over with chairs and metal foldout tables; beer flows as roadside stalls barbecue shellfish, calamari and the rest of the day's catch. It is a delicious way to end the day – with only the nagging thought that the return flight back to the mainland is all too soon.