Ever since antiquity, mankind has been fretfully besotted with strange creatures from the oceans' depths. A terrible sea monster called Leviathan is mentioned in the Old Testament. The prophet Jonah is famously swallowed by an enormous whale-like fish. Watery behemoths feature in the creation myths of many cultures. The Loch Ness Monster still generates headlines today. So it is no surprise that artists have frequently attempted to depict awe-inspiring deep-sea beasts, which, of course, they could never have seen in reality.

In the ancient world, fantastical and threatening sea creatures embodied the many dangers of maritime trade

In the ancient world, fantastical and threatening sea creatures embodied the many dangers of maritime trade upon which so many Mediterranean societies depended. Homer's Odyssey described two sea monsters called Scylla and Charybdis that lurked on either side of a narrow channel of water (perhaps the Strait of Messina between Italy and Sicily).

In a natural grotto near the coastal Italian town of Sperlonga, where he had a seaside villa, the Roman Emperor Tiberius displayed several dramatic marble sculptures – including a terrifying vision of Scylla, set on an "island" within a pool in the middle of this enormous cave, devouring Odysseus's companions. Although only fragments of the sculpture survive, Scylla's top half took the form of a gigantic, bare-breasted woman, while her nether regions consisted of several vicious dogs, emerging from her genitals, clawing at Odysseus's crew.

Intriguingly, one of the fragments is inscribed with the names of the artists who carved the sculpture. It turns out that they were the same trio, from the island of Rhodes, credited by the Roman writer Pliny the Elder as the artists responsible for the famous Laocoon, now in the Vatican. This impressive marble group, which Pliny praised as "a work to be preferred above all others", also contains sea monsters: two sharp-fanged sea serpents that writhe around the agonised Trojan priest Laocoon and his innocent sons.

In Book II of the Aeneid, Virgil offers a chilling account of the death of Laocoon, who was eliminated by Minerva because he realised that the Trojan Horse was a trick. (It is Virgil's Laocoon who coins the famous saying about not trusting Greeks who bear gifts.) Two immense serpents, with fiery eyes, flickering tongues, hissing mouths and blood-red crests, suddenly emerge from the waves, before wreathing Laocoon and his sons in massive coils and devouring their wretched limbs. What a spectacularly gruesome way to go.

Beware the Kraken

By the Renaissance, during the great age of maritime exploration, improbable sea monsters commonly decorated maps. Most were frightening – a colossal squid-like creature known as the Kraken terrorised seafarers for centuries – but many were whimsical too. For instance, strange-looking whales, with improbably large teeth and lupine faces, as well as waterspouts, can be seen in the Carta marina ("Sea Map") of the Nordic countries of Northern Europe, drawn by the Swedish ecclesiastic Olaus Magnus in the early 16th Century. At the same time, there was a new desire to make accurate studies of these much-mythologised cetaceans whenever possible.

In 1520, the German artist Albrecht Dürer, who was capable of extraordinary realism, travelled to Zeeland in the hope of seeing a stranded whale. "It is much more than 100 fathoms long and no man living in Zeeland has seen one even a third as long as this is," he wrote in his diary. However, when Dürer arrived, the "great fish" was gone, carried off by the tide. For his troubles, Dürer fell ill with a fever. It is possible that he contracted a malarial infection, which eventually finished him off in 1528.

In the 16th and 17th centuries, stranded whales were significant events in Northern Europe. "Sometimes, they were taken as omens of God's ill will," explains Philip Hoare, whose brilliant book Leviathan (2008), a memoir-cum-history about his obsession with whales, won the prestigious Samuel Johnson non-fiction prize. "In 1658, a white whale swam up the Thames as far as John Evelyn the diarist's estate in Deptford. It was hacked to death, but its appearance was seen as an augury of the death of [Oliver] Cromwell."

Hoare continues: "But stranded sperm whales also offered good fortune, because, of course, they represented food and money – something we would never think about today." During the 17th Century, in particular, there were many reports of beached whales in the Netherlands. As a result, dramatic scenes of stranded whales became a staple of art in the region. Two years ago, for instance, a conservator in the Fitzwilliam Museum in Cambridge discovered a beached whale beneath layers of overpainting in Dutch marine artist Hendrick van Anthonissen's View of Scheveningen Sands (c 1641).

'Alien and unconquerable'

By the 19th Century, the whaling industry was big business. "It oiled and lubricated the Industrial Revolution," explains Hoare, whose most recent travelogue, The Sea Inside, was published in 2013. "The streets of New York, Berlin, Paris, London were lit by whale oil, which was America's second-biggest export after timber. The Millennium Dome in London sits on the site of a whale-rendering plant." Hunting sperm whales on the open ocean was dangerous and dirty – adult males can weigh up to 60 tonnes (60,000kg) – but, for the general public, the practice seemed exciting.

In part, this explains why JMW Turner, the British landscapist and marine painter par excellence, created a quartet of pictures on the theme of whaling during the 1840s. For the first time, an ongoing exhibition at the Metropolitan Museum of Art in New York unites these four seascapes, which were first shown at the Royal Academy in 1845 and 1846.

Turner wished to imagine whaling as an elemental struggle between man and the sublime power of nature

It is likely that Turner chose whaling as a subject because he hoped to please his patron Elhanan Bicknell, who had made his fortune in the whale-oil business. As a source for his paintings, Turner drew upon The Natural History of the Sperm Whale (1839), an authoritative treatise written by the former whale-ship surgeon Thomas Beale. However, a glance at Turner's pictures suggests that he wasn't interested in precisely rendering a sperm whale's form, which he probably never encountered in reality. Instead, he wished to imagine whaling as an elemental struggle between man and the sublime power of nature.

This was especially true of Turner's earlier pair of whaling paintings. The Met's Whalers, which the museum purchased in 1896 (the other three pictures are in the collection of the Tate), contains the dark, ominous form of a hunted sperm whale in the foreground, rearing and thrashing amid a maelstrom of foam, spray, spuming blood and churning surf. In the words of the English novelist William Thackeray, who saw the painting at the Royal Academy, "That is not a smear of purple you see yonder, but a beautiful whale, whose tail has just slapped a half-dozen whale-boats into perdition." Turner's whales boast the capricious ferocity of the sea monsters that bedevilled the imaginations of sailors in the ancient world.

According to Alison Hokanson, who organised the Met's exhibition, it is even possible that Turner's whaling paintings influenced Herman Melville when he was writing Moby-Dick. Melville's masterpiece was published in 1851, a few months before the artist's death. Its third chapter contains a passage describing a large oil painting of a whaling scene in a Massachusetts hostelry called the Spouter-Inn: "A boggy, soggy, squitchy picture truly, enough to drive a nervous man distracted," says the narrator, Ishmael. "Yet was there a sort of indefinite, half-attained, unimaginable sublimity about it that fairly froze you to it." This sounds like one of Turner's paintings, which were notorious for their perilously free brushwork and lack of traditional finish.

"Look at Turner's Sunrise with Sea Monsters [an unfinished canvas from c 1845 with an ambiguous pink form in the middle]," says Hoare. "There's all this bubbling, strange stuff going on underneath. It's psychologically uncanny. It's about what's below, what's underneath the relentless movement towards progress in the 19th Century: the wilderness."

He pauses. "Below the ocean's skin, as Melville put it, everything is other – alien and unconquerable. I'm not sure what Turner really knew about whales. He just went for the symbol."

Alastair Sooke is art critic of The Daily Telegraph

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.

And if you liked this story, sign up for the weekly bbc.com features newsletter, called "If You Only Read 6 Things This Week". A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.