Frederick Koch's Upper East Side townhouse sits on a quiet street off Fifth Avenue, near the Metropolitan Museum of Art. It is January 2013, and in a little over a week's time, Frederick's hard-hat-wearing younger brother will dip a ceremonial shovel into a mound of dirt in front of the museum, officially breaking ground on the David H. Koch Plaza. Frederick, the eldest of the four Koch brothers, has devoted his life to the arts, but it's David's name that's plastered on some of New York's most prestigious cultural real estate, including the former New York State Theater at Lincoln Center.

The world of the Upper East Side elite is a small one, and Frederick and David occasionally bump into each other at galas or charity functions. These short, awkward exchanges ("Oh, hi, Freddie") are pretty much the extent of their contact. Unlike David, who enjoys the status that comes with his high-profile philanthropy, Frederick conducts his life as if almost striving for obscurity. Thanks to the recent infamy achieved by David and Charles Koch through their sprawling political operation, he is now thought of as one of the "other" Koch brothers (along with another brother, Bill).

Frederick, 80, is so private about his affairs that during the 1980s, after underwriting the $2.8 million construction of England's Swan Theatre in Stratford-upon-Avon (in the shell of a fire-ravaged Victorian playhouse), he kept quiet about his gift for several years, as the British press tried to dig up the name of the angel donor. When Frederick's role was finally revealed, he told the BBC in a rare interview, "Never ask from where I came, nor what is my rank or name." He was quoting Lohengrin's warning to Elsa, when the knight comes to her aid in Wagner's romantic opera. When Elsa later poses the forbidden question, her savior disappears in a boat pulled by a dove.

Built of white marble, Frederick's seven-story neoclassical townhouse is one of a trio commissioned, in the early 1900s, by dime-store magnate Frank Winfield Woolworth, and designed by Charles Pierrepont Henry Gilbert, one of several architects favored by New York's industrialists during the Gilded Age. Woolworth gave the townhouses to each of his three daughters as wedding gifts. Six East 80th Street, the property Frederick now owns, belonged to the tycoon's youngest daughter, Jessie. Woolworth was said to have wept on the day he gave Jessie away to James Donahue, the Irish-American scion of a less prominent family that made its money in the fat-rendering business. Donahue was a man of vices—gambling, booze, and young men, in particular—and Woolworth's doubts about the pairing proved well placed. In 1931, during a luncheon at the couple's home, the 44-year-old Donahue excused himself from the table and locked himself in the downstairs bathroom. He staggered out a few minutes later exclaiming, "I've done it." Despondent over his finances and recently spurned by a young sailor, Donahue had gulped down seven mercury-bichloride pills. He died within days.

Frederick acquired the property for $5 million in 1986, three years after cashing in his stock in the family company, Koch Industries, and in the throes of a frenetic buying spree of historic homes and artwork. Frederick's longtime architect, Charles T. Young, spent the next decade restoring the townhouse to its former splendor, and in many cases, surpassing it. Frederick spared no expense. He continued a marble balustrade that had ended after the first floor up the staircase and through the remaining six floors of the house. He replaced the crumbling plaster walls with limestone from the quarries of Caen in northwestern France, the kind that would be found in a Parisian townhouse of the French Régence period. In one case, Frederick made structural alterations just to create enough wall space to hang one of the masterpieces of his art collection, widening a pair of stone columns to fit William-Adolphe Bouguereau's The Abduction of Psyche.

"Sotheby's for years has been trying to take this away from me," Frederick says, as he shows me around his home. "And I keep telling them, 'I can't move it. It's part of the architecture of this room.' " Frederick's office is located in what was James Donahue's bedroom. Ornate paneling that once adorned the Palace of Versailles lines the walls. A bookcase, as in some murder-mystery thriller, conceals a hidden passageway, designed, Frederick explains, "so that the six servants in this house would not be aware of Mr. Donahue's comings and goings." It leads past a row of stained-glass windows inlaid with the initials of the Donahue clan, to what was Jessie Donahue's bedroom and is now Frederick's.

Not that he actually sleeps there.

Frederick spent millions getting every hand-wrought, filigreed detail just right, but he doesn't actually live in the townhouse. When he's in New York—and he moves frequently among his collection of homes, which includes Schloss Blühnbach, a palatial hunting lodge just south of Salzburg, Austria, that once belonged to Archduke Franz Ferdinand—he resides at 825 Fifth Avenue. He just entertains the occasional guest at the Woolworth mansion, his private museum.

Despite the scale of his wealth and the opulence of his surroundings, Frederick also has a reputation for frugality, sometimes growing testy at his staff, if they add extra postage to letters and packages. He spent lavishly on refurbishing his homes, but he prefers taking the public bus in New York and typically flies commercial.

One associate recalls strolling down East 80th Street with Frederick on a sweltering summer afternoon in the mid-1990s. Crossing Fifth Avenue, Frederick noticed a nickel in the middle of the crosswalk; it had been run over so many times that it was embedded in the asphalt. His companion looked on in shock as Frederick took out his keys, stooped down and began trying to pry the coin loose. The multi-millionaire continued to work as the traffic light changed. Traffic bore down and horns blared, but Frederick kept digging, finally dislodging the nickel. "I got it," he said, holding the coin up with a beatific expression on his face. "I just was dumbfounded," his companion recalled. ("I never pick up coins in the street," Frederick responds, "despite this apocryphal account.")

Dressed in a navy blazer, striped shirt, and thin cobalt tie, Frederick moves through the rooms of his home like a curator, his arms folded behind him. "If you pull the carpet back, you'll see a Versailles parquet," he says, touring the dining room. He pauses to point out paintings by Alexandre-Evariste Fragonard, Edward Burne‑Jones, and Gilbert Stuart; Aubusson carpets; a set of 10 mahogany dining-room chairs that once belonged to financier J. P. Morgan.

In one bedroom, he shows off a canopied bed that he calls "the most important piece of furniture in the house." It belonged to Marie Antoinette, a wedding gift from the mayor of Paris, when she married Louis XVI. "I'm amazed even to this day that I was able to get it out of France. I bought it in Monte Carlo, which may have been the reason."

He is at ease talking at length about every object in the house, and the intricacies of procuring or rehabilitating each of them. But when the topic turns briefly to his brothers, and whether they share his artistic sensibilities, Frederick tenses. "William does—David's twin brother," he replies. "He has a wonderful collection of all kinds of things." Then he changes the subject.

It's no surprise that Frederick isn't eager to talk about Charles and David. He and younger brother Bill spent nearly 15 years locked in a series of bare-knuckle legal brawls with their brothers, who they accused of cheating them on the 1983 sale of their Koch Industries stock, which together netted them $800 million. (In 1998, a jury deemed Koch Industries had misrepresented a refinery's productivity in the buyout negotiations, but it also determined that any omissions were not material, so Frederick and Bill could not recoup the massive damages they sought.)

Frederick displays one of his most treasured pieces on the top floor, in a sun-drenched, glass-enclosed conservatory that was once Jessie Donahue's studio. On a pedestal beneath a circular skylight ringed with the signs of the zodiac sits a marble head wearing the headdress of an Egyptian pharaoh. The sculpture dates back to A.D. 130. "That is Antinous," Frederick explains, "who was the lover of Emperor Hadrian . . . Antinous accompanied Hadrian on a voyage down the Nile and at some point they stopped. Antinous went swimming and drowned in the Nile. The emperor was so grief stricken that he decided to immortalize Antinous and name him god of the Nile—and required people to worship him."