[Transcript]
Hello, and welcome to America’s First Ladies. I’m your host, Risa Weaver-Enion.
Episode 4.13 Retirement (Again) to Montpelier
We left off last week at the end of 1815, which was a banner year for the U.S. in its struggle for recognition by the European powers as a real country with real national interests.
In April 1816, Congress passed, and James signed, a flurry of bills: rechartering the national bank, setting mildly protective tariffs in specific industries, retaining many war taxes, and maintaining the army and navy at sensible levels to protect the country. Most notably, for the first time, Congress voted its members an actual annual salary. Up until this point, Congressmen had been given a palty per diem amount, which is why so many of them didn’t bring their families to Washington. It was hardly enough for one man to live on, let alone a family. Their new annual salary would be $1500 (about $34,000 in today’s money) vs. the $6 per day they received before, and only when Congress was in session (which is about $138 today.)
Also, the Republican Congressional Caucus nominated Secretary of State James Monroe to be presidential candidate for the coming 1816 election. Because the Federalist party was in such disarray—remember, they didn’t even bother to nominate their own candidate in 1812, they just supported the Republican challenger to Madison—the Republicans were virtually guaranteed to win the 1816 election.
With all these accomplishments out of the way, James and Dolley left Washington on June 5, 1816 to head to Montpelier. They would not return to Washington until October 9, making it their longest-ever absence from the capital. I’m sure you’ve noticed that both James Madison and Thomas Jefferson made extended visits to their homes while serving as president, despite both of them being highly critical of John Adams for doing the same. Granted, Quincy was further from Philadelphia than Monticello and Montpelier were from Washington, but still. It just goes to show that all these dudes are hypocrites of the first order.
The summer of 1816 was extraordinarily cold. In fact, in Europe it’s known as the Year Without a Summer. They experienced the coldest summer temperatures recorded between 1766 and 2000. I’m guessing no one realized this at the time, but the weird weather was caused by a massive 1815 volcanic eruption in Indonesia (which was called the Dutch East Indies at the time). Crop failures were rampant throughout the entire Western Hemisphere, including at Montpelier. And here’s a fun fact for you: Mary Shelley wrote Frankenstein during the Year Without a Summer while on a holiday with her friends. Lord Byron challenged them all to write a scary story, and she delivered.
Despite the unpleasant weather, the Madisons gathered 90 people at Montpelier for a Fourth of July celebration. Strangely, aside from Dolley, James’s mother Nelly, his sister, and one of his nieces, all the guests were men—mostly local neighbors. Who apparently did not bring their wives to this shindig.
When the Madisons returned to Washington in October for their last several months as president and first lady, they did not return to Octagon House. They were never able to live in the President’s House again, as it was not rebuilt until September 1817, but they moved into a home in a building called Seven Buildings. It was the corner residence of a three-and-a-half story building, at the corner of 19th Street and Pennsylvania Avenue. Congress allocated $7,577 to refurbish it, which is a weirdly specific number, equal to about $174,000 today).
As expected, James Monroe practically walked away with the election of 1816, winning all states except Massachusetts, Connecticut, and Delaware. His running mate was New York Governor Daniel D. Tompkins. The Federalists had once again not bothered to nominate a candidate, but Rufus King garnered the votes of 34 unpledged electors. It was to be the last election for the Federalists, who dissipated into nothingness over the next few years.
Knowing that he was about to leave Washington and political life for good, James seemed to transform over the winter holidays. He attended a Christmas ball at the French Embassy, and the Secretary of the Navy wrote, “I think the President never seemed so happy as now, his term of service nearly expired, and with the applause of the nation, what more can a good man hope for!”
Dolley was kept busy packing and entertaining. Her final two drawing room receptions were packed to the gills. They were so crowded that no one could even sit down. One attendee wrote, “coffee and wine and punch were handed about…to all classes and conditions of society, from the minister plenipotentiary of the Emperor of Russia to the underclerks of the post office. …Members of Congress and officers of the army and navy, greasy boots and silk stockings, Virginia buckskins and Yankee cowhides, all mingled in ill-assorted and fantastic groups.”
Congressman Elijah Mills wrote to his wife to describe the final reception, telling her that the crowd was more genteel than usual and it was “no small evidence of approbation for [Mrs. Madison’s] past conduct, and regret of her retirement. …It is said her liberality to the indigent and unfortunate is unprecedented in this part of the country. …A coterie of ladies…were lamenting, as I thought sincerely, her approaching retirement; and recounting to each other instances that had come within their own knowledge of her kindness and munificence.”
Then Mills took a parting shot at incoming First Lady Elizabeth Kortright Monroe, telling his wife, “from her successor…neither the fashionable world nor the suffering poor have much to expect.” I guess we’ll have to wait until next season to find out if he’s right!
Amidst the bustle of leaving Washington, James and Dolley each sat for a portrait by Joseph Wood. Many said that his portrait of James was the best one ever painted. Dolley’s life-long friend Eliza Collins Lee declared, “The likeness of your dear Husband almost breathes, and expresses much of the serenity of his feelings at the moment it was taken. In short, it is, himself.” I’ll put the portraits on the webpage for this episode; you can find a link in the show description.
On March 4, 1817, James was delighted to see his longtime friend James Monroe sworn into office as the fifth President. Dolley and James attended the inaugural ball later that night, but left early so as not to steal thunder from Mr. and Mrs. Monroe.
Dolley’s friend Eliza Collins Lee wrote her, “On this day eight years ago I wrote…to congratulate you on the joyful event that placed you in the highest station your country can bestow. I then enjoyed the proudest feelings—that my friend—the friend of my youth, who had never forsaken me, should be thus distinguished and so peculiarly fitted to fill it. How much greater cause have I to congratulate you, at this period for having so filled it as to render yourself more enviable this day, than your successor, as it is more difficult to deserve the gratitude and thanks of the community than their congratulations—You have deservedly received of all.”
The Madisons were not able to escape Washington right away after the inauguration, because there were still more parties to attend. They moved out of Seven Buildings so the Monroes could move in—it was serving as the official presidential residence while construction continued on the President’s House. They returned to live with Anna Cutts and her family at the house on F Street, where they had lived so many years before.
At one post-inauguration reception, an attendee noted, “Mrs. Madison behaved with her usual civility, she always finds time & occasion to speak to everybody, & what is a little remarkable never forgets the name of any one to who she has been introduced—she as well dressed rather more pensive than common, but more interesting we all thought than ever. Mr. Madison was quite cheerful and conversible even complimentary—apparently glad to be rid of his Laborious State.”
When the citizens of Washington were finally ready to let go of the Madisons, they boarded a steamboat at the Potomac wharf in Washington. Ralph Ketcham writes in James Madison: A Biography, “To Madison, whose first journey along the Potomac had been made slowly on horseback, over poor roads and across many ferries, on his way to Princeton nearly fifty years earlier, the swift, comfortable steamer ride was both a welcome pleasure and a measure of the incredible changes wrought in his lifetime.”
In his book James and Dolley Madison: America’s First Power Couple, Bruce Chadwick offers a nice summary.
“Everything had changed in America during Madison’s two terms and his eight years as secretary of state. The nation’s population boomed. The total number of people living in America in 1817, when Madison retired, was about four times as great as when he wrote the Constitution. The three western states of Ohio, Kentucky, and Tennessee had developed rapidly, and the 1820 census would show their growth from 370,000 in 1800 to 1.7 million in 1820. The country’s churches had flourished and were now found all over the land, and not just in the large cities. When the revolution ended, there were forty-four newspapers in America; in the 1820s there were more than five hundred. The one-party system that began with George Washington had folded, and there were two strong parties. Then the Federalists died and were to be replaced by the Whigs. Men’s and women’s fashions had changed, again and again, during his [sixteen] years in Washington and would change yet again in his retirement. James Madison had, in the twilight of his presidency, established a happy balance between a powerful federal government over a collection of vibrant state governments. He had moved a long way from the Madison of the mid-1790s, and so had the nation.”
As Dolley and James returned to Montpelier for good, they probably had rather different outlooks on the situation. James had grown up there, and he was eager to return to the countryside, his family, and his country friends. He was also eager to get away from the albatross of politics. Naval Officer James Paulding accompanied the Madisons on their steamboat ride down the Potomac, and he wrote of James, “if ever a man rejoiced sincerely at being freed from the cares of public life it was he. During the voyage he was as playful as a child; talked and joked with everybody on board, and reminded me of a schoolboy on a long vacation.”
Dolley, on the other hand, had only lived at Montpelier for a short stint after James’s retirement from Congress in the spring of 1797 until he was drafted back into service as Jefferson’s Secretary of State in May 1801. Although she had spent her girlhood in the countryside of Virginia, she had primarily lived in cities, first Philadelphia and then Washington, since she was 15. She was almost 49 as they returned to Montpelier. One imagines it might have been difficult to go from a glittering life of parties, dinners, and official events back to the seclusion of a Virginia plantation. A plantation where she wouldn’t even technically be mistress of the house, because James’s mother Nelly was still alive. She was 86 at this point, so no longer in the prime of life, but she lived another 12 years, so she’s not going anywhere anytime soon.
Little of Dolley’s correspondence from the time they left Washington and returned to Montpelier survives, so we don’t have many of her own thoughts and are left to conjecture about what she may have been feeling as they stepped out of the limelight and into the seclusion of retirement. She did often write to her sister and nieces still in Washington, asking for news, so it’s probably safe to assume that she didn’t love being cut off from the heart of society. Some years later, she wrote to her niece, Dolley, that the enjoyed the “quiet retreat” of Montpelier but missed “the maneuverings and gossip of the old days.”
The Madisons settled into a quiet country life. They continued to have visitors often, and each morning, after breakfast with Dolley and their guests, James would walk to the stables, saddle a horse, and begin a long ride around the plantation. It was good exercise, which he had not gotten enough of during his 16 years of service in Washington, but it also gave him a chance to check in on various projects being carried out by the enslaved workforce.
Dolley and James also spent part of each afternoon visiting with his mother. After the renovation he undertook during his presidency, Nelly essentially had her own wing of the house with her own entrance. Montpelier was like a very large duplex, with James and Dolley living in one half and Nelly in the other. One notable addition to Montpelier was the telescope that was installed on one of the porticoes. They could see the long road to Montpelier from here, and be on the lookout for approaching company. They also had an icehouse in the yard, which was a rarity. They could store ice, ice cream, and cold drinks for summer parties.
Dinner at Montpelier was served at 4 pm and usually lasted until 6 pm, depending on how many guests they had. James enjoyed regaling guests with stories of his life, featuring many famous people. Margaret Bayard Smith described one dinner, “Mr. Madison was the chief speaker. …He spoke of scenes in which he himself had acted a conspicuous part and of great men, who had been actors in the same theatre. …Franklin, Washington, Hamilton, John Adams, Jefferson, Jay, Patrick Henry and a host of other great men were spoken of and characteristic anecdotes of all related. It was living History!”
After dinner, everyone would gather in the parlor during cold weather or on the porch or in the gardens in warm weather. They would remain there, talking and drinking, until it was time to go to bed.
Dolley’s sister Lucy Payne Jackson came with her family most summers and spent the entire summer at Montpelier. Her husband, the Supreme Court Justice, was only required to be in Washington for a short portion of the year, and she didn’t love living in Western Kentucky the rest of the year. Montpelier gave her a welcome respite.
Dolley’s sister Anna Cutts and her family still lived in Washington where Richard Cutts was in government service. They came and visited Dolley and James for several weeks of every year.
For her part, Dolley seems to have enjoyed gardening at Montpelier. Near the house there was a kitchen garden, for vegetables used in everyday meals, plus a fruit orchard. Beyond that was a formal garden designed by a French gardener. Dolley focused on the vegetable garden and the orchard. She wrote one of her many nieces, “our garden promises grapes and figs in abundance but I shall not enjoy them unless your mamma comes and brings you to help us with them.”
Gardening seemed to suit her. Margaret Bayard Smith wrote of her in 1827, “Time seems to favor her as much as fortune. She looks young and she says she feels so. I can believe her, nor do I think she will ever look or feel like an old woman.”
Montpelier was still a working plantation, and James dove back into farming with a vengeance. Not that he had much choice; they had to make a living somehow. As President, James had earned a salary of $25,000 a year, but it’s easy to imagine a lot of that being spent on Dolley’s elaborate parties and dinners, not to mention her expensive Parisian wardrobes. It costs a lot of money to provide food and drinks for hundreds of people every week. Add to that the need to cover debts incurred by Payne Todd while he was abroad, plus business losses accumulated during the years of embargoes and war, not to mention James’s penchant, like his friend Jefferson, for spending lots of money on books and wine—and what you end up with is a financially strapped family.
During the war, James had shifted the crops at Montpelier to wheat and other grains, which were in high demand at the time. But after the war, that demand plummeted, so many plantations, including Montpelier, went back to tobacco. James was elected president of the Agricultural Society of Albemarle in 1818, and he threw himself into research, as he did with everything.
In a speech to the society, he recommended leaving some fields fallow every year, to aid in soil recuperation, he extolled the benefits of contour plowing and irrigation, and he advised getting rid of “scraggly animals” whose costs in feeding outweighed the income produced by their milk, manure, and hides. He also advised that each plantation keep ten acres of woodland for every fireplace in their home. He calculated that each fireplace required an average of ten cords of wood annually, that an acre of woods yielded twenty cords, and that it took 20 years for a wood lot to restore itself. So he was not only preaching regenerative agriculture, but sustainability too, many, many decades before those concepts would have names.
Ralph Ketcham succinctly outlines some of the difficulties facing James and other plantation owners: “Despite the confident tone of his address, Madison in fact faced, along with his neighbors, the calamitous consequences of agricultural depression. With the end of war-heightened demand for wheat in Europe, the bottom fell out of the grain market, and the revival in the tobacco trade immediately after the war proved shortlived. An unusual number of poor harvests during the eighteen-twenties, the steady exhaustion of the soil, and competition from rapidly opening rich farm lands in the West intensified the distress. Always short of cash, Virginia planters found it increasingly difficult to derive anything like the needed amount of coin from their crops. Farmers, large and small, faced grim alternatives: sale of land, sale of slaves, emigration, or impoverishment. …
Madison possessed in 1817 not only clear title to five thousand fertile acres at Montpelier, but also over a thousand acres of Kentucky lands, some stock in a turnpike company, a house in Washington, and other assets able to cushion him from the shock of crop failures. On the other hand a gracious way of life filled with company and entertainment, a refusal to traffic in slaves, and the vast expenditures needed to keep Payne Todd out of debtor’s prison depleted Madison’s cash reserves. Through his retirement he gradually sold his Kentucky lands, disposed of his stock and other convertible assets, and even mortgaged, with little hope of regaining clear title, nearly half of his Montpelier lands. …
Though he was spared the pain and humiliation of bankruptcy, the threat of economic ruin grew stronger during his years of retirement.”
No doubt this all sounds familiar to you from past seasons.
While his work on farm practices was forward-looking and commendable, James got involved in another, less commendable organization. He was one of the founding members of the American Colonization Society. The long and short of it is that this group wanted the federal government to take the proceeds of selling land in the West (land they were stealing from the Native Americans, I’ll remind you) and use those proceeds to purchase slaves from their owners. While eliminating slavery and manumitting the existing slaves is a worthy goal, the “colonization” part of the American Colonization Society involved shipping the freed former slaves to a new colony to be established on the West Coast of Africa, Liberia.
James felt that “existing and probably unalterable prejudices in the United States [require that] freed blacks…be permanently removed beyond the region occupied by, or alotted to, a white population.” He basically believed that Blacks and whites could not live peacefully together in the same physical space. The whites were too racist, and the Blacks were too, well, black. When Dolley’s cousin, Edward Coles, freed all his slaves, he took them all to Illinois, gave them land, and helped them learn how to farm. Madison admired his efforts, but wrote that unless they could change the color of their skin, “they seem destined to a privation of that moral rank and those social participations which give to freedom more than half its value.”
James seems to have been fundamentally incapable of recognizing any agency, competency, or drive for success among Blacks, whether freed or still enslaved. Ralph Ketcham writes, “Coles so strongly felt the moral evil of slavery, and had enough faith in the ability of his blacks, with such help as he could give, to become genuinely free men, that he went ahead, with great personal sacrifice, to manumit his slaves; Madison, on the other hand, ‘lulled,’ as Coles put it, by a lifetime of existence with slavery, and without Coles’s zealous faith in the ultimate equality of the races, was unwilling to undertake the suffering and disruption, for him and his slaves, of individual manumission. Furthermore, he placed his faith in a colonization plan that had serious practical difficulties and had as well supporters who saw it as a way to preserve slavery by getting rid of free blacks. That a man of Madison’s realism and integrity should in this instance adhere to such an insufficient and compromised program is painful evidence of the virtually insoluble dilemma slavery posed for him.”
Eighteen years after its founding, the American Colonization Society had resettled only between two and three thousand former slaves, while the slave population of the U.S. was 60,000. It was never going to be a success, and it was founded on faulty logic.
In 1829, James was invited to come to Richmond to help draft a new state constitution. He and Dolley spent the winter of 1829-1830 in Richmond. It was the first time they had spent any time away from Montpelier since returning there in 1817. Dolley enjoyed the social life of the city, and they were guests of honor at many parties and dinners. A woman named Anne Royall came to interview Dolley, the famous hostess, and was shocked to find a “tall, young, active and elegant [woman, instead of] a little old dried-up [one].” Dolley was 61 at this point but “as active on her feet as a girl. Her power to please, the irresistible grace of her every movement shed such a charm over all she says and does that it is impossible not to admire her.” Clearly, Dolley had not lost her touch during retirement.
Sadly, James’s political influence was waning. His views on gradually abolishing slavery were jeered at, and his efforts at compromise when it came to whether slaves should be counted for purposes of allocating legislative seats failed.
As Bruce Chadwick writes in James & Dolley Madison: America’s First Power Couple, “Madison realized that while he was venerated, he was not followed anymore. He knew, too, that his efforts at age-old compromise, which had always worked, now failed. And finally, he new that slavery had catapulted forward in the social consciousness as an issue. It was becoming a runaway train.”
The years after James retired in 1817 saw a steady stream of friends and family members pass away. First came James’s dear friend Thomas Jefferson in July 1826. Dolley’s brother-in-law, Thomas Todd, married to her sister Lucy, also died in 1826. Her deceased sister Mary’s husband had died the year before. James’s nephew Robert Madison died in 1828, and his three children came to primarily live at Montpelier. One of James’s uncles died in 1827, and then James’s mother Nelly died at the age of 97 in February 1829. Even James Monroe, who was six years younger than James, predeceased him, dying in 1831.
But the greatest loss for Dolley came in 1832 when her dear sister-daughter Anna Cutts died of dropsy to the heart, which is what congestive heart failure was called at that time. Dolley and James took in Anna’s two daughters, Anna and Mary, after the death of their mother.
The biggest project James worked on during his retirement was the collection of his official papers. It was a gargantuan task, because he was a prolific writer and had over 50 years’ worth of letters, speeches, treatises, notes, and research. To help him with this work, he enlisted the aid of both Dolley and her brother, John C. Payne. You may recall that John Payne had been a wayward son earlier in his life, but he eventually married and settled down. He was a lifelong alcoholic, but apparently he was helpful enough to James in this project.
They started work on the project in 1821, and in 1824 Dolley complained in a letter, “this is the third winter in which he has been engaged in the arrangement of papers and the [business] appears to accumulate as he proceeds, so that I calculate its outlasting my patience and yet I cannot press him to forsake a duty so important or find it in my heart to leave him during its fulfillment.”
One of the most historically important aspects of James’s official papers was the collection of notes he took at the Constitutional Convention. As I mentioned back in episode 4.4, when his notes were published, they were not entirely as he wrote them down during the convention. In these later years, which were many decades after the convention, he revised, annotated, crossed out, and obfuscated some of what he had written. If you’re interested in a deep dive into what, exactly, was changed, I’ll direct you once again to the book Madison’s Hand by Mary Sarah Bilder. I’ll put a link in the show notes for this episode. [https://www.amazon.com/Madisons-Hand-Revising-Constitutional-Convention/dp/0674979745]
James hoped that the eventual publication of his notes from the convention would raise enough money to support Dolley after his death.
Despite remaining mentally active during his retirement years, James did start declining in the 1830s. After returning to Montpelier from Richmond in early 1830, James rarely left it again. In the autumn of 1831, when James was 80 years old, Dolley wrote that he was so weak from rheumatism and fever that she had hardly left his side for a month. She was writing letters on his behalf because his hands were too swollen to write. Even well into spring 1832, James was still confined to bed.
Summers usually brought relief, and he was able to ride his horse every day in September 1833. Dolley wanted to take him to the warm springs, but the journey would have been too much for him, so they never went.
In February 1835, British writer Harriet Martineau came to Montpelier to spend three days interviewing James for a multi-volume book she was writing about the United States. She described him as having difficulty seeing and hearing, but still willing and able to take part in conversation. She wrote that he sat in his chair from nine in the morning until ten at night “with a pillow behind him,...his little person wrapped in a black silk gown; a warm gray and white cap upon his head, which [Mrs. Madison] took care should always sit becomingly; and gray worsted gloves, his hands having been rheumatic.”
Martineau described Dolley as “a strong-minded woman, fully capable of entering into her husband’s occupations and cares; and there is little doubt that he owed much to her intellectual companionship, as well as her ability in sustaining the outward dignity of his office.”
They spent the entire three days of her visit talking about politics and history, and she was amazed at how engaged he was and able to recall details of events from decades earlier. She found that his overriding belief was faith in the American people, writing that he had “an inexhaustible faith, faith that a well-founded commonwealth may, as our motto declares, be immortal; not only because the people, its constituency, never die, but because the principles of justice in which such a commonwealth originates never die out of the people’s heart and mind.”
Next week, we’ll say goodbye to James Madison, and follow Dolley for the remaining 13 years of her life.
Thanks for listening to this episode. It was produced by me, and the music is by Matt Dull. Please share the podcast with a friend you think would enjoy it. Thanks!
