Link to H.R. 630 https://www.congress.gov/bill/119th-congress/house-bill/630
Link to S. 193 https://www.congress.gov/bill/119th-congress/senate-bill/193
[Transcript]
Hello, and welcome to America’s First Ladies. I’m your host, Risa Weaver-Enion.
Episode 4.6 Retirement to Montpelier
We left off last week in the spring of 1797. After not standing for re-election to the U.S. House of Representatives, James and Dolley Madison left Philadelphia and headed to his family plantation in Virginia. Even though they had been married for nearly three years at that point, Dolley hadn’t had the chance to meet James Sr. and Nelly Madison, or to see Montpelier.
James was relieved to get out of the viper’s nest that was Congress, and Montpelier needed his assistance. James Sr. was 74 years old at this point, and no longer able to manage day-to-day oversight of the large plantation. While James was busy serving his country, his brother Ambrose had taken on a leading role at Montpelier, but Ambrose had died in 1793.
Another advantage of Montpelier was its proximity to Monticello, where Thomas Jefferson lived when he wasn’t needed in Philadelphia (remember, he was Vice President under John Adams). The Madisons made twice-yearly lengthy visits to Monticello when Jefferson was there, which is how Dolley and Martha Jefferson Randolph became close friends.
Both Monticello and James Monroe’s plantation called Highland were within a day’s ride of Montpelier. Monroe returned from France in 1797 and moved to Highland in 1799. Jefferson and Monroe were Madison’s closest friends and ideological kindred spirits.
For the first time in her life, Dolley settled into the role of prominent southern planter’s wife. If her parents hadn’t been Quakers, and if they hadn’t freed all their slaves and moved to Philadelphia, this might have been a role she was quite familiar with. But she had James’s mother Nelly to serve as role model. Dolley and James were kept quite busy making visits to various friends and family connections that James hadn’t had time to see during his 20+ years of service in American government.
He also was busy as a farmer. He, Jefferson, and Monroe traded many letters on the subject, sharing ideas and advice. After a difficult season in 1797, James sent a letter to James Monroe in February 1798, writing, “Calling to mind the difficulty you may experience from the general failure of the potato crop last year, I beg you to accept by the bearer a couple of bushels, which may furnish the seed for your garden, if nothing more. Mrs. Madison insists on adding for Mrs. Monroe a few pickles and preserves, with half a dozen bottles of gooseberries and a bag of dried cherries.”
Dolley and James’s family became a little larger in March 1798 when his niece Nelly became their legal ward. As I mentioned, James’s brother Ambrose had died in 1793, leaving behind young Nelly and her mother, who then died in 1798, leaving teenaged Nelly an orphan. With the younger set of Madisons now expanded to a family of five, and the elder pair of Madisons also living in the same house, James decided to expand Montpelier.
The existing house had four rooms on the ground floor, two apiece on either side of a central passage, with a staircase leading to a second floor with four additional rooms. During this first expansion in the late 1790s, James added a 30-foot addition to the north end of the house, providing an additional two rooms on both the ground floor and the floor above. He also added a new front door and new staircase, making the house more of a duplex than a mansion. There was no access between the two halves until a later renovation during Madison’s presidency.
The expanded Montpelier gave the Madisons a chance to show off all those fine French furnishings they had asked James Monroe to purchase on their behalf. And Dolley had a chance to show off her fine hosting skills to the numerous guests they entertained for dinners, teas, and parties during the four years they spent at Montpelier.
Although James was out of government service from 1797 until 1801, he was not ignorant of what was happening in Philadelphia during the Adams Administration. Jefferson kept him apprised via long letters, and they undoubtedly spent most of their time together in person discussing politics.
On the subject of the XYZ Affair, which we discussed in episode 2.12, Madison and Jefferson both allowed their affinity for France to cloud their judgment, and they refused to see what was so outrageous about a foreign minister demanding a personal bribe in order to meet with the appointed representatives of a peer nation. But when it came to the Alien and Sedition Acts, which we also discussed in episode 2.12, they were more clear-eyed than Congress and President Adams.
To refresh your memory, the Alien and Sedition Acts were four different laws. There were three that together made up the Alien Acts:
The Naturalization Act increased the required period of residence to qualify for citizenship from five years to 14 years. The Alien Friends Act allowed the President to imprison and deport any foreigners he considered to be dangerous. The Alien Enemies Act gave the President additional powers to detain foreigners during times of war or invasion.
The Sedition Act criminalized any “False, scandalous, and malicious” writing against the Government, Congress, or the President, or any attempt “to excite against them…the hatred of the good people of the United States, or to stir up sedition.” The punishment was a fine and imprisonment.
On its face, the Sedition Act was a violation of the First Amendment. The Alien Acts weren’t per se unconstitutional, but they certainly were worrisome and against the spirit of the founding of America. Madison and Jefferson weren’t wrong to oppose these acts.
However, how they chose to register their disapproval of the Alien and Sedition Acts was problematic, to put it mildly. They decided that the best avenue to attack the Alien and Sedition Acts was through the state legislatures. Thus, the man who had fought so hard at the Constitutional Convention to prevent the states from being able to invalidate any federally passed laws or measures, turned to the states to do just that when he disagreed with the federally passed laws in question.
In secret, Jefferson drafted what came to be known as the Kentucky Resolves, because they were intended to be adopted by the Kentucky legislature. (Side note: Kentucky became a state in 1792, made up of land that had previously been the far western territory of Virginia.) At the same time, James drafted similar resolves for the Virginia Assembly.
Jefferson’s main argument was that the federal government was delegated specific powers by the states, and when the federal government exceeded its mandate and assumed undelegated powers, its acts were “unauthoritative, void, and of no force.” He went further and added language that was later removed before the Kentucky legislature approved the Resolves, but it introduces a concept that we are going to come back to during Andrew Jackson’s presidency and again in the lead up to the Civil War: nullification. To nullify something is to render or declare it legally void.
Jefferson wrote, “where powers are assumed which have not been delegated, a nullification of the act is the rightful remedy; that every state has a natural right in cases not within the compact to nullify of their own authority, all assumptions of power by others within their limits.” He was basically saying that the states had the right to ignore any federal law that they didn’t think the federal government had the authority to enact. It was a dangerous concept, and it’s not hard to draw a straight line from nullification to secession. We tend to think of secession mostly as a Civil War problem, but various states threatened secession throughout early American history. It was especially problematic when the Union was still fairly new, as it was in 1798.
To his credit, Madison talked Jefferson out of including the nullification language, correctly pointing out the logical consequences of such a position. The Kentucky and Virginia Resolves were both passed by their respective legislatures, but other states declined to join them, and the Resolves were themselved challenged as unconstitutional. Massachusetts and other states passed competing resolutions declaring that the Alien and Sedition Acts were constitutional and within Congress’s right to pass.
Side note: for anyone wondering why the states are giving themselves the right to decide which federal laws are and are not constitutional, when we understand that today to clearly be the job of the Supreme Court, you need to know that the Supreme Court didn’t formally take on that role until 1803 in a famous constitutional law case known as Marbury v. Madison. And yes, the Madison is James Madison, who was being sued in his capacity as Secretary of State.
This is the case where Chief Justice John Marshall famously declared that it’s the judiciary’s role to “say what the law is” and asserted the Supreme Court's duty, assigned to it by the Constitution, to decide when laws passed by Congress are unconstitutional and therefore should be struck down. But in 1798, this had not yet been established, hence we have the various states declaring on their own what is and is not constitutional. You can see how this would create chaos.
Even George Washington knew how much trouble this could cause, writing to Patrick Henry, that the doctrines outlined in the Virginia Resolves “if systematically and pertinaciously pursued [would] dissolve the Union or produce coercion.”
Madison did not include the phrase “void and of no force” in his Virginia Resolves, because he understood that saying something was void was the same as saying it was nullified, and he didn’t actually want an individual state to have the power to ignore or void a federal law. What he wanted was for a collection of states to band together to oppose an offending federal law. He later claimed that by “oppose” he didn’t mean “ignore” but rather take collective action to convince Congress to repeal the offending law. Either way, it was a slippery slope to secession and disunion. At the Constitutional Convention, James had argued that the federal government should have the power to veto state laws. Now he was essentially making the opposite argument, showing that these guys didn’t really have any strong principles—they only believed certain things when it was convenient for them to believe them.
Other than making him an unprincipled hypocrite, Madison’s involvement in this scheme was not overly problematic because he was a private citizen. But Jefferson was serving in the administration as Vice President. He was actively scheming against the government he had sworn an oath to uphold and defend. It makes him more than a hypocrite. It makes him a traitor. It’s no wonder he kept his authorship a secret.
Pulitzer Prize winning Jefferson biographer Dumas Malone pointed out that Jefferson could have been impeached for treason if his involvement had been known. Referring to Jefferson’s statement that the Alien and Sedition Acts would “necessarily drive these states into revolution and blood,” Ron Chernow writes in his book Alexander Hamilton that Jefferson “wasn’t calling for peaceful protests or civil disobedience: he was calling for outright rebellion, if needed, against the federal government of which he was vice president.” We call that inciting an insurrection. We also call it treason. Or at least we should.
And just to put a cap on the Alien and Sedition Acts, both the Sedition Act and the Alien Friends Act expired in 1800. The Naturalization Act was repealed in 1802. But the Alien Enemies Act is still in force today, after being amended in 1918. That’s the one that gives the president the authority to detain foreigners in times of war or invasion. It was used in the War of 1812, which we’ll get to eventually this season, and also in World War I and World War II, most infamously to inter Americans of Japanese descent.
It’s also worth noting that as I write this in February 2026 there are active bills in both the Senate and the House of Representatives that would repeal the Alien Enemies Act. It’s called the Neighbors Not Enemies Act and is known as S.193 in the Senate and H.R.630 in the House. If you think giving the president the power to detain foreigners is a terrible idea, you should let your Congresspeople know that you support these bills. I’ll put a link to them in the show notes so you don’t have to write down those numbers.
Okay, so that was a pretty long detour into the Alien and Sedition Acts! But much like the background on the Constitution, this discussion lays the groundwork for things that are going to come up again, and fairly soon. Like later this season. So consider yourself warned.
Back to our main storyline: during this whole Virginia Resolves situation, James was persuaded to stand for election to the Virginia Assembly. He won easily, and he and Dolley relocated to Richmond for the short legislative session from December 1799 through January 1800. During this time, James defended the Virginia Resolves against attack, and mourned with the rest of the nation when George Washington died that December.
Dolley was excited to travel to Richmond, especially because she thought she might see her best friend, Eliza Collins Lee there. She was disappointed to find out that Eliza had not been able to travel to Richmond, writing her, “A prospect of meeting thee which is now clouded with disappointment, was my first inducement to visit Richmond—having heard that Mr. Lee’s family would certainly be with him. I have found the place, however, to my surprise, a most agreeable one—many ladies joine [sic] in my regret that you are not with us….Mr. L tells me you have 2 little ones—I would give the world to see them—why will you not come to Orange? Make an Effort my dear Girl to visit us—& if your heart is unaltered towards me you will Imagine the happiness such a favour would occasion—My time is so short that I can only add our affectionate salutations. Pray write me and kiss the children.”
The Madisons spent the remainder of 1800 back home at Montpelier. James fretted about crops and worked to ensure Jefferson’s election as president in the fall 1800 election. Dolley presumably continued hosting and entertaining guests.
As we discussed in both episode 2.13 and 3.5, Jefferson and Aaron Burr each received 73 electoral college votes in the 1800 election, throwing the election to the House of Representatives, as outlined in the Constitution. It took them 16 days and 36 rounds of voting, but they finally elected Jefferson on February 17, 1801.
James was unable to attend his dear friend’s inauguration on March 4, because his father had died on February 27 at the age of 77. Jefferson wanted James for his Secretary of State, and made the official appointment the day after his inauguration. But James’s ill health and the need to address his father’s estate prevented James and Dolley from leaving Montpelier until late April. They finally arrived in the capital city, which you’ll recall was now Washington, or Federal City, on May 1.
As we know from season 3, Jefferson had no wife or children to bring with him to Washington, so he invited the Madison family to live with him in the President’s House until they could find their own home. Nine-year-old Payne and 21-year-old Anna came with them. Dolley was just about to turn 33, and James was 50, and the spotlight was about to shine on them in a whole new way.
They stayed with Jefferson in the President’s House for their first three weeks in Washington, then moved to a row of houses called Six Buildings, which also housed the offices of the State Department, which was convenient for James. After two months at Six Buildings, they returned to Montpelier during Congress’s summer recess. When the Madisons returned to Washington in October 1801, they moved into a new home on F Street, NW, just two blocks from the President’s House.
In Strength and Honor: The Life of Dolley Madison, Richard Côté writes, “In Washington, [Dolley] would develop new political skills, prove her mettle, and ultimately demonstrate the heroism that would spread her fame throughout the world. Dolley Madison’s first official call to national service came from the President’s House just twenty-five days after her husband took office. Mr. Jefferson, it seemed, had a problem that neither he nor his secretary of state could solve. In his moment of need, the president turned to Mrs. Madison.”
Jefferson’s problem was that he was hosting an official dinner party, and two of the guests were women. Protocol demanded that a woman be on hand to act as hostess to the ladies, but as we know, Jefferson was widowed, and his daughters were not in Washington at this time. His Vice President, Aaron Burr, was also a widower, and Burr’s adult daughter had just gotten married and moved to South Carolina. So Jefferson turned to a woman he knew well to take on the hostess duties: Dolley.
He also would have accepted Dolley’s sister Anna, writing a note to Dolley that said, “Thomas Jefferson begs that either Mrs. Madison or Miss Payne will be so good as to dine with him to-day, to take care of his female friends expected.”
One of the guests at that first dinner party was Margaret Bayard Smith, the wife of Samuel Harrison Smith, who was the editor of a local Washington newspaper called The National Intelligencer. Martha Jefferson Randolph also befriended Margaret Bayard Smith, so you may recognize her name from last season.
Margaret wrote to her sister about the Madisons, “I am highly pleased with her; she has good humour and sprightliness, united to the most affable and agreeable manners. I admire the simplicity and mildness of Mr. Madison’ manners, and his smile has so much benevolence in it, that it cannot fail of inspiring good will and esteem.”
Dolley’s niece Mary Cutts later wrote of Dolley, “She was humble-minded, tolerant, and sincere, but with a desire to please, and a willingness to be pleased, which made her popular, and always a great friend and support to her husband. The power of adaptation was a life-giving principle in her nature, while an unusually attentive memory prevented her ever forgetting either names, faces, or the slightest incident connected with the personal history of any one.”
She may have been good with names and faces, but she was terrible with dates. For many years historians weren’t sure how old Dolley was at any given time or when she was born because she mixed up her dates and years when recounting her childhood history. The mystery of her birth year and also the spelling of her name were cleared up when the original Quaker records from the Paynes’ meetinghouse in North Carolina were discovered.
As we know from seasons 2 and 3, the city of Washington was not much to speak of in 1801 when the Madisons moved there. There were about 3000 residents, about 100 brick buildings and more than 250 shoddily constructed wooden buildings. The city was small, but growing, and builders were putting up residences as fast as they could, which meant attention to detail and sound construction fell by the wayside.
The roads in Washington were atrocious—filled with ruts and tree stumps. There was only one tailor, one shoemaker, one printer, one washingwoman, one grocery shop, one stationery shop, and one small dry-goods shop. And there was an oyster house, in case you’re not aware of how much the residents of the Chesapeake Bay area love their seafood.
The fact that Washington didn’t have much to recommend it turned out to be an advantage for Dolley. As Richard Côté puts it, “The relocation of the capital to Washington City was an enormous social boon for Dolley. Had either of the nation’s first two capital cities, New York and Philadelphia, become the permanent seat of government, Dolley would have been at a serious disadvantage. Both of those places had long-established power structures and well-entrenched dynastic families who ruled political and social life. As a former Virginia country girl who had recently married into the political and plantation aristocracy, Dolley would have been an outsider, excluded from spheres of influence.
Washington City was perfect for her because it was virtually a blank slate. …She came to Washington in 1801 a perfect stranger. By the time her husband finished his service as president sixteen years later, Dolley had met an astonishing number of people and was arguably the best-known woman in America.”
Their closest friends during the early days in Washington, besides Jefferson, were the Gallatins and the Thorntons. Albert Gallatin was a Swiss-born U.S. resident who served as Jefferson’s Treasury Secretary for both of his terms. His wife Hannah was a member of New York society. She was described as “the most stylish woman in the drawing room [and dressed with] more splendour than any of the noblesse.”
Dr. William Thornton was a native of the West Indies. He was a Quaker multi-hyphenate, even though they didn’t have that word back then. He was an architect, physician, painter, and inventor. He designed both the U.S. Capitol Building and a residence called The Octagon House, where the Madisons will live after the President’s House is burned by the British in the War of 1812. Thornton’s wife Anna once referred to Washington City as “dull tho’ great.”
As I mentioned in episode 3.6, Jefferson attempted to dramatically scale back the style of presidential entertaining. He felt that the weekly gentlemen’s levées and ladies’ drawing rooms were far too monarchical for his fine Republican sensibilities. Never mind that those social engagements had never been George and Martha Washington’s idea in the first place. Anything from the first presidential administration that smacked of monarchy had been put there by Congress when everyone was making their way in the brave new world formed after the ratification of the Constitution. John and Abigail Adams had merely continued the traditions established by the Washingtons, despite what Jefferson may have said about it.
But Jefferson did away with both the levées and the drawing room receptions, and he hosted small dinner parties, usually with only 10 to 12 guests in total. He hosted dinners nearly every night, and Dolley and James were often in attendance. They dined with Jefferson both in James’s official capacity as Secretary of State and also as personal friends.
In case I didn’t make clear last season how much I dislike Thomas Jefferson and his sexism, here’s another quote to make my point: “The tender breasts of ladies were not formed for political convulsions, and the French ladies miscalculate much their own happiness when they wander the true field of their influence into that of politicks.[sic] Our good ladies, I trust, have been too wise to wrinkle their foreheads with politics. They are contented to soothe and calm the minds of their husbands returning from political debate. They have the good sense to value domestic happiness above all other.”
If this weren’t a family podcast, I’d have some choice words for old Thomas Jefferson. It never ceases to amaze me that both Abigail Adams and Dolley Madison considered Jefferson a good friend for long periods of time. Leaving aside the fact that Abigail and Jefferson fell out over political differences, she reconciled with him later in life. Abigail was clearly a woman who had many thoughts about politics, and was happy to wrinkle her forehead over them. Dolley was less overtly political, but she was most certainly interested in politics, even if she didn’t voice her opinions as freely as Abigail did.
In 1805, during Jefferson’s second term, Dolley and James were separated for a time while she was in Philadelphia being treated for a medical condition. We’ll get to that next week. But what’s relevant now is that she wrote to James, asking for information about the current political climate:
“I wish you would indulge me with some information respecting the war with Spain and disagreement with England, as it is so generally expected here that I am at a loss what to surmise—you know I am not much of a politician but I am extremely anxious to hear (as far as you think proper) what is going forward in the Cabinet—on this subject, I believe you would not desire your wife the active partizan [sic], such as her neighbor Mrs. L., nor will there be the slightest danger whilst she is conscious of her want of talents, and her diffidence in expressing her opinions always imperfectly understood by her sex.”
So clearly, Dolley was no Abigail, who so famously counseled her husband to “remember the ladies” when the laws of a newly independent America were being constructed. Dolley is much more self-deprecating when it comes to her ability to understand politics, and she even refers to herself in the third person to put some distance between her and her request for information. But she clearly is interested in politics, and that probably contributed to her skills as a hostess and as a provider of welcoming spaces where politicians of all stripes could feel comfortable together. It also gives her personality an important dimension, because without it, she might just be a politician’s wife who cared about nothing but fancy clothes and interior decorating.
And even Dolley knew that Jefferson sometimes went too far in trying to impose his strict Republicanism on diplomatic protocols. There was a bit of a kerfuffle after Jefferson pointedly offended the British minister’s wife. In his efforts to make America as little like Europe as possible, Jefferson published what he referred to as the Canons of Etiquette to be Observed by the Executive. One of the canons was that guests would be seated to dinner in what was known as the pêle-mêle style. This is a French term meaning “mixed up” and it’s been brought over to English with a different spelling but basically the same pronunciation. Anything done pell-mell is done in a disorderly fashion.
Instead of guests being escorted into dinner in accordance with centuries of diplomatic protocol and then being seated at carefully determined assigned seats, Jefferson wanted everyone to proceed into dinner in whatever order they wanted, and then to find their own seats at the table. You can see how this would cause both confusion and offense to people who are used to doing things “a certain way.”
To give you a sense of what the protocols were, I’ll turn to Richard Côté once again. “The European diplomatic corps operated by complicated rules of protocol based upon diplomatic rank and the current friend-or-foe status of each country. High-ranking ministers and their wives, for example, would be seated closest to the host, with deputies and aides at the far end of the table. Titled nobility would be seated by descending rank, with untitled guests at the end of the table or the back of the room. Ministers of nations at war with each other would not usually be invited to the same event. …These social protocols were followed at both private and state events. The rules of etiquette dictated who visited whom first, who was invited to what event, who was introduced first and by whom, who escorted whom where, and who was seated where. Violations of protocol could be, and often were, taken as national insults, and a failed introduction or improper invitation or seating arrangement could provoke an international incident.”
It all sounds rather exhausting, doesn’t it?
But back to the kerfuffle. When British minister Anthony Merry called on President Jefferson in November 1803 in full regalia to present his credentials, he was flabbergasted to discover that the President received him wearing bedroom slippers. A few days later, Ambassador and Mrs. Merry were guests at a dinner party where Dolley was serving as hostess. They were first flummoxed to find the French chargé d’affaires there, because France and Britain were currently at war. And then Mrs. Merry was offended beyond belief when it came time to go in to dinner and Jefferson offered his arm to Dolley instead of to Mrs. Merry, as protocol demanded.
Dolley knew that he was in error, and even whispered to him, “Take Mrs. Merry.” But Jefferson stubbornly persisted and it was left to James to escort Mrs. Merry to dinner. We’re lucky the Brits didn’t declare war on us right then and there over this overt snub.
I’ve now gone on way too long, so next week, we’ll pick up with the rest of James and Dolley’s time as Secretary of State and unofficial Presidential hostess. We still have a lot to cover before James even becomes President!
Thanks for listening to this episode. It was produced by me, and the music is by Matt Dull. If you’re enjoying it, please recommend it to a friend. Thanks!
