Episode 4.1 Little Dolley Payne

[Transcript]

Hello, and welcome to America’s First Ladies. I’m your host, Risa Weaver-Enion.

Episode 4.1 Little Dolley Payne

Welcome back to season four of the podcast! We are diving into the life of Dolley Madison and her husband James Madison this season. I am going to switch up the format just a bit this season though, because Dolley and James didn’t meet until fairly late in his life. A lot of the interesting things he did, such as his service in the Continental Congress and at the Constitutional Convention, happened before he even met Dolley. I don’t want to skip over those things, especially the Constitutional Convention.

So we’re going to spend today and next week covering Dolley’s early life and first marriage, and then we’re going to say a temporary goodbye to Dolley and focus solely on James for episodes 3 and 4. Then we’ll pick up the narrative of Dolley again in episode 5 when she finally meets James.

I feel like I had to miss out on some pretty important aspects of Thomas Jefferson’s presidency last season because his wife Patty was already dead, and his daughter Martha didn’t spend all that much time with him in Washington during his presidency. There wasn’t a good way to work in the Jefferson presidency to Martha’s life.

But the good news is that Dolley Madison served as President Jefferson’s hostess when his daughter wasn’t in town, which was most of his presidency, and James Madison was his Secretary of State for all eight years. So I plan to cover the Jefferson presidency in some detail because the Madisons were so involved.

First though, let’s get to know Dolley Payne Todd Madison. As you can probably guess by the number of names I just included, James Madison was her second husband, a fairly common situation among our subjects thus far. At this point, Abigail Adams has been the only First Lady who was only married once. But let’s start at the beginning, in the woods of the Piedmont of North Carolina, where Dolley was born in 1768.

The Payne family was not native to North Carolina. Like most of our subjects so far, they were Virginians. Also, for the record, since this is a podcast, Payne is spelled P-A-Y-N-E.

Dolley Payne was born May 20, 1768, the third child and first daughter of John Payne and Mary Coles Payne, who had married in 1761. John and Mary were both from well established Virginia families. 

John Payne’s parents, Josias and Anna Payne, were Anglicans, like most Virginians, and had settled in Goochland County, along the northern shore of the James River, north of Richmond. Josias was a wealthy plantation owner, which of course also means that he was a slave owner. His English family had been in Virginia for at least two generations.

Anna Payne’s grandfather had been Scottish, and he had emigrated to the new colony of James Town, Virginia in around 1616. Anna Payne also claimed that she was descended from Pocahontas, but that’s not confirmed.

Mary Coles was the daughter of an Irish immigrant named William Coles, who had come to Virginia with his brother John Coles, probably early in the 1720s or 30s. They were originally from County Wexford. Mary’s parents, William and Lucy, settled in Hanover County, and were also plantation owners.

Interesting side note, Mary’s mother, Lucy Winston, had a sister named Mary Winston. Mary Winston married a Virginian named John Henry and gave birth to a child whose name will be very familiar to you—Patrick Henry. Mr. “Give me liberty or give me death” himself.

Historians often refer to Patrick Henry as Dolley Madison’s cousin, but to be technical about it, he was her first cousin, once removed. Dolley’s mother was first cousins with Patrick Henry, not Dolley herself. And I drew the geneaology charts just to make sure!

So back to Dolley’s parents. John Payne was born in February 1741 on his parents’ plantation in Goochland County. Mary Coles was born in October 1745 on her parents’ plantation, Coles Hill, in Hanover County. Goochland and Hanover Counties were near one another, and presumably all the wealthy plantation owners ran in the same circles, which is probably how John and Mary met. They were married in 1761, when John was 20 and Mary was just 16. The newlyweds were given 200 acres of land by John’s uncle, so they got off to a strong financial start.

Now, we need to take a slight detour to talk about Quakerism, because Dolley was born a Quaker, and it is an important part of her story, but we need to understand more about it for it to all make sense. The Coles family and the Winston family were both Quakers, which means Mary Coles Payne was born a Quaker. However, John Payne was Anglican like the rest of his ancestors, and the Quakers did not take kindly to marrying outside the faith. Upon marrying John, Mary was disowned by her Quaker “meeting,” which is what Quaker congregations were called, even though they didn’t have churches and clergy like most other religions.

So, what is Quakerism? As we discussed in Season 2 when I explained John and Abigail Adams’s Congregationalist faith, 17th century England saw the rise of a number of religious offshoots, aiming to “purify” the Anglican Church of its Catholic vestiges. The Society of Friends was founded around 1650 by a man named George Fox. He preached that all people have an “inner light,” and that no one was greater or lesser than another. To demonstrate equality among people, he referred to everyone as “Friend.” 

As with most small religious offshoots, the Society of Friends was ridiculed and derided by the English public, who for some reason decided to call them Quakers. Wikipedia claims that it’s because George Fox once told a judge to “quake before the authority of God,” but unless he said it repeatedly, that seems like a thin reed to support calling the entire religion Quakers. But that’s how they became known, and eventually the Society of Friends embraced the term themselves.

Quakers believed in the equality of women, compassion for the mentally ill, the abolition of slavery, and a refusal to bear arms. Those last two beliefs put them at a distinct disadvantage in Virginia and also during the Revolutionary War, when they refused to fight for either side.

Quakers also believed in simplicity. The Quaker founder of the colony of Pennsylvania, William Penn, famously said, “Let thy garments be plain and simple….If thou art clean and warm, thy end is accomplished; to do more is to rob the poor.” In keeping with this tenet, Quaker Friends were not supposed to wear expensive clothes, jewelry, or wigs, were not allowed to dance, were prohibited from owning taverns and selling liquor (but they could drink alcohol sparingly), and were expected to adhere strictly to the truth at all times. 

They did not tip their hats or bow or curtsey to anyone, and they also used no titles. No “My Lord,” “Reverend,” or even “Sir” or “Madam.” Everyone was addressed simply as Friend and was referred to as thee or thou. They even wrote the date differently because the names of the months were derived from paganism. So instead of writing January 15, they would write 15th day, first month.

They also didn’t meet in churches for guided sermons given by a clergyperson. They had meetinghouses, which were, of course, plain and simple, and the Friends gathered there for long meetings where anyone was free to speak if the Holy Spirit moved them. There were no hymns sung, no formal prayers recited, no scripture readings. Long periods of the meeting were spent in silence and quiet reflection by the members.

Members could be disowned by their “meeting” for a number of reasons. The Quakers kept detailed record books, so we have documentary evidence of some of those reasons, outlined by Richard Côté in his book, Strength and Honor: the Life of Dolley Madison: neglecting to attend meetings, joining the Masonic society, attending a Baptist church, fighting, using ill words, uttering profane expressions, drunkenness, adultery, moving out of state without first settling with creditors, taking solemn oaths, engaging in military service, attending muster, marching in procession with music and weapons of war, following in the corrupt ways of the world, partaking of the vain fashions and customs of the world, laying a wager, hiring a slave, purchasing or receiving a manumitted former slave, abusing a slave, selling a fellow human being into slavery, or working as an overseer of slaves.

Friends also were not permitted to marry at home, only in a meetinghouse. They were not permitted to marry a non-Quaker. And if their spouse died, they were required to wait a suitable amount of time, usually at least a year, before remarrying. Those rules would eventually become very relevant to Dolley.

As for her parents, like I mentioned earlier, Mary Coles was disowned by her meeting when she married John Payne, a non-Quaker. And one could not simply rejoin one’s former meeting. There was a very involved application process, one that John and Mary Payne undertook in 1764, about three years after their marriage. Applying for membership to a meeting required a formal letter of application, confession of one’s trangressions, proof of repentance for those transgressions, and an investigation into the applicant’s life and background. A women’s committee was appointed to look into Mary, and a separate men’s committee was appointed to look into John.

They were finally admitted to the Cedar Creek meeting, which was Mary’s former meeting, on May 10, 1765. John and Mary were both officially Quakers, and all their subsequent children would be born into the Quaker faith, including Dolley.

About six months later, in the autumn of 1765, the Payne family moved from the colony of Virginia to the colony of North Carolina. They had a young son already, and Mary was pregnant with their second child. The reasons for the move probably related to the growing population of Virginia, which not only drove up land prices, but also forced Quakers into closer society with non-Quakers, which they didn’t love. And also because the Anglican Virginians were about as tolerant of the Quakers as the English had been. Which is to say, not very. Granted, the Quakers weren’t very tolerant of other religions either. See the list of reasons for disowning a member, which included attending a Baptist church.

The Paynes were not alone in relocating to North Carolina, and they applied to be members of the New Garden Meeting, which was located in what is today Guilford County, near Greensboro, North Carolina. The Paynes brought with them a letter of good standing from their Virginia meeting, which smoothed the transition. Unlike the months-long process to join the Cedar Creek Meeting, their application to the New Garden Meeting was accepted the same day they presented their certificate of good standing, November 30, 1765.

The following June, Mary gave birth to her second child, another boy. And then in 1768, she gave birth to our subject, Dolley. Once again, for the record, Dolley is spelled D-O-L-L-E-Y. If you’re familiar with the well-known brand of snacks called Dolly Madison, which, yes, is named after the woman Dolley Madison, you may think that Dolley is spelled without the E, because that’s how Dolly Madison the brand is spelled. It turns out that there was a lot of historical confusion over what Dolley’s name actually was and how it was spelled. But this mystery has subsequently been cleared up, and we can say with certainty that her name was always Dolley, never Dorothy or Dorothea, and that it was always spelled with an E.

Glad we cleared that up!

When Dolley was born, she had two older brothers: Walter, who had been born in Virginia, and William, who had been born in North Carolina. The family lived in a log cabin that comprised two rooms with a fireplace in between them. Presumably one room was for cooking and living and the other was for sleeping. There was also a staircase leading to an attic level where there was room for storage and additional sleeping space.

Baby Dolley was taken care of by an enslaved nursemaid called Mother Amy. Wait, I can hear you asking: the Paynes owned slaves but were still allowed to be Quakers? I had the same question. Despite being in favor of the abolition of slavery, and despite all those slave-related activities that could get one disowned by their meeting, apparently owning slaves wasn’t prohibited, as long as you didn’t abuse them.

Because the Paynes had a 200-acre plantation in Virginia, they also had slaves. There’s no solid record of how many slaves they owned, but historians have estimated that it could have been as many as 50, which would have made them major Virginia slaveowners. In the 1760s, it was actually illegal in Virginia to free a slave, which was also known as emancipation or manumission. I use all terms interchangeably.

The Paynes had taken at least some of their slaves with them to North Carolina, even though they presumably did not have 200 acres of land in their new home. The slaves returned with them to Virginia in 1769, at which point John Payne wrote, “I am persuaded that liberty is the natural condition of all mankind.” He wanted to free his slaves at that point, but legally could not.

It’s unclear why the Payne family only stayed in North Carolina for three years, but there is plenty of speculation. John Payne had sold his Virginia land before leaving that colony, and with land prices rising due to demand, he probably made out pretty well from the sale. He purchased three large tracts of land in North Carolina, and probably used his enslaved workers to farm those lands. He also held himself out as a merchant, but there’s no record of what he might have been dealing in. 

The Paynes had to apply to the New Garden Meeting for a certificate of good standing to take with them back to Virginia, and of course that required a whole new investigation. Because the certificate wasn’t issued for three months, it seems that the investigation might have turned up something untoward, perhaps debts or some sort of failed business endeavor. At any rate, once the certificate was issued, John sold his North Carolina properties at a loss, and the family headed back to Virginia, where they were received back into the Cedar Creek Meeting on April 8, 1769.

So although Dolley was born in North Carolina, she only lived there for less than a year, and always identified herself as a Virginian.

Upon their return to Virginia, in reduced financial circumstances, they lived at Patrick Henry’s plantation called Scotchtown in Hanover County. Scotchtown was massive—960 acres of land, and a house of at least 9000 square feet, more than enough room to accommodate a few extra people. The first floor was divided into eight rooms, and the second story was one large, open attic space. It also had an underground basement, which was divided into another eight rooms, including a wine cellar.

The family stayed at Scotchtown probably until September 1771, when records show that John Payne purchased 176 acres of the Coles Hill plantation from his father-in-law, William Coles. This is where Dolley lived from age 3 until age 14 or 15. The house was a modest two-story dwelling. The ground floor was divided into two large rooms separated by a wide hallway. Bedrooms with dormer windows were on the second floor. There was a broad porch where the family could enjoy summer evenings, and, of course, slave cabins.

Living on a working farm, Dolley would have had plenty of chores and housework to do as a child. She probably learned to cook and sew, as well as garden and tend to farm animals. Quaker children had a strict upbringing. Dancing, racing, and competitive games were prohibited. Girls were allowed to have dolls, but there were few toys other than that. 

Dressing as a Quaker was even less fun. Plain, drab clothes were required. No ruffles, frills, ribbons, petticoats, or colors. This probably goes a long way toward explaining why Dolley was such a clotheshorse later in her life, after she left the Quaker religion.

Dolley’s Anglican grandmother, Anna Fleming, seems to have given Dolley gifts of old-fashioned jewelry from time to time. This would definitely have been prohibited by Quaker rules, and according to family tradition, Dolley carried the jewelry in a small bag tied around her neck where no one could see it. She was devastated when the bag came loose one day, and all her treasures were lost.

The Payne family’s second term of residence in Virginia lasted from 1769 until 1783, when they moved to Philadelphia. There are very few records of the family from this time period, and nothing relating directly to Dolley. These years would have been prime learning years, but Dolley probably received very little formal schooling. She almost certainly learned the usual housekeeping, cooking, and sewing skills from her mother. And we know she learned fancy needlework because some of her needlepoint work survives. 

According to family tradition, Dolley attended a log schoolhouse in Hanover County when she was young, and her parents are listed on the subscribers’ list for the school offered by their meetinghouse, but there’s no indication of which child or children were enrolled at the school. Richard Côté describes Dolley’s writing and penmanship this way: 

“Dolley’s earliest letters reflect a strong, legible, but ragged and unpolished hand. Her letters display the random capitalization and phonetic spelling common to middle-class citizens of her time. … Her later handwriting became more regular, practiced, and disciplined, but her letters still contained run-on sentences, frequent use of terminal dashes to the near-exclusion of periods, and sentences that frequently ran very much downhill.” After reading some of her letters for myself, I can confirm what he says about the run-on sentences!

As Dolley’s parents continued to have more children, Dolley undoubtedly would have helped to take care of them. Mary Coles Payne gave birth to at least five more children during these 14 years in Virginia: Isaac in 1769, Lucy in 1777, Anna in 1779, Mary in 1781, and John in 1782. One of Dolley’s early biographers posits that three children with the last name of Paine, spelled P-A-I-N-E, and buried in a local Quaker meeting burial ground were quite possibly deceased siblings of Dolley’s. We know how high the infant mortality rate was in these days, so it would be surprising if the Payne family didn’t have any children die as babies or toddlers.

You’ll note that the years the Payne family spent in Virginia overlap with the tensions leading up to the Revolutionary War and the war itself. Because Quakers were prohibited from engaging in military service, Dolley’s father played no part in the war. Shortly after the Declaration of Independence was signed in 1776, a national gathering of Quaker leaders directed all meetings that their members should neither perform military service nor pay a fine in lieu of said service, should not engage in any trade or business that promoted war, and should help all war victims. Quakers were also supposed to withdraw from all political activities—even voting.

The national meeting of Quaker leaders also decided that enough was enough when it came to slavery. All local meetings were directed to “discourage the iniquitous trade” of selling and owning slaves. John Payne took this to heart and spent the next six years gradually freeing all his slaves, despite the fact that it was still illegal under Virginia law to do so. A number of his fellow Quakers did the same.

This was morally the right thing to do, but economically it was a problem. Without an enslaved workforce, running plantations became economically infeasible. Most plantation owners, John Payne included, did not have the kind of cash flow that would enable them to hire paid laborers. It’s quite possible that some formerly enslaved workers stayed on at the plantations and worked under sharecropping arrangements. They could continue to live in their former quarters on the property and receive seeds and farm tools from the plantation owner, but in lieu of being paid wages, they would keep a portion of the crops produced as payment for their services. They could then sell the crops for cash. The rest of the crop would be sold by the plantation owner and the proceeds kept by him. 

Many Quaker Virginians who freed their slaves decided to abandon Virginia plantation life and move west into sparsely populated areas. The Paynes did the opposite—they moved to the second largest city in America: Philadelphia. At least one of their emancipated slaves chose to go with them—Dolley’s former nursemaid, Mother Amy. She lived with and worked for the Paynes for the rest of her life, earning wages as a paid servant.

Philadelphia had been founded in 1682, shortly after the Quaker William Penn was given the royal land grant to Pennsylvania. It was still a bastion of Quakerism, and John and Mary Payne thought it was the ideal place for their children to be “educated in their religion” and for John to transition from slave-holding plantation owner to urban merchant.

The family applied for certificates of good standing from the Cedar Creek Meeting, which were granted on April 12, 1783. Sometime that summer, the Payne family headed north. There are no records of their journey, so we don’t know which route they took or how long it took them to travel the 200 miles to Philadelphia. But there’s no question it was an arduous journey. Roads were atrocious in those days, and travel by wagon was slow, painful, and dangerous. 

One observer at the time wrote, “Outside Philadelphia lay black and treacherous quagmires in which the horses floundered and struggled for hours, making no progress towards getting out, while some of the hills were so steep that wagons must pause till other teams came to their assistance. These wagons had no springs, and the unlucky passengers were jolted from side to side as the wheels of the vehicle rolled over rocks or sank to the hubs in mud.”

Now imagine doing this while eight or nine months pregnant at the age of almost 40 and you’ll put yourself in Mary Payne’s shoes. No thank you.

Dolley would have just turned 15 by the time they arrived in Philadelphia, and her world was about to get a lot bigger. By 1783, Philadelphia had a population of about 22,000 people, 4,000 grand homes, rows of trees lining the paved streets, and actual sidewalks made of raised brick. It must have been amazing for a country girl to behold such magnificence.

On the other hand, it was a large city, and with large cities come large problems and lots of smells. Richard Côté quotes a vivid description of Philadelphia from John Harvey Powell’s book about the 1793 yellow fever epidemic:

“A low, level town, hottest and dampest of all the American seacoast, hotter even than Charleston, Savannah, or the West India cities. Wharves jutted out into the river and cut off the current; high tide deposited rotting stuff on the banks and in the mud. Below the city were swamps, marshes, pools in clay pits, stagnant water. Most of the streets were unpaved. There was no water system, and only one sewer, under the serpentine Dock Street. Elsewhere holes were dug, as at Market and Fourth streets, to receive water from the gutters. These ‘sinks’ exhaled a noxious effluvia, for dead animals and all kinds of nauseous matters were hurled into them to putrefy. All the wells were shallow; citizens continually pronounced them polluted.”

If time travel is ever invented, I can assure you that I will not be going back to pre-indoor plumbing Philadelphia, or anywhere else for that matter.

Upon their arrival in the city, the Payne family stayed with Elizabeth and Henry Drinker and their family. The Drinkers were a prominent Quaker family, and Henry Drinker was a partner in a shipping and importing firm. Mary Payne had visited Elizabeth Drinker in Philadelphia at least twice before the Payne family decided to move there. Shortly after arriving at the Drinkers’ in Philadelphia, Mary Payne gave birth to her final child, a girl they named Philadelphia in honor of their new home. 

Sadly, within a few years of moving to Philadelphia, the Paynes lost two of their children. First, their eldest son Walter went missing and was presumed lost at sea on his way to Britain in December 1784. And then their daughter Philadelphia died, probably some time in 1786, because that’s the last year in which there is any mention of her.

But on the upside, the Payne family moved into their own home in 1785. It was a row home consisting of several three-story dwellings. Each home had a small porch out front where one could sit and people-watch. John Payne had gone into business selling laundry starch, and he probably conducted business from the front rooms on the ground floor. The second floor likely contained a parlor and the master bedroom, with additional bedrooms on the third floor. The kitchen was located in a separate building at the back of the house.

At the time the Paynes moved into this house, they had eight living children ranging in age from infant to age 17. The house was probably not large enough to comfortably hold them all, and the Payne family would never be as prosperous as they had been before the war, when they were still plantation owners.

Dolley was an immediate success with her fellow Quakers. Years later, her lifelong friend Anthony Morris recalled her entry onto the Philadelphia scene. I’ll note in advance that he refers to her as being 16 and the month being May, but she was actually just 15 when they first arrived in Philadelphia, and they probably arrived in late June or early July. So he might not have met her until the following year. Also, when he says “Society of Philadelphia” the word “Society” is capitalized because he’s referring to the Quaker Society of Friends specifically, not the general society of Philadelphia. The limits of audio make themselves known once again.

“She came upon our comparatively cold hearts in Philadelphia, suddenly and unexpectedly with all the delightful influences of a summer sun, from the Sweet South, in the season of May and at the age of sixteen, bringing with her all the warm feelings, & glowing fancies of her Native State. She was the first and fairest representative of Virginia in the female Society of Philadelphia, and she soon raised the mercury there in the thermometers of the Heart to fever heat. But she was not altogether in appearance a Virginian, her complexion seemed from Scotland and her soft blue eyes from Saxony.”

It sounds like someone had a crush!

Next week, we’ll dive into the Society of Philadelphia and young Dolley. The documentary evidence of her life picks up at this point, and we have quite a bit of information about how she spent her teenage years.

Thanks for listening to this episode. If you enjoy the podcast, please recommend it to a friend. As always, this episode was produced by me, and the music is by Matt Dull.