Prince Harry reportedly considered changing his family’s surname to his late mother Princess Diana’s maiden name, Spencer, but was advised against it by his uncle, Charles Spencer.

According to The Mail on Sunday, the Duke of Sussex, 40, had contemplated abandoning the Mountbatten-Windsor surname — the one currently used by his children, Prince Archie and Princess Lilibet — in favor of his mother’s family name. During a rare visit to the U.K., Harry discussed the possibility with Diana’s brother, the 9th Earl Spencer, but was told it would be legally complicated.

“They had a very amicable conversation and Spencer advised him against taking such a step,” a friend of Harry told the publication.

Representatives for both Prince Harry and Earl Spencer did not comment when contacted by PEOPLE.

Harry and Meghan’s two children, Archie, 6, and Lilibet, who turns 4 on June 4, are officially registered with the surname Mountbatten-Windsor on their birth certificates. However, the family now informally goes by Sussex, following royal tradition — much like Harry was known as “Harry Wales” growing up, based on his father’s title at the time.

In a past interview with PEOPLE, Meghan reflected on how significant the Sussex name had become for their family. “It’s our shared name as a family, and I guess I hadn’t recognized how meaningful that would be to me until we had children,” she said. “I love that that is something that Archie, Lili, H and I all have together. It means a lot to me.”