By: sam doak
arron williams
November 7 2022
Benford’s Law: The questionable statistics of election deniers
Share Article:
By Sam Doak and Arron Williams
On November 30, 2022, Brazilians voted in the second round of the presidential election. Ultimately, incumbent Jair Bolsonaro was defeated by his leftist opponent Luiz Inácio Lula da Silva, known simply as Lula, by a margin of over two million votes. Lula's victory was swiftly recognized by both domestic institutions and figures on the international stage. Despite this, figures on the American right have denounced the result and made unfounded claims about voter fraud and suppression. With the U.S. midterms this week, many fear that anti-democracy forces will attempt to use bogus statistical analysis to drum up fake evidence for election fraud. It's worth examining how these claims hold up in the wake of the Brazilian elections, and how they might be used again.
Much of the right-wing discourse concerning the Brazilian election features an anonymous report concerning voting patterns and Benford's law, a mathematical rule that can be used to analyze and detect potential fraud cases; in this case, supposed election fraud. The report first surfaced on Creative Destruction Media (CDM), a fringe right-wing news site, accompanied by an article by Matt Tyrmand. Tyrmand explains that it was "created by a small group of technical experts in the fields of mathematics, political science, and forensic analysis, all of who are well versed in election statistics and electoral anomalies." In a nutshell, the report claims that the analysis of results from the first round of the Brazilian presidential election suggests that this vote was unduly influenced.
Tyrmand, a former contributor to Breitbart and board member at Project Veritas, subsequently discussed this report on a livestream hosted by Steve Bannon on GETTR. During this conversation, he told viewers that the same anonymous experts had found similar irregularities in the second round of voting, stating, "I'm talking to my experts who penned this study that this dossier that we posted on CDM Press after the polls closed about round one. They say it's the same, same exact curve, same exact numbers. It looks exactly the same. Same programming language, so to speak, that was executed in this round two as in round one."
Plagiarism, Sources, and Framework
While Tyrmand and Bannon have framed this report as an electoral smoking gun, even a small amount of scrutiny is enough to cast serious doubt on the qualifications of its authors and the validity of its findings. One particularly blatant issue is that sections of the report appear to have been copied verbatim from existing extracts.
In an opening section written to explain the use of Benford's law, the authors write, "Benford's Law, also called the law of the first digit, or Newcomb-Benford’s Law, and the law of anomalous numbers, refers to the distribution of digits in various sources of real cases. Instead of expected homogeneity, the law states in many naturally occurring numbers collections the first significant digit is likely to be small. Without homogeneity, the distribution shows digit 1 has probability of appearing 30% in a statistical data set, while larger values are less likely to appear."
While the exact text of the definition they use in the anonymous report is unremarkable to a layperson, what is notable is that it appears to be reworded from writing posted on Github by user cjph8914 in November 2020. This post employs a similar analysis, applying Benford's Law to the results of the U.S. presidential election. Screenshots of the charts included were subsequently shared widely by Trump supporters, who erroneously claimed they were proof of electoral fraud. Perhaps unsurprisingly, a search of the text included in this post indicates that it was lifted from the Wikipedia entry for Benford's Law.
|
Brazil report |
||
|
Benford's law, also known as the Newcomb–Benford law, the law of anomalous numbers, or the first-digit law, is an observation that in many real-life sets of numerical data, the leading digit is likely to be small.[1] In sets that obey the law, the number 1 appears as the leading significant digit about 30% of the time, while 9 appears as the leading significant digit less than 5% of the time. |
Benford's Law, also called the Newcomb–Benford law, the law of anomalous numbers, or the first-digit law, is an observation about the frequency distribution of leading digits in many real-life sets of numerical data. The law states that in many naturally occurring collections of numbers, the leading digit is likely to be small. For example, in sets that obey the law, the number 1 appears as the leading significant digit about 30% of the time, while 9 appears as the leading significant digit less than 5% of the time. |
Benford’s Law, also called the law of the first digit, or Newcomb-Benford’s Law, and the law of anomalous numbers, refers to the distribution of digits in various sources of real cases. Instead of expected homogeneity, the law states in many naturally occurring numbers collections the first significant digit is likely to be small. Without homogeneity, the distribution shows digit 1 has probability of appearing 30% in a statistical data set, while larger values are less likely to appear. |
A table comparing the report's text with Github and Wikipedia.
While copying from a text that is itself ripped from Wikipedia is not conclusive evidence that a report has not been drafted by a "group of technical experts in the fields of mathematics, political science, and forensic analysis," it does raise questions. It seems unlikely that a group that fits the description provided by Tyrman would commit such obvious plagiarism, least of all from a Github post about supposed U.S. election fraud. This is not the only instance of plagiarism in this report. Another piece of text in the introductory section is lifted from Wikipedia, albeit directly in this instance.
|
Second paragraph of the report’s introduction |
|
|
It has been shown that this result applies to a wide variety of data sets, including electricity bills, street addresses, stock prices, house prices, population numbers, death rates, lengths of rivers, and physical and mathematical constants. |
Frank Benford has shown this result applies to a wide variety of data sets, including electricity bills, addresses, stock prices, private equity prices, population numbers, death rates, river lengths, physical and mathematical constants, by power laws (which are very common in nature). |
A table comparing the report's text and Wikipedia.
The report also uses the New York Times as a source when discussing Benford's Law, provides no sources to support its methodology, and uses few sources elsewhere. The only sources that are provided are public, suggesting a lack of authenticity. While using a government website to obtain voting data is not suspicious, as it contains relevant information, there are no academic or research sources to discuss either Benford's Law or the report's methodology. This is suspicious, as experts would likely have access to research resources and wider knowledge. It is unlikely an expert on Benford's Law or mathematics, for example, would rely on citations from the New York Times. This lack of citation and discussion casts doubt on the validity of the alleged report as it shows no greater awareness of the framework it uses.
Another flaw in the report is that it does not reveal who was behind the analysis beyond the article on CDM, claiming that it was a group of technical experts. While the information was possibly removed for the sake of anonymity, this has not been stated as such. As it stands, the lack of recognized authors gives further suspicion to the credibility of the paper as "experts" supposedly wrote it, but there is no ability to verify this.
Benford's Law and Fraud Stats
Benford's Law is never definitive proof of fraud, a fact that Tyrmand acknowledges. However, the report itself does not address this, instead asserting it should be considered definitive evidence, and declaring that the best course of action would be an audit. However, more specifically outside this omission, it is not just the case that Benford's Law cannot give a definitive answer, but rather that academic sources and research suggest more significant issues of its validity, especially when applied to election fraud.
For example, a 2010 paper in the German Economic Review states that Benford's Law has issues with false positives and negatives, and is questionable when discriminating between manipulated and unmanipulated data. A 2011 paper in the journal Political Analysis looked at simulations modeling both fair and fraudulent elections and found that Benford's Law attributed judgment essentially at random. The paper states that as a forensic tool, Benford's Law is problematic at best when analyzing elections, specifically questioning its capacity to judge election fraud as a forensic tool.
A 2020 Reuters fact check investigated claims of U.S. election fraud being proven by Benford's Law and found that these claims were not supported. In order to investigate this, they reached out to a variety of experts, who echoed similar statements to those made in the Journal of Accountancy: that the law cannot be used to prove fraud and that a significant result can be caused by other complications. Reuters quotes Dr. Jen Goldbeck, a professor in computer science, stating that "there is no solid proof that Benford works in elections at all." Ultimately, the outlet concluded that the extent of Benford's Law's application to election fraud is still debated in academic circles, and its application is considered problematic.
Further doubt is cast on the report's use of Benford's Law when looking at the data sources cited on the 2004 Venezuelan election. The report claims the data is derived from a documentary, however, the link directs readers to a YouTube video with a low view count and no reference to it being an official or otherwise reliable source. The report does little to cite the video's direct findings, preferring to discuss it in general terms and provide screenshots. When quoting the source, the authors merely provide a Google Translate version of the video's description. Clearly, this video is not a credible source for a forensic report, even presumably to most experts "well versed in election statistics and electoral anomalies," not least because it mistakenly presents an image of an Argentinian lottery terminal as a voting machine.
Claims concerning election fraud in Brazil and Benford's Law have surfaced before, as a 2021 BBC article reports. These were contested by experts, who found no evidence of consequential election tampering and concluded that methods of mathematical analysis based upon Benford's Law cannot establish fraud.
This previous coverage demonstrates that the forensic application of Benford's Law is not only incapable of definitively proving fraud, but subject to a variety of other criticisms about its application to election interference. Together, these cast significant doubt on the substance of the analysis of the alleged report.
CDM Policy
Given the multitude of issues and irregularities concerning this report, questions ought to be asked of Matthew Tyrmand and the editorial team at Creative Destruction Media. The outlet, which has published its policies concerning confidential sources online, states, "we use confidential sources sparingly to provide important information that cannot be obtained through on-the-record sources. Reporters should disclose the identity of unnamed sources to at least one editor." If Tyrmand and Creative Destruction Media determined that allowing unnamed individuals to anonymously publish flawed, partially plagiarized analysis of public data falls within these guidelines, they are being interpreted generously.
Logically reached out to Creative Destruction Media for comment, but received no reply. It has since published a second anonymous report on the second round of voting in the Brazilian election, the content of which is very similar to that concerning the first round.