Prevalence of nonsensical algorithmically generated papers in the scientific literature

In 2014 leading publishers withdrew more than 120 nonsensical publications automatically generated with the SCIgen program. Casual observations suggested that similar problematic papers are still published and sold, without follow‐up retractions. No systematic screening has been performed and the prevalence of such nonsensical publications in the scientific literature is unknown. Our contribution is 2‐fold. First, we designed a detector that combs the scientific literature for grammar‐based computer‐generated papers. Applied to SCIgen, it has a 83.6% precision. Second, we performed a scientometric study of the 243 detected SCIgen‐papers from 19 publishers. We estimate the prevalence of SCIgen‐papers to be 75 per million papers in Information and Computing Sciences. Only 19% of the 243 problematic papers were dealt with: formal retraction (12) or silent removal (34). Publishers still serve and sometimes sell the remaining 197 papers without any caveat. We found evidence of citation manipulation via edited SCIgen bibliographies. This work reveals metric gaming up to the point of absurdity: fraudsters publish nonsensical algorithmically generated papers featuring genuine references. It stresses the need to screen papers for nonsense before peer‐review and chase citation manipulation in published papers. Overall, this is yet another illustration of the harmful effects of the pressure to publish or perish.

outputs a PDF looking like a scientific paper … but full of gobbledegook.
The scientific community must wipe these nonsensical papers out of the published literature to enhance the credibility of science. Following the reporting of more than 120 nonsensical papers (Labbé & Labbé, 2013;Van Noorden, 2014b), the two concerned publishers reacted differently. Springer retracted the 18 incriminated papers. 6 They updated the publication records and the proceedings available from SpringerLink, prefixing each title with "Retracted" and erasing the content of the paper. This episode had severe consequences: the Open Access Scholarly Publishers Association placed Springer under review between April and August 2014. 7 Springer funded a project to design the SCIDetect software 8 classifying a given paper as generated or not (Nguyen, 2018). This software has been integrated into the editorial workflow to flag such computer-generated submissions. The arXiv preprint server also screens the incoming submissions to reject generated papers (Ginsparg, 2014). In contrast, IEEE removed most of the reported papers without providing any clue about their past existence, their metadata, or their full-text.
Screening the entire literature for computer-generated papers with methods by (Ginsparg, 2014;Labbé & Labbé, 2013;Nguyen, 2018) requires the harvesting of each paper in its full-text version so to classify it as generated or not. To date, only a few stakeholders such as Google Scholar and the Jawaharlal Nehru University in India (Pulla, 2019) would be able to do so. This is a daunting endeavor as scientists publish more than 1.4 million papers a year (Schneegans, 2015, p. 36). The systematic identification of algorithmically generated nonsensical papers (henceforth simply referred to as "nonsensical papers") in the scientific literature is thus still an open issue. This paper introduces two contributions: • C1. We design a method to identify nonsensical papers and assess their prevalence in the entire scientific literature. We focus on nonsensical papers generated with a probabilistic context-free grammar, such as SCIgen and Mathgen. 9 We extract characteristic patterns from the grammars that are run against an academic search engine indexing scientific papers in full-text. This approach is applied to identify SCIgen-based nonsensical papers from the entire literature. We evaluate its effectiveness and report performance in terms of recall and precision. • C2. We present a scientometric report on the prevalence of SCIgen-generated papers to answer the following questions. Which publishers are concerned? What are the main characteristics of the nonsensical papers? Which venues (among journals and conference proceedings) and under which license (open/closed access) those papers were published? Has anyone noticed these absurd contents and reported this outrageous situation where people are charged for absurd senseless contents? Either authors (through author processing charges) or readers (through subscriptions)? Are these problematic papers cited and what do they cite?
The paper is organized as follows. We first review the existing approaches to detect grammar-generated papers. Then we introduce our first contribution: the detector of grammar-generated nonsensical papers and its performance in combing the scientific literature for SCIgen papers. Our code is realized as Supporting information, which enables anyone to re-run the nonsensical paper detection in the future. As a second contribution, we then perform a scientometric study of SCIgen papers in the entire scientific literature and discuss their prevalence.

| HOW TO DETECT COMPUTER-GENERATED PAPERS?
The SCIgen designers submitted a generated nonsensical paper to a 2005 conference in computer science (Ball, 2005). Surprisingly, it was accepted! This unfortunate event triggered a new research question: Is it possible to automatically detect such computer-generated papers? We focus on the generation method implemented in SCIgen: a probabilistic context-free grammar (i.e., textgeneration rules applied at random). Several stylometric methods have been investigated (Labbé et al., 2016). They relate to authorship attribution and profiling tasks, and we refer to (Savoy, 2020) for a comprehensive review on that topic. Four types of methods were designed to tag a candidate paper as generated or genuine.

| Word-based detection methods
Most of the published work analyzes the generated text. Some approaches build a binary classifier to determine if a candidate document should be tagged as generated or genuine. The features of such classifiers include statistics on words (Amancio, 2015;Avros & Volkovich, 2018;Lavoie & Krishnamoorthy, 2010;Williams & Giles, 2015) or compression factor (Dalkilic et al., 2006), as the theory of information states that random texts are less prone to compression than genuine ones. This method is challenged when a scammer rewrites passages or includes genuine text (Antkare, 2020, p. 186).
2.2 | Grammar-based detection method Nguyen and Labbé (2018) aimed at flagging a given sentence as being generated or not. The method computes a grammatical structure similarity between the sentence and a set of known generated sentences. The two syntax/ parse trees are processed with a tree matching algorithm to identify their largest common subtree. The probability of a sentence being generated is higher when this subtree overlaps the syntax tree of a non-generated sentence. Nguyen (2018, p. 22) described an internal method Springer had designed to detect SCIgen papers, which extracts characteristic phrases from a set of SCIgen papers. A paper is flagged as generated based on the wordlevel n-gram overlap between the tested paper and a reference set of generated papers. This method requires tuning the n-gram threshold.

| Citation-based detection method
The method of Xiong and Huang (2009) tests the existence of the references with a web search engine. As SCIgen generates all references by concatenating fixed strings picked at random, this approach proved successful on a dataset of 50 SCIgen and 50 genuine papers. As opposed to the three previous methods, this citationbased method does not require any example of generated papers or any knowledge on the associated grammar. However, this method is challenged when a scammer inserts his/her own references in the generated text (Antkare, 2020, p. 181).
To assess the prevalence of nonsensical papers in the entire scientific literature, one needs to run one of the aforementioned methods on each and every paper. Many practical barriers need to be overcome: one needs to collect papers in full-text (a pitfall when not published as open access), get the right to text-mine them, and devote sufficient computer power to process the resulting massive amount of data.
In the next section, we present an original and more efficient method that we designed to measure the prevalence of SCIgen papers in the entire scientific literature.
3 | NONSENSICAL PAPER DETECTOR TARGETING THE SCIENTIFIC LITERATURE Section 3.1 describes our approach for day-to-day detection of computer-generated papers from the scientific literature. We then assess the effectiveness of our approach to detected SCIgen-generated papers (Section 3.2).
3.1 | The "search and prune" method to detect nonsensical papers The "search and prune" method we designed to identify papers generated by grammar has two steps.

| Searching for nonsensical candidate papers
In this first step, an academic search engine is used to retrieve all papers most likely to be generated by a given grammar (and only them). These grammar-generated papers contain fixed word sequences specified in the grammar rules. The idea is to consider some of these sequences as "fingerprints" of the grammar. Fingerprints need to be specific of the grammar: these selected sequences of words unlikely occur in non-generated genuine papers. Word sequences with lower likelihood to occur in the scientific literature are the most effective fingerprints. This likelihood decreases as, on the one hand, the sequence length increases and as, on the other hand, the words are seldom used in the literature or they feature the typos inadvertently made by the (human) authors of the grammar. Examples of improbable word sequences extracted from the SCIgen grammar include: • "in fact, few futurists would disagree with".
• "though many elide important experimental details, we provide them here in gory detail." • "A well designed system that has bad performance is of no use to any man, woman or animal." • Featuring typos: "but without all the unnecssary complexity." "holds suprising results for patient reader." The designer of the nonsensical paper detector identifies such fingerprints by running through the grammar rules. The aim is to collect fingerprints so that at least one fingerprint matches any possible generated paper. As a guidance: one needs to find the subset of derivation rules that will always be executed. For instance, the fingerprints in Figure 1 cover all possible generated text. Each fingerprint translates into a query that is submitted to the academic search engine.
One ends up with a set of "fingerprint-queries" that will match papers generated with the grammar under study. This candidate set of potential nonsensical papers is then passed to step 2, detailed in the next section.

| Pruning the set of candidate papers
A fingerprint-query retrieves many papers; most of them are expected to be grammar-generated. Nonetheless, it is also likely that fingerprints retrieve some genuine papers. The pruning step intends to identify and remove these false positives. One may use one of the methods listed in Section 2.
Leveraging the results of the first step of our method, we propose a new strategy implementing an approach of burden of proof. The first step lists suspect documents, that is, potentially grammar-generated. Each suspect is ranked according to the number of hits, namely the number of fingerprint-queries that matched it. The more fingerprints-queries retrieve a given document, the more this document is likely to be grammar-generated (true positive).

| Application: Detection of SCIgengenerated papers
We apply the "search and prune" method to retrieve SCIgen-generated papers from the scientific literature.
3.2.1 | Leveraging an academic search engine indexing the literature in full-text Our method requires the searching of fingerprint-queries in the full-text of scholarly papers. We leverage a thirdparty academic search engine to do so. The currently available options are reviewed in (Harzing, 2019;Visser et al., 2020). Crossref, Scopus, Microsoft Academic, and the Web of Science index the metadata of publications without providing search capabilities on the full-texts. Thus they are not relevant for our purpose. In contrast, Google Scholar and Dimensions index papers in full-text. Google Scholar has the largest coverage of the peerreviewed and non-peer-reviewed literature (e.g., preprints, dissertations, reports, white papers) but it does not provide any API to programmatically access its data (Else, 2018;Van Noorden, 2014a).
We opted for Dimensions (Herzog et al., 2020) as it provides an API to query metadata and full-text. 10 Its coverage of the peer-reviewed literature is one of the most comprehensive (Visser et al., 2020). Each bibliographic record comes with a set of metadata: title, byline, venue, publisher, publication year, DOI when available, among others. The document type (e.g., article, proceedings paper, monograph, book, and preprint), citation count, and Altmetric Attention Score are also provided.
The next section details the method used to identify the characteristic text chunks in the SCIgen grammar, expected to be used as fingerprint-queries submitted to Dimensions.

| Analyzing the SCIgen grammar to identify fingerprint-queries
We downloaded the SCIgen grammar 11 and analyzed it to spot fingerprints. Two types of fingerprints were selected. On the one hand, long text chucks such as the aforementioned examples were selected. On the other hand, we selected several shorter text chunks of a given rule turned as a conjunctive Boolean query, using Dimensions' search capabilities. A typical example is the rule SCI_INTRO_A producing: Many SCI_PEOPLE would agree that, had it not been for SCI_GENERIC_NOUN, the SCI_ACT might never have occurred XXX where: • SCI_PEOPLE yields information theorists, cyberneticists, cryptographers, futurists … • SCI_GENERIC_NOUN expands as SCI_BUZZWORD_ADJ followed by SCI_BUZZWORD_NOUN: F I G U R E 1 Candidate fingerprints inferred from selected rules of the Mathgen grammar SCI_BUZZWORD_ADJ yields psychoacoustic, empathic, symbiotic, stochastic … SCI_BUZZWORD_NOUN yields methodologies, archetypes, epistemologies … • SCI_ACT expands as understanding of SCI_THING (Note: we stop expanding here). • XXX is list of zero to seven references followed by a full stop.
The fixed part of the rule is a promising fingerprintquery: ["would agree that, had it not been for" AND "might never have occurred"]. The AND conjunctive Boolean operator combines two phrases delineated with quotes. For each promising fingerprint-query we submitted the following query 12 to the Dimensions API: where year >= 2005 and type in [ "article", "chapter", "preprint", "proceeding" ] return publications [id+year+type+doi+title +journal+proceedings_title+ The search targets the full_data index corresponding to publications indexed in full-text. The where clause filters out publications published before 2005, which is when SCIgen was created. We only retain publications that are journal articles, proceedings papers, book chapters, and preprints. This leaves out two other types Dimension indexes: monographs and books. The return clause specifies the metadata to output for the matching publications: ID, year, type, and so on. The top 20 results are provided unless one specifies a higher number with the limit clause; we increased it to the maximum value of 1,000.
The authors reviewed the results of each promising fingerprint-query to drop those returning too many false positives. For instance ["We ran" AND "on commodity operating systems, such as"] seemed characteristic of SCIgen to us but we found several clearly genuine papers matching it and we disqualified this fingerprintquery as a result. Finding false positives was easier when sorting results by decreasing citation counts or Altmetric attention scores: genuine papers (i.e., false positives) were usually at the top of the list. This assessment task led to the delineation of the final set of 258 fingerprint-queries (see Supporting information). On May 20, 2020, we submitted these queries to the Dimensions API which retrieved 3,755 search results corresponding to 298 publications when grouping by publication ID.
The next section evaluates this result both in terms of precision and recall.

| Evaluating the list of flagged nonsensical papers
The authors jointly assessed each of the 298 publications, tagging it with either "contains nonsensical SCIgen text" or "entirely genuine" (see Supporting information). We relied on several evidence conveyed by both full-text and figures. First, we looked up the matching fingerprint-queries in the full-text. Second, SCIgen-generated figures usually stand out: (a) the labels on X and Y axes refer to units (e.g., CPU cycles) that most of the time do not appear in the full-text, (b) the graphs show random data points, (c) incoherent diagrams connect boxes with arrows without any meaning. We were unable to access the full-text for 12 documents (4% of the corpus) and marked them as "contains nonsensical SCIgen text" based on titles and abstracts only. This section reports qualitative observations and a quantitative analysis of the effectiveness of the proposed method.
Qualitative observations. We used the DOIs included in the Dimensions results to get to the landing pages of the publishers. These usually provide a link to download the paper as PDF that might be subscription-based. We downloaded the PDFs from our universities' network (when these subscribed to the contents) and we used other sources such as open archives, social networks, and personal archives (Labbé & Labbé, 2013). When none of these methods was successful, we used Libgen and Sci-Hub (Cabanac, 2016) as last resort. Table 1 lists the examples we discuss in this section, we refer to each case with its letter, for example, Table 1C for case "C".
Some papers appeared to be retracted, see the Quantitative Analysis paragraph for a comprehensive report. We noticed a great variability in reporting retractions among publishers, such as: • The IEEE, as the Institute of Electrical and Electronics Engineers, did not present all retractions the same way, with landing pages showing: A retraction notice with the original paper (Table 1A and Figure 2). A retraction notice with abstract only and no full paper. Some notices indicate: • a violation of quality criteria (e.g., Table 1B).
• other reasons such as "due to non-receipt of a completed Copyright form" in Table 1C. An empty page (Table 1D, i.e., with no information at all even not an error message). Most of these papers can still be found in the table of contents of the conference proceedings, such as Table 1E.
• SPIE, as the Society of Photo-Optical Instrumentation Engineers, provides retraction notices without any access to the abstract or original text (Table 1F). Springer-Nature also adopted this approach in the past, see Table 1G and the publishers' statement. 13 Note that the publisher even rebuilt the online proceedings. • Science domain International provides no content at all: the paper was silently removed from the table of contents and no retraction notice is to be found (Table 1H). We note, however, that the abstract of this paper was deposited at Crossref and is still available. 14 Among the 298 retrieved papers that we jointly assessed, 249 papers (83.6%) contain nonsensical SCIgen text. Most of the authors appear to have used the PDF as generated by SCIgen without any modifications. For some papers, however, authors or typesetters made changes on: • The text: some paragraphs were rewritten in the abstract of Table 1I to match the topic of the venue: materials research. The abstract mentions "the shock wave damage on structures in the roadway during gas explosion" while the entire body of the paper is SCIgen-generated. F I G U R E 2 Example of the IEEE retracted paper Table 1A with retraction notice and original text • The layout was changed for Table 1J and Table 1K, probably during copy-editing. • The figures: SCIgen figures were redrawn in Table 1L. The famous Lena picture (Munson, 1996) was added in Table 1M.
All 243 remaining papers (98%) feature SCIgen materials non-explicitly: they do not warn the reader that SCIgen-generated text appears in these papers. There is no caveat about the meaningless nature of some passages. We hypothesize that such paragraphs were used as padding: • Two SCIgen paragraphs were used for padding in Table 1N. The introduction features SCIgen text (in bold) that seems to have been edited to enhance coherence.
Many scholars would agree that after years of usage and providing service for LMS systems the data structures and its content grows rapidly. (…) Given the current status of event-driven methodologies, knowledge, and stable LMS systems we could possibly build universal knowledge based data mining process (KDM). KDM will be the proper … But the "analysis" section contains unmodified SCIgen text: Continuing with this rationale, we performed a trace for verification that our design is unfounded. This is a technical property of our application. We ran a month-long trace verifying that our model is not feasible.
Another example includes a 16-line SCIgen paragraph in the middle of paper referenced in Table 1O.
• The SCIgen figures were added to Table 1P: the units and labels of the axis are not coherent with the text. Only one fingerprint-query retrieved this paper: We use our previously evaluated results as a basis for all of these assumptions.
Tongue-in-cheek, Crossref used a SCIgen paper (Table 1Q) to showcase its technology, indexing a SCIgen paper under the name of the fake author Josiah Carberry. 15 Another case features a plagiarism report that Dimension indexed in lieu of the original SCIgen paper (Table 1R) whose text was annotated and linked to 38 potentially plagiarized sources (Figure 3). Each individual source appears to be 1% plagiarized only, which can happen for genuine papers when authors reuse a definition, use standard sentences (e.g., "This paper is organised as follows") or when the plagiarism detector does not discard headers, footers, and references that are common to many genuine papers. Despite a reported 36% similarity index, there is no clear evidence of plagiarism since all sources represent 1% only each. The user of the plagiarism detector might not flag this paper as nonsensical. At the time of writing, this paper is still available at the publisher's without any mention of its SCIgen nature.
This section discussed the qualitative aspects of the dataset collected. We now study the performance of the proposed method from a quantitative viewpoint.
Quantitative analysis. Figure 4 shows the number of fingerprint-queries matching each of the 298 retrieved documents. It reveals the precision of the nonsensical paper detector: • With only one matching fingerprint-query, 48% of the documents retrieved contain SCIgen materials. The remaining part is made of genuine papers (false positives) matching fingerprint-queries such as "Our design avoids this overhead" in (Yu and Vahdat, 2005, p. 34). • With more than one matching fingerprint-query, the detector is 100% correct and no false positives were found.
The bi-modal distribution in Figure 4 suggests that most detected papers contain either few (1-4) SCIgen extracts or a lot of them . We hypothesize that the first case corresponds to papers where SCIgen was used as padding (for a few paragraphs only) or papers that Dimensions indexed with their abstract only, having no access to their full-text.
Overall, our method has a 83.6% precision (249 papers "containing SCIgen") and an unknown recall as the total number of papers containing SCIgen text is unknown ( Figure 5). As a way to estimate recall, we collected the entire catalog of Atlantis Press (N = 123,259 publications) as of April 19, 2020, an open-access publisher whose papers appeared in the results (Table 1O). We ran the classifier of (Labbé & Labbé, 2013) that has a 100% precision and 100% recall on unmodified SCIgen-generated papers. This detector did not yield any new Atlantis paper compared to the 39 problematic papers we identified with the fingerprint-query method ( Figure 6). We noted that this classifier failed to flag papers containing only a few SCIgen sentences used as padding (e.g., Table 1N,O). This suggests that the fingerprint-query method is effective to identify all unmodified SCIgen-generated papers and some (recall to be assessed) papers with SCIgen padding. A caveat applies: Dimensions indexes the titles and abstracts of the papers in its catalog, two-thirds of these being indexed in full-text (Herzog et al., 2020, p. 392). Our fingerprint-query method is likely to miss some papers containing SCIgen in the body of the text only (neither in the title nor in the abstract). F I G U R E 3 A plagiarism report (right) that Dimensions indexed in lieu of the original SCIgen-generated paper (left) shown with the allegedly plagiarized passages (Table 1R). This is an illustration of a plagiarism software failing to unambiguously detect a SCIgen-generated paper

| Limitations of the nonsensical publication detector
Based on the assessment of the papers retrieved, we performed a failure analysis per fingerprint-query (Table 2). Most fingerprint-queries show a 100% precision (87.6%, 226/258). A positive yet weaker precision in range [47%, 97%] (median 83%) is achieved by 21 fingerprint-queries. A detailed study of the pros and cons of each fingerprintquery could lead to increase precision without recall loss. Eleven fingerprint-queries retrieved zero papers (see Supporting information). Albeit retrieving no papers via Dimensions, these 11 fingerprint-queries match documents via Google Scholar. Manual examination suggests that the results correspond to publications in venues with ISSNs but without DOIs. This limitation reflects the lower coverage of Dimensions compared to Google Scholar (see Section 3.2.1).
The proposed detection method relies on the examination of the generative grammar to identify fingerprint-queries. With undisclosed grammars (e.g., the SBIR grant proposal generator mentioned in Section 1), one can infer the fingerprint-queries from a sample of generated texts.
The next section presents a scientometric study of the nonsensical papers we found with the proposed method.

| SCIENTOMETRIC STUDY OF THE NONSENSICAL SCIGEN-GENERATED PUBLICATIONS
The detector retrieved 249 papers containing SCIgen text out of which 6 are genuine publications. These 6 publications explicitly mention the generated nature of the text and acknowledge its SCIgen origin. In this section, we performed a scientometrics study on the remaining 243 papers only. F I G U R E 6 Evaluation of the fingerprint-queries approach on the complete catalog of the Atlantis Press publisher (N = 123,259). The number of true negatives was estimated by a classifier that proved to detect unmodified SCIgen-generated papers with a 100% recall and 100% precision (Labbé & Labbé, 2013)

| When and which type of published nonsensical papers
As commented in Section 1, Springer and IEEE retracted or erased the nonsensical conference publications reported in 2014 (Van Noorden, 2014b). Since then, empirical data shows that this problem has not been tackled by the entire publishing industry. Figure 4 shows the yearly number of detected nonsensical documents. This recurring problem occurred with a small number of cases (ranging from 1 to 59) over the years. Questionable preprints, chapters, and journal articles were detected, in addition to papers in conference proceedings. The two peaks of 2014 and 2019 correspond to articles mostly: journals are not immune to nonsensical publications. Year 2020 should be considered with caution as data collection occurred in May. All 2019 cases were journal articles (Figure 7) that were mostly (91.5%) 142 "the question is, will" AND "satisfy all of these assumptions?" 51 51 1.00

219
"our experiments soon proved that" AND "was more effective than" AND "them, as previous work suggested" 51 51 1.00

237
"the results come from only" AND "trial runs, and were not reproducible" 51 51 1.00

232
"the many discontinuities in the graphs point to" AND "introduced with our hardware upgrades" "is defined not only by the" AND "but also by the key need for" 1 2 0.50 128 "do not necessarily obviate the need for" 9 19 0.47 Notes: Fingerprint-queries with 100% precision sorted by decreasing number of captured papers (top 10 over a set of 226 fingerprint-queries) and the bottom 10 fingerprint-queries. Each row shows the ID of a fingerprint-query and its syntax (see Supporting information). TP stands for "true positives" (i.e., number of papers retrieved by the fingerprint-query and containing SCIgen text). FP stands for "false positives" (i.e., number of papers retrieved by the fingerprint-query and not containing SCIgen text). "Prec" stands for precision: TP/(TP + FP).
published by BEIESP, a publisher founded in 2010 in India. The 54 BEIESP nonsensical publications found appeared in three journals indexed by Scopus in years 2018-2019 and now marked with "coverage discontinued in Scopus" 16 : • International Journal of Innovative Technology and Exploring Engineering (27 cases). Number of SCIgen-generated articles found journal article chapter preprint proceedings paper F I G U R E 7 Yearly distribution of the 243 documents containing SCIgen text that we detected as of May 20, 2020, with yearly share of document types F I G U R E 8 Yearly number of papers containing SCIgen-generated text by publisher, whose acronyms are given in full in Table 3.
Year 2020 is incomplete as the data were collected on May 20, 2020 32 nonsensical publications, 27 were published in the journal Applied Mechanics and Materials indexed by Scopus over 2005-2015 and now marked with "coverage discontinued in Scopus." A third smaller peak of Figure 7 appears in 2010 with IEEE (Institute of Electrical and Electronics Engineers) proceeding papers mostly (72.0%, 18/25). Founded in 1963 in the United States, the IEEE is a professional association and leading publisher in these fields. Most of these papers are still indexed by search engines but not available anymore from the IEEE online library. This phenomenon is described in more detail later (Table 3). After being publicly exposed (Van Noorden, 2014b), only two problematic publications were published after 2014 (Figure 8).
The fourth and last peak of Figure 7 in 2009 concerns two publishers that appear to have published problematic papers in 2009 only. ASME International is the American Society of Mechanical Engineers, founded in 1880 in the United States. Nine problematic ASME publications appeared in the proceedings of ICACTE 2009, the International Conference on Advanced Computer Theory and Engineering, including more than 230 papers and are still sold as book chapters. SPIE stands for the Society of Photo-Optical Instrumentation Engineers, founded in 1955 in the United States. Nine SPIE problematic publications appeared in the proceedings of PIAGENG 2009 (Intelligent Information, Control, and Communication Technology for Agricultural Engineering) including 111 proceeding papers. These nine papers were retracted in November 2015. Notes: The "Access" column is "C" for closed access versus "O" for open access papers. Each paper was accessed by resolving its DOI. The publisher's landing page either (a) provided the paper without any notice ("Unnoticed"), or (b) provided a retraction notice, with or without the paper ("Retracted"), or (c) was a blank page providing neither metadata nor paper ("Disappeared" The top four publishers of Figure 8 include Atlantis Press, founded in 2006 in France, which has published 39 problematic open access publications between 2012 and 2018. Most of the publishers in Figure 8 were not listed in the Beall's list of potential predatory journals and publishers, 17 a notable exception being BEIESP. At this point, we studied the yearly occurrence of problematic papers per publisher. A remaining question is: how many of these papers have been retracted already? Table 3 shows that 4.9% (12/243) of the problematic papers appear as retracted on the publishers' platforms. We labeled the remaining 231 papers as "Disappeared" (34, 14.0%) for papers not available anymore and "Unnoticed" (197, 81.1%) for papers provided without any notice or caveat. As of November 5, 2020, none of these 197 unnoticed problematic papers have been commented on PubPeer, 18 which suggests that neither the publishers nor whistleblowers reported these nonsensical papers. Most of them were neither cited nor mentioned on social media (Section 4.5.2).

| Retraction status of the published nonsensical papers
The per-publisher analysis shows that only five publishers took action. The leading two were SPIE, retracting nine papers that now appear with a withdrawn notice 19 and IEEE whose problematic publications were either erased without any notice (81.0%, 30/37) or retracted as in Figure 2

| Origin of the published nonsensical papers
All 243 papers were affiliated to one country each, none featured a cross-country collaboration. The author affiliations of the 243 papers were in China (156, 64.2%), India (54, 22.2%), Indonesia (3, 1.2%), as well as Belgium, Iran, Poland, Slovakia, and the United States with one paper each (0.4% each). For 25 papers (10.3%), no affiliations appeared or the country could not be determined (e.g., complete affiliation reading as "Independent"). We did not contact the authors to check if they were aware of these papers published with their names in the bylines. It cannot be excluded that malicious people have generated a SCIgen paper with other individuals' identity as authors. With this caveat in mind, these problematic papers seem to come from a few places in the world only.

| Impact of the published nonsensical papers
The following sections consider outward and inward links to the nonsensical papers, in terms of citations and Altmetric Attention Score. 4.5.1 | References: Outward links from nonsensical publications Some problematic papers appear to reference genuine publications in their bibliographies. This section discusses selected cases without being exhaustive.
We found several instances of this in BEIESP journals and on the SSRN preprint server. These references from nonsensical papers effectively count in bibliographic databases such as Google Scholar. The BEIESP bibliographies overlap and share some common references from recurring authors, suggesting a citation cartel aiming at h-index manipulation (Antkare, 2020).
As anecdotal evidence, 29 of the 54 BEIESP problematic papers cite the paper titled "Routing alogrithm over semiregular tessellations" that appeared in 2013 in the IEEE Conference on Information & Communication Technologies under doi: 10.110910. /cict.2013. Surprisingly, the title features a typo: alogrithm instead of algorithm. Note that this paper contains no SCIgen materials, however. While the authors of all 29 citing BEIESP SCIgen-generated papers are different from the "alogrithm" authors, they all belong to the same research institute. According to IEEE Xplore, the paper had "50 Full Text Views" only as of November 13, 2020. Scopus reported 252 citations to this "alogrithm" paper, this total number including the 29 problematic papers (11.5%). Moreover, only two citations (less than 1%) came from papers whose authors were affiliated to another institute.

| Citations: Inward links to nonsensical publications
Dimensions reported citations for 3.3% (N = 8) of the 243 problematic publications. Two papers have three citations, two papers have two citations, and six papers were cited once.
We checked the retracted SPIE paper titled A methodology for the simulation of digital-to-analog converters (doi: 10.1117/12.836788) for which Dimensions listed three citing IEEE papers. After manual examination, none of these three papers cited the SPIE retracted paper, which highlights flaws in the citation index of Dimensions. Google Scholar reported no citations to this SPIE retracted paper. Martín-Martín et al. (2020) published a recent comparison of the coverage of citation indexes. They found that 7% of the citations found by Dimensions are not indexed by Google Scholar. Our case study illustrates a situation where Google Scholar is more accurate than Dimensions.
We also checked the paper Contrasting Congestion Control and the Producer-Consumer Problem Using Pleaseman (doi: 10.1166/asl.2013.4571) for which Dimensions reported one citation. This citation, also reported by Google Scholar, comes from a disappeared IEEE SCIgengenerated publication (doi: 10.1109/qr2mse.2013.6626013) entitled Evaluation of the Producer-Consumer Problem. The original bibliography of the citing paper, generated by SCIgen with random references, was intentionally replaced by references to both genuine and SCIgengenerated publications. For instance, the SCIgensentence "a litany of previous work supports our use of sensor networks [1, 2]" mentions the "Pleaseman" problematic paper (reference 1) and a genuine paper (reference 2). Reference 24 is a retracted SPIE paper. This citing publication Evaluation of the Producer-Consumer Problem is an example of a SCIgengenerated paper with a completely modified bibliography that cites already published publications. We see two motivations for this. First, citation manipulation to increase citation counts. Second, a game of whack-amole, as publications with genuine references fool the citation-based method (Xiong & Huang, 2009) that detects the non-existing references generated by SCIgen (see Section 2).
We also reviewed the Altmetric Attention Score provided by Dimensions for each paper. Only 3 papers out of 243 (1.2%) have a positive score of one, reflecting one mention each in Twitter. One tweet was posted by a journal to promote the problematic paper it published. Two tweets were critical comments by readers who reported the generated nature of the papers. 20

| Prevalence of SCIgen-generated publications
Let us now estimate the prevalence of nonsensical publications, that is, the share of such published papers in the literature. We assume that the 243 papers we found constitute the entire set of existing nonsensical papers (i.e., no false negatives in Dimension, see Figure 5). This assumption allows a conservative estimation: we thus compute the lower bound of the prevalence of nonsensical papers. Dimensions indexing 58 million articles, chapters, and proceedings papers between 2005 and 2020, we can estimate the prevalence of nonsensical papers as 4.29 papers every one million papers. Since most SCIgen papers were published in Computer Science venues, we can reassess this estimation. Dimensions reports 3.3 million indexed documents of the three aforementioned types, on the same period, in the field of "Information and Computing Sciences." This translates into 75 nonsensical papers for every one million papers in computer science.

| CONCLUSION
Unexpectedly, the scientific literature includes computergenerated nonsensical papers getting published and sold by various publishers. Our contribution to the understanding of this detrimental phenomenon is twofold. We proposed a detection method and a scientometric study of the detected nonsensical papers.
The detection method identifies grammar-generated text in publications using a third-party academic search engine. It yields a 83.6% precision when applied to the Dimensions search engine to detect SCIgen-generated papers. Most of the 243 nonsensical papers detected went unnoticed to date: 197 papers are available, sometimes sold, without any warning or withdrawal notice. The scientometric portrait of these 197 papers shows they are affiliated to China and India mostly. Both closed and open access venues include such generated papers. They appeared in journals, conference proceedings, as book chapters, and preprints. The prevalence of SCIgen-generated publications in the computing literature is estimated to 75 nonsensical papers per million (a conservative estimate). Bibliography analysis reveals striking examples of citation manipulation. SCIgen-generated references were intentionally modified to include selected papers/authors/journals/ publishers. This misconduct increases the citation counts fraudulently.
Our method is suited to detect all papers issued by grammar-based generators. With the advances of artificial intelligence writing capabilities (New chapter in intelligence writing [Editorial], 2020) and the remaining pressure to publish, new AI-powered generators are yet to be exposed. Flagging nonsensical, incoherent, redundant, poorly informative, or contradictory publications calls for enhanced tools to screen the scientific literature.