In July 2026, the American Society for Indexing AI Committee published an “IndexerLabs Review” and concluded that IndexerLabs should not be used to create indexes. We had previously contacted the Committee ourselves because we wanted professional scrutiny of our work and hoped to collaborate on a better benchmark for AI indexing.
We respect ASI’s expertise, which is precisely why we approached them. However, we do not believe a review based on a nearly nine-month-old research artifact, conducted without ever using IndexerLabs and containing factual claims about a process the Committee had never examined, provides a reasonable basis for recommending against IndexerLabs as a product. We have therefore asked ASI to correct the factual errors in the report and reconsider its recommendation.
1. ASI did not evaluate the IndexerLabs platform
The most basic problem with the review is that nobody involved appears to have actually used IndexerLabs. The Committee did not create an account, upload a book, run an indexing job, use the editor, inspect the evidence behind an entry, add omitted material, remove unwanted material, modify headings, work with subentries or cross-references, or test any of the controls through which an author reviews and refines an index.
That matters because IndexerLabs is a tool, not a system designed around immediately exporting an untouched model completion. The generated index is obviously important, but so is the environment built around it. Authors can inspect what the system selected, change terminology, add concepts they believe deserve treatment, remove entries they do not want, restructure material and adjust the result to match their preferences. Our authors generally spend fewer than two or three hours reviewing and lightly editing an index before export, and much of that work reflects ordinary editorial preference rather than correction of an objective mistake. Indexing is subjective enough that two authors can reasonably want different indexes of the same book.
None of this workflow was evaluated. ASI instead evaluated a static output and then issued a recommendation about the software that produced it. An exported model can tell you something about a 3D modeling application, but evaluating that model without opening the application does not amount to evaluating the application itself. ASI acknowledged essentially this distinction in its review of Indexia, and we believe the same standard should apply to IndexerLabs.
2. The artifact was old and its limitations were already disclosed
The Oxford index reviewed by ASI was generated nearly nine months ago by a substantially earlier version of IndexerLabs. More than 2,000 updates have been made to the system since then, covering term selection, names and works, subentries, cross-references, locator verification, the editor, history, exports and the controls available to authors.
More importantly, this particular index came from our early research into overlap between AI-generated and professionally generated indexes. In the accompanying paper, we explicitly limited the conclusions we thought could be drawn from the experiment. Much of the work was concerned with a narrower question, whether the system identified similar concepts as indexable, while locators, subentries, cross-references, formatting and human editing were treated as separate parts of the indexing process. The Oxford output was intentionally published without human editing, and we said so.
ASI is entirely within its rights to criticize that artifact, including its subheadings, cross-references, terminology and locators. What we dispute is the much broader inference that shortcomings in an intentionally unedited research output from an early version of the system establish that IndexerLabs, including the editor and workflow that ASI never tested, should not be used.
3. The report makes a false claim about our training process
ASI states that “nothing in its training tells the AI which indexes are well-done and which are poorly done.”
The Committee had no basis to claim this. It has never had access to our training dataset, our selection and filtering process, our quality controls, the full provenance of our data or our broader training methodology. More importantly, the statement is factually incorrect. Our training process includes negative examples, including poor indexing decisions and undesirable model completions, specifically to teach distinctions between stronger and weaker outputs.
We keep much of our training corpus and methodology private for commercial reasons. ASI could reasonably have said that it did not know how we distinguish between higher and lower quality examples because those details have not been publicly disclosed. Instead, it supplied its own answer and presented it as fact. We have asked the Committee to explain the evidentiary basis for the statement and to correct it on the review page.
4. The published evidence is insufficient to reproduce several of the findings
ASI reports that it randomly sampled 102 entries and classified 23 as containing inaccurate locators, another 34 as problematic for other reasons and 45 as adequate. It does not publish the 102 sampled entries, the individual judgments behind those classifications or the professional index created for the 22-page comparison chapter.
That omission is particularly important because many of the disputed classifications are editorial judgments rather than mechanical measurements. Without the underlying examples, readers cannot examine those judgments for themselves and we cannot independently reproduce the reported analysis.
We have asked ASI for the professional comparison index, the sampled entries, the individual classifications and further details about the methodology. As of publication, we are awaiting a response and will update this page if the requested materials are provided. Indexia has said that it previously requested replication data from the Committee and was refused, although we will not assume ASI will respond to our request in the same way.
We still want ASI to evaluate IndexerLabs
Our original invitation remains open. We will give the ASI AI Committee unlimited free access to the current version of IndexerLabs and let them choose the books, run the jobs, use the editor, inspect the evidence, change the index and publish whatever they find.
We would also still like to collaborate on the benchmark we originally proposed, ideally using multiple independently produced professional indexes so that disagreement between an AI system and a professional indexer can be interpreted alongside the disagreement that naturally exists between professional indexers themselves.
If ASI finds serious problems with the current system, we want to know about them. That was the reason we contacted the Committee in the first place. What they have published so far is a detailed critique of one unedited output produced by an early version of IndexerLabs. We do not believe that supports their much broader recommendation against IndexerLabs as a product.
Paul Estrada
Founder of IndexerLabs