In July 2026, the American Society for Indexing AI Committee published an “IndexerLabs Review” and concluded that IndexerLabs should not be used to create indexes. We had previously contacted the Committee ourselves because we wanted a professional review of our work and hoped to collaborate on a better benchmark for AI indexing.
We respect ASI’s expertise, which is why we approached them. However, we do not believe a review based on a roughly five-month-old research artifact, conducted without ever using IndexerLabs and containing entirely unfounded claims about a process the Committee had never examined, provides a basis for recommending against IndexerLabs as a product. We have therefore asked ASI to correct the factual errors in the report and reconsider its recommendation.
1. The index reviewed appears to have been deliberately “cherry-picked” from an archived version of our website.
On May 27, the Committee said it planned to review IndexerLabs that summer, by which point we had released several substantially newer public demos.
May 27 message from ASI AI Committee Chair Elizabeth Bartmess announcing a planned summer review of IndexerLabs.
The version they ultimately reviewed was created around February of 2026 and was no longer available on our website, so they appear to have deliberately used a previously saved or archived version (from either manually saving or from an archiving engine) while ignoring the current public demo.
The Committee appears to have gone out of its way to find that old index while ignoring our newer public demos.
Since February, more than 900 updates have been made to our pipeline and underlying models.
2. ASI did not evaluate the IndexerLabs platform
Nobody from the Committee appears to have actually used IndexerLabs. The Committee did not create an account, upload a book, run an indexing job, use the editor, or test any of the controls through which an author reviews and refines an index.
IndexerLabs is a tool, not a system designed around immediately exporting an untouched index. The generated index is obviously important, but so is the platform built around it. Authors can inspect what the system selected and adjust the result to match their preferences. Our authors generally spend fewer than two or three hours reviewing and lightly editing an index before export, and much of that work reflects ordinary editorial preference rather than correction of an objective mistake. Indexing is subjective enough that two authors can reasonably want different indexes of the same book.
None of this workflow was evaluated. ASI instead evaluated a standalone output and then issued a recommendation about the software that produced it. ASI acknowledged essentially this distinction in its review of Indexia, and we believe the same standard should apply to IndexerLabs.
3. The report makes a false claim about our training process
ASI states that “nothing in its training tells the AI which indexes are well-done and which are poorly done.”
The Committee had no basis to claim this. It has never had access to our training dataset, our selection and filtering process, our quality controls, the full provenance of our data or our broader training methodology. More importantly, the statement is factually incorrect. Our training process includes negative examples, including poor indexing decisions and undesirable model completions, specifically to teach distinctions between stronger and weaker outputs.
We keep much of our training corpus and methodology private for commercial reasons. ASI could reasonably have said that it did not know how we distinguish between higher and lower quality examples because those details have not been publicly disclosed. Instead, it supplied its own answer and presented it as fact. We have asked the Committee to explain the evidentiary basis for the statement and to correct it on the review page.
4. The published evidence is insufficient to reproduce several of the findings
ASI reports that it randomly sampled 102 entries and classified 23 as containing inaccurate locators, another 34 as problematic for other reasons and 45 as adequate. It does not publish the 102 sampled entries, the individual judgments behind those classifications or the professional index created for the 22-page comparison chapter.
That omission is particularly important because many of the disputed classifications are subjective judgments rather than objective measurements. Without the underlying examples, readers cannot examine those judgments for themselves and we cannot independently reproduce the reported analysis.
We emailed the Committee to request the professional comparison index, the sampled entries, the individual classifications and further details about the methodology. The Committee refused to provide the specific data, instead pointing us to a generic evaluation guide without answering any of our specific questions.
The National Academies of Sciences, Engineering, and Medicine call reproducibility a hallmark of good science. ASI refused to share even the sampled entries and classifications needed to check its numbers. When an organization refuses to share the data behind its findings, readers should be wary of trusting its conclusions.
We still want ASI to evaluate IndexerLabs
Our original invitation remains open. We will give the ASI AI Committee unlimited and free access to the current version of IndexerLabs and let them choose the books, run the jobs, use the editor, inspect the evidence, change the index and publish whatever they find.
We would also still like to collaborate on the benchmark we originally proposed, ideally using multiple independently produced professional indexes so that disagreement between an AI system and a professional indexer can be interpreted alongside the disagreement that naturally exists between professional indexers themselves.
If ASI finds serious problems with the current system, we want to know about them. That was the reason we contacted the Committee in the first place. What they have published so far is a cherry-picked critique of one unedited output produced by an early version of IndexerLabs. We do not believe that supports their much broader recommendation against IndexerLabs as a product.
Paul Estrada
Founder of IndexerLabs