Google has introduced a new control feature in its robots.txt indexing file, offering publishers the ability to decide how their content contributes to the improvement of Bard and Vertex AI generative APIs. This control, known as Google-Extended, provides publishers with the option to specify that their content should not be used for these two APIs, including future iterations of the models that power these products.
Danielle Romain, Google’s Vice President of “Trust,” emphasized that this development stems from feedback received from web publishers who seek more choice and control over how their content is utilized in emerging generative AI applications. Romain stated that Google-Extended represents an essential stride towards greater transparency and control, principles that should be upheld by all providers of AI models.
Generative AI chatbots and similar systems have gained prominence and deeper integration into search results. However, how these systems digest content from various sources and present it within conversations has raised concerns among publishers. While they often cite the sources, the aggregation of content can potentially divert significant traffic away from individual publishers, impacting aspects like ad revenue and business models.
Google clarified that the opt-outs provided by Google-Extended will apply to the next generation of models for Bard and Vertex AI. For publishers looking to keep their content out of entities like the Search Generative Experience (SGE), the use of the Googlebot user agent and the NOINDEX meta tag in the robots.txt document remains the appropriate method.
Romain also highlighted the growing complexity web publishers will face in managing different uses of their content at scale as AI applications expand. With 2023 witnessing a proliferation of generative AI-based tools and search remaining a primary discovery method for content, the internet landscape is poised for a significant transformation. Google’s introduction of this control feature is both timely and reflective of its commitment to addressing the evolving impact of its products on the web.
