Google has joined OpenAI, Anthropic and Microsoft in promising to support a common signal in robots.txt that tells web crawlers not to scrape a website’s content for AI training. For businesses, this is an important step towards clearer choices about how online material may be collected and used.
But a signal is only one part of the picture. If you run an online store, booking service, clinic or other digital business, you need to understand what the change can—and cannot—do. At Best Choice, we help businesses make practical decisions about website access, automation, data and AI, based on their own goals.
1. What does the new AI scraping signal mean?
robots.txt is a text file that website owners use to communicate crawler preferences. Search engines and other automated services may check it before accessing pages. The new common signal is intended to give site owners a clearer way to say that their content should not be collected for AI training.
Support from major technology companies could make the signal more useful: a shared approach is easier for website owners to manage than separate instructions for every crawler. However, a pledge to support a standard does not mean every crawler will follow it, or that the signal will control every use of content.
2. A robots.txt file is not a security barrier
It is important to treat robots.txt as a request, not a technical lock. It does not prevent a determined visitor or crawler from accessing a publicly available page, and it should not be used to protect confidential information.
For sensitive business data, use proper access controls such as authentication, permissions, rate limits and server-side security. A clinic, for example, should never rely on crawler instructions to protect patient details. Private information should not be publicly accessible in the first place.
3. Decide what you want to protect—and what you want discovered
Not every business benefits from blocking every automated crawler. A retailer may want its product pages to appear in search results while preferring that certain content is not used for AI training. A booking company might want public service descriptions indexed, but keep customer accounts and internal availability tools private.
Start by mapping your website content into categories: public marketing pages, original editorial material, customer data and internal systems. Then decide what should be discoverable, what should require permission, and what should not be collected. Clear choices help avoid accidentally limiting search visibility while trying to manage AI scraping.
4. Build a website policy that works in practice
At Best Choice, we can help create a tailored approach that combines crawler instructions with real technical safeguards. This may include reviewing your site structure, configuring access rules, identifying sensitive endpoints, and setting up logs or alerts to help your team understand unusual traffic.
We can also connect these controls with your existing website, webshop, CRM or analytics setup. For a warehouse business, for instance, public product information can remain accessible while stock management tools and supplier data stay behind secure login. The aim is to protect business assets without disrupting everyday operations.
5. Make scraping controls part of your data strategy
Web scraping is not always harmful. Businesses may use approved scraping to monitor public prices, collect market information or automate research. The key is to know what is being collected, whether the activity is permitted, and how the data will be used.
We can build custom data workflows that gather information from appropriate sources, apply validation rules and send useful results to your team. For a café group, this could mean tracking public competitor menus; for an online store, it could mean monitoring selected market prices. Good automation saves staff time while keeping the process controlled and documented.
6. Review your setup as standards and rules evolve
AI crawler practices and technical standards are changing, so review your website policies regularly. Check that your instructions match your current business needs, test that private areas require authentication, and make sure your team knows who can approve changes to data access.
Businesses operating in Denmark and across Europe should also consider privacy, copyright and contractual responsibilities when collecting or sharing data. A crawler signal does not replace legal advice or a clear data governance process. When in doubt, assess the specific content, purpose and permissions involved.
Take control of how your website is accessed
The emerging common signal gives businesses a more consistent way to express their preferences about AI training data, but it is not a complete security solution. The best approach combines clear policies with technical controls and automation designed around your operations.
We at Best Choice can review your website, improve crawler and access settings, and build custom integrations or data workflows for your business. Contact us to discuss a practical solution for your website, webshop or digital services.
