Robots.txt and Crawler Politeness in the Age of Generative AI
Poster providing an overview of the robot.txt standard including results from a study of its current use and potential uses for generative artificial intelligence (AI). It was presented at the IIPC General Assembly and Web Archiving Conference held on April 8-10, 2025 in Oslo, Norway.