> I wish the IA were little more aggressive about not retroactively applying rob...

		wl on April 29, 2024 \| parent \| context \| favorite \| on: The Internet Archive's last-ditch effort to save i... > I wish the IA were little more aggressive about not retroactively applying robots.txt rules on archived content. They've stopped doing that. They now ignore robots.txt completely and you have to email them to stop them.