Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> I wish the IA were little more aggressive about not retroactively applying robots.txt rules on archived content.

They've stopped doing that. They now ignore robots.txt completely and you have to email them to stop them.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: