Until these bots become good citizens (eg: respecting robots.txt), I will be ser... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		dmitrygr 10 months ago \| parent \| context \| favorite \| on: The Cost of Being Crawled: LLM Bots and Vercel Ima... Until these bots become good citizens (eg: respecting robots.txt), I will be serving them gzipped gibberish that decompresses to terabytes. The ball is in their court. You don’t get to demand civility AFTER being a dick. You apologize and HOPE you’re forgiven.

randunel 10 months ago | [–]

What do you reckon, does OP in this post respect robots.txt or do they "borrow" content in a similar manner, without respecting such standards?

AlienRobot 10 months ago | [–]

I thought the AI wars would be fought with bombs vs. bots not with ZIP bombs vs. bots.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact