The only thing that would make that even more perfect would be if there was some... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		marcusb 10 months ago \| parent \| context \| favorite \| on: The Cost of Being Crawled: LLM Bots and Vercel Ima... The only thing that would make that even more perfect would be if there was some way for the site owner to signal to prospective bots which parts of the site are open to the bots to visit. I know this seems really complicated, but I really think it could be expressed in a simple text file.

tough 10 months ago [–]

i dont know, robots.txt sounds too complicated for 2025

thwarted 10 months ago | [–]

I would have worded this as "it sounds to simple for 2025".

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact