slipknotfana day agohttps://sr.ht/robots.txt is a robots.txtgus_massa7 hours agoIgnoring that AI bot will ignore it, ...Doesn't "User-agent: AwarioRssBot" needs it's own "Disallow: /" line?zionsatia day agoDo LLM crawlers adide by this?
gus_massa7 hours agoIgnoring that AI bot will ignore it, ...Doesn't "User-agent: AwarioRssBot" needs it's own "Disallow: /" line?