Substack,
It is frustrating that AI agents, including GPTBot, cannot access my content. As a writer, I should have the right to decide whether my work is indexed by AI, not you.
Your robots.txt file restricts AI crawlers from reading my articles, making it impossible for them to be indexed or discovered through AI-powered search. Meanwhile, traditional search engines and social media platforms can still access parts of Substack freely.
Search engines have revolutionized content discovery. AI-powered search is the next step. By blocking AI crawlers, Substack limits writers' ability to:
Reach AI-driven audiences who rely on AI summarization and search.
Be referenced in AI-generated discussions and research.
Adapt to the future of digital content consumption.
If a writer wants their work hidden from AI, they should be able to opt out. If a writer wants AI indexing, they should be able to opt in. The choice should not be dictated by the platform.
AI indexing is not a threat to writers—it is an opportunity. It enables our work to be part of the larger digital conversation. Please give us the freedom to decide whether AI can read our words.
Sincerely,
We