SA news sites block AI scrapers at half the rate of Global North publishers — and smaller outlets are barely protected at all
Share of news websites explicitly blocking at least one AI crawler via robots.txt; 263 South African sites examined, May 2026; international comparators for context
Blocks at least one AI crawler
Does not block any AI crawler
+0.7pp
change in SA blocking rate
since December 2025 baseline
263
South African news websites
examined in the report
South Africa's news sector blocks AI crawlers at more than four times the rate of Brazil — but at barely half the rate recorded among top-tier Global North publishers by the Reuters Institute in 2023. Yet the aggregate 30.4% figure conceals a deeper structural inequality: the Protocol Gap report finds that the ability to block AI scrapers belongs overwhelmingly to large, well-resourced publishers, while smaller, community, vernacular and independent outlets — which make up the majority of the 263 sites examined — remain almost entirely exposed. The rate has shifted by just 0.7 percentage points in six months, meaning for most SA news publishers, meaningful protection against AI content extraction is, in practice, still out of reach.