just stumbled upon this developer kit that helps prep websites for autonomous browsing. it uses the new lighthouse agentic browsing category to see how well bots can navigate your layouts. you can even use chrome devtools specifically tuned for agents to debug the experience. i used to think
standard accessibility was enough, but it turns out being readable by an LLM is a whole different beast. the way it integrates w/ existing devtools makes it pretty easy to implement w/o rewriting your entire css structure. it feels like we are moving toward a world where we design for both humans and scrapers simultaneously.
it's kind of terrifying if you think about it . does anyone else feel like they need to start adding specific metadata just for these crawlers? i am curious if this will eventually become a standard part of our basic seo workflow
more here:
https://developer.chrome.com/blog/agent-ready-toolkit?hl=en