The Cartographer of the Crawl
I once met a developer who described his work not in terms of features or performance metrics, but as a form of cartography. His medium wasn't parchment or satellite imagery; it was the crawl budget. His job, as he saw it, was to draw maps so clear and logical that even a machine with the barest context could understand the lay of the land. He was a cartographer of the crawl, and his primary tool was the humble, often overlooked, internal link.
Most of us treat internal links as a means to an end—a way to guide a user from point A to point B. But for him, each link was a deliberate stroke on a vast, invisible canvas. He wasn't just connecting pages; he was defining relationships, establishing hierarchies, and whispering to the algorithms that wander our sites. A link from a broad category page to a specific article wasn't just navigation; it was a parent introducing a child. A link between two related articles in a sidebar wasn't clutter; it was a handshake between peers, suggesting a shared conversation.
His work began not with code, but with a whiteboard covered in circles and arrows. He would spend hours sketching the ‘neighbourhoods’ of the site. Which pages were the central plazas, the hubs of activity? Which were the quiet residential streets, the deep-dive articles that deserved a peaceful existence? The goal was to ensure no page was an isolated island. A visitor, or a crawler, should always feel the pull of the next logical step, the invitation to explore just a little further.
This philosophy extends to the very structure of the site, the URLs themselves. He saw a clean URL hierarchy as the grid system of a well-planned city. A path like `/library/fiction/1984` isn't just an address; it’s a map legend. It tells you precisely where you are in the information metropolis. The cartographer understands that when a URL’s structure reflects the site’s architecture, the internal links become more than just pathways—they become affirmations of that structure, reinforcing the map with every click.
Of course, maps need updates. Cities grow and change. This is where his work intersected with the melancholy art of redirects. When a page was retired or moved, the 301 redirect wasn't just a technical fix to him. It was an act of historical preservation. It was the cartographer carefully pasting a small label on an old map that reads, “The treasure you seek is now housed in the new museum, two blocks east.” The link is broken, but the intent to guide remains perfectly intact.
In an age of complex algorithms and opaque ranking factors, there’s a quiet elegance to this approach. It’s a reminder that beneath the layers of code and strategy, the web is fundamentally a collection of documents connected by references. The cartographer of the crawl doesn't try to game the system. Instead, he builds a system so inherently sensible, so gracefully interconnected, that its value is self-evident. He builds a world where both humans and machines can find their way home.
Notes & further reading
A few pages I came back to while writing this: