How to Research Your Competition with a SEO Crawler

I was crawling a rival website the other day… Oh, you didn’t know you could crawl someone else’s website?  Well, you can, and it’s one of the best ways you can gain a competitive advantage.  Competitive crawling may sound like a silly contest of babies racing each other across the floor, but it’s actually an SEO strategy to ensure that you’re staying ahead of websites that you’re competing with. The premise is fairly simple, you crawl a competitors site just as if you were about to perform an SEO audit on your own site.  You can then analyze the results and even benchmark your competitors’ progress as they add new content.  But if you’ve never crawled a rival site you may be wondering what sites you can crawl and when you should fire up your trusted spider.  Here are a few tips.

Is competitive crawling important?

Yes.  Competitive crawling is a form of competitor research, and you’d have a difficult time convincing anyone that competitor research isn’t important.  Your competitors’ website is a significant part of their business strategy and you’ll lose serious ground if you can’t dissect the competition’s game plan.

Is crawling someone else’s site legal?

Yes.  You’re crawling information that has been published for the public.  Your crawler won’t be able to access anything that’s protected with passwords or authentications anyway, plus you’re not doing anything malicious.

What sites will I be able to crawl?

Whatever site you want!  Unless, of course, that site requires a log in or the servers block your IP address.  Most websites are open to everyone which means your crawler will be able to navigate freely.  Scout a few sites that are competing for your search terms or vying for the attention of your audience and try scanning them first.  If you crawl often then a savvy webmaster may identify your IP address and block your bot, so just don’t be obvious.

When should I crawl my competitor’s sites?

Ideally you should research your competitors’ sites during the planning and development phase of your site so that you don’t miss any strategic components from the beginning.  If not, don’t worry, just start now so you can respond to, and get ahead of, your competitor’s strategy.  Once you know how your competitors behave you shouldn’t need to spy every day, just when you suspect that the site’s been updated or that they are coming out with something new.

What information can I get from website espionage?

As I mentioned above, it’s just like performing an SEO audit on your own site or a new client’s site.  In general, you’re looking for any information that can help identify a competitor’s strategy.  This can include:

• Keyword trends and recurring headline topics
• Link profile including pages they deem most important
• Landing pages and demographics they are targeting
• Resources they link to that you may be able to utilize

You could even uncover weaknesses that you may be able to exploit such as a lack of keywords in certain areas that you can develop on your site, or broken URLs on topics that you can now hijack.

What can I do with the information once I have it?

Adjust your strategy.  After careful consideration, start making small and measurable changes and continue changing based on outcomes.  You may wish to push into areas of your competitor’s weakness (like sparse keywords on a certain topic) or you may beef up your activity where you can see your competitor is focusing.  The determining factor will be the results of your change, don’t just throw in some keywords or external links and call it good.  You may even find that expanding your resources into one of your competitor’s areas results in negative ROI so you just pull out, only testing will tell.

So where do I start?

Simply fire up your crawler and type in a website that’s in your space.  If you don’t have a crawler, or your subscription to an expensive crawler has expired, you can download Beam Us Up Crawler for free, just look to your right and you should see the download buttons.  You may try crawling a site that you’ve visited before so you have a better idea of what to expect when you open the report.  It’s certainly not a difficult process, and it can improve your strategy greatly so I’d recommend not waiting long before your first competitive crawl.