How to Automate Competitor Analysis with Python?

Competitor analysis shows us the issues that we often overlook. Competitor analysis is a great priority to see what competitors are doing better and where we can improve ourselves.

When doing competitor analysis, we usually follow our rankings on a keyword basis. For example, we regularly check the rankings in the word "laptop" and try to make various optimizations when we lose rankings.

Importance of Competitor Analysis on Pages with Loss of Ranking

When we lose ranking, the first thing we look at is usually the title-description fields. Because title-description elements are a ranking factor. The more optimized these elements are, the higher we can rank on the relevant page. At this point, one of the most effective methods for title-description optimization is to determine how competitors construct titles and descriptions and which keywords they focus on. 

When we look at the title and description of competitors in the SERP and compare them with the title and description of our site, we can see the details we missed. For example, the keyword "X" appears in the title of competitor "A" but not in ours. Detecting this is a good method for title-description optimization.

Sometimes Google may choose not to use the title-description information specified in the source code. For example, competitor "A" may have given the "Z" title in the source code, but Google may have created the "T" title itself and shown it in the SERP. If we can see in which cases these also happen, it can help us while doing our title-description optimization.

Doing these analyses is often a handful. Because for this analysis, a Google search should be done for each keyword one by one. Then titles and descriptions should be analyzed one after another. Of course, Google may prefer not to use the title and description in the source code. Therefore, it is also necessary to look at what is written in the source code.

When this is tried to be done for each keyword, a long analysis time emerges. For example, let's have 100 keywords. Let's have 4 competitors. That means 5 domains including our domain. It takes 25-30 seconds per domain to look at the title-description information in the SERP and source code. In other words, it takes about 2-2.5 minutes to copy and paste only the data of 1 keyword for 5 domains into an Excel. If we do this for 100 keywords, it takes about 3.5-4 hours. Of course, after copying them without any interruption, you should have time to analyze them :)

Python Optimization in Competitor Analysis

What needs to be done here is to pull this data autonomously. If we can get the computer to do it automatically, we can skip the operational part and just spend time on analysis. Of course, I used Python for this, so let's see how I did it.

First of all, we need to search keywords on Google. For this, I used the Google API to search the keywords one by one on Google. Then we need to capture the results of the domains we want from the results on Google. For this, I defined a for loop and pulled the URL information of the domains I wanted.

In other words, I searched the keyword "laptop" on Google using the API, and from the results I captured the results of mainexample.com, example2.com, example3.com, example4.com, and example5.com domains. In this way, I captured the result that someone searching the keyword "laptop" on Google sees from the example4.com domain. Here I had to scrape 3 elements; title, description, and URL. The title and description I scraped at this stage are the title-description information that Google prefers to show. We take the URL to access the source code.

Then I accessed the source code of these URLs and scraped the title and description information specified in the source code. If you want more detailed information about the part up to this point, I described a similar project in the Google SERP and Website Scraping with Python article.

I now had the URLs that appeared in the relevant keyword, their title and description information in the source code, and the title and description information in Google SERP. In this way, we were able to capture in which cases Google does not use the title and description information in the source code.

Of course, I didn't want to end the code here. I also integrated the code to pull data from PageSpeed Insights, which I recently developed with Ata Mert Kansoy. Thus, we could see each page's page speed data.

Conclusion

As a result, we could now autonomously scrape the following information for any keyword;

Which title-description texts our competitors used

How Google shows them in SERPs

What are the pagespeed data of pages ranked in SERP

When we print all this to an Excel file, it looks like this;

We can contact you for details :)

Thanks to Adem Yıldız , Nefise Taş , Esad Bozkaya , Tuğba Evrensel , Duygu Garip and Zehra Türksoy for their support in the making of this study.