How to conduct a Website Carbon Audit using Screaming Frog SEO

My name is Ben and I am an actor turned web professional. Throughout the pandemic, I taught myself to code and then found myself working as a Project Manager at Wholegrain Digital – one of the pioneers of digital sustainability and the creators of websitecarbon.com. Over my time there, I learnt a lot about digital sustainability – but the process of auditing an entire website was very cumbersome. Now I’m excited to share what I’ve learned and what I’ve observed from the Screaming Frog SEO software.

Screaming Frog is, to my knowledge and at this time, the only website crawling software that includes the Co2.js package by The Green Web Foundation. Their software is able to retrieve the total transferred bytes of every URL that is returned in the crawl and from this, it calculates the Co2 emissions using the Co2.js package.

More importantly, it is also capable of executing custom Javascript on each page – allowing you to simulate user interaction with the webpage. This part is fundamentally crucial to loading the full assets of a webpage and thereby accounting for all of its assets – not just what’s loaded on the initial page load. Personally, I don’t know of any other performance tool that is able to do this at this point in time. You can read more about this problem, aptly termed the 1mm gap, here.

Finally, it also offers an integration with Google Analytics 4 which many consider to be the industry standard analytics platform. By utilising this, you are able to cross reference views or hits with every URL’s carbon emissions score.

Tracking cookies have a carbon cost. They are just more code! Code which has a file size which needs to be loaded onto the user’s device, which therefore has a carbon cost. With Screaming Frog, you have the option to count or discount these in your crawl under “Configuration > Extraction > Cookies”.

The case for counting them in your crawl is that arguably, most users will opt-in to cookies which will impact the overall size of their data transferred per view. I.e. this is the “worst case scenario”. The case against, is that if users don’t opt in or decline, then they won’t (and shouldn’t) be loaded in and therefore shouldn’t be counted.

You can toggle this by going to “Configuration > Crawl Config > Extraction > Cookies”.

This is a fairly straightforward one. If your website runs on Green Hosting, turn it on! If not, turn it off. You can do this under “Configuration > Crawl Config > Spider > Advanced”.

This feature also has an interesting use case in that you can see the carbon improvements that could be made if you switched to a green host. For example, if you’re not on a green host and are trying to persuade your boss or your client that they should move to a green web host – you could run the crawl once with it turned off and then again with it turned on. The scores will improve across the board because the Co2 calculation allows for a green web hosting variable to be set to true or false. It favours true and you will see the scores improve when this is turned on, therefore supporting your argument to change hosts to a green provider.

This is the most important and perhaps impressive feature that Screaming Frog can offer. Until now, it was pretty much impossible to have a website carbon test which could iterate through a website’s pages at scale, and load the full assets of each page to make the most accurate measurement.

This is because most performance testing platforms simulate a bot visiting the site and initiating a page load, rather than a real human who may move their mouse, scroll the page or make another interaction before, during or after the page loads.

This human interaction loads additional assets which are not counted in these performance tests because they are simply not initiated. For example, a large image or video embed that is being lazy loaded (i.e. doesn’t load in until it’s actually visible) three full scrolls down the page will not be counted in the initial page load because it simply won’t be loaded in. Therefore, when measuring this page, the weight of said asset will not be counted in the total bytes transferred measurement.

This is a false reading and is why you may occasionally find a website that looks very carbon heavy (e.g. full of images, slow to load, maybe uses a video header), but is scoring A’s across the board on website carbon measurement tools. Again, this problem is detailed in this article about the 1mm gap.

But now with Screaming Frog, you are able to use custom Javascript snippets on a page by page basis in your crawl – including the ability to scroll the page and therefore load all those additional assets! This is very exciting.

I have confirmed with the team at Screaming Frog and verified myself that if you enable this feature, the readings from the total bytes transferred metric become larger and it is from this reading that the Co2 calculation can be made.

I’d encourage you to enable this setting in your crawl. This will allow the software to scroll each page it encounters a set number of times (which you can configure) to ensure you have accounted for as much of each page’s content as possible.

You can do this by:

• firstly, going to “Configuration > Crawl Config > Rendering” and changing the option to use Javascript rendering.

• going to “Configuration > Crawl Config > Custom > Custom Javascript > Scroll Page”.

Further to this, you can configure the number of times you wish to scroll by opening the snippet and changing the Javascript constant “numberOfTimesToScroll” from 5 to whatever you wish.

Finally, if you’re savvy with Javascript or know someone who is, you could also conduct a crawl where you measure the impact of any other sort of user interaction. For example, you could configure it to click play on all videos it encounters and watch them for 20 seconds before moving on, or you could simulate clicking through a checkout path on an e-commerce site.

In the previous section, I recommend to use the custom Javascript snippets feature. This is only available to use if you change the rendering settings to enable Javascript rendering. Assuming you have this enabled, you can further customise your website carbon audit and crawl settings by selecting which device to emulate i.e. which device characteristics and screen widths will be used to load each page of your crawl.

When you enable Javascript rendering, the default settings are to use the “Googlebot Mobile: Smartphone” preset. You can read about how this works in Screaming Frog’s documentation here. But you are free to choose another device by selecting from the drop down list.

In theory, choosing different devices should result in different values being returned in the total bytes transferred metric. This will be because the viewport (i.e. the size) of the device will directly impact both the number and size of the assets loaded onto the page.

For example, an image on a mobile device may be rendered as 250px by 250px and weigh 180kb. But on a desktop device, that same image may be rendered as 600px by 600px and weigh perhaps 320kb, due to the increase in size. This is a very simplistic example, but the principle remains the same.

However, in practice, I have noticed some inconsistencies with how this is currently working in the software. In general, I’m still finding that the default Googlebot Mobile: Smartphone preset is returning the most accurate results – that is – the total transferred bytes metric being returned on a given webpage is closest to what I can verify the full weight of a loaded page is, through my Chrome browser’s developer tools when I use this preset.

Therefore, at this point in time, my recommendation is to stick with the default settings but feel free to experiment with this.

Again, if you notice that the majority of your audience are using desktops or tablets to view your website, it may be more appropriate to configure the crawl with that in mind.

You can alter all these settings in “Configuration > Crawl Config > Rendering”.

Screaming Frog’s default settings use a User-Agent of “Screaming Frog SEO Spider”. This means that when it crawls your website, it does so as a bot and identifies itself to the web server as the “Screaming Frog SEO Spider” bot.

However, you can change this by using one of many of their available preset User-Agents.

This is useful because different browsers will render web pages differently and you may find variations in the total bytes transferred metric depending on which browser or bot you choose to emulate.

Screaming Frog recommended I use Chrome as some websites vary greatly depending on User-Agents, but to get the most realistic reading, I’d actually recommend choosing whichever User-Agent makes up the majority of your website traffic. You can get this information from your Google Analytics account by viewing the browser information of the majority of your users.

You can do this by going to “Configuration > Crawl Config > User-Agent”.

The robots.txt file informs bots – both good and bad ones – which parts of the website they are allowed access to. There are sections which are just meant for bots and others that really aren’t. For example, in WordPress, you will often find that bots are not allowed access to the WordPress REST API for security reasons, but they’ll also be directed to the site’s sitemap so that search engines like Google can index your content. You can also use the robots.txt file to block particular bots e.g. to opt out of being crawled by OpenAI’s ChatGPT.

The decision to make here is whether you feel these mostly hidden parts of your website are valid URLs to account for in your website carbon audit – or not.

On the one hand, they do absolutely exist. They are still public facing pages, that can be found and accessed by anyone on the web, so they still have a carbon implication – albeit a small one as it’s usually only text based content. For example, here’s my sitemap.

On the other hand, usually, the only people that are even aware that such pages exist are developers and robots. It may be more helpful to your use case to only account for publicly facing URLs. It’s also usually the case that these sorts of URLs are not counted in tracking platforms like Google Analytics, so cross referencing the hits from visitors (or bots) becomes a lot harder in the final stages of this process.

In Screaming Frog, you are able to respect or ignore robots.txt. If you ignore it, all pages will be crawled, regardless of whether the crawler is meant to have access to those pages or not.

You can do this by going to “Configuration > Crawl Config > robots.txt”.

You can optionally configure the Google Analytics integration natively supported by Screaming Frog.

If you enable this, you can later cross-reference the number of views metric with the “Co2 (mg)” metric which will be returned in the crawl on a per url basis. This will allow you to account for the total Co2 emitted by every page of your website in the given time period!

I will cover how to do this part later in the article.

To do enable the Google Analytics integration, follow the usual steps to connect your Google Analytics 4 account. Then navigate to the next tab and choose the time period you are interested in. For a first crawl, I’d recommend just 1 month to check it’s working properly, but you can select any available period you have data for in your GA4 account.

Go to “Configuration > Crawl Config > API Access > Google Analytics 4 > Account Information & Date Range”.

The “Views” metric will automatically be collected in the crawl if using the GA4 integration.

Again, you could get super custom with this by filtering down your audience segments and events in your crawl settings. You could, for example, isolate the data from GA4 so that you account for the carbon emissions emitted only by users in a certain audience e.g. purchasers, or perhaps only those who have triggered certain events e.g. watched a video.