How to connect Screaming Frog with ChatGPT to use in your Crawls

The latest version of Screaming Frog is amazing. The inclusion of custom JavaScript to run during the crawl is an incredible advancement for the tool.

This new update allows us to connect ChatGPT directly to the tool to use GPT-3.5 on the URLs.

In this post, I will walk you through the steps you need to follow to set it up.

If you prefer to watch it in a video, you can find it below:

https://www.youtube.com/watch?v=GaHoj4AKXys

These are the steps you need to complete to use ChatGPT in Screaming Frog:

1. Enable JavaScript rendering in Screaming Frog. Go to Crawl Configuration > Rendering.

2. You need to have credits in the API of the ChatGPT Business account. It is very likely that you will need to add 10 dollars for the first test, just like I did. Remember that it only works with paid ChatGPT accounts.

3. Before starting the crawl, go to Crawl Configuration > Custom JavaScript and click on “Add from library.”

Here you can see the default options and use a custom one. To use a custom one, you will need to choose the one called: (ChatGPT) Template

4. You will need to add your API key once you select the ChatGPT JavaScript. To do this, click the JS button. From that screen, you will need to enter your API key on the left side as shown in the image. On the right side, you can test the prompt you have selected.

Remember that the crawl you perform consumes API calls, so I recommend you carefully choose what you are going to crawl.