As anyone who has ever managed a website knows, it can be difficult to keep up with, all the time.
This is especially true when software updates are applied, pages are updated, images change, or old pages are moved and / or removed. This is just some of the maintenance we deal with on a daily basis with our monthly support clients.
While one website is difficult enough to keep up with, it’s exponentially more complex to apply updates and maintain the quality of dozens.
As Louisville Web Nerds has grown over the past two years, the manual process of checking for broken links, missing images, broken javascript or CSS files, 404 and 500 errors, and the other issues that can come up with routine updates, has become far too cumbersome to be a manual process.
This is why we chose to license the Screaming Frog web crawler, one of the top web crawlers in the world, to increase the quality of the websites we manage for our clients.
How do we use Screaming Frog?
Well, this is a pretty sophisticated piece of software with an extensive feature list. You can use it for dozens of different things, including taking down a website if you’re not careful with the crawl rate. 🙂
We use it to do a full crawl of all of the links on our clients’ websites once a week, in a slow and controlled manner.
We check for things like:
- Broken links that are causing 404 errors
- Server errors that are causing 500 errors
- Redirect loops that are causing memory errors
- Missing images that may be causing a strange appearance
- External links that may be pointing to malicous sites (this occurs when a website may be hacked)
- CSS and Javascript files that may not be loading correctly
- SEO issues including blocked URLs, blocked websites (gasp!), and many others
All of these things can happen after routine updates, so we check for them often, once a week.
This process would take a human hours for a single website. Believe me, I used to do it. And that human still may not find everything.
By using the Screaming Frog software, we can run through a dozen or more websites with thousands and even tens of thousands of URLs (when configured correctly), in a safe manner that will show us everything we need to know.
We can do this with an automated process that runs in the background while we’re busy building websites, and then check it when it’s finished. We’ve even done a few scans with over 1 million URLs, which were quite exciting to be a part of.
Sometimes we’ll find a few broken links and fix them. Most of the time things are pretty good. Since we run this software weekly, quality issues tend to diminish over time for each website where we deploy this process.
Many sites often have dozens of broken links that they’re not even aware of. This can not only contribute to search engine rankings falling, but can also cause users to lose trust, and be unable to find important resources on these websites.
Personally, since I started using this software on my own websites, I’ve slept a lot better knowing that I have a full understanding of what every link is doing on my website.
You’ll feel equally at ease with Screaming Frog and Louisville Web Nerds in your corner, which is our goal. We want to take the worry of your website out of your hands so you can focus on building your business.
All of our monthly support plans include this service, which is a $200 value for the software alone, and an extensive amount of training and experience to know how to use it.
If you need help with your WordPress website, get in touch with us and we’ll be happy to see how we can help.