Crawl won't start, gets stuck or finds 0 pages

SEOAsist starts from your home page and sitemap, then follows links on the same domain. The crawler identifies itself as SEOAsistBot and respects robots.txt. Each site is crawled automatically every week, and you can crawl it by hand at any time.

#Symptoms

• A Could not start crawl warning: “You don’t have permission or crawling is disabled.”
• A A crawl is already in progress warning, but no crawl seems to be running.
• The crawl finishes with only 0 or 1 pages.
• Technical Audit lists many “Page unreachable (timeout / connection error)” or “Redirect loop” issues.

#Fix

1. Is crawling off? Open the site from Sites. In General settings on the General tab, tick Weekly automatic crawl enabled and click Save. With this box off, manual crawls don't start either.
2. Do you have permission? Owners, Admins and Members can start crawls. Viewers can't.
3. Is the quota used up? Check the Monthly crawl quota meter in the left menu. When the monthly page quota is used up, you can't start new crawls by hand, and the panel may show “A crawl is already in progress”. The quota resets at the start of the month, or you can upgrade your plan.
4. Is the site URL right? The Site URL field must be the site's final address (correct https, correct domain). If the home page redirects to another domain, the crawl stops there, because only pages on the same domain are followed.
5. Is robots.txt blocking it? If example.com/robots.txt has a rule like Disallow: /, blocked pages are skipped. If the home page is blocked, you get 0 pages. Allow SEOAsistBot.
6. Is bot protection blocking it? A firewall or bot protection (WAF, “bot fight” modes) may block the SEOAsistBot user agent. Add it to the allow list.
7. Is the page limit too low? Crawl page limit is the most pages one crawl visits. The server-side cap is 500.
8. Once fixed, click Crawl now on the site page or Crawl this site on Technical Audit.

#Still not working?

#The crawl has shown “running” for a long time. What should I do?

You can't start a new crawl for the same site until the current one ends. When the system restarts, unfinished crawls go back into the queue automatically. If nothing moves for a long time, email destek@seoasist.com.

#Can I get notified when a crawl finishes?

Yes. Under Settings → Notifications, select the Crawl finished event. It's off by default.

#When does the automatic crawl run?

The weekly crawl runs every Sunday morning (03:00 UTC). Sites with the box turned off are skipped.