Search engines work in stages: they discover a page, crawl it (read it), index it (store and organise what they learn) and later rank it when someone searches. A new or small site may take time to appear, and Google does not promise to index every page. You can help by making pages reachable, linking them together and submitting a sitemap.
Discover
Follows links and reads sitemaps.
Crawl
Visits and reads the page.
Index
Stores what it learned.
Rank
Chooses what to show for a search.
The stages
- Discovering. Google finds a page through links from other pages or from a sitemap you provide.
- Crawling. A program called a crawler visits the page and reads it.
- Indexing. Google stores and organises what it learned. Only indexed pages can appear in results.
- Ranking. When someone searches, Google picks and orders the pages it thinks fit best.
Google describes these basics in its SEO Starter Guide (external site, opens in a new tab), including that there is no guarantee a site will be crawled, indexed or served.
Why a page may not appear
- It is new. Discovery can take days or weeks.
- Nothing links to it. A page with no links in and no sitemap entry is hard to find.
- The site is blocked. A setting or file may tell crawlers to stay away.
- The page is a copy. Google picks one main address. See canonical URLs (external site, opens in a new tab).
- The content is thin. Very little useful text gives search engines little to work with.
How to help Google
- Link your pages together, so every page can be reached from another.
- Give each page a clear title and its own address.
- Provide an XML sitemap.
- Keep one main address for each page, and make the mobile version complete. See mobile-first indexing (external site, opens in a new tab).
- Claim a Google Business Profile for local searches.
Common mistakes
- Expecting instant results. Indexing is not immediate.
- Paying for a guarantee. Nobody can guarantee indexing or position.
- Leaving a “do not index” setting on after launch. Developers sometimes use it while building. Check it was removed.
Key terms and where the definitions come from
Show key terms and sources
These explanations are ADA's own words, adapted from the independent sources named. Those organisations do not endorse or work with ADA, and the examples are illustrations, not real clients.
What is an XML sitemap?
An XML sitemap is a file that lists the pages of a website you want search engines to know about. It helps them discover pages. It does not guarantee that a page is crawled or shown.
Source: Adapted from Google Search Central, Learn about sitemaps.
Reference: Google Search Central: Learn about sitemaps (external site, opens in a new tab)
For a Namibian business: A lodge adding new seasonal pages can list them in its sitemap so search engines find them sooner, without waiting to follow a link.
Related ADA service: Search visibility (SEO)
What is a canonical URL?
A canonical URL is the address you tell search engines is the main version of a page when the same content can be reached at more than one address.
Source: Adapted from Google Search Central, What is URL canonicalization.
Reference: Google Search Central: What is URL canonicalization (external site, opens in a new tab)
For a Namibian business: A shop whose page opens at both its www and non-www address should name one of them as the main address so the two are not treated as competing copies.
Related ADA service: Business website design
What is mobile-first indexing?
Mobile-first indexing means Google mainly uses the mobile version of a page's content to index and rank it.
Source: Adapted from Google Search Central, Mobile-first indexing best practices.
Reference: Google Search Central: Mobile-first indexing best practices (external site, opens in a new tab)
For a Namibian business: If a company's phone version hides its services or contact details, search engines may not see them either.
Related ADA service: Website redesign
Common questions
How long does Google take to find a new website?
It can be days or weeks. Links from other sites and a submitted sitemap can help. There is no set time.
What is the difference between crawling and indexing?
Crawling is reading a page. Indexing is storing and organising what was read so it can be shown in results.
Why is my page not on Google?
It may be new, blocked, not linked from anywhere, a copy of another page or of low value. Check each in turn.
Do I have to submit my site to Google?
No, but a sitemap can help Google find your pages faster, especially on new sites.
Is being indexed the same as ranking well?
No. Indexed means stored. Ranking is where the page appears for a search.
Need help with this?
Optional. You do not need any ADA service to use what you have just read.
- Search visibility (SEO)
Checks that crawlers can reach and understand your pages. - Customers cannot find us on Google
What to check first. - Website care plans
Ongoing checks after launch.