Should my sitemap and robots.txt have HTTP or https URLs for the given scenario?

Battery Exchange Network

The Stack Exchange network includes 175 question-and-answer communities, including Stack Overflow, the largest and most reliable online community on which developers can learn, share knowledge and build their careers.

Visit Stack Exchange

seo – How to change the structure of my URLs with the sitemap for an active website?

I change the URL of my website to improve SEO. The current sitemap contains the following URLs:

mysite/browse/1    /* 1 is the ID for fashion */
mysite/browse/2    /* 2 is the ID for real estate */

Now I have changed the URLs to the following format:


The code is ready to be published with the new site map. If I publish the new code, the old URLs would no longer be valid, namely: mysite/browse/2 would return "404 – page not found".

I think I need to release the code and send the new site map to Google Search Console. But I think it would take some time for Google to analyze the new sitemap. Therefore, during this time, all my old URLs that appear in the search result would return the error "404 – page not found".

How can I mitigate this situation?

Sitemap in Bing waiting for two weeks

A month ago, I built a new website. Two weeks ago, I sent the sitemap in the Bing search console.

At this point, he is still waiting. What should I do for Bing to move?

seo – Hierarchical vs Flat sitemap

Your question is divided into two categories, each with different considerations:

XML Sitemap
In your XML sitemap, you can specify both the frequency of change and the ranking of priorities for different pages. This helps search engines determine the importance of a page on the site and the frequency with which it must be searched for changes. Avoid listing duplicate content pages such as / fashion / women / shoes and / fashion / shoes / women. In this case, choose the canonical URL to submit to the search engines.

URL structure
Respecting a minimum hierarchy in your URL structure for each page or faceted search results will allow a search engine to index your site and view top-level category pages. (For example, the result of searching a department store may indicate different departments).

User experience
A menu too deep can hinder the user experience, which might be overwhelmed by choices or simply tired of exploring many levels of navigation. Having a good faceted search interface with the most common categories as predefined filters can help avoid a too complex menu. In the same way, having an Ariane thread on the page can help users to return to the top-level category or sub-category pages.

Search Engjne Optimization for the products themselves
For each product on your site, consider adding schema microdata in your markup. This will help search engines identify the individual products available for sale on your e-comm site. See

Also remember to add an RSS or XML feed and submit it to several sources to improve the reputation of your site.

seo – Hierarchical vs Flat site-map

I want to create a site map for my ecommerce site.

On the homepage, there is a multilevel menu like this:

enter the description of the image here

The user can click on any of the following links since the Home page:

  1. Fashion -> enter the fashion department
  2. Fashion> Women -> enter the sub-department of women under Fashion
  3. Fashion> women> Shoes -> enter the unit of shoes under the women

There are also page links Help me and about us this use can range from Home Page.

When it comes to SEO, what is the best way to create a site map?

enter the description of the image here

So, the user has 2 ways to navigate to Womensub-department, either directly via Home page or while sailing from home to Fashion and then Women page.

sitemap – How to tell bots to forget about a website and reindex it from scratch

This is not possible You need to map your old URLs on the news with referrals for SEO and user experience.

Google never forgets old URLs even after a decade. When you migrate to a new CMS, you must implement page-level redirects.

If there is no equivalent for a particular page, you can leave it 404 and Google will remove it from the index. Instead, using "deleted 410" allows Google to remove index URLs as soon as they are crawled without the 24-hour grace period used by Google for "404 no found".

There is no directive to tell robots not to use an old site in the search console or robots.txt file.

And if you do not redirect?

It may be that the redirection is too much work or that your new CMS does not make it easy to implement the redirection.

If you choose not to implement redirects, it will be something like starting over again. Google will see that your old URLs return the 404 status and remove them from the search index.

Your new URLs will eventually be indexed, but this may take some time. Changing any of your URLs without redirects is a serious sign that your site is not stable and can not be trusted. All your rankings will be lost and your site will start again.

Googlebot will continue to explore old URLs for years. For that, hope is eternal that you can one day go back these pages.

If you redirect, all your incoming links, user bookmarks, and most of your current rankings will be preserved.


So, why do search engines have no "reset" button? Because there are almost always better options. In your case, it is much better to redirect.

In the event that a site is penalized, Google does not provide a reset button, as this could remove all penalties.


So, how do you implement redirects? You need a list of your old URLs. You can start with a sitemap of your old site. You can also get the list from your server logs, Google Analytics, or even the Google Search console.

If you have planned in advance, your URLs in your new CMS will be similar and you can implement a rewrite rule to manage them. If there is a pattern between the old and the new URL, it can act from one line in one .htaccess file to send redirects for the entire site.

If you need to manually search for new URLs and map thousands of them one by one, you can consider: RewriteCarte functionality.

seo – GSC: Unable to retrieve the sitemap

I'm trying to submit a very simple sitemap (for test purposes only) to Google Search Console but, unfortunately, I'm constantly getting the following error message:

╔══════════════╤═════════╤══════════════╤═════════ ══╤══════════════════╤═════════════════╗
Site Map │ Type │ Submitted Last Reading │ State │ URL Discoveries ║
╠══════════════╪═════════╪══════════════╪═════════ ══╪══════════════════╪═════════════════╣
/Sitemap.txt │ Unknown 17 Jul 2019 │ * Unable to retrieve * 0 ║
╚══════════════╧═════════╧══════════════╧═════════ ══╧══════════════════╧═════════════════╝

By clicking on it, there is an additional error message: "(!) Can not read site map".
However, if you click on "OPEN SITEMAP", it will open normally.

An idea of ​​what's going on?

Sitemap file: sitemap.txt
Server: Apache (Debian)

How to warm the server cache without using a sitemap?

I need to write a script that warms the cache when my server reboots. And I want to cache the first 10 pages of my website. I have these first 10 pages in one .txt file each URL on a new line in the same directory.
Can any one help me write a script to warm up the cache with these pages with the help of a bash script, please?
Any help is appreciated.

seo – Should I use an xml sitemap instead of a txt for a site with deeply nested product pages?

The context

I have a B2B parts website with about:

  • 25 parent categories (organized hierarchically)
  • 150 leaf categories (models)
  • 250 products (unique articles, each with quantity = 1)

Target visitors are looking for a specific spare part.
Generally, they do not hesitate between several brands and products, as in the consumer segment.

The site is intended for specialists (niche market).

Despite several optimizations, the site remains poorly referenced in the search results, compared to those of the competitors.

I must admit that I am not a fan of social networks, so there are only few links to the site, coming from specialized forums.

Publishing many products on the homepage can help reference the site, but would also create duplicate content with dedicated product pages.

In this thread, the general consensus is that there is no problem using a txt sitemap instead of an XML. However, I am not sure of this in the context where the pages to be indexed are buried in the middle of the hierarchy and that the search engines ignore the intermediate levels.

How pages are currently indexed

Google was able to index pages for sheet categories and products, which were provided via two text sitemaps (URL lists):

Site map with leaf categories:

Site map with products:

The products are mainly accessible via a search field, where the visitor enters the model he wishes to acquire for spare part (s). The model name is used as Friendly URL and .htaccess file redirects directly to the sheet category page.

# Currently, no friendly URL for intermediate categories (branches).

# Friendly URL for sheet categories (templates)
RewriteRule ^ A_model $ /index.php?cmd=category&cat_id=123 [L]
RewriteRule ^ Another_model $ /index.php?cmd=category&cat_id=124 [L]

Category pages contain links to unique spares.

Friendly URLs are also used and redirection is done with .htaccess file.

# Friendly URL for unique products
RewriteRule ^ A_product $ /index.php?cmd=products&prod_id=456
RewriteRule ^ Another_product $ /index.php?cmd=products&prod_id=789

For the convenience of the user, if only one spare part is available for a given model, there is an automatic redirection of the sheet category page to the single product page, so that the address of the category behaves like a tiny URL (or a gateway if you prefer). on the product page.

If the visitor wants Browse Categorieshe can do it if a Ajaxified tree whose developed nodes load the subcategories on the fly. (For this, the site uses dynatree.js with a delayed load.)

So, the robots are aware of the relevant landing pages for sale (sheet categories and product pages) but – because they do not have an XML sitemap -, the site may appear to them as unstructured (no hierarchical structure they know).

Why I used .txt sitemaps rather than the .xml so far:

  • Simplified Maintenance: I just need to add a new link when a new product or category is released.
  • Targeted visitors are experts in their field,
    who from the beginning know which model / piece they are looking for.
  • Intermediate categories (tree branches) are almost irrelevant – apart from
    see the different families of products available – and therefore does not need to be referenced.


  1. Do I have to create friendly URLs for intermediate categories and add
    the site map
    in order to make the site more structured, given
    that these pages would create duplicate content with the sheet
    categories and product pages?
  2. In this particular case, should I to switch from .txt sitemaps to XML? (although the interview would be much more difficult).
  3. I plan to replace ajaxified tree with an ajaxified navigation based on tags (filters). Would that make SEO even worse?
  4. Since the homepage looks more or less like a search engine (ie, with little content), would you recommend adding "blah blah blah" to it – even if it is useless for the visitor – to attract more traffic?

information architecture – Are the tabs and / or steps of an assistant displayed as separate areas in a sitemap diagram?

I create a sitemap for a business application.

For a section of the application, there is a calendar editing feature. Once clicked, there are three sections / or different types of calendars to configure.

  1. Start / end dates for the entire project
  2. Blocked dates (public holidays, etc.)
  3. Start / end dates for specific tasks within the project

We are currently using a step-by-step wizard to modify the calendar. The user must therefore define the dates in this order.

In my sitemap, do I map each step as a separate area or would it be in a separate user flow diagram?