Plan crawl, indexing, and canonical URLs for directory pages
- Thread started
- Last updated
What you’ll achieve
Check directory pages’ robots and canonical tags, and choose which city pages search engines should index.
My site
Choose which directory pages search engines should index, then check their canonical URLs and robots tags.
Create one city-and-category landing page
Keep cities in location custom fields so visitors can filter the directory across categories. Create a maintained Custom List for a city-and-category combination you want to link from your city guide, such as Hotels in Chicago.
-
Open your Hotels category on the site. Use the city field or search form to select Chicago. Confirm that the results include Chicago hotels and exclude hotels in other cities.
-
Copy the search query string after
?from a working results URL. Use the field option value in that URL, which can differ from the displayed city name. -
Create a Joomla menu item of type JReviews > List Pages. Set Type of Menu to Custom List. Select the Hotels category and paste the city query string into Custom Parameters. Set the menu alias for the city page.
-
Create a JReviews Custom List page in the WordPress menu editor. Select the Hotels category and paste the city query string into Custom Parameters. Set the page's city-specific permalink.
-
Remove ordering, layout, pagination, and tracking parameters from the copied query string. Keep the city filter. For a field named
jr_citywith option valuechicago, the city filter isjr_city=chicago. -
Give the page a title such as "Hotels in Chicago", a description of the result, and useful visible city content. Link to the page from your Chicago guide and your site's navigation.
-
Open the saved page while logged out. Confirm the intended hotels appear before you apply robots or canonical changes. Repeat for a second city with its own field value and page text.
The menu item can remain in a menu you do not display, while links from your city guide lead to it.
A city landing page does not change the URL structure of each hotel detail page.
Collect the directory URLs to check
Start at your Hotels category. Filter to Chicago, change the order, and open the next results page. Copy the actual URLs. Add your "Hotels in Chicago" Custom List page and one hotel detail page to the list.
Record the pages in a table like this. Use your own URLs. The paths shown here are examples:
| Page | Intended purpose | What to check |
|---|---|---|
/hotels |
All hotels | Category content and page metadata |
/chicago/hotels |
Maintained city guide and hotels | City content, correct results, and stable navigation link |
| A filtered Hotels URL | Temporary visitor selection | Actual filters, robots tag, and canonical |
| An ordered Hotels URL | Same collection in another order | Whether membership changes and which canonical is emitted |
| Page 2 | More hotels | Different results and a page-specific URL |
| A hotel detail URL | One hotel | Hotel-specific canonical, not the city's list URL |
Record the page title and the search-engine instructions on each page. The steps below show where to find those instructions. JReviews and an installed SEO extension can both add them, so check the page visitors actually receive.
Choose the page visitors should find
Link your city guide and menus to the maintained Custom List. Give that page useful city content and a distinct title. Keep the same destination in internal navigation and your sitemap.
A filtered page and a Custom List are not automatically interchangeable. Compare their complete content and results. Use a cross-page canonical only for duplicate or very similar pages. A different city, category, or later results page needs a separate decision. Google treats canonical annotations as signals, so they do not guarantee your preferred URL will be selected.
Read the page's search-engine instructions
Two instructions matter for this example:
- A canonical URL tells search engines which URL you prefer when two pages have duplicate or very similar content. For example, you may prefer
/chicago/hotelsover a search URL that shows the same hotels. Google can choose another URL. See Google's canonical URL guidance. - A robots tag can tell search engines not to include a page in their search results. The value
noindexmeans that the page should stay out of those results. Visitors can still open it on your site. See Google's robots tag reference.
Before adding code, read what your site already sends:
- Open your Hotels in Chicago page. Right-click the page and choose View page source.
- Use the browser's Find command to search for
canonical. Look for a line like this, using your own domain and page URL:
<link rel="canonical" href="https://www.example.com/chicago/hotels">
- Search for
robots. If the tag containsnoindex, the page is asking search engines to leave it out of search results. For a public city guide you want people to find through search, that instruction needs attention. A missing robots tag does not by itself mean the page is blocked. - Repeat these checks on the filtered Hotels results and page 2. Write down each page's canonical URL and whether it contains
noindex.
An SEO extension can also send noindex outside the page HTML. To check, open your browser's developer tools, select Network, and reload the page. Select the main page request, open Headers, and look under Response Headers for X-Robots-Tag. Record its value if present. An instruction there still matters even when the HTML robots tag looks correct.
What JReviews 6 already handles
JReviews 6 already removes sort-order and layout choices from the canonical URL. These choices can change how a hotel list looks without changing which hotels belong to it.
A city filter or a later results page can show different hotels. JReviews keeps filters it recognizes and valid later page numbers in the canonical URL. It can also add noindex, follow to filtered results, so check the actual tag before assuming that Google can list that page.
If your saved Hotels in Chicago page already has the canonical URL you want and no instruction to exclude it from search, you may not need custom code. Use the checks above to decide.
Check the tags on JReviews 5
Use the same page-source checks above. If an SEO extension manages your page titles and search settings, open its settings for Hotels in Chicago too. A setting there may explain a canonical URL pointing elsewhere or a noindex instruction. Correct that setting before adding another code example that changes the same tag.
Set page metadata and inspect canonical links
A filtered search does not become indexable just because its results match a Custom List page. JReviews does not automatically choose your preferred city-guide URL as the canonical. Saved Custom List conditions and extra filters added to a page URL are handled differently. Check the actual response for your category, curated list, filtered search, and later results pages.
Joomla. Edit the Custom List menu item's Page Display and Metadata settings.
WordPress. Edit the Custom List page's Page & SEO Settings and page content.
Set a title and description for Hotels in Chicago. Check the category page too. Its title and description should describe all hotels rather than only Chicago.
Inspect the actual canonical on each page. If your installed release already produces the desired link, keep that output. If the CMS or an SEO extension adds a conflicting canonical, use that component's supported page controls to resolve the conflict. Change one component's settings at a time, then inspect the HTML head again.
If custom canonical logic is needed, base it on the page URL and all active filters. Do not choose a canonical from the first listing in the results. Sorting, empty results, or pagination can change that listing. Do not add a canonical inside a listing-card loop.
Keep page 2 distinct when it contains additional results. Do not point every results page or every hotel detail page at the first city landing page. Canonical URLs describe equivalent pages, not a desired navigation hierarchy.
Map an equivalent filtered page to its city guide
Use this example when a public filtered page and your maintained city Custom List show equivalent content and results. The example changes the canonical to the maintained page and removes noindex for the same exact source page. It keeps other robots directives. Do not use it for a private page or a page you deliberately exclude from search.
Copy the complete source path and query string from the URL worksheet. Include the site's subdirectory, if any. Keep every filter that affects membership. Replace the example source paths, category ID, city value, and destination in the map below with your own values. Use the final public HTTPS URL for the destination.
The code matches the complete URL after your domain, including its path and query string. It also checks that this is the expected kind of results page, that it is page 1, and that it contains results. A different path, an extra query parameter, a different kind of page, page 2, or empty results keep their normal tags. Copy the path and query exactly, including parameter order and encoded characters. Add another entry only after comparing that page's content and listings with its destination.
Save this file as city-canonical.php in your overrides folder.
Use templates/jreviews_overrides/city-canonical.php.
Use jreviews_overrides/city-canonical.php.
<?php
class GuideCityCanonical
{
public static function destination(string $identity, array $pagination): ?string
{
if ((int) ($pagination['page'] ?? 0) !== 1 || (int) ($pagination['total'] ?? 0) < 1) {
return null;
}
$pages = [
'/hotels/tag/city/chicago' => [
'identities' => ['tag', 'categories.search'],
'canonical' => 'https://www.example.com/chicago/hotels',
],
'/advanced-search/search-results/?cat=23&jr_city=chicago' => [
'identities' => ['search', 'categories.search'],
'canonical' => 'https://www.example.com/chicago/hotels',
],
];
$row = $pages[(string) ($_SERVER['REQUEST_URI'] ?? '')] ?? null;
if (! $row || ! in_array($identity, $row['identities'], true)) {
return null;
}
return $row['canonical'];
}
public static function robots(string $robots): string
{
$directives = array_filter(array_map('trim', explode(',', $robots)));
$directives = array_filter($directives, fn (string $directive): bool => ! in_array(
strtolower($directive), ['index', 'noindex', 'follow', 'nofollow', 'none', 'all'], true
));
return implode(', ', array_merge(['index', 'follow'], array_values($directives)));
}
}
Each identities line names the kind of results page for that URL. Use the values for your selected setup below. Keep only map entries for URLs you actually use.
Use tag for a URL opened by clicking a searchable field value, such as Chicago on a hotel. Use search for a URL produced by submitting the search form.
Use categories.search for both URLs opened by clicking a searchable field value and URLs produced by submitting the search form.
Register both metadata filters
Enable Listing List Pages in Early Access before using these modern hooks. Register the filters in hooks.php in the same overrides folder.
Use templates/jreviews_overrides/hooks.php.
Use jreviews_overrides/hooks.php.
Add the callbacks marked with +. They load city-canonical.php and register both filters.
- If
hooks.phpalready exists, add the marked lines at the end of the file. The copy button copies only those lines. - If you are creating
hooks.php, type the two dimmed lines first, then add the marked lines below them.
<?php
defined('MVC_FRAMEWORK') or die;
require_once __DIR__.'/city-canonical.php';
fwd_add_filter('jreviews:page_meta:canonical_url', function ($url, $name, $pageType, $attributes) {
$destination = GuideCityCanonical::destination((string) $pageType, $attributes['pagination'] ?? []);
return $destination ?? $url;
}, 20, 4);
fwd_add_filter('jreviews:page_meta:robots', function ($robots, $name, $pageType, $attributes) {
$destination = GuideCityCanonical::destination((string) $pageType, $attributes['pagination'] ?? []);
return $destination ? GuideCityCanonical::robots((string) $robots) : $robots;
}, 20, 4);
Register both metadata filters
Register the filters in filters/filter_functions.php in your overrides folder. JReviews 6 Legacy list pages use these hooks too; their automatic canonical handling is described under What JReviews 6 already handles above.
Use templates/jreviews_overrides/filters/filter_functions.php.
Use jreviews_overrides/filters/filter_functions.php.
Add the callbacks marked with +. They load city-canonical.php and register both filters.
- If
filter_functions.phpalready exists, add the marked lines at the end of the file. The copy button copies only those lines. - If you are creating
filter_functions.php, type the two dimmed lines first, then add the marked lines below them.
<?php
defined('MVC_FRAMEWORK') or die;
use Clickfwd\Hook\Filter as GuideCityCanonicalFilter;
require_once dirname(__DIR__).'/city-canonical.php';
GuideCityCanonicalFilter::add('page_canonical_metatag', function ($url, array $params) {
$pagination = $params['vars']['pagination'] ?? $params['pagination'] ?? [];
$destination = GuideCityCanonical::destination((string) ($params['route'] ?? ''), $pagination);
return $destination ?? $url;
}, 20);
GuideCityCanonicalFilter::add('page_robots_metatag', function ($robots, array $params) {
$pagination = $params['vars']['pagination'] ?? $params['pagination'] ?? [];
$destination = GuideCityCanonical::destination((string) ($params['route'] ?? ''), $pagination);
return $destination ? GuideCityCanonical::robots((string) $robots) : $robots;
}, 20);
Clear the JReviews file registry and page caches. On the mapped filtered page, confirm that the canonical is your city guide and robots permits indexing and following links. Confirm that the destination returns a public page with its own canonical and no noindex meta tag or response header. If another SEO component supplies conflicting metadata, resolve it through that component's supported controls before using the map.
Open the same source with an additional city, rating, or keyword filter. The city-guide mapping must not apply. Repeat with another category, a different sort or layout, page 2, empty results, and one hotel detail URL. Each unlisted URL keeps the normal canonical and robots policy. Add a sort or layout variant only as its own exact entry after confirming equivalence. A later results page needs its own equivalent destination, so do not add it to a map that targets the first city page.
Separate indexing from crawl control
For a public utility page you want excluded from search, use the CMS or SEO integration's page-specific robots control to emit:
<meta name="robots" content="noindex">
Inspect the resulting response to confirm that the intended page receives the tag and the city landing page does not. Google must be able to visit the page, also called crawling, to read its noindex. Do not block that visit through your site's robots.txt file while relying on the tag. See Google's noindex instructions.
Review robots.txt separately. Blocking every URL with a query string can block useful pages and prevent search engines from reading their metadata. Keep resources required to render public pages accessible. Robots.txt controls crawling; it is not a dependable way to remove a URL from search or protect private content. See Google's robots.txt guide.
Verify every affected page
Revisit each worksheet URL while logged out. Confirm that filters still work and page 2 shows its own listings. Check that each response contains the intended robots instruction and one consistent canonical link. Confirm the canonical destination resolves to the intended public page.
When changes reach your public site, use Search Console URL Inspection to compare the declared canonical with Google's selected canonical and inspect rendered content. Recheck after recrawling. A local HTML check proves emitted metadata, not Google's indexing decision or a ranking improvement.
Learn more
- Search Engine Optimization (v5)
- Custom Lists (v5)
- Search Engine Optimization (v6)
- Custom Lists (v6)
- Early Access (v6)