In this release, we introduce a new Nuxt config header which allows you to control whether or not the X-Robots-Tag header will be added.
We also made improvements around matching the robots.txt rules to add the meta tag and header automatically. Previously only limited path matching was done, leading to issues when using query params or wildcards. This now also gives us the ability to see exactly which robots.txt rule is blocking the page from being crawled.
🚀 Features
🐞 Bug Fixes
- Improved robots.txt matching - by @harlan-zw in #136 (e5b91)