I have already shared my opinion on the variety of various Robots.txt checkers and verifiers: be aware of them but use those with caution. They are full of errors and misinterpretations. However I ...