Why assume someone who wants to index all your site is going to play by the non-mandatory rules. Its not even law that you have to write a search engine that respects those rules.

And a criminal would not respect the law, even if it was the law, and could easily ignore robots and nofollow directives if they wrote their own spider.