User-agent: * Disallow: /redirect Disallow: /images User-agent: grub-client Disallow: / User-agent: grub Disallow: / User-agent: looksmart Disallow: / User-agent: ia_archiver Disallow: / User-agent: thesubot Disallow: / User-agent: InsurancoBot Disallow: /