RolesIndexBot

A research crawler that reads public job postings on employers' careers sites.

What it does

RolesIndexBot reads the job postings that employers publish on their own careers sites and the public pages of the jobs systems those sites use. It is a research project that studies hiring across listed companies: how many roles are open, where, and of what kind. It reads only pages anyone can open without signing in.

How it behaves

How to stop it

Add this to your site's robots.txt; it takes effect within a day:

User-agent: RolesIndexBot
Disallow: /

To limit it to parts of a site, disallow the paths you want left alone.

Data

It keeps the text of public job postings and their details (title, place, date, type of role) for research. It does not collect applicants' data. Where a posting itself names a recruiter or the role a job reports to, that text is kept as part of the posting; anyone named can ask, at the contact below, for it to be removed.

Contact

Questions, problems or removal requests: weork.work@gmail.com (the project's own mailbox). Blocking the crawler in robots.txt works without writing to us.