RolesIndexBot
A research crawler that reads public job postings on employers' careers sites.
What it does
RolesIndexBot reads the job postings that employers publish on their own careers sites and the public pages of the jobs systems those sites use. It is a research project that studies hiring across listed companies: how many roles are open, where, and of what kind. It reads only pages anyone can open without signing in.
How it behaves
- It reads
robots.txtbefore every site and obeys it, including rules written forRolesIndexBotby name. - Its user agent always says what it is:
Mozilla/5.0 (compatible; RolesIndexBot/0.1; job postings research crawler; obeys robots.txt; +this page). - It visits each site about once a day, in the evening at the employer's head office, and skips pages that have not changed since its last visit.
- It keeps its request rate low, slows down when a site answers 429 or 503, and stops when a site keeps refusing it.
- It does not sign in, create accounts, solve CAPTCHAs or other challenges, or use proxy networks.
- Its requests may carry a Web Bot Auth
signature; the public key is at
/.well-known/http-message-signatures-directory.
How to stop it
Add this to your site's robots.txt; it takes effect within a day:
User-agent: RolesIndexBot Disallow: /
To limit it to parts of a site, disallow the paths you want left alone.
Data
It keeps the text of public job postings and their details (title, place, date, type of role) for research. It does not collect applicants' data. Where a posting itself names a recruiter or the role a job reports to, that text is kept as part of the posting; anyone named can ask, at the contact below, for it to be removed.
Contact
Questions, problems or removal requests: weork.work@gmail.com
(the project's own mailbox). Blocking the crawler in robots.txt works without writing to us.