The Hiring Record

About our crawler

We operate an automated crawler that reads publicly published job listings and public government records. This page explains exactly what it does, how to identify it, and how to contact us or ask us to stop.

If you administer a site our crawler visits and something we are doing is causing you a problem, email crawler@hiringrecord.com. We read every message and we respond. In almost every case we can solve the problem by slowing down rather than by you having to block us.

How to identify our crawler

User agent

HiringRecordBot/1.0 (+https://hiringrecord.com/crawler; crawler@hiringrecord.com)

Source addresses

Address	Region	Reverse DNS
155.138.160.201	Chicago, US	crawl-01.hiringrecord.com

Every address we crawl from has a reverse DNS record resolving to hiringrecord.com, and the forward DNS confirms it. If a request claims to be us but does not verify in both directions, it is not us. We keep this table current, and anything not listed here is either stale or an impersonator. We would like to know about it.

We do not use residential proxy networks, rotating consumer IP pools, or any other infrastructure designed to disguise the origin of our traffic. This is a deliberate and permanent commitment, not a description of current practice.

What we collect

• Job listings from applicant tracking system endpoints that vendors publish for syndication
• JobPosting structured data that sites publish in their own page source for search engines
• Public filings from federal and state agencies

We record when a listing appears, when its content changes, and when it is removed. That history is the basis of our research.

What we do not collect

• Anything behind a login. We do not create accounts, submit applications, use credentials, or access any page that requires authentication.
• Personal information. We do not collect names, contact details, or any identifying information about recruiters, hiring managers, employees or applicants. Where a listing contains a person's name, we discard it before storage.
• Applicant or candidate data of any kind.
• Anything a site's robots.txt disallows for our user agent.

How our crawler behaves

• Once per day, in a fixed overnight window, US Eastern time
• No more than one request at a time to any single host, with a delay and randomised spacing between them
• Conditional requests. We send If-Modified-Since and ETag headers and honour 304 Not Modified, so we transfer almost nothing when nothing has changed
• Backoff. We honour Retry-After and back off exponentially on 429 and 5xx responses
• Spread out. We distribute requests across the crawl window rather than issuing them in a burst

If our traffic is still too much for you, tell us a number and we will crawl at that rate.

robots.txt

We read and honour robots.txt on every host we visit, and we cache it for no more than 24 hours.

To set a specific rule for our crawler:

User-agent: HiringRecordBot
Crawl-delay: 10
Disallow: /some-path

To block us entirely:

User-agent: HiringRecordBot
Disallow: /

We also honour a blanket User-agent: * disallow.

Opting out, and what it means

You can block our crawler at any time using robots.txt, or by emailing crawler@hiringrecord.com. We will honour the request and confirm it in writing.

We want to be straightforward about what happens next, because being vague about this would be worse than stating it plainly.

If you block our crawler, we stop reading your job listings. We will not route around the block, and we will not use another identity to collect the same data.

We continue to use public government records. Filings with federal and state agencies are public records. Blocking our crawler has no effect on those, and we could not remove them from our research even if we wanted to.

Your company page will note that our listing data is limited at your request. We will not describe this as suspicious or draw any inference from it, because we do not think one is warranted. But we will not present incomplete data as though it were complete, so the page will say plainly that listing coverage is limited because you asked us not to crawl, and it will show the public-record information we still have.

Slowing down is almost always better than blocking, for both of us. Email us before you block and we will very likely be able to accommodate whatever the underlying problem is.

Corrections and right of reply

If we have published something about your company that is factually wrong, email corrections@hiringrecord.com. We investigate every report and correct confirmed errors within two business days. Corrections are logged publicly.

Any company may also publish a response on its own page, at no cost, with no account required. We do not edit responses and we do not gate them behind any commercial relationship.

Data retention

We retain listing observation history indefinitely, because the historical record is the entire point of the research. We retain raw fetched content for 30 days and then keep only the extracted structured fields and content hashes.

We do not sell or share raw crawled content.

Contact

Purpose	Address
Crawler issues, rate limiting, blocking	crawler@hiringrecord.com
Factual corrections	corrections@hiringrecord.com
Right of reply	rightofreply@hiringrecord.com
Security disclosure	security@hiringrecord.com
Everything else	hello@hiringrecord.com

We aim to respond to crawler and security email within one business day.