HireVue Candidate Scraper

Paste supported HireVue interview links into a small web app and copy the resulting candidate table into Excel. The production deployment runs in Docker behind Pangolin authentication and does not store result rows.

Production deployment

1. Export the HireVue session on your Mac

The browser authentication file is a secret: it grants access to the same HireVue session and must never be committed or shared casually.

If the existing .browser_profile/ session is current:

source .venv/bin/activate
python scrape.py export-auth 'https://providence.hirevue-app.com/interview/review?...'

If it has expired, log in again using a review URL that normally takes you to Providence HireVue:

python scrape.py login 'https://providence.hirevue-app.com/interview/review?...'

Keep the opened browser window open. After confirming that a candidate page loads, return to the terminal and press Enter. Either command writes the portable session to auth.json.

2. Prepare the VPS

On the VPS, clone this repository and create the private data directory:

git clone https://forge.penguinarcade.com/alexm/Interview-Scrape.git
cd Interview-Scrape
mkdir -p data
chmod 700 data

From your Mac, upload the session file, then protect it on the VPS:

scp auth.json your-vps:/path/to/Interview-Scrape/data/auth.json
ssh your-vps 'chmod 600 /path/to/Interview-Scrape/data/auth.json'

Alternatively, while connected to the VPS, verify the result with:

chmod 600 data/auth.json

Run docker network ls on the VPS and identify the network used by Pangolin/Newt. The default in compose.yaml is pangolin_default; override it in an uncommitted .env file when necessary:

PANGOLIN_NETWORK=your_pangolin_network

Build and start the app:

docker compose up -d --build
docker compose ps

The container intentionally publishes no host port. In Pangolin, create an authenticated public HTTP resource with:

• Method: http
• Hostname: interview-scrape
• Port: 8000
• Authentication: Pangolin SSO with only the intended users or role

The app and Newt/Pangolin must share the Docker network selected by PANGOLIN_NETWORK.

3. Operate and update

Check status and update, saving the current commit for a possible rollback:

docker inspect --format '{{.State.Health.Status}}' interview-scrape
git rev-parse HEAD
git pull --ff-only
docker compose up -d --build

Follow logs separately with docker compose logs -f --tail=100.

To roll back, use the commit printed before the update:

git switch --detach PREVIOUS_COMMIT
docker compose up -d --build

After resolving the issue, return to the maintained branch with git switch main.

When HireVue authentication expires, repeat the local login/export. Stop the app before replacing its live session, upload to a temporary name, and atomically install it:

ssh your-vps 'cd /path/to/Interview-Scrape && docker compose stop'
scp auth.json your-vps:/path/to/Interview-Scrape/data/auth.json.new
ssh your-vps 'chmod 600 /path/to/Interview-Scrape/data/auth.json.new && mv /path/to/Interview-Scrape/data/auth.json.new /path/to/Interview-Scrape/data/auth.json && cd /path/to/Interview-Scrape && docker compose start'

The service accepts at most 25 unique URLs per batch. One scrape runs at a time; a simultaneous request receives a retry message. This keeps one authenticated browser session reliable without adding a database or job queue.

Local development

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
playwright install chromium
python scrape.py login 'https://providence.hirevue-app.com/interview/review?...'
python scrape.py serve

Open http://127.0.0.1:8000, paste any block containing supported HireVue links, click Scrape links, then click Copy for Excel. Repeated Markdown label/destination links are de-duplicated automatically. Copying excludes the table headers.

Run the checks with:

python -m unittest -v
python -m py_compile scrape.py
docker compose config --quiet

Command line

Put one HireVue interview URL per line in urls.txt, then run:

python scrape.py run

Output is appended to candidates.csv; reruns skip URLs already present. These local files, the browser profile, and authentication state are excluded from Git and the Docker image.

Output columns

URL, Application Status, Interview Time (start), Interview End, ID Candidate, Phone Number, Email, Cohort Month, Cohort Year, Applying For, Unit Applying for, First Name, Last Name

Application status is Canceled, Completed, or Scheduled - Missing Resume. Applying For is Residency or Fellowship; Residency always uses unit Multiple. Phone numbers contain digits only, and interview start/end values are date-times.