Scraping LinkedIn with Python and Playwright (2026 Guide)

LinkedIn is notoriously difficult to scrape. It's a dynamic, JavaScript-heavy site that requires a logged-in session for most data, and it actively detects and blocks bots. This guide shows you the practical approach that actually works in 2026.

Important disclaimer: Scraping LinkedIn may violate their Terms of Service. This guide is for educational purposes. Use only on data you have legitimate access to (e.g., your own profile data, publicly available job postings, or with LinkedIn's explicit permission via their API).

Why Playwright Over BeautifulSoup for LinkedIn

LinkedIn renders almost all of its content via JavaScript — the initial HTML response is mostly an empty shell. requests + BeautifulSoup won't work because by the time you parse the response, none of the profile or job data has been injected yet.

Playwright launches a real Chromium browser that executes JavaScript, handles dynamic rendering, and lets you interact with the page like a human would.

pip install playwright
playwright install chromium

Step 1: A Basic Playwright Setup

from playwright.sync_api import sync_playwright
import time

def scrape_linkedin_job(job_url: str) -> dict:
    with sync_playwright() as p:
        browser = p.chromium.launch(
            headless=True,
            args=[
                "--no-sandbox",
                "--disable-dev-shm-usage",
                "--disable-blink-features=AutomationControlled",
            ]
        )

        context = browser.new_context(
            user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
            viewport={"width": 1280, "height": 800},
        )

        page = context.new_page()

        # Navigate to the job listing
        page.goto(job_url, wait_until="networkidle", timeout=30000)
        time.sleep(2)  # Let JS finish rendering

        # Extract job details
        job_data = {
            "title": page.locator("h1.job-title").text_content(timeout=5000),
            "company": page.locator(".company-name").text_content(timeout=5000),
            "location": page.locator(".job-criteria-text").first.text_content(timeout=5000),
        }

        browser.close()
        return job_data

Step 2: Handling LinkedIn's Login Wall

Many LinkedIn pages require authentication. Here's how to handle login with Playwright:

import os

def get_authenticated_page(playwright_instance):
    browser = playwright_instance.chromium.launch(headless=True)
    context = browser.new_context(
        user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ...",
    )
    page = context.new_page()

    # Navigate to login
    page.goto("https://www.linkedin.com/login")
    page.wait_for_selector("#username", timeout=10000)

    # Fill credentials from environment variables (never hardcode!)
    page.fill("#username", os.environ.get("LINKEDIN_EMAIL"))
    page.fill("#password", os.environ.get("LINKEDIN_PASSWORD"))
    page.click("[type='submit']")

    # Wait for dashboard to confirm login
    page.wait_for_url("**/feed/**", timeout=15000)
    print("Login successful.")

    return browser, context, page

Step 3: Scraping Job Search Results

Once authenticated, you can scrape search results. Here's a function that pulls job listings from a keyword search:

def scrape_job_search(keyword: str, location: str, max_jobs: int = 20) -> list:
    results = []

    with sync_playwright() as p:
        browser, context, page = get_authenticated_page(p)

        # Search URL
        search_url = (
            f"https://www.linkedin.com/jobs/search/"
            f"?keywords={keyword.replace(' ', '%20')}"
            f"&location={location.replace(' ', '%20')}"
            f"&sortBy=DD"  # Sort by most recent
        )
        page.goto(search_url, wait_until="networkidle")

        job_cards = page.locator(".job-card-container")
        count = min(job_cards.count(), max_jobs)

        for i in range(count):
            card = job_cards.nth(i)
            try:
                title = card.locator(".job-card-list__title").text_content(timeout=3000).strip()
                company = card.locator(".job-card-container__company-name").text_content(timeout=3000).strip()
                location = card.locator(".job-card-container__metadata-item").first.text_content(timeout=3000).strip()
                job_url = card.locator("a").first.get_attribute("href")

                results.append({
                    "title": title,
                    "company": company,
                    "location": location,
                    "url": f"https://www.linkedin.com{job_url}" if job_url else None,
                })
            except Exception as e:
                print(f"Skipped card {i}: {e}")
                continue

        browser.close()

    return results

Step 4: Anti-Detection Techniques

LinkedIn's bot detection looks for:

1. Headless browser signatures: Override the navigator.webdriver property
2. Inhuman speed: Add random delays between actions
3. Fingerprinting: Rotate viewport sizes and user agents

# Override WebDriver detection
page.add_init_script("""
    Object.defineProperty(navigator, 'webdriver', {
        get: () => undefined
    });
""")

# Human-like delays
import random

def human_delay(min_ms=500, max_ms=2000):
    time.sleep(random.randint(min_ms, max_ms) / 1000)

# Use between every action
page.fill("#username", email)
human_delay()
page.fill("#password", password)
human_delay(300, 800)
page.click("[type='submit']")

Step 5: The Full Handler for Scheduled Execution

Here is the complete, deployable handler that runs on a schedule:

import os
import json
import requests
from playwright.sync_api import sync_playwright
import time
import random

def human_delay(min_ms=800, max_ms=2500):
    time.sleep(random.randint(min_ms, max_ms) / 1000)

def send_to_webhook(data: list):
    """Push scraped jobs to a webhook (Notion, Airtable, Slack, etc.)"""
    webhook_url = os.environ.get("WEBHOOK_URL")
    if not webhook_url:
        return
    requests.post(webhook_url, json={"jobs": data}, timeout=10)

def handler(event, context):
    """
    Runs every day at 9:00 AM UTC.
    Schedule: 0 9 * * *

    Required env vars:
    - LINKEDIN_EMAIL
    - LINKEDIN_PASSWORD  
    - WEBHOOK_URL (optional, for pushing results)
    - SEARCH_KEYWORD (e.g., "Python Developer")
    - SEARCH_LOCATION (e.g., "Remote")
    """
    keyword = os.environ.get("SEARCH_KEYWORD", "Python Developer")
    location = os.environ.get("SEARCH_LOCATION", "Remote")

    print(f"Searching LinkedIn for: {keyword} in {location}")

    with sync_playwright() as p:
        browser = p.chromium.launch(
            headless=True,
            args=["--no-sandbox", "--disable-dev-shm-usage"]
        )
        context = browser.new_context(
            user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
            viewport={"width": 1280, "height": 800},
        )
        page = context.new_page()

        # Add anti-detection
        page.add_init_script("Object.defineProperty(navigator,'webdriver',{get:()=>undefined})")

        # Login
        page.goto("https://www.linkedin.com/login")
        page.fill("#username", os.environ["LINKEDIN_EMAIL"])
        human_delay()
        page.fill("#password", os.environ["LINKEDIN_PASSWORD"])
        human_delay(300, 800)
        page.click("[type='submit']")
        page.wait_for_url("**/feed/**", timeout=20000)
        print("Logged in successfully.")

        # Search
        search_url = f"https://www.linkedin.com/jobs/search/?keywords={keyword}&location={location}&sortBy=DD"
        page.goto(search_url, wait_until="networkidle")
        human_delay(1000, 2000)

        # Extract
        jobs = []
        cards = page.locator(".job-card-container")
        for i in range(min(cards.count(), 15)):
            try:
                card = cards.nth(i)
                title = card.locator(".job-card-list__title").text_content(timeout=3000).strip()
                company = card.locator(".job-card-container__company-name").text_content(timeout=3000).strip()
                jobs.append({"title": title, "company": company})
                human_delay(200, 600)
            except:
                continue

        browser.close()

    print(f"Found {len(jobs)} jobs.")

    if jobs:
        send_to_webhook(jobs)

    return {"status": "success", "jobs_found": len(jobs), "keyword": keyword}

Step 6: pip Packages Required

In LiteLambda's Pip Packages field, add:

playwright==1.44.0
requests==2.32.3

LiteLambda's sandbox environment supports Playwright's Chromium runtime out of the box — no extra install steps.

Common Errors and Fixes

Scheduling This Scraper

Set your cron schedule to run once per day: 0 9 * * *

Running it more than once per day significantly increases your chances of being temporarily blocked. LinkedIn's bot detection is traffic-pattern based — consistent daily scraping at reasonable volumes is far safer than running every 15 minutes.

Run this Playwright scraper for free on LiteLambda →