Process Pending Links

Fetch product details from URLs and store extracted data for downstream content workflows.

Step 2
Before You Start Processing
  1. Verify WED2C Store Theme: Log in to your WED2C dashboard and navigate to Online Store >> Theme >> Website Theme.
  2. Select Default Theme: Ensure the Default Theme is active. This guarantees clean product structure parsing and prevents extraction errors during HTML scraping.
Scraper Engine & Data Extraction Pipeline

When you initiate a processing batch, the background engine executes a structured 3-tier fetch and extraction routine for each pending URL:

  • Multi-Tiered Fetching Fallback: 1. Playwright (Headless Chromium): Renders single-page applications (SPAs) and waits for dynamic elements like titles, prices, and variant boxes to settle. 2. Cloudscraper: Acts as a secondary backup to bypass Cloudflare anti-bot challenges if browser automation is blocked. 3. Requests: Standard HTTP client fallback for static response handling.
  • Structured Parsing (BeautifulSoup4): Extracted HTML is cleaned of scripts and styles. Key data nodes parsed include:
    Title .good-title.good-title-default
    Price .currency-money.notranslate
    Description .commodity-desc-box
    Media Filtered <img> tags (ignoring tracking assets)
  • Database Sync: Saves raw page text, parsed image arrays, price points, and metadata into LinkData records, then flags the LinkSource as processed=True.
Batch Execution Progress 100% Completed
Total Queued
511
Processed
511
Remaining Pending
0