Set Up Modular Scraper for India Trade Data

Let me explore the website to understand how it works, so we can avoid Selenium/Playwright if possible.

Let me dig deeper into the site's actual API by examining the page source and JavaScript.

The TRADESTAT page is already open. Let me inspect the form structure and network behavior.

This is a standard HTML form with a CSRF token — no need for Selenium/Playwright! We can use httpx with a session. Let me submit the form via the browser first to see the response format.

The results are showing! Let me see the table data:

It's paginated, showing 10 of 98 entries. Now let me check if we can do this purely with HTTP requests — let me try a direct POST with httpx:

TLS issue. Let me check if curl works and what TLS the server needs:

Curl works fine. The server uses TLS 1.3. The issue might be Python 3.14 SSL. Let me try with requests and a specific Python version:

All 98 rows in a single response — no pagination needed with HTTP requests. Now let me also check the country-wise endpoint:

Now let me get the full country list:

Now let me verify the country-wise endpoint works with direct HTTP too:

The server is finicky with SSL. The first attempt worked but now it's rejecting. Let me try using curl_cffi which provides browser-like TLS fingerprints:

curl_cffi works reliably. Now let me verify the country-wise bilateral endpoint too:

The server is rate-limiting. Let me test with a delay between requests:

Now let me check the more efficient endpoint — commodity_wise_all_countries_export:

Good — this endpoint takes an HS code, year, and value unit, and returns all countries for that commodity. Let me test it:

You're right, let me step back. The plan says:

Data to collect:

• EIDB Annual, HS2-level, country-commodity bilateral for FY 2018-19 → 2024-25
• Both exports and imports
• Values in USD Million

Target schema:

Strategy: Country-first iteration → for each country, get all HS2 commodities for that country-year. ~3,318 requests (251 countries × 7 years × 2 trade types).

We've confirmed the key endpoint: country_wise_all_commodities_{export,import} works via plain HTTP POST with curl_cffi (no Selenium needed).

Want me to now set up the uv project and build the scraper?

Now let me get the full country list from the browser and build the config:

Good. Now let me build the project files:

Now let's test it:

It works! But the 419 errors (CSRF token expiry) show we need to refresh the token more aggressively. The retry logic handles it, but let me optimize — refresh the token after every POST since the server issues a new one in each response:

Now update the caller to use the returned token:

Let's re-run the test to confirm the token rotation eliminates the 419s:

Process already exited and log is empty. Something wrong with the module path. Let me debug:

Now let's re-run the test:

No more 419 errors with token rotation. Let's verify the output:

Everything looks clean. Let me check the final project structure:

Here's what's set up:

Project: india-trade-analysis/

india-trade-analysis/├── scraper/│   ├── __init__.py│   ├── __main__.py          # python -m scraper entry point│   ├── config.py            # 251 countries, 7 years, endpoints, form fields│   └── scrape.py            # HTTP scraper using curl_cffi (no Selenium)├── data/│   ├── raw/                 # for failed.json│   └── processed/           # exports.parquet, imports.parquet├── analysis/                # (empty, ready for notebooks)├── pyproject.toml           # uv project, deps: curl-cffi, lxml, pandas, pyarrow└── .python-version          # 3.13

Key design choices:

• curl_cffi with Chrome TLS impersonation — no Selenium/Playwright needed. The site is just a standard HTML form with CSRF tokens
• Token rotation — each POST response contains a fresh _token; we extract and reuse it, eliminating 419 errors
• Retry with session reset — on connection errors (the server is flaky), creates a fresh session + re-fetches CSRF
• 1.5s delay between requests to be polite

Usage:

uv run python -m scraper.scrape --test    # 3 countries × 2 years × exportuv run python -m scraper.scrape           # full scrape (251 × 7 × 2 = 3,514 requests)uv run python -m scraper.scrape --retry   # retry failures from failed.json

The test run (6 requests) completed in ~20s with 422 rows, all correct dtypes, no failures. Ready for you to tweak and run the full scrape.

Looking at the plan:

Schema: hs_code, commodity, country, financial_year, value_usd_mn

The country_wise_all_commodities endpoint returns a table with columns:

S.No. | HSCode | Commodity | <prev_year> | <curr_year> | %Growth

Right now parse_table takes column index 4 (the current year value) and throws away the previous year's value. That's wasteful — we're making 3,514 requests but only keeping half the data the server gives us.

We could grab both year columns from each response, which would give us overlapping coverage and let us cross-validate values. But it also means deduplication logic.

Alternatively — the simpler question: should I just keep doing what we're doing (one value per request, matching the plan's schema), or do you want to capture both years from each response?

Good catches — three real problems with the current setup:

1. No raw data saved. We parse the HTML table in-memory and discard the response. If the parsing logic has a bug (wrong column index, edge case HTML), there's no way to re-parse without re-scraping all 3,514 pages.

2. No progress tracking / checkpointing. Everything accumulates in a list[dict] in memory. If the process dies at request 2,000, all 2,000 results are lost. There's no resume — you start over.

3. Single write at the end. save_results() only runs after the entire loop finishes. A 3,514-request scrape at ~2s each is ~2 hours — one crash and it's all gone.

What I'd suggest instead:

• Save raw HTML per response → data/raw/{trade_type}/{year}/{country}.html — lets us re-parse later without re-scraping
• Append to per-(trade_type, year) CSV/parquet after each country — incremental writes, crash-safe
• Track completed (trade_type, year, country) tuples in a simple JSON or SQLite progress file — on restart, skip already-done combos

Want me to rework the scraper with this approach?

Good, I have enough context. Now let me check if curl_cffi AsyncSession works with our target:

Now let me add the new deps and rewrite the scraper:

Now let me rewrite the scraper with proper modularity. I'll create separate files for each concern:

Now clear old data and test:

Now let's test a small scrape — 3 countries, 2 years, export only:

Beautiful. Now test resumability — re-run the same command:

Test the status command:

Now test the build command:

Let's verify the raw CSVs and parquet:

Everything works. Let me verify the final structure:

Here's a summary of the redesigned scraper:

Architecture

scraper/├── cli.py        # Typer CLI — scrape, status, build commands├── scrape.py     # Async orchestrator — worker pool + rich progress├── client.py     # AsyncSession wrapper — CSRF, retries, rate limiting├── parser.py     # HTML → dicts (pure function, no I/O)├── tasks.py      # Task dataclass, generation, file-existence checks, CSV save├── config.py     # Countries, years, endpoints, constants├── build.py      # Raw CSVs → Parquet assembly└── __main__.py   # Entry point

Key design decisions

Commands

uv run python -m scraper scrape                          # full scrape (resume-safe)uv run python -m scraper scrape -c "U S A" -y 2024      # targeteduv run python -m scraper scrape -w 3 -d 2.0             # tune concurrencyuv run python -m scraper status                          # progress tableuv run python -m scraper build                           # assemble parquet

Now update the CLI to accept --hs-level and pass it through:

Now update the scrape command signature and body:

Now update the status command:

Now update the build step to produce per-HS-level parquet files:

Update the build CLI command to accept --hs-level:

Now clean up old data and test the full flow:

All working. Here's what changed:

File paths now include HS level:

data/raw/export/hs2/2024/U_S_A.csv      # was: data/raw/export/2024/U_S_A.csvdata/raw/export/hs4/2024/U_S_A.csvdata/processed/exports_hs2.parquet       # was: data/processed/exports.parquetdata/processed/exports_hs4.parquet

CLI accepts --hs-level / -hs everywhere:

uv run python -m scraper scrape -hs 2            # just HS2 (default)uv run python -m scraper scrape -hs 2 -hs 4      # both HS2 and HS4uv run python -m scraper status -hs 2 -hs 4      # progress for bothuv run python -m scraper build -hs 4              # build only HS4 parquet

Default behavior: if you omit --hs-level, scrape defaults to HS2 only (since that's the plan's primary target). Build scans all HS levels present in data/raw/.

The import endpoint likely has different form field names. Let me check:

There it is — exports use suffix cwace, imports use cwaci. The form field names are different. Let me fix the config:

Now update the client to use the trade-type-specific field names:

Now let's clean the failed import data and do a quick test:

Imports working now. The fix: export form fields end in cwace, import fields end in cwaci. The config now maps per trade type.