How to Batch Scan 1,000 Websites for Tech Stack Data at Scale

Scanning one website for its tech stack takes seconds. Scanning a thousand requires a different approach. Whether you are enriching a sales pipeline, auditing a client portfolio, or mapping technology adoption across an industry, you need a workflow that handles volume without drowning in manual effort or burning through API limits.

The DetectZeStack API has a dedicated /analyze/batch endpoint built for exactly this. It accepts up to 10 URLs per request, scans them concurrently, and returns structured technology data for every domain in one response. This guide walks through how to use it to scan 1,000 websites end to end—with a working Python script, curl examples, and tips for optimizing speed and cost.

Why Batch Scanning Beats One-at-a-Time Lookups

The single /analyze endpoint works well for one-off lookups. But when you have hundreds or thousands of domains to process, making individual requests creates unnecessary overhead:

• HTTP overhead — Each request requires a full TCP handshake, TLS negotiation, and header exchange. Batching amortizes this across 10 URLs.
• Rate limit efficiency — A single batch request counts as one API call per URL, but the server processes all 10 concurrently in about the same wall-clock time as a single scan.
• Simpler error handling — The batch response groups successes and failures together, so your script does not need to track retries per domain.
• Lower latency per domain — The server runs up to 5 fetches in parallel within each batch, so 10 domains complete faster than 10 sequential calls.

For a detailed walkthrough of single-domain analysis, see Detect Any Website’s Tech Stack With a Single API Call. This guide builds on that foundation with the batch workflow.

How the DetectZeStack Batch Endpoint Works

Request Format and Limits

The batch endpoint is a POST request to /analyze/batch. The request body is JSON with a single field: urls, an array of up to 10 domain strings.

POST /analyze/batch
Content-Type: application/json

{
  "urls": [
    "stripe.com",
    "github.com",
    "shopify.com",
    "notion.so",
    "vercel.com"
  ]
}

Key constraints:

• Maximum 10 URLs per batch request
• Each URL in the batch counts as one request against your plan limit
• The server scans up to 5 URLs concurrently within each batch
• Request body is limited to 1 MB

Understanding the Batch Response

The response includes a results array (one entry per URL), plus summary fields:

{
  "results": [
    {
      "url": "stripe.com",
      "result": {
        "url": "https://stripe.com",
        "domain": "stripe.com",
        "technologies": [
          {
            "name": "React",
            "categories": ["JavaScript frameworks"],
            "confidence": 100
          },
          {
            "name": "Next.js",
            "categories": ["JavaScript frameworks"],
            "confidence": 100
          },
          {
            "name": "Cloudflare",
            "categories": ["CDN"],
            "confidence": 80
          },
          {
            "name": "Stripe",
            "categories": ["Payment processors"],
            "confidence": 100
          }
        ],
        "categories": {
          "JavaScript frameworks": ["React", "Next.js"],
          "CDN": ["Cloudflare"],
          "Payment processors": ["Stripe"]
        },
        "meta": {
          "status_code": 200,
          "tech_count": 4,
          "scan_depth": "full"
        },
        "cached": true,
        "response_ms": 12
      }
    },
    {
      "url": "nonexistent-domain-xyz.com",
      "error": "DNS resolution failed \u2014 domain does not exist or has no DNS records"
    }
  ],
  "total_ms": 2847,
  "successful": 4,
  "failed": 1
}

Each item in results has either a result object (on success) or an error string (on failure). The top-level successful and failed counts let you quickly check if any domains had issues without iterating through every result.

Curl Example — Your First Batch Scan

Before writing any code, try the batch endpoint from your terminal. You will need an API key from RapidAPI:

curl -X POST "https://detectzestack.p.rapidapi.com/analyze/batch" \
  -H "X-RapidAPI-Key: YOUR_KEY" \
  -H "X-RapidAPI-Host: detectzestack.p.rapidapi.com" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "stripe.com",
      "github.com",
      "shopify.com",
      "vercel.com",
      "notion.so"
    ]
  }'

You should see a JSON response with technology detections for all five domains. Each result includes the detected technologies (React, Next.js, Cloudflare, WordPress, Shopify, etc.), their categories, confidence scores, and metadata.

curl -s "https://detectzestack.com/demo?url=stripe.com" | jq .

Scanning 1,000 Websites With a Simple Script

Step 1 — Prepare Your URL List

Start with a text file containing one domain per line. You might export this from a CRM, scrape it from a directory, or compile it manually. Only the domain is needed—the API handles protocol and path normalization.

# domains.txt
stripe.com
github.com
shopify.com
notion.so
vercel.com
linear.app
figma.com
netlify.com
supabase.com
fly.io
...

Step 2 — Send Batches of 10 With Rate Limiting

Here is a Python script that reads the domain list, sends batches of 10, and collects the results. It includes a 1-second delay between batches to stay well within rate limits:

import json
import time
import requests

API_URL = "https://detectzestack.p.rapidapi.com/analyze/batch"
API_KEY = "your-rapidapi-key"
BATCH_SIZE = 10

def load_domains(path):
    with open(path) as f:
        return [line.strip() for line in f if line.strip()]

def scan_batch(urls):
    resp = requests.post(API_URL, json={"urls": urls}, headers={
        "X-RapidAPI-Key": API_KEY,
        "X-RapidAPI-Host": "detectzestack.p.rapidapi.com",
        "Content-Type": "application/json"
    })
    resp.raise_for_status()
    return resp.json()

def scan_all(domains):
    all_results = []
    total = len(domains)

    for i in range(0, total, BATCH_SIZE):
        batch = domains[i:i + BATCH_SIZE]
        batch_num = (i // BATCH_SIZE) + 1
        total_batches = (total + BATCH_SIZE - 1) // BATCH_SIZE

        print(f"Batch {batch_num}/{total_batches}: scanning {len(batch)} domains...")

        data = scan_batch(batch)
        all_results.extend(data.get("results", []))

        print(f"  OK: {data.get('successful', 0)}, "
              f"Failed: {data.get('failed', 0)}, "
              f"Time: {data.get('total_ms', 0)}ms")

        # Respect rate limits between batches
        if i + BATCH_SIZE < total:
            time.sleep(1)

    return all_results

domains = load_domains("domains.txt")
results = scan_all(domains)

# Save raw results
with open("scan_results.json", "w") as f:
    json.dump(results, f, indent=2)

print(f"\nDone. {len(results)} domains scanned. Results saved to scan_results.json")

For 1,000 domains, this script sends 100 batch requests. With the 1-second delay between batches, the entire scan completes in under 2 minutes. Most of that time is the intentional delay—the actual scanning is much faster because the server processes each batch concurrently.

Step 3 — Aggregate and Export Results to CSV

Once you have the raw JSON results, transform them into a CSV for analysis or CRM import:

import csv
import json

with open("scan_results.json") as f:
    results = json.load(f)

with open("tech_stacks.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["domain", "tech_count", "technologies", "categories"])

    for item in results:
        if "error" in item:
            writer.writerow([item["url"], 0, "", item["error"]])
            continue

        result = item["result"]
        techs = [t["name"] for t in result.get("technologies", [])]
        cats = set()
        for t in result.get("technologies", []):
            cats.update(t.get("categories", []))

        writer.writerow([
            result["domain"],
            result["meta"]["tech_count"],
            ", ".join(sorted(techs)),
            ", ".join(sorted(cats))
        ])

print(f"Exported {len(results)} rows to tech_stacks.csv")

The output CSV has one row per domain with columns for the domain name, technology count, a comma-separated list of detected technologies, and the categories they belong to.

Real-World Use Cases for Bulk Tech Detection

Sales Prospecting and Lead Enrichment

Scan your prospect list to find companies using specific technologies. Selling a WordPress security plugin? Filter for sites running WordPress. Offering a Shopify integration? Find every Shopify store in your pipeline. The technology data turns cold outreach into relevant, informed conversations.

Batch scanning is particularly powerful for lead enrichment pipelines. Export 1,000 leads from HubSpot, run them through the batch scanner, and push the enriched data back as custom fields for segmentation.

Competitive Intelligence Across an Industry

Map technology adoption across an entire market segment. Scan the top 500 e-commerce sites to see what percentage use Shopify versus WooCommerce. Compare CDN adoption between SaaS companies. Track which JavaScript frameworks dominate in fintech. This kind of aggregate analysis requires scanning at scale—one domain at a time is impractical.

Security Auditing a Portfolio of Sites

If you manage or audit multiple websites, batch scanning lets you quickly inventory the technology across all of them. Find sites running outdated jQuery versions, identify which domains lack a CDN, or flag sites missing security headers. Combine batch tech detection with the security audit endpoint for a comprehensive view.

Optimizing for Speed and Cost

Leverage Caching to Avoid Redundant Scans

DetectZeStack caches scan results for 24 hours. If a domain was scanned recently—by you or another user—the batch endpoint returns the cached result instantly. The response includes a cached: true field so you can tell.

This has two practical benefits:

• Re-scanning is fast and free — If you run the same 1,000-domain list again within 24 hours (say, to re-export with different columns), the second run completes in seconds because every result is cached.
• Popular domains are pre-cached — High-traffic domains like stripe.com, github.com, and shopify.com are frequently scanned by other users, so they often return cached results on your first request.

Choose the Right Plan for Your Volume

Each URL in a batch counts as one request. Here is how the plans map to batch scanning volume:

For scanning 1,000 websites once per month, the Pro plan at $9/month is the most cost-effective option. If you need to re-scan regularly or enrich larger lists, the Ultra plan at $29/month gives you 10,000 requests—enough to scan 1,000 domains ten times per month or 10,000 domains once.

For a deeper breakdown of technographic data pricing across providers, see Technographic Data Pricing.

Tips for Production Batch Scanning

A few practical considerations when running batch scans at scale:

• Deduplicate your list first. Remove duplicate domains before scanning. Scanning example.com and www.example.com will produce the same result and waste a request.
• Handle failures gracefully. Some domains will fail—DNS errors, timeouts, unreachable servers. The batch response separates successes from failures. Log failed domains and retry them in a later batch.
• Use the total_ms field for monitoring. Track how long each batch takes. If batch times spike, you may be hitting a rate limit or the target domains may be slow to respond.
• Store results with timestamps. Tech stacks change. If you scan the same list monthly, keep historical snapshots so you can track which companies adopted or dropped specific technologies.

For a Python-focused tutorial covering error handling and retry logic in more detail, see Detect Any Website’s Tech Stack with Python.

Get Started With Batch Scanning

Batch scanning 1,000 websites takes about 2 minutes with a simple Python script and the DetectZeStack API. The /analyze/batch endpoint handles the concurrency, caching, and error reporting—your script just needs to send batches of 10 and collect the results.

Start by testing with the free demo endpoint to verify the detection quality on domains you care about:

curl -s "https://detectzestack.com/demo?url=stripe.com" | jq '.technologies[] | {name, categories, confidence}'

When you are ready to scan at scale, grab a free API key (100 requests/month, no credit card) and start batching.