Every client who walks through our door wants the same answer. They want to know exactly how rivals grab the rankings, the clicks and the affiliate commissions they keep losing out on.
At AFFiNCO, we answer with Apify Actors, not guesswork. We pull live SERP data, scrape competitor pages, track paid ads and feed the lot into AI for fast read-outs.
No bloated spy-tool subscriptions, no stale exports. Just real-time competitor intelligence at scale, built around the affiliate niches we work in daily.
Below, we walk through the exact Actors we use, our step-by-step process, and how we turn raw scrapes into client-ready insight. Grab a coffee. Let's get into it.
How AFFiNCO Setup Competitor Research Before a Single Keyword Gets Touched
Most affiliate teams open a spy tool, screenshot a few rankings, and call it research. We don't work like that.
Before we pitch any client a content plan, we build a picture of who actually owns their market. Not the brands they think are winning, but the domains Google and AI engines reward right now.
Apify gives us that picture without us babysitting proxies or fixing broken scripts. Each Actor handles its own rotating proxies, CAPTCHA solving and retries.
We point it at a target, set the inputs, and pull clean structured JSON data back.
Here's what we like about working this way:
That last point matters in our world. We work across proxies, antidetect browsers, eSIMs, AI tools and coupon niches, and margins differ wildly between them. Apify scales down to a five-page crawl and up to a full-market sweep.
The Exact Apify Actors We Run on Every Client Project
Here is the core kit. We mix and match per client, but most projects use four or five of these Actors.
| Apify Actor | What we pull | Best AFFiNCO use |
|---|---|---|
| Google Search Results Scraper | Organic ranks, paid ads, People Also Ask, AI Overview text and cited domains | Rank tracking on SaaS and AI tool keywords |
| Website Content Crawler | Full page copy as clean markdown | Competitor content audits across review and coupon pages |
| Facebook Ads Scraper (Meta Ad Library) | Ad creatives, copy, CTAs, run dates, spend ranges | Offer spying on eSIM and VPN brands |
| G2 Scraper | Software reviews, ratings, vendor and category data | Sentiment checks on SaaS tools we promote |
| Reddit Scraper | Posts, comments, community chatter | Organic buzz versus paid messaging gaps |
| Google Ads Transparency scraper | Search, Display, Shopping and YouTube ad creatives | Catch affiliates bidding on a client's brand |
| SimilarWeb Scraper | Traffic estimates, top countries, similar sites | Sizing rival reach before we pitch a client |
Each Actor returns the same shape of output. So our team learns one pattern and applies it everywhere. New niche, same muscle memory.
Mapping SERP Ownership with The Google Search Results Scraper
This Actor is where every project starts. The official apify/google-search-scraper reads Google's results pages and hands back a tidy dataset.
We feed it a list of the client's target keywords. Think “best residential proxies“, “eSIM for Europe”, or “Semrush coupon”. One query per line.
What comes back is gold for affiliates. The Actor extracts organic results with title, URL, snippet and exact organic search rankings position. It also captures the SERP blocks that decide visibility in 2026.
Here's what we map for every keyword set:
That AI Overview data changed how we plan content. Google's core updates pushed AI answers to the top, and a page one ranking means little if the AI box ignores you. The Actor tells us whether a client, or a rival, gets named inside that answer.
We select the client's country and language too. Rankings shift heavily by region, and our eSIM and proxy clients sell worldwide. A query localised to India looks nothing like the same query in Germany.
SERP Signals we Extract and How We Use Them
| SERP signal | What we learn | Client action |
|---|---|---|
| Top 10 organic URLs | Who to beat on each keyword | Build better, more useful pages |
| People Also Ask | Real questions buyers ask | Add FAQ blocks and H2s that match |
| Featured snippet owner | The page Google trusts most | Reverse-engineer the format and answer cleaner |
| AI Overview citations | Brands AI engines name | Push generative engine optimisation work |
| Paid ads shown | Rivals spending on the term | Decide organic vs paid priority |
A quick word on scale. To pull around 100 results per keyword, we raise the pages-per-query setting so the Actor walks through ten result pages. For a coupon client tracking 300 deal keywords, that sweep finishes while we make tea.
Reading Competitor Pages Without Copy-pasting All Day
Knowing who ranks is half the job. Next we read what those pages actually say, and the Website Content Crawler (apify/website-content-crawler) does the heavy lifting.
We drop in the rival URLs our SERP scrape surfaced. The crawler then performs a deep crawl, follows same-domain links, and pulls every page as clean Markdown output.
Why Markdown? Because it strips the junk. The Actor removes navigation, headers, footers, ads and cookie banners, then keeps only the main content. That clean text uses fewer tokens when we hand it to AI, and reads far better in a report.
Here's what we get out of a single crawl:
We control the crawl tightly with URL glob patterns. On a SaaS review client, we include **/review/** and **/comparison/** and exclude the blog noise. The crawler walks breadth-first, so we cap depth and page count to keep runs cheap.
The output plugs straight into our LLM tooling. The Actor was built to feed AI models, vector databases and RAG pipelines, and it works neatly with LangChain and LlamaIndex. For us, that means a competitor's entire content library becomes searchable in minutes.
One thing we stay sharp on. Scraped competitor copy informs our strategy, but we never republish it. We read the structure, spot competitor content gaps, and write something genuinely better. Google rewards that, and our clients keep their trust signals clean.
Spying on The Paid Traffic Affiliates Push
Organic is only one battlefield. In affiliate, paid ads often tell you where the real money flows, and two Actors expose that.
The Facebook Ads Scraper reads Meta's Ad Library across Facebook, Instagram, Messenger, Threads and Audience Network. We paste a rival's page URL or an Ad Library URL and pull every live ad.
What we extract per advertiser:
A long-running ad is a winning ad. Nobody pays to keep a loser live. So the competitor ad creatives that have run for months become our swipe file for offers worth promoting.
The Google Ads Transparency Center scraper does the same job on Google's side. We pull every ad an advertiser runs across Search, Display, Shopping and YouTube, with creatives, run dates and target regions.
This one earns its keep on brand bidding detection. If an affiliate or rogue partner bids on a client's brand name, the Transparency Center shows it. We hand the client a list of who's poaching their branded traffic, complete with first-shown dates.
Both Actors export to the same clean formats, and both can run on a schedule. We set weekly runs so a client sees new rival campaigns land in their swipe file automatically.
Tracking Offers, Coupons and Deals Across Rival Sites
Our coupon and deals properties live or die on fresh offers. So we run a focused layer of e-commerce and store scrapers on top of the core stack.
For Shopify and WooCommerce rivals, store-monitoring Actors pull catalogues, prices, variants and collection changes with no API key. We watch for:
We pair that with the Website Content Crawler aimed at rival coupon pages. Together they tell us when a competitor adds a deal, so our pages stay current and our affiliate offer tracking never lags behind theirs.
Here's the actual sequence we follow on a new project. Same order, every time.
- Step 1. Define the keyword universe: We list the client's money keywords and group them by intent. Review terms, comparison terms, coupon terms and “best of” terms each get their own bucket.
- Step 2. Run the Google Search Results Scraper: We load every keyword, set the country and language, and raise the pages-per-query for deeper coverage. Then we hit start and let it map the full SERP.
- Step 3. Pull the competitor URL list: From the SERP dataset, we export the top-ranking domains per keyword. Duplicates get merged, so we end up with a clean shortlist of who actually wins the market.
- Step 4. Crawl those competitors: We feed the shortlist into the Website Content Crawler, set include and exclude URL patterns, and pull every relevant page as Markdown.
- Step 5. Scrape the paid side: We run the Facebook Ads Scraper and the Google Ads Transparency scraper on the same competitor set. Now we have their organic and paid playbook side by side.
- Step 6. Layer in offers: For coupon and e-commerce clients, we add the store and deals scrapers so live promotions land in the dataset too.
- Step 7. Pipe everything into AI: We send the combined dataset to Claude or ChatGPT through the Apify MCP server and ask pointed questions. More on that next.
- Step 8. Schedule the repeat runs: Inside the Apify Console, we set the SERP and ad Actors to run weekly. A one-off audit becomes a living feed.
- Step 9. Ship the client report: We pull the AI summary and the raw datasets into a clean deck, flag the quick wins, and hand it over with a prioritised action list.
That whole sequence takes us a day for a mid-size client, where it used to take a week of manual work.
Piping Apify data into Claude and ChatGPT with MCP
This is the step that turned our research from slow to genuinely fast.
Apify ships an official Apify MCP server. So we connect Claude Desktop or Cursor straight to our Actors and datasets, with no custom backend. The AI reads the scraped data live and answers in plain language.
A few prompts we run on every dataset:
The AI handles the volume a human can't. We've fed it hundreds of scraped pages and asked it to surface competitor content gaps in minutes. The output becomes our content brief.
This is also where answer engine optimization comes in. Once we know which queries trigger AI citations and which rivals get named, we shape client pages to earn those mentions. Ranking on page one no longer cuts it alone. We optimise for the AI box too.
Turning Raw Scrapes Into Client-ready Insight
A dataset full of JSON helps nobody. The value sits in what we do next.
We keep our reporting tight and honest, the same way we'd want a partner to report to us. Every client gets three things:
We also flag the share of voice in AI answers, since that's the metric most affiliate teams still ignore. If a client gets named in zero AI Overviews while a rival appears in nine, that's the first fire we put out.
Scheduling closes the loop. Weekly Apify runs feed a Google Sheet through Apify's integrations, and any big rival move pings our team. Clients stop reacting late and start moving first.
Final Words from The AFFiNCO Team
Search has shifted more in two years than in the previous ten. Rankings, AI answers and paid offers now move together, and affiliate teams that watch only one of them fall behind fast.
At AFFiNCO, Apify Actors let us watch all three at once, at a scale that off-the-shelf spy tools simply can't match. The Google Search Results Scraper maps the battlefield, the Website Content Crawler reads the playbook, and the ad Actors expose where rivals spend. AI ties it together.
If you take one thing from this guide, make it this. Start with the SERP, follow the data into competitor pages, then let AI do the reading. Your reports get sharper, and your clients win the keywords they keep losing.
So, which competitor would you map first if the data was sitting in front of you right now? Tell us in the comments, and we'll point you at the right Actor to start with.
We use this exact stack across our own network of affiliate sites before we ever run it for a client. Everything here is built on tools we rely on daily.
Ali
Ali is a digital marketing expert with 7+ years of experience in SEO-optimized blogging. Skilled in reviewing SaaS tools, social media marketing, and email campaigns, we craft content that ranks well and engages audiences. Known for providing genuine information, Ali is a reliable source for businesses seeking to boost their online presence effectively.
