Course proposal: Web scraping basics with AI

[honzajavorek]

Learn how to use AI to extract information from websites in this practical course, starting from the absolute basics.

In this course we'll use AI assistants to create an application for watching prices. It'll be able to scrape all product pages of an e-commerce website and record prices. Data from several runs of such program would be useful for seeing trends in price changes, detecting discounts, etc. The end product will, unlike programs vibe-coded carelessly, reach the level of quality allowing for further extensibility and comfortable maintenance, so that it can be published to Apify Store.

What we'll do

• Use LLM_CHAT (TBD) to create a program which extracts data from a web page.
• Save extracted data in various formats, e.g. CSV which MS Excel or Google Sheets can open.
• Use LLM_AGENT_GUI_OR_TUI (TBD) to improve the program so that it is robust and maintainable.
• Save time and effort with Apify's scraping platform.

Who this course is for

Anyone with basic knowledge of chatting with an AI assistant and affinity to building digital products who wants to start with web scraping can take this course. The course does not expect you to have any prior knowledge of web technologies or scraping.

Requirements

• A macOS, Linux, or Windows machine with a web browser.
• Prior experience chatting with AI assistants, such as OpenAI ChatGPT, Google Gemini, or Anthropic Claude.
• Familiarity with running commands in Terminal (macOS/Linux) or Command Prompt (Windows).

Lessons

1. Chatting to code - Copying from chat, pasting to local files. Running with uv (should be easy-ish?). Creating a basic scraper which does what we need.
2. Using an agent - Explaining benefits (delegation and independent work, AGENTS.md). Getting environment ready, learning the ropes with a GUI/TUI. Using the apify CLI to start a project. Creating a basic scraper which does what we need.
3. Docs-driven prompting - Improving the README, e.g. input output. Pointing the agent to the README and turning the design to reality.
4. Test-driven prompting - Adding fixtures, expectations. Setting up tests and teaching the agent to run tests. Dealing with corner cases by pointing the agent to the fixtures.
5. Using a platform - Deploying to Apify and reaping the benefits of the platform. Running the scraper periodically, adding support for proxies...

[honzajavorek]

What tools we should use in the course

As per Ignore options and alternatives, we'll need to choose one tool which we'll use for teaching, and only mention that in others, the approach would be similar. This makes the learning path clear and the course more maintainable.

Conclusion of my analysis

• The first lesson should use ChatGPT. It's mainstream. For chatting, it's free with limits, but many people might already pay for it, and for those it's gonna be without perceivable limits.
• The second lesson should use Cursor (or Google Antigravity). Only if it truly scales to zero as they claim and it is not required to have a paid account to try an agent. Minimal friction, just install – beats any other decision factors.
• If previous bullet point turns out being a wrong direction, we should use VS Code and tell people to spend $10 to try Copilot. VS Code is mainstream. Paying for Copilot is the cheapest agent offering, and it's quite powerful.
• The implementation language can be JS or Python, it doesn't matter. However, the apify create template for Python is outdated, so I think (with a tear in my eye) that we should generate JavaScript – compared to outdated Python stack, I expect less issues to run it locally.

Full analysis

Open here

There is no future-proof choice

As we're touching a field which develops rapidly, whatever tool we choose, we cannot expect that our choice is gonna be future-proof. We can expect that outputs will soon differ, or that screenshots we'll make will need reworking in perhaps less then a month after publishing the course. We'd better fight this by being carefully vague in directing the student through the UI – we should be specific enough so that they know what to do, but vague enough so that they still know what to do even if the UI changes.

Random thought: Outputs of TUI might be less tedious to fix in the future than screenshots.

We shouldn't require several providers

We can point out to the reader that whatever we teach should be doable with any AI provider, but we should focus on one. The course goes from chatting with an assistant to using an agent. This transition should be possible using the same provider, so that if someone want to pay for AI usage, they don't have to pay for multiple services. That would be annoying friction and unnecessary requirement.

ChatGPT is probably the most mainstream choice

If we're to tap on what's in many people's phones and computers already as an AI assistant of their choice, we'd go with ChatGPT. However, it's agent offering, Codex, isn't included in the cheapest paid plan, so most people who already pay for ChatGPT won't have it available.

TUI vs GUI

• The two frontier tools for developers are OpenAI Codex and Claude Code. Both are TUI, which might pose a barrier for people who are not used to work in a terminal environment.
• Claude Code has a desktop app. Both Codex and Claude have their own VS Code extension, so this way can be turned into GUI. Both can be used on the web.
• And then there are IDEs with a selection of multiple models, such as VS Code (built-in GitHub Copilot), Cursor, Google Antigravity.

Pricing

To upgrade to an agent, the student must pay for the AI:

• Codex ~$24/mo (it requires Plus, which is a higher plan than Go, which most people might already have)
• Claude Code $20/mo
• Cursor: free with limits or $20/mo, access to many models
• GitHub Copilot: $10/mo, access to many models
• Google Antigravity: free with limits, access to many models

Ok, so it seems there are free options! That's a big factor, because for educational purposes, we'll be always better off with something that scales down to zero.

Programming language

The programming language we'll choose for implementation should be JavaScript or Python. Apify CLI and Claude Code have one-line installation bash command, so the student isn't required to fiddle with npm or anything. The implementation language could be either JS or Python. The Codex CLI tool installation is npm-only (or Homebrew). In such case, for the purposes of the course, JS would be the only sensible option.

In Python, uv allows to run a script in a super simple and accessible way, in milliseconds, without setting up anything (it even downloads and installs Python), but the apify create template is outdated and doesn't seem to use it. It doesn't even use pyproject.toml, it still sports requirements.txt. This holds us back. The apify/actor-templates#350 is still open. I'm not willing to die on this particular hill, so I guess JavaScript it is…

[honzajavorek]

Here is a rough proposal of what the lessons could be about.

---

1️⃣ Chatting to code

Creating first scraper

Prompt ChatGPT to get a simple JavaScript program which downloads https://warehouse-theme-metal.myshopify.com/collections/sales and lists the product names:

Create a scraper in JavaScript which downloads https://warehouse-theme-metal.myshopify.com/collections/sales, extracts all the products in Sales and saves a CSV file, which contains:

• Product name
• Product detail page URL
• Price

Running code

Save it as a scraper.js, setup Node/npm environment, run it, get results. If the student gets stuck setting up Node/npm, they ask ChatGPT. Roughly explaining what the program does, establishing basic terms.

Scraping stock units

Prompt ChatGPT to modify the program so that it scrapes stock units. Technically, modifying the program like this proves to be cumbersome, but doable. Run the program again, get better results.

Teaser: In the next lesson we'll get rid of copying and pasting and updating the files ourselves.

2️⃣ Using an agent

Installing development environment

Explaining benefits (delegation and independent work, AGENTS.md). Getting environment ready. Use https://docs.apify.com/platform/actors/development/quick-start/build-with-ai

Scraping vendor names

Learning the ropes with a GUI/TUI, prompting the agent to update the code so that it scrapes vendor names. Run the program again, get better results.

Teaser: Explain why this is fragile. In the next lesson we'll learn how to develop features of the scraper in a robust way by first specifying them as documentation.

3️⃣ Docs-driven prompting

Creating README.md

Create simple README.md where we document how the scraper behaves, what it produces as an output, etc. Primer to Markdown.

Dealing with price intervals

Explain focus on product and domain knowledge. In the README explain how the scraper should handle prices like "From $1,398.00", introduce minPrice, keep prices as numbers, etc.

Let the agent implement handling of variants based on the README. Run updated code, see results.

Teaser: Imagine the target website changes something (happens often!). In such case the README won't help. Let's deal with that.

4️⃣ Test-driven prompting

Creating tests

Explain tests, expectations. Create test folder, drop downloaded HTML of the listing there, let the agent set up tests around it. Let the agent run tests.

Scrape product variants

Change README so that it describes how the product variants should be handled. Drop HTML of the product detail page to the tests folder. Prompt the agent to implement scraping product variants. Run the program, get results.

Teaser: In the next lesson we'll learn how to deploy a similar scraper to a platform, schedule it to run regularly, automatically produce various formats, have a history of data, etc.

5️⃣ Using a platform

Creating an Actor

Explain what is Actor. Use Apify CLI template - already provides AGENTS.md, README.md, all the basics. Get the Apify CLI and use it to start a new project. Explain that this is how a professional scraper should be started.

Turning the Actor to a scraper

Take contents of the old README to the template and tests and let the agent implement the same scraper again according to all of that. Explain that this is the shortest and most robust way from idea to result - use template, provide docs, tests, and let the agent do its job.

Deploying

Deploying to Apify and reaping the benefits of the platform. Running the scraper periodically, getting JSONs and CSVs, seeing history, monitoring, adding support for proxies... Mention Apify Store.