Skip to main content

Understanding XPath and CSS Selectors

Scrapy leverages two types of selectors to extract data from websites: XPath and CSS. This tutorial will help you understand these selectors and how to use them effectively.

Understanding XPath​

XPath, or XML Path Language, is a query language for selecting nodes from an XML document. HTML can be handled as XML, so we can use XPath to navigate through elements and attributes in HTML.

Basic XPath Syntax​

  • Nodes: In XPath, nodes are majorly elements, attributes, and text from an XML document. For instance, in <title>My Title</title>, title is a node.

  • Absolute Path: The absolute path starts from the root node and ends at the desired node. For example, /html/head/title.

  • Relative Path: The relative path starts from the node we're currently at. For example, if we're at the head node, the relative path to title would be title.

  • Predicates: Predicates are used to find a specific node or a node that contains a specific value. Predicates are always embedded in square brackets. For example, //a[@href='https://www.example.com'] selects all the a tags whose href attribute equals https://www.example.com.

Understanding CSS​

Cascading Style Sheets (CSS) is a style sheet language used for describing the look and formatting of a document written in HTML. Scrapy uses CSS to select data from HTML documents, similar to XPath.

Basic CSS Selectors​

  • Elements: To select elements by their type, simply use the name of the element. For example, p selects all p tags.

  • Classes: To select elements by class, use a period followed by the class name. For example, .myclass selects all elements with myclass as a class.

  • IDs: To select elements by ID, use a hash followed by the ID name. For example, #myid selects the element with myid as an ID.

  • Attributes: To select elements based on their attribute value, use the attribute name and value in square brackets. For example, a[href="https://www.example.com"] selects all a tags with href set to https://www.example.com.

Using XPath and CSS in Scrapy​

In Scrapy, you can use the .xpath() and .css() methods on a Selector to apply the XPath or CSS expression respectively.

For example, 

response.xpath('//title/text()').get()

This will return the text inside the title tag of the HTML document.

Similarly, 

response.css('title::text').get()

This will do the same, but with CSS selectors.

Remember, get() returns a single result. If you want all the results that match, use getall().

Conclusion​

XPath and CSS selectors are powerful tools in your Scrapy toolkit. Practice using these selectors to become proficient in web scraping with Scrapy. Don't be afraid to experiment and happy scraping!