Category: Python

Beautiful Soup

Web Scraping with Beautiful Soup — CSS Class and Strings

Post author By John Au-Yeung
Post date October 9, 2020
No Comments on Web Scraping with Beautiful Soup — CSS Class and Strings

We can get data from web pages with Beautiful Soup.

It lets us parse the DOM and extract the data we want.

In this article, we’ll look at how to scrape HTML documents with Beautiful Soup.

Searching by CSS Class

We can get an element with the given CSS class with Beautiful Soup.

For example, we can write:

from bs4 import BeautifulSoup
import re

html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
print(soup.find_all("a", class_="sister"))

We get all the a tags with class sister , so we see:

[<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

printed.

Also, we can search with a regex:

from bs4 import BeautifulSoup
import re

html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
print(soup.find_all(class_=re.compile("itl")))

We get all the elements with class having the 'itl' substring, so we get:

[<p class="title"><b>The Dormouse's story</b></p>]

printed.

Also, we can set a function:

from bs4 import BeautifulSoup
import re

html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""

def has_six_characters(css_class):
    return css_class is not None and len(css_class) == 6

soup = BeautifulSoup(html_doc, 'html.parser')
print(soup.find_all(class_=has_six_characters))

We set the class_ parameter to the has_six_characters function, so we can get all the elements with classes with 6 characters.

So we see:

[<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

printed.

The class attribute can have more than one value, and we can search for them all.

For example, we can write:

from bs4 import BeautifulSoup

css_soup = BeautifulSoup('<p class="body strikeout"></p>', 'html.parser')
print(css_soup.find_all("p", class_="body strikeout"))

to search for nodes with the class set to body strikeout .

They have to be in the same order.

The `string` Argument

We can also search for string content instead of tags.

So we can write:

from bs4 import BeautifulSoup
import re
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""

soup = BeautifulSoup(html_doc, 'html.parser')
print(soup.find_all(string=["Tillie", "Elsie", "Lacie"]))

Then we see:

[u'Elsie', u'Lacie', u'Tillie']

logged.

Conclusion

We can search for elements with the CSS class and strings with Beautiful Soup.

Beautiful Soup

Web Scraping with Beautiful Soup — Siblings and Selectors

Post author By John Au-Yeung
Post date October 9, 2020
No Comments on Web Scraping with Beautiful Soup — Siblings and Selectors

We can get data from web pages with Beautiful Soup.

It lets us parse the DOM and extract the data we want.

In this article, we’ll look at how to scrape HTML documents with Beautiful Soup.

`.next_element` and `.previous_element`

We can get sibling elements with the .next_element and .previous_element properties.

For example, we can write:

from bs4 import BeautifulSoup

html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
last_a_tag = soup.find("a", id="link3")
print(last_a_tag.next_element)

We get the a element with the ID link3 .

Then we get the element next to it with the next_element property.

So we see:

Tillie

printed.

We can also get the previous element with the previous_element property:

from bs4 import BeautifulSoup

html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
last_a_tag = soup.find("a", id="link3")
print(last_a_tag.previous_element)

And we see:

and

printed.

`find_all()`

We can find all elements with the given selector with the find_all method.

For example, we can write:

from bs4 import BeautifulSoup

html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
print(soup.find_all("title"))

to get all the title elements, so we see:

[<title>The Dormouse's story</title>]

printed.

We can get more than one kind of element. For example, we can write:

from bs4 import BeautifulSoup

html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
print(soup.find_all("p", "title"))

Then we get:

[<p class="title"><b>The Dormouse's story</b></p>]

logged.

The Keyword Arguments

We can pass in other selectors.

For example, we can write:

from bs4 import BeautifulSoup

html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
print(soup.find_all(id='link2'))

and get the a element with ID link2 .

We can also pass in a regex object to select nodes:

from bs4 import BeautifulSoup
import re

html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
print(soup.find_all(href=re.compile("elsie")))

We get all the elements with href that has the substring 'elsie' .

So we get:

[<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]

printed.

We can also search for nodes with the given attributes.

To do that, we write:

from bs4 import BeautifulSoup
import re

soup = BeautifulSoup('<div data-foo="value">foo!</div>', 'html.parser')
print(soup.find_all(attrs={"data-foo": "value"}))

We get the nodes with the data-foo attribute set to value .

So we see:

[<div data-foo="value">foo!</div>]

printed.

To search for node with a given name element value, we can write:

from bs4 import BeautifulSoup

name_soup = BeautifulSoup('<input name="email"/>', 'html.parser')
print(name_soup.find_all(attrs={"name": "email"}))

Then we get:

[<input name="email"/>]

logged.

Conclusion

We can get nodes at various locations and with various attributes with Beautiful Soup.

Beautiful Soup

Web Scraping with Beautiful Soup — Parent and Sibling Elements

Post author By John Au-Yeung
Post date October 9, 2020
2 Comments on Web Scraping with Beautiful Soup — Parent and Sibling Elements

We can get data from web pages with Beautiful Soup.

It lets us parse the DOM and extract the data we want.

In this article, we’ll look at how to scrape HTML documents with Beautiful Soup.

Going Up

We can move up the tree with Beautiful Soup.

For example, we can write:

from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
title_tag = soup.title
print(title_tag)
print(title_tag.parent)

Then the first print call prints:

<title>The Dormouse's story</title>

And the 2nd print call prints:

<head><title>The Dormouse's story</title></head>

So we see the head tag with the parent property.

`.parents`

We can iterate over all element’s parents with the .parents property.

For example, we can write:

from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
link = soup.a
print(link)
for parent in link.parents:
    print(parent.name)

We get the first a element with soup.a .

So the first print call is:

<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>

The 2nd print calls prints all the parents of the a element, so we get:

p
body
html
[document]

Going Sideways

We can get siblings elements with Beautiful Soup.

For example, we can write:

from bs4 import BeautifulSoup

sibling_soup = BeautifulSoup(
    "<a><b>text1</b><c>text2</c></b></a>", 'html.parser')
print(sibling_soup.prettify())

Then we get:

<a>
 <b>
  text1
 </b>
 <c>
  text2
 </c>
</a>

printed.

`.next_sibling` and `.previous_sibling`

We can get the next sibling with the .next_sibling property and the previous sibling with the .previous_sibling property.

For example, we can write:

from bs4 import BeautifulSoup

sibling_soup = BeautifulSoup(
    "<a><b>text1</b><c>text2</c></b></a>", 'html.parser')
print(sibling_soup.b.next_sibling)
print(sibling_soup.c.previous_sibling)

We see:

<c>text2</c>
<b>text1</b>

printed from the print calls.

The strings in the tags aren’t siblings since they don’t have the same parent.

`.next_siblings` and `.previous_siblings`

We can get multiple siblings with the .next_siblings and .previous_siblings properties.

For example, if we have:

from bs4 import BeautifulSoup

html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
for sibling in soup.a.next_siblings:
    print(repr(sibling))

Then we see all the siblings next to the first a element:

u',n'
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
u' andn'
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
u';nand they lived at the bottom of a well.'

printed.

We can do the same with the previous_siblings property:

from bs4 import BeautifulSoup

html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
for sibling in soup.find(id="link3").previous_siblings:
    print(repr(sibling))

And we see:

u' andn'
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
u',n'
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>
u'Once upon a time there were three little sisters; and their names weren'

printed.

Conclusion

We can get parent and sibling nodes with Beautiful Soup.

Beautiful Soup

Web Scraping with Beautiful Soup — Strings

Post author By John Au-Yeung
Post date October 9, 2020
No Comments on Web Scraping with Beautiful Soup — Strings

brown rodent on brown rock during daytime

We can get data from web pages with Beautiful Soup.

It lets us parse the DOM and extract the data we want.

In this article, we’ll look at how to scrape HTML documents with Beautiful Soup.

`.string`

We can get the text content from the element if there’s only one child and the child is a NavigableString .

For example, we can write:

from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
title_tag = soup.head.title
print(title_tag.string)

to get the title from the head tag.

Then we get:

The Dormouse's story

printed.

If the only child has another tag and that tag has a string, then the parent yag is considered to have the same string as the child.

So if we write:

from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
head_tag = soup.head
print(head_tag.string)

We also get:

The Dormouse's story

printed.

And if we replace tyhe last line with:

print(head_tag.contents)

We see:

[<title>The Dormouse's story</title>]

printed.

If there’s more than one thing inside a tag, we can use the .strings generator to look at all the contents:

from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
for string in soup.strings:
    print(repr(string))

Then we see:

u"The Dormouse's story"
u'n'
u'n'
u"The Dormouse's story"
u'n'
u'Once upon a time there were three little sisters; and their names weren'
u'Elsie'
u',n'
u'Lacie'
u' andn'
u'Tillie'
u';nand they lived at the bottom of a well.'
u'n'
u'...'
u'n'

printed.

To get the strings without the extra whitespace, we can use the stripped_strings generator.

For example, we can write:

from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
for string in soup.stripped_strings:
    print(repr(string))

And we get:

u"The Dormouse's story"
u"The Dormouse's story"
u'Once upon a time there were three little sisters; and their names were'
u'Elsie'
u','
u'Lacie'
u'and'
u'Tillie'
u';nand they lived at the bottom of a well.'
u'...'

printed.

Conclusion

We can get strings at various locations with Beautiful Soup.

Beautiful Soup

Web Scraping with Beautiful Soup — Child Nodes

Post author By John Au-Yeung
Post date October 9, 2020
No Comments on Web Scraping with Beautiful Soup — Child Nodes

We can get data from web pages with Beautiful Soup.

It lets us parse the DOM and extract the data we want.

In this article, we’ll look at how to scrape HTML documents with Beautiful Soup.

CData

We can get the CData from a document with Beautiful Soup.

For example, wen can write:

from bs4 import BeautifulSoup, CData
markup = "<b><!--Hey, buddy. Want to buy a used parser?--></b>"
soup = BeautifulSoup(markup, 'html.parser')
comment = soup.b.string
cdata = CData("A CDATA block")
comment.replace_with(cdata)

print(soup.b.prettify())

We replaced the comment inside the b tag with the CData block, so the print function will print:

<b>
 <![CDATA[A CDATA block]]>
</b>

Going Down

We can get tags with other tags.

For example, we can write:

from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
print(soup.head)
print(soup.title)

The first print call gets the head element’s content.

And the 2nd print call gets the title element’s content.

So we get:

<head><title>The Dormouse's story</title></head>

and:

<title>The Dormouse's story</title>

respectively.

We can also get the b element by writing:

print(soup.body.b)

to get the first b element in body .

So we get:

<b>The Dormouse's story</b>

printed.

And:

print(soup.a)

to get the first a element.

So we tet:

<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>

printed.

We can use the find_all method to find all elements with the given selector.

For example, we can write:

print(soup.find_all('a'))

And we get:

[<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

printed.

`.contents` and `.children`

We can get the contents of tags with the contents property.

For exam[ple, we can write:

from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
head_tag = soup.head
print(head_tag.contents)

And we see:

[<title>The Dormouse's story</title>]

printed.

We can get the content of the title tag by writing:

from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
head_tag = soup.head
title_tag = head_tag.contents[0]
print(title_tag.contents)

We get the head element with soup.head .

And we get the content of it with head_tag.contents[0] .

And we get the title tag’s content with title_tag.contents .

So we see:

[u"The Dormouse's story"]

printed.

We can also loop through the title_tag ‘s content with a for loop:

from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
head_tag = soup.head
title_tag = head_tag.contents[0]
for child in title_tag.children:
    print(child)

Then we see ‘The Dormouse’s story’ logged.

`.descendants`

We can get the descendants of an elemnt with the descendants property.

For example, we can write:

from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
head_tag = soup.head
for child in head_tag.descendants:
    print(child)

Then we see:

<title>The Dormouse's story</title>
The Dormouse's story

logged.

We get the title element and the content of it, so it goes through the tree.

Conclusion

Beautiful Soup can work with CData and child nodes.

Searching by CSS Class

The string Argument

Conclusion

.next_element and .previous_element

find_all()

The Keyword Arguments

Conclusion

Going Up

.parents

Going Sideways

.next_sibling and .previous_sibling

.next_siblings and .previous_siblings

Conclusion

.string

Conclusion

CData

Going Down

.contents and .children

.descendants

Conclusion

The `string` Argument

`.next_element` and `.previous_element`

`find_all()`

`.parents`

`.next_sibling` and `.previous_sibling`

`.next_siblings` and `.previous_siblings`

`.string`

`.contents` and `.children`

`.descendants`