Author: John Au-Yeung

Sometimes, we want to grab visible webpage text with BeautifulSoup.

In this article, we’ll look at how to grab visible webpage text with BeautifulSoup.

How to grab visible webpage text with BeautifulSoup?

To grab visible webpage text with BeautifulSoup, we can call filter when we’re grabbing the webpage content.

For instance, we write:

from bs4 import BeautifulSoup
from bs4.element import Comment
import urllib.request


def tag_visible(element):
    if element.parent.name in [
            'style', 'script', 'head', 'title', 'meta', '[document]'
    ]:
        return False
    if isinstance(element, Comment):
        return False
    return True


def text_from_html(body):
    soup = BeautifulSoup(body, 'html.parser')
    texts = soup.findAll(text=True)
    visible_texts = filter(tag_visible, texts)
    return u" ".join(t.strip() for t in visible_texts)


html = urllib.request.urlopen('https://yahoo.com').read()
print(text_from_html(html))

We have the tag_visible function that checks for tags for invisible elements by checking the element.parent.name for the tags that aren’t displayed.

We return True for the visible tags and False otherwise.

Then we define the text_from_html function to grab the text.

We use the BeautifulSoup constructor with body to get the content.

Then we call soup.findAll with text set to True to get all the nodes with text content.

And then we call filter with tag_visible and texts to get the visible nodes.

And finally, we call join to join all the results together.

We then get the HTML with urllib.request.urlopen and call text_from_html with the returned HTML.

Conclusion

To grab visible webpage text with BeautifulSoup, we can call filter when we’re grabbing the webpage content.

Sometimes, we want to redirect print output to a file with Python.

In this article, we’ll look at how to redirect print output to a file with Python.

How to redirect print output to a file with Python?

To redirect print output to a file with Python, we can set the file argument of print.

For instance, we write:

with open('out.txt', 'w') as f:
    print('foo', file=f)

We call open with the path of the file to write to and 'w' permission to let us write to the file.

Then we set the file parameter to f when we call print.

Now 'foo' will be written to out.txt.

The file will be closed automatically when writing is done since we used with when we open the file.

Conclusion

To redirect print output to a file with Python, we can set the file argument of print.

Sometimes, we want to convert seconds to hours, minutes and seconds with Python.

In this article, we’ll look at how to convert seconds to hours, minutes and seconds with Python.

How to convert seconds to hours, minutes and seconds with Python?

To convert seconds to hours, minutes and seconds with Python, we can use the datetime module.

For instance, we write:

import datetime

t = str(datetime.timedelta(seconds=10000))
print(t)

We call datetime.timedelta with the seconds set.

Then we call str on the returned object to get the seconds in hours, minutes, and seconds format.

Therefore, t is '2:46:40'.

Conclusion

To convert seconds to hours, minutes and seconds with Python, we can use the datetime module.

Sometimes, we want to get a list of values from a list of dicts with Python.

In this article, we’ll look at how to get a list of values from a list of dicts with Python.

How to get a list of values from a list of dicts with Python?

To get a list of values from a list of dicts with Python, we can use list comprehension.

For instance, we write:

dicts = [{
    'value': 'apple',
    'blah': 2
}, {
    'value': 'banana',
    'blah': 3
}, {
    'value': 'cars',
    'blah': 4
}]

values = [d['value'] for d in dicts if 'value' in d]
print(values)

We get the entries in dicts with for d in dicts.

Then we get the value of each entry d with d['value'].

And we only return the entries that has the value key with if 'value' in d.

Therefore, values is ['apple', 'banana', 'cars'].

Conclusion

To get a list of values from a list of dicts with Python, we can use list comprehension.

Sometimes, we want to find the MIME type of a file in Python.

In this article, we’ll look at how to find the MIME type of a file in Python.

How to find the MIME type of a file in Python?

To find the MIME type of a file in Python, we can use the python-magic package.

To install it, we run:

pip install python-magic

Then we use it by writing:

import magic

mime = magic.Magic(mime=True)
t = mime.from_file("foo.csv")
print(t)

We invoke the Magic constructor with mime set to True to let us get the MIME type of the file.

Then we call mime.from_file with the path of the file to get the MIME type of to return the MIME type of the file as a string.

Therefore, t is something like 'text/plain'.

Conclusion

To find the MIME type of a file in Python, we can use the python-magic package.