{"id":65125,"date":"2022-06-16T09:00:36","date_gmt":"2022-06-16T09:00:36","guid":{"rendered":"https:\/\/www.cryptocabaret.com\/?p=65125"},"modified":"2022-06-16T09:00:36","modified_gmt":"2022-06-16T09:00:36","slug":"analyze-web-pages-with-python-requests-and-beautiful-soup","status":"publish","type":"post","link":"https:\/\/www.cryptocabaret.com\/?p=65125","title":{"rendered":"Analyze web pages with Python requests and Beautiful Soup"},"content":{"rendered":"

Analyze web pages with Python requests and Beautiful Soup<\/span>
\nSeth Kenlon<\/a><\/span>
\nThu, 06\/16\/2022 – 03:00<\/span><\/p>\n

\n
1 reader likes this<\/div>\n
1 reader likes this<\/div>\n<\/div>\n
\n

Browsing the web probably accounts for much of your day. But it’s an awfully manual process, isn’t it? You have to open a browser. Go to a website. Click buttons, move a mouse. It’s a lot of work. Wouldn’t it be nicer to interact with the Internet through code?<\/p>\n

You can get data from the Internet using Python with the help of the Python module requests<\/code>:<\/p>\n

\n
import<\/span> requests

\nDATA =<\/span> \"https:\/\/opensource.com\/article\/22\/5\/document-source-code-doxygen-linux\"<\/span>
\nPAGE =<\/span> requests.get<\/span>(<\/span>DATA)<\/span>

print<\/span>(<\/span>PAGE.text<\/span>)<\/span><\/div><\/div><\/pre>\n
In this code sample, you first import the module requests<\/code>. Then you create two variables: one called DATA<\/code> to hold the URL you want to download. In later versions of this code, you’ll be able to provide a different URL each time you run your application. For now, though, it’s easiest to just “hard code\u201d a test URL for demonstration purposes.<\/p>\n
The other variable is PAGE<\/code>, which you set to the response <\/em>of the requests.get<\/code> function when it reads the URL stored in DATA<\/code>. The requests<\/code> module and its .get<\/code> function is pre-programmed to “read\u201d an Internet address (a URL), access the Internet, and download whatever is located at that address.<\/p>\n
That’s a lot of steps you don’t have to figure out on your own, and that’s exactly why Python modules exist. Finally, you tell Python to print<\/code> everything that requests.get<\/code> has stored in the .text<\/code> field of the PAGE<\/code> variable.<\/p>\n
Beautiful Soup<\/h2>\n
If you run the sample code above, you get the contents of the example URL dumped indiscriminately into your terminal. It does that because the only thing your code does with the data that requests<\/code> has gathered is print it. It’s more interesting to parse the text.<\/p>\n
Python can “read\u201d text with its most basic functions, but parsing text allows you to search for patterns, specific words, HTML tags, and so on. You could parse the text returned by requests<\/code> yourself, but using a specialized module is much easier. For HTML and XML, there’s the Beautiful Soup<\/a> library.<\/p>\n
This code accomplishes the same thing, but it uses Beautiful Soup to parse the downloaded text. Because Beautiful Soup recognizes HTML entities, you can use some of its built-in features to make the output a little easier for the human eye to parse.<\/p>\n
For instance, instead of printing raw text at the end of your program, you can run the text through the .prettify<\/code> function of Beautiful Soup:<\/p>\n
\n
from<\/span> bs4 import<\/span> BeautifulSoup
import<\/span> requests

\nPAGE =<\/span> requests.get<\/span>(<\/span>\"https:\/\/opensource.com\/article\/22\/5\/document-source-code-doxygen-linux\"<\/span>)<\/span>
\nSOUP =<\/span> BeautifulSoup(<\/span>PAGE.text<\/span>,<\/span> 'html.parser'<\/span>)<\/span>

# Press the green button in the gutter to run the script.<\/span>
if<\/span> __name__ ==<\/span> '__main__'<\/span>:
\n\u00a0 \u00a0 # do a thing here<\/span>
\n\u00a0 \u00a0 print<\/span>(<\/span>SOUP.prettify<\/span>(<\/span>)<\/span>)<\/span><\/div><\/div><\/pre>\n
The output of this version of your program ensures that every opening HTML tag starts on its own line, with indentation to help demonstrate which tag is a parent of another tag. Beautiful Soup is aware of HTML tags in more ways than just how it prints it out.<\/p>\n
Instead of printing the whole page, you can single out a specific kind of tag. For instance, try changing the print selector from print(SOUP.prettify()<\/span> to this:<\/p>\n
\n\u00a0 print<\/span>(<\/span>SOUP.p<\/span>)<\/span><\/code><\/span><\/pre>\n
This prints just a <\/p>\n
<\/code> tag. Specifically, it prints just the first <\/p>\n
<\/code> tag encountered. To print all <\/p>\n
<\/code> tags, you need a loop.<\/p>\n<\/p>\n
\n
More Python resources<\/div>\n
\n
What is an IDE?<\/a><\/div>\n
Cheat sheet: Python 3.7 for beginners<\/a><\/div>\n
Top Python GUI frameworks<\/a><\/div>\n
Download: 7 essential PyPI libraries<\/a><\/div>\n
Red Hat Developers<\/a><\/div>\n
Latest Python articles<\/a><\/div>\n<\/p><\/div>\n<\/p><\/div>\n
Looping<\/h2>\n
Create a for loop to cycle over the entire webpage contained in the SOUP<\/code> variable, using the find_all<\/code> function of Beautiful Soup. It’s not unreasonable to want to use your loop for other tags besides just the <\/p>\n
<\/code> tag, so build it as a custom function, designated by the def<\/code> keyword (for “define\u201d) in Python.<\/p>\n
\n
def<\/span> loopit(<\/span>)<\/span>:
\n\u00a0 \u00a0 for<\/span> TAG in<\/span> SOUP.find_all<\/span>(<\/span>'p'<\/span>)<\/span>:
\n\u00a0 \u00a0 \u00a0 \u00a0 print<\/span>(<\/span>TAG)<\/span><\/div><\/div><\/pre>\n
The temporary variable TAG<\/code> is arbitrary. You can use any term, such as ITEM<\/code> or i<\/code> or whatever you want. Each time the loop runs, TAG<\/code> contains the search results of the find_all<\/code> function. In this code, the <\/p>\n
<\/code> tag is being searched.<\/p>\n
A function doesn’t run unless it’s explicitly called. You can call your function at the end of your code:<\/p>\n
\n
# Press the green button in the gutter to run the script.<\/span>
if<\/span> __name__ ==<\/span> '__main__'<\/span>:
\n\u00a0 \u00a0 # do a thing here<\/span>
\n\u00a0 \u00a0 loopit(<\/span>)<\/span><\/div><\/div><\/pre>\n
Run your code to see all <\/p>\n
<\/code> tags and each one’s contents.<\/p>\n
Getting just the content<\/h2>\n
You can exclude tags from being printed by specifying that you want just the “string\u201d (programming lingo for “words\u201d).<\/p>\n
\n
def<\/span> loopit(<\/span>)<\/span>:
\n\u00a0 \u00a0 for<\/span> TAG in<\/span> SOUP.find_all<\/span>(<\/span>'p'<\/span>)<\/span>:
\n\u00a0 \u00a0 \u00a0 \u00a0 print<\/span>(<\/span>TAG.string<\/span>)<\/span><\/div><\/div><\/pre>\n
Of course, once you have the text of a webpage, you can parse it further with the standard Python string libraries. For instance, you can get a word count using len<\/code> and split<\/code>:<\/p>\n
\n
def<\/span> loopit(<\/span>)<\/span>:
\n\u00a0 \u00a0 for<\/span> TAG in<\/span> SOUP.find_all<\/span>(<\/span>'p'<\/span>)<\/span>:
\n\u00a0 \u00a0 \u00a0 \u00a0 if<\/span> TAG.string<\/span> is<\/span> not<\/span> None<\/span>:
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 print<\/span>(<\/span>len<\/span>(<\/span>TAG.string<\/span>.split<\/span>(<\/span>)<\/span>)<\/span>)<\/span><\/div><\/div><\/pre>\n
This prints the number of strings within each paragraph element, omitting those paragraphs that don’t have any strings. To get a grand total, use a variable and some basic math:<\/p>\n
\n
def<\/span> loopit(<\/span>)<\/span>:
\n\u00a0 \u00a0 NUM =<\/span> 0<\/span>
\n\u00a0 \u00a0 for<\/span> TAG in<\/span> SOUP.find_all<\/span>(<\/span>'p'<\/span>)<\/span>:
\n\u00a0 \u00a0 \u00a0 \u00a0 if<\/span> TAG.string<\/span> is<\/span> not<\/span> None<\/span>:
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 NUM =<\/span> NUM + len<\/span>(<\/span>TAG.string<\/span>.split<\/span>(<\/span>)<\/span>)<\/span>
\n\u00a0 \u00a0 print<\/span>(<\/span>\"Grand total is \"<\/span>,<\/span> NUM)<\/span><\/div><\/div><\/pre>\n
Python homework<\/h2>\n
There’s a lot more information you can extract with Beautiful Soup and Python. Here are some ideas on how to improve your application:<\/p>\n