Python Web Scraping: Scrap number of tweets of a given Twitter account
Python Web Scraping: Exercise-20 with Solution
Write a Python program to scrap number of tweets of a given Twitter account.
Sample Solution:
Python Code:
from bs4 import BeautifulSoup
import requests
handle = input('Input your account name on Twitter: ')
ctr = int(input('Input number of tweets to scrape: '))
res=requests.get('https://twitter.com/'+ handle)
bs=BeautifulSoup(res.content,'lxml')
all_tweets = bs.find_all('div',{'class':'tweet'})
if all_tweets:
for tweet in all_tweets[:ctr]:
context = tweet.find('div',{'class':'context'}).text.replace("\n"," ").strip()
content = tweet.find('div',{'class':'content'})
header = content.find('div',{'class':'stream-item-header'})
user = header.find('a',{'class':'account-group js-account-group js-action-profile js-user-profile-link js-nav'}).text.replace("\n"," ").strip()
time = header.find('a',{'class':'tweet-timestamp js-permalink js-nav js-tooltip'}).find('span').text.replace("\n"," ").strip()
message = content.find('div',{'class':'js-tweet-text-container'}).text.replace("\n"," ").strip()
footer = content.find('div',{'class':'stream-item-footer'})
stat = footer.find('div',{'class':'ProfileTweet-actionCountList u-hiddenVisually'}).text.replace("\n"," ").strip()
if context:
print(context)
print(user,time)
print(message)
print(stat)
print()
else:
print("List is empty/account name not found.")
Sample Output:
Input your account name on Twitter: @prasanta Input number of tweets to scrape: 2 Prasanta Kundu Retweeted WorkApps @WorkAppsInc 13 Nov 2017 A Dedicated Mobile App for your Sales Teams to keep All Product Knowledge handy. Works even when Offline!! Log on to https://www.KnowledgeApps.ai to know more!pic.twitter.com/5F7v1CRHma 0 replies 3 retweets 2 likes Prasanta Kundu Retweeted MVS Murthy @mittispeaks 8 Nov 2017 This one is for all the #BFSI folks ... #HRManager #HRTech Help your sales team sell better. Complete #Product #Knowledge on a dedicated mobile app which works even in #offline mode. Know more @ https://www.knowledgeapps.ai pic.twitter.com/dwsOpFgX0A 0 replies 2 retweets 0 likes
Flowchart:
Python Code Editor:
Have another way to solve this solution? Contribute your code (and comments) through Disqus.
Previous: Write a Python program to count number of tweets by a given Twitter account.
Next: Write a Python program to find the live weather report (temperature, wind speed, description and weather) of a given city.
What is the difficulty level of this exercise?
Python: Tips of the Day
Find current directory and file's directory:
To get the full path to the directory a Python file is contained in, write this in that file:
import os dir_path = os.path.dirname(os.path.realpath(__file__))
(Note that the incantation above won't work if you've already used os.chdir() to change your current working directory, since the value of the __file__ constant is relative to the current working directory and is not changed by an os.chdir() call.)
To get the current working directory use
import os cwd = os.getcwd()
Documentation references for the modules, constants and functions used above:
- The os and os.path modules.
- The __file__ constant
- os.path.realpath(path) (returns "the canonical path of the specified filename, eliminating any symbolic links encountered in the path")
- os.path.dirname(path) (returns "the directory name of pathname path")
- os.getcwd() (returns "a string representing the current working directory")
- os.chdir(path) ("change the current working directory to path")
Ref: https://bit.ly/3fy0R6m
- New Content published on w3resource:
- HTML-CSS Practical: Exercises, Practice, Solution
- Java Regular Expression: Exercises, Practice, Solution
- Scala Programming Exercises, Practice, Solution
- Python Itertools exercises
- Python Numpy exercises
- Python GeoPy Package exercises
- Python Pandas exercises
- Python nltk exercises
- Python BeautifulSoup exercises
- Form Template
- Composer - PHP Package Manager
- PHPUnit - PHP Testing
- Laravel - PHP Framework
- Angular - JavaScript Framework
- Vue - JavaScript Framework
- Jest - JavaScript Testing Framework