lexifdev
yeah. many websites are cannot crawl without real browser based tool.
but I always try this once.
every time I crawl Facebook, I use mobile website (https://iphone.facebook.com/ or https://iphone.facebook.com/). it has also ‘noscript’ version.
some sites that blocked by ‘User-Agent’ or ‘Referer’ are can avoid by
```session = requests.Session()
session.headers.update('Referer', 'https://~~~') # or
session.headers.update('User-Agent', 'Mozilla/5.0 ~~~~')```
(maybe you already know)
but I always try this once.
every time I crawl Facebook, I use mobile website (https://iphone.facebook.com/ or https://iphone.facebook.com/). it has also ‘noscript’ version.
some sites that blocked by ‘User-Agent’ or ‘Referer’ are can avoid by
```session = requests.Session()
session.headers.update('Referer', 'https://~~~') # or
session.headers.update('User-Agent', 'Mozilla/5.0 ~~~~')```
(maybe you already know)
iphone.facebook.com
Create an account or log into Facebook. Connect with friends, family and other people you know. Share photos and videos, send messages and get updates.
中時電子報 
4
3
3
facebook.com 
1
芋傳媒 TaroNews 
nooho.net 
1
GitHub 
數位時代 
YouTube 
twreporter.org 
HackMD 
Stack Overflow 
自由電子報 
2
Google Docs 
