You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

501 lines
20 KiB

Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
  1. # coding: utf-8
  2. from __future__ import unicode_literals
  3. import re
  4. import socket
  5. from .common import InfoExtractor
  6. from ..compat import (
  7. compat_etree_fromstring,
  8. compat_http_client,
  9. compat_urllib_error,
  10. compat_urllib_parse_unquote,
  11. compat_urllib_parse_unquote_plus,
  12. )
  13. from ..utils import (
  14. clean_html,
  15. error_to_compat_str,
  16. ExtractorError,
  17. get_element_by_id,
  18. int_or_none,
  19. js_to_json,
  20. limit_length,
  21. parse_count,
  22. sanitized_Request,
  23. try_get,
  24. urlencode_postdata,
  25. )
  26. class FacebookIE(InfoExtractor):
  27. _VALID_URL = r'''(?x)
  28. (?:
  29. https?://
  30. (?:[\w-]+\.)?(?:facebook\.com|facebookcorewwwi\.onion)/
  31. (?:[^#]*?\#!/)?
  32. (?:
  33. (?:
  34. video/video\.php|
  35. photo\.php|
  36. video\.php|
  37. video/embed|
  38. story\.php
  39. )\?(?:.*?)(?:v|video_id|story_fbid)=|
  40. [^/]+/videos/(?:[^/]+/)?|
  41. [^/]+/posts/|
  42. groups/[^/]+/permalink/
  43. )|
  44. facebook:
  45. )
  46. (?P<id>[0-9]+)
  47. '''
  48. _LOGIN_URL = 'https://www.facebook.com/login.php?next=http%3A%2F%2Ffacebook.com%2Fhome.php&login_attempt=1'
  49. _CHECKPOINT_URL = 'https://www.facebook.com/checkpoint/?next=http%3A%2F%2Ffacebook.com%2Fhome.php&_fb_noscript=1'
  50. _NETRC_MACHINE = 'facebook'
  51. IE_NAME = 'facebook'
  52. _CHROME_USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36'
  53. _VIDEO_PAGE_TEMPLATE = 'https://www.facebook.com/video/video.php?v=%s'
  54. _VIDEO_PAGE_TAHOE_TEMPLATE = 'https://www.facebook.com/video/tahoe/async/%s/?chain=true&isvideo=true&payloadtype=primary'
  55. _TESTS = [{
  56. 'url': 'https://www.facebook.com/video.php?v=637842556329505&fref=nf',
  57. 'md5': '6a40d33c0eccbb1af76cf0485a052659',
  58. 'info_dict': {
  59. 'id': '637842556329505',
  60. 'ext': 'mp4',
  61. 'title': 're:Did you know Kei Nishikori is the first Asian man to ever reach a Grand Slam',
  62. 'uploader': 'Tennis on Facebook',
  63. 'upload_date': '20140908',
  64. 'timestamp': 1410199200,
  65. },
  66. 'skip': 'Requires logging in',
  67. }, {
  68. 'url': 'https://www.facebook.com/video.php?v=274175099429670',
  69. 'info_dict': {
  70. 'id': '274175099429670',
  71. 'ext': 'mp4',
  72. 'title': 're:^Asif Nawab Butt posted a video',
  73. 'uploader': 'Asif Nawab Butt',
  74. 'upload_date': '20140506',
  75. 'timestamp': 1399398998,
  76. 'thumbnail': r're:^https?://.*',
  77. },
  78. 'expected_warnings': [
  79. 'title'
  80. ]
  81. }, {
  82. 'note': 'Video with DASH manifest',
  83. 'url': 'https://www.facebook.com/video.php?v=957955867617029',
  84. 'md5': 'b2c28d528273b323abe5c6ab59f0f030',
  85. 'info_dict': {
  86. 'id': '957955867617029',
  87. 'ext': 'mp4',
  88. 'title': 'When you post epic content on instagram.com/433 8 million followers, this is ...',
  89. 'uploader': 'Demy de Zeeuw',
  90. 'upload_date': '20160110',
  91. 'timestamp': 1452431627,
  92. },
  93. 'skip': 'Requires logging in',
  94. }, {
  95. 'url': 'https://www.facebook.com/maxlayn/posts/10153807558977570',
  96. 'md5': '037b1fa7f3c2d02b7a0d7bc16031ecc6',
  97. 'info_dict': {
  98. 'id': '544765982287235',
  99. 'ext': 'mp4',
  100. 'title': '"What are you doing running in the snow?"',
  101. 'uploader': 'FailArmy',
  102. },
  103. 'skip': 'Video gone',
  104. }, {
  105. 'url': 'https://m.facebook.com/story.php?story_fbid=1035862816472149&id=116132035111903',
  106. 'md5': '1deb90b6ac27f7efcf6d747c8a27f5e3',
  107. 'info_dict': {
  108. 'id': '1035862816472149',
  109. 'ext': 'mp4',
  110. 'title': 'What the Flock Is Going On In New Zealand Credit: ViralHog',
  111. 'uploader': 'S. Saint',
  112. },
  113. 'skip': 'Video gone',
  114. }, {
  115. 'note': 'swf params escaped',
  116. 'url': 'https://www.facebook.com/barackobama/posts/10153664894881749',
  117. 'md5': '97ba073838964d12c70566e0085c2b91',
  118. 'info_dict': {
  119. 'id': '10153664894881749',
  120. 'ext': 'mp4',
  121. 'title': 'Average time to confirm recent Supreme Court nominees: 67 days Longest it\'s t...',
  122. 'thumbnail': r're:^https?://.*',
  123. 'timestamp': 1456259628,
  124. 'upload_date': '20160223',
  125. 'uploader': 'Barack Obama',
  126. },
  127. }, {
  128. # have 1080P, but only up to 720p in swf params
  129. 'url': 'https://www.facebook.com/cnn/videos/10155529876156509/',
  130. 'md5': '9571fae53d4165bbbadb17a94651dcdc',
  131. 'info_dict': {
  132. 'id': '10155529876156509',
  133. 'ext': 'mp4',
  134. 'title': 'She survived the holocaust — and years later, she’s getting her citizenship s...',
  135. 'timestamp': 1477818095,
  136. 'upload_date': '20161030',
  137. 'uploader': 'CNN',
  138. 'thumbnail': r're:^https?://.*',
  139. 'view_count': int,
  140. },
  141. }, {
  142. # bigPipe.onPageletArrive ... onPageletArrive pagelet_group_mall
  143. 'url': 'https://www.facebook.com/yaroslav.korpan/videos/1417995061575415/',
  144. 'info_dict': {
  145. 'id': '1417995061575415',
  146. 'ext': 'mp4',
  147. 'title': 'md5:1db063d6a8c13faa8da727817339c857',
  148. 'timestamp': 1486648217,
  149. 'upload_date': '20170209',
  150. 'uploader': 'Yaroslav Korpan',
  151. },
  152. 'params': {
  153. 'skip_download': True,
  154. },
  155. }, {
  156. 'url': 'https://www.facebook.com/LaGuiaDelVaron/posts/1072691702860471',
  157. 'info_dict': {
  158. 'id': '1072691702860471',
  159. 'ext': 'mp4',
  160. 'title': 'md5:ae2d22a93fbb12dad20dc393a869739d',
  161. 'timestamp': 1477305000,
  162. 'upload_date': '20161024',
  163. 'uploader': 'La Guía Del Varón',
  164. 'thumbnail': r're:^https?://.*',
  165. },
  166. 'params': {
  167. 'skip_download': True,
  168. },
  169. }, {
  170. 'url': 'https://www.facebook.com/groups/1024490957622648/permalink/1396382447100162/',
  171. 'info_dict': {
  172. 'id': '1396382447100162',
  173. 'ext': 'mp4',
  174. 'title': 'md5:19a428bbde91364e3de815383b54a235',
  175. 'timestamp': 1486035494,
  176. 'upload_date': '20170202',
  177. 'uploader': 'Elisabeth Ahtn',
  178. },
  179. 'params': {
  180. 'skip_download': True,
  181. },
  182. }, {
  183. 'url': 'https://www.facebook.com/video.php?v=10204634152394104',
  184. 'only_matching': True,
  185. }, {
  186. 'url': 'https://www.facebook.com/amogood/videos/1618742068337349/?fref=nf',
  187. 'only_matching': True,
  188. }, {
  189. 'url': 'https://www.facebook.com/ChristyClarkForBC/videos/vb.22819070941/10153870694020942/?type=2&theater',
  190. 'only_matching': True,
  191. }, {
  192. 'url': 'facebook:544765982287235',
  193. 'only_matching': True,
  194. }, {
  195. 'url': 'https://www.facebook.com/groups/164828000315060/permalink/764967300301124/',
  196. 'only_matching': True,
  197. }, {
  198. 'url': 'https://zh-hk.facebook.com/peoplespower/videos/1135894589806027/',
  199. 'only_matching': True,
  200. }, {
  201. 'url': 'https://www.facebookcorewwwi.onion/video.php?v=274175099429670',
  202. 'only_matching': True,
  203. }, {
  204. # no title
  205. 'url': 'https://www.facebook.com/onlycleverentertainment/videos/1947995502095005/',
  206. 'only_matching': True,
  207. }, {
  208. 'url': 'https://www.facebook.com/WatchESLOne/videos/359649331226507/',
  209. 'info_dict': {
  210. 'id': '359649331226507',
  211. 'ext': 'mp4',
  212. 'title': '#ESLOne VoD - Birmingham Finals Day#1 Fnatic vs. @Evil Geniuses',
  213. 'uploader': 'ESL One Dota 2',
  214. },
  215. 'params': {
  216. 'skip_download': True,
  217. },
  218. }]
  219. @staticmethod
  220. def _extract_urls(webpage):
  221. urls = []
  222. for mobj in re.finditer(
  223. r'<iframe[^>]+?src=(["\'])(?P<url>https?://www\.facebook\.com/(?:video/embed|plugins/video\.php).+?)\1',
  224. webpage):
  225. urls.append(mobj.group('url'))
  226. # Facebook API embed
  227. # see https://developers.facebook.com/docs/plugins/embedded-video-player
  228. for mobj in re.finditer(r'''(?x)<div[^>]+
  229. class=(?P<q1>[\'"])[^\'"]*\bfb-(?:video|post)\b[^\'"]*(?P=q1)[^>]+
  230. data-href=(?P<q2>[\'"])(?P<url>(?:https?:)?//(?:www\.)?facebook.com/.+?)(?P=q2)''', webpage):
  231. urls.append(mobj.group('url'))
  232. return urls
  233. def _login(self):
  234. useremail, password = self._get_login_info()
  235. if useremail is None:
  236. return
  237. login_page_req = sanitized_Request(self._LOGIN_URL)
  238. self._set_cookie('facebook.com', 'locale', 'en_US')
  239. login_page = self._download_webpage(login_page_req, None,
  240. note='Downloading login page',
  241. errnote='Unable to download login page')
  242. lsd = self._search_regex(
  243. r'<input type="hidden" name="lsd" value="([^"]*)"',
  244. login_page, 'lsd')
  245. lgnrnd = self._search_regex(r'name="lgnrnd" value="([^"]*?)"', login_page, 'lgnrnd')
  246. login_form = {
  247. 'email': useremail,
  248. 'pass': password,
  249. 'lsd': lsd,
  250. 'lgnrnd': lgnrnd,
  251. 'next': 'http://facebook.com/home.php',
  252. 'default_persistent': '0',
  253. 'legacy_return': '1',
  254. 'timezone': '-60',
  255. 'trynum': '1',
  256. }
  257. request = sanitized_Request(self._LOGIN_URL, urlencode_postdata(login_form))
  258. request.add_header('Content-Type', 'application/x-www-form-urlencoded')
  259. try:
  260. login_results = self._download_webpage(request, None,
  261. note='Logging in', errnote='unable to fetch login page')
  262. if re.search(r'<form(.*)name="login"(.*)</form>', login_results) is not None:
  263. error = self._html_search_regex(
  264. r'(?s)<div[^>]+class=(["\']).*?login_error_box.*?\1[^>]*><div[^>]*>.*?</div><div[^>]*>(?P<error>.+?)</div>',
  265. login_results, 'login error', default=None, group='error')
  266. if error:
  267. raise ExtractorError('Unable to login: %s' % error, expected=True)
  268. self._downloader.report_warning('unable to log in: bad username/password, or exceeded login rate limit (~3/min). Check credentials or wait.')
  269. return
  270. fb_dtsg = self._search_regex(
  271. r'name="fb_dtsg" value="(.+?)"', login_results, 'fb_dtsg', default=None)
  272. h = self._search_regex(
  273. r'name="h"\s+(?:\w+="[^"]+"\s+)*?value="([^"]+)"', login_results, 'h', default=None)
  274. if not fb_dtsg or not h:
  275. return
  276. check_form = {
  277. 'fb_dtsg': fb_dtsg,
  278. 'h': h,
  279. 'name_action_selected': 'dont_save',
  280. }
  281. check_req = sanitized_Request(self._CHECKPOINT_URL, urlencode_postdata(check_form))
  282. check_req.add_header('Content-Type', 'application/x-www-form-urlencoded')
  283. check_response = self._download_webpage(check_req, None,
  284. note='Confirming login')
  285. if re.search(r'id="checkpointSubmitButton"', check_response) is not None:
  286. self._downloader.report_warning('Unable to confirm login, you have to login in your browser and authorize the login.')
  287. except (compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
  288. self._downloader.report_warning('unable to log in: %s' % error_to_compat_str(err))
  289. return
  290. def _real_initialize(self):
  291. self._login()
  292. def _extract_from_url(self, url, video_id, fatal_if_no_video=True):
  293. req = sanitized_Request(url)
  294. req.add_header('User-Agent', self._CHROME_USER_AGENT)
  295. webpage = self._download_webpage(req, video_id)
  296. video_data = None
  297. def extract_video_data(instances):
  298. for item in instances:
  299. if item[1][0] == 'VideoConfig':
  300. video_item = item[2][0]
  301. if video_item.get('video_id'):
  302. return video_item['videoData']
  303. server_js_data = self._parse_json(self._search_regex(
  304. r'handleServerJS\(({.+})(?:\);|,")', webpage,
  305. 'server js data', default='{}'), video_id, fatal=False)
  306. if server_js_data:
  307. video_data = extract_video_data(server_js_data.get('instances', []))
  308. def extract_from_jsmods_instances(js_data):
  309. if js_data:
  310. return extract_video_data(try_get(
  311. js_data, lambda x: x['jsmods']['instances'], list) or [])
  312. if not video_data:
  313. server_js_data = self._parse_json(
  314. self._search_regex(
  315. r'bigPipe\.onPageletArrive\(({.+?})\)\s*;\s*}\s*\)\s*,\s*["\']onPageletArrive\s+(?:stream_pagelet|pagelet_group_mall|permalink_video_pagelet)',
  316. webpage, 'js data', default='{}'),
  317. video_id, transform_source=js_to_json, fatal=False)
  318. video_data = extract_from_jsmods_instances(server_js_data)
  319. if not video_data:
  320. if not fatal_if_no_video:
  321. return webpage, False
  322. m_msg = re.search(r'class="[^"]*uiInterstitialContent[^"]*"><div>(.*?)</div>', webpage)
  323. if m_msg is not None:
  324. raise ExtractorError(
  325. 'The video is not available, Facebook said: "%s"' % m_msg.group(1),
  326. expected=True)
  327. elif '>You must log in to continue' in webpage:
  328. self.raise_login_required()
  329. # Video info not in first request, do a secondary request using
  330. # tahoe player specific URL
  331. tahoe_data = self._download_webpage(
  332. self._VIDEO_PAGE_TAHOE_TEMPLATE % video_id, video_id,
  333. data=urlencode_postdata({
  334. '__a': 1,
  335. '__pc': self._search_regex(
  336. r'pkg_cohort["\']\s*:\s*["\'](.+?)["\']', webpage,
  337. 'pkg cohort', default='PHASED:DEFAULT'),
  338. '__rev': self._search_regex(
  339. r'client_revision["\']\s*:\s*(\d+),', webpage,
  340. 'client revision', default='3944515'),
  341. 'fb_dtsg': self._search_regex(
  342. r'"DTSGInitialData"\s*,\s*\[\]\s*,\s*{\s*"token"\s*:\s*"([^"]+)"',
  343. webpage, 'dtsg token', default=''),
  344. }),
  345. headers={
  346. 'Content-Type': 'application/x-www-form-urlencoded',
  347. })
  348. tahoe_js_data = self._parse_json(
  349. self._search_regex(
  350. r'for\s+\(\s*;\s*;\s*\)\s*;(.+)', tahoe_data,
  351. 'tahoe js data', default='{}'),
  352. video_id, fatal=False)
  353. video_data = extract_from_jsmods_instances(tahoe_js_data)
  354. if not video_data:
  355. raise ExtractorError('Cannot parse data')
  356. formats = []
  357. for f in video_data:
  358. format_id = f['stream_type']
  359. if f and isinstance(f, dict):
  360. f = [f]
  361. if not f or not isinstance(f, list):
  362. continue
  363. for quality in ('sd', 'hd'):
  364. for src_type in ('src', 'src_no_ratelimit'):
  365. src = f[0].get('%s_%s' % (quality, src_type))
  366. if src:
  367. preference = -10 if format_id == 'progressive' else 0
  368. if quality == 'hd':
  369. preference += 5
  370. formats.append({
  371. 'format_id': '%s_%s_%s' % (format_id, quality, src_type),
  372. 'url': src,
  373. 'preference': preference,
  374. })
  375. dash_manifest = f[0].get('dash_manifest')
  376. if dash_manifest:
  377. formats.extend(self._parse_mpd_formats(
  378. compat_etree_fromstring(compat_urllib_parse_unquote_plus(dash_manifest))))
  379. if not formats:
  380. raise ExtractorError('Cannot find video formats')
  381. self._sort_formats(formats)
  382. video_title = self._html_search_regex(
  383. r'<h2\s+[^>]*class="uiHeaderTitle"[^>]*>([^<]*)</h2>', webpage,
  384. 'title', default=None)
  385. if not video_title:
  386. video_title = self._html_search_regex(
  387. r'(?s)<span class="fbPhotosPhotoCaption".*?id="fbPhotoPageCaption"><span class="hasCaption">(.*?)</span>',
  388. webpage, 'alternative title', default=None)
  389. if not video_title:
  390. video_title = self._html_search_meta(
  391. 'description', webpage, 'title', default=None)
  392. if video_title:
  393. video_title = limit_length(video_title, 80)
  394. else:
  395. video_title = 'Facebook video #%s' % video_id
  396. uploader = clean_html(get_element_by_id(
  397. 'fbPhotoPageAuthorName', webpage)) or self._search_regex(
  398. r'ownerName\s*:\s*"([^"]+)"', webpage, 'uploader',
  399. default=None) or self._og_search_title(webpage, fatal=False)
  400. timestamp = int_or_none(self._search_regex(
  401. r'<abbr[^>]+data-utime=["\'](\d+)', webpage,
  402. 'timestamp', default=None))
  403. thumbnail = self._og_search_thumbnail(webpage)
  404. view_count = parse_count(self._search_regex(
  405. r'\bviewCount\s*:\s*["\']([\d,.]+)', webpage, 'view count',
  406. default=None))
  407. info_dict = {
  408. 'id': video_id,
  409. 'title': video_title,
  410. 'formats': formats,
  411. 'uploader': uploader,
  412. 'timestamp': timestamp,
  413. 'thumbnail': thumbnail,
  414. 'view_count': view_count,
  415. }
  416. return webpage, info_dict
  417. def _real_extract(self, url):
  418. video_id = self._match_id(url)
  419. real_url = self._VIDEO_PAGE_TEMPLATE % video_id if url.startswith('facebook:') else url
  420. webpage, info_dict = self._extract_from_url(real_url, video_id, fatal_if_no_video=False)
  421. if info_dict:
  422. return info_dict
  423. if '/posts/' in url:
  424. entries = [
  425. self.url_result('facebook:%s' % vid, FacebookIE.ie_key())
  426. for vid in self._parse_json(
  427. self._search_regex(
  428. r'(["\'])video_ids\1\s*:\s*(?P<ids>\[.+?\])',
  429. webpage, 'video ids', group='ids'),
  430. video_id)]
  431. return self.playlist_result(entries, video_id)
  432. else:
  433. _, info_dict = self._extract_from_url(
  434. self._VIDEO_PAGE_TEMPLATE % video_id,
  435. video_id, fatal_if_no_video=True)
  436. return info_dict
  437. class FacebookPluginsVideoIE(InfoExtractor):
  438. _VALID_URL = r'https?://(?:[\w-]+\.)?facebook\.com/plugins/video\.php\?.*?\bhref=(?P<id>https.+)'
  439. _TESTS = [{
  440. 'url': 'https://www.facebook.com/plugins/video.php?href=https%3A%2F%2Fwww.facebook.com%2Fgov.sg%2Fvideos%2F10154383743583686%2F&show_text=0&width=560',
  441. 'md5': '5954e92cdfe51fe5782ae9bda7058a07',
  442. 'info_dict': {
  443. 'id': '10154383743583686',
  444. 'ext': 'mp4',
  445. 'title': 'What to do during the haze?',
  446. 'uploader': 'Gov.sg',
  447. 'upload_date': '20160826',
  448. 'timestamp': 1472184808,
  449. },
  450. 'add_ie': [FacebookIE.ie_key()],
  451. }, {
  452. 'url': 'https://www.facebook.com/plugins/video.php?href=https%3A%2F%2Fwww.facebook.com%2Fvideo.php%3Fv%3D10204634152394104',
  453. 'only_matching': True,
  454. }, {
  455. 'url': 'https://www.facebook.com/plugins/video.php?href=https://www.facebook.com/gov.sg/videos/10154383743583686/&show_text=0&width=560',
  456. 'only_matching': True,
  457. }]
  458. def _real_extract(self, url):
  459. return self.url_result(
  460. compat_urllib_parse_unquote(self._match_id(url)),
  461. FacebookIE.ie_key())