You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

253 lines
9.9 KiB

Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
  1. from __future__ import unicode_literals
  2. import re
  3. from .common import InfoExtractor
  4. from ..compat import (
  5. compat_parse_qs,
  6. compat_urllib_parse,
  7. compat_urllib_parse_unquote,
  8. )
  9. from ..utils import (
  10. determine_ext,
  11. ExtractorError,
  12. int_or_none,
  13. sanitized_Request,
  14. )
  15. class MetacafeIE(InfoExtractor):
  16. _VALID_URL = r'http://(?:www\.)?metacafe\.com/watch/([^/]+)/([^/]+)/.*'
  17. _DISCLAIMER = 'http://www.metacafe.com/family_filter/'
  18. _FILTER_POST = 'http://www.metacafe.com/f/index.php?inputType=filter&controllerGroup=user'
  19. IE_NAME = 'metacafe'
  20. _TESTS = [
  21. # Youtube video
  22. {
  23. 'add_ie': ['Youtube'],
  24. 'url': 'http://metacafe.com/watch/yt-_aUehQsCQtM/the_electric_company_short_i_pbs_kids_go/',
  25. 'info_dict': {
  26. 'id': '_aUehQsCQtM',
  27. 'ext': 'mp4',
  28. 'upload_date': '20090102',
  29. 'title': 'The Electric Company | "Short I" | PBS KIDS GO!',
  30. 'description': 'md5:2439a8ef6d5a70e380c22f5ad323e5a8',
  31. 'uploader': 'PBS',
  32. 'uploader_id': 'PBS'
  33. }
  34. },
  35. # Normal metacafe video
  36. {
  37. 'url': 'http://www.metacafe.com/watch/11121940/news_stuff_you_wont_do_with_your_playstation_4/',
  38. 'md5': '6e0bca200eaad2552e6915ed6fd4d9ad',
  39. 'info_dict': {
  40. 'id': '11121940',
  41. 'ext': 'mp4',
  42. 'title': 'News: Stuff You Won\'t Do with Your PlayStation 4',
  43. 'uploader': 'ign',
  44. 'description': 'Sony released a massive FAQ on the PlayStation Blog detailing the PS4\'s capabilities and limitations.',
  45. },
  46. },
  47. # AnyClip video
  48. {
  49. 'url': 'http://www.metacafe.com/watch/an-dVVXnuY7Jh77J/the_andromeda_strain_1971_stop_the_bomb_part_3/',
  50. 'info_dict': {
  51. 'id': 'an-dVVXnuY7Jh77J',
  52. 'ext': 'mp4',
  53. 'title': 'The Andromeda Strain (1971): Stop the Bomb Part 3',
  54. 'uploader': 'anyclip',
  55. 'description': 'md5:38c711dd98f5bb87acf973d573442e67',
  56. },
  57. },
  58. # age-restricted video
  59. {
  60. 'url': 'http://www.metacafe.com/watch/5186653/bbc_internal_christmas_tape_79_uncensored_outtakes_etc/',
  61. 'md5': '98dde7c1a35d02178e8ab7560fe8bd09',
  62. 'info_dict': {
  63. 'id': '5186653',
  64. 'ext': 'mp4',
  65. 'title': 'BBC INTERNAL Christmas Tape \'79 - UNCENSORED Outtakes, Etc.',
  66. 'uploader': 'Dwayne Pipe',
  67. 'description': 'md5:950bf4c581e2c059911fa3ffbe377e4b',
  68. 'age_limit': 18,
  69. },
  70. },
  71. # cbs video
  72. {
  73. 'url': 'http://www.metacafe.com/watch/cb-8VD4r_Zws8VP/open_this_is_face_the_nation_february_9/',
  74. 'info_dict': {
  75. 'id': '8VD4r_Zws8VP',
  76. 'ext': 'flv',
  77. 'title': 'Open: This is Face the Nation, February 9',
  78. 'description': 'md5:8a9ceec26d1f7ed6eab610834cc1a476',
  79. 'duration': 96,
  80. },
  81. 'params': {
  82. # rtmp download
  83. 'skip_download': True,
  84. },
  85. },
  86. # Movieclips.com video
  87. {
  88. 'url': 'http://www.metacafe.com/watch/mv-Wy7ZU/my_week_with_marilyn_do_you_love_me/',
  89. 'info_dict': {
  90. 'id': 'mv-Wy7ZU',
  91. 'ext': 'mp4',
  92. 'title': 'My Week with Marilyn - Do You Love Me?',
  93. 'description': 'From the movie My Week with Marilyn - Colin (Eddie Redmayne) professes his love to Marilyn (Michelle Williams) and gets her to promise to return to set and finish the movie.',
  94. 'uploader': 'movie_trailers',
  95. 'duration': 176,
  96. },
  97. 'params': {
  98. 'skip_download': 'requires rtmpdump',
  99. }
  100. }
  101. ]
  102. def report_disclaimer(self):
  103. self.to_screen('Retrieving disclaimer')
  104. def _real_initialize(self):
  105. # Retrieve disclaimer
  106. self.report_disclaimer()
  107. self._download_webpage(self._DISCLAIMER, None, False, 'Unable to retrieve disclaimer')
  108. # Confirm age
  109. disclaimer_form = {
  110. 'filters': '0',
  111. 'submit': "Continue - I'm over 18",
  112. }
  113. request = sanitized_Request(self._FILTER_POST, compat_urllib_parse.urlencode(disclaimer_form))
  114. request.add_header('Content-Type', 'application/x-www-form-urlencoded')
  115. self.report_age_confirmation()
  116. self._download_webpage(request, None, False, 'Unable to confirm age')
  117. def _real_extract(self, url):
  118. # Extract id and simplified title from URL
  119. mobj = re.match(self._VALID_URL, url)
  120. if mobj is None:
  121. raise ExtractorError('Invalid URL: %s' % url)
  122. video_id = mobj.group(1)
  123. # the video may come from an external site
  124. m_external = re.match('^(\w{2})-(.*)$', video_id)
  125. if m_external is not None:
  126. prefix, ext_id = m_external.groups()
  127. # Check if video comes from YouTube
  128. if prefix == 'yt':
  129. return self.url_result('http://www.youtube.com/watch?v=%s' % ext_id, 'Youtube')
  130. # CBS videos use theplatform.com
  131. if prefix == 'cb':
  132. return self.url_result('theplatform:%s' % ext_id, 'ThePlatform')
  133. # Retrieve video webpage to extract further information
  134. req = sanitized_Request('http://www.metacafe.com/watch/%s/' % video_id)
  135. # AnyClip videos require the flashversion cookie so that we get the link
  136. # to the mp4 file
  137. mobj_an = re.match(r'^an-(.*?)$', video_id)
  138. if mobj_an:
  139. req.headers['Cookie'] = 'flashVersion=0;'
  140. webpage = self._download_webpage(req, video_id)
  141. # Extract URL, uploader and title from webpage
  142. self.report_extraction(video_id)
  143. video_url = None
  144. mobj = re.search(r'(?m)&mediaURL=([^&]+)', webpage)
  145. if mobj is not None:
  146. mediaURL = compat_urllib_parse_unquote(mobj.group(1))
  147. video_ext = mediaURL[-3:]
  148. # Extract gdaKey if available
  149. mobj = re.search(r'(?m)&gdaKey=(.*?)&', webpage)
  150. if mobj is None:
  151. video_url = mediaURL
  152. else:
  153. gdaKey = mobj.group(1)
  154. video_url = '%s?__gda__=%s' % (mediaURL, gdaKey)
  155. if video_url is None:
  156. mobj = re.search(r'<video src="([^"]+)"', webpage)
  157. if mobj:
  158. video_url = mobj.group(1)
  159. video_ext = 'mp4'
  160. if video_url is None:
  161. flashvars = self._search_regex(
  162. r' name="flashvars" value="(.*?)"', webpage, 'flashvars',
  163. default=None)
  164. if flashvars:
  165. vardict = compat_parse_qs(flashvars)
  166. if 'mediaData' not in vardict:
  167. raise ExtractorError('Unable to extract media URL')
  168. mobj = re.search(
  169. r'"mediaURL":"(?P<mediaURL>http.*?)",(.*?)"key":"(?P<key>.*?)"', vardict['mediaData'][0])
  170. if mobj is None:
  171. raise ExtractorError('Unable to extract media URL')
  172. mediaURL = mobj.group('mediaURL').replace('\\/', '/')
  173. video_url = '%s?__gda__=%s' % (mediaURL, mobj.group('key'))
  174. video_ext = determine_ext(video_url)
  175. if video_url is None:
  176. player_url = self._search_regex(
  177. r"swfobject\.embedSWF\('([^']+)'",
  178. webpage, 'config URL', default=None)
  179. if player_url:
  180. config_url = self._search_regex(
  181. r'config=(.+)$', player_url, 'config URL')
  182. config_doc = self._download_xml(
  183. config_url, video_id,
  184. note='Downloading video config')
  185. smil_url = config_doc.find('.//properties').attrib['smil_file']
  186. smil_doc = self._download_xml(
  187. smil_url, video_id,
  188. note='Downloading SMIL document')
  189. base_url = smil_doc.find('./head/meta').attrib['base']
  190. video_url = []
  191. for vn in smil_doc.findall('.//video'):
  192. br = int(vn.attrib['system-bitrate'])
  193. play_path = vn.attrib['src']
  194. video_url.append({
  195. 'format_id': 'smil-%d' % br,
  196. 'url': base_url,
  197. 'play_path': play_path,
  198. 'page_url': url,
  199. 'player_url': player_url,
  200. 'ext': play_path.partition(':')[0],
  201. })
  202. if video_url is None:
  203. raise ExtractorError('Unsupported video type')
  204. video_title = self._html_search_regex(
  205. r'(?im)<title>(.*) - Video</title>', webpage, 'title')
  206. description = self._og_search_description(webpage)
  207. thumbnail = self._og_search_thumbnail(webpage)
  208. video_uploader = self._html_search_regex(
  209. r'submitter=(.*?);|googletag\.pubads\(\)\.setTargeting\("(?:channel|submiter)","([^"]+)"\);',
  210. webpage, 'uploader nickname', fatal=False)
  211. duration = int_or_none(
  212. self._html_search_meta('video:duration', webpage))
  213. age_limit = (
  214. 18
  215. if re.search(r'"contentRating":"restricted"', webpage)
  216. else 0)
  217. if isinstance(video_url, list):
  218. formats = video_url
  219. else:
  220. formats = [{
  221. 'url': video_url,
  222. 'ext': video_ext,
  223. }]
  224. self._sort_formats(formats)
  225. return {
  226. 'id': video_id,
  227. 'description': description,
  228. 'uploader': video_uploader,
  229. 'title': video_title,
  230. 'thumbnail': thumbnail,
  231. 'age_limit': age_limit,
  232. 'formats': formats,
  233. 'duration': duration,
  234. }