You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

423 lines
16 KiB

10 years ago
Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
10 years ago
Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
10 years ago
  1. # coding: utf-8
  2. from __future__ import unicode_literals
  3. import re
  4. import json
  5. import itertools
  6. from .common import InfoExtractor
  7. from ..utils import (
  8. determine_ext,
  9. error_to_compat_str,
  10. ExtractorError,
  11. int_or_none,
  12. parse_iso8601,
  13. sanitized_Request,
  14. str_to_int,
  15. unescapeHTML,
  16. )
  17. class DailymotionBaseInfoExtractor(InfoExtractor):
  18. @staticmethod
  19. def _build_request(url):
  20. """Build a request with the family filter disabled"""
  21. request = sanitized_Request(url)
  22. request.add_header('Cookie', 'family_filter=off; ff=off')
  23. return request
  24. def _download_webpage_handle_no_ff(self, url, *args, **kwargs):
  25. request = self._build_request(url)
  26. return self._download_webpage_handle(request, *args, **kwargs)
  27. def _download_webpage_no_ff(self, url, *args, **kwargs):
  28. request = self._build_request(url)
  29. return self._download_webpage(request, *args, **kwargs)
  30. class DailymotionIE(DailymotionBaseInfoExtractor):
  31. _VALID_URL = r'(?i)(?:https?://)?(?:(www|touch)\.)?dailymotion\.[a-z]{2,3}/(?:(embed|#)/)?video/(?P<id>[^/?_]+)'
  32. IE_NAME = 'dailymotion'
  33. _FORMATS = [
  34. ('stream_h264_ld_url', 'ld'),
  35. ('stream_h264_url', 'standard'),
  36. ('stream_h264_hq_url', 'hq'),
  37. ('stream_h264_hd_url', 'hd'),
  38. ('stream_h264_hd1080_url', 'hd180'),
  39. ]
  40. _TESTS = [
  41. {
  42. 'url': 'https://www.dailymotion.com/video/x2iuewm_steam-machine-models-pricing-listed-on-steam-store-ign-news_videogames',
  43. 'md5': '2137c41a8e78554bb09225b8eb322406',
  44. 'info_dict': {
  45. 'id': 'x2iuewm',
  46. 'ext': 'mp4',
  47. 'title': 'Steam Machine Models, Pricing Listed on Steam Store - IGN News',
  48. 'description': 'Several come bundled with the Steam Controller.',
  49. 'thumbnail': 're:^https?:.*\.(?:jpg|png)$',
  50. 'duration': 74,
  51. 'timestamp': 1425657362,
  52. 'upload_date': '20150306',
  53. 'uploader': 'IGN',
  54. 'uploader_id': 'xijv66',
  55. 'age_limit': 0,
  56. 'view_count': int,
  57. 'comment_count': int,
  58. }
  59. },
  60. # Vevo video
  61. {
  62. 'url': 'http://www.dailymotion.com/video/x149uew_katy-perry-roar-official_musi',
  63. 'info_dict': {
  64. 'title': 'Roar (Official)',
  65. 'id': 'USUV71301934',
  66. 'ext': 'mp4',
  67. 'uploader': 'Katy Perry',
  68. 'upload_date': '20130905',
  69. },
  70. 'params': {
  71. 'skip_download': True,
  72. },
  73. 'skip': 'VEVO is only available in some countries',
  74. },
  75. # age-restricted video
  76. {
  77. 'url': 'http://www.dailymotion.com/video/xyh2zz_leanna-decker-cyber-girl-of-the-year-desires-nude-playboy-plus_redband',
  78. 'md5': '0d667a7b9cebecc3c89ee93099c4159d',
  79. 'info_dict': {
  80. 'id': 'xyh2zz',
  81. 'ext': 'mp4',
  82. 'title': 'Leanna Decker - Cyber Girl Of The Year Desires Nude [Playboy Plus]',
  83. 'uploader': 'HotWaves1012',
  84. 'age_limit': 18,
  85. }
  86. },
  87. # geo-restricted, player v5
  88. {
  89. 'url': 'http://www.dailymotion.com/video/xhza0o',
  90. 'only_matching': True,
  91. },
  92. # with subtitles
  93. {
  94. 'url': 'http://www.dailymotion.com/video/x20su5f_the-power-of-nightmares-1-the-rise-of-the-politics-of-fear-bbc-2004_news',
  95. 'only_matching': True,
  96. }
  97. ]
  98. def _real_extract(self, url):
  99. video_id = self._match_id(url)
  100. webpage = self._download_webpage_no_ff(
  101. 'https://www.dailymotion.com/video/%s' % video_id, video_id)
  102. age_limit = self._rta_search(webpage)
  103. description = self._og_search_description(webpage) or self._html_search_meta(
  104. 'description', webpage, 'description')
  105. view_count = str_to_int(self._search_regex(
  106. [r'<meta[^>]+itemprop="interactionCount"[^>]+content="UserPlays:(\d+)"',
  107. r'video_views_count[^>]+>\s+([\d\.,]+)'],
  108. webpage, 'view count', fatal=False))
  109. comment_count = int_or_none(self._search_regex(
  110. r'<meta[^>]+itemprop="interactionCount"[^>]+content="UserComments:(\d+)"',
  111. webpage, 'comment count', fatal=False))
  112. player_v5 = self._search_regex(
  113. [r'buildPlayer\(({.+?})\);\n', # See https://github.com/rg3/youtube-dl/issues/7826
  114. r'playerV5\s*=\s*dmp\.create\([^,]+?,\s*({.+?})\);',
  115. r'buildPlayer\(({.+?})\);'],
  116. webpage, 'player v5', default=None)
  117. if player_v5:
  118. player = self._parse_json(player_v5, video_id)
  119. metadata = player['metadata']
  120. self._check_error(metadata)
  121. formats = []
  122. for quality, media_list in metadata['qualities'].items():
  123. for media in media_list:
  124. media_url = media.get('url')
  125. if not media_url:
  126. continue
  127. type_ = media.get('type')
  128. if type_ == 'application/vnd.lumberjack.manifest':
  129. continue
  130. ext = determine_ext(media_url)
  131. if type_ == 'application/x-mpegURL' or ext == 'm3u8':
  132. formats.extend(self._extract_m3u8_formats(
  133. media_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
  134. elif type_ == 'application/f4m' or ext == 'f4m':
  135. formats.extend(self._extract_f4m_formats(
  136. media_url, video_id, preference=-1, f4m_id='hds', fatal=False))
  137. else:
  138. f = {
  139. 'url': media_url,
  140. 'format_id': quality,
  141. }
  142. m = re.search(r'H264-(?P<width>\d+)x(?P<height>\d+)', media_url)
  143. if m:
  144. f.update({
  145. 'width': int(m.group('width')),
  146. 'height': int(m.group('height')),
  147. })
  148. formats.append(f)
  149. self._sort_formats(formats)
  150. title = metadata['title']
  151. duration = int_or_none(metadata.get('duration'))
  152. timestamp = int_or_none(metadata.get('created_time'))
  153. thumbnail = metadata.get('poster_url')
  154. uploader = metadata.get('owner', {}).get('screenname')
  155. uploader_id = metadata.get('owner', {}).get('id')
  156. subtitles = {}
  157. subtitles_data = metadata.get('subtitles', {}).get('data', {})
  158. if subtitles_data and isinstance(subtitles_data, dict):
  159. for subtitle_lang, subtitle in subtitles_data.items():
  160. subtitles[subtitle_lang] = [{
  161. 'ext': determine_ext(subtitle_url),
  162. 'url': subtitle_url,
  163. } for subtitle_url in subtitle.get('urls', [])]
  164. return {
  165. 'id': video_id,
  166. 'title': title,
  167. 'description': description,
  168. 'thumbnail': thumbnail,
  169. 'duration': duration,
  170. 'timestamp': timestamp,
  171. 'uploader': uploader,
  172. 'uploader_id': uploader_id,
  173. 'age_limit': age_limit,
  174. 'view_count': view_count,
  175. 'comment_count': comment_count,
  176. 'formats': formats,
  177. 'subtitles': subtitles,
  178. }
  179. # vevo embed
  180. vevo_id = self._search_regex(
  181. r'<link rel="video_src" href="[^"]*?vevo.com[^"]*?video=(?P<id>[\w]*)',
  182. webpage, 'vevo embed', default=None)
  183. if vevo_id:
  184. return self.url_result('vevo:%s' % vevo_id, 'Vevo')
  185. # fallback old player
  186. embed_page = self._download_webpage_no_ff(
  187. 'https://www.dailymotion.com/embed/video/%s' % video_id,
  188. video_id, 'Downloading embed page')
  189. timestamp = parse_iso8601(self._html_search_meta(
  190. 'video:release_date', webpage, 'upload date'))
  191. info = self._parse_json(
  192. self._search_regex(
  193. r'var info = ({.*?}),$', embed_page,
  194. 'video info', flags=re.MULTILINE),
  195. video_id)
  196. self._check_error(info)
  197. formats = []
  198. for (key, format_id) in self._FORMATS:
  199. video_url = info.get(key)
  200. if video_url is not None:
  201. m_size = re.search(r'H264-(\d+)x(\d+)', video_url)
  202. if m_size is not None:
  203. width, height = map(int_or_none, (m_size.group(1), m_size.group(2)))
  204. else:
  205. width, height = None, None
  206. formats.append({
  207. 'url': video_url,
  208. 'ext': 'mp4',
  209. 'format_id': format_id,
  210. 'width': width,
  211. 'height': height,
  212. })
  213. self._sort_formats(formats)
  214. # subtitles
  215. video_subtitles = self.extract_subtitles(video_id, webpage)
  216. title = self._og_search_title(webpage, default=None)
  217. if title is None:
  218. title = self._html_search_regex(
  219. r'(?s)<span\s+id="video_title"[^>]*>(.*?)</span>', webpage,
  220. 'title')
  221. return {
  222. 'id': video_id,
  223. 'formats': formats,
  224. 'uploader': info['owner.screenname'],
  225. 'timestamp': timestamp,
  226. 'title': title,
  227. 'description': description,
  228. 'subtitles': video_subtitles,
  229. 'thumbnail': info['thumbnail_url'],
  230. 'age_limit': age_limit,
  231. 'view_count': view_count,
  232. 'duration': info['duration']
  233. }
  234. def _check_error(self, info):
  235. if info.get('error') is not None:
  236. raise ExtractorError(
  237. '%s said: %s' % (self.IE_NAME, info['error']['title']), expected=True)
  238. def _get_subtitles(self, video_id, webpage):
  239. try:
  240. sub_list = self._download_webpage(
  241. 'https://api.dailymotion.com/video/%s/subtitles?fields=id,language,url' % video_id,
  242. video_id, note=False)
  243. except ExtractorError as err:
  244. self._downloader.report_warning('unable to download video subtitles: %s' % error_to_compat_str(err))
  245. return {}
  246. info = json.loads(sub_list)
  247. if (info['total'] > 0):
  248. sub_lang_list = dict((l['language'], [{'url': l['url'], 'ext': 'srt'}]) for l in info['list'])
  249. return sub_lang_list
  250. self._downloader.report_warning('video doesn\'t have subtitles')
  251. return {}
  252. class DailymotionPlaylistIE(DailymotionBaseInfoExtractor):
  253. IE_NAME = 'dailymotion:playlist'
  254. _VALID_URL = r'(?:https?://)?(?:www\.)?dailymotion\.[a-z]{2,3}/playlist/(?P<id>.+?)/'
  255. _MORE_PAGES_INDICATOR = r'(?s)<div class="pages[^"]*">.*?<a\s+class="[^"]*?icon-arrow_right[^"]*?"'
  256. _PAGE_TEMPLATE = 'https://www.dailymotion.com/playlist/%s/%s'
  257. _TESTS = [{
  258. 'url': 'http://www.dailymotion.com/playlist/xv4bw_nqtv_sport/1#video=xl8v3q',
  259. 'info_dict': {
  260. 'title': 'SPORT',
  261. 'id': 'xv4bw_nqtv_sport',
  262. },
  263. 'playlist_mincount': 20,
  264. }]
  265. def _extract_entries(self, id):
  266. video_ids = set()
  267. processed_urls = set()
  268. for pagenum in itertools.count(1):
  269. page_url = self._PAGE_TEMPLATE % (id, pagenum)
  270. webpage, urlh = self._download_webpage_handle_no_ff(
  271. page_url, id, 'Downloading page %s' % pagenum)
  272. if urlh.geturl() in processed_urls:
  273. self.report_warning('Stopped at duplicated page %s, which is the same as %s' % (
  274. page_url, urlh.geturl()), id)
  275. break
  276. processed_urls.add(urlh.geturl())
  277. for video_id in re.findall(r'data-xid="(.+?)"', webpage):
  278. if video_id not in video_ids:
  279. yield self.url_result('http://www.dailymotion.com/video/%s' % video_id, 'Dailymotion')
  280. video_ids.add(video_id)
  281. if re.search(self._MORE_PAGES_INDICATOR, webpage) is None:
  282. break
  283. def _real_extract(self, url):
  284. mobj = re.match(self._VALID_URL, url)
  285. playlist_id = mobj.group('id')
  286. webpage = self._download_webpage(url, playlist_id)
  287. return {
  288. '_type': 'playlist',
  289. 'id': playlist_id,
  290. 'title': self._og_search_title(webpage),
  291. 'entries': self._extract_entries(playlist_id),
  292. }
  293. class DailymotionUserIE(DailymotionPlaylistIE):
  294. IE_NAME = 'dailymotion:user'
  295. _VALID_URL = r'https?://(?:www\.)?dailymotion\.[a-z]{2,3}/(?!(?:embed|#|video|playlist)/)(?:(?:old/)?user/)?(?P<user>[^/]+)'
  296. _PAGE_TEMPLATE = 'http://www.dailymotion.com/user/%s/%s'
  297. _TESTS = [{
  298. 'url': 'https://www.dailymotion.com/user/nqtv',
  299. 'info_dict': {
  300. 'id': 'nqtv',
  301. 'title': 'Rémi Gaillard',
  302. },
  303. 'playlist_mincount': 100,
  304. }, {
  305. 'url': 'http://www.dailymotion.com/user/UnderProject',
  306. 'info_dict': {
  307. 'id': 'UnderProject',
  308. 'title': 'UnderProject',
  309. },
  310. 'playlist_mincount': 1800,
  311. 'expected_warnings': [
  312. 'Stopped at duplicated page',
  313. ],
  314. 'skip': 'Takes too long time',
  315. }]
  316. def _real_extract(self, url):
  317. mobj = re.match(self._VALID_URL, url)
  318. user = mobj.group('user')
  319. webpage = self._download_webpage(
  320. 'https://www.dailymotion.com/user/%s' % user, user)
  321. full_user = unescapeHTML(self._html_search_regex(
  322. r'<a class="nav-image" title="([^"]+)" href="/%s">' % re.escape(user),
  323. webpage, 'user'))
  324. return {
  325. '_type': 'playlist',
  326. 'id': user,
  327. 'title': full_user,
  328. 'entries': self._extract_entries(user),
  329. }
  330. class DailymotionCloudIE(DailymotionBaseInfoExtractor):
  331. _VALID_URL_PREFIX = r'http://api\.dmcloud\.net/(?:player/)?embed/'
  332. _VALID_URL = r'%s[^/]+/(?P<id>[^/?]+)' % _VALID_URL_PREFIX
  333. _VALID_EMBED_URL = r'%s[^/]+/[^\'"]+' % _VALID_URL_PREFIX
  334. _TESTS = [{
  335. # From http://www.francetvinfo.fr/economie/entreprises/les-entreprises-familiales-le-secret-de-la-reussite_933271.html
  336. # Tested at FranceTvInfo_2
  337. 'url': 'http://api.dmcloud.net/embed/4e7343f894a6f677b10006b4/556e03339473995ee145930c?auth=1464865870-0-jyhsm84b-ead4c701fb750cf9367bf4447167a3db&autoplay=1',
  338. 'only_matching': True,
  339. }, {
  340. # http://www.francetvinfo.fr/societe/larguez-les-amarres-le-cobaturage-se-developpe_980101.html
  341. 'url': 'http://api.dmcloud.net/player/embed/4e7343f894a6f677b10006b4/559545469473996d31429f06?auth=1467430263-0-90tglw2l-a3a4b64ed41efe48d7fccad85b8b8fda&autoplay=1',
  342. 'only_matching': True,
  343. }]
  344. @classmethod
  345. def _extract_dmcloud_url(self, webpage):
  346. mobj = re.search(r'<iframe[^>]+src=[\'"](%s)[\'"]' % self._VALID_EMBED_URL, webpage)
  347. if mobj:
  348. return mobj.group(1)
  349. mobj = re.search(
  350. r'<input[^>]+id=[\'"]dmcloudUrlEmissionSelect[\'"][^>]+value=[\'"](%s)[\'"]' % self._VALID_EMBED_URL,
  351. webpage)
  352. if mobj:
  353. return mobj.group(1)
  354. def _real_extract(self, url):
  355. video_id = self._match_id(url)
  356. webpage = self._download_webpage_no_ff(url, video_id)
  357. title = self._html_search_regex(r'<title>([^>]+)</title>', webpage, 'title')
  358. video_info = self._parse_json(self._search_regex(
  359. r'var\s+info\s*=\s*([^;]+);', webpage, 'video info'), video_id)
  360. # TODO: parse ios_url, which is in fact a manifest
  361. video_url = video_info['mp4_url']
  362. return {
  363. 'id': video_id,
  364. 'url': video_url,
  365. 'title': title,
  366. 'thumbnail': video_info.get('thumbnail_url'),
  367. }