You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

354 lines
12 KiB

11 years ago
Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
11 years ago
11 years ago
11 years ago
10 years ago
  1. # coding: utf-8
  2. from __future__ import unicode_literals
  3. import json
  4. import time
  5. import hmac
  6. import hashlib
  7. import itertools
  8. from .common import InfoExtractor
  9. from ..utils import (
  10. ExtractorError,
  11. int_or_none,
  12. parse_age_limit,
  13. parse_iso8601,
  14. sanitized_Request,
  15. )
  16. class VikiBaseIE(InfoExtractor):
  17. _VALID_URL_BASE = r'https?://(?:www\.)?viki\.(?:com|net|mx|jp|fr)/'
  18. _API_QUERY_TEMPLATE = '/v4/%sapp=%s&t=%s&site=www.viki.com'
  19. _API_URL_TEMPLATE = 'http://api.viki.io%s&sig=%s'
  20. _APP = '65535a'
  21. _APP_VERSION = '2.2.5.1428709186'
  22. _APP_SECRET = '-$iJ}@p7!G@SyU/je1bEyWg}upLu-6V6-Lg9VD(]siH,r.,m-r|ulZ,U4LC/SeR)'
  23. _NETRC_MACHINE = 'viki'
  24. _token = None
  25. _ERRORS = {
  26. 'geo': 'Sorry, this content is not available in your region.',
  27. 'upcoming': 'Sorry, this content is not yet available.',
  28. # 'paywall': 'paywall',
  29. }
  30. def _prepare_call(self, path, timestamp=None, post_data=None):
  31. path += '?' if '?' not in path else '&'
  32. if not timestamp:
  33. timestamp = int(time.time())
  34. query = self._API_QUERY_TEMPLATE % (path, self._APP, timestamp)
  35. if self._token:
  36. query += '&token=%s' % self._token
  37. sig = hmac.new(
  38. self._APP_SECRET.encode('ascii'),
  39. query.encode('ascii'),
  40. hashlib.sha1
  41. ).hexdigest()
  42. url = self._API_URL_TEMPLATE % (query, sig)
  43. return sanitized_Request(
  44. url, json.dumps(post_data).encode('utf-8')) if post_data else url
  45. def _call_api(self, path, video_id, note, timestamp=None, post_data=None):
  46. resp = self._download_json(
  47. self._prepare_call(path, timestamp, post_data), video_id, note)
  48. error = resp.get('error')
  49. if error:
  50. if error == 'invalid timestamp':
  51. resp = self._download_json(
  52. self._prepare_call(path, int(resp['current_timestamp']), post_data),
  53. video_id, '%s (retry)' % note)
  54. error = resp.get('error')
  55. if error:
  56. self._raise_error(resp['error'])
  57. return resp
  58. def _raise_error(self, error):
  59. raise ExtractorError(
  60. '%s returned error: %s' % (self.IE_NAME, error),
  61. expected=True)
  62. def _check_errors(self, data):
  63. for reason, status in data.get('blocking', {}).items():
  64. if status and reason in self._ERRORS:
  65. raise ExtractorError('%s said: %s' % (
  66. self.IE_NAME, self._ERRORS[reason]), expected=True)
  67. def _real_initialize(self):
  68. self._login()
  69. def _login(self):
  70. (username, password) = self._get_login_info()
  71. if username is None:
  72. return
  73. login_form = {
  74. 'login_id': username,
  75. 'password': password,
  76. }
  77. login = self._call_api(
  78. 'sessions.json', None,
  79. 'Logging in as %s' % username, post_data=login_form)
  80. self._token = login.get('token')
  81. if not self._token:
  82. self.report_warning('Unable to get session token, login has probably failed')
  83. @staticmethod
  84. def dict_selection(dict_obj, preferred_key):
  85. if preferred_key in dict_obj:
  86. return dict_obj.get(preferred_key)
  87. filtered_dict = list(filter(None, [dict_obj.get(k) for k in dict_obj.keys()]))
  88. return filtered_dict[0] if filtered_dict else None
  89. class VikiIE(VikiBaseIE):
  90. IE_NAME = 'viki'
  91. _VALID_URL = r'%s(?:videos|player)/(?P<id>[0-9]+v)' % VikiBaseIE._VALID_URL_BASE
  92. _TESTS = [{
  93. 'url': 'http://www.viki.com/videos/1023585v-heirs-episode-14',
  94. 'info_dict': {
  95. 'id': '1023585v',
  96. 'ext': 'mp4',
  97. 'title': 'Heirs Episode 14',
  98. 'uploader': 'SBS',
  99. 'description': 'md5:c4b17b9626dd4b143dcc4d855ba3474e',
  100. 'upload_date': '20131121',
  101. 'age_limit': 13,
  102. },
  103. 'skip': 'Blocked in the US',
  104. }, {
  105. # clip
  106. 'url': 'http://www.viki.com/videos/1067139v-the-avengers-age-of-ultron-press-conference',
  107. 'md5': '86c0b5dbd4d83a6611a79987cc7a1989',
  108. 'info_dict': {
  109. 'id': '1067139v',
  110. 'ext': 'mp4',
  111. 'title': "'The Avengers: Age of Ultron' Press Conference",
  112. 'description': 'md5:d70b2f9428f5488321bfe1db10d612ea',
  113. 'duration': 352,
  114. 'timestamp': 1430380829,
  115. 'upload_date': '20150430',
  116. 'uploader': 'Arirang TV',
  117. 'like_count': int,
  118. 'age_limit': 0,
  119. }
  120. }, {
  121. 'url': 'http://www.viki.com/videos/1048879v-ankhon-dekhi',
  122. 'info_dict': {
  123. 'id': '1048879v',
  124. 'ext': 'mp4',
  125. 'title': 'Ankhon Dekhi',
  126. 'duration': 6512,
  127. 'timestamp': 1408532356,
  128. 'upload_date': '20140820',
  129. 'uploader': 'Spuul',
  130. 'like_count': int,
  131. 'age_limit': 13,
  132. },
  133. 'params': {
  134. # m3u8 download
  135. 'skip_download': True,
  136. }
  137. }, {
  138. # episode
  139. 'url': 'http://www.viki.com/videos/44699v-boys-over-flowers-episode-1',
  140. 'md5': '190f3ef426005ba3a080a63325955bc3',
  141. 'info_dict': {
  142. 'id': '44699v',
  143. 'ext': 'mp4',
  144. 'title': 'Boys Over Flowers - Episode 1',
  145. 'description': 'md5:52617e4f729c7d03bfd4bcbbb6e946f2',
  146. 'duration': 4155,
  147. 'timestamp': 1270496524,
  148. 'upload_date': '20100405',
  149. 'uploader': 'group8',
  150. 'like_count': int,
  151. 'age_limit': 13,
  152. }
  153. }, {
  154. # youtube external
  155. 'url': 'http://www.viki.com/videos/50562v-poor-nastya-complete-episode-1',
  156. 'md5': '216d1afdc0c64d1febc1e9f2bd4b864b',
  157. 'info_dict': {
  158. 'id': '50562v',
  159. 'ext': 'mp4',
  160. 'title': 'Poor Nastya [COMPLETE] - Episode 1',
  161. 'description': '',
  162. 'duration': 607,
  163. 'timestamp': 1274949505,
  164. 'upload_date': '20101213',
  165. 'uploader': 'ad14065n',
  166. 'uploader_id': 'ad14065n',
  167. 'like_count': int,
  168. 'age_limit': 13,
  169. }
  170. }, {
  171. 'url': 'http://www.viki.com/player/44699v',
  172. 'only_matching': True,
  173. }, {
  174. # non-English description
  175. 'url': 'http://www.viki.com/videos/158036v-love-in-magic',
  176. 'md5': '1713ae35df5a521b31f6dc40730e7c9c',
  177. 'info_dict': {
  178. 'id': '158036v',
  179. 'ext': 'mp4',
  180. 'uploader': 'I Planet Entertainment',
  181. 'upload_date': '20111122',
  182. 'timestamp': 1321985454,
  183. 'description': 'md5:44b1e46619df3a072294645c770cef36',
  184. 'title': 'Love In Magic',
  185. 'age_limit': 13,
  186. },
  187. }]
  188. def _real_extract(self, url):
  189. video_id = self._match_id(url)
  190. video = self._call_api(
  191. 'videos/%s.json' % video_id, video_id, 'Downloading video JSON')
  192. self._check_errors(video)
  193. title = self.dict_selection(video.get('titles', {}), 'en')
  194. if not title:
  195. title = 'Episode %d' % video.get('number') if video.get('type') == 'episode' else video.get('id') or video_id
  196. container_titles = video.get('container', {}).get('titles', {})
  197. container_title = self.dict_selection(container_titles, 'en')
  198. title = '%s - %s' % (container_title, title)
  199. description = self.dict_selection(video.get('descriptions', {}), 'en')
  200. duration = int_or_none(video.get('duration'))
  201. timestamp = parse_iso8601(video.get('created_at'))
  202. uploader = video.get('author')
  203. like_count = int_or_none(video.get('likes', {}).get('count'))
  204. age_limit = parse_age_limit(video.get('rating'))
  205. thumbnails = []
  206. for thumbnail_id, thumbnail in video.get('images', {}).items():
  207. thumbnails.append({
  208. 'id': thumbnail_id,
  209. 'url': thumbnail.get('url'),
  210. })
  211. subtitles = {}
  212. for subtitle_lang, _ in video.get('subtitle_completions', {}).items():
  213. subtitles[subtitle_lang] = [{
  214. 'ext': subtitles_format,
  215. 'url': self._prepare_call(
  216. 'videos/%s/subtitles/%s.%s' % (video_id, subtitle_lang, subtitles_format)),
  217. } for subtitles_format in ('srt', 'vtt')]
  218. result = {
  219. 'id': video_id,
  220. 'title': title,
  221. 'description': description,
  222. 'duration': duration,
  223. 'timestamp': timestamp,
  224. 'uploader': uploader,
  225. 'like_count': like_count,
  226. 'age_limit': age_limit,
  227. 'thumbnails': thumbnails,
  228. 'subtitles': subtitles,
  229. }
  230. streams = self._call_api(
  231. 'videos/%s/streams.json' % video_id, video_id,
  232. 'Downloading video streams JSON')
  233. if 'external' in streams:
  234. result.update({
  235. '_type': 'url_transparent',
  236. 'url': streams['external']['url'],
  237. })
  238. return result
  239. formats = []
  240. for format_id, stream_dict in streams.items():
  241. height = int_or_none(self._search_regex(
  242. r'^(\d+)[pP]$', format_id, 'height', default=None))
  243. for protocol, format_dict in stream_dict.items():
  244. if format_id == 'm3u8':
  245. formats.extend(self._extract_m3u8_formats(
  246. format_dict['url'], video_id, 'mp4', 'm3u8_native',
  247. m3u8_id='m3u8-%s' % protocol, fatal=False))
  248. else:
  249. formats.append({
  250. 'url': format_dict['url'],
  251. 'format_id': '%s-%s' % (format_id, protocol),
  252. 'height': height,
  253. })
  254. self._sort_formats(formats)
  255. result['formats'] = formats
  256. return result
  257. class VikiChannelIE(VikiBaseIE):
  258. IE_NAME = 'viki:channel'
  259. _VALID_URL = r'%s(?:tv|news|movies|artists)/(?P<id>[0-9]+c)' % VikiBaseIE._VALID_URL_BASE
  260. _TESTS = [{
  261. 'url': 'http://www.viki.com/tv/50c-boys-over-flowers',
  262. 'info_dict': {
  263. 'id': '50c',
  264. 'title': 'Boys Over Flowers',
  265. 'description': 'md5:ecd3cff47967fe193cff37c0bec52790',
  266. },
  267. 'playlist_count': 70,
  268. }, {
  269. 'url': 'http://www.viki.com/tv/1354c-poor-nastya-complete',
  270. 'info_dict': {
  271. 'id': '1354c',
  272. 'title': 'Poor Nastya [COMPLETE]',
  273. 'description': 'md5:05bf5471385aa8b21c18ad450e350525',
  274. },
  275. 'playlist_count': 127,
  276. }, {
  277. 'url': 'http://www.viki.com/news/24569c-showbiz-korea',
  278. 'only_matching': True,
  279. }, {
  280. 'url': 'http://www.viki.com/movies/22047c-pride-and-prejudice-2005',
  281. 'only_matching': True,
  282. }, {
  283. 'url': 'http://www.viki.com/artists/2141c-shinee',
  284. 'only_matching': True,
  285. }]
  286. _PER_PAGE = 25
  287. def _real_extract(self, url):
  288. channel_id = self._match_id(url)
  289. channel = self._call_api(
  290. 'containers/%s.json' % channel_id, channel_id,
  291. 'Downloading channel JSON')
  292. self._check_errors(channel)
  293. title = self.dict_selection(channel['titles'], 'en')
  294. description = self.dict_selection(channel['descriptions'], 'en')
  295. entries = []
  296. for video_type in ('episodes', 'clips', 'movies'):
  297. for page_num in itertools.count(1):
  298. page = self._call_api(
  299. 'containers/%s/%s.json?per_page=%d&sort=number&direction=asc&with_paging=true&page=%d'
  300. % (channel_id, video_type, self._PER_PAGE, page_num), channel_id,
  301. 'Downloading %s JSON page #%d' % (video_type, page_num))
  302. for video in page['response']:
  303. video_id = video['id']
  304. entries.append(self.url_result(
  305. 'http://www.viki.com/videos/%s' % video_id, 'Viki'))
  306. if not page['pagination']['next']:
  307. break
  308. return self.playlist_result(entries, channel_id, title, description)