You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

586 lines
24 KiB

Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
10 years ago
10 years ago
10 years ago
Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
  1. # coding: utf-8
  2. from __future__ import unicode_literals
  3. import re
  4. import json
  5. import base64
  6. import zlib
  7. from hashlib import sha1
  8. from math import pow, sqrt, floor
  9. from .common import InfoExtractor
  10. from ..compat import (
  11. compat_etree_fromstring,
  12. compat_urllib_parse_urlencode,
  13. compat_urllib_request,
  14. compat_urlparse,
  15. )
  16. from ..utils import (
  17. ExtractorError,
  18. bytes_to_intlist,
  19. intlist_to_bytes,
  20. int_or_none,
  21. lowercase_escape,
  22. remove_end,
  23. sanitized_Request,
  24. unified_strdate,
  25. urlencode_postdata,
  26. xpath_text,
  27. extract_attributes,
  28. )
  29. from ..aes import (
  30. aes_cbc_decrypt,
  31. )
  32. class CrunchyrollBaseIE(InfoExtractor):
  33. _LOGIN_URL = 'https://www.crunchyroll.com/login'
  34. _LOGIN_FORM = 'login_form'
  35. _NETRC_MACHINE = 'crunchyroll'
  36. def _login(self):
  37. (username, password) = self._get_login_info()
  38. if username is None:
  39. return
  40. login_page = self._download_webpage(
  41. self._LOGIN_URL, None, 'Downloading login page')
  42. def is_logged(webpage):
  43. return '<title>Redirecting' in webpage
  44. # Already logged in
  45. if is_logged(login_page):
  46. return
  47. login_form_str = self._search_regex(
  48. r'(?P<form><form[^>]+?id=(["\'])%s\2[^>]*>)' % self._LOGIN_FORM,
  49. login_page, 'login form', group='form')
  50. post_url = extract_attributes(login_form_str).get('action')
  51. if not post_url:
  52. post_url = self._LOGIN_URL
  53. elif not post_url.startswith('http'):
  54. post_url = compat_urlparse.urljoin(self._LOGIN_URL, post_url)
  55. login_form = self._form_hidden_inputs(self._LOGIN_FORM, login_page)
  56. login_form.update({
  57. 'login_form[name]': username,
  58. 'login_form[password]': password,
  59. })
  60. response = self._download_webpage(
  61. post_url, None, 'Logging in', 'Wrong login info',
  62. data=urlencode_postdata(login_form),
  63. headers={'Content-Type': 'application/x-www-form-urlencoded'})
  64. # Successful login
  65. if is_logged(response):
  66. return
  67. error = self._html_search_regex(
  68. '(?s)<ul[^>]+class=["\']messages["\'][^>]*>(.+?)</ul>',
  69. response, 'error message', default=None)
  70. if error:
  71. raise ExtractorError('Unable to login: %s' % error, expected=True)
  72. raise ExtractorError('Unable to log in')
  73. def _real_initialize(self):
  74. self._login()
  75. def _download_webpage(self, url_or_request, *args, **kwargs):
  76. request = (url_or_request if isinstance(url_or_request, compat_urllib_request.Request)
  77. else sanitized_Request(url_or_request))
  78. # Accept-Language must be set explicitly to accept any language to avoid issues
  79. # similar to https://github.com/rg3/youtube-dl/issues/6797.
  80. # Along with IP address Crunchyroll uses Accept-Language to guess whether georestriction
  81. # should be imposed or not (from what I can see it just takes the first language
  82. # ignoring the priority and requires it to correspond the IP). By the way this causes
  83. # Crunchyroll to not work in georestriction cases in some browsers that don't place
  84. # the locale lang first in header. However allowing any language seems to workaround the issue.
  85. request.add_header('Accept-Language', '*')
  86. return super(CrunchyrollBaseIE, self)._download_webpage(request, *args, **kwargs)
  87. @staticmethod
  88. def _add_skip_wall(url):
  89. parsed_url = compat_urlparse.urlparse(url)
  90. qs = compat_urlparse.parse_qs(parsed_url.query)
  91. # Always force skip_wall to bypass maturity wall, namely 18+ confirmation message:
  92. # > This content may be inappropriate for some people.
  93. # > Are you sure you want to continue?
  94. # since it's not disabled by default in crunchyroll account's settings.
  95. # See https://github.com/rg3/youtube-dl/issues/7202.
  96. qs['skip_wall'] = ['1']
  97. return compat_urlparse.urlunparse(
  98. parsed_url._replace(query=compat_urllib_parse_urlencode(qs, True)))
  99. class CrunchyrollIE(CrunchyrollBaseIE):
  100. _VALID_URL = r'https?://(?:(?P<prefix>www|m)\.)?(?P<url>crunchyroll\.(?:com|fr)/(?:media(?:-|/\?id=)|[^/]*/[^/?&]*?)(?P<video_id>[0-9]+))(?:[/?&]|$)'
  101. _TESTS = [{
  102. 'url': 'http://www.crunchyroll.com/wanna-be-the-strongest-in-the-world/episode-1-an-idol-wrestler-is-born-645513',
  103. 'info_dict': {
  104. 'id': '645513',
  105. 'ext': 'mp4',
  106. 'title': 'Wanna be the Strongest in the World Episode 1 – An Idol-Wrestler is Born!',
  107. 'description': 'md5:2d17137920c64f2f49981a7797d275ef',
  108. 'thumbnail': 'http://img1.ak.crunchyroll.com/i/spire1-tmb/20c6b5e10f1a47b10516877d3c039cae1380951166_full.jpg',
  109. 'uploader': 'Yomiuri Telecasting Corporation (YTV)',
  110. 'upload_date': '20131013',
  111. 'url': 're:(?!.*&amp)',
  112. },
  113. 'params': {
  114. # rtmp
  115. 'skip_download': True,
  116. },
  117. }, {
  118. 'url': 'http://www.crunchyroll.com/media-589804/culture-japan-1',
  119. 'info_dict': {
  120. 'id': '589804',
  121. 'ext': 'flv',
  122. 'title': 'Culture Japan Episode 1 – Rebuilding Japan after the 3.11',
  123. 'description': 'md5:2fbc01f90b87e8e9137296f37b461c12',
  124. 'thumbnail': r're:^https?://.*\.jpg$',
  125. 'uploader': 'Danny Choo Network',
  126. 'upload_date': '20120213',
  127. },
  128. 'params': {
  129. # rtmp
  130. 'skip_download': True,
  131. },
  132. 'skip': 'Video gone',
  133. }, {
  134. 'url': 'http://www.crunchyroll.com/rezero-starting-life-in-another-world-/episode-5-the-morning-of-our-promise-is-still-distant-702409',
  135. 'info_dict': {
  136. 'id': '702409',
  137. 'ext': 'mp4',
  138. 'title': 'Re:ZERO -Starting Life in Another World- Episode 5 – The Morning of Our Promise Is Still Distant',
  139. 'description': 'md5:97664de1ab24bbf77a9c01918cb7dca9',
  140. 'thumbnail': r're:^https?://.*\.jpg$',
  141. 'uploader': 'TV TOKYO',
  142. 'upload_date': '20160508',
  143. },
  144. 'params': {
  145. # m3u8 download
  146. 'skip_download': True,
  147. },
  148. }, {
  149. 'url': 'http://www.crunchyroll.com/konosuba-gods-blessing-on-this-wonderful-world/episode-1-give-me-deliverance-from-this-judicial-injustice-727589',
  150. 'info_dict': {
  151. 'id': '727589',
  152. 'ext': 'mp4',
  153. 'title': "KONOSUBA -God's blessing on this wonderful world! 2 Episode 1 – Give Me Deliverance from this Judicial Injustice!",
  154. 'description': 'md5:cbcf05e528124b0f3a0a419fc805ea7d',
  155. 'thumbnail': r're:^https?://.*\.jpg$',
  156. 'uploader': 'Kadokawa Pictures Inc.',
  157. 'upload_date': '20170118',
  158. 'series': "KONOSUBA -God's blessing on this wonderful world!",
  159. 'season': "KONOSUBA -God's blessing on this wonderful world! 2",
  160. 'season_number': 2,
  161. 'episode': 'Give Me Deliverance from this Judicial Injustice!',
  162. 'episode_number': 1,
  163. },
  164. 'params': {
  165. # m3u8 download
  166. 'skip_download': True,
  167. },
  168. }, {
  169. 'url': 'http://www.crunchyroll.fr/girl-friend-beta/episode-11-goodbye-la-mode-661697',
  170. 'only_matching': True,
  171. }, {
  172. # geo-restricted (US), 18+ maturity wall, non-premium available
  173. 'url': 'http://www.crunchyroll.com/cosplay-complex-ova/episode-1-the-birth-of-the-cosplay-club-565617',
  174. 'only_matching': True,
  175. }, {
  176. # A description with double quotes
  177. 'url': 'http://www.crunchyroll.com/11eyes/episode-1-piros-jszaka-red-night-535080',
  178. 'info_dict': {
  179. 'id': '535080',
  180. 'ext': 'mp4',
  181. 'title': '11eyes Episode 1 – Piros éjszaka - Red Night',
  182. 'description': 'Kakeru and Yuka are thrown into an alternate nightmarish world they call "Red Night".',
  183. 'uploader': 'Marvelous AQL Inc.',
  184. 'upload_date': '20091021',
  185. },
  186. 'params': {
  187. # Just test metadata extraction
  188. 'skip_download': True,
  189. },
  190. }, {
  191. # make sure we can extract an uploader name that's not a link
  192. 'url': 'http://www.crunchyroll.com/hakuoki-reimeiroku/episode-1-dawn-of-the-divine-warriors-606899',
  193. 'info_dict': {
  194. 'id': '606899',
  195. 'ext': 'mp4',
  196. 'title': 'Hakuoki Reimeiroku Episode 1 – Dawn of the Divine Warriors',
  197. 'description': 'Ryunosuke was left to die, but Serizawa-san asked him a simple question "Do you want to live?"',
  198. 'uploader': 'Geneon Entertainment',
  199. 'upload_date': '20120717',
  200. },
  201. 'params': {
  202. # just test metadata extraction
  203. 'skip_download': True,
  204. },
  205. }, {
  206. # A video with a vastly different season name compared to the series name
  207. 'url': 'http://www.crunchyroll.com/nyarko-san-another-crawling-chaos/episode-1-test-590532',
  208. 'info_dict': {
  209. 'id': '590532',
  210. 'ext': 'mp4',
  211. 'title': 'Haiyoru! Nyaruani (ONA) Episode 1 – Test',
  212. 'description': 'Mahiro and Nyaruko talk about official certification.',
  213. 'uploader': 'TV TOKYO',
  214. 'upload_date': '20120305',
  215. 'series': 'Nyarko-san: Another Crawling Chaos',
  216. 'season': 'Haiyoru! Nyaruani (ONA)',
  217. },
  218. 'params': {
  219. # Just test metadata extraction
  220. 'skip_download': True,
  221. },
  222. }]
  223. _FORMAT_IDS = {
  224. '360': ('60', '106'),
  225. '480': ('61', '106'),
  226. '720': ('62', '106'),
  227. '1080': ('80', '108'),
  228. }
  229. def _decrypt_subtitles(self, data, iv, id):
  230. data = bytes_to_intlist(base64.b64decode(data.encode('utf-8')))
  231. iv = bytes_to_intlist(base64.b64decode(iv.encode('utf-8')))
  232. id = int(id)
  233. def obfuscate_key_aux(count, modulo, start):
  234. output = list(start)
  235. for _ in range(count):
  236. output.append(output[-1] + output[-2])
  237. # cut off start values
  238. output = output[2:]
  239. output = list(map(lambda x: x % modulo + 33, output))
  240. return output
  241. def obfuscate_key(key):
  242. num1 = int(floor(pow(2, 25) * sqrt(6.9)))
  243. num2 = (num1 ^ key) << 5
  244. num3 = key ^ num1
  245. num4 = num3 ^ (num3 >> 3) ^ num2
  246. prefix = intlist_to_bytes(obfuscate_key_aux(20, 97, (1, 2)))
  247. shaHash = bytes_to_intlist(sha1(prefix + str(num4).encode('ascii')).digest())
  248. # Extend 160 Bit hash to 256 Bit
  249. return shaHash + [0] * 12
  250. key = obfuscate_key(id)
  251. decrypted_data = intlist_to_bytes(aes_cbc_decrypt(data, key, iv))
  252. return zlib.decompress(decrypted_data)
  253. def _convert_subtitles_to_srt(self, sub_root):
  254. output = ''
  255. for i, event in enumerate(sub_root.findall('./events/event'), 1):
  256. start = event.attrib['start'].replace('.', ',')
  257. end = event.attrib['end'].replace('.', ',')
  258. text = event.attrib['text'].replace('\\N', '\n')
  259. output += '%d\n%s --> %s\n%s\n\n' % (i, start, end, text)
  260. return output
  261. def _convert_subtitles_to_ass(self, sub_root):
  262. output = ''
  263. def ass_bool(strvalue):
  264. assvalue = '0'
  265. if strvalue == '1':
  266. assvalue = '-1'
  267. return assvalue
  268. output = '[Script Info]\n'
  269. output += 'Title: %s\n' % sub_root.attrib['title']
  270. output += 'ScriptType: v4.00+\n'
  271. output += 'WrapStyle: %s\n' % sub_root.attrib['wrap_style']
  272. output += 'PlayResX: %s\n' % sub_root.attrib['play_res_x']
  273. output += 'PlayResY: %s\n' % sub_root.attrib['play_res_y']
  274. output += """
  275. [V4+ Styles]
  276. Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
  277. """
  278. for style in sub_root.findall('./styles/style'):
  279. output += 'Style: ' + style.attrib['name']
  280. output += ',' + style.attrib['font_name']
  281. output += ',' + style.attrib['font_size']
  282. output += ',' + style.attrib['primary_colour']
  283. output += ',' + style.attrib['secondary_colour']
  284. output += ',' + style.attrib['outline_colour']
  285. output += ',' + style.attrib['back_colour']
  286. output += ',' + ass_bool(style.attrib['bold'])
  287. output += ',' + ass_bool(style.attrib['italic'])
  288. output += ',' + ass_bool(style.attrib['underline'])
  289. output += ',' + ass_bool(style.attrib['strikeout'])
  290. output += ',' + style.attrib['scale_x']
  291. output += ',' + style.attrib['scale_y']
  292. output += ',' + style.attrib['spacing']
  293. output += ',' + style.attrib['angle']
  294. output += ',' + style.attrib['border_style']
  295. output += ',' + style.attrib['outline']
  296. output += ',' + style.attrib['shadow']
  297. output += ',' + style.attrib['alignment']
  298. output += ',' + style.attrib['margin_l']
  299. output += ',' + style.attrib['margin_r']
  300. output += ',' + style.attrib['margin_v']
  301. output += ',' + style.attrib['encoding']
  302. output += '\n'
  303. output += """
  304. [Events]
  305. Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
  306. """
  307. for event in sub_root.findall('./events/event'):
  308. output += 'Dialogue: 0'
  309. output += ',' + event.attrib['start']
  310. output += ',' + event.attrib['end']
  311. output += ',' + event.attrib['style']
  312. output += ',' + event.attrib['name']
  313. output += ',' + event.attrib['margin_l']
  314. output += ',' + event.attrib['margin_r']
  315. output += ',' + event.attrib['margin_v']
  316. output += ',' + event.attrib['effect']
  317. output += ',' + event.attrib['text']
  318. output += '\n'
  319. return output
  320. def _extract_subtitles(self, subtitle):
  321. sub_root = compat_etree_fromstring(subtitle)
  322. return [{
  323. 'ext': 'srt',
  324. 'data': self._convert_subtitles_to_srt(sub_root),
  325. }, {
  326. 'ext': 'ass',
  327. 'data': self._convert_subtitles_to_ass(sub_root),
  328. }]
  329. def _get_subtitles(self, video_id, webpage):
  330. subtitles = {}
  331. for sub_id, sub_name in re.findall(r'\bssid=([0-9]+)"[^>]+?\btitle="([^"]+)', webpage):
  332. sub_page = self._download_webpage(
  333. 'http://www.crunchyroll.com/xml/?req=RpcApiSubtitle_GetXml&subtitle_script_id=' + sub_id,
  334. video_id, note='Downloading subtitles for ' + sub_name)
  335. id = self._search_regex(r'id=\'([0-9]+)', sub_page, 'subtitle_id', fatal=False)
  336. iv = self._search_regex(r'<iv>([^<]+)', sub_page, 'subtitle_iv', fatal=False)
  337. data = self._search_regex(r'<data>([^<]+)', sub_page, 'subtitle_data', fatal=False)
  338. if not id or not iv or not data:
  339. continue
  340. subtitle = self._decrypt_subtitles(data, iv, id).decode('utf-8')
  341. lang_code = self._search_regex(r'lang_code=["\']([^"\']+)', subtitle, 'subtitle_lang_code', fatal=False)
  342. if not lang_code:
  343. continue
  344. subtitles[lang_code] = self._extract_subtitles(subtitle)
  345. return subtitles
  346. def _real_extract(self, url):
  347. mobj = re.match(self._VALID_URL, url)
  348. video_id = mobj.group('video_id')
  349. if mobj.group('prefix') == 'm':
  350. mobile_webpage = self._download_webpage(url, video_id, 'Downloading mobile webpage')
  351. webpage_url = self._search_regex(r'<link rel="canonical" href="([^"]+)" />', mobile_webpage, 'webpage_url')
  352. else:
  353. webpage_url = 'http://www.' + mobj.group('url')
  354. webpage = self._download_webpage(self._add_skip_wall(webpage_url), video_id, 'Downloading webpage')
  355. note_m = self._html_search_regex(
  356. r'<div class="showmedia-trailer-notice">(.+?)</div>',
  357. webpage, 'trailer-notice', default='')
  358. if note_m:
  359. raise ExtractorError(note_m)
  360. mobj = re.search(r'Page\.messaging_box_controller\.addItems\(\[(?P<msg>{.+?})\]\)', webpage)
  361. if mobj:
  362. msg = json.loads(mobj.group('msg'))
  363. if msg.get('type') == 'error':
  364. raise ExtractorError('crunchyroll returned error: %s' % msg['message_body'], expected=True)
  365. if 'To view this, please log in to verify you are 18 or older.' in webpage:
  366. self.raise_login_required()
  367. video_title = self._html_search_regex(
  368. r'(?s)<h1[^>]*>((?:(?!<h1).)*?<span[^>]+itemprop=["\']title["\'][^>]*>(?:(?!<h1).)+?)</h1>',
  369. webpage, 'video_title')
  370. video_title = re.sub(r' {2,}', ' ', video_title)
  371. video_description = self._parse_json(self._html_search_regex(
  372. r'<script[^>]*>\s*.+?\[media_id=%s\].+?({.+?"description"\s*:.+?})\);' % video_id,
  373. webpage, 'description', default='{}'), video_id).get('description')
  374. if video_description:
  375. video_description = lowercase_escape(video_description.replace(r'\r\n', '\n'))
  376. video_upload_date = self._html_search_regex(
  377. [r'<div>Availability for free users:(.+?)</div>', r'<div>[^<>]+<span>\s*(.+?\d{4})\s*</span></div>'],
  378. webpage, 'video_upload_date', fatal=False, flags=re.DOTALL)
  379. if video_upload_date:
  380. video_upload_date = unified_strdate(video_upload_date)
  381. video_uploader = self._html_search_regex(
  382. # try looking for both an uploader that's a link and one that's not
  383. [r'<a[^>]+href="/publisher/[^"]+"[^>]*>([^<]+)</a>', r'<div>\s*Publisher:\s*<span>\s*(.+?)\s*</span>\s*</div>'],
  384. webpage, 'video_uploader', fatal=False)
  385. available_fmts = []
  386. for a, fmt in re.findall(r'(<a[^>]+token=["\']showmedia\.([0-9]{3,4})p["\'][^>]+>)', webpage):
  387. attrs = extract_attributes(a)
  388. href = attrs.get('href')
  389. if href and '/freetrial' in href:
  390. continue
  391. available_fmts.append(fmt)
  392. if not available_fmts:
  393. for p in (r'token=["\']showmedia\.([0-9]{3,4})p"', r'showmedia\.([0-9]{3,4})p'):
  394. available_fmts = re.findall(p, webpage)
  395. if available_fmts:
  396. break
  397. video_encode_ids = []
  398. formats = []
  399. for fmt in available_fmts:
  400. stream_quality, stream_format = self._FORMAT_IDS[fmt]
  401. video_format = fmt + 'p'
  402. streamdata_req = sanitized_Request(
  403. 'http://www.crunchyroll.com/xml/?req=RpcApiVideoPlayer_GetStandardConfig&media_id=%s&video_format=%s&video_quality=%s'
  404. % (video_id, stream_format, stream_quality),
  405. compat_urllib_parse_urlencode({'current_page': url}).encode('utf-8'))
  406. streamdata_req.add_header('Content-Type', 'application/x-www-form-urlencoded')
  407. streamdata = self._download_xml(
  408. streamdata_req, video_id,
  409. note='Downloading media info for %s' % video_format)
  410. stream_info = streamdata.find('./{default}preload/stream_info')
  411. video_encode_id = xpath_text(stream_info, './video_encode_id')
  412. if video_encode_id in video_encode_ids:
  413. continue
  414. video_encode_ids.append(video_encode_id)
  415. video_file = xpath_text(stream_info, './file')
  416. if not video_file:
  417. continue
  418. if video_file.startswith('http'):
  419. formats.extend(self._extract_m3u8_formats(
  420. video_file, video_id, 'mp4', entry_protocol='m3u8_native',
  421. m3u8_id='hls', fatal=False))
  422. continue
  423. video_url = xpath_text(stream_info, './host')
  424. if not video_url:
  425. continue
  426. metadata = stream_info.find('./metadata')
  427. format_info = {
  428. 'format': video_format,
  429. 'format_id': video_format,
  430. 'height': int_or_none(xpath_text(metadata, './height')),
  431. 'width': int_or_none(xpath_text(metadata, './width')),
  432. }
  433. if '.fplive.net/' in video_url:
  434. video_url = re.sub(r'^rtmpe?://', 'http://', video_url.strip())
  435. parsed_video_url = compat_urlparse.urlparse(video_url)
  436. direct_video_url = compat_urlparse.urlunparse(parsed_video_url._replace(
  437. netloc='v.lvlt.crcdn.net',
  438. path='%s/%s' % (remove_end(parsed_video_url.path, '/'), video_file.split(':')[-1])))
  439. if self._is_valid_url(direct_video_url, video_id, video_format):
  440. format_info.update({
  441. 'url': direct_video_url,
  442. })
  443. formats.append(format_info)
  444. continue
  445. format_info.update({
  446. 'url': video_url,
  447. 'play_path': video_file,
  448. 'ext': 'flv',
  449. })
  450. formats.append(format_info)
  451. self._sort_formats(formats)
  452. metadata = self._download_xml(
  453. 'http://www.crunchyroll.com/xml', video_id,
  454. note='Downloading media info', query={
  455. 'req': 'RpcApiVideoPlayer_GetMediaMetadata',
  456. 'media_id': video_id,
  457. })
  458. subtitles = self.extract_subtitles(video_id, webpage)
  459. # webpage provide more accurate data than series_title from XML
  460. series = self._html_search_regex(
  461. r'id=["\']showmedia_about_episode_num[^>]+>\s*<a[^>]+>([^<]+)',
  462. webpage, 'series', fatal=False)
  463. season = xpath_text(metadata, 'series_title')
  464. episode = xpath_text(metadata, 'episode_title')
  465. episode_number = int_or_none(xpath_text(metadata, 'episode_number'))
  466. season_number = int_or_none(self._search_regex(
  467. r'(?s)<h4[^>]+id=["\']showmedia_about_episode_num[^>]+>.+?</h4>\s*<h4>\s*Season (\d+)',
  468. webpage, 'season number', default=None))
  469. return {
  470. 'id': video_id,
  471. 'title': video_title,
  472. 'description': video_description,
  473. 'thumbnail': xpath_text(metadata, 'episode_image_url'),
  474. 'uploader': video_uploader,
  475. 'upload_date': video_upload_date,
  476. 'series': series,
  477. 'season': season,
  478. 'season_number': season_number,
  479. 'episode': episode,
  480. 'episode_number': episode_number,
  481. 'subtitles': subtitles,
  482. 'formats': formats,
  483. }
  484. class CrunchyrollShowPlaylistIE(CrunchyrollBaseIE):
  485. IE_NAME = 'crunchyroll:playlist'
  486. _VALID_URL = r'https?://(?:(?P<prefix>www|m)\.)?(?P<url>crunchyroll\.com/(?!(?:news|anime-news|library|forum|launchcalendar|lineup|store|comics|freetrial|login))(?P<id>[\w\-]+))/?(?:\?|$)'
  487. _TESTS = [{
  488. 'url': 'http://www.crunchyroll.com/a-bridge-to-the-starry-skies-hoshizora-e-kakaru-hashi',
  489. 'info_dict': {
  490. 'id': 'a-bridge-to-the-starry-skies-hoshizora-e-kakaru-hashi',
  491. 'title': 'A Bridge to the Starry Skies - Hoshizora e Kakaru Hashi'
  492. },
  493. 'playlist_count': 13,
  494. }, {
  495. # geo-restricted (US), 18+ maturity wall, non-premium available
  496. 'url': 'http://www.crunchyroll.com/cosplay-complex-ova',
  497. 'info_dict': {
  498. 'id': 'cosplay-complex-ova',
  499. 'title': 'Cosplay Complex OVA'
  500. },
  501. 'playlist_count': 3,
  502. 'skip': 'Georestricted',
  503. }, {
  504. # geo-restricted (US), 18+ maturity wall, non-premium will be available since 2015.11.14
  505. 'url': 'http://www.crunchyroll.com/ladies-versus-butlers?skip_wall=1',
  506. 'only_matching': True,
  507. }]
  508. def _real_extract(self, url):
  509. show_id = self._match_id(url)
  510. webpage = self._download_webpage(self._add_skip_wall(url), show_id)
  511. title = self._html_search_regex(
  512. r'(?s)<h1[^>]*>\s*<span itemprop="name">(.*?)</span>',
  513. webpage, 'title')
  514. episode_paths = re.findall(
  515. r'(?s)<li id="showview_videos_media_(\d+)"[^>]+>.*?<a href="([^"]+)"',
  516. webpage)
  517. entries = [
  518. self.url_result('http://www.crunchyroll.com' + ep, 'Crunchyroll', ep_id)
  519. for ep_id, ep in episode_paths
  520. ]
  521. entries.reverse()
  522. return {
  523. '_type': 'playlist',
  524. 'id': show_id,
  525. 'title': title,
  526. 'entries': entries,
  527. }