You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

361 lines
13 KiB

Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
Switch codebase to use sanitized_Request instead of compat_urllib_request.Request [downloader/dash] Use sanitized_Request [downloader/http] Use sanitized_Request [atresplayer] Use sanitized_Request [bambuser] Use sanitized_Request [bliptv] Use sanitized_Request [brightcove] Use sanitized_Request [cbs] Use sanitized_Request [ceskatelevize] Use sanitized_Request [collegerama] Use sanitized_Request [extractor/common] Use sanitized_Request [crunchyroll] Use sanitized_Request [dailymotion] Use sanitized_Request [dcn] Use sanitized_Request [dramafever] Use sanitized_Request [dumpert] Use sanitized_Request [eitb] Use sanitized_Request [escapist] Use sanitized_Request [everyonesmixtape] Use sanitized_Request [extremetube] Use sanitized_Request [facebook] Use sanitized_Request [fc2] Use sanitized_Request [flickr] Use sanitized_Request [4tube] Use sanitized_Request [gdcvault] Use sanitized_Request [extractor/generic] Use sanitized_Request [hearthisat] Use sanitized_Request [hotnewhiphop] Use sanitized_Request [hypem] Use sanitized_Request [iprima] Use sanitized_Request [ivi] Use sanitized_Request [keezmovies] Use sanitized_Request [letv] Use sanitized_Request [lynda] Use sanitized_Request [metacafe] Use sanitized_Request [minhateca] Use sanitized_Request [miomio] Use sanitized_Request [meovideo] Use sanitized_Request [mofosex] Use sanitized_Request [moniker] Use sanitized_Request [mooshare] Use sanitized_Request [movieclips] Use sanitized_Request [mtv] Use sanitized_Request [myvideo] Use sanitized_Request [neteasemusic] Use sanitized_Request [nfb] Use sanitized_Request [niconico] Use sanitized_Request [noco] Use sanitized_Request [nosvideo] Use sanitized_Request [novamov] Use sanitized_Request [nowness] Use sanitized_Request [nuvid] Use sanitized_Request [played] Use sanitized_Request [pluralsight] Use sanitized_Request [pornhub] Use sanitized_Request [pornotube] Use sanitized_Request [primesharetv] Use sanitized_Request [promptfile] Use sanitized_Request [qqmusic] Use sanitized_Request [rtve] Use sanitized_Request [safari] Use sanitized_Request [sandia] Use sanitized_Request [shared] Use sanitized_Request [sharesix] Use sanitized_Request [sina] Use sanitized_Request [smotri] Use sanitized_Request [sohu] Use sanitized_Request [spankwire] Use sanitized_Request [sportdeutschland] Use sanitized_Request [streamcloud] Use sanitized_Request [streamcz] Use sanitized_Request [tapely] Use sanitized_Request [tube8] Use sanitized_Request [tubitv] Use sanitized_Request [twitch] Use sanitized_Request [twitter] Use sanitized_Request [udemy] Use sanitized_Request [vbox7] Use sanitized_Request [veoh] Use sanitized_Request [vessel] Use sanitized_Request [vevo] Use sanitized_Request [viddler] Use sanitized_Request [videomega] Use sanitized_Request [viewvster] Use sanitized_Request [viki] Use sanitized_Request [vk] Use sanitized_Request [vodlocker] Use sanitized_Request [voicerepublic] Use sanitized_Request [wistia] Use sanitized_Request [xfileshare] Use sanitized_Request [xtube] Use sanitized_Request [xvideos] Use sanitized_Request [yandexmusic] Use sanitized_Request [youku] Use sanitized_Request [youporn] Use sanitized_Request [youtube] Use sanitized_Request [patreon] Use sanitized_Request [extractor/common] Remove unused import [nfb] PEP 8
9 years ago
10 years ago
  1. # coding: utf-8
  2. from __future__ import unicode_literals
  3. import random
  4. import time
  5. import re
  6. from .common import InfoExtractor
  7. from ..utils import (
  8. sanitized_Request,
  9. strip_jsonp,
  10. unescapeHTML,
  11. clean_html,
  12. ExtractorError,
  13. )
  14. class QQMusicIE(InfoExtractor):
  15. IE_NAME = 'qqmusic'
  16. IE_DESC = 'QQ音乐'
  17. _VALID_URL = r'https?://y.qq.com/#type=song&mid=(?P<id>[0-9A-Za-z]+)'
  18. _TESTS = [{
  19. 'url': 'http://y.qq.com/#type=song&mid=004295Et37taLD',
  20. 'md5': '9ce1c1c8445f561506d2e3cfb0255705',
  21. 'info_dict': {
  22. 'id': '004295Et37taLD',
  23. 'ext': 'mp3',
  24. 'title': '可惜没如果',
  25. 'release_date': '20141227',
  26. 'creator': '林俊杰',
  27. 'description': 'md5:d327722d0361576fde558f1ac68a7065',
  28. 'thumbnail': 're:^https?://.*\.jpg$',
  29. }
  30. }, {
  31. 'note': 'There is no mp3-320 version of this song.',
  32. 'url': 'http://y.qq.com/#type=song&mid=004MsGEo3DdNxV',
  33. 'md5': 'fa3926f0c585cda0af8fa4f796482e3e',
  34. 'info_dict': {
  35. 'id': '004MsGEo3DdNxV',
  36. 'ext': 'mp3',
  37. 'title': '如果',
  38. 'release_date': '20050626',
  39. 'creator': '李季美',
  40. 'description': 'md5:46857d5ed62bc4ba84607a805dccf437',
  41. 'thumbnail': 're:^https?://.*\.jpg$',
  42. }
  43. }, {
  44. 'note': 'lyrics not in .lrc format',
  45. 'url': 'http://y.qq.com/#type=song&mid=001JyApY11tIp6',
  46. 'info_dict': {
  47. 'id': '001JyApY11tIp6',
  48. 'ext': 'mp3',
  49. 'title': 'Shadows Over Transylvania',
  50. 'release_date': '19970225',
  51. 'creator': 'Dark Funeral',
  52. 'description': 'md5:ed14d5bd7ecec19609108052c25b2c11',
  53. 'thumbnail': 're:^https?://.*\.jpg$',
  54. },
  55. 'params': {
  56. 'skip_download': True,
  57. },
  58. }]
  59. _FORMATS = {
  60. 'mp3-320': {'prefix': 'M800', 'ext': 'mp3', 'preference': 40, 'abr': 320},
  61. 'mp3-128': {'prefix': 'M500', 'ext': 'mp3', 'preference': 30, 'abr': 128},
  62. 'm4a': {'prefix': 'C200', 'ext': 'm4a', 'preference': 10}
  63. }
  64. # Reference: m_r_GetRUin() in top_player.js
  65. # http://imgcache.gtimg.cn/music/portal_v3/y/top_player.js
  66. @staticmethod
  67. def m_r_get_ruin():
  68. curMs = int(time.time() * 1000) % 1000
  69. return int(round(random.random() * 2147483647) * curMs % 1E10)
  70. def _real_extract(self, url):
  71. mid = self._match_id(url)
  72. detail_info_page = self._download_webpage(
  73. 'http://s.plcloud.music.qq.com/fcgi-bin/fcg_yqq_song_detail_info.fcg?songmid=%s&play=0' % mid,
  74. mid, note='Download song detail info',
  75. errnote='Unable to get song detail info', encoding='gbk')
  76. song_name = self._html_search_regex(
  77. r"songname:\s*'([^']+)'", detail_info_page, 'song name')
  78. publish_time = self._html_search_regex(
  79. r'发行时间:(\d{4}-\d{2}-\d{2})', detail_info_page,
  80. 'publish time', default=None)
  81. if publish_time:
  82. publish_time = publish_time.replace('-', '')
  83. singer = self._html_search_regex(
  84. r"singer:\s*'([^']+)", detail_info_page, 'singer', default=None)
  85. lrc_content = self._html_search_regex(
  86. r'<div class="content" id="lrc_content"[^<>]*>([^<>]+)</div>',
  87. detail_info_page, 'LRC lyrics', default=None)
  88. if lrc_content:
  89. lrc_content = lrc_content.replace('\\n', '\n')
  90. thumbnail_url = None
  91. albummid = self._search_regex(
  92. [r'albummid:\'([0-9a-zA-Z]+)\'', r'"albummid":"([0-9a-zA-Z]+)"'],
  93. detail_info_page, 'album mid', default=None)
  94. if albummid:
  95. thumbnail_url = "http://i.gtimg.cn/music/photo/mid_album_500/%s/%s/%s.jpg" \
  96. % (albummid[-2:-1], albummid[-1], albummid)
  97. guid = self.m_r_get_ruin()
  98. vkey = self._download_json(
  99. 'http://base.music.qq.com/fcgi-bin/fcg_musicexpress.fcg?json=3&guid=%s' % guid,
  100. mid, note='Retrieve vkey', errnote='Unable to get vkey',
  101. transform_source=strip_jsonp)['key']
  102. formats = []
  103. for format_id, details in self._FORMATS.items():
  104. formats.append({
  105. 'url': 'http://cc.stream.qqmusic.qq.com/%s%s.%s?vkey=%s&guid=%s&fromtag=0'
  106. % (details['prefix'], mid, details['ext'], vkey, guid),
  107. 'format': format_id,
  108. 'format_id': format_id,
  109. 'preference': details['preference'],
  110. 'abr': details.get('abr'),
  111. })
  112. self._check_formats(formats, mid)
  113. self._sort_formats(formats)
  114. actual_lrc_lyrics = ''.join(
  115. line + '\n' for line in re.findall(
  116. r'(?m)^(\[[0-9]{2}:[0-9]{2}(?:\.[0-9]{2,})?\][^\n]*|\[[^\]]*\])', lrc_content))
  117. info_dict = {
  118. 'id': mid,
  119. 'formats': formats,
  120. 'title': song_name,
  121. 'release_date': publish_time,
  122. 'creator': singer,
  123. 'description': lrc_content,
  124. 'thumbnail': thumbnail_url
  125. }
  126. if actual_lrc_lyrics:
  127. info_dict['subtitles'] = {
  128. 'origin': [{
  129. 'ext': 'lrc',
  130. 'data': actual_lrc_lyrics,
  131. }]
  132. }
  133. return info_dict
  134. class QQPlaylistBaseIE(InfoExtractor):
  135. @staticmethod
  136. def qq_static_url(category, mid):
  137. return 'http://y.qq.com/y/static/%s/%s/%s/%s.html' % (category, mid[-2], mid[-1], mid)
  138. @classmethod
  139. def get_entries_from_page(cls, page):
  140. entries = []
  141. for item in re.findall(r'class="data"[^<>]*>([^<>]+)</', page):
  142. song_mid = unescapeHTML(item).split('|')[-5]
  143. entries.append(cls.url_result(
  144. 'http://y.qq.com/#type=song&mid=' + song_mid, 'QQMusic',
  145. song_mid))
  146. return entries
  147. class QQMusicSingerIE(QQPlaylistBaseIE):
  148. IE_NAME = 'qqmusic:singer'
  149. IE_DESC = 'QQ音乐 - 歌手'
  150. _VALID_URL = r'https?://y.qq.com/#type=singer&mid=(?P<id>[0-9A-Za-z]+)'
  151. _TEST = {
  152. 'url': 'http://y.qq.com/#type=singer&mid=001BLpXF2DyJe2',
  153. 'info_dict': {
  154. 'id': '001BLpXF2DyJe2',
  155. 'title': '林俊杰',
  156. 'description': 'md5:870ec08f7d8547c29c93010899103751',
  157. },
  158. 'playlist_count': 12,
  159. }
  160. def _real_extract(self, url):
  161. mid = self._match_id(url)
  162. singer_page = self._download_webpage(
  163. self.qq_static_url('singer', mid), mid, 'Download singer page')
  164. entries = self.get_entries_from_page(singer_page)
  165. singer_name = self._html_search_regex(
  166. r"singername\s*:\s*'([^']+)'", singer_page, 'singer name',
  167. default=None)
  168. singer_id = self._html_search_regex(
  169. r"singerid\s*:\s*'([0-9]+)'", singer_page, 'singer id',
  170. default=None)
  171. singer_desc = None
  172. if singer_id:
  173. req = sanitized_Request(
  174. 'http://s.plcloud.music.qq.com/fcgi-bin/fcg_get_singer_desc.fcg?utf8=1&outCharset=utf-8&format=xml&singerid=%s' % singer_id)
  175. req.add_header(
  176. 'Referer', 'http://s.plcloud.music.qq.com/xhr_proxy_utf8.html')
  177. singer_desc_page = self._download_xml(
  178. req, mid, 'Donwload singer description XML')
  179. singer_desc = singer_desc_page.find('./data/info/desc').text
  180. return self.playlist_result(entries, mid, singer_name, singer_desc)
  181. class QQMusicAlbumIE(QQPlaylistBaseIE):
  182. IE_NAME = 'qqmusic:album'
  183. IE_DESC = 'QQ音乐 - 专辑'
  184. _VALID_URL = r'https?://y.qq.com/#type=album&mid=(?P<id>[0-9A-Za-z]+)'
  185. _TESTS = [{
  186. 'url': 'http://y.qq.com/#type=album&mid=000gXCTb2AhRR1',
  187. 'info_dict': {
  188. 'id': '000gXCTb2AhRR1',
  189. 'title': '我们都是这样长大的',
  190. 'description': 'md5:179c5dce203a5931970d306aa9607ea6',
  191. },
  192. 'playlist_count': 4,
  193. }, {
  194. 'url': 'http://y.qq.com/#type=album&mid=002Y5a3b3AlCu3',
  195. 'info_dict': {
  196. 'id': '002Y5a3b3AlCu3',
  197. 'title': '그리고...',
  198. 'description': 'md5:a48823755615508a95080e81b51ba729',
  199. },
  200. 'playlist_count': 8,
  201. }]
  202. def _real_extract(self, url):
  203. mid = self._match_id(url)
  204. album = self._download_json(
  205. 'http://i.y.qq.com/v8/fcg-bin/fcg_v8_album_info_cp.fcg?albummid=%s&format=json' % mid,
  206. mid, 'Download album page')['data']
  207. entries = [
  208. self.url_result(
  209. 'http://y.qq.com/#type=song&mid=' + song['songmid'], 'QQMusic', song['songmid']
  210. ) for song in album['list']
  211. ]
  212. album_name = album.get('name')
  213. album_detail = album.get('desc')
  214. if album_detail is not None:
  215. album_detail = album_detail.strip()
  216. return self.playlist_result(entries, mid, album_name, album_detail)
  217. class QQMusicToplistIE(QQPlaylistBaseIE):
  218. IE_NAME = 'qqmusic:toplist'
  219. IE_DESC = 'QQ音乐 - 排行榜'
  220. _VALID_URL = r'https?://y\.qq\.com/#type=toplist&p=(?P<id>(top|global)_[0-9]+)'
  221. _TESTS = [{
  222. 'url': 'http://y.qq.com/#type=toplist&p=global_123',
  223. 'info_dict': {
  224. 'id': 'global_123',
  225. 'title': '美国iTunes榜',
  226. },
  227. 'playlist_count': 10,
  228. }, {
  229. 'url': 'http://y.qq.com/#type=toplist&p=top_3',
  230. 'info_dict': {
  231. 'id': 'top_3',
  232. 'title': '巅峰榜·欧美',
  233. 'description': 'QQ音乐巅峰榜·欧美根据用户收听行为自动生成,集结当下最流行的欧美新歌!:更新时间:每周四22点|统'
  234. '计周期:一周(上周四至本周三)|统计对象:三个月内发行的欧美歌曲|统计数量:100首|统计算法:根据'
  235. '歌曲在一周内的有效播放次数,由高到低取前100名(同一歌手最多允许5首歌曲同时上榜)|有效播放次数:'
  236. '登录用户完整播放一首歌曲,记为一次有效播放;同一用户收听同一首歌曲,每天记录为1次有效播放'
  237. },
  238. 'playlist_count': 100,
  239. }, {
  240. 'url': 'http://y.qq.com/#type=toplist&p=global_106',
  241. 'info_dict': {
  242. 'id': 'global_106',
  243. 'title': '韩国Mnet榜',
  244. },
  245. 'playlist_count': 50,
  246. }]
  247. def _real_extract(self, url):
  248. list_id = self._match_id(url)
  249. list_type, num_id = list_id.split("_")
  250. toplist_json = self._download_json(
  251. 'http://i.y.qq.com/v8/fcg-bin/fcg_v8_toplist_cp.fcg?type=%s&topid=%s&format=json'
  252. % (list_type, num_id),
  253. list_id, 'Download toplist page')
  254. entries = [
  255. self.url_result(
  256. 'http://y.qq.com/#type=song&mid=' + song['data']['songmid'], 'QQMusic', song['data']['songmid']
  257. ) for song in toplist_json['songlist']
  258. ]
  259. topinfo = toplist_json.get('topinfo', {})
  260. list_name = topinfo.get('ListName')
  261. list_description = topinfo.get('info')
  262. return self.playlist_result(entries, list_id, list_name, list_description)
  263. class QQMusicPlaylistIE(QQPlaylistBaseIE):
  264. IE_NAME = 'qqmusic:playlist'
  265. IE_DESC = 'QQ音乐 - 歌单'
  266. _VALID_URL = r'https?://y\.qq\.com/#type=taoge&id=(?P<id>[0-9]+)'
  267. _TESTS = [{
  268. 'url': 'http://y.qq.com/#type=taoge&id=3462654915',
  269. 'info_dict': {
  270. 'id': '3462654915',
  271. 'title': '韩国5月新歌精选下旬',
  272. 'description': 'md5:d2c9d758a96b9888cf4fe82f603121d4',
  273. },
  274. 'playlist_count': 40,
  275. 'skip': 'playlist gone',
  276. }, {
  277. 'url': 'http://y.qq.com/#type=taoge&id=1374105607',
  278. 'info_dict': {
  279. 'id': '1374105607',
  280. 'title': '易入人心的华语民谣',
  281. 'description': '民谣的歌曲易于传唱、、歌词朗朗伤口、旋律简单温馨。属于那种才入耳孔。却上心头的感觉。没有太多的复杂情绪。简单而直接地表达乐者的情绪,就是这样的简单才易入人心。',
  282. },
  283. 'playlist_count': 20,
  284. }]
  285. def _real_extract(self, url):
  286. list_id = self._match_id(url)
  287. list_json = self._download_json(
  288. 'http://i.y.qq.com/qzone-music/fcg-bin/fcg_ucc_getcdinfo_byids_cp.fcg?type=1&json=1&utf8=1&onlysong=0&disstid=%s'
  289. % list_id, list_id, 'Download list page',
  290. transform_source=strip_jsonp)
  291. if not len(list_json.get('cdlist', [])):
  292. if list_json.get('code'):
  293. raise ExtractorError(
  294. 'QQ Music said: error %d in fetching playlist info' % list_json['code'],
  295. expected=True)
  296. raise ExtractorError('Unable to get playlist info')
  297. cdlist = list_json['cdlist'][0]
  298. entries = [
  299. self.url_result(
  300. 'http://y.qq.com/#type=song&mid=' + song['songmid'], 'QQMusic', song['songmid']
  301. ) for song in cdlist['songlist']
  302. ]
  303. list_name = cdlist.get('dissname')
  304. list_description = clean_html(unescapeHTML(cdlist.get('desc')))
  305. return self.playlist_result(entries, list_id, list_name, list_description)