Add documentation about supported sites (Fixes #4503)

10 years ago · 416c7fcbce
--- a/+ 5
+++ b/+ 5
@ -1,4 +1,4 @@
 all: youtube-dl README.md CONTRIBUTING.md README.txt youtube-dl.1 youtube-dl.bash-completion youtube-dl.zsh youtube-dl.fish
 all: youtube-dl README.md CONTRIBUTING.md README.txt youtube-dl.1 youtube-dl.bash-completion youtube-dl.zsh youtube-dl.fish supportedsites

 clean:
 	rm -rf youtube-dl.1.temp.md youtube-dl.1 youtube-dl.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dl.tar.gz youtube-dl.zsh youtube-dl.fish *.dump *.part *.info.json CONTRIBUTING.md.tmp
@ -50,7 +50,7 @@ offlinetest: codetest

 tar: youtube-dl.tar.gz

 .PHONY: all clean install test tar bash-completion pypi-files zsh-completion fish-completion ot offlinetest codetest
 .PHONY: all clean install test tar bash-completion pypi-files zsh-completion fish-completion ot offlinetest codetest supportedsites

 pypi-files: youtube-dl.bash-completion README.txt youtube-dl.1 youtube-dl.fish

@ -68,6 +68,9 @@ README.md: youtube_dl/*.py youtube_dl/*/*.py
 CONTRIBUTING.md: README.md
 	python devscripts/make_contributing.py README.md CONTRIBUTING.md

 supportedsites:
 	python devscripts/make_supportedsites.py docs/supportedsites.md

 README.txt: README.md
 	pandoc -f markdown -t plain README.md -o README.txt

--- a/README.md
+++ b/README.md
@ -449,6 +449,14 @@ Since June 2012 (#342) youtube-dl is packed as an executable zipfile, simply unz

 To run the exe you need to install first the [Microsoft Visual C++ 2008 Redistributable Package](http://www.microsoft.com/en-us/download/details.aspx?id=29).

 ### How can I detect whether a given URL is supported by youtube-dl?

 For one, have a look at the [list of supported sites](docs/supportedsites). Note that it can sometimes happen that the site changes its URL scheme (say, from http://example.com/v/1234567 to http://example.com/v/1234567 ) and youtube-dl reports an URL of a service in that list as unsupported. In that case, simply report a bug.

 It is *not* possible to detect whether a URL is supported or not. That's because youtube-dl contains a generic extractor which maches **all** URLs. You may be tempted to disable, exclude, or remove the generic extractor, but the generic extractor not only allows users to extract videos from lots of websites that embed a video from another service, but may also be used to extract video from a service that it's hosting itself. Therefore, we neither recommend nor support disabling, excluding, or removing the generic extractor.

 If you want to find out, simply call youtube-dl. If you get no videos back, chances are the URL is either not referring to a video or unsupported. You can find out which by examining the output (if you run youtube-dl on the console) or catching an `UnsupportedError` exception if you run it from a Python program.

 # DEVELOPER INSTRUCTIONS

 Most users do not need to build youtube-dl and can [download the builds](http://rg3.github.io/youtube-dl/download.html) or get them from their distribution.
--- a/devscripts/make_supportedsites.py
+++ b/devscripts/make_supportedsites.py
@ -0,0 +1,45 @@
 #!/usr/bin/env python
 from __future__ import unicode_literals

 import io
 import optparse
 import os
 import sys


 # Import youtube_dl
 ROOT_DIR = os.path.join(os.path.dirname(__file__), '..')
 sys.path.append(ROOT_DIR)
 import youtube_dl


 def main():
    parser = optparse.OptionParser(usage='%prog OUTFILE.md')
    options, args = parser.parse_args()
    if len(args) != 1:
        parser.error('Expected an output filename')

    outfile, = args

    def gen_ies_md(ies):
        for ie in ies:
            ie_md = '**{}**'.format(ie.IE_NAME)
            ie_desc = getattr(ie, 'IE_DESC', None)
            if ie_desc is False:
                continue
            if ie_desc is not None:
                ie_md += ': {}'.format(ie.IE_DESC)
            if not ie.working():
                ie_md += ' (Currently broken)'
            yield ie_md

    ies = sorted(youtube_dl.gen_extractors(), key=lambda i: i.IE_NAME.lower())
    out = '# Supported sites\n' + ''.join(
        ' - ' + md + '\n'
        for md in gen_ies_md(ies))

    with io.open(outfile, 'w', encoding='utf-8') as outf:
        outf.write(out)

 if __name__ == '__main__':
    main()
--- a/docs/supportedsites.md
+++ b/docs/supportedsites.md
@ -0,0 +1,500 @@
 # Supported sites
 - **1up.com**
 - **220.ro**
 - **24video**
 - **3sat**
 - **4tube**
 - **56.com**
 - **5min**
 - **8tracks**
 - **9gag**
 - **abc.net.au**
 - **AcademicEarth:Course**
 - **AddAnime**
 - **AdobeTV**
 - **AdultSwim**
 - **Aftonbladet**
 - **AlJazeera**
 - **Allocine**
 - **anitube.se**
 - **AnySex**
 - **Aparat**
 - **AppleTrailers**
 - **archive.org**: archive.org videos
 - **ARD**
 - **ARD:mediathek**
 - **arte.tv**
 - **arte.tv:+7**
 - **arte.tv:concert**
 - **arte.tv:creative**
 - **arte.tv:ddc**
 - **arte.tv:embed**
 - **arte.tv:future**
 - **audiomack**
 - **AUEngine**
 - **Azubu**
 - **bambuser**
 - **bambuser:channel**
 - **Bandcamp**
 - **Bandcamp:album**
 - **bbc.co.uk**: BBC iPlayer
 - **Beeg**
 - **BehindKink**
 - **Bet**
 - **Bild**: Bild.de
 - **BiliBili**
 - **blinkx**
 - **blip.tv:user**
 - **BlipTV**
 - **Bloomberg**
 - **Bpb**: Bundeszentrale für politische Bildung
 - **BR**: Bayerischer Rundfunk Mediathek
 - **Break**
 - **Brightcove**
 - **BuzzFeed**
 - **BYUtv**
 - **Canal13cl**
 - **canalc2.tv**
 - **Canalplus**: canalplus.fr, piwiplus.fr and d8.tv
 - **CBS**
 - **CBSNews**: CBS News
 - **CeskaTelevize**
 - **channel9**: Channel 9
 - **Chilloutzone**
 - **Cinchcast**
 - **Cinemassacre**
 - **clipfish**
 - **cliphunter**
 - **Clipsyndicate**
 - **Cloudy**
 - **Clubic**
 - **cmt.com**
 - **CNET**
 - **CNN**
 - **CNNBlogs**
 - **CollegeHumor**
 - **ComCarCoff**
 - **ComedyCentral**
 - **ComedyCentralShows**: The Daily Show / The Colbert Report
 - **CondeNast**: Condé Nast media group: Condé Nast, GQ, Glamour, Vanity Fair, Vogue, W Magazine, WIRED
 - **Cracked**
 - **Criterion**
 - **Crunchyroll**
 - **crunchyroll:playlist**
 - **CSpan**: C-SPAN
 - **culturebox.francetvinfo.fr**
 - **dailymotion**
 - **dailymotion:playlist**
 - **dailymotion:user**
 - **daum.net**
 - **DBTV**
 - **DeezerPlaylist**
 - **defense.gouv.fr**
 - **Discovery**
 - **divxstage**: DivxStage
 - **Dotsub**
 - **Dropbox**
 - **DrTuber**
 - **DRTV**
 - **Dump**
 - **dvtv**: http://video.aktualne.cz/
 - **EbaumsWorld**
 - **eHow**
 - **Einthusan**
 - **eitb.tv**
 - **EllenTV**
 - **EllenTV:clips**
 - **ElPais**: El País
 - **EMPFlix**
 - **Engadget**
 - **Eporner**
 - **Escapist**
 - **EveryonesMixtape**
 - **exfm**: ex.fm
 - **ExpoTV**
 - **ExtremeTube**
 - **facebook**
 - **faz.net**
 - **fc2**
 - **fernsehkritik.tv**
 - **fernsehkritik.tv:postecke**
 - **Firedrive**
 - **Firstpost**
 - **firsttv**: Видеоархив - Первый канал
 - **Flickr**
 - **Folketinget**: Folketinget (ft.dk; Danish parliament)
 - **Foxgay**
 - **FoxNews**
 - **france2.fr:generation-quoi**
 - **FranceCulture**
 - **FranceInter**
 - **francetv**: France 2, 3, 4, 5 and Ô
 - **francetvinfo.fr**
 - **Freesound**
 - **freespeech.org**
 - **FreeVideo**
 - **FunnyOrDie**
 - **Gamekings**
 - **GameOne**
 - **gameone:playlist**
 - **GameSpot**
 - **GameStar**
 - **Gametrailers**
 - **GDCVault**
 - **generic**: Generic downloader that works on some sites
 - **GiantBomb**
 - **Glide**: Glide mobile video messages (glide.me)
 - **Globo**
 - **GodTube**
 - **GoldenMoustache**
 - **Golem**
 - **GorillaVid**: GorillaVid.in, daclips.in, movpod.in and fastvideo.in
 - **Goshgay**
 - **Grooveshark**
 - **Groupon**
 - **Hark**
 - **Heise**
 - **Helsinki**: helsinki.fi
 - **HentaiStigma**
 - **HornBunny**
 - **HostingBulk**
 - **HotNewHipHop**
 - **Howcast**
 - **HowStuffWorks**
 - **HuffPost**: Huffington Post
 - **Hypem**
 - **Iconosquare**
 - **ign.com**
 - **imdb**: Internet Movie Database trailers
 - **imdb:list**: Internet Movie Database lists
 - **Ina**
 - **InfoQ**
 - **Instagram**
 - **instagram:user**: Instagram user profile
 - **InternetVideoArchive**
 - **IPrima**
 - **ivi**: ivi.ru
 - **ivi:compilation**: ivi.ru compilations
 - **Izlesene**
 - **JadoreCettePub**
 - **JeuxVideo**
 - **Jove**
 - **jpopsuki.tv**
 - **Jukebox**
 - **Kankan**
 - **keek**
 - **KeezMovies**
 - **KhanAcademy**
 - **KickStarter**
 - **kontrtube**: KontrTube.ru - Труба зовёт
 - **KrasView**: Красвью
 - **Ku6**
 - **la7.tv**
 - **Laola1Tv**
 - **lifenews**: LIFE | NEWS
 - **LiveLeak**
 - **livestream**
 - **livestream:original**
 - **lrt.lt**
 - **lynda**: lynda.com videos
 - **lynda:course**: lynda.com online courses
 - **m6**
 - **macgamestore**: MacGameStore trailers
 - **mailru**: Видео@Mail.Ru
 - **Malemotion**
 - **MDR**
 - **metacafe**
 - **Metacritic**
 - **Mgoon**
 - **Minhateca**
 - **MinistryGrid**
 - **mitele.es**
 - **mixcloud**
 - **MLB**
 - **MoeVideo**: LetitBit video services: moevideo.net, playreplay.net and videochart.net
 - **Mofosex**
 - **Mojvideo**
 - **Moniker**: allmyvideos.net and vidspot.net
 - **mooshare**: Mooshare.biz
 - **Morningstar**: morningstar.com
 - **Motherless**
 - **Motorsport**: motorsport.com
 - **MovieClips**
 - **Moviezine**
 - **movshare**: MovShare
 - **MPORA**
 - **MTV**
 - **mtviggy.com**
 - **mtvservices:embedded**
 - **MuenchenTV**: münchen.tv
 - **MusicPlayOn**
 - **MusicVault**
 - **muzu.tv**
 - **MySpace**
 - **MySpace:album**
 - **MySpass**
 - **myvideo**
 - **MyVidster**
 - **Naver**
 - **NBA**
 - **NBC**
 - **NBCNews**
 - **ndr**: NDR.de - Mediathek
 - **NDTV**
 - **NerdCubedFeed**
 - **Newgrounds**
 - **Newstube**
 - **nfb**: National Film Board of Canada
 - **nfl.com**
 - **nhl.com**
 - **nhl.com:videocenter**: NHL videocenter category
 - **niconico**: ニコニコ動画
 - **NiconicoPlaylist**
 - **Noco**
 - **Normalboots**
 - **NosVideo**
 - **novamov**: NovaMov
 - **Nowness**
 - **nowvideo**: NowVideo
 - **npo.nl**
 - **NRK**
 - **NRKTV**
 - **NTV**
 - **Nuvid**
 - **NYTimes**
 - **ocw.mit.edu**
 - **OktoberfestTV**
 - **on.aol.com**
 - **Ooyala**
 - **orf:oe1**: Radio Österreich 1
 - **orf:tvthek**: ORF TVthek
 - **ORFFM4**: radio FM4
 - **parliamentlive.tv**: UK parliament videos
 - **Patreon**
 - **PBS**
 - **Phoenix**
 - **Photobucket**
 - **PlanetaPlay**
 - **play.fm**
 - **played.to**
 - **Playvid**
 - **plus.google**: Google Plus
 - **pluzz.francetv.fr**
 - **podomatic**
 - **PornHd**
 - **PornHub**
 - **Pornotube**
 - **PornoXO**
 - **PromptFile**
 - **prosiebensat1**: ProSiebenSat.1 Digital
 - **Pyvideo**
 - **QuickVid**
 - **radio.de**
 - **radiofrance**
 - **Rai**
 - **RBMARadio**
 - **RedTube**
 - **Restudy**
 - **ReverbNation**
 - **RingTV**
 - **RottenTomatoes**
 - **Roxwel**
 - **RTBF**
 - **RTLnow**
 - **rtlxl.nl**
 - **RTP**
 - **RTS**: RTS.ch
 - **rtve.es:alacarta**: RTVE a la carta
 - **rtve.es:live**: RTVE.es live streams
 - **RUHD**
 - **rutube**: Rutube videos
 - **rutube:channel**: Rutube channels
 - **rutube:movie**: Rutube movies
 - **rutube:person**: Rutube person videos
 - **RUTV**: RUTV.RU
 - **Sapo**: SAPO Vídeos
 - **savefrom.net**
 - **SBS**: sbs.com.au
 - **SciVee**
 - **screen.yahoo:search**: Yahoo screen search
 - **Screencast**
 - **ScreencastOMatic**
 - **ScreenwaveMedia**
 - **ServingSys**
 - **Sexu**
 - **SexyKarma**: Sexy Karma and Watch Indian Porn
 - **Shared**
 - **ShareSix**
 - **Sina**
 - **Slideshare**
 - **Slutload**
 - **smotri**: Smotri.com
 - **smotri:broadcast**: Smotri.com broadcasts
 - **smotri:community**: Smotri.com community videos
 - **smotri:user**: Smotri.com user videos
 - **Snotr**
 - **Sockshare**
 - **Sohu**
 - **soundcloud**
 - **soundcloud:playlist**
 - **soundcloud:set**
 - **soundcloud:user**
 - **Soundgasm**
 - **southpark.cc.com**
 - **southpark.de**
 - **Space**
 - **Spankwire**
 - **Spiegel**
 - **Spiegel:Article**: Articles on spiegel.de
 - **Spiegeltv**
 - **Spike**
 - **Sport5**
 - **SportBox**
 - **SportDeutschland**
 - **SRMediathek**: Süddeutscher Rundfunk
 - **stanfordoc**: Stanford Open ClassRoom
 - **Steam**
 - **streamcloud.eu**
 - **StreamCZ**
 - **SunPorno**
 - **SWRMediathek**
 - **Syfy**
 - **SztvHu**
 - **Tagesschau**
 - **Tapely**
 - **Tass**
 - **teachertube**: teachertube.com videos
 - **teachertube:user:collection**: teachertube.com user and collection videos
 - **TeachingChannel**
 - **Teamcoco**
 - **TeamFour**
 - **TechTalks**
 - **techtv.mit.edu**
 - **TED**
 - **tegenlicht.vpro.nl**
 - **TeleBruxelles**
 - **telecinco.es**
 - **TeleMB**
 - **TenPlay**
 - **TF1**
 - **TheOnion**
 - **ThePlatform**
 - **TheSixtyOne**
 - **ThisAV**
 - **THVideo**
 - **THVideoPlaylist**
 - **tinypic**: tinypic.com videos
 - **tlc.com**
 - **tlc.de**
 - **TMZ**
 - **TNAFlix**
 - **tou.tv**
 - **Toypics**: Toypics user profile
 - **ToypicsUser**: Toypics user profile
 - **TrailerAddict** (Currently broken)
 - **Trilulilu**
 - **TruTube**
 - **Tube8**
 - **Tudou**
 - **Tumblr**
 - **TuneIn**
 - **Turbo**
 - **Tutv**
 - **tv.dfb.de**
 - **tvigle**: Интернет-телевидение Tvigle.ru
 - **tvp.pl**
 - **TVPlay**: TV3Play and related services
 - **Twitch**
 - **Ubu**
 - **udemy**
 - **udemy:course**
 - **Unistra**
 - **Urort**: NRK P3 Urørt
 - **ustream**
 - **ustream:channel**
 - **Vbox7**
 - **VeeHD**
 - **Veoh**
 - **Vesti**: Вести.Ru
 - **Vevo**
 - **VGTV**
 - **vh1.com**
 - **Vice**
 - **Viddler**
 - **video.google:search**: Google Video search
 - **video.mit.edu**
 - **VideoBam**
 - **VideoDetective**
 - **videofy.me**
 - **videolectures.net**
 - **VideoMega**
 - **VideoPremium**
 - **VideoTt**: video.tt - Your True Tube
 - **videoweed**: VideoWeed
 - **Vidme**
 - **Vidzi**
 - **viki**
 - **vimeo**
 - **vimeo:album**
 - **vimeo:channel**
 - **vimeo:group**
 - **vimeo:likes**: Vimeo user likes
 - **vimeo:review**: Review pages on vimeo
 - **vimeo:user**
 - **vimeo:watchlater**: Vimeo watch later list, "vimeowatchlater" keyword (requires authentication)
 - **Vimple**: Vimple.ru
 - **Vine**
 - **vine:user**
 - **vk.com**
 - **vk.com:user-videos**: vk.com:All of a user's videos
 - **Vodlocker**
 - **Vporn**
 - **VRT**
 - **vube**: Vube.com
 - **VuClip**
 - **vulture.com**
 - **Walla**
 - **WashingtonPost**
 - **wat.tv**
 - **WayOfTheMaster**
 - **WDR**
 - **wdr:mobile**
 - **WDRMaus**: Sendung mit der Maus
 - **Weibo**
 - **Wimp**
 - **Wistia**
 - **WorldStarHipHop**
 - **wrzuta.pl**
 - **XBef**
 - **XboxClips**
 - **XHamster**
 - **XMinus**
 - **XNXX**
 - **XTube**
 - **XTubeUser**: XTube user profile
 - **XVideos**
 - **Yahoo**: Yahoo screen and movies
 - **YesJapan**
 - **Ynet**
 - **YouJizz**
 - **Youku**
 - **YouPorn**
 - **YourUpload**
 - **youtube**: YouTube.com
 - **youtube:channel**: YouTube.com channels
 - **youtube:favorites**: YouTube.com favourite videos, ":ytfav" for short (requires authentication)
 - **youtube:history**: Youtube watch history, ":ythistory" for short (requires authentication)
 - **youtube:playlist**: YouTube.com playlists
 - **youtube:recommended**: YouTube.com recommended videos, ":ytrec" for short (requires authentication)
 - **youtube:search**: YouTube.com searches
 - **youtube:search:date**: YouTube.com searches, newest videos first
 - **youtube:search_url**: YouTube.com search URLs
 - **youtube:show**: YouTube.com (multi-season) shows
 - **youtube:subscriptions**: YouTube.com subscriptions feed, "ytsubs" keyword (requires authentication)
 - **youtube:toplist**: YouTube.com top lists, "yttoplist:{channel}:{list title}" (Example: "yttoplist:music:Top Tracks")
 - **youtube:user**: YouTube.com user videos (URL or "ytuser" keyword)
 - **youtube:watch_later**: Youtube watch later list, ":ytwatchlater" for short (requires authentication)
 - **ZDF**
 - **ZDFChannel**
 - **zingmp3:album**: mp3.zing.vn albums
 - **zingmp3:song**: mp3.zing.vn songs
--- a/youtube_dl/extractor/generic.py
+++ b/youtube_dl/extractor/generic.py
@ -23,6 +23,7 @@ from ..utils import (
    unescapeHTML,
    unified_strdate,
    unsmuggle_url,
    UnsupportedError,
    url_basename,
 )
 from .brightcove import BrightcoveIE
@ -1057,7 +1058,7 @@ class GenericIE(InfoExtractor):
                    'url': new_url,
                }
        if not found:
            raise ExtractorError('Unsupported URL: %s' % url)
            raise UnsupportedError(url)

        entries = []
        for video_url in found:
--- a/youtube_dl/utils.py
+++ b/youtube_dl/utils.py
@ -464,6 +464,13 @@ class ExtractorError(Exception):
        return ''.join(traceback.format_tb(self.traceback))


 class UnsupportedError(ExtractorError):
    def __init__(self, url):
        super(UnsupportedError, self).__init__(
            'Unsupported URL: %s' % url, expected=True)
        self.url = url


 class RegexNotFoundError(ExtractorError):
    """Error when a regex didn't match"""
    pass