Bump version to v0.3.2

Introduce primary_client, fallback_client, and last_resort_client variables for better configurability.
Replace hardcoded 'android_vr' with primary_client in fetch_player_response call.

.gitea/workflows/ci.yaml

@@ -0,0 +1,23 @@
+name: CI
+
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.11
+
+      - name: Install dependencies
+        run: |
+          pip install --upgrade pip
+          pip install -r requirements-dev.txt
+
+      - name: Run tests
+        run: pytest

.gitea/workflows/git-sync.yaml

@@ -0,0 +1,40 @@
+name: git-sync-with-mirror
+
+on:
+  push:
+    branches: [ master ]
+  workflow_dispatch:
+
+jobs:
+  git-sync:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: git-sync
+        env:
+          git_sync_source_repo: git@git.fridu.us:heckyel/yt-local.git
+          git_sync_destination_repo: ssh://git@c.fridu.us/software/yt-local.git
+        if: env.git_sync_source_repo && env.git_sync_destination_repo
+        uses: astounds/git-sync@v1
+        with:
+          source_repo: git@git.fridu.us:heckyel/yt-local.git
+          source_branch: "master"
+          destination_repo: ssh://git@c.fridu.us/software/yt-local.git
+          destination_branch: "master"
+          source_ssh_private_key: ${{ secrets.GIT_SYNC_SOURCE_SSH_PRIVATE_KEY }}
+          destination_ssh_private_key:  ${{ secrets.GIT_SYNC_DESTINATION_SSH_PRIVATE_KEY }}
+
+      - name: git-sync-sourcehut
+        env:
+          git_sync_source_repo: git@git.fridu.us:heckyel/yt-local.git
+          git_sync_destination_repo: git@git.sr.ht:~heckyel/yt-local
+        if: env.git_sync_source_repo && env.git_sync_destination_repo
+        uses: astounds/git-sync@v1
+        with:
+          source_repo: git@git.fridu.us:heckyel/yt-local.git
+          source_branch: "master"
+          destination_repo: git@git.sr.ht:~heckyel/yt-local
+          destination_branch: "master"
+          source_ssh_private_key: ${{ secrets.GIT_SYNC_SOURCE_SSH_PRIVATE_KEY }}
+          destination_ssh_private_key:  ${{ secrets.GIT_SYNC_DESTINATION_SSH_PRIVATE_KEY }}
+        continue-on-error: true

.gitignore

@@ -4,7 +4,7 @@ debug/
 data/
 python/
 release/
-youtube-local/
+yt-local/
 banned_addresses.txt
 settings.txt
 get-pip.py
@@ -12,3 +12,4 @@ latest-dist.zip
 *.7z
 *.zip
 *venv*
+flycheck_*

README.md

@@ -1,5 +1,3 @@
-[![builds.sr.ht status](https://builds.sr.ht/~heckyel/yt-local/commits/.build.yml.svg)](https://builds.sr.ht/~heckyel/yt-local/commits/.build.yml?)
-
 # yt-local
 
 Fork of [youtube-local](https://github.com/user234683/youtube-local)
@@ -24,10 +22,10 @@ The YouTube API is not used, so no keys or anything are needed. It uses the same
 * Local playlists: These solve the two problems with creating playlists on YouTube: (1) they're datamined and (2) videos frequently get deleted by YouTube and lost from the playlist, making it very difficult to find a reupload as the title of the deleted video is not displayed.
 * Themes: Light, Gray, and Dark
 * Subtitles
-* Easily download videos or their audio
+* Easily download videos or their audio. (Disabled by default)
 * No ads
 * View comments
-* Javascript not required
+* JavaScript not required
 * Theater and non-theater mode
 * Subscriptions that are independent from YouTube
   * Can import subscriptions from YouTube
@@ -46,6 +44,7 @@ The YouTube API is not used, so no keys or anything are needed. It uses the same
 * Optionally skip sponsored segments using [SponsorBlock](https://github.com/ajayyy/SponsorBlock)'s API
 * Custom video speeds
 * Video transcript
+* Supports all available video qualities: 144p through 2160p
 
 ## Planned features
 - [ ] Putting videos from subscriptions or local playlists into the related videos
@@ -54,8 +53,7 @@ The YouTube API is not used, so no keys or anything are needed. It uses the same
 - [ ] Auto-saving of local playlist videos
 - [ ] Import youtube playlist into a local playlist
 - [ ] Rearrange items of local playlist
-- [ ] Video qualities other than 360p and 720p by muxing video and audio
-- [ ] Corrected .m4a downloads
+- [x] Video qualities other than 360p and 720p by muxing video and audio
 - [x] Indicate if comments are disabled
 - [x] Indicate how many comments a video has
 - [ ] Featured channels page
@@ -89,15 +87,15 @@ Download the tarball under the Releases page and extract it. `cd` into the direc
 
 ## Usage
 
-Firstly, if you wish to run this in portable mode, create the empty file "settings.txt" in the program's main directory. If the file is there, settings and data will be stored in the same directory as the program. Otherwise, settings and data will be stored in `C:\Users\[your username]\.youtube-local` on Windows and `~/.youtube-local` on GNU+Linux/MacOS.
+Firstly, if you wish to run this in portable mode, create the empty file "settings.txt" in the program's main directory. If the file is there, settings and data will be stored in the same directory as the program. Otherwise, settings and data will be stored in `C:\Users\[your username]\.yt-local` on Windows and `~/.yt-local` on GNU+Linux/MacOS.
 
 To run the program on windows, open `run.bat`. On GNU+Linux/MacOS, run `python3 server.py`.
 
+Access youtube URLs by prefixing them with `http://localhost:9010/`.
+For instance, `http://localhost:9010/https://www.youtube.com/watch?v=vBgulDeV2RU`
+You can use an addon such as Redirector ([Firefox](https://addons.mozilla.org/en-US/firefox/addon/redirector/)|[Chrome](https://chrome.google.com/webstore/detail/redirector/ocgpenflpmgnfapjedencafcfakcekcd)) to automatically redirect YouTube URLs to yt-local. I use the include pattern `^(https?://(?:[a-zA-Z0-9_-]*\.)?(?:youtube\.com|youtu\.be|youtube-nocookie\.com)/.*)` and redirect pattern `http://localhost:9010/$1` (Make sure you're using regular expression mode).
 
-Access youtube URLs by prefixing them with `http://localhost:8080/`, For instance, `http://localhost:8080/https://www.youtube.com/watch?v=vBgulDeV2RU`
-You can use an addon such as Redirector ([Firefox](https://addons.mozilla.org/en-US/firefox/addon/redirector/)|[Chrome](https://chrome.google.com/webstore/detail/redirector/ocgpenflpmgnfapjedencafcfakcekcd)) to automatically redirect YouTube URLs to yt-local. I use the include pattern `^(https?://(?:[a-zA-Z0-9_-]*\.)?(?:youtube\.com|youtu\.be|youtube-nocookie\.com)/.*)` and the redirect pattern `http://localhost:8080/$1` (Make sure you're using regular expression mode).
-
-If you want embeds on the web to also redirect to yt-local, make sure "Iframes" is checked under advanced options in your redirector rule. Check test `http://localhost:8080/youtube.com/embed/vBgulDeV2RU`
+If you want embeds on web to also redirect to yt-local, make sure "Iframes" is checked under advanced options in your redirector rule. Check test `http://localhost:9010/youtube.com/embed/vBgulDeV2RU`
 
 yt-local can be added as a search engine in firefox to make searching more convenient. See [here](https://support.mozilla.org/en-US/kb/add-or-remove-search-engine-firefox) for information on firefox search plugins.
 
@@ -105,13 +103,21 @@ yt-local can be added as a search engine in firefox to make searching more conve
 
 In the settings page, set "Route Tor" to "On, except video" (the second option). Be sure to save the settings.
 
-Ensure Tor is listening for Socks5 connections on port 9150 (a simple way to accomplish this is by opening the Tor Browser Bundle and leaving it open). Your connections should now be routed through Tor.
+Ensure Tor is listening for Socks5 connections on port 9150. A simple way to accomplish this is by opening the Tor Browser Bundle and leaving it open. However, you will not be accessing the program (at https://localhost:8080) through the Tor Browser. You will use your regular browser for that. Rather, this is just a quick way to give the program access to Tor routing.
+
+### Standalone Tor
+
+If you don't want to waste system resources leaving the Tor Browser open in addition to your regular browser, you can configure standalone Tor to run instead using the following instructions.
+
+For Windows, to make standalone Tor run at startup, press Windows Key + R and type `shell:startup` to open the Startup folder. Create a new shortcut there. For the command of the shortcut, enter `"C:\[path-to-Tor-Browser-directory]\Tor\tor.exe" SOCKSPort 9150 ControlPort 9151`. You can then launch this shortcut to start it. Alternatively, if something isn't working, to see what's wrong, open `cmd.exe` and go to the directory `C:\[path-to-Tor-Browser-directory]\Tor`. Then run `tor SOCKSPort 9150 ControlPort 9151 | more`. The `more` part at the end is just to make sure any errors are displayed, to fix a bug in Windows cmd where tor doesn't display any output. You can stop tor in the task manager.
+
+For Debian/Ubuntu, you can `sudo apt install tor` to install the command line version of Tor, and then run `sudo systemctl start tor` to run it as a background service that will get started during boot as well. However, Tor on the command line uses the port `9050` by default (rather than the 9150 used by the Tor Browser). So you will need to change `Tor port` to 9050 and `Tor control port` to `9051` in yt-local settings page. Additionally, you will need to enable the Tor control port by uncommenting the line `ControlPort 9051`, and setting `CookieAuthentication` to 0 in `/etc/tor/torrc`. If no Tor package is available for your distro, you can configure the `tor` binary located at `./Browser/TorBrowser/Tor/tor` inside the Tor Browser installation location to run at start time, or create a service to do it.
 
 ### Tor video routing
 
 If you wish to route the video through Tor, set "Route Tor" to "On, including video". Because this is bandwidth-intensive, you are strongly encouraged to donate to the [consortium of Tor node operators](https://torservers.net/donate.html). For instance, donations to [NoiseTor](https://noisetor.net/) go straight towards funding nodes. Using their numbers for bandwidth costs, together with an average of 485 kbit/sec for a diverse sample of videos, and assuming n hours of video watched per day, gives $0.03n/month. A $1/month donation will be a very generous amount to not only offset losses, but help keep the network healthy.
 
-In general, Tor video routing will be slower (for instance, moving around in the video is quite slow). I've never seen any signs that watch history in yt-local affects on-site YouTube recommendations. It's likely that requests to googlevideo are logged for some period of time, but are not integrated into YouTube's larger advertisement/recommendation systems, since those presumably depend more heavily on in-page tracking through Javascript rather than CDN requests to googlevideo.
+In general, Tor video routing will be slower (for instance, moving around in the video is quite slow). I've never seen any signs that watch history in yt-local affects on-site Youtube recommendations. It's likely that requests to googlevideo are logged for some period of time, but are not integrated into Youtube's larger advertisement/recommendation systems, since those presumably depend more heavily on in-page tracking through Javascript rather than CDN requests to googlevideo.
 
 ### Importing subscriptions
 
@@ -120,8 +126,14 @@ In general, Tor video routing will be slower (for instance, moving around in the
 3. Click on "All data included", then on "Deselect all", then select only "subscriptions" and click "OK".
 4. Click on "Next step" and then on "Create export".
 5. Click on the "Download" button after it appears.
-6. From the downloaded takeout zip extract the .json file. It is usually located under `YouTube and YouTube Music/subscriptions/subscriptions.json`
-7. Go to the subscriptions manager in yt-local. In the import area, select your .json file, then press import.
+6. From the downloaded takeout zip extract the .csv file. It is usually located under `YouTube and YouTube Music/subscriptions/subscriptions.csv`
+7. Go to the subscriptions manager in yt-local. In the import area, select your .csv file, then press import.
+
+Supported subscriptions import formats:
+- NewPipe subscriptions export JSON
+- Google Takeout CSV
+- Old Google Takeout JSON
+- OPML format from now-removed YouTube subscriptions manager
 
 ## Contributing
 
@@ -129,6 +141,18 @@ Pull requests and issues are welcome
 
 For coding guidelines and an overview of the software architecture, see the [HACKING.md](docs/HACKING.md) file.
 
+## GPG public KEY
+
+```bash
+72CFB264DFC43F63E098F926E607CE7149F4D71C
+```
+
+## Public instances
+
+yt-local is not made to work in public mode, however there is an instance of yt-local in public mode but with less features
+
+- <https://m.fridu.us/https://youtube.com>
+
 ## License
 
 This project is licensed under the GNU Affero General Public License v3 (GNU AGPLv3) or any later version.

docs/basic-script-openrc/README.md

@@ -73,4 +73,4 @@ after, modified execute permissions:
 - disable: `doas rc-update del ytlocal`
 
 When yt-local is run with administrator privileges,
-the configuration file is stored in /root/.youtube-local
+the configuration file is stored in /root/.yt-local

generate_release.py

@@ -1,7 +1,8 @@
 # Generate a windows release and a generated embedded distribution of python
-# Latest python version is the argument of the script
+# Latest python version is the argument of the script (or oldwin for
+# vista, 7 and 32-bit versions)
 # Requirements: 7z, git
-# wine 32-bit is required in order to build on Linux
+# wine is required in order to build on Linux
 
 import sys
 import urllib
@@ -12,22 +13,28 @@ import os
 import hashlib
 
 latest_version = sys.argv[1]
+if len(sys.argv) > 2:
+    bitness = sys.argv[2]
+else:
+    bitness = '64'
 
+if latest_version == 'oldwin':
+    bitness = '32'
+    latest_version = '3.7.9'
+    suffix = 'windows-vista-7-only'
+else:
+    suffix = 'windows'
 
 def check(code):
     if code != 0:
         raise Exception('Got nonzero exit code from command')
-
-
 def check_subp(x):
     if x.returncode != 0:
         raise Exception('Got nonzero exit code from command')
 
-
 def log(line):
     print('[generate_release.py] ' + line)
 
-
 # https://stackoverflow.com/questions/7833715/python-deleting-certain-file-extensions
 def remove_files_with_extensions(path, extensions):
     for root, dirs, files in os.walk(path):
@@ -35,7 +42,6 @@ def remove_files_with_extensions(path, extensions):
             if os.path.splitext(file)[1] in extensions:
                 os.remove(os.path.join(root, file))
 
-
 def download_if_not_exists(file_name, url, sha256=None):
     if not os.path.exists('./' + file_name):
         log('Downloading ' + file_name + '..')
@@ -51,7 +57,6 @@ def download_if_not_exists(file_name, url, sha256=None):
     else:
         log('Using existing ' + file_name)
 
-
 def wine_run_shell(command):
     if os.name == 'posix':
         check(os.system('wine ' + command.replace('\\', '/')))
@@ -60,14 +65,12 @@ def wine_run_shell(command):
     else:
         raise Exception('Unsupported OS')
 
-
 def wine_run(command_parts):
     if os.name == 'posix':
-        command_parts = ['wine', ] + command_parts
+        command_parts = ['wine',] + command_parts
     if subprocess.run(command_parts).returncode != 0:
         raise Exception('Got nonzero exit code from command')
 
-
 # ---------- Get current release version, for later ----------
 log('Getting current release version')
 describe_result = subprocess.run(['git', 'describe', '--tags'], stdout=subprocess.PIPE)
@@ -77,40 +80,54 @@ if describe_result.returncode != 0:
 release_tag = describe_result.stdout.strip().decode('ascii')
 
 
-# ----------- Make copy of youtube-local files using git -----------
+# ----------- Make copy of yt-local files using git -----------
 
-if os.path.exists('./youtube-local'):
+if os.path.exists('./yt-local'):
     log('Removing old release')
-    shutil.rmtree('./youtube-local')
+    shutil.rmtree('./yt-local')
 
 # Export git repository - this will ensure .git and things in gitignore won't
 # be included. Git only supports exporting archive formats, not into
-# directories, so pipe into 7z to put it into .\youtube-local (not to be
+# directories, so pipe into 7z to put it into .\yt-local (not to be
 # confused with working directory. I'm calling it the same thing so it will
 # have that name when extracted from the final release zip archive)
-log('Making copy of youtube-local files')
-check(os.system('git archive --format tar master | 7z x -si -ttar -oyoutube-local'))
+log('Making copy of yt-local files')
+check(os.system('git archive --format tar master | 7z x -si -ttar -oyt-local'))
 
-if len(os.listdir('./youtube-local')) == 0:
-    raise Exception('Failed to copy youtube-local files')
+if len(os.listdir('./yt-local')) == 0:
+    raise Exception('Failed to copy yt-local files')
 
 
 # ----------- Generate embedded python distribution -----------
 os.environ['PYTHONDONTWRITEBYTECODE'] = '1'     # *.pyc files double the size of the distribution
 get_pip_url = 'https://bootstrap.pypa.io/get-pip.py'
-latest_dist_url = 'https://www.python.org/ftp/python/' + latest_version + '/python-' + latest_version + '-embed-win32.zip'
+latest_dist_url = 'https://www.python.org/ftp/python/' + latest_version + '/python-' + latest_version
+if bitness == '32':
+    latest_dist_url += '-embed-win32.zip'
+else:
+    latest_dist_url += '-embed-amd64.zip'
 
 # I've verified that all the dlls in the following are signed by Microsoft.
 # Using this because Microsoft only provides installers whose files can't be
 # extracted without a special tool.
-visual_c_runtime_url = 'https://github.com/yuempek/vc-archive/raw/master/archives/vc15_(14.10.25017.0)_2017_x86.7z'
-visual_c_runtime_sha256 = '2549eb4d2ce4cf3a87425ea01940f74368bf1cda378ef8a8a1f1a12ed59f1547'
+if bitness == '32':
+    visual_c_runtime_url = 'https://github.com/yuempek/vc-archive/raw/master/archives/vc15_(14.10.25017.0)_2017_x86.7z'
+    visual_c_runtime_sha256 = '2549eb4d2ce4cf3a87425ea01940f74368bf1cda378ef8a8a1f1a12ed59f1547'
+    visual_c_name = 'vc15_(14.10.25017.0)_2017_x86.7z'
+    visual_c_path_to_dlls = 'runtime_minimum/System'
+else:
+    visual_c_runtime_url = 'https://github.com/yuempek/vc-archive/raw/master/archives/vc15_(14.10.25017.0)_2017_x64.7z'
+    visual_c_runtime_sha256 = '4f00b824c37e1017a93fccbd5775e6ee54f824b6786f5730d257a87a3d9ce921'
+    visual_c_name = 'vc15_(14.10.25017.0)_2017_x64.7z'
+    visual_c_path_to_dlls = 'runtime_minimum/System64'
 
 download_if_not_exists('get-pip.py', get_pip_url)
-download_if_not_exists('python-dist-' + latest_version + '.zip', latest_dist_url)
-download_if_not_exists('vc15_(14.10.25017.0)_2017_x86.7z',
-                       visual_c_runtime_url,
-                       sha256=visual_c_runtime_sha256)
+
+python_dist_name = 'python-dist-' + latest_version + '-' + bitness + '.zip'
+
+download_if_not_exists(python_dist_name, latest_dist_url)
+download_if_not_exists(visual_c_name,
+    visual_c_runtime_url, sha256=visual_c_runtime_sha256)
 
 if os.path.exists('./python'):
     log('Removing old python distribution')
@@ -119,7 +136,7 @@ if os.path.exists('./python'):
 
 log('Extracting python distribution')
 
-check(os.system(r'7z -y x -opython python-dist-' + latest_version + '.zip'))
+check(os.system(r'7z -y x -opython ' + python_dist_name))
 
 log('Executing get-pip.py')
 wine_run(['./python/python.exe', '-I', 'get-pip.py'])
@@ -176,14 +193,14 @@ with open(r'./python/path_fixes.pth', 'w', encoding='utf-8') as f:
 
 '''# python3x._pth file tells the python executable where to look for files
 #  Need to add the directory where packages are installed,
-# and the parent directory (which is where the youtube-local files are)
+# and the parent directory (which is where the yt-local files are)
 major_release = latest_version.split('.')[1]
 with open('./python/python3' + major_release + '._pth', 'a', encoding='utf-8') as f:
     f.write('.\\Lib\\site-packages\n')
     f.write('..\n')'''
 
 log('Inserting Microsoft C Runtime')
-check_subp(subprocess.run([r'7z', '-y', 'e', '-opython', 'vc15_(14.10.25017.0)_2017_x86.7z', 'runtime_minimum/System']))
+check_subp(subprocess.run([r'7z', '-y', 'e', '-opython', visual_c_name, visual_c_path_to_dlls]))
 
 log('Installing dependencies')
 wine_run(['./python/python.exe', '-I', '-m', 'pip', 'install', '--no-compile', '-r', './requirements.txt'])
@@ -216,15 +233,15 @@ log('Finished generating python distribution')
 
 # ----------- Copy generated distribution into release folder -----------
 log('Copying python distribution into release folder')
-shutil.copytree(r'./python', r'./youtube-local/python')
+shutil.copytree(r'./python', r'./yt-local/python')
 
 # ----------- Create release zip -----------
-output_filename = 'youtube-local-' + release_tag + '-windows.zip'
+output_filename = 'yt-local-' + release_tag + '-' + suffix + '.zip'
 if os.path.exists('./' + output_filename):
     log('Removing previous zipped release')
     os.remove('./' + output_filename)
 log('Zipping release')
-check(os.system(r'7z -mx=9 a ' + output_filename + ' ./youtube-local'))
+check(os.system(r'7z -mx=9 a ' + output_filename + ' ./yt-local'))
 
 print('\n')
 log('Finished')

requirements-dev.txt

@@ -1,28 +1,5 @@
-attrs>=20.3.0
-Brotli>=1.0.9
-cachetools>=4.2.2
-click>=8.0.1
-dataclasses>=0.6
-defusedxml>=0.7.1
-Flask>=2.0.1
-gevent>=21.8.0
-greenlet>=1.1.1
-importlib-metadata>=4.6.4
-iniconfig>=1.1.1
-itsdangerous>=2.0.1
-Jinja2>=3.0.1
-MarkupSafe>=2.0.1
-packaging>=20.9
-pluggy>=0.13.1
-py>=1.10.0
-pyparsing>=2.4.7
-PySocks>=1.7.1
-pytest>=6.2.2
-stem>=1.8.0
-toml>=0.10.2
-typing-extensions>=3.10.0.0
-urllib3>=1.26.6
-Werkzeug>=2.0.1
-zipp>=3.5.0
-zope.event>=4.5.0
-zope.interface>=5.4.0
+# Include all production requirements
+-r requirements.txt
+
+# Development requirements
+pytest>=6.2.1

requirements.txt

@@ -1,20 +1,8 @@
-Brotli>=1.0.9
-cachetools>=4.2.2
-click>=8.0.1
-dataclasses>=0.6
-defusedxml>=0.7.1
-Flask>=2.0.1
-gevent>=21.8.0
-greenlet>=1.1.1
-importlib-metadata>=4.6.4
-itsdangerous>=2.0.1
-Jinja2>=3.0.1
-MarkupSafe>=2.0.1
-PySocks>=1.7.1
+Flask>=1.0.3
+gevent>=1.2.2
+Brotli>=1.0.7
+PySocks>=1.6.8
+urllib3>=1.24.1
+defusedxml>=0.5.0
+cachetools>=4.0.0
 stem>=1.8.0
-typing-extensions>=3.10.0.0
-urllib3>=1.26.6
-Werkzeug>=2.0.1
-zipp>=3.5.0
-zope.event>=4.5.0
-zope.interface>=5.4.0

server.py

@@ -84,9 +84,12 @@ def proxy_site(env, start_response, video=False):
         else:
             response, cleanup_func = util.fetch_url_response(url, send_headers)
 
-        response_headers = response.getheaders()
+        response_headers = response.headers
         if isinstance(response_headers, urllib3._collections.HTTPHeaderDict):
             response_headers = response_headers.items()
+        if video:
+            response_headers = (list(response_headers)
+                                +[('Access-Control-Allow-Origin', '*')])
 
         if first_attempt:
             start_response(str(response.status) + ' ' + response.reason,
@@ -166,8 +169,8 @@ site_handlers = {
     'youtube-nocookie.com': yt_app,
     'youtu.be': youtu_be,
     'ytimg.com': proxy_site,
-    'yt3.ggpht.com': proxy_site,
-    'lh3.googleusercontent.com': proxy_site,
+    'ggpht.com': proxy_site,
+    'googleusercontent.com': proxy_site,
     'sponsor.ajay.app': proxy_site,
     'googlevideo.com': proxy_video,
 }
@@ -247,12 +250,14 @@ def site_dispatch(env, start_response):
 
 class FilteredRequestLog:
     '''Don't log noisy thumbnail and avatar requests'''
-    filter_re = re.compile(r"""(?x)^
-                            "GET /https://(i[.]ytimg[.]com/|
+    filter_re = re.compile(r'''(?x)
+                            "GET\ /https://(
+                            i[.]ytimg[.]com/|
                             www[.]youtube[.]com/data/subscription_thumbnails/|
                             yt3[.]ggpht[.]com/|
-                            www[.]youtube[.]com/api/timedtext).*" 200
-                            """)
+                            www[.]youtube[.]com/api/timedtext|
+                            [-\w]+[.]googlevideo[.]com/).*"\ (200|206)
+                            ''')
 
     def __init__(self):
         pass

settings.py

@@ -39,7 +39,7 @@ SETTINGS_INFO = collections.OrderedDict([
 
     ('tor_port', {
         'type': int,
-        'default': 9150,
+        'default': 9050,
         'comment': '',
         'category': 'network',
     }),
@@ -53,7 +53,7 @@ SETTINGS_INFO = collections.OrderedDict([
 
     ('port_number', {
         'type': int,
-        'default': 8080,
+        'default': 9010,
         'comment': '',
         'category': 'network',
     }),
@@ -151,18 +151,75 @@ For security reasons, enabling this is not recommended.''',
         'category': 'interface',
     }),
 
+    ('autoplay_videos', {
+        'type': bool,
+        'default': False,
+        'comment': '',
+        'category': 'playback',
+    }),
+
     ('default_resolution', {
         'type': int,
         'default': 720,
         'comment': '',
         'options': [
+            (144, '144p'),
+            (240, '240p'),
             (360, '360p'),
             (480, '480p'),
             (720, '720p'),
+            (1080, '1080p'),
+            (1440, '1440p'),
+            (2160, '2160p'),
         ],
         'category': 'playback',
     }),
 
+    ('codec_rank_av1', {
+        'type': int,
+        'default': 1,
+        'label': 'AV1 Codec Ranking',
+        'comment': '',
+        'options': [(1, '#1'), (2, '#2'), (3, '#3')],
+        'category': 'playback',
+    }),
+
+    ('codec_rank_vp', {
+        'type': int,
+        'default': 2,
+        'label': 'VP8/VP9 Codec Ranking',
+        'comment': '',
+        'options': [(1, '#1'), (2, '#2'), (3, '#3')],
+        'category': 'playback',
+    }),
+
+    ('codec_rank_h264', {
+        'type': int,
+        'default': 3,
+        'label': 'H.264 Codec Ranking',
+        'comment': '',
+        'options': [(1, '#1'), (2, '#2'), (3, '#3')],
+        'category': 'playback',
+        'description': (
+            'Which video codecs to prefer. Codecs given the same '
+            'ranking will use smaller file size as a tiebreaker.'
+        )
+    }),
+
+    ('prefer_uni_sources', {
+        'label': 'Use integrated sources',
+        'type': int,
+        'default': 1,
+        'comment': '',
+        'options': [
+            (0, 'Prefer not'),
+            (1, 'Prefer'),
+            (2, 'Always'),
+        ],
+        'category': 'playback',
+        'description': 'If set to Prefer or Always and the default resolution is set to 360p or 720p, uses the unified (integrated) video files which contain audio and video, with buffering managed by the browser. If set to prefer not, uses the separate audio and video files through custom buffer management in av-merge via MediaSource unless they are unavailable.',
+    }),
+
     ('use_video_player', {
         'type': int,
         'default': 1,
@@ -175,6 +232,20 @@ For security reasons, enabling this is not recommended.''',
         'category': 'interface',
     }),
 
+    ('use_video_download', {
+        'type': int,
+        'default': 0,
+        'comment': '',
+        'options': [
+            (0, 'Disabled'),
+            (1, 'Enabled'),
+        ],
+        'category': 'interface',
+        'comment': '''If enabled, you may incur legal issues with RIAA. Disabled by default.
+More info: https://torrentfreak.com/riaa-thwarts-youts-attempt-to-declare-youtube-ripping-legal-221002/
+Archive: https://archive.ph/OZQbN''',
+    }),
+
     ('proxy_images', {
         'label': 'Route images',
         'type': bool,
@@ -239,11 +310,16 @@ For security reasons, enabling this is not recommended.''',
         'comment': '',
     }),
 
-    ('gather_googlevideo_domains', {
+    ('include_shorts_in_subscriptions', {
         'type': bool,
-        'default': False,
-        'comment': '''Developer use to debug 403s''',
-        'hidden': True,
+        'default': 0,
+        'comment': '',
+    }),
+
+    ('include_shorts_in_channel', {
+        'type': bool,
+        'default': 1,
+        'comment': '',
     }),
 
     ('debugging_save_responses', {
@@ -255,14 +331,16 @@ For security reasons, enabling this is not recommended.''',
 
     ('settings_version', {
         'type': int,
-        'default': 3,
+        'default': 6,
         'comment': '''Do not change, remove, or comment out this value, or else your settings may be lost or corrupted''',
         'hidden': True,
     }),
 ])
 
 program_directory = os.path.dirname(os.path.realpath(__file__))
-acceptable_targets = SETTINGS_INFO.keys() | {'enable_comments', 'enable_related_videos'}
+acceptable_targets = SETTINGS_INFO.keys() | {
+    'enable_comments', 'enable_related_videos', 'preferred_video_codec'
+}
 
 
 def comment_string(comment):
@@ -309,9 +387,45 @@ def upgrade_to_3(settings_dict):
     return new_settings
 
 
+def upgrade_to_4(settings_dict):
+    new_settings = settings_dict.copy()
+    if 'preferred_video_codec' in settings_dict:
+        pref = settings_dict['preferred_video_codec']
+        if pref == 0:
+            new_settings['codec_rank_h264'] = 1
+            new_settings['codec_rank_vp'] = 2
+            new_settings['codec_rank_av1'] = 3
+        else:
+            new_settings['codec_rank_h264'] = 3
+            new_settings['codec_rank_vp'] = 2
+            new_settings['codec_rank_av1'] = 1
+        del new_settings['preferred_video_codec']
+    new_settings['settings_version'] = 4
+    return new_settings
+
+
+def upgrade_to_5(settings_dict):
+    new_settings = settings_dict.copy()
+    if 'prefer_uni_sources' in settings_dict:
+        new_settings['prefer_uni_sources'] = int(settings_dict['prefer_uni_sources'])
+    new_settings['settings_version'] = 5
+    return new_settings
+
+
+def upgrade_to_6(settings_dict):
+    new_settings = settings_dict.copy()
+    if 'gather_googlevideo_domains' in new_settings:
+        del new_settings['gather_googlevideo_domains']
+    new_settings['settings_version'] = 6
+    return new_settings
+
+
 upgrade_functions = {
     1: upgrade_to_2,
     2: upgrade_to_3,
+    3: upgrade_to_4,
+    4: upgrade_to_5,
+    5: upgrade_to_6,
 }
 
 
@@ -325,8 +439,8 @@ if os.path.isfile("settings.txt"):
     data_dir = os.path.normpath('./data')
 else:
     print("Running in non-portable mode")
-    settings_dir = os.path.expanduser(os.path.normpath("~/.youtube-local"))
-    data_dir = os.path.expanduser(os.path.normpath("~/.youtube-local/data"))
+    settings_dir = os.path.expanduser(os.path.normpath("~/.yt-local"))
+    data_dir = os.path.expanduser(os.path.normpath("~/.yt-local/data"))
     if not os.path.exists(settings_dir):
         os.makedirs(settings_dir)
 

youtube/__init__.py

@@ -54,7 +54,10 @@ def commatize(num):
     if num is None:
         return ''
     if isinstance(num, str):
-        num = int(num)
+        try:
+            num = int(num)
+        except ValueError:
+            return num
     return '{:,}'.format(num)
 
 
@@ -115,7 +118,18 @@ def error_page(e):
                     error_message=exc_info()[1].error_message,
                     slim=slim
                 ), 502)
-    return flask.render_template('error.html', traceback=traceback.format_exc(), slim=slim), 500
+    elif (exc_info()[0] == util.FetchError
+        and exc_info()[1].code == '404'
+    ):
+        error_message = ('Error: The page you are looking for isn\'t here.')
+        return flask.render_template('error.html',
+                                     error_code=exc_info()[1].code,
+                                     error_message=error_message,
+                                     slim=slim), 404
+    return flask.render_template('error.html', traceback=traceback.format_exc(),
+                                 error_code=exc_info()[1].code,
+                                 slim=slim), 500
+    # return flask.render_template('error.html', traceback=traceback.format_exc(), slim=slim), 500
 
 
 font_choices = {

youtube/channel.py

@@ -1,6 +1,8 @@
 import base64
-from youtube import util, yt_data_extract, local_playlist, subscriptions
+from youtube import (util, yt_data_extract, local_playlist, subscriptions,
+                     playlist)
 from youtube import yt_app
+import settings
 
 import urllib
 import json
@@ -31,6 +33,132 @@ headers_mobile = (
 real_cookie = (('Cookie', 'VISITOR_INFO1_LIVE=8XihrAcN1l4'),)
 generic_cookie = (('Cookie', 'VISITOR_INFO1_LIVE=ST1Ti53r4fU'),)
 
+# added an extra nesting under the 2nd base64 compared to v4
+# added tab support
+# changed offset field to uint id 1
+def channel_ctoken_v5(channel_id, page, sort, tab, view=1):
+    new_sort = (2 if int(sort) == 1 else 1)
+    offset = 30*(int(page) - 1)
+    if tab == 'videos':
+        tab = 15
+    elif tab == 'shorts':
+        tab = 10
+    elif tab == 'streams':
+        tab = 14
+    pointless_nest = proto.string(80226972,
+        proto.string(2, channel_id)
+        + proto.string(3,
+            proto.percent_b64encode(
+                proto.string(110,
+                    proto.string(3,
+                        proto.string(tab,
+                            proto.string(1,
+                                proto.string(1,
+                                    proto.unpadded_b64encode(
+                                        proto.string(1,
+                                        proto.string(1,
+                                            proto.unpadded_b64encode(
+                                                proto.string(2,
+                                                    b"ST:"
+                                                    + proto.unpadded_b64encode(
+                                                        proto.uint(1, offset)
+                                                    )
+                                                )
+                                            )
+                                        )
+                                        )
+                                    )
+                                )
+                                 # targetId, just needs to be present but
+                                 # doesn't need to be correct
+                                + proto.string(2, "63faaff0-0000-23fe-80f0-582429d11c38")
+                            )
+                            # 1 - newest, 2 - popular
+                            + proto.uint(3, new_sort)
+                        )
+                    )
+                )
+            )
+        )
+    )
+
+    return base64.urlsafe_b64encode(pointless_nest).decode('ascii')
+
+
+def channel_about_ctoken(channel_id):
+    return proto.make_protobuf(
+        ('base64p',
+         [
+          [2, 80226972,
+           [
+            [2, 2, channel_id],
+            [2, 3,
+             ('base64p',
+              [
+               [2, 110,
+                [
+                 [2, 3,
+                  [
+                   [2, 19,
+                    [
+                     [2, 1, b'66b0e9e9-0000-2820-9589-582429a83980'],
+                    ]
+                   ],
+                  ]
+                 ],
+                ]
+               ],
+              ]
+             )
+            ],
+           ]
+          ],
+         ]
+        )
+    )
+
+
+# https://github.com/user234683/youtube-local/issues/151
+def channel_ctoken_v4(channel_id, page, sort, tab, view=1):
+    new_sort = (2 if int(sort) == 1 else 1)
+    offset = str(30*(int(page) - 1))
+    pointless_nest = proto.string(80226972,
+        proto.string(2, channel_id)
+        + proto.string(3,
+            proto.percent_b64encode(
+                proto.string(110,
+                    proto.string(3,
+                        proto.string(15,
+                            proto.string(1,
+                                proto.string(1,
+                                    proto.unpadded_b64encode(
+                                        proto.string(1,
+                                            proto.unpadded_b64encode(
+                                                proto.string(2,
+                                                    b"ST:"
+                                                    + proto.unpadded_b64encode(
+                                                        proto.string(2, offset)
+                                                    )
+                                                )
+                                            )
+                                        )
+                                    )
+                                )
+                                 # targetId, just needs to be present but
+                                 # doesn't need to be correct
+                                + proto.string(2, "63faaff0-0000-23fe-80f0-582429d11c38")
+                            )
+                            # 1 - newest, 2 - popular
+                            + proto.uint(3, new_sort)
+                        )
+                    )
+                )
+            )
+        )
+    )
+
+    return base64.urlsafe_b64encode(pointless_nest).decode('ascii')
+
 # SORT:
 # videos:
 #    Popular - 1
@@ -75,15 +203,15 @@ def channel_ctoken_v2(channel_id, page, sort, tab, view=1):
         2: 17254859483345278706,
         1: 16570086088270825023,
     }[int(sort)]
-    page_token = proto.string(61, proto.unpadded_b64encode(
-        proto.string(1, proto.uint(1, schema_number) + proto.string(
-            2,
-            proto.string(1, proto.unpadded_b64encode(proto.uint(1, offset)))
-        ))))
+    page_token = proto.string(61, proto.unpadded_b64encode(proto.string(1,
+            proto.uint(1, schema_number) + proto.string(2,
+                proto.string(1, proto.unpadded_b64encode(proto.uint(1,offset)))
+            )
+    )))
 
     tab = proto.string(2, tab)
     sort = proto.uint(3, int(sort))
-    # page = proto.string(15, str(page) )
+    #page = proto.string(15, str(page))
 
     shelf_view = proto.uint(4, 0)
     view = proto.uint(6, int(view))
@@ -118,8 +246,12 @@ def get_channel_tab(channel_id, page="1", sort=3, tab='videos', view=1,
     message = 'Got channel tab' if print_status else None
 
     if not ctoken:
-        ctoken = channel_ctoken_v3(channel_id, page, sort, tab, view)
+        if tab in ('videos', 'shorts', 'streams'):
+            ctoken = channel_ctoken_v5(channel_id, page, sort, tab, view)
+        else:
+            ctoken = channel_ctoken_v3(channel_id, page, sort, tab, view)
         ctoken = ctoken.replace('=', '%3D')
+
     # Not sure what the purpose of the key is or whether it will change
     # For now it seems to be constant for the API endpoint, not dependent
     # on the browsing session or channel
@@ -132,7 +264,7 @@ def get_channel_tab(channel_id, page="1", sort=3, tab='videos', view=1,
                 'hl': 'en',
                 'gl': 'US',
                 'clientName': 'WEB',
-                'clientVersion': '2.20180830',
+                'clientVersion': '2.20240327.00.00',
             },
         },
         'continuation': ctoken,
@@ -147,7 +279,8 @@ def get_channel_tab(channel_id, page="1", sort=3, tab='videos', view=1,
 
 
 # cache entries expire after 30 minutes
-@cachetools.func.ttl_cache(maxsize=128, ttl=30*60)
+number_of_videos_cache = cachetools.TTLCache(128, 30*60)
+@cachetools.cached(number_of_videos_cache)
 def get_number_of_videos_channel(channel_id):
     if channel_id is None:
         return 1000
@@ -159,7 +292,7 @@ def get_number_of_videos_channel(channel_id):
     try:
         response = util.fetch_url(url, headers_mobile,
             debug_name='number_of_videos', report_text='Got number of videos')
-    except urllib.error.HTTPError as e:
+    except (urllib.error.HTTPError, util.FetchError) as e:
         traceback.print_exc()
         print("Couldn't retrieve number of videos")
         return 1000
@@ -172,18 +305,20 @@ def get_number_of_videos_channel(channel_id):
         return int(match.group(1).replace(',',''))
     else:
         return 0
+def set_cached_number_of_videos(channel_id, num_videos):
+    @cachetools.cached(number_of_videos_cache)
+    def dummy_func_using_same_cache(channel_id):
+        return num_videos
+    dummy_func_using_same_cache(channel_id)
 
 
 channel_id_re = re.compile(r'videos\.xml\?channel_id=([a-zA-Z0-9_-]{24})"')
-
-
 @cachetools.func.lru_cache(maxsize=128)
 def get_channel_id(base_url):
     # method that gives the smallest possible response at ~4 kb
     # needs to be as fast as possible
     base_url = base_url.replace('https://www', 'https://m') # avoid redirect
-    response = util.fetch_url(
-        base_url + '/about?pbj=1', headers_mobile,
+    response = util.fetch_url(base_url + '/about?pbj=1', headers_mobile,
         debug_name='get_channel_id', report_text='Got channel id').decode('utf-8')
     match = channel_id_re.search(response)
     if match:
@@ -191,6 +326,31 @@ def get_channel_id(base_url):
     return None
 
 
+metadata_cache = cachetools.LRUCache(128)
+@cachetools.cached(metadata_cache)
+def get_metadata(channel_id):
+    base_url = 'https://www.youtube.com/channel/' + channel_id
+    polymer_json = util.fetch_url(base_url + '/about?pbj=1',
+                                  headers_desktop,
+                                  debug_name='gen_channel_about',
+                                  report_text='Retrieved channel metadata')
+    info = yt_data_extract.extract_channel_info(json.loads(polymer_json),
+                                                'about',
+                                                continuation=False)
+    return extract_metadata_for_caching(info)
+def set_cached_metadata(channel_id, metadata):
+    @cachetools.cached(metadata_cache)
+    def dummy_func_using_same_cache(channel_id):
+        return metadata
+    dummy_func_using_same_cache(channel_id)
+def extract_metadata_for_caching(channel_info):
+    metadata = {}
+    for key in ('approx_subscriber_count', 'short_description', 'channel_name',
+                'avatar'):
+        metadata[key] = channel_info[key]
+    return metadata
+
+
 def get_number_of_videos_general(base_url):
     return get_number_of_videos_channel(get_channel_id(base_url))
 
@@ -211,7 +371,7 @@ def get_channel_search_json(channel_id, query, page):
                 'hl': 'en',
                 'gl': 'US',
                 'clientName': 'WEB',
-                'clientVersion': '2.20180830',
+                'clientVersion': '2.20240327.00.00',
             },
         },
         'continuation': ctoken,
@@ -229,19 +389,20 @@ def post_process_channel_info(info):
     info['avatar'] = util.prefix_url(info['avatar'])
     info['channel_url'] = util.prefix_url(info['channel_url'])
     for item in info['items']:
+        item['thumbnail'] = "https://i.ytimg.com/vi/{}/hqdefault.jpg".format(item['id'])
         util.prefix_urls(item)
         util.add_extra_html_info(item)
     if info['current_tab'] == 'about':
         for i, (text, url) in enumerate(info['links']):
-            if util.YOUTUBE_URL_RE.fullmatch(url):
+            if isinstance(url, str) and util.YOUTUBE_URL_RE.fullmatch(url):
                 info['links'][i] = (text, util.prefix_url(url))
 
 
-def get_channel_first_page(base_url=None, channel_id=None):
+def get_channel_first_page(base_url=None, tab='videos', channel_id=None):
     if channel_id:
         base_url = 'https://www.youtube.com/channel/' + channel_id
-    return util.fetch_url(base_url + '/videos?pbj=1&view=0', headers_desktop,
-                          debug_name='gen_channel_videos')
+    return util.fetch_url(base_url + '/' + tab + '?pbj=1&view=0',
+                          headers_desktop, debug_name='gen_channel_' + tab)
 
 
 playlist_sort_codes = {'2': "da", '3': "dd", '4': "lad"}
@@ -250,63 +411,159 @@ playlist_sort_codes = {'2': "da", '3': "dd", '4': "lad"}
 # youtube.com/user/[username]/[tab]
 # youtube.com/c/[custom]/[tab]
 # youtube.com/[custom]/[tab]
-
-
 def get_channel_page_general_url(base_url, tab, request, channel_id=None):
 
     page_number = int(request.args.get('page', 1))
-    sort = request.args.get('sort', '3')
+    # sort 1: views
+    # sort 2: oldest
+    # sort 3: newest
+    # sort 4: newest - no shorts (Just a kludge on our end, not internal to yt)
+    default_sort = '3' if settings.include_shorts_in_channel else '4'
+    sort = request.args.get('sort', default_sort)
     view = request.args.get('view', '1')
     query = request.args.get('query', '')
     ctoken = request.args.get('ctoken', '')
-    default_params = (page_number == 1 and sort == '3' and view == '1')
+    include_shorts = (sort != '4')
+    default_params = (page_number == 1 and sort in ('3', '4') and view == '1')
+    continuation = bool(ctoken) # whether or not we're using a continuation
+    page_size = 30
+    try_channel_api = True
+    polymer_json = None
 
-    if tab == 'videos' and channel_id and not default_params:
-        tasks = (
-            gevent.spawn(get_number_of_videos_channel, channel_id),
-            gevent.spawn(get_channel_tab, channel_id, page_number, sort,
-                         'videos', view, ctoken)
-        )
-        gevent.joinall(tasks)
-        util.check_gevent_exceptions(*tasks)
-        number_of_videos, polymer_json = tasks[0].value, tasks[1].value
-    elif tab == 'videos':
+    # Use the special UU playlist which contains all the channel's uploads
+    if tab == 'videos' and sort in ('3', '4'):
+        if not channel_id:
+            channel_id = get_channel_id(base_url)
+        if page_number == 1 and include_shorts:
+            tasks = (
+                gevent.spawn(playlist.playlist_first_page,
+                             'UU' + channel_id[2:],
+                             report_text='Retrieved channel videos'),
+                gevent.spawn(get_metadata, channel_id),
+            )
+            gevent.joinall(tasks)
+            util.check_gevent_exceptions(*tasks)
+
+            # Ignore the metadata for now, it is cached and will be
+            # recalled later
+            pl_json = tasks[0].value
+            pl_info = yt_data_extract.extract_playlist_info(pl_json)
+            number_of_videos = pl_info['metadata']['video_count']
+            if number_of_videos is None:
+                number_of_videos = 1000
+            else:
+                set_cached_number_of_videos(channel_id, number_of_videos)
+        else:
+            tasks = (
+                gevent.spawn(playlist.get_videos, 'UU' + channel_id[2:],
+                             page_number, include_shorts=include_shorts),
+                gevent.spawn(get_metadata, channel_id),
+                gevent.spawn(get_number_of_videos_channel, channel_id),
+            )
+            gevent.joinall(tasks)
+            util.check_gevent_exceptions(*tasks)
+
+            pl_json = tasks[0].value
+            pl_info = yt_data_extract.extract_playlist_info(pl_json)
+            number_of_videos = tasks[2].value
+
+        info = pl_info
+        info['channel_id'] = channel_id
+        info['current_tab'] = 'videos'
+        if info['items']:   # Success
+            page_size = 100
+            try_channel_api = False
+        else:   # Try the first-page method next
+            try_channel_api = True
+
+    # Use the regular channel API
+    if tab in ('shorts', 'streams') or (tab=='videos' and try_channel_api):
         if channel_id:
             num_videos_call = (get_number_of_videos_channel, channel_id)
         else:
             num_videos_call = (get_number_of_videos_general, base_url)
+
+        # Use ctoken method, which YouTube changes all the time
+        if channel_id and not default_params:
+            if sort == 4:
+                _sort = 3
+            else:
+                _sort = sort
+            page_call = (get_channel_tab, channel_id, page_number, _sort,
+                         tab, view, ctoken)
+        # Use the first-page method, which won't break
+        else:
+            page_call = (get_channel_first_page, base_url, tab)
+
         tasks = (
             gevent.spawn(*num_videos_call),
-            gevent.spawn(get_channel_first_page, base_url=base_url),
+            gevent.spawn(*page_call),
         )
         gevent.joinall(tasks)
         util.check_gevent_exceptions(*tasks)
         number_of_videos, polymer_json = tasks[0].value, tasks[1].value
+
     elif tab == 'about':
-        polymer_json = util.fetch_url(base_url + '/about?pbj=1', headers_desktop, debug_name='gen_channel_about')
+        # polymer_json = util.fetch_url(base_url + '/about?pbj=1', headers_desktop, debug_name='gen_channel_about')
+        channel_id = get_channel_id(base_url)
+        ctoken = channel_about_ctoken(channel_id)
+        polymer_json = util.call_youtube_api('web', 'browse', {
+            'continuation': ctoken,
+        })
+        continuation=True
     elif tab == 'playlists' and page_number == 1:
         polymer_json = util.fetch_url(base_url+ '/playlists?pbj=1&view=1&sort=' + playlist_sort_codes[sort], headers_desktop, debug_name='gen_channel_playlists')
     elif tab == 'playlists':
         polymer_json = get_channel_tab(channel_id, page_number, sort,
                                        'playlists', view)
+        continuation = True
     elif tab == 'search' and channel_id:
         polymer_json = get_channel_search_json(channel_id, query, page_number)
     elif tab == 'search':
         url = base_url + '/search?pbj=1&query=' + urllib.parse.quote(query, safe='')
         polymer_json = util.fetch_url(url, headers_desktop, debug_name='gen_channel_search')
+    elif tab == 'videos':
+        pass
     else:
         flask.abort(404, 'Unknown channel tab: ' + tab)
 
-    info = yt_data_extract.extract_channel_info(json.loads(polymer_json), tab)
+    if polymer_json is not None:
+        info = yt_data_extract.extract_channel_info(
+            json.loads(polymer_json), tab, continuation=continuation
+        )
+
     if info['error'] is not None:
         return flask.render_template('error.html', error_message=info['error'])
 
-    post_process_channel_info(info)
-    if tab == 'videos':
+    if channel_id:
+        info['channel_url'] = 'https://www.youtube.com/channel/' + channel_id
+        info['channel_id'] = channel_id
+    else:
+        channel_id = info['channel_id']
+
+    # Will have microformat present, cache metadata while we have it
+    if channel_id and default_params and tab not in ('videos', 'about'):
+        metadata = extract_metadata_for_caching(info)
+        set_cached_metadata(channel_id, metadata)
+    # Otherwise, populate with our (hopefully cached) metadata
+    elif channel_id and info.get('channel_name') is None:
+        metadata = get_metadata(channel_id)
+        for key, value in metadata.items():
+            yt_data_extract.conservative_update(info, key, value)
+        # need to add this metadata to the videos/playlists
+        additional_info = {
+            'author': info['channel_name'],
+            'author_id': info['channel_id'],
+            'author_url': info['channel_url'],
+        }
+        for item in info['items']:
+            item.update(additional_info)
+
+    if tab in ('videos', 'shorts', 'streams'):
         info['number_of_videos'] = number_of_videos
-        info['number_of_pages'] = math.ceil(number_of_videos/30)
+        info['number_of_pages'] = math.ceil(number_of_videos/page_size)
         info['header_playlist_names'] = local_playlist.get_playlist_names()
-    if tab in ('videos', 'playlists'):
+    if tab in ('videos', 'shorts', 'streams', 'playlists'):
         info['current_sort'] = sort
     elif tab == 'search':
         info['search_box_value'] = query
@@ -315,9 +572,10 @@ def get_channel_page_general_url(base_url, tab, request, channel_id=None):
         info['page_number'] = page_number
     info['subscribed'] = subscriptions.is_subscribed(info['channel_id'])
 
-    return flask.render_template(
-        'channel.html',
-        parameters_dictionary=request.args,
+    post_process_channel_info(info)
+
+    return flask.render_template('channel.html',
+        parameters_dictionary = request.args,
         **info
     )
 

youtube/comments.py

@@ -78,7 +78,7 @@ def single_comment_ctoken(video_id, comment_id):
 
 def post_process_comments_info(comments_info):
     for comment in comments_info['comments']:
-        comment['author'] = strip_non_ascii(comment['author'])
+        comment['author'] = strip_non_ascii(comment['author']) if comment.get('author') else ""
         comment['author_url'] = concat_or_none(
             '/', comment['author_url'])
         comment['author_avatar'] = concat_or_none(
@@ -97,7 +97,7 @@ def post_process_comments_info(comments_info):
             ctoken = comment['reply_ctoken']
             ctoken, err = proto.set_protobuf_value(
                 ctoken,
-                'base64p', 6, 3, 9, value=250)
+                'base64p', 6, 3, 9, value=200)
             if err:
                 print('Error setting ctoken value:')
                 print(err)
@@ -127,7 +127,7 @@ def post_process_comments_info(comments_info):
             # change max_replies field to 250 in ctoken
             new_ctoken, err = proto.set_protobuf_value(
                 ctoken,
-                'base64p', 6, 3, 9, value=250)
+                'base64p', 6, 3, 9, value=200)
             if err:
                 print('Error setting ctoken value:')
                 print(err)
@@ -150,7 +150,7 @@ def post_process_comments_info(comments_info):
         util.URL_ORIGIN, '/watch?v=', comments_info['video_id'])
     comments_info['video_thumbnail'] = concat_or_none(
         settings.img_prefix, 'https://i.ytimg.com/vi/',
-        comments_info['video_id'], '/mqdefault.jpg'
+        comments_info['video_id'], '/hqdefault.jpg'
     )
 
 
@@ -189,10 +189,10 @@ def video_comments(video_id, sort=0, offset=0, lc='', secret_key=''):
                 comments_info['error'] += '\n\n' + e.error_message
             comments_info['error'] += '\n\nExit node IP address: %s' % e.ip
         else:
-            comments_info['error'] = 'YouTube blocked the request. IP address: %s' % e.ip
+            comments_info['error'] = 'YouTube blocked the request. Error: %s' % str(e)
 
     except Exception as e:
-        comments_info['error'] = 'YouTube blocked the request. IP address: %s' % e.ip
+        comments_info['error'] = 'YouTube blocked the request. Error: %s' % str(e)
 
     if comments_info.get('error'):
         print('Error retrieving comments for ' + str(video_id) + ':\n' +

youtube/get_app_version/get_app_version.py

@@ -11,17 +11,10 @@ import subprocess
 def app_version():
     def minimal_env_cmd(cmd):
         # make minimal environment
-        env = {}
-        for k in ['SYSTEMROOT', 'PATH']:
-            v = os.environ.get(k)
-            if v is not None:
-                env[k] = v
+        env = {k: os.environ[k] for k in ['SYSTEMROOT', 'PATH'] if k in os.environ}
+        env.update({'LANGUAGE': 'C', 'LANG': 'C', 'LC_ALL': 'C'})
 
-        env['LANGUAGE'] = 'C'
-        env['LANG'] = 'C'
-        env['LC_ALL'] = 'C'
-        out = subprocess.Popen(
-            cmd, stdout=subprocess.PIPE, env=env).communicate()[0]
+        out = subprocess.Popen(cmd, stdout=subprocess.PIPE, env=env).communicate()[0]
         return out
 
     subst_list = {
@@ -31,24 +24,21 @@ def app_version():
     }
 
     if os.system("command -v git > /dev/null 2>&1") != 0:
-        subst_list
-    else:
-        if call(["git", "branch"], stderr=STDOUT,
-                stdout=open(os.devnull, 'w')) != 0:
-            subst_list
-        else:
-            # version
-            describe = minimal_env_cmd(["git", "describe", "--always"])
-            git_revision = describe.strip().decode('ascii')
-            # branch
-            branch = minimal_env_cmd(["git", "branch"])
-            git_branch = branch.strip().decode('ascii').replace('* ', '')
+        return subst_list
 
-            subst_list = {
-                "version": __version__,
-                "branch": git_branch,
-                "commit": git_revision
-            }
+    if call(["git", "branch"], stderr=STDOUT, stdout=open(os.devnull, 'w')) != 0:
+        return subst_list
+
+    describe = minimal_env_cmd(["git", "describe", "--tags", "--always"])
+    git_revision = describe.strip().decode('ascii')
+
+    branch = minimal_env_cmd(["git", "branch"])
+    git_branch = branch.strip().decode('ascii').replace('* ', '')
+
+    subst_list.update({
+        "branch": git_branch,
+        "commit": git_revision
+    })
 
     return subst_list
 

youtube/local_playlist.py

@@ -153,6 +153,12 @@ def path_edit_playlist(playlist_name):
         number_of_videos_remaining = remove_from_playlist(playlist_name, videos_to_remove)
         redirect_page_number = min(int(request.values.get('page', 1)), math.ceil(number_of_videos_remaining/50))
         return flask.redirect(util.URL_ORIGIN + request.path + '?page=' + str(redirect_page_number))
+    elif request.values['action'] == 'remove_playlist':
+        try:
+            os.remove(os.path.join(playlists_directory, playlist_name + ".txt"))
+        except OSError:
+            pass
+        return flask.redirect(util.URL_ORIGIN + '/playlists')
     elif request.values['action'] == 'export':
         videos = read_playlist(playlist_name)
         fmt = request.values['export_format']

youtube/playlist.py

@@ -12,12 +12,13 @@ from flask import request
 import flask
 
 
-def playlist_ctoken(playlist_id, offset):
+def playlist_ctoken(playlist_id, offset, include_shorts=True):
 
     offset = proto.uint(1, offset)
-    # this is just obfuscation as far as I can tell. It doesn't even follow protobuf
     offset = b'PT:' + proto.unpadded_b64encode(offset)
     offset = proto.string(15, offset)
+    if not include_shorts:
+        offset += proto.string(104, proto.uint(2, 1))
 
     continuation_info = proto.string(3, proto.percent_b64encode(offset))
 
@@ -26,47 +27,46 @@ def playlist_ctoken(playlist_id, offset):
 
     return base64.urlsafe_b64encode(pointless_nest).decode('ascii')
 
-# initial request types:
-#   polymer_json: https://m.youtube.com/playlist?list=PLv3TTBr1W_9tppikBxAE_G6qjWdBljBHJ&pbj=1&lact=0
-#   ajax json:    https://m.youtube.com/playlist?list=PLv3TTBr1W_9tppikBxAE_G6qjWdBljBHJ&pbj=1&lact=0 with header X-YouTube-Client-Version: 1.20180418
 
-
-# continuation request types:
-#   polymer_json: https://m.youtube.com/playlist?&ctoken=[...]&pbj=1
-#   ajax json:    https://m.youtube.com/playlist?action_continuation=1&ajax=1&ctoken=[...]
-
-
-headers_1 = (
-    ('Accept', '*/*'),
-    ('Accept-Language', 'en-US,en;q=0.5'),
-    ('X-YouTube-Client-Name', '2'),
-    ('X-YouTube-Client-Version', '2.20180614'),
-)
-
-
-def playlist_first_page(playlist_id, report_text="Retrieved playlist"):
-    url = 'https://m.youtube.com/playlist?list=' + playlist_id + '&pbj=1'
-    content = util.fetch_url(url, util.mobile_ua + headers_1, report_text=report_text, debug_name='playlist_first_page')
-    content = json.loads(content.decode('utf-8'))
+def playlist_first_page(playlist_id, report_text="Retrieved playlist",
+                        use_mobile=False):
+    if use_mobile:
+        url = 'https://m.youtube.com/playlist?list=' + playlist_id + '&pbj=1'
+        content = util.fetch_url(
+            url, util.mobile_xhr_headers,
+            report_text=report_text, debug_name='playlist_first_page'
+        )
+        content = json.loads(content.decode('utf-8'))
+    else:
+        url = 'https://www.youtube.com/playlist?list=' + playlist_id + '&pbj=1'
+        content = util.fetch_url(
+            url, util.desktop_xhr_headers,
+            report_text=report_text, debug_name='playlist_first_page'
+        )
+        content = json.loads(content.decode('utf-8'))
 
     return content
 
 
-#https://m.youtube.com/playlist?itct=CBMQybcCIhMIptj9xJaJ2wIV2JKcCh3Idwu-&ctoken=4qmFsgI2EiRWTFBMT3kwajlBdmxWWlB0bzZJa2pLZnB1MFNjeC0tN1BHVEMaDmVnWlFWRHBEUWxFJTNE&pbj=1
-def get_videos(playlist_id, page):
-
-    url = "https://m.youtube.com/playlist?ctoken=" + playlist_ctoken(playlist_id, (int(page)-1)*20) + "&pbj=1"
-    headers = {
-        'User-Agent': '  Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1',
-        'Accept': '*/*',
-        'Accept-Language': 'en-US,en;q=0.5',
-        'X-YouTube-Client-Name': '2',
-        'X-YouTube-Client-Version': '2.20180508',
-    }
+def get_videos(playlist_id, page, include_shorts=True, use_mobile=False,
+               report_text='Retrieved playlist'):
+    # mobile requests return 20 videos per page
+    if use_mobile:
+        page_size = 20
+        headers = util.mobile_xhr_headers
+    # desktop requests return 100 videos per page
+    else:
+        page_size = 100
+        headers = util.desktop_xhr_headers
 
+    url = "https://m.youtube.com/playlist?ctoken="
+    url += playlist_ctoken(playlist_id, (int(page)-1)*page_size,
+                           include_shorts=include_shorts)
+    url += "&pbj=1"
     content = util.fetch_url(
-        url, headers,
-        report_text="Retrieved playlist", debug_name='playlist_videos')
+        url, headers, report_text=report_text,
+        debug_name='playlist_videos'
+    )
 
     info = json.loads(content.decode('utf-8'))
     return info
@@ -85,7 +85,10 @@ def get_playlist_page():
         this_page_json = first_page_json
     else:
         tasks = (
-            gevent.spawn(playlist_first_page, playlist_id, report_text="Retrieved playlist info" ),
+            gevent.spawn(
+                playlist_first_page, playlist_id,
+                report_text="Retrieved playlist info", use_mobile=True
+            ),
             gevent.spawn(get_videos, playlist_id, page)
         )
         gevent.joinall(tasks)
@@ -104,7 +107,7 @@ def get_playlist_page():
         util.prefix_urls(item)
         util.add_extra_html_info(item)
         if 'id' in item:
-            item['thumbnail'] = settings.img_prefix + 'https://i.ytimg.com/vi/' + item['id'] + '/default.jpg'
+            item['thumbnail'] = f"{settings.img_prefix}https://i.ytimg.com/vi/{item['id']}/hqdefault.jpg"
 
         item['url'] += '&list=' + playlist_id
         if item['index']:
@@ -112,13 +115,13 @@ def get_playlist_page():
 
     video_count = yt_data_extract.deep_get(info, 'metadata', 'video_count')
     if video_count is None:
-        video_count = 40
+        video_count = 1000
 
     return flask.render_template(
         'playlist.html',
         header_playlist_names=local_playlist.get_playlist_names(),
         video_list=info.get('items', []),
-        num_pages=math.ceil(video_count/20),
+        num_pages=math.ceil(video_count/100),
         parameters_dictionary=request.args,
 
         **info['metadata']

youtube/proto.py

@@ -141,6 +141,17 @@ base64_enc_funcs = {
 
 
 def _make_protobuf(data):
+    '''
+    Input: Recursive list of protobuf objects or base-64 encodings
+    Output: Protobuf bytestring
+    Each protobuf object takes the form [wire_type, field_number, field_data]
+    If a string protobuf has a list/tuple of length 2, this has the form
+    (base64 type, data)
+    The base64 types are
+    - base64 means a base64 encode with equals sign paddings
+    - base64s means a base64 encode without padding
+    - base64p means a url base64 encode with equals signs replaced with %3D
+    '''
     # must be dict mapping field_number to [wire_type, value]
     if isinstance(data, dict):
         new_data = []

youtube/search.py

@@ -64,6 +64,8 @@ def get_search_page():
     query = request.args.get('search_query') or request.args.get('query')
     if query is None:
         return flask.render_template('home.html', title='Search')
+    elif query.startswith('https://www.youtube.com') or query.startswith('https://www.youtu.be'):
+         return flask.redirect(f'/{query}')
 
     page = request.args.get("page", "1")
     autocorrect = int(request.args.get("autocorrect", "1"))

youtube/static/channel.css

@@ -33,6 +33,8 @@ input[type="search"] {
     padding: 0.4rem 0.4rem;
     font-size: 15px;
     color: var(--search-text);
+    outline: none;
+    box-shadow: none;
 }
 
 input[type='search'] {
@@ -200,8 +202,10 @@ label[for=options-toggle-cbox] {
 }
 
 #options-toggle-cbox:checked ~ .dropdown-content {
-    display: inline-grid;
+    display: block;
     white-space: nowrap;
+    background: var(--secondary-background);
+    padding: 0.5rem 1rem;
 }
 /*- ----------- End Menu Mobile sin JS ------------- */
 
@@ -252,7 +256,8 @@ hr {
     padding-top: 6px;
     text-align: center;
     white-space: nowrap;
-    border: none;
+    border: 1px solid;
+    border-color: var(--button-border);
     border-radius: 0.2rem;
 }
 
@@ -504,15 +509,19 @@ hr {
     .dropdown {
         display: grid;
         grid-gap: 1px;
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: 100px auto;
         grid-template-areas:
             "dropdown-label"
             "dropdown-content";
         grid-area: dropdown;
-        background: var(--background);
-        padding-right: 4rem;
-        z-index: 1;
         position: absolute;
+        z-index: 1;
+    }
+
+    #options-toggle-cbox:checked ~ .dropdown-content {
+        width: calc(100% + 100px);
+        max-height: 80vh;
+        overflow-y: scroll;
     }
 
     .author-container {
@@ -528,7 +537,7 @@ hr {
         grid-area: playlist;
     }
     .play-clean {
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: 100px auto;
     }
     .play-clean > button {
         padding-left: 0px;

youtube/static/comments.css

@@ -39,6 +39,8 @@ input[type="search"] {
     padding: 0.4rem 0.4rem;
     font-size: 15px;
     color: var(--search-text);
+    outline: none;
+    box-shadow: none;
 }
 
 input[type='search'] {
@@ -105,9 +107,7 @@ header {
         "dropdown-label"
         "dropdown-content";
     grid-area: dropdown;
-    background: var(--background);
     padding-right: 4rem;
-    z-index: 1;
 }
 .dropdown-label {
     grid-area: dropdown-label;
@@ -272,7 +272,7 @@ label[for=options-toggle-cbox] {
     .dropdown {
         display: grid;
         grid-gap: 1px;
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: minmax(50px, 120px);
         grid-template-areas:
             "dropdown-label"
             "dropdown-content";
@@ -280,6 +280,12 @@ label[for=options-toggle-cbox] {
         z-index: 1;
         position: absolute;
     }
+    #options-toggle-cbox:checked ~ .dropdown-content {
+        padding: 0rem 3rem 1rem 1rem;
+        width: 100%;
+        max-height: 45vh;
+        overflow-y: scroll;
+    }
 
     .footer {
         display: grid;

youtube/static/dark_theme.css

@@ -1,19 +1,22 @@
 :root {
-    --background:           #212121;
+    --background:           #121113;
     --text:                 #FFFFFF;
-    --secondary-hover:      #73828c;
-    --secondary-focus:      #606060;
-    --secondary-inverse:    #FFF;
-    --primary-background:   #757575;
-    --secondary-background: #424242;
-    --thumb-background:     #757575;
+    --secondary-hover:      #222222;
+    --secondary-focus:      #121113;
+    --secondary-inverse:    #FFFFFF;
+    --primary-background:   #242424;
+    --secondary-background: #222222;
+    --thumb-background:     #222222;
     --link:                 #00B0FF;
     --link-visited:         #40C4FF;
-    --buttom:               #dcdcdb;
-    --buttom-text:          #415462;
-    --button-border:        #91918c;
-    --buttom-hover:         #BBB;
-    --search-text:          #FFF;
-    --time-background:      #212121;
-    --time-text:            #FFF;
+    --border-bg:            #222222;
+    --border-bg-settings:   #000000;
+    --border-bg-license:    #000000;
+    --buttom:               #121113;
+    --buttom-text:          #FFFFFF;
+    --button-border:        #222222;
+    --buttom-hover:         #222222;
+    --search-text:          #FFFFFF;
+    --time-background:      #121113;
+    --time-text:            #FFFFFF;
 }

youtube/static/gray_theme.css

@@ -1,18 +1,21 @@
 :root {
-    --background:            #2d3743;
+    --background:            #2D3743;
     --text:                  #FFFFFF;
-    --secondary-hover:       #73828c;
+    --secondary-hover:       #73828C;
     --secondary-focus:       rgba(115, 130, 140, 0.125);
     --secondary-inverse:     #FFFFFF;
-    --primary-background:    #2d3743;
+    --primary-background:    #2D3743;
     --secondary-background:  #102027;
     --thumb-background:      #35404D;
-    --link:                  #22aaff;
-    --link-visited:          #7755ff;
-    --buttom:                #DCDCDC;
-    --buttom-text:           #415462;
-    --button-border:         #91918c;
-    --buttom-hover:          #BBBBBB;
+    --link:                  #22AAFF;
+    --link-visited:          #7755FF;
+    --border-bg:             #FFFFFF;
+    --border-bg-settings:    #FFFFFF;
+    --border-bg-license:     #FFFFFF;
+    --buttom:                #2D3743;
+    --buttom-text:           #FFFFFF;
+    --button-border:         #102027;
+    --buttom-hover:          #102027;
     --search-text:           #FFFFFF;
     --time-background:       #212121;
     --time-text:             #FFFFFF;

youtube/static/home.css

@@ -29,6 +29,8 @@ input[type="search"] {
     padding: 0.4rem 0.4rem;
     font-size: 15px;
     color: var(--search-text);
+    outline: none;
+    box-shadow: none;
 }
 
 input[type='search'] {
@@ -95,7 +97,6 @@ header {
         "dropdown-label"
         "dropdown-content";
     grid-area: dropdown;
-    z-index: 1;
 }
 .dropdown-label {
     grid-area: dropdown-label;
@@ -135,8 +136,10 @@ label[for=options-toggle-cbox] {
 }
 
 #options-toggle-cbox:checked ~ .dropdown-content {
-    display: inline-grid;
+    display: block;
     white-space: nowrap;
+    background: var(--secondary-background);
+    padding: 0.5rem 1rem;
 }
 /*- ----------- End Menu Mobile sin JS ------------- */
 
@@ -186,15 +189,20 @@ label[for=options-toggle-cbox] {
     .dropdown {
         display: grid;
         grid-gap: 1px;
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: 100px auto;
         grid-template-areas:
             "dropdown-label"
             "dropdown-content";
         grid-area: dropdown;
-        z-index: 1;
         position: absolute;
         background: var(--background);
         padding-right: 4rem;
+        z-index: 1;
+    }
+    #options-toggle-cbox:checked ~ .dropdown-content {
+        width: calc(100% + 100px);
+        max-height: 80vh;
+        overflow-y: scroll;
     }
 
     .footer {

youtube/static/js/comments.js

@@ -1,9 +1,9 @@
 function onClickReplies(e) {
-  var details = e.target.parentElement;
+  let details = e.target.parentElement;
   // e.preventDefault();
   console.log("loading replies ..");
   doXhr(details.getAttribute("data-src") + "&slim=1", (html) => {
-    var div = details.querySelector(".comment_page");
+    let div = details.querySelector(".comment_page");
     div.innerHTML = html;
   });
   details.removeEventListener('click', onClickReplies);

youtube/static/js/common.js

@@ -1,16 +1,19 @@
 const Q = document.querySelector.bind(document);
 const QA = document.querySelectorAll.bind(document);
 const QId = document.getElementById.bind(document);
+let seconds,
+    minutes,
+    hours;
 function text(msg) { return document.createTextNode(msg); }
 function clearNode(node) { while (node.firstChild) node.removeChild(node.firstChild); }
 function toTimestamp(seconds) {
-  var seconds = Math.floor(seconds);
+  seconds = Math.floor(seconds);
 
-  var minutes = Math.floor(seconds/60);
-  var seconds = seconds % 60;
+  minutes = Math.floor(seconds/60);
+  seconds = seconds % 60;
 
-  var hours = Math.floor(minutes/60);
-  var minutes = minutes % 60;
+  hours = Math.floor(minutes/60);
+  minutes = minutes % 60;
 
   if (hours) {
     return `0${hours}:`.slice(-3) + `0${minutes}:`.slice(-3) + `0${seconds}`.slice(-2);
@@ -18,8 +21,7 @@ function toTimestamp(seconds) {
   return `0${minutes}:`.slice(-3) + `0${seconds}`.slice(-2);
 }
 
-
-var cur_track_idx = 0;
+let cur_track_idx = 0;
 function getActiveTranscriptTrackIdx() {
     let textTracks = QId("js-video-player").textTracks;
     if (!textTracks.length) return;
@@ -39,7 +41,7 @@ function getDefaultTranscriptTrackIdx() {
 }
 
 function doXhr(url, callback=null) {
-    var xhr = new XMLHttpRequest();
+    let xhr = new XMLHttpRequest();
     xhr.open("GET", url);
     xhr.onload = (e) => {
       callback(e.currentTarget.response);
@@ -50,7 +52,7 @@ function doXhr(url, callback=null) {
 
 // https://stackoverflow.com/a/30810322
 function copyTextToClipboard(text) {
-  var textArea = document.createElement("textarea");
+  let textArea = document.createElement("textarea");
 
   //
   // *** This styling is an extra step which is likely not required. ***
@@ -92,19 +94,20 @@ function copyTextToClipboard(text) {
 
   textArea.value = text;
 
-  document.body.appendChild(textArea);
+  let parent_el = video.parentElement;
+  parent_el.appendChild(textArea);
   textArea.focus();
   textArea.select();
 
   try {
-    var successful = document.execCommand('copy');
-    var msg = successful ? 'successful' : 'unsuccessful';
+    let successful = document.execCommand('copy');
+    let msg = successful ? 'successful' : 'unsuccessful';
     console.log('Copying text command was ' + msg);
   } catch (err) {
     console.log('Oops, unable to copy');
   }
 
-  document.body.removeChild(textArea);
+  parent_el.removeChild(textArea);
 }
 
 

youtube/static/js/hotkeys.js

@@ -3,6 +3,7 @@ function onKeyDown(e) {
 
     // console.log(e);
     let v = QId("js-video-player");
+    if (!e.isTrusted) return;  // plyr CustomEvent
     let c = e.key.toLowerCase();
     if (e.ctrlKey) return;
     else if (c == "k") {
@@ -26,8 +27,17 @@ function onKeyDown(e) {
     }
     else if (c == "f") {
         e.preventDefault();
-        if (document.fullscreenElement && document.fullscreenElement.nodeName == 'VIDEO') {document.exitFullscreen();}
-        else {v.requestFullscreen()};
+        if (data.settings.use_video_player == 2) {
+            player.fullscreen.toggle()
+        }
+        else {
+            if (document.fullscreen) {
+                document.exitFullscreen()
+            }
+            else {
+                v.requestFullscreen()
+            }
+        }
     }
     else if (c == "m") {
         if (v.muted == false) {v.muted = true;}

youtube/static/js/playlistadd.js

@@ -37,7 +37,7 @@
     }
     // https://developer.mozilla.org/en-US/docs/Learn/HTML/Forms/Sending_forms_through_JavaScript
     function sendData(event){
-        var clicked_button = document.activeElement;
+        let clicked_button = document.activeElement;
         if(clicked_button === null || clicked_button.getAttribute('type') !== 'submit' || clicked_button.parentElement != event.target){
             console.log('ERROR: clicked_button not valid');
             return;
@@ -46,8 +46,8 @@
             return;     // video(s) are being removed from playlist, just let it refresh the page
         }
         event.preventDefault();
-        var XHR = new XMLHttpRequest();
-        var FD = new FormData(playlistAddForm);
+        let XHR = new XMLHttpRequest();
+        let FD = new FormData(playlistAddForm);
 
         if(FD.getAll('video_info_list').length === 0){
             displayMessage('Error: No videos selected', true);

youtube/static/js/plyr-start.js

@@ -1,36 +1,136 @@
-let captionsActive;
+(function main() {
+  'use strict';
 
-switch(true) {
-case data.settings.subtitles_mode == 2:
-  captionsActive = true;
-  break;
-case data.settings.subtitles_mode == 1 && data.has_manual_captions:
-  captionsActive = true;
-  break;
-default:
-  captionsActive = false;
-}
+  // Captions
+  let captionsActive = false;
+  if (data.settings.subtitles_mode === 2 || (data.settings.subtitles_mode === 1 && data.has_manual_captions)) {
+    captionsActive = true;
+  }
 
-const player = new Plyr(document.getElementById('js-video-player'), {
-  disableContextMenu: false,
-  captions: {
-    active: captionsActive,
-    language: data.settings.subtitles_language,
-  },
-  controls: [
-    'play-large',
-    'play',
-    'progress',
-    'current-time',
-    'duration',
-    'mute',
-    'volume',
-    'captions',
-    'settings',
-    'fullscreen'
-  ],
-  iconUrl: "/youtube.com/static/modules/plyr/plyr.svg",
-  blankVideo: "/youtube.com/static/modules/plyr/blank.webm",
-  debug: false,
-  storage: {enabled: false}
-});
+  // AutoPlay
+  let autoplayActive = data.settings.autoplay_videos || false;
+
+  let qualityOptions = [];
+  let qualityDefault;
+
+  for (let src of data.uni_sources) {
+    qualityOptions.push(src.quality_string);
+  }
+
+  for (let src of data.pair_sources) {
+    qualityOptions.push(src.quality_string);
+  }
+
+  if (data.using_pair_sources) {
+    qualityDefault = data.pair_sources[data.pair_idx].quality_string;
+  } else if (data.uni_sources.length !== 0) {
+    qualityDefault = data.uni_sources[data.uni_idx].quality_string;
+  } else {
+    qualityDefault = 'None';
+  }
+
+  // Fix plyr refusing to work with qualities that are strings
+  Object.defineProperty(Plyr.prototype, 'quality', {
+    set: function (input) {
+      const config = this.config.quality;
+      const options = this.options.quality;
+      let quality = input;
+      let updateStorage = true;
+
+      if (!options.length) {
+        return;
+      }
+
+      if (!options.includes(quality)) {
+        return;
+      }
+
+      // Update config
+      config.selected = quality;
+
+      // Set quality
+      this.media.quality = quality;
+
+      // Save to storage
+      if (updateStorage) {
+        this.storage.set({ quality });
+      }
+    },
+  });
+
+  const playerOptions = {
+    // Learning about autoplay permission https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permissions-Policy/autoplay#syntax
+    autoplay: autoplayActive,
+    disableContextMenu: false,
+    captions: {
+      active: captionsActive,
+      language: data.settings.subtitles_language,
+    },
+    controls: [
+      'play-large',
+      'play',
+      'progress',
+      'current-time',
+      'duration',
+      'mute',
+      'volume',
+      'captions',
+      'settings',
+      'pip',
+      'airplay',
+      'fullscreen',
+    ],
+    iconUrl: '/youtube.com/static/modules/plyr/plyr.svg',
+    blankVideo: '/youtube.com/static/modules/plyr/blank.webm',
+    debug: false,
+    storage: { enabled: false },
+    quality: {
+      default: qualityDefault,
+      options: qualityOptions,
+      forced: true,
+      onChange: function (quality) {
+        if (quality == 'None') {
+          return;
+        }
+        if (quality.includes('(integrated)')) {
+          for (let i = 0; i < data.uni_sources.length; i++) {
+            if (data.uni_sources[i].quality_string == quality) {
+              changeQuality({ type: 'uni', index: i });
+              return;
+            }
+          }
+        } else {
+          for (let i = 0; i < data.pair_sources.length; i++) {
+            if (data.pair_sources[i].quality_string == quality) {
+              changeQuality({ type: 'pair', index: i });
+              return;
+            }
+          }
+        }
+      },
+    },
+    previewThumbnails: {
+      enabled: storyboard_url !== null,
+      src: [storyboard_url],
+    },
+    settings: ['captions', 'quality', 'speed', 'loop'],
+    tooltips: {
+      controls: true,
+    },
+  }
+
+  const player = new Plyr(document.getElementById('js-video-player'), playerOptions);
+
+  // disable double click to fullscreen
+  // https://github.com/sampotts/plyr/issues/1370#issuecomment-528966795
+  player.eventListeners.forEach(function(eventListener) {
+    if(eventListener.type === 'dblclick') {
+        eventListener.element.removeEventListener(eventListener.type, eventListener.callback, eventListener.options);
+    }
+  });
+
+  // Add .started property, true after the playback has been started
+  // Needed so controls won't be hidden before playback has started
+  player.started = false;
+  player.once('playing', function(){this.started = true});
+})();

youtube/static/js/speedyplay.js

@@ -1,13 +0,0 @@
-(function main() {
-    'use strict';
-    const video = document.getElementById('js-video-player');
-    const speedInput = document.getElementById('speed-control');
-    speedInput.addEventListener('keyup', (event) => {
-        if (event.key === 'Enter') {
-            let speed = parseFloat(speedInput.value);
-            if(!isNaN(speed)){
-                video.playbackRate = speed;
-            }
-        }
-    });
-}());

youtube/static/js/sponsorblock.js

@@ -2,7 +2,7 @@
 
 // from: https://git.gir.st/subscriptionfeed.git/blob/59a590d:/app/youtube/templates/watch.html.j2#l28
 
-var sha256=function a(b){function c(a,b){return a>>>b|a<<32-b}for(var d,e,f=Math.pow,g=f(2,32),h="length",i="",j=[],k=8*b[h],l=a.h=a.h||[],m=a.k=a.k||[],n=m[h],o={},p=2;64>n;p++)if(!o[p]){for(d=0;313>d;d+=p)o[d]=p;l[n]=f(p,.5)*g|0,m[n++]=f(p,1/3)*g|0}for(b+="\x80";b[h]%64-56;)b+="\x00";for(d=0;d<b[h];d++){if(e=b.charCodeAt(d),e>>8)return;j[d>>2]|=e<<(3-d)%4*8}for(j[j[h]]=k/g|0,j[j[h]]=k,e=0;e<j[h];){var q=j.slice(e,e+=16),r=l;for(l=l.slice(0,8),d=0;64>d;d++){var s=q[d-15],t=q[d-2],u=l[0],v=l[4],w=l[7]+(c(v,6)^c(v,11)^c(v,25))+(v&l[5]^~v&l[6])+m[d]+(q[d]=16>d?q[d]:q[d-16]+(c(s,7)^c(s,18)^s>>>3)+q[d-7]+(c(t,17)^c(t,19)^t>>>10)|0),x=(c(u,2)^c(u,13)^c(u,22))+(u&l[1]^u&l[2]^l[1]&l[2]);l=[w+x|0].concat(l),l[4]=l[4]+w|0}for(d=0;8>d;d++)l[d]=l[d]+r[d]|0}for(d=0;8>d;d++)for(e=3;e+1;e--){var y=l[d]>>8*e&255;i+=(16>y?0:"")+y.toString(16)}return i}; /*https://geraintluff.github.io/sha256/sha256.min.js (public domain)*/
+let sha256=function a(b){function c(a,b){return a>>>b|a<<32-b}for(var d,e,f=Math.pow,g=f(2,32),h="length",i="",j=[],k=8*b[h],l=a.h=a.h||[],m=a.k=a.k||[],n=m[h],o={},p=2;64>n;p++)if(!o[p]){for(d=0;313>d;d+=p)o[d]=p;l[n]=f(p,.5)*g|0,m[n++]=f(p,1/3)*g|0}for(b+="\x80";b[h]%64-56;)b+="\x00";for(d=0;d<b[h];d++){if(e=b.charCodeAt(d),e>>8)return;j[d>>2]|=e<<(3-d)%4*8}for(j[j[h]]=k/g|0,j[j[h]]=k,e=0;e<j[h];){var q=j.slice(e,e+=16),r=l;for(l=l.slice(0,8),d=0;64>d;d++){var s=q[d-15],t=q[d-2],u=l[0],v=l[4],w=l[7]+(c(v,6)^c(v,11)^c(v,25))+(v&l[5]^~v&l[6])+m[d]+(q[d]=16>d?q[d]:q[d-16]+(c(s,7)^c(s,18)^s>>>3)+q[d-7]+(c(t,17)^c(t,19)^t>>>10)|0),x=(c(u,2)^c(u,13)^c(u,22))+(u&l[1]^u&l[2]^l[1]&l[2]);l=[w+x|0].concat(l),l[4]=l[4]+w|0}for(d=0;8>d;d++)l[d]=l[d]+r[d]|0}for(d=0;8>d;d++)for(e=3;e+1;e--){var y=l[d]>>8*e&255;i+=(16>y?0:"")+y.toString(16)}return i}; /*https://geraintluff.github.io/sha256/sha256.min.js (public domain)*/
 
 window.addEventListener("load", load_sponsorblock);
 document.addEventListener('DOMContentLoaded', ()=>{

youtube/static/js/watch.js

@@ -0,0 +1,200 @@
+const video = document.getElementById('js-video-player');
+
+function changeQuality(selection) {
+    let currentVideoTime = video.currentTime;
+    let videoPaused = video.paused;
+    let videoSpeed = video.playbackRate;
+    let srcInfo;
+    if (avMerge && typeof avMerge.close === 'function') {
+        avMerge.close();
+    }
+    if (selection.type == 'uni'){
+        srcInfo = data['uni_sources'][selection.index];
+        video.src = srcInfo.url;
+    } else {
+        srcInfo = data['pair_sources'][selection.index];
+        avMerge = new AVMerge(video, srcInfo, currentVideoTime);
+    }
+    video.currentTime = currentVideoTime;
+    if (!videoPaused){
+        video.play();
+    }
+    video.playbackRate = videoSpeed;
+}
+
+// Initialize av-merge
+let avMerge;
+if (data.using_pair_sources) {
+    let srcPair = data['pair_sources'][data['pair_idx']];
+    // Do it dynamically rather than as the default in jinja
+    // in case javascript is disabled
+    avMerge = new AVMerge(video, srcPair, 0);
+}
+
+// Quality selector
+const qs = document.getElementById('quality-select');
+if (qs) {
+  qs.addEventListener('change', function(e) {
+    changeQuality(JSON.parse(this.value))
+  });
+}
+
+// Set up video start time from &t parameter
+if (data.time_start != 0 && video) {video.currentTime = data.time_start};
+
+// External video speed control
+let speedInput = document.getElementById('speed-control');
+speedInput.addEventListener('keyup', (event) => {
+    if (event.key === 'Enter') {
+        let speed = parseFloat(speedInput.value);
+        if(!isNaN(speed)){
+            video.playbackRate = speed;
+        }
+    }
+});
+
+
+// Playlist lazy image loading
+if (data.playlist && data.playlist['id'] !== null) {
+    // lazy load playlist images
+    // copied almost verbatim from
+    // https://css-tricks.com/tips-for-rolling-your-own-lazy-loading/
+    // IntersectionObserver isn't supported in pre-quantum
+    // firefox versions, but the alternative of making it
+    // manually is a performance drain, so oh well
+    let observer = new IntersectionObserver(lazyLoad, {
+
+      // where in relation to the edge of the viewport, we are observing
+      rootMargin: "100px",
+
+      // how much of the element needs to have intersected
+      // in order to fire our loading function
+      threshold: 1.0
+
+    });
+
+    function lazyLoad(elements) {
+      elements.forEach(item => {
+        if (item.intersectionRatio > 0) {
+
+          // set the src attribute to trigger a load
+          item.target.src = item.target.dataset.src;
+
+          // stop observing this element. Our work here is done!
+          observer.unobserve(item.target);
+        };
+      });
+    };
+
+    // Tell our observer to observe all img elements with a "lazy" class
+    let lazyImages = document.querySelectorAll('img.lazy');
+    lazyImages.forEach(img => {
+      observer.observe(img);
+    });
+}
+
+
+// Autoplay
+if (data.settings.related_videos_mode !== 0 || data.playlist !== null) {
+    let playability_error = !!data.playability_error;
+    let isPlaylist = false;
+    if (data.playlist !== null && data.playlist['current_index'] !== null)
+        isPlaylist = true;
+
+    // read cookies on whether to autoplay
+    // https://developer.mozilla.org/en-US/docs/Web/API/Document/cookie
+    let cookieValue;
+    let playlist_id;
+    if (isPlaylist) {
+        // from https://stackoverflow.com/a/6969486
+        function escapeRegExp(string) {
+            // $& means the whole matched string
+            return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+        }
+        playlist_id = data.playlist['id'];
+        playlist_id = escapeRegExp(playlist_id);
+
+        cookieValue = document.cookie.replace(new RegExp(
+            '(?:(?:^|.*;\\s*)autoplay_'
+            + playlist_id + '\\s*\\=\\s*([^;]*).*$)|^.*$'
+        ), '$1');
+    } else {
+        cookieValue = document.cookie.replace(new RegExp(
+            '(?:(?:^|.*;\\s*)autoplay\\s*\\=\\s*([^;]*).*$)|^.*$'
+        ),'$1');
+    }
+
+    let autoplayEnabled = 0;
+    if(cookieValue.length === 0){
+        autoplayEnabled = 0;
+    } else {
+        autoplayEnabled = Number(cookieValue);
+    }
+
+    // check the checkbox if autoplay is on
+    let checkbox = document.querySelector('.autoplay-toggle');
+    if(autoplayEnabled){
+        checkbox.checked = true;
+    }
+
+    // listen for checkbox to turn autoplay on and off
+    let cookie = 'autoplay'
+    if (isPlaylist)
+        cookie += '_' + playlist_id;
+
+    checkbox.addEventListener( 'change', function() {
+        if(this.checked) {
+            autoplayEnabled = 1;
+            document.cookie = cookie + '=1; SameSite=Strict';
+        } else {
+            autoplayEnabled = 0;
+            document.cookie = cookie + '=0; SameSite=Strict';
+        }
+    });
+
+    if(!playability_error){
+        // play the video if autoplay is on
+        if(autoplayEnabled){
+            video.play();
+        }
+    }
+
+    // determine next video url
+    let nextVideoUrl;
+    if (isPlaylist) {
+        let currentIndex = data.playlist['current_index'];
+        if (data.playlist['current_index']+1 == data.playlist['items'].length)
+            nextVideoUrl = null;
+        else
+            nextVideoUrl = data.playlist['items'][data.playlist['current_index']+1]['url'];
+
+        // scroll playlist to proper position
+        // item height + gap == 100
+        let pl = document.querySelector('.playlist-videos');
+        pl.scrollTop = 100*currentIndex;
+    } else {
+        if (data.related.length === 0)
+            nextVideoUrl = null;
+        else
+            nextVideoUrl = data.related[0]['url'];
+    }
+    let nextVideoDelay = 1000;
+
+    // go to next video when video ends
+    // https://stackoverflow.com/a/2880950
+    if (nextVideoUrl) {
+        if(playability_error){
+            videoEnded();
+        } else {
+            video.addEventListener('ended', videoEnded, false);
+        }
+        function nextVideo(){
+            if(autoplayEnabled){
+                window.location.href = nextVideoUrl;
+            }
+        }
+        function videoEnded(e) {
+            window.setTimeout(nextVideo, nextVideoDelay);
+        }
+    }
+}

youtube/static/license.css

@@ -29,6 +29,8 @@ input[type="search"] {
     padding: 0.4rem 0.4rem;
     font-size: 15px;
     color: var(--search-text);
+    outline: none;
+    box-shadow: none;
 }
 
 input[type='search'] {
@@ -95,7 +97,6 @@ header {
         "dropdown-label"
         "dropdown-content";
     grid-area: dropdown;
-    z-index: 1;
 }
 .dropdown-label {
     grid-area: dropdown-label;
@@ -135,8 +136,10 @@ label[for=options-toggle-cbox] {
 }
 
 #options-toggle-cbox:checked ~ .dropdown-content {
-    display: inline-grid;
+    display: block;
     white-space: nowrap;
+    background: var(--secondary-background);
+    padding: 0.5rem 1rem;
 }
 /*- ----------- End Menu Mobile sin JS ------------- */
 
@@ -178,7 +181,7 @@ label[for=options-toggle-cbox] {
 
 .table td,.table th {
     padding: 10px 10px;
-    border: 1px solid var(--secondary-background);
+    border: 1px solid var(--border-bg-license);
     text-align: center;
 }
 
@@ -263,7 +266,7 @@ label[for=options-toggle-cbox] {
     .dropdown {
         display: grid;
         grid-gap: 1px;
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: minmax(50px, 120px);
         grid-template-areas:
             "dropdown-label"
             "dropdown-content";
@@ -271,7 +274,12 @@ label[for=options-toggle-cbox] {
         z-index: 1;
         position: absolute;
     }
-
+    #options-toggle-cbox:checked ~ .dropdown-content {
+        padding: 0rem 3rem 1rem 1rem;
+        width: 100%;
+        max-height: 45vh;
+        overflow-y: scroll;
+    }
     .footer {
         display: grid;
         grid-template-columns: repeat(3, 1fr);

youtube/static/light_theme.css

@@ -8,10 +8,13 @@
     --secondary-background: #EEEEEE;
     --thumb-background:     #F5F5F5;
     --link:                 #212121;
-    --link-visited:         #606060;
-    --buttom:               #DCDCDC;
+    --link-visited:         #808080;
+    --border-bg:            #212121;
+    --border-bg-settings:   #91918C;
+    --border-bg-license:    #91918C;
+    --buttom:               #FFFFFF;
     --buttom-text:          #212121;
-    --button-border:        #91918c;
+    --button-border:        #91918C;
     --buttom-hover:         #BBBBBB;
     --search-text:          #212121;
     --time-background:      #212121;

youtube/static/local_playlist.css

@@ -33,6 +33,8 @@ input[type="search"] {
     padding: 0.4rem 0.4rem;
     font-size: 15px;
     color: var(--search-text);
+    outline: none;
+    box-shadow: none;
 }
 
 input[type='search'] {
@@ -100,7 +102,6 @@ header {
         "dropdown-label"
         "dropdown-content";
     grid-area: dropdown;
-    z-index: 1;
 }
 .dropdown-label {
     grid-area: dropdown-label;
@@ -200,8 +201,10 @@ label[for=options-toggle-cbox] {
 }
 
 #options-toggle-cbox:checked ~ .dropdown-content {
-    display: inline-grid;
+    display: block;
     white-space: nowrap;
+    background: var(--secondary-background);
+    padding: 0.5rem 1rem;
 }
 /*- ----------- End Menu Mobile sin JS ------------- */
 
@@ -474,17 +477,19 @@ hr {
     .dropdown {
         display: grid;
         grid-gap: 1px;
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: 100px auto;
         grid-template-areas:
             "dropdown-label"
             "dropdown-content";
         grid-area: dropdown;
-        background: var(--background);
-        padding-right: 4rem;
-        z-index: 1;
         position: absolute;
+        z-index: 1;
+    }
+    #options-toggle-cbox:checked ~ .dropdown-content {
+        width: calc(100% + 100px);
+        max-height: 80vh;
+        overflow-y: scroll;
     }
-
     .playlist-metadata {
         max-width: 50vw;
     }
@@ -498,7 +503,7 @@ hr {
         grid-area: playlist;
     }
     .play-clean {
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: 100px auto;
     }
     .play-clean > button {
         padding-left: 0px;

youtube/static/modules/plyr/custom_plyr.css

@@ -0,0 +1,77 @@
+/* Prevent this div from blocking right-click menu for video
+e.g. Firefox playback speed options */
+.plyr__poster {
+    display: none;
+}
+
+/* plyr fix */
+.plyr:-moz-full-screen video {
+    max-height: initial;
+}
+
+.plyr:-webkit-full-screen video {
+    max-height: initial;
+}
+
+.plyr:-ms-fullscreen video {
+    max-height: initial;
+}
+
+.plyr:fullscreen video {
+    max-height: initial;
+}
+
+.plyr__preview-thumb__image-container {
+    width: 158px;
+    height: 90px;
+}
+
+.plyr__preview-thumb {
+    bottom: 100%;
+}
+
+.plyr__menu__container [role="menu"],
+.plyr__menu__container [role="menucaptions"] {
+    /* Set vertical scroll */
+    /* issue https://github.com/sampotts/plyr/issues/1420 */
+    max-height: 320px;
+    overflow-y: auto;
+}
+
+/*
+* Custom styles similar to youtube
+*/
+.plyr__controls {
+    display: flex;
+    justify-content: center;
+}
+
+.plyr__progress__container {
+    position: absolute;
+    bottom: 0;
+    width: 100%;
+    margin-bottom: -10px;
+}
+
+.plyr__controls .plyr__controls__item:first-child {
+    margin-left: 0;
+    margin-right: 0;
+    z-index: 5;
+}
+
+.plyr__controls .plyr__controls__item.plyr__volume {
+    margin-left: auto;
+}
+
+.plyr__controls .plyr__controls__item.plyr__progress__container {
+    padding-left: 10px;
+    padding-right: 10px;
+}
+
+.plyr__progress input[type="range"] {
+    margin-bottom: 50px;
+}
+
+/*
+* End custom styles
+*/

youtube/static/playlist.css

@@ -33,6 +33,8 @@ input[type="search"] {
     padding: 0.4rem 0.4rem;
     font-size: 15px;
     color: var(--search-text);
+    outline: none;
+    box-shadow: none;
 }
 
 input[type='search'] {
@@ -100,7 +102,6 @@ header {
         "dropdown-label"
         "dropdown-content";
     grid-area: dropdown;
-    z-index: 1;
 }
 .dropdown-label {
     grid-area: dropdown-label;
@@ -200,8 +201,10 @@ label[for=options-toggle-cbox] {
 }
 
 #options-toggle-cbox:checked ~ .dropdown-content {
-    display: inline-grid;
+    display: block;
     white-space: nowrap;
+    background: var(--secondary-background);
+    padding: 0.5rem 1rem;
 }
 /*- ----------- End Menu Mobile sin JS ------------- */
 
@@ -484,17 +487,19 @@ hr {
     .dropdown {
         display: grid;
         grid-gap: 1px;
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: 100px auto;
         grid-template-areas:
             "dropdown-label"
             "dropdown-content";
         grid-area: dropdown;
-        background: var(--background);
-        padding-right: 4rem;
-        z-index: 1;
         position: absolute;
+        z-index: 1;
+    }
+    #options-toggle-cbox:checked ~ .dropdown-content {
+        width: calc(100% + 100px);
+        max-height: 80vh;
+        overflow-y: scroll;
     }
-
     .playlist-metadata {
         max-width: 50vw;
     }
@@ -508,7 +513,7 @@ hr {
         grid-area: playlist;
     }
     .play-clean {
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: 100px auto;
     }
     .play-clean > button {
         padding-left: 0px;

youtube/static/search.css

@@ -33,6 +33,8 @@ input[type="search"] {
     padding: 0.4rem 0.4rem;
     font-size: 15px;
     color: var(--search-text);
+    outline: none;
+    box-shadow: none;
 }
 
 input[type='search'] {
@@ -100,7 +102,6 @@ header {
         "dropdown-label"
         "dropdown-content";
     grid-area: dropdown;
-    z-index: 1;
 }
 .dropdown-label {
     grid-area: dropdown-label;
@@ -200,8 +201,10 @@ label[for=options-toggle-cbox] {
 }
 
 #options-toggle-cbox:checked ~ .dropdown-content {
-    display: inline-grid;
+    display: block;
     white-space: nowrap;
+    background: var(--secondary-background);
+    padding: 0.5rem 1rem;
 }
 /*- ----------- End Menu Mobile sin JS ------------- */
 
@@ -252,7 +255,7 @@ hr {
 /* Video list item */
 .video-container {
     display: grid;
-    grid-row-gap: 0.5rem;
+    grid-gap: 0.5rem;
 }
 
 .length {
@@ -295,6 +298,12 @@ hr {
     cursor: pointer;
 }
 
+.item-video address {
+    white-space: nowrap;
+    overflow: hidden;
+    text-overflow: ellipsis;
+}
+
 .item-video.channel-item .thumbnail.channel {
     display: flex;
     justify-content: center;
@@ -430,7 +439,7 @@ hr {
 @media (min-width: 600px) {
     .video-container {
         display: grid;
-        grid-row-gap: 0.5rem;
+        grid-gap: 0.5rem;
         grid-template-columns: 1fr 1fr;
     }
 }
@@ -456,17 +465,19 @@ hr {
     .dropdown {
         display: grid;
         grid-gap: 1px;
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: 100px auto;
         grid-template-areas:
             "dropdown-label"
             "dropdown-content";
         grid-area: dropdown;
-        background: var(--background);
-        padding-right: 4rem;
-        z-index: 1;
         position: absolute;
+        z-index: 1;
+    }
+    #options-toggle-cbox:checked ~ .dropdown-content {
+        width: calc(100% + 100px);
+        max-height: 80vh;
+        overflow-y: scroll;
     }
-
     /* playlist */
     .playlist {
         display: grid;
@@ -476,7 +487,7 @@ hr {
         grid-area: playlist;
     }
     .play-clean {
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: 100px auto;
     }
     .play-clean > button {
         padding-left: 0px;
@@ -494,7 +505,7 @@ hr {
     .video-container {
         display: grid;
         grid-template-columns: repeat(4, 1fr);
-        grid-row-gap: 1rem;
+        grid-gap: 1rem;
         grid-column-gap: 1rem;
     }
 

youtube/static/settings.css

@@ -29,6 +29,8 @@ input[type="search"] {
     padding: 0.4rem 0.4rem;
     font-size: 15px;
     color: var(--search-text);
+    outline: none;
+    box-shadow: none;
 }
 
 input[type='search'] {
@@ -95,7 +97,6 @@ header {
         "dropdown-label"
         "dropdown-content";
     grid-area: dropdown;
-    z-index: 1;
 }
 .dropdown-label {
     grid-area: dropdown-label;
@@ -135,8 +136,10 @@ label[for=options-toggle-cbox] {
 }
 
 #options-toggle-cbox:checked ~ .dropdown-content {
-    display: inline-grid;
+    display: block;
     white-space: nowrap;
+    background: var(--secondary-background);
+    padding: 0.5rem 1rem;
 }
 /*- ----------- End Menu Mobile sin JS ------------- */
 
@@ -151,6 +154,11 @@ label[for=options-toggle-cbox] {
     padding: 1rem;
 }
 
+.settings-form > h2 {
+    border-bottom: 2px solid var(--border-bg-settings);
+    padding-bottom: 0.5rem;
+}
+
 .settings-list {
     display: grid;
     grid-row-gap: 1rem;
@@ -161,7 +169,6 @@ label[for=options-toggle-cbox] {
 .setting-item {
     display: grid;
     grid-template-columns: auto auto;
-    background-color: var(--secondary-focus);
     align-items: center;
     padding: 5px;
 }
@@ -215,15 +222,18 @@ label[for=options-toggle-cbox] {
     .dropdown {
         display: grid;
         grid-gap: 1px;
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: 100px auto;
         grid-template-areas:
             "dropdown-label"
             "dropdown-content";
         grid-area: dropdown;
-        z-index: 1;
         position: absolute;
-        background: var(--background);
-        padding-right: 4rem;
+        z-index: 1;
+    }
+    #options-toggle-cbox:checked ~ .dropdown-content {
+        width: calc(100% + 100px);
+        max-height: 80vh;
+        overflow-y: scroll;
     }
     .main {
         display: grid;

youtube/static/subscription.css

@@ -33,6 +33,8 @@ input[type="search"] {
     padding: 0.4rem 0.4rem;
     font-size: 15px;
     color: var(--search-text);
+    outline: none;
+    box-shadow: none;
 }
 
 input[type='search'] {
@@ -100,7 +102,6 @@ header {
         "dropdown-label"
         "dropdown-content";
     grid-area: dropdown;
-    z-index: 1;
 }
 .dropdown-label {
     grid-area: dropdown-label;
@@ -200,8 +201,10 @@ label[for=options-toggle-cbox] {
 }
 
 #options-toggle-cbox:checked ~ .dropdown-content {
-    display: inline-grid;
+    display: block;
     white-space: nowrap;
+    background: var(--secondary-background);
+    padding: 0.5rem 1rem;
 }
 /*- ----------- End Menu Mobile sin JS ------------- */
 
@@ -477,17 +480,20 @@ hr {
     .dropdown {
         display: grid;
         grid-gap: 1px;
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: 100px auto;
         grid-template-areas:
             "dropdown-label"
             "dropdown-content";
         grid-area: dropdown;
-        background: var(--background);
         padding-right: 4rem;
         z-index: 1;
         position: absolute;
     }
-
+    #options-toggle-cbox:checked ~ .dropdown-content {
+        width: calc(100% + 100px);
+        max-height: 80vh;
+        overflow-y: scroll;
+    }
     .sidebar-links {
         max-width: 50vw;
     }
@@ -501,7 +507,7 @@ hr {
         grid-area: playlist;
     }
     .play-clean {
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: 100px auto;
     }
     .play-clean > button {
         padding-left: 0px;

youtube/static/subscription_manager.css

@@ -33,6 +33,8 @@ input[type="search"] {
     padding: 0.4rem 0.4rem;
     font-size: 15px;
     color: var(--search-text);
+    outline: none;
+    box-shadow: none;
 }
 
 input[type='search'] {
@@ -100,7 +102,6 @@ header {
         "dropdown-label"
         "dropdown-content";
     grid-area: dropdown;
-    z-index: 1;
 }
 .dropdown-label {
     grid-area: dropdown-label;
@@ -120,66 +121,6 @@ header {
     background-color: var(--buttom-hover);
 }
 
-/* playlist */
-.playlist {
-    display: grid;
-    grid-gap: 4px;
-    grid-template-areas:
-        "play-box"
-        "play-hidden"
-        "play-add"
-        "play-clean";
-    grid-area: playlist;
-}
-.play-box {
-    grid-area: play-box;
-}
-
-.play-hidden {
-    grid-area: play-hidden;
-}
-
-.play-add {
-    grid-area: play-add;
-    cursor: pointer;
-
-    padding-bottom: 6px;
-    padding-left: .75em;
-    padding-right: .75em;
-    padding-top: 6px;
-    text-align: center;
-    white-space: nowrap;
-    background-color: var(--buttom);
-    border: 1px solid var(--button-border);
-    color: var(--buttom-text);
-    border-radius: 5px;
-}
-.play-add:hover {
-    background-color: var(--buttom-hover);
-}
-
-.play-clean {
-    display: grid;
-    grid-area: play-clean;
-}
-
-.play-clean > button {
-    padding-bottom: 6px;
-    padding-left: .75em;
-    padding-right: .75em;
-    padding-top: 6px;
-    text-align: center;
-    white-space: nowrap;
-    background-color: var(--buttom);
-    border: 1px solid var(--button-border);
-    color: var(--buttom-text);
-    border-radius: 5px;
-}
-.play-clean > button:hover {
-    background-color: var(--buttom-hover);
-}
-/* /playlist */
-
 /* ------------- Menu Mobile sin JS ---------------- */
 /* input hidden */
 .opt-box {
@@ -200,8 +141,10 @@ label[for=options-toggle-cbox] {
 }
 
 #options-toggle-cbox:checked ~ .dropdown-content {
-    display: inline-grid;
+    display: block;
     white-space: nowrap;
+    background: var(--secondary-background);
+    padding: 0.5rem 1rem;
 }
 /*- ----------- End Menu Mobile sin JS ------------- */
 
@@ -385,17 +328,19 @@ hr {
     .dropdown {
         display: grid;
         grid-gap: 1px;
-        grid-template-columns: minmax(50px, 100px);
+        grid-template-columns: 100px auto;
         grid-template-areas:
             "dropdown-label"
             "dropdown-content";
         grid-area: dropdown;
-        background: var(--background);
-        padding-right: 4rem;
-        z-index: 1;
         position: absolute;
+        z-index: 1;
+    }
+    #options-toggle-cbox:checked ~ .dropdown-content {
+        width: calc(100% + 100px);
+        max-height: 80vh;
+        overflow-y: scroll;
     }
-
     .import-export {
         max-width: 50vw;
     }
@@ -408,37 +353,6 @@ hr {
         align-items: center;
     }
 
-    /* playlist */
-    .playlist {
-        display: grid;
-        grid-gap: 1px;
-        grid-template-columns: 1fr 1.4fr 0.3fr 1.3fr;
-        grid-template-areas: ". play-box play-add play-clean";
-        grid-area: playlist;
-    }
-    .play-clean {
-        grid-template-columns: minmax(50px, 100px);
-    }
-    .play-clean > button {
-        padding-left: 0px;
-        padding-right: 0px;
-        padding-bottom: 6px;
-        padding-top: 6px;
-        text-align: center;
-        white-space: nowrap;
-        background-color: var(--buttom);
-        color: var(--buttom-text);
-        border-radius: 5px;
-        cursor: pointer;
-    }
-
-    .video-container {
-        display: grid;
-        grid-template-columns: repeat(4, 1fr);
-        grid-row-gap: 1rem;
-        grid-column-gap: 1rem;
-    }
-
     .footer {
         display: grid;
         grid-template-columns: repeat(3, 1fr);

youtube/static/unsubscribe.css

@@ -0,0 +1,269 @@
+body {
+    display: grid;
+    grid-gap: 20px;
+    grid-template-areas:
+        "header"
+        "main"
+        "footer";
+    /* Fix height */
+    height: 100vh;
+    grid-template-rows: auto 1fr auto;
+    /* fix top and bottom */
+    margin-left: 1rem;
+    margin-right: 1rem;
+}
+
+img {
+    width: 100%;
+    height: auto;
+}
+
+a:link {
+    color: var(--link);
+}
+
+a:visited {
+    color: var(--link-visited);
+}
+
+input[type="text"],
+input[type="search"] {
+    background: var(--background);
+    border: 1px solid var(--button-border);
+    padding: 0.4rem 0.4rem;
+    font-size: 15px;
+    color: var(--search-text);
+    outline: none;
+    box-shadow: none;
+}
+
+input[type='search'] {
+    border-bottom: 1px solid var(--button-border);
+    border-top: 0px;
+    border-left: 0px;
+    border-right: 0px;
+    border-radius: 0px;
+}
+
+header {
+    display: grid;
+    grid-gap: 1px;
+    grid-template-areas:
+        "home"
+        "form"
+        "playlist";
+    grid-area: header;
+}
+
+.home {
+    grid-area: home;
+    margin-left: auto;
+    margin-right: auto;
+    margin-bottom: 1rem;
+    margin-top: 1rem;
+}
+
+.form {
+    display: grid;
+    grid-gap: 4px;
+    grid-template-areas:
+        "search-box"
+        "search-button"
+        "dropdown";
+    grid-area: form;
+}
+
+.search-box {
+    grid-area: search-box;
+}
+.search-button {
+    grid-area: search-button;
+
+    cursor: pointer;
+    padding-bottom: 6px;
+    padding-left: .75em;
+    padding-right: .75em;
+    padding-top: 6px;
+    text-align: center;
+    white-space: nowrap;
+    background-color: var(--buttom);
+    border: 1px solid var(--button-border);
+    color: var(--buttom-text);
+    border-radius: 5px;
+}
+.search-button:hover {
+    background-color: var(--buttom-hover);
+}
+
+.dropdown {
+    display: grid;
+    grid-gap: 1px;
+    grid-template-areas:
+        "dropdown-label"
+        "dropdown-content";
+    grid-area: dropdown;
+}
+.dropdown-label {
+    grid-area: dropdown-label;
+
+    padding-bottom: 6px;
+    padding-left: .75em;
+    padding-right: .75em;
+    padding-top: 6px;
+    text-align: center;
+    white-space: nowrap;
+    background-color: var(--buttom);
+    border: 1px solid var(--button-border);
+    color: var(--buttom-text);
+    border-radius: 5px;
+}
+.dropdown-label:hover {
+    background-color: var(--buttom-hover);
+}
+
+/* ------------- Menu Mobile sin JS ---------------- */
+/* input hidden */
+.opt-box {
+    display: none;
+}
+.dropdown-content {
+    display: none;
+    grid-area: dropdown-content;
+}
+label[for=options-toggle-cbox] {
+    cursor: pointer;
+    -webkit-touch-callout: none;
+      -webkit-user-select: none;
+       -khtml-user-select: none;
+         -moz-user-select: none;
+          -ms-user-select: none;
+              user-select: none;
+}
+
+#options-toggle-cbox:checked ~ .dropdown-content {
+    display: block;
+    white-space: nowrap;
+    background: var(--secondary-background);
+    padding: 0.5rem 1rem;
+}
+/*- ----------- End Menu Mobile sin JS ------------- */
+
+.main {
+    grid-area: main;
+    display: flex;
+    flex-direction: column;
+    align-items: center;
+}
+
+/* fix hr when is children of grid */
+hr {
+    width: 100%;
+}
+
+.list-channel {
+    padding: 0;
+}
+
+.list-channel > li {
+    list-style: none;
+}
+/* pagination */
+.main .pagination-container {
+    display: grid;
+    justify-content: center;
+}
+
+.main .pagination-container .pagination-list {
+    display: flex;
+    flex-direction: row;
+    flex-wrap: wrap;
+    justify-content: center;
+}
+
+.main .pagination-container .pagination-list .page-link {
+    border-style: none;
+    font-weight: bold;
+    text-align: center;
+    background: var(--secondary-focus);
+    text-decoration: none;
+    align-self: center;
+    padding: .5rem;
+    width: 1rem;
+}
+
+.main .pagination-container .pagination-list .page-link.is-current {
+    background: var(--secondary-background);
+}
+
+.footer {
+    grid-area: footer;
+    display: grid;
+    grid-template-columns: auto;
+    align-items: center;
+    justify-content: center;
+    margin: auto;
+    text-align: center;
+}
+
+.footer > p {
+    text-align: center;
+}
+
+@media (min-width: 480px) {
+    .item-video {
+        font-size: 0.85rem;
+    }
+    .info-box {
+        grid-gap: 2px;
+    }
+    .title {
+        font-size: 1rem;
+    }
+}
+
+@media (min-width: 992px) {
+    body {
+        display: grid;
+        grid-template-columns: 0.3fr 2fr 1fr 0.3fr;
+        grid-template-rows: auto 1fr auto;
+        grid-template-areas:
+            "header header header header"
+            "main   main   main   main"
+            "footer footer footer footer";
+    }
+    .form {
+        display: grid;
+        grid-gap: 1px;
+        grid-template-columns: 1fr 1.4fr 0.3fr 1.3fr;
+        grid-template-areas: ". search-box search-button dropdown";
+        grid-area: form;
+        position: relative;
+    }
+    .dropdown {
+        display: grid;
+        grid-gap: 1px;
+        grid-template-columns: 100px auto;
+        grid-template-areas:
+            "dropdown-label"
+            "dropdown-content";
+        grid-area: dropdown;
+        z-index: 1;
+        position: absolute;
+    }
+    #options-toggle-cbox:checked ~ .dropdown-content {
+        width: calc(100% + 100px);
+        max-height: 80vh;
+        overflow-y: scroll;
+    }
+
+    .footer {
+        display: grid;
+        grid-template-columns: repeat(3, 1fr);
+        grid-column-gap: 2rem;
+        align-items: center;
+        justify-content: center;
+        text-align: center;
+        margin-top: 1rem;
+        margin-bottom: 1rem;
+    }
+}

youtube/static/watch.css

@@ -21,21 +21,7 @@ img {
 video {
     width: 100%;
     height: auto;
-    max-height: 480px;
-}
-
-/* plyr fix */
-.plyr:-moz-full-screen video {
-    max-height: initial;
-}
-.plyr:-webkit-full-screen video {
-    max-height: initial;
-}
-.plyr:-ms-fullscreen video {
-    max-height: initial;
-}
-.plyr:fullscreen video {
-    max-height: initial;
+    max-height: calc(100vh/1.5);
 }
 
 a:link {
@@ -54,6 +40,8 @@ input[type="search"] {
     padding: 0.4rem 0.4rem;
     font-size: 15px;
     color: var(--search-text);
+    outline: none;
+    box-shadow: none;
 }
 
 input[type='search'] {
@@ -121,7 +109,6 @@ header {
         "dropdown-label"
         "dropdown-content";
     grid-area: dropdown;
-    z-index: 1;
 }
 .dropdown-label {
     grid-area: dropdown-label;
@@ -141,6 +128,29 @@ header {
     background-color: var(--buttom-hover);
 }
 
+.live-url-choices {
+    background-color: var(--thumb-background);
+    margin: 1rem 0;
+    padding: 1rem;
+}
+
+.playability-error {
+    position: relative;
+    box-sizing: border-box;
+    height: 30vh;
+    margin: 1rem 0;
+}
+
+.playability-error > span {
+    display: flex;
+    background-color: var(--thumb-background);
+    height: 100%;
+    object-fit: cover;
+    justify-content: center;
+    align-items: center;
+    text-align: center;
+}
+
 .playlist {
     display: grid;
     grid-gap: 4px;
@@ -219,9 +229,10 @@ label[for=options-toggle-cbox] {
 }
 
 #options-toggle-cbox:checked ~ .dropdown-content {
-    display: inline-grid;
+    display: block;
     white-space: nowrap;
-    padding-left: 1rem;
+    background: var(--secondary-background);
+    padding: 0.5rem 1rem;
 }
 /*- ----------- End Menu Mobile sin JS ------------- */
 
@@ -236,6 +247,9 @@ label[for=options-toggle-cbox] {
         "sc-video"
         "sc-info";
 }
+figure.sc-video {
+    margin: 1rem 0px;
+}
 .sc-video { grid-area: sc-video; }
 .sc-info {
     display: grid;
@@ -618,17 +632,21 @@ label[for=options-toggle-cbox] {
     .dropdown {
         display: grid;
         grid-gap: 1px;
-        grid-template-columns: minmax(50px, 120px);
+        grid-template-columns: 100px auto;
         grid-template-areas:
             "dropdown-label"
             "dropdown-content";
         grid-area: dropdown;
-
-        background: var(--background);
-        padding-right: 4rem;
-
-        z-index: 1;
         position: absolute;
+        z-index: 1;
+    }
+    #options-toggle-cbox:checked ~ .dropdown-content {
+        width: calc(100% + 100px);
+        max-height: 80vh;
+        overflow-y: scroll;
+    }
+    .playability-error {
+        height: 60vh;
     }
     .playlist {
         display: grid;
@@ -638,7 +656,7 @@ label[for=options-toggle-cbox] {
         grid-area: playlist;
     }
     .play-clean {
-        grid-template-columns: minmax(50px, 120px);
+        grid-template-columns: 100px auto;
     }
     .play-clean > button {
         padding-bottom: 6px;

youtube/subscriptions.py

@@ -1,4 +1,4 @@
-from youtube import util, yt_data_extract, channel, local_playlist
+from youtube import util, yt_data_extract, channel, local_playlist, playlist
 from youtube import yt_app
 import settings
 
@@ -15,6 +15,8 @@ import math
 import secrets
 import collections
 import calendar # bullshit! https://bugs.python.org/issue6280
+import csv
+import re
 
 import flask
 from flask import request
@@ -106,8 +108,7 @@ def _subscribe(channels):
         with connection as cursor:
             channel_ids_to_check = [channel[0] for channel in channels if not _is_subscribed(cursor, channel[0])]
 
-            rows = ((channel_id, channel_name, 0, 0) for channel_id,
-                    channel_name in channels)
+            rows = ((channel_id, channel_name, 0, 0) for channel_id, channel_name in channels)
             cursor.executemany('''INSERT OR IGNORE INTO subscribed_channels (yt_channel_id, channel_name, time_last_checked, next_check_time)
                                   VALUES (?, ?, ?, ?)''', rows)
 
@@ -234,8 +235,7 @@ def _get_channel_names(cursor, channel_ids):
     return result
 
 
-def _channels_with_tag(cursor, tag, order=False, exclude_muted=False,
-                       include_muted_status=False):
+def _channels_with_tag(cursor, tag, order=False, exclude_muted=False, include_muted_status=False):
     ''' returns list of (channel_id, channel_name) '''
 
     statement = '''SELECT yt_channel_id, channel_name'''
@@ -432,8 +432,10 @@ def autocheck_setting_changed(old_value, new_value):
         stop_autocheck_system()
 
 
-settings.add_setting_changed_hook('autocheck_subscriptions',
-                                  autocheck_setting_changed)
+settings.add_setting_changed_hook(
+    'autocheck_subscriptions',
+    autocheck_setting_changed
+)
 if settings.autocheck_subscriptions:
     start_autocheck_system()
 # ----------------------------
@@ -461,7 +463,24 @@ def _get_atoma_feed(channel_id):
 
 def _get_channel_videos_first_page(channel_id, channel_status_name):
     try:
-        return channel.get_channel_first_page(channel_id=channel_id)
+        # First try the playlist method
+        pl_json = playlist.get_videos(
+            'UU' + channel_id[2:],
+            1,
+            include_shorts=settings.include_shorts_in_subscriptions,
+            report_text=None
+        )
+        pl_info = yt_data_extract.extract_playlist_info(pl_json)
+        if pl_info.get('items'):
+            pl_info['items'] = pl_info['items'][0:30]
+            return pl_info
+
+        # Try the channel api method
+        channel_json = channel.get_channel_first_page(channel_id=channel_id)
+        channel_info = yt_data_extract.extract_channel_info(
+            json.loads(channel_json), 'videos'
+        )
+        return channel_info
     except util.FetchError as e:
         if e.code == '429' and settings.route_tor:
             error_message = ('Error checking channel ' + channel_status_name
@@ -495,7 +514,7 @@ def _get_upstream_videos(channel_id):
     )
     gevent.joinall(tasks)
 
-    channel_tab, feed = tasks[0].value, tasks[1].value
+    channel_info, feed = tasks[0].value, tasks[1].value
 
     # extract published times from atoma feed
     times_published = {}
@@ -533,9 +552,8 @@ def _get_upstream_videos(channel_id):
     except defusedxml.ElementTree.ParseError:
         print('Failed to read atoma feed for ' + channel_status_name)
 
-    if channel_tab is None: # there was an error
+    if channel_info is None: # there was an error
         return
-    channel_info = yt_data_extract.extract_channel_info(json.loads(channel_tab), 'videos')
     if channel_info['error']:
         print('Error checking channel ' + channel_status_name + ': ' + channel_info['error'])
         return
@@ -550,14 +568,38 @@ def _get_upstream_videos(channel_id):
         if video_item['id'] in times_published:
             video_item['time_published'] = times_published[video_item['id']]
             video_item['is_time_published_exact'] = True
-        else:
+        elif video_item.get('time_published'):
             video_item['is_time_published_exact'] = False
             try:
                 video_item['time_published'] = youtube_timestamp_to_posix(video_item['time_published']) - i  # subtract a few seconds off the videos so they will be in the right order
-            except KeyError:
+            except Exception:
                 print(video_item)
-
+        else:
+            video_item['is_time_published_exact'] = False
+            video_item['time_published'] = None
         video_item['channel_id'] = channel_id
+    if len(videos) > 1:
+        # Go back and fill in any videos that don't have a time published
+        # using the time published of the surrounding ones
+        for i in range(len(videos)-1):
+            if (videos[i+1]['time_published'] is None
+                and videos[i]['time_published'] is not None
+            ):
+                videos[i+1]['time_published'] = videos[i]['time_published'] - 1
+        for i in reversed(range(1,len(videos))):
+            if (videos[i-1]['time_published'] is None
+                and videos[i]['time_published'] is not None
+            ):
+                videos[i-1]['time_published'] = videos[i]['time_published'] + 1
+    # Special case: none of the videos have a time published.
+    # In this case, make something up
+    if videos and videos[0]['time_published'] is None:
+        assert all(v['time_published'] is None for v in videos)
+        now = time.time()
+        for i in range(len(videos)):
+            # 1 month between videos
+            videos[i]['time_published'] = now - i*3600*24*30
+
 
     if len(videos) == 0:
         average_upload_period = 4*7*24*3600  # assume 1 month for channel with no videos
@@ -576,26 +618,31 @@ def _get_upstream_videos(channel_id):
     with open_database() as connection:
         with connection as cursor:
 
-            # calculate how many new videos there are
-            existing_vids = set(row[0] for row in cursor.execute(
-                '''SELECT video_id
+            # Get video ids and duration of existing vids so we
+            # can see how many new ones there are and update
+            # livestreams/premiers
+            existing_vids = list(cursor.execute(
+                '''SELECT video_id, duration
                    FROM videos
                    INNER JOIN subscribed_channels
                        ON videos.sql_channel_id = subscribed_channels.id
                    WHERE yt_channel_id=?
                    ORDER BY time_published DESC
                    LIMIT 30''', [channel_id]).fetchall())
+            existing_vid_ids = set(row[0] for row in existing_vids)
+            existing_durs = dict(existing_vids)
 
             # new videos the channel has uploaded since last time we checked
             number_of_new_videos = 0
             for video in videos:
-                if video['id'] in existing_vids:
+                if video['id'] in existing_vid_ids:
                     break
                 number_of_new_videos += 1
 
             is_first_check = cursor.execute('''SELECT time_last_checked FROM subscribed_channels WHERE yt_channel_id=?''', [channel_id]).fetchone()[0] in (None, 0)
             time_videos_retrieved = int(time.time())
             rows = []
+            update_rows = []
             for i, video_item in enumerate(videos):
                 if (is_first_check
                         or number_of_new_videos > 6
@@ -611,16 +658,34 @@ def _get_upstream_videos(channel_id):
                     time_noticed = video_item['time_published']
                 else:
                     time_noticed = time_videos_retrieved
-                rows.append((
-                    video_item['channel_id'],
-                    video_item['id'],
-                    video_item['title'],
-                    video_item['duration'],
-                    video_item['time_published'],
-                    video_item['is_time_published_exact'],
-                    time_noticed,
-                    video_item['description'],
-                ))
+
+                # videos which need durations updated
+                non_durations = ('upcoming', 'none', 'live', '')
+                v_id = video_item['id']
+                if (existing_durs.get(v_id) is not None
+                    and existing_durs[v_id].lower() in non_durations
+                    and video_item['duration'] not in non_durations
+                ):
+                    update_rows.append((
+                        video_item['title'],
+                        video_item['duration'],
+                        video_item['time_published'],
+                        video_item['is_time_published_exact'],
+                        video_item['description'],
+                        video_item['id'],
+                    ))
+                # all other videos
+                else:
+                    rows.append((
+                        video_item['channel_id'],
+                        video_item['id'],
+                        video_item['title'],
+                        video_item['duration'],
+                        video_item['time_published'],
+                        video_item['is_time_published_exact'],
+                        time_noticed,
+                        video_item['description'],
+                    ))
 
             cursor.executemany('''INSERT OR IGNORE INTO videos (
                                       sql_channel_id,
@@ -633,6 +698,13 @@ def _get_upstream_videos(channel_id):
                                       description
                                   )
                                   VALUES ((SELECT id FROM subscribed_channels WHERE yt_channel_id=?), ?, ?, ?, ?, ?, ?, ?)''', rows)
+            cursor.executemany('''UPDATE videos SET
+                                      title=?,
+                                      duration=?,
+                                      time_published=?,
+                                      is_time_published_exact=?,
+                                      description=?
+                                  WHERE video_id=?''', update_rows)
             cursor.execute('''UPDATE subscribed_channels
                               SET time_last_checked = ?, next_check_time = ?
                               WHERE yt_channel_id=?''', [int(time.time()), next_check_time, channel_id])
@@ -682,7 +754,7 @@ def check_specific_channels(channel_ids):
     channel_names.update(channel_id_name_list)
     check_channels_if_necessary(channel_ids)
 
-
+CHANNEL_ID_RE = re.compile(r'UC[-_\w]{22}')
 @yt_app.route('/import_subscriptions', methods=['POST'])
 def import_subscriptions():
 
@@ -700,15 +772,36 @@ def import_subscriptions():
     mime_type = file.mimetype
 
     if mime_type == 'application/json':
-        file = file.read().decode('utf-8')
+        info = file.read().decode('utf-8')
+        if info == '':
+            return '400 Bad Request: File is empty', 400
         try:
-            file = json.loads(file)
+            info = json.loads(info)
         except json.decoder.JSONDecodeError:
             traceback.print_exc()
             return '400 Bad Request: Invalid json file', 400
 
+        channels = []
         try:
-            channels = ((item['snippet']['resourceId']['channelId'], item['snippet']['title']) for item in file)
+            if 'app_version_int' in info:   # NewPipe Format
+                for item in info['subscriptions']:
+                    # Other service, such as SoundCloud
+                    if item.get('service_id', 0) != 0:
+                        continue
+                    channel_url = item['url']
+                    channel_id_match = CHANNEL_ID_RE.search(channel_url)
+                    if channel_id_match:
+                        channel_id = channel_id_match.group(0)
+                    else:
+                        print('WARNING: Could not find channel id in url',
+                              channel_url)
+                        continue
+                    channels.append((channel_id, item['name']))
+            else:   # Old Google Takeout format
+                for item in info:
+                    snippet = item['snippet']
+                    channel_id = snippet['resourceId']['channelId']
+                    channels.append((channel_id, snippet['title']))
         except (KeyError, IndexError):
             traceback.print_exc()
             return '400 Bad Request: Unknown json structure', 400
@@ -729,8 +822,25 @@ def import_subscriptions():
 
         except (AssertionError, IndexError, defusedxml.ElementTree.ParseError) as e:
             return '400 Bad Request: Unable to read opml xml file, or the file is not the expected format', 400
+    elif mime_type in ('text/csv', 'application/vnd.ms-excel'):
+        content = file.read().decode('utf-8')
+        reader = csv.reader(content.splitlines())
+        channels = []
+        for row in reader:
+            if not row or row[0].lower().strip() == 'channel id':
+                continue
+            elif len(row) > 1 and CHANNEL_ID_RE.fullmatch(row[0].strip()):
+                channels.append( (row[0], row[-1]) )
+            else:
+                print('WARNING: Unknown row format:', row)
     else:
-            return '400 Bad Request: Unsupported file format: ' + mime_type + '. Only subscription.json files (from Google Takeouts) and XML OPML files exported from YouTube\'s subscription manager page are supported', 400
+        error = 'Unsupported file format: ' + mime_type
+        error += (' . Only subscription.json, subscriptions.csv files'
+                  ' (from Google Takeouts)'
+                  ' and XML OPML files exported from YouTube\'s'
+                  ' subscription manager page are supported')
+        return (flask.render_template('error.html', error_message=error),
+                400)
 
     _subscribe(channels)
 
@@ -747,7 +857,7 @@ def export_subscriptions():
                     _get_subscribed_channels(cursor)):
                 if muted and not include_muted:
                     continue
-                if request.values['export_format'] == 'json':
+                if request.values['export_format'] == 'json_google_takeout':
                     sub_list.append({
                         'kind': 'youtube#subscription',
                         'snippet': {
@@ -760,21 +870,38 @@ def export_subscriptions():
                             'title': channel_name,
                         },
                     })
+                elif request.values['export_format'] == 'json_newpipe':
+                    sub_list.append({
+                        'service_id': 0,
+                        'url': 'https://www.youtube.com/channel/' + channel_id,
+                        'name': channel_name,
+                    })
                 elif request.values['export_format'] == 'opml':
                     sub_list.append({
                         'channel_name': channel_name,
                         'channel_id': channel_id,
                     })
-    if request.values['export_format'] == 'json':
+    date_time = time.strftime('%Y%m%d%H%M', time.localtime())
+    if request.values['export_format'] == 'json_google_takeout':
         r = flask.Response(json.dumps(sub_list), mimetype='text/json')
-        cd = 'attachment; filename="subscriptions.json"'
+        cd = 'attachment; filename="subscriptions_%s.json"' % date_time
+        r.headers['Content-Disposition'] = cd
+        return r
+    elif request.values['export_format'] == 'json_newpipe':
+        r = flask.Response(json.dumps({
+            'app_version': '0.21.9',
+            'app_version_int': 975,
+            'subscriptions': sub_list,
+        }), mimetype='text/json')
+        file_name = 'newpipe_subscriptions_%s_youtube-local.json' % date_time
+        cd = 'attachment; filename="%s"' % file_name
         r.headers['Content-Disposition'] = cd
         return r
     elif request.values['export_format'] == 'opml':
         r = flask.Response(
             flask.render_template('subscriptions.xml', sub_list=sub_list),
             mimetype='text/xml')
-        cd = 'attachment; filename="subscriptions.xml"'
+        cd = 'attachment; filename="subscriptions_%s.xml"' % date_time
         r.headers['Content-Disposition'] = cd
         return r
     else:
@@ -905,7 +1032,8 @@ def get_subscriptions_page():
                     'muted': muted,
                 })
 
-    return flask.render_template('subscriptions.html',
+    return flask.render_template(
+        'subscriptions.html',
         header_playlist_names=local_playlist.get_playlist_names(),
         videos=videos,
         num_pages=math.ceil(number_of_videos_in_db/60),
@@ -961,7 +1089,7 @@ def serve_subscription_thumbnail(thumbnail):
             f.close()
             return flask.Response(image, mimetype='image/jpeg')
 
-    url = "https://i.ytimg.com/vi/" + video_id + "/mqdefault.jpg"
+    url = f"https://i.ytimg.com/vi/{video_id}/hqdefault.jpg"
     try:
         image = util.fetch_url(url, report_text="Saved thumbnail: " + video_id)
     except urllib.error.HTTPError as e:

youtube/templates/base.html

@@ -1,18 +1,31 @@
+{% if settings.app_public %}
+    {% set app_url = settings.app_url|string %}
+{% else %}
+    {% set app_url = settings.app_url|string + ':' + settings.port_number|string %}
+{% endif %}
 <!DOCTYPE html>
 <html lang="en">
     <head>
-        <meta charset="UTF-8"/>
-        <meta name="viewport" content="width=device-width, initial-scale=1"/>
-        <meta http-equiv="Content-Security-Policy" content="default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' https://*.googlevideo.com; {{ "img-src 'self' https://*.googleusercontent.com https://*.ggpht.com https://*.ytimg.com;" if not settings.proxy_images else "" }}"/>
+        <meta charset="UTF-8">
+        <meta name="viewport" content="width=device-width, initial-scale=1">
+        <meta http-equiv="Content-Security-Policy" content="default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' blob: {{ app_url }}/* data: https://*.googlevideo.com;  {{ "img-src 'self' https://*.googleusercontent.com https://*.ggpht.com https://*.ytimg.com;" if not settings.proxy_images else "" }}">
         <title>{{ page_title }}</title>
-        <link title="YT Local" href="/youtube.com/opensearch.xml" rel="search" type="application/opensearchdescription+xml"/>
-        <link href="/youtube.com/static/favicon.ico" type="image/x-icon" rel="icon"/>
-        <link href="/youtube.com/static/normalize.css" rel="stylesheet"/>
-        <link href="{{ theme_path }}" rel="stylesheet"/>
+        <link title="YT Local" href="/youtube.com/opensearch.xml" rel="search" type="application/opensearchdescription+xml">
+        <link href="/youtube.com/static/favicon.ico" type="image/x-icon" rel="icon">
+        <link href="/youtube.com/static/normalize.css" rel="stylesheet">
+        <link href="{{ theme_path }}" rel="stylesheet">
         <link href="/youtube.com/shared.css" rel="stylesheet">
         {% block style %}
             {{ style }}
         {% endblock %}
+
+        {% if js_data %}
+            <script>
+             // @license magnet:?xt=urn:btih:0b31508aeb0634b347b8270c7bee4d411b5d4109&dn=agpl-3.0.txt AGPL-v3-or-Later
+             data = {{ js_data|tojson }};
+             // @license-end
+            </script>
+        {% endif %}
     </head>
 
     <body>
@@ -22,7 +35,7 @@
             </nav>
             <form class="form" id="site-search" action="/youtube.com/results">
                 <input type="search" name="search_query" class="search-box" value="{{ search_box_value }}"
-                       {{ "autofocus" if request.path == "/" else "" }} required placeholder="Type to search...">
+                       {{ "autofocus" if (request.path in ("/", "/results") or error_message) else "" }} required placeholder="Type to search...">
                 <button type="submit" value="Search" class="search-button">Search</button>
                 <!-- options -->
                 <div class="dropdown">
@@ -120,7 +133,7 @@
 
             {% if header_playlist_names is defined %}
                 <form class="playlist" id="playlist-edit" action="/youtube.com/edit_playlist" method="post" target="_self">
-                    <input class="play-box" name="playlist_name" id="playlist-name-selection" list="playlist-options" type="search" placeholder="I added your playlist...">
+                    <input class="play-box" name="playlist_name" id="playlist-name-selection" list="playlist-options" type="search" placeholder="Add name of your playlist...">
                     <datalist class="play-hidden" id="playlist-options">
                         {% for playlist_name in header_playlist_names %}
                             <option value="{{ playlist_name }}">{{ playlist_name }}</option>
@@ -128,7 +141,7 @@
                     </datalist>
                     <button class="play-add" type="submit" id="playlist-add-button" name="action" value="add">+List</button>
                     <div class="play-clean">
-                        <button type="reset" id="item-selection-reset">Clear selection</button>
+                        <button type="reset" id="item-selection-reset">Clear</button>
                     </div>
                 </form>
                 <script src="/youtube.com/static/js/playlistadd.js"></script>
@@ -151,8 +164,8 @@
             </div>
             <div>
                 <p>This site is Free/Libre Software</p>
-                {% if current_commit and current_version %}
-                    <p>Current version: {{ current_version }}-{{ current_commit }} @ {{ current_branch }}</p>
+                {% if current_commit != None %}
+                    <p>Current version: {{ current_commit }} @ {{ current_branch }}</p>
                 {% else %}
                     <p>Current version: {{ current_version }}</p>
                 {% endif %}

youtube/templates/channel.html

@@ -1,21 +1,21 @@
 {% if current_tab == 'search' %}
     {% set page_title = search_box_value + ' - Page ' + page_number|string %}
 {% else %}
-    {% set page_title = channel_name + ' - Channel' %}
+    {% set page_title = channel_name|string + ' - Channel' %}
 {% endif %}
 
 {% extends "base.html" %}
 {% import "common_elements.html" as common_elements %}
 {% block style %}
-    <link href="/youtube.com/static/message_box.css" rel="stylesheet"/>
-    <link href="/youtube.com/static/channel.css" rel="stylesheet"/>
+    <link href="/youtube.com/static/message_box.css" rel="stylesheet">
+    <link href="/youtube.com/static/channel.css" rel="stylesheet">
 {% endblock style %}
 
 {% block main %}
 
     <div class="author-container">
         <div class="author">
-            <img alt="{{ channel_name }}" src="{{ avatar }}"/>
+            <img alt="{{ channel_name }}" src="{{ avatar }}">
             <h2>{{ channel_name }}</h2>
         </div>
         <div class="summary">
@@ -33,7 +33,7 @@
     <hr/>
 
     <nav class="channel-tabs">
-        {% for tab_name in ('Videos', 'Playlists', 'About') %}
+        {% for tab_name in ('Videos', 'Shorts', 'Streams', 'Playlists', 'About') %}
             {% if tab_name.lower() == current_tab %}
                 <a class="tab page-button">{{ tab_name }}</a>
             {% else %}
@@ -51,8 +51,11 @@
             <ul>
                 {% for (before_text, stat, after_text) in [
                     ('Joined ', date_joined, ''),
-                    ('', view_count|commatize, ' views'),
+                    ('', approx_view_count, ' views'),
                     ('', approx_subscriber_count, ' subscribers'),
+                    ('', approx_video_count, ' videos'),
+                    ('Country: ', country, ''),
+                    ('Canonical Url: ', canonical_url, ''),
                 ] %}
                     {% if stat %}
                         <li>{{ before_text + stat|string + after_text }}</li>
@@ -65,7 +68,11 @@
             <hr>
             <ul>
                 {% for text, url in links %}
-                    <li><a href="{{ url }}">{{ text }}</a></li>
+                    {% if url %}
+                        <li><a href="{{ url }}">{{ text }}</a></li>
+                    {% else %}
+                        <li>{{ text }}</li>
+                    {% endif %}
                 {% endfor %}
             </ul>
         </div>
@@ -73,8 +80,8 @@
 
         <!-- new-->
         <div id="links-metadata">
-                {% if current_tab == 'videos' %}
-                    {% set sorts = [('1', 'views'), ('2', 'oldest'), ('3', 'newest')] %}
+                {% if current_tab in ('videos', 'shorts', 'streams') %}
+                    {% set sorts = [('1', 'views'), ('2', 'oldest'), ('3', 'newest'), ('4', 'newest - no shorts'),] %}
                     <div id="number-of-results">{{ number_of_videos }} videos</div>
                 {% elif current_tab == 'playlists' %}
                     {% set sorts = [('2', 'oldest'), ('3', 'newest'), ('4', 'last video added')] %}
@@ -110,13 +117,9 @@
         <hr/>
 
         <footer class="pagination-container">
-            {% if current_tab == 'videos' and current_sort.__str__() == '2' %}
-                <nav class="next-previous-button-row">
-                    {{ common_elements.next_previous_ctoken_buttons(None, ctoken, channel_url + '/' + current_tab, parameters_dictionary) }}
-                </nav>
-            {% elif current_tab == 'videos' %}
+            {% if current_tab in ('videos', 'shorts', 'streams') %}
                 <nav class="pagination-list">
-                    {{ common_elements.page_buttons(number_of_pages, channel_url + '/' + current_tab, parameters_dictionary, include_ends=(current_sort.__str__() == '3')) }}
+                    {{ common_elements.page_buttons(number_of_pages, channel_url + '/' + current_tab, parameters_dictionary, include_ends=(current_sort.__str__() in '34')) }}
                 </nav>
             {% elif current_tab == 'playlists' or current_tab == 'search' %}
                 <nav class="next-previous-button-row">

youtube/templates/common_elements.html

@@ -38,10 +38,21 @@
                 <h4 class="title"><a href="{{ info['url'] }}" title="{{ info['title'] }}">{{ info['title'] }}</a></h4>
 
                 {% if include_author %}
+                    {% set author_description = info['author'] %}
+                    {% set AUTHOR_DESC_LENGTH = 35 %}
+                    {% if author_description != None %}
+                        {% if author_description|length >= AUTHOR_DESC_LENGTH %}
+                            {% set author_description = author_description[:AUTHOR_DESC_LENGTH].split(' ')[:-1]|join(' ') %}
+                            {% if not author_description[-1] in ['.', '?', ':', '!'] %}
+                                {% set author_more = author_description + '…' %}
+                                {% set author_description = author_more|replace('"','') %}
+                            {% endif %}
+                        {% endif %}
+                    {% endif %}
                     {% if info.get('author_url') %}
-                        <address title="{{ info['author'] }}"><b><a href="{{ info['author_url'] }}">{{ info['author'] }}</a></b></address>
+                        <address title="{{ info['author'] }}"><b><a href="{{ info['author_url'] }}">{{ author_description }}</a></b></address>
                     {% else %}
-                        <address title="{{ info['author'] }}"><b>{{ info['author'] }}</b></address>
+                        <address title="{{ info['author'] }}"><b>{{ author_description }}</b></address>
                     {% endif %}
                 {% endif %}
 

youtube/templates/embed.html

@@ -1,14 +1,14 @@
 <!DOCTYPE html>
-<html lang="es">
+<html lang="en">
     <head>
-        <meta charset="UTF-8"/>
-        <meta name="viewport" content="width=device-width, initial-scale=1"/>
-        <meta http-equiv="Content-Security-Policy" content="default-src 'self' 'unsafe-inline'; media-src 'self' https://*.googlevideo.com; {{ "img-src 'self' https://*.googleusercontent.com https://*.ggpht.com https://*.ytimg.com;" if not settings.proxy_images else "" }}"/>
+        <meta charset="UTF-8">
+        <meta name="viewport" content="width=device-width, initial-scale=1">
+        <meta http-equiv="Content-Security-Policy" content="default-src 'self' 'unsafe-inline'; media-src 'self' https://*.googlevideo.com; {{ "img-src 'self' https://*.googleusercontent.com https://*.ggpht.com https://*.ytimg.com;" if not settings.proxy_images else "" }}">
         <title>{{ title }}</title>
-        <link href="/youtube.com/static/favicon.ico" type="image/x-icon" rel="icon"/>
+        <link href="/youtube.com/static/favicon.ico" type="image/x-icon" rel="icon">
         {% if settings.use_video_player == 2 %}
             <!-- plyr -->
-            <link href="/youtube.com/static/modules/plyr/plyr.css" rel="stylesheet"/>
+            <link href="/youtube.com/static/modules/plyr/plyr.css" rel="stylesheet">
             <!--/ plyr -->
         {% endif %}
         <style>
@@ -26,13 +26,20 @@
              display: none !important;
          }
         </style>
+        {% if js_data %}
+            <script>
+             // @license magnet:?xt=urn:btih:0b31508aeb0634b347b8270c7bee4d411b5d4109&dn=agpl-3.0.txt AGPL-v3-or-Later
+             data = {{ js_data|tojson }};
+             // @license-end
+            </script>
+        {% endif %}
     </head>
     <body>
         <video id="js-video-player" controls autofocus onmouseleave="{{ title }}"
                oncontextmenu="{{ title }}" onmouseenter="{{ title }}" title="{{ title }}">
-            {% for video_source in video_sources %}
-                <source src="{{ video_source['src'] }}" type="{{ video_source['type'] }}">
-            {% endfor %}
+            {% if uni_sources %}
+                <source src="{{ uni_sources[uni_idx]['url'] }}" type="{{ uni_sources[uni_idx]['type'] }}" data-res="{{ uni_sources[uni_idx]['quality'] }}">
+            {% endif %}
             {% for source in subtitle_sources %}
                 {% if source['on'] %}
                     <track label="{{ source['label'] }}" src="{{ source['url'] }}" kind="subtitles" srclang="{{ source['srclang'] }}" default>
@@ -48,10 +55,15 @@
              // @license-end
             </script>
         {% endif %}
+        <script>
+         // @license magnet:?xt=urn:btih:0b31508aeb0634b347b8270c7bee4d411b5d4109&dn=agpl-3.0.txt AGPL-v3-or-Later
+         let storyboard_url = {{ storyboard_url | tojson }};
+         // @license-end
+        </script>
         {% if settings.use_video_player == 2 %}
             <!-- plyr -->
             <script src="/youtube.com/static/modules/plyr/plyr.min.js"
-                    integrity="sha512-LxSGuB4I2iAln3VLWi8t3RYhEks4/2rtcCw6kqiBghbqBJHXg5ikpeRxEOm0luiIuKDiqwNI3rsCXI/d+MPPAA=="
+                    integrity="sha512-l6ZzdXpfMHRfifqaR79wbYCEWjLDMI9DnROvb+oLkKq6d7MGroGpMbI7HFpicvmAH/2aQO+vJhewq8rhysrImw=="
                     crossorigin="anonymous"></script>
             <script src="/youtube.com/static/js/plyr-start.js"></script>
             <!-- /plyr -->

youtube/templates/error.html

@@ -1,4 +1,8 @@
-{% set page_title = 'Error' %}
+{% if error_code %}
+    {% set page_title = 'Error: ' ~ error_code %}
+{% else %}
+    {% set page_title = 'Error' %}
+{% endif %}
 
 {% if not slim %}
     {% extends "base.html" %}

youtube/templates/home.html

@@ -1,7 +1,7 @@
 {% set page_title = title %}
 {% extends "base.html" %}
 {% block style %}
-    <link href="/youtube.com/static/home.css" rel="stylesheet"/>
+    <link href="/youtube.com/static/home.css" rel="stylesheet">
 {% endblock style %}
 {% block main %}
     <ul>

youtube/templates/licenses.html

@@ -1,7 +1,7 @@
 {% set page_title = title %}
 {% extends "base.html" %}
 {% block style %}
-    <link href="/youtube.com/static/license.css" rel="stylesheet"/>
+    <link href="/youtube.com/static/license.css" rel="stylesheet">
 {% endblock style %}
 {% block main %}
     <table id="jslicense-labels1" class="table">
@@ -14,6 +14,11 @@
             </tr>
         </thead>
         <tbody>
+            <tr>
+                <td data-label="File"><a href="/youtube.com/static/js/av-merge.js">av-merge.js</a></td>
+                <td data-label="License"><a href="http://www.gnu.org/licenses/agpl-3.0.html">AGPL-3.0 or later</a></td>
+                <td data-label="Source"><a href="/youtube.com/static/js/av-merge.js">av-merge.js</a></td>
+            </tr>
             <tr>
                 <td data-label="File"><a href="/youtube.com/static/js/comments.js">comments.js</a></td>
                 <td data-label="License"><a href="http://www.gnu.org/licenses/agpl-3.0.html">AGPL-3.0 or later</a></td>
@@ -45,15 +50,15 @@
                 <td data-label="Source"><a href="/youtube.com/static/modules/plyr/plyr.js">plyr.js</a></td>
             </tr>
             <tr>
-                <td data-label="File"><a href="/youtube.com/static/js/speedyplay.js">speedyplay.js</a></td>
-                <td data-label="License"><a href="http://www.gnu.org/licenses/agpl-3.0.html">AGPL-3.0 or later</a></td>
-                <td data-label="Source"><a href="/youtube.com/static/js/speedyplay.js">speedyplay.js</a></td>
-            </tr>
-             <tr>
                 <td data-label="File"><a href="/youtube.com/static/js/transcript-table.js">transcript-table.js</a></td>
                 <td data-label="License"><a href="http://www.gnu.org/licenses/agpl-3.0.html">AGPL-3.0 or later</a></td>
                 <td data-label="Source"><a href="/youtube.com/static/js/transcript-table.js">transcript-table.js</a></td>
-             </tr>
+            </tr>
+            <tr>
+                <td data-label="File"><a href="/youtube.com/static/js/watch.js">watch.js</a></td>
+                <td data-label="License"><a href="http://www.gnu.org/licenses/agpl-3.0.html">AGPL-3.0 or later</a></td>
+                <td data-label="Source"><a href="/youtube.com/static/js/watch.js">watch.js</a></td>
+            </tr>
         </tbody>
     </table>
 {% endblock main %}

youtube/templates/local_playlist.html

@@ -2,8 +2,8 @@
 {% extends "base.html" %}
 {% import "common_elements.html" as common_elements %}
 {% block style %}
-    <link href="/youtube.com/static/message_box.css" rel="stylesheet"/>
-    <link href="/youtube.com/static/local_playlist.css" rel="stylesheet"/>
+    <link href="/youtube.com/static/message_box.css" rel="stylesheet">
+    <link href="/youtube.com/static/local_playlist.css" rel="stylesheet">
 {% endblock style %}
 
 {% block main %}
@@ -22,7 +22,9 @@
         </div>
     </div>
 
+    <form id="playlist-remove" action="/youtube.com/edit_playlist" method="post" target="_self"></form>
     <div class="playlist-metadata" id="video-remove-container">
+        <button id="removePlayList" type="submit" name="action" value="remove_playlist" form="playlist-remove" formaction="">Remove playlist</button>
         <input type="hidden" name="playlist_page" value="{{ playlist_name }}" form="playlist-edit">
         <button class="play-action" type="submit" id="playlist-remove-button" name="action" value="remove" form="playlist-edit" formaction="">Remove from playlist</button>
     </div>
@@ -31,6 +33,14 @@
             {{ common_elements.item(video_info) }}
         {% endfor %}
     </div>
+    <script>
+     // @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt GPL-v3-or-Later
+     const deletePlayList = document.getElementById('removePlayList');
+     deletePlayList.addEventListener('click', (event) => {
+         return confirm('You are about to permanently delete {{ playlist_name }}\n\nOnce a playlist is permanently deleted, it cannot be recovered.')
+     });
+     // @license-end
+    </script>
     <footer class="pagination-container">
         <nav class="pagination-list">
             {{ common_elements.page_buttons(num_pages, '/https://www.youtube.com/playlists/' + playlist_name, parameters_dictionary) }}

youtube/templates/playlist.html

@@ -2,15 +2,15 @@
 {% extends "base.html" %}
 {% import "common_elements.html" as common_elements %}
 {% block style %}
-    <link href="/youtube.com/static/message_box.css" rel="stylesheet"/>
-    <link href="/youtube.com/static/playlist.css" rel="stylesheet"/>
+    <link href="/youtube.com/static/message_box.css" rel="stylesheet">
+    <link href="/youtube.com/static/playlist.css" rel="stylesheet">
 {% endblock style %}
 
 {% block main %}
 
     <div class="playlist-metadata">
         <div class="author">
-            <img alt="{{ title }}" src="{{ thumbnail }}"/>
+            <img alt="{{ title }}" src="{{ thumbnail }}">
             <h2>{{ title }}</h2>
         </div>
         <div class="summary">

youtube/templates/search.html

@@ -3,8 +3,8 @@
 {% extends "base.html" %}
 {% import "common_elements.html" as common_elements %}
 {% block style %}
-    <link href="/youtube.com/static/message_box.css" rel="stylesheet"/>
-    <link href="/youtube.com/static/search.css" rel="stylesheet"/>
+    <link href="/youtube.com/static/message_box.css" rel="stylesheet">
+    <link href="/youtube.com/static/search.css" rel="stylesheet">
 {% endblock style %}
 
 {% block main %}

youtube/templates/settings.html

@@ -1,7 +1,7 @@
 {% set page_title = 'Settings' %}
 {% extends "base.html" %}
 {% block style %}
-    <link href="/youtube.com/static/settings.css" rel="stylesheet"/>
+    <link href="/youtube.com/static/settings.css" rel="stylesheet">
 {% endblock style %}
 
 {% block main %}
@@ -13,9 +13,9 @@
                     {% if not setting_info.get('hidden', false) %}
                         <li class="setting-item">
                             {% if 'label' is in(setting_info) %}
-                                <label for="{{ 'setting_' + setting_name }}">{{ setting_info['label'] }}</label>
+                                <label for="{{ 'setting_' + setting_name }}" {% if 'comment' is in(setting_info) %}title="{{ setting_info['comment'] }}" {% endif %}>{{ setting_info['label'] }}</label>
                             {% else %}
-                                <label for="{{ 'setting_' + setting_name }}">{{ setting_name.replace('_', ' ')|capitalize }}</label>
+                                <label for="{{ 'setting_' + setting_name }}" {% if 'comment' is in(setting_info) %}title="{{ setting_info['comment'] }}" {% endif %}>{{ setting_name.replace('_', ' ')|capitalize }}</label>
                             {% endif %}
 
                             {% if setting_info['type'].__name__ == 'bool' %}

youtube/templates/subscription_manager.html

@@ -1,7 +1,7 @@
 {% set page_title = 'Subscription Manager' %}
 {% extends "base.html" %}
 {% block style %}
-    <link href="/youtube.com/static/subscription_manager.css" rel="stylesheet"/>
+    <link href="/youtube.com/static/subscription_manager.css" rel="stylesheet">
 {% endblock style %}
 
 
@@ -20,7 +20,7 @@
         <form class="subscriptions-import-form" enctype="multipart/form-data" action="/youtube.com/import_subscriptions" method="POST">
             <h2>Import subscriptions</h2>
             <div class="subscriptions-import-options">
-                <input type="file" id="subscriptions-import" accept="application/json, application/xml, text/x-opml" name="subscriptions_file">
+                <input type="file" id="subscriptions-import" accept="application/json, application/xml, text/x-opml, text/csv" name="subscriptions_file" required>
                 <input type="submit" value="Import">
             </div>
         </form>
@@ -29,7 +29,8 @@
             <h2>Export subscriptions</h2>
             <div class="subscriptions-export-options">
                 <select id="export-type" name="export_format" title="Export format">
-                    <option value="json">JSON</option>
+                    <option value="json_newpipe">JSON (NewPipe)</option>
+                    <option value="json_google_takeout">JSON (Old Google Takeout Format)</option>
                     <option value="opml">OPML (RSS, no tags)</option>
                 </select>
                 <label for="include-muted">Include muted</label>

youtube/templates/subscriptions.html

@@ -7,8 +7,8 @@
 {% import "common_elements.html" as common_elements %}
 
 {% block style %}
-    <link href="/youtube.com/static/message_box.css" rel="stylesheet"/>
-    <link href="/youtube.com/static/subscription.css" rel="stylesheet"/>
+    <link href="/youtube.com/static/message_box.css" rel="stylesheet">
+    <link href="/youtube.com/static/subscription.css" rel="stylesheet">
 {% endblock style %}
 
 {% block main %}

youtube/templates/unsubscribe_verify.html

@@ -1,17 +1,19 @@
 {% set page_title = 'Unsubscribe?' %}
 {% extends "base.html" %}
+{% block style %}
+    <link href="/youtube.com/static/unsubscribe.css" rel="stylesheet"/>
+{% endblock style %}
 
 {% block main %}
-    <span>Are you sure you want to unsubscribe from these channels?</span>
+    <p>Are you sure you want to unsubscribe from these channels?</p>
     <form class="subscriptions-import-form" action="/youtube.com/subscription_manager" method="POST">
         {% for channel_id, channel_name in unsubscribe_list %}
             <input type="hidden" name="channel_ids" value="{{ channel_id }}">
         {% endfor %}
-
         <input type="hidden" name="action" value="unsubscribe">
         <input type="submit" value="Yes, unsubscribe">
     </form>
-    <ul>
+    <ul class="list-channel">
         {% for channel_id, channel_name in unsubscribe_list %}
             <li><a href="{{ '/https://www.youtube.com/channel/' + channel_id }}" title="{{ channel_name }}">{{ channel_name }}</a></li>
         {% endfor %}

youtube/templates/watch.html

@@ -3,19 +3,13 @@
 {% import "common_elements.html" as common_elements %}
 {% import "comments.html" as comments with context %}
 {% block style %}
-    <link href="/youtube.com/static/message_box.css" rel="stylesheet"/>
-    <link href="/youtube.com/static/watch.css" rel="stylesheet"/>
+    <link href="/youtube.com/static/message_box.css" rel="stylesheet">
+    <link href="/youtube.com/static/watch.css" rel="stylesheet">
     {% if settings.use_video_player == 2 %}
         <!-- plyr -->
-        <link href="/youtube.com/static/modules/plyr/plyr.css" rel="stylesheet"/>
+        <link href="/youtube.com/static/modules/plyr/plyr.css" rel="stylesheet">
+        <link href="/youtube.com/static/modules/plyr/custom_plyr.css" rel="stylesheet">
         <!--/ plyr -->
-        <style>
-         /* Prevent this div from blocking right-click menu for video
-         e.g. Firefox playback speed options */
-         .plyr__poster {
-             display: none !important;
-         }
-        </style>
     {% endif %}
 {% endblock style %}
 
@@ -29,7 +23,7 @@
                 {% endif %}
             </span>
         </div>
-    {% elif (video_sources.__len__() == 0 or live) and hls_formats.__len__() != 0 %}
+    {% elif (uni_sources.__len__() == 0 or live) and hls_formats.__len__() != 0 %}
         <div class="live-url-choices">
             <span>Copy a url into your video player:</span>
             <ol>
@@ -40,10 +34,10 @@
         </div>
     {% else %}
         <figure class="sc-video">
-            <video id="js-video-player" playsinline controls>
-                {% for video_source in video_sources %}
-                    <source src="{{ video_source['src'] }}" type="{{ video_source['type'] }}" data-res="{{ video_source['quality'] }}">
-                {% endfor %}
+            <video id="js-video-player" playsinline controls {{ 'autoplay' if settings.autoplay_videos }}>
+                {% if uni_sources %}
+                    <source src="{{ uni_sources[uni_idx]['url'] }}" type="{{ uni_sources[uni_idx]['type'] }}" data-res="{{ uni_sources[uni_idx]['quality'] }}">
+                {% endif %}
 
                 {% for source in subtitle_sources %}
                     {% if source['on'] %}
@@ -54,12 +48,6 @@
                 {% endfor %}
             </video>
         </figure>
-
-        {% if time_start != 0 %}
-            <script>
-             document.getElementById('js-video-player').currentTime = {{ time_start|tojson }};
-            </script>
-        {% endif %}
     {% endif %}
 
     <div class="sc-info">
@@ -84,17 +72,26 @@
             <address class="v-uploaded">Uploaded by <a href="{{ uploader_channel_url }}">{{ uploader }}</a></address>
             <span class="v-views">{{ view_count }} views</span>
             <time class="v-published" datetime="{{ time_published_utc }}">Published on {{ time_published }}</time>
-            <span class="v-likes-dislikes">{{ like_count }} likes {{ dislike_count }} dislikes</span>
+            <span class="v-likes-dislikes">{{ like_count }} likes</span>
 
             <div class="external-player-controls">
                 <input class="speed" id="speed-control" type="text" title="Video speed">
-                <script src="/youtube.com/static/js/speedyplay.js"></script>
+                {% if settings.use_video_player != 2 %}
+                    <select id="quality-select" autocomplete="off">
+                        {% for src in uni_sources %}
+                            <option value='{"type": "uni", "index": {{ loop.index0 }}}' {{ 'selected' if loop.index0 == uni_idx and not using_pair_sources else '' }} >{{ src['quality_string'] }}</option>
+                        {% endfor %}
+                        {% for src_pair in pair_sources %}
+                            <option value='{"type": "pair", "index": {{ loop.index0}}}' {{ 'selected' if loop.index0 == pair_idx and using_pair_sources else '' }} >{{ src_pair['quality_string'] }}</option>
+                        {% endfor %}
+                    </select>
+                {% endif %}
             </div>
-
             <input class="v-checkbox" name="video_info_list" value="{{ video_info }}" form="playlist-edit" type="checkbox">
 
             <span class="v-direct-link"><a href="https://youtu.be/{{ video_id }}" rel="noopener noreferrer" target="_blank">Direct Link</a></span>
 
+            {% if settings.use_video_download != 0 %}
             <details class="v-download">
                 <summary class="download-dropdown-label">Download</summary>
                 <ul class="download-dropdown-content">
@@ -114,6 +111,9 @@
                     {% endfor %}
                 </ul>
             </details>
+            {% else %}
+            <span class="v-download"></span>
+            {% endif %}
             <span class="v-description">{{ common_elements.text_runs(description)|escape|urlize|timestamps|safe }}</span>
 
             <div class="v-music-list">
@@ -129,7 +129,11 @@
                         {% for track in music_list %}
                             <tr>
                                 {% for attribute in music_attributes %}
-                                    <td>{{ track.get(attribute.lower(), '') }}</td>
+                                    {% if attribute.lower() == 'title' and track['url'] is not none %}
+                                        <td><a href="{{ track['url'] }}">{{ track.get(attribute.lower(), '') }}</a></td>
+                                    {% else %}
+                                        <td>{{ track.get(attribute.lower(), '') }}</td>
+                                    {% endif %}
                                 {% endfor %}
                             </tr>
                         {% endfor %}
@@ -161,7 +165,7 @@
                     <div class="playlist-header">
                         <a href="{{ playlist['url'] }}" title="{{ playlist['title'] }}"><h3>{{ playlist['title'] }}</h3></a>
                         <ul class="playlist-metadata">
-                            <li><label for="playlist-autoplay-toggle">Autoplay: </label><input type="checkbox" id="playlist-autoplay-toggle"></li>
+                            <li><label for="playlist-autoplay-toggle">Autoplay: </label><input id="playlist-autoplay-toggle" type="checkbox" class="autoplay-toggle"></li>
                             {% if playlist['current_index'] is none %}
                                 <li>[Error!]/{{ playlist['video_count'] }}</li>
                             {% else %}
@@ -182,173 +186,11 @@
                             {% endif %}
                         {% endfor %}
                     </nav>
-                    {% if playlist['id'] is not none %}
-                        <script>
-                         // @license magnet:?xt=urn:btih:0b31508aeb0634b347b8270c7bee4d411b5d4109&dn=agpl-3.0.txt AGPL-v3-or-Later
-                         // lazy load playlist images
-                         // copied almost verbatim from
-                         // https://css-tricks.com/tips-for-rolling-your-own-lazy-loading/
-                         // IntersectionObserver isn't supported in pre-quantum
-                         // firefox versions, but the alternative of making it
-                         // manually is a performance drain, so oh well
-                         let observer = new IntersectionObserver(lazyLoad, {
-
-                             // where in relation to the edge of the viewport, we are observing
-                             rootMargin: "100px",
-                             // how much of the element needs to have intersected
-                             // in order to fire our loading function
-                             threshold: 1.0
-                         });
-
-                         function lazyLoad(elements) {
-                             elements.forEach(item => {
-                                 if (item.intersectionRatio > 0) {
-                                     // set the src attribute to trigger a load
-                                     item.target.src = item.target.dataset.src;
-                                     // stop observing this element. Our work here is done!
-                                     observer.unobserve(item.target);
-                                 };
-                             });
-                         };
-
-                         // Tell our observer to observe all img elements with a "lazy" class
-                         let lazyImages = document.querySelectorAll('img.lazy');
-                         lazyImages.forEach(img => {
-                             observer.observe(img);
-                         });
-                         // @license-end
-                        </script>
-                    {% endif %}
                 </div>
             {% elif settings.related_videos_mode != 0 %}
-                <div class="related-autoplay"><label for="related-autoplay-toggle">Autoplay: </label><input type="checkbox" id="related-autoplay-toggle"></div>
+                <div class="related-autoplay"><label for="related-autoplay-toggle">Autoplay: </label><input id="related-autoplay-toggle" type="checkbox" class="autoplay-toggle"></div>
             {% endif %}
 
-            {% if settings.related_videos_mode != 0 or playlist %}
-                <script>
-                 // @license magnet:?xt=urn:btih:0b31508aeb0634b347b8270c7bee4d411b5d4109&dn=agpl-3.0.txt AGPL-v3-or-Later
-                 let playability_error = {{ 'true' if playability_error else 'false' }};
-                 {% if playlist and playlist['current_index'] is not none %}
-                 {% set isPlaylist = true %}
-                 {% endif %}
-
-                 {% if isPlaylist %}
-                 // from https://stackoverflow.com/a/6969486
-                 function escapeRegExp(string) {
-                     return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string
-                 }
-                 let playlist_id = {{ playlist['id']|tojson }};
-                 playlist_id = escapeRegExp(playlist_id);
-                 {% endif %}
-
-                 // read cookies on whether to autoplay
-                 // pain in the ass:
-                 // https://developer.mozilla.org/en-US/docs/Web/API/Document/cookie
-                 {% if isPlaylist %}
-                 let cookieValue = document.cookie.replace(new RegExp(
-                     '(?:(?:^|.*;\\s*)autoplay_' + playlist_id + '\\s*\\=\\s*([^;]*).*$)|^.*$'), '$1');
-                 {% else %}
-                 let cookieValue = document.cookie.replace(new RegExp(
-                     '(?:(?:^|.*;\\s*)autoplay\\s*\\=\\s*([^;]*).*$)|^.*$'), '$1');
-                 {% endif %}
-
-                 let autoplayEnabled = 0;
-                 if(cookieValue.length === 0){
-                     autoplayEnabled = 0;
-                 } else {
-                     autoplayEnabled = Number(cookieValue);
-                 }
-
-                 // check the autoplay checkbox if autoplay is on
-                 {% if isPlaylist %}
-                 let PlaylistAutoplayCheck = document.getElementById('playlist-autoplay-toggle');
-                 if(autoplayEnabled){
-                     PlaylistAutoplayCheck.checked = true;
-                 }
-                 {% else %}
-                 let RelatedAutoplayCheck = document.getElementById('related-autoplay-toggle');
-                 if(autoplayEnabled){
-                     RelatedAutoplayCheck.checked = true;
-                 }
-                 {% endif %}
-
-                 // listen for checkbox to turn autoplay on and off
-                 let cookie = 'autoplay'
-                 {% if isPlaylist %}
-                 cookie += '_' + playlist_id;
-                 PlaylistAutoplayCheck.addEventListener( 'change', function() {
-                     if(this.checked) {
-                         autoplayEnabled = 1;
-                         document.cookie = cookie + '=1; SameSite=Strict';
-                     } else {
-                         autoplayEnabled = 0;
-                         document.cookie = cookie + '=0; SameSite=Strict';
-                     }
-                 });
-                 {% else %}
-                 RelatedAutoplayCheck.addEventListener( 'change', function() {
-                     if(this.checked) {
-                         autoplayEnabled = 1;
-                         document.cookie = cookie + '=1; SameSite=Strict';
-                     } else {
-                         autoplayEnabled = 0;
-                         document.cookie = cookie + '=0; SameSite=Strict';
-                     }
-                 });
-                 {% endif %}
-
-                 const vid = document.getElementById('js-video-player');
-                 if(!playability_error){
-                     // play the video if autoplay is on
-                     if(autoplayEnabled){
-                         vid.play();
-                     }
-                 }
-
-                 // determine next video url
-                 {% if isPlaylist %}
-                 let currentIndex = {{ playlist['current_index']|tojson }};
-                 {% if playlist['current_index']+1 == playlist['items']|length %}
-                 let nextVideoUrl = null;
-                 {% else %}
-                 let nextVideoUrl = {{ (playlist['items'][playlist['current_index']+1]['url'])|tojson }};
-                 {% endif %}
-
-                 // scroll playlist to proper position
-                 // item height + gap == 100
-                 let pl = document.querySelector('.playlist-videos');
-                 pl.scrollTop = 100*currentIndex;
-                 {% else %}
-                 {% if related|length == 0 %}
-                 let nextVideoUrl = null;
-                 {% else %}
-                 let nextVideoUrl = {{ (related[0]['url'])|tojson }};
-                 {% endif %}
-                 {% endif %}
-                 let nextVideoDelay = 1000;
-
-                 // go to next video when video ends
-                 // https://stackoverflow.com/a/2880950
-                 if (nextVideoUrl) {
-                     if(playability_error){
-                         videoEnded();
-                     } else {
-                         vid.addEventListener('ended', videoEnded, false);
-                     }
-                     function nextVideo(){
-                         if(autoplayEnabled){
-                             window.location.href = nextVideoUrl;
-                         }
-                     }
-                     function videoEnded(e) {
-                         window.setTimeout(nextVideo, nextVideoDelay);
-                     }
-                 }
-                 // @license-end
-                </script>
-            {% endif %}
-            <!-- /playlist -->
-
             {% if subtitle_sources %}
                 <details id="transcript-details">
                     <summary>Transcript</summary>
@@ -385,7 +227,7 @@
                 <div class="comments-area-outer comments-disabled">Comments disabled</div>
             {% else %}
                 <details class="comments-area-outer" {{'open' if settings.comments_mode == 1 else ''}}>
-                    <summary>{{ comment_count|commatize }} comment{{'s' if comment_count != 1 else ''}}</summary>
+                    <summary>{{ comment_count|commatize }} comment{{'s' if comment_count != '1' else ''}}</summary>
                     <div class="comments-area-inner comments-area">
                         {% if comments_info %}
                             {{ comments.video_comments(comments_info) }}
@@ -397,9 +239,11 @@
 
     </div>
 
+    <script src="/youtube.com/static/js/av-merge.js"></script>
+    <script src="/youtube.com/static/js/watch.js"></script>
     <script>
      // @license magnet:?xt=urn:btih:0b31508aeb0634b347b8270c7bee4d411b5d4109&dn=agpl-3.0.txt AGPL-v3-or-Later
-     data = {{ js_data|tojson }};
+     let storyboard_url = {{ storyboard_url | tojson }};
      // @license-end
     </script>
     <script src="/youtube.com/static/js/common.js"></script>
@@ -407,7 +251,7 @@
     {% if settings.use_video_player == 2 %}
         <!-- plyr -->
         <script src="/youtube.com/static/modules/plyr/plyr.min.js"
-                integrity="sha512-LxSGuB4I2iAln3VLWi8t3RYhEks4/2rtcCw6kqiBghbqBJHXg5ikpeRxEOm0luiIuKDiqwNI3rsCXI/d+MPPAA=="
+                integrity="sha512-l6ZzdXpfMHRfifqaR79wbYCEWjLDMI9DnROvb+oLkKq6d7MGroGpMbI7HFpicvmAH/2aQO+vJhewq8rhysrImw=="
                 crossorigin="anonymous"></script>
         <script src="/youtube.com/static/js/plyr-start.js"></script>
         <!-- /plyr -->

youtube/util.py

@@ -268,14 +268,15 @@ def fetch_url_response(url, headers=(), timeout=15, data=None,
         # According to the documentation for urlopen, a redirect counts as a
         # retry. So there are 3 redirects max by default.
         if max_redirects:
-            retries = urllib3.Retry(3+max_redirects, redirect=max_redirects)
+            retries = urllib3.Retry(3+max_redirects, redirect=max_redirects, raise_on_redirect=False)
         else:
-            retries = urllib3.Retry(3)
+            retries = urllib3.Retry(3, raise_on_redirect=False)
         pool = get_pool(use_tor and settings.route_tor)
         try:
             response = pool.request(method, url, headers=headers, body=data,
                                     timeout=timeout, preload_content=False,
                                     decode_content=False, retries=retries)
+            response.retries = retries
         except urllib3.exceptions.MaxRetryError as e:
             exception_cause = e.__context__.__context__
             if (isinstance(exception_cause, socks.ProxyConnectionError)
@@ -317,10 +318,11 @@ def fetch_url(url, headers=(), timeout=15, report_text=None, data=None,
         cleanup_func(response)  # release_connection for urllib3
         content = decode_content(
             content,
-            response.getheader('Content-Encoding', default='identity'))
+            response.headers.get('Content-Encoding', default='identity'))
 
         if (settings.debugging_save_responses
-                and debug_name is not None and content):
+                and debug_name is not None
+                and content):
             save_dir = os.path.join(settings.data_dir, 'debug')
             if not os.path.exists(save_dir):
                 os.makedirs(save_dir)
@@ -328,11 +330,22 @@ def fetch_url(url, headers=(), timeout=15, report_text=None, data=None,
             with open(os.path.join(save_dir, debug_name), 'wb') as f:
                 f.write(content)
 
-        if response.status == 429:
+        if response.status == 429 or (
+            response.status == 302 and (response.getheader('Location') == url
+                or response.getheader('Location').startswith(
+                       'https://www.google.com/sorry/index'
+                   )
+            )
+        ):
+            print(response.status, response.reason, response.headers)
             ip = re.search(
                 br'IP address: ((?:[\da-f]*:)+[\da-f]+|(?:\d+\.)+\d+)',
                 content)
             ip = ip.group(1).decode('ascii') if ip else None
+            if not ip:
+                ip = re.search(r'IP=((?:\d+\.)+\d+)',
+                               response.getheader('Set-Cookie') or '')
+                ip = ip.group(1) if ip else None
 
             # don't get new identity if we're not using Tor
             if not use_tor:
@@ -382,7 +395,6 @@ def head(url, use_tor=False, report_text=None, max_redirects=10):
             round(time.monotonic() - start_time, 3))
     return response
 
-
 mobile_user_agent = 'Mozilla/5.0 (Linux; Android 7.0; Redmi Note 4 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Mobile Safari/537.36'
 mobile_ua = (('User-Agent', mobile_user_agent),)
 desktop_user_agent = 'Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0'
@@ -392,13 +404,13 @@ desktop_xhr_headers = (
     ('Accept', '*/*'),
     ('Accept-Language', 'en-US,en;q=0.5'),
     ('X-YouTube-Client-Name', '1'),
-    ('X-YouTube-Client-Version', '2.20180830'),
+    ('X-YouTube-Client-Version', '2.20240304.00.00'),
 ) + desktop_ua
 mobile_xhr_headers = (
     ('Accept', '*/*'),
     ('Accept-Language', 'en-US,en;q=0.5'),
     ('X-YouTube-Client-Name', '2'),
-    ('X-YouTube-Client-Version', '2.20180830'),
+    ('X-YouTube-Client-Version', '2.20240304.08.00'),
 ) + mobile_ua
 
 
@@ -450,7 +462,7 @@ class RateLimitedQueue(gevent.queue.Queue):
 
 
 def download_thumbnail(save_directory, video_id):
-    url = "https://i.ytimg.com/vi/" + video_id + "/mqdefault.jpg"
+    url = f"https://i.ytimg.com/vi/{video_id}/hqdefault.jpg"
     save_location = os.path.join(save_directory, video_id + ".jpg")
     try:
         thumbnail = fetch_url(url, report_text="Saved thumbnail: " + video_id)
@@ -492,7 +504,7 @@ def video_id(url):
 
 # default, sddefault, mqdefault, hqdefault, hq720
 def get_thumbnail_url(video_id):
-    return settings.img_prefix + "https://i.ytimg.com/vi/" + video_id + "/mqdefault.jpg"
+    return f"{settings.img_prefix}https://i.ytimg.com/vi/{video_id}/hqdefault.jpg"
 
 
 def seconds_to_timestamp(seconds):
@@ -653,8 +665,183 @@ def to_valid_filename(name):
     return name
 
 
+# https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/youtube.py#L72
+INNERTUBE_CLIENTS = {
+    'android': {
+        'INNERTUBE_API_KEY': 'AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w',
+        'INNERTUBE_CONTEXT': {
+            'client': {
+                'hl': 'en',
+                'gl': 'US',
+                'clientName': 'ANDROID',
+                'clientVersion': '19.09.36',
+                'osName': 'Android',
+                'osVersion': '12',
+                'androidSdkVersion': 31,
+                'platform': 'MOBILE',
+                'userAgent': 'com.google.android.youtube/19.09.36 (Linux; U; Android 12; US) gzip'
+            },
+            # https://github.com/yt-dlp/yt-dlp/pull/575#issuecomment-887739287
+            #'thirdParty': {
+            #    'embedUrl': 'https://google.com',  # Can be any valid URL
+            #}
+        },
+        'INNERTUBE_CONTEXT_CLIENT_NAME': 3,
+        'REQUIRE_JS_PLAYER': False,
+    },
+
+    'android-test-suite': {
+        'INNERTUBE_API_KEY': 'AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w',
+        'INNERTUBE_CONTEXT': {
+            'client': {
+                'hl': 'en',
+                'gl': 'US',
+                'clientName': 'ANDROID_TESTSUITE',
+                'clientVersion': '1.9',
+                'osName': 'Android',
+                'osVersion': '12',
+                'androidSdkVersion': 31,
+                'platform': 'MOBILE',
+                'userAgent': 'com.google.android.youtube/1.9 (Linux; U; Android 12; US) gzip'
+            },
+            # https://github.com/yt-dlp/yt-dlp/pull/575#issuecomment-887739287
+            #'thirdParty': {
+            #    'embedUrl': 'https://google.com',  # Can be any valid URL
+            #}
+        },
+        'INNERTUBE_CONTEXT_CLIENT_NAME': 3,
+        'REQUIRE_JS_PLAYER': False,
+    },
+
+    'ios': {
+        'INNERTUBE_API_KEY': 'AIzaSyB-63vPrdThhKuerbB2N_l7Kwwcxj6yUAc',
+        'INNERTUBE_CONTEXT': {
+            'client': {
+                'hl': 'en',
+                'gl': 'US',
+                'clientName': 'IOS',
+                'clientVersion': '19.09.3',
+                'deviceModel': 'iPhone14,3',
+                'userAgent': 'com.google.ios.youtube/19.09.3 (iPhone14,3; U; CPU iOS 15_6 like Mac OS X)'
+            }
+        },
+        'INNERTUBE_CONTEXT_CLIENT_NAME': 5,
+        'REQUIRE_JS_PLAYER': False
+    },
+
+    # This client can access age restricted videos (unless the uploader has disabled the 'allow embedding' option)
+    # See: https://github.com/zerodytrash/YouTube-Internal-Clients
+    'tv_embedded': {
+        'INNERTUBE_API_KEY': 'AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8',
+        'INNERTUBE_CONTEXT': {
+            'client': {
+                'hl': 'en',
+                'gl': 'US',
+                'clientName': 'TVHTML5_SIMPLY_EMBEDDED_PLAYER',
+                'clientVersion': '2.0',
+                'clientScreen': 'EMBED',
+            },
+            # https://github.com/yt-dlp/yt-dlp/pull/575#issuecomment-887739287
+            'thirdParty': {
+                'embedUrl': 'https://google.com',  # Can be any valid URL
+            }
+
+        },
+        'INNERTUBE_CONTEXT_CLIENT_NAME': 85,
+        'REQUIRE_JS_PLAYER': True,
+    },
+
+    'web': {
+        'INNERTUBE_API_KEY': 'AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8',
+        'INNERTUBE_CONTEXT': {
+            'client': {
+                'clientName': 'WEB',
+                'clientVersion': '2.20220801.00.00',
+                'userAgent': desktop_user_agent,
+            }
+        },
+        'INNERTUBE_CONTEXT_CLIENT_NAME': 1
+    },
+    'android_vr': {
+        'INNERTUBE_API_KEY': 'AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w',
+        'INNERTUBE_CONTEXT': {
+            'client': {
+                'clientName': 'ANDROID_VR',
+                'clientVersion': '1.60.19',
+                'deviceMake': 'Oculus',
+                'deviceModel': 'Quest 3',
+                'androidSdkVersion': 32,
+                'userAgent': 'com.google.android.apps.youtube.vr.oculus/1.60.19 (Linux; U; Android 12L; eureka-user Build/SQ3A.220605.009.A1) gzip',
+                'osName': 'Android',
+                'osVersion': '12L',
+            },
+        },
+        'INNERTUBE_CONTEXT_CLIENT_NAME': 28,
+        'REQUIRE_JS_PLAYER': False,
+    },
+}
+
+def get_visitor_data():
+    visitor_data = None
+    visitor_data_cache = os.path.join(settings.data_dir, 'visitorData.txt')
+    if not os.path.exists(settings.data_dir):
+        os.makedirs(settings.data_dir)
+    if os.path.isfile(visitor_data_cache):
+        with open(visitor_data_cache, 'r') as file:
+            print('Getting visitor_data from cache')
+            visitor_data = file.read()
+        max_age = 12*3600
+        file_age = time.time() - os.path.getmtime(visitor_data_cache)
+        if file_age > max_age:
+            print('visitor_data cache is too old. Removing file...')
+            os.remove(visitor_data_cache)
+        return visitor_data
+
+    print('Fetching youtube homepage to get visitor_data')
+    yt_homepage = 'https://www.youtube.com'
+    yt_resp = fetch_url(yt_homepage, headers={'User-Agent': mobile_user_agent}, report_text='Getting youtube homepage')
+    visitor_data_re = r'''"visitorData":\s*?"(.+?)"'''
+    visitor_data_match = re.search(visitor_data_re, yt_resp.decode())
+    if visitor_data_match:
+        visitor_data = visitor_data_match.group(1)
+        print(f'Got visitor_data: {len(visitor_data)}')
+        with open(visitor_data_cache, 'w') as file:
+            print('Saving visitor_data cache...')
+            file.write(visitor_data)
+        return visitor_data
+    else:
+        print('Unable to get visitor_data value')
+    return visitor_data
+
+def call_youtube_api(client, api, data):
+    client_params = INNERTUBE_CLIENTS[client]
+    context = client_params['INNERTUBE_CONTEXT']
+    key = client_params['INNERTUBE_API_KEY']
+    host = client_params.get('INNERTUBE_HOST') or 'www.youtube.com'
+    user_agent = context['client'].get('userAgent') or mobile_user_agent
+    visitor_data = get_visitor_data()
+
+    url = 'https://' + host + '/youtubei/v1/' + api + '?key=' + key
+    if visitor_data:
+        context['client'].update({'visitorData': visitor_data})
+    data['context'] = context
+
+    data = json.dumps(data)
+    headers = (('Content-Type', 'application/json'),('User-Agent', user_agent))
+    if visitor_data:
+        headers = ( *headers, ('X-Goog-Visitor-Id', visitor_data ))
+    response = fetch_url(
+        url, data=data, headers=headers,
+        debug_name='youtubei_' + api + '_' + client,
+        report_text='Fetched ' + client + ' youtubei ' + api
+    ).decode('utf-8')
+    return response
+
+
 def strip_non_ascii(string):
     ''' Returns the string without non ASCII characters'''
+    if string is None:
+        return ""
     stripped = (c for c in string if 0 < ord(c) < 127)
     return ''.join(stripped)
 

youtube/version.py

@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
 
-__version__ = '0.1.0'
+__version__ = 'v0.3.2'

youtube/watch.py

@@ -15,6 +15,10 @@ import traceback
 import urllib
 import re
 import urllib3.exceptions
+from urllib.parse import parse_qs, urlencode
+from types import SimpleNamespace
+from math import ceil
+
 
 try:
     with open(os.path.join(settings.data_dir, 'decrypt_function_cache.json'), 'r') as f:
@@ -23,26 +27,148 @@ except FileNotFoundError:
     decrypt_cache = {}
 
 
-def get_video_sources(info):
-    video_sources = []
-    max_resolution = settings.default_resolution
+def codec_name(vcodec):
+    if vcodec.startswith('avc'):
+        return 'h264'
+    elif vcodec.startswith('av01'):
+        return 'av1'
+    elif vcodec.startswith('vp'):
+        return 'vp'
+    else:
+        return 'unknown'
+
+
+def get_video_sources(info, target_resolution):
+    '''return dict with organized sources: {
+        'uni_sources': [{}, ...],   # video and audio in one file
+        'uni_idx': int,     # default unified source index
+        'pair_sources': [{video: {}, audio: {}, quality: ..., ...}, ...],
+        'pair_idx': int,    # default pair source index
+    }
+    '''
+    audio_sources = []
+    video_only_sources = {}
+    uni_sources = []
+    pair_sources = []
+
+
     for fmt in info['formats']:
-        if not all(fmt[attr] for attr in ('quality', 'width', 'ext', 'url')):
+        if not all(fmt[attr] for attr in ('ext', 'url', 'itag')):
             continue
-        if (fmt['acodec'] and fmt['vcodec']
-                and fmt['quality'] <= max_resolution):
-            video_sources.append({
-                'src': fmt['url'],
+
+        # unified source
+        if fmt['acodec'] and fmt['vcodec']:
+            source = {
                 'type': 'video/' + fmt['ext'],
-                'quality': fmt['quality'],
-                'height': fmt['height'],
-                'width': fmt['width'],
-            })
+                'quality_string': short_video_quality_string(fmt),
+            }
+            source['quality_string'] += ' (integrated)'
+            source.update(fmt)
+            uni_sources.append(source)
+            continue
 
-    # order the videos sources so the preferred resolution is first #
-    video_sources.sort(key=lambda source: source['quality'], reverse=True)
+        if not (fmt['init_range'] and fmt['index_range']):
+            continue
 
-    return video_sources
+        # audio source
+        if fmt['acodec'] and not fmt['vcodec'] and (
+                fmt['audio_bitrate'] or fmt['bitrate']):
+            if fmt['bitrate']:  # prefer this one, more accurate right now
+                fmt['audio_bitrate'] = int(fmt['bitrate']/1000)
+            source = {
+                'type': 'audio/' + fmt['ext'],
+                'quality_string': audio_quality_string(fmt),
+            }
+            source.update(fmt)
+            source['mime_codec'] = (source['type'] + '; codecs="'
+                                    + source['acodec'] + '"')
+            audio_sources.append(source)
+        # video-only source
+        elif all(fmt[attr] for attr in ('vcodec', 'quality', 'width', 'fps',
+                                        'file_size')):
+            if codec_name(fmt['vcodec']) == 'unknown':
+                continue
+            source = {
+                'type': 'video/' + fmt['ext'],
+                'quality_string': short_video_quality_string(fmt),
+            }
+            source.update(fmt)
+            source['mime_codec'] = (source['type'] + '; codecs="'
+                                    + source['vcodec'] + '"')
+            quality = str(fmt['quality']) + 'p' + str(fmt['fps'])
+            if quality in video_only_sources:
+                video_only_sources[quality].append(source)
+            else:
+                video_only_sources[quality] = [source]
+
+    audio_sources.sort(key=lambda source: source['audio_bitrate'])
+    uni_sources.sort(key=lambda src: src['quality'])
+
+    webm_audios = [a for a in audio_sources if a['ext'] == 'webm']
+    mp4_audios = [a for a in audio_sources if a['ext'] == 'mp4']
+
+    for quality_string, sources in video_only_sources.items():
+        # choose an audio source to go with it
+        # 0.5 is semiarbitrary empirical constant to spread audio sources
+        # between 144p and 1080p. Use something better eventually.
+        quality, fps = map(int, quality_string.split('p'))
+        target_audio_bitrate = quality*fps/30*0.5
+        pair_info = {
+            'quality_string': quality_string,
+            'quality': quality,
+            'height': sources[0]['height'],
+            'width': sources[0]['width'],
+            'fps': fps,
+            'videos': sources,
+            'audios': [],
+        }
+        for audio_choices in (webm_audios, mp4_audios):
+            if not audio_choices:
+                continue
+            closest_audio_source = audio_choices[0]
+            best_err = target_audio_bitrate - audio_choices[0]['audio_bitrate']
+            best_err = abs(best_err)
+            for audio_source in audio_choices[1:]:
+                err = abs(audio_source['audio_bitrate'] - target_audio_bitrate)
+                # once err gets worse we have passed the closest one
+                if err > best_err:
+                    break
+                best_err = err
+                closest_audio_source = audio_source
+            pair_info['audios'].append(closest_audio_source)
+
+        if not pair_info['audios']:
+            continue
+
+        def video_rank(src):
+            ''' Sort by settings preference. Use file size as tiebreaker '''
+            setting_name = 'codec_rank_' + codec_name(src['vcodec'])
+            return (settings.current_settings_dict[setting_name],
+                    src['file_size'])
+        pair_info['videos'].sort(key=video_rank)
+
+        pair_sources.append(pair_info)
+
+    pair_sources.sort(key=lambda src: src['quality'])
+
+    uni_idx = 0 if uni_sources else None
+    for i, source in enumerate(uni_sources):
+        if source['quality'] > target_resolution:
+            break
+        uni_idx = i
+
+    pair_idx = 0 if pair_sources else None
+    for i, pair_info in enumerate(pair_sources):
+        if pair_info['quality'] > target_resolution:
+            break
+        pair_idx = i
+
+    return {
+        'uni_sources': uni_sources,
+        'uni_idx': uni_idx,
+        'pair_sources': pair_sources,
+        'pair_idx': pair_idx,
+    }
 
 
 def make_caption_src(info, lang, auto=False, trans_lang=None):
@@ -52,7 +178,7 @@ def make_caption_src(info, lang, auto=False, trans_lang=None):
     if trans_lang:
         label += ' -> ' + trans_lang
     return {
-        'url': '/' + yt_data_extract.get_caption_url(info, lang, 'vtt', auto, trans_lang),
+        'url': util.prefix_url(yt_data_extract.get_caption_url(info, lang, 'vtt', auto, trans_lang)),
         'label': label,
         'srclang': trans_lang[0:2] if trans_lang else lang[0:2],
         'on': False,
@@ -96,6 +222,8 @@ def get_subtitle_sources(info):
     pref_lang (Automatic)
     pref_lang (Manual)'''
     sources = []
+    if not yt_data_extract.captions_available(info):
+        return []
     pref_lang = settings.subtitles_language
     native_video_lang = None
     if info['automatic_caption_languages']:
@@ -182,14 +310,6 @@ def save_decrypt_cache():
     f.close()
 
 
-watch_headers = (
-    ('Accept', '*/*'),
-    ('Accept-Language', 'en-US,en;q=0.5'),
-    ('X-YouTube-Client-Name', '2'),
-    ('X-YouTube-Client-Version', '2.20180830'),
-) + util.mobile_ua
-
-
 def decrypt_signatures(info, video_id):
     '''return error string, or False if no errors'''
     if not yt_data_extract.requires_decryption(info):
@@ -220,7 +340,13 @@ def _add_to_error(info, key, additional_message):
         info[key] = additional_message
 
 
-def extract_info(video_id, use_invidious, playlist_id=None, index=None):
+def fetch_player_response(client, video_id):
+    return util.call_youtube_api(client, 'player', {
+        'videoId': video_id,
+    })
+
+
+def fetch_watch_page_info(video_id, playlist_id, index):
     # bpctr=9999999999 will bypass are-you-sure dialogs for controversial
     # videos
     url = 'https://m.youtube.com/embed/' + video_id + '?bpctr=9999999999'
@@ -228,56 +354,55 @@ def extract_info(video_id, use_invidious, playlist_id=None, index=None):
         url += '&list=' + playlist_id
     if index:
         url += '&index=' + index
-    watch_page = util.fetch_url(url, headers=watch_headers,
+
+    headers = (
+        ('Accept', '*/*'),
+        ('Accept-Language', 'en-US,en;q=0.5'),
+        ('X-YouTube-Client-Name', '2'),
+        ('X-YouTube-Client-Version', '2.20180830'),
+    ) + util.mobile_ua
+
+    watch_page = util.fetch_url(url, headers=headers,
                                 debug_name='watch')
     watch_page = watch_page.decode('utf-8')
-    info = yt_data_extract.extract_watch_info_from_html(watch_page)
+    return yt_data_extract.extract_watch_info_from_html(watch_page)
 
-    # request player urls if it's missing
-    # see https://github.com/user234683/youtube-local/issues/22#issuecomment-706395160
-    if info['age_restricted'] or info['player_urls_missing']:
-        if info['age_restricted']:
-            print('Age restricted video. Fetching /youtubei/v1/player page')
-        else:
-            print('Missing player. Fetching /youtubei/v1/player page')
 
-        # https://github.com/yt-dlp/yt-dlp/issues/574#issuecomment-887171136
-        # ANDROID is used instead because its urls don't require decryption
-        # The URLs returned with WEB for videos requiring decryption
-        # couldn't be decrypted with the base.js from the web page for some
-        # reason
-        url ='https://youtubei.googleapis.com/youtubei/v1/player'
-        url += '?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8'
-        data = {
-            'videoId': video_id,
-            'context': {
-                'client': {
-                    'clientName': 'ANDROID',
-                    'clientVersion': '16.20',
-                    'clientScreen': 'EMBED',
-                    'gl': 'US',
-                    'hl': 'en',
-                },
-                # https://github.com/yt-dlp/yt-dlp/pull/575#issuecomment-887739287
-                'thirdParty': {
-                    'embedUrl': 'https://google.com',  # Can be any valid URL
-                }
-            }
-        }
-        data = json.dumps(data)
-        content_header = (('Content-Type', 'application/json'),)
-        player_response = util.fetch_url(
-            url, data=data, headers=util.mobile_ua + content_header,
-            debug_name='youtubei_player',
-            report_text='Fetched youtubei player page').decode('utf-8')
-        yt_data_extract.update_with_age_restricted_info(info,
-                                                            player_response)
+def extract_info(video_id, use_invidious, playlist_id=None, index=None):
+    primary_client = 'android_vr'
+    fallback_client = 'ios'
+    last_resort_client = 'tv_embedded'
+
+    tasks = (
+        # Get video metadata from here
+        gevent.spawn(fetch_watch_page_info, video_id, playlist_id, index),
+        gevent.spawn(fetch_player_response, primary_client, video_id)
+    )
+    gevent.joinall(tasks)
+    util.check_gevent_exceptions(*tasks)
+
+    info = tasks[0].value or {}
+    player_response = tasks[1].value or {}
+
+    yt_data_extract.update_with_new_urls(info, player_response)
+
+    # Fallback to 'ios' if no valid URLs are found
+    if not info.get('formats') or info.get('player_urls_missing'):
+        print(f"No URLs found in '{primary_client}', attempting with '{fallback_client}'.")
+        player_response = fetch_player_response(fallback_client, video_id) or {}
+        yt_data_extract.update_with_new_urls(info, player_response)
+
+    # Final attempt with 'tv_embedded' if there are still no URLs
+    if not info.get('formats') or info.get('player_urls_missing'):
+        print(f"No URLs found in '{fallback_client}', attempting with '{last_resort_client}'")
+        player_response = fetch_player_response(last_resort_client, video_id) or {}
+        yt_data_extract.update_with_new_urls(info, player_response)
 
     # signature decryption
-    decryption_error = decrypt_signatures(info, video_id)
-    if decryption_error:
-        decryption_error = 'Error decrypting url signatures: ' + decryption_error
-        info['playability_error'] = decryption_error
+    if info.get('formats'):
+        decryption_error = decrypt_signatures(info, video_id)
+        if decryption_error:
+            info['playability_error'] = 'Error decrypting url signatures: ' + decryption_error
 
     # check if urls ready (non-live format) in former livestream
     # urls not ready if all of them have no filesize
@@ -291,26 +416,26 @@ def extract_info(video_id, use_invidious, playlist_id=None, index=None):
 
     # livestream urls
     # sometimes only the livestream urls work soon after the livestream is over
-    if (info['hls_manifest_url']
-        and (info['live'] or not info['formats'] or not info['urls_ready'])
-    ):
-        manifest = util.fetch_url(
-            info['hls_manifest_url'],
-            debug_name='hls_manifest.m3u8',
-            report_text='Fetched hls manifest'
-        ).decode('utf-8')
-
-        info['hls_formats'], err = yt_data_extract.extract_hls_formats(manifest)
-        if not err:
-            info['playability_error'] = None
-        for fmt in info['hls_formats']:
-            fmt['video_quality'] = video_quality_string(fmt)
-    else:
-        info['hls_formats'] = []
+    info['hls_formats'] = []
+    if info.get('hls_manifest_url') and (info.get('live') or not info.get('formats') or not info['urls_ready']):
+        try:
+            manifest = util.fetch_url(info['hls_manifest_url'],
+                debug_name='hls_manifest.m3u8',
+                report_text='Fetched hls manifest'
+            ).decode('utf-8')
+            info['hls_formats'], err = yt_data_extract.extract_hls_formats(manifest)
+            if not err:
+                info['playability_error'] = None
+            for fmt in info['hls_formats']:
+                fmt['video_quality'] = video_quality_string(fmt)
+        except Exception as e:
+            print(f"Error obteniendo HLS manifest: {e}")
+            info['hls_formats'] = []
 
     # check for 403. Unnecessary for tor video routing b/c ip address is same
     info['invidious_used'] = False
     info['invidious_reload_button'] = False
+    info['tor_bypass_used'] = False
     if (settings.route_tor == 1
             and info['formats'] and info['formats'][0]['url']):
         try:
@@ -324,6 +449,7 @@ def extract_info(video_id, use_invidious, playlist_id=None, index=None):
         if response.status == 403:
             print('Access denied (403) for video urls.')
             print('Routing video through Tor')
+            info['tor_bypass_used'] = True
             for fmt in info['formats']:
                 fmt['url'] += '&use_tor=1'
         elif 300 <= response.status < 400:
@@ -343,15 +469,30 @@ def video_quality_string(format):
     return '?'
 
 
-def audio_quality_string(format):
-    if format['acodec']:
-        result = str(format['audio_bitrate'] or '?') + 'k'
-        if format['audio_sample_rate']:
-            result += ' ' + str(format['audio_sample_rate']) + ' Hz'
-        return result
-    elif format['vcodec']:
-        return 'video only'
+def short_video_quality_string(fmt):
+    result = str(fmt['quality'] or '?') + 'p'
+    if fmt['fps']:
+        result += str(fmt['fps'])
+    if fmt['vcodec'].startswith('av01'):
+        result += ' AV1'
+    elif fmt['vcodec'].startswith('avc'):
+        result += ' h264'
+    else:
+        result += ' ' + fmt['vcodec']
+    return result
 
+
+def audio_quality_string(fmt):
+    if fmt['acodec']:
+        if fmt['audio_bitrate']:
+            result = '%d' % fmt['audio_bitrate'] + 'k'
+        else:
+            result = '?k'
+        if fmt['audio_sample_rate']:
+            result += ' ' + '%.3G' % (fmt['audio_sample_rate']/1000) + 'kHz'
+        return result
+    elif fmt['vcodec']:
+        return 'video only'
     return '?'
 
 
@@ -370,12 +511,71 @@ def format_bytes(bytes):
     return '%.2f%s' % (converted, suffix)
 
 
+@yt_app.route('/ytl-api/storyboard.vtt')
+def get_storyboard_vtt():
+    """
+    See:
+        https://github.com/iv-org/invidious/blob/9a8b81fcbe49ff8d88f197b7f731d6bf79fc8087/src/invidious.cr#L3603
+        https://github.com/iv-org/invidious/blob/3bb7fbb2f119790ee6675076b31cd990f75f64bb/src/invidious/videos.cr#L623
+    """
+
+    spec_url = request.args.get('spec_url')
+    url, *boards = spec_url.split('|')
+    base_url, q = url.split('?')
+    q = parse_qs(q)  # for url query
+
+    storyboard = None
+    wanted_height = 90
+
+    for i, board in enumerate(boards):
+        *t, _, sigh = board.split("#")
+        width, height, count, width_cnt, height_cnt, interval = map(int, t)
+        if height != wanted_height: continue
+        q['sigh'] = [sigh]
+        url = f"{base_url}?{urlencode(q, doseq=True)}"
+        storyboard = SimpleNamespace(
+            url               = url.replace("$L", str(i)).replace("$N", "M$M"),
+            width             = width,
+            height            = height,
+            interval          = interval,
+            width_cnt         = width_cnt,
+            height_cnt        = height_cnt,
+            storyboard_count  = ceil(count / (width_cnt * height_cnt))
+        )
+
+    if not storyboard:
+        flask.abort(404)
+
+    def to_ts(ms):
+        s, ms = divmod(ms, 1000)
+        h, s = divmod(s, 3600)
+        m, s = divmod(s, 60)
+        return f"{h:02}:{m:02}:{s:02}.{ms:03}"
+
+    r = "WEBVTT"  # result
+    ts = 0  # current timestamp
+
+    for i in range(storyboard.storyboard_count):
+        url = '/' + storyboard.url.replace("$M", str(i))
+        interval = storyboard.interval
+        w, h = storyboard.width, storyboard.height
+        w_cnt, h_cnt = storyboard.width_cnt, storyboard.height_cnt
+
+        for j in range(h_cnt):
+            for k in range(w_cnt):
+                r += f"{to_ts(ts)} --> {to_ts(ts+interval)}\n"
+                r += f"{url}#xywh={w * k},{h * j},{w},{h}\n\n"
+                ts += interval
+
+    return flask.Response(r, mimetype='text/vtt')
+
+
 time_table = {'h': 3600, 'm': 60, 's': 1}
-
-
 @yt_app.route('/watch')
 @yt_app.route('/embed')
 @yt_app.route('/embed/<video_id>')
+@yt_app.route('/shorts')
+@yt_app.route('/shorts/<video_id>')
 def get_watch_page(video_id=None):
     video_id = request.args.get('v') or video_id
     if not video_id:
@@ -425,8 +625,11 @@ def get_watch_page(video_id=None):
 
     # prefix urls, and other post-processing not handled by yt_data_extract
     for item in info['related_videos']:
+        item['thumbnail'] = "https://i.ytimg.com/vi/{}/hqdefault.jpg".format(item['id']) # set HQ relateds thumbnail videos
         util.prefix_urls(item)
         util.add_extra_html_info(item)
+    for song in info['music_list']:
+        song['url'] = util.prefix_url(song['url'])
     if info['playlist']:
         playlist_id = info['playlist']['id']
         for item in info['playlist']['items']:
@@ -438,10 +641,11 @@ def get_watch_page(video_id=None):
                 item['url'] += '&index=' + str(item['index'])
         info['playlist']['author_url'] = util.prefix_url(
             info['playlist']['author_url'])
-    # Don't prefix hls_formats for now because the urls inside the manifest
-    # would need to be prefixed as well.
-    for fmt in info['formats']:
-        fmt['url'] = util.prefix_url(fmt['url'])
+    if settings.img_prefix:
+        # Don't prefix hls_formats for now because the urls inside the manifest
+        # would need to be prefixed as well.
+        for fmt in info['formats']:
+            fmt['url'] = util.prefix_url(fmt['url'])
 
     # Add video title to end of url path so it has a filename other than just
     # "videoplayback" when downloaded
@@ -455,12 +659,6 @@ def get_watch_page(video_id=None):
             '/videoplayback',
             '/videoplayback/name/' + filename)
 
-    if settings.gather_googlevideo_domains:
-        with open(os.path.join(settings.data_dir, 'googlevideo-domains.txt'), 'a+', encoding='utf-8') as f:
-            url = info['formats'][0]['url']
-            subdomain = url[0:url.find(".googlevideo.com")]
-            f.write(subdomain + "\n")
-
     download_formats = []
 
     for format in (info['formats'] + info['hls_formats']):
@@ -477,9 +675,53 @@ def get_watch_page(video_id=None):
             'codecs': codecs_string,
         })
 
-    video_sources = get_video_sources(info)
-    video_height = yt_data_extract.deep_get(video_sources, 0, 'height', default=360)
-    video_width = yt_data_extract.deep_get(video_sources, 0, 'width', default=640)
+    if (settings.route_tor == 2) or info['tor_bypass_used']:
+        target_resolution = 240
+    else:
+        target_resolution = settings.default_resolution
+
+    source_info = get_video_sources(info, target_resolution)
+    uni_sources = source_info['uni_sources']
+    pair_sources = source_info['pair_sources']
+    uni_idx, pair_idx = source_info['uni_idx'], source_info['pair_idx']
+
+    pair_quality = yt_data_extract.deep_get(pair_sources, pair_idx, 'quality')
+    uni_quality = yt_data_extract.deep_get(uni_sources, uni_idx, 'quality')
+
+    pair_error = abs((pair_quality or 360) - target_resolution)
+    uni_error = abs((uni_quality or 360) - target_resolution)
+    if uni_error == pair_error:
+        # use settings.prefer_uni_sources as a tiebreaker
+        closer_to_target = 'uni' if settings.prefer_uni_sources else 'pair'
+    elif uni_error < pair_error:
+        closer_to_target = 'uni'
+    else:
+        closer_to_target = 'pair'
+
+    if settings.prefer_uni_sources == 2:
+        # Use uni sources unless there's no choice.
+        using_pair_sources = (
+            bool(pair_sources) and (not uni_sources)
+        )
+    else:
+        # Use the pair sources if they're closer to the desired resolution
+        using_pair_sources = (
+            bool(pair_sources)
+            and (not uni_sources or closer_to_target == 'pair')
+        )
+    if using_pair_sources:
+        video_height = pair_sources[pair_idx]['height']
+        video_width = pair_sources[pair_idx]['width']
+    else:
+        video_height = yt_data_extract.deep_get(
+            uni_sources, uni_idx, 'height', default=360
+        )
+        video_width = yt_data_extract.deep_get(
+            uni_sources, uni_idx, 'width', default=640
+        )
+
+
+
     # 1 second per pixel, or the actual video width
     theater_video_target_width = max(640, info['duration'] or 0, video_width)
 
@@ -512,19 +754,16 @@ def get_watch_page(video_id=None):
         template_name = 'embed.html'
     else:
         template_name = 'watch.html'
-    return flask.render_template(
-        template_name,
-        header_playlist_names = local_playlist.get_playlist_names(),
-        uploader_channel_url = ('/' + info['author_url']) if info['author_url'] else '',
-        time_published = info['time_published'],
-        time_published_utc=time_utc_isoformat(info['time_published']),
+    return flask.render_template(template_name,
+        header_playlist_names   = local_playlist.get_playlist_names(),
+        uploader_channel_url    = ('/' + info['author_url']) if info['author_url'] else '',
+        time_published             = info['time_published'],
         view_count    = (lambda x: '{:,}'.format(x) if x is not None else "")(info.get("view_count", None)),
         like_count    = (lambda x: '{:,}'.format(x) if x is not None else "")(info.get("like_count", None)),
         dislike_count = (lambda x: '{:,}'.format(x) if x is not None else "")(info.get("dislike_count", None)),
         download_formats        = download_formats,
         other_downloads         = other_downloads,
         video_info              = json.dumps(video_info),
-        video_sources           = video_sources,
         hls_formats             = info['hls_formats'],
         subtitle_sources        = subtitle_sources,
         related                 = info['related_videos'],
@@ -554,15 +793,25 @@ def get_watch_page(video_id=None):
         invidious_reload_button = info['invidious_reload_button'],
         video_url = util.URL_ORIGIN + '/watch?v=' + video_id,
         video_id = video_id,
-        time_start = time_start,
+        storyboard_url = (util.URL_ORIGIN + '/ytl-api/storyboard.vtt?' +
+            urlencode([('spec_url', info['storyboard_spec_url'])])
+            if info['storyboard_spec_url'] else None),
 
         js_data = {
-            'video_id': video_info['id'],
+            'video_id': info['id'],
+            'video_duration': info['duration'],
             'settings': settings.current_settings_dict,
             'has_manual_captions': any(s.get('on') for s in subtitle_sources),
+            **source_info,
+            'using_pair_sources': using_pair_sources,
+            'time_start': time_start,
+            'playlist': info['playlist'],
+            'related': info['related_videos'],
+            'playability_error': info['playability_error'],
         },
-        # for embed page
-        font_family=youtube.font_choices[settings.font],
+        font_family = youtube.font_choices[settings.font], # for embed page
+        **source_info,
+        using_pair_sources = using_pair_sources,
     )
 
 

youtube/yt_data_extract/__init__.py

@@ -7,7 +7,7 @@ from .everything_else import (extract_channel_info, extract_search_info,
     extract_playlist_metadata, extract_playlist_info, extract_comments_info)
 
 from .watch_extraction import (extract_watch_info, get_caption_url,
-    update_with_age_restricted_info, requires_decryption,
+    update_with_new_urls, requires_decryption,
     extract_decryption_function, decrypt_signatures, _formats,
     update_format_with_type_info, extract_hls_formats,
-    extract_watch_info_from_html)
+    extract_watch_info_from_html, captions_available)

youtube/yt_data_extract/common.py

@@ -109,7 +109,7 @@ def concat_or_none(*strings):
 def remove_redirect(url):
     if url is None:
         return None
-    if re.fullmatch(r'(((https?:)?//)?(www.)?youtube.com)?/redirect\?.*', url) is not None: # youtube puts these on external links to do tracking
+    if re.fullmatch(r'(((https?:)?//)?(www.)?youtube.com)?/redirect\?.*', url) is not None: # YouTube puts these on external links to do tracking
         query_string = url[url.find('?')+1: ]
         return urllib.parse.parse_qs(query_string)['q'][0]
     return url
@@ -133,11 +133,11 @@ def _recover_urls(runs):
     for run in runs:
         url = deep_get(run, 'navigationEndpoint', 'urlEndpoint', 'url')
         text = run.get('text', '')
-        # second condition is necessary because youtube makes other things into urls, such as hashtags, which we want to keep as text
+        # second condition is necessary because YouTube makes other things into urls, such as hashtags, which we want to keep as text
         if url is not None and (text.startswith('http://') or text.startswith('https://')):
             url = remove_redirect(url)
             run['url'] = url
-            run['text'] = url # youtube truncates the url text, use actual url instead
+            run['text'] = url # YouTube truncates the url text, use actual url instead
 
 def extract_str(node, default=None, recover_urls=False):
     '''default is the value returned if the extraction fails. If recover_urls is true, will attempt to fix YouTube's truncation of url text (most prominently seen in descriptions)'''
@@ -166,14 +166,17 @@ def extract_formatted_text(node):
         return [{'text': node['simpleText']}]
     return []
 
-def extract_int(string, default=None):
+def extract_int(string, default=None, whole_word=True):
     if isinstance(string, int):
         return string
     if not isinstance(string, str):
         string = extract_str(string)
     if not string:
         return default
-    match = re.search(r'\b(\d+)\b', string.replace(',', ''))
+    if whole_word:
+        match = re.search(r'\b(\d+)\b', string.replace(',', ''))
+    else:
+        match = re.search(r'(\d+)', string.replace(',', ''))
     if match is None:
         return default
     try:
@@ -182,7 +185,7 @@ def extract_int(string, default=None):
         return default
 
 def extract_approx_int(string):
-    '''e.g. "15.1M" from "15.1M subscribers"'''
+    '''e.g. "15.1M" from "15.1M subscribers" or '4,353' from 4353'''
     if not isinstance(string, str):
         string = extract_str(string)
     if not string:
@@ -190,7 +193,10 @@ def extract_approx_int(string):
     match = re.search(r'\b(\d+(?:\.\d+)?[KMBTkmbt]?)\b', string.replace(',', ''))
     if match is None:
         return None
-    return match.group(1)
+    result = match.group(1)
+    if re.fullmatch(r'\d+', result):
+        result = '{:,}'.format(int(result))
+    return result
 
 MONTH_ABBREVIATIONS = {'jan':'1', 'feb':'2', 'mar':'3', 'apr':'4', 'may':'5', 'jun':'6', 'jul':'7', 'aug':'8', 'sep':'9', 'oct':'10', 'nov':'11', 'dec':'12'}
 def extract_date(date_text):
@@ -246,6 +252,9 @@ def extract_item_info(item, additional_info={}):
     primary_type = type_parts[-2]
     if primary_type == 'video':
         info['type'] = 'video'
+    elif type_parts[0] == 'reel': # shorts
+        info['type'] = 'video'
+        primary_type = 'video'
     elif primary_type in ('playlist', 'radio', 'show'):
         info['type'] = 'playlist'
         info['playlist_type'] = primary_type
@@ -292,7 +301,11 @@ def extract_item_info(item, additional_info={}):
             info['time_published'] = timestamp.group(1)
 
     if primary_type == 'video':
-        info['id'] = item.get('videoId')
+        info['id'] = multi_deep_get(item,
+            ['videoId'],
+            ['navigationEndpoint', 'watchEndpoint', 'videoId'],
+            ['navigationEndpoint', 'reelWatchEndpoint', 'videoId'] # shorts
+        )
         info['view_count'] = extract_int(item.get('viewCountText'))
 
         # dig into accessibility data to get view_count for videos marked as recommended, and to get time_published
@@ -310,17 +323,35 @@ def extract_item_info(item, additional_info={}):
         if info['view_count']:
             info['approx_view_count'] = '{:,}'.format(info['view_count'])
         else:
-            info['approx_view_count'] = extract_approx_int(item.get('shortViewCountText'))
+            info['approx_view_count'] = extract_approx_int(multi_get(item,
+                'shortViewCountText',
+                'viewCountText' # shorts
+            ))
 
         # handle case where it is "No views"
         if not info['approx_view_count']:
             if ('No views' in item.get('shortViewCountText', '')
-                    or 'no views' in accessibility_label.lower()):
+                    or 'no views' in accessibility_label.lower()
+                    or 'No views' in extract_str(item.get('viewCountText', '')) # shorts
+            ):
                 info['view_count'] = 0
                 info['approx_view_count'] = '0'
 
         info['duration'] = extract_str(item.get('lengthText'))
 
+        # dig into accessibility data to get duration for shorts
+        accessibility_label = deep_get(item,
+            'accessibility', 'accessibilityData', 'label',
+            default='')
+        duration = re.search(r'(\d+) (second|seconds|minute) - play video$',
+                            accessibility_label)
+        if duration:
+            if duration.group(2) == 'minute':
+                conservative_update(info, 'duration', '1:00')
+            else:
+                conservative_update(info,
+                    'duration', '0:' + duration.group(1).zfill(2))
+
         # if it's an item in a playlist, get its index
         if 'index' in item: # url has wrong index on playlist page
             info['index'] = extract_int(item.get('index'))
@@ -392,6 +423,8 @@ _item_types = {
     'gridVideoRenderer',
     'playlistVideoRenderer',
 
+    'reelItemRenderer',
+
     'playlistRenderer',
     'compactPlaylistRenderer',
     'gridPlaylistRenderer',
@@ -539,9 +572,13 @@ def extract_items(response, item_types=_item_types,
                     item_types=item_types)
                 if items:
                     break
-    elif 'onResponseReceivedEndpoints' in response:
-        for endpoint in response.get('onResponseReceivedEndpoints', []):
-            items, ctoken = extract_items_from_renderer_list(
+    if ('onResponseReceivedEndpoints' in response
+          or 'onResponseReceivedActions' in response):
+        for endpoint in multi_get(response,
+                                  'onResponseReceivedEndpoints',
+                                  'onResponseReceivedActions',
+                                  []):
+            new_items, new_ctoken = extract_items_from_renderer_list(
                 multi_deep_get(
                     endpoint,
                     ['reloadContinuationItemsCommand', 'continuationItems'],
@@ -550,13 +587,17 @@ def extract_items(response, item_types=_item_types,
                 ),
                 item_types=item_types,
             )
-            if items:
-                break
-    elif 'contents' in response:
+            items += new_items
+            if (not ctoken) or (new_ctoken and new_items):
+                ctoken = new_ctoken
+    if 'contents' in response:
         renderer = get(response, 'contents', {})
-        items, ctoken = extract_items_from_renderer(
+        new_items, new_ctoken = extract_items_from_renderer(
             renderer,
             item_types=item_types)
+        items += new_items
+        if (not ctoken) or (new_ctoken and new_items):
+            ctoken = new_ctoken
 
     if search_engagement_panels and 'engagementPanels' in response:
         new_items, new_ctoken = extract_items_from_renderer_list(

youtube/yt_data_extract/everything_else.py

@@ -9,7 +9,7 @@ import re
 import urllib
 from math import ceil
 
-def extract_channel_info(polymer_json, tab):
+def extract_channel_info(polymer_json, tab, continuation=False):
     response, err = extract_response(polymer_json)
     if err:
         return {'error': err}
@@ -23,7 +23,8 @@ def extract_channel_info(polymer_json, tab):
 
     # channel doesn't exist or was terminated
     # example terminated channel: https://www.youtube.com/channel/UCnKJeK_r90jDdIuzHXC0Org
-    if not metadata:
+    # metadata and microformat are not present for continuation requests
+    if not metadata and not continuation:
         if response.get('alerts'):
             error_string = ' '.join(
                 extract_str(deep_get(alert, 'alertRenderer', 'text'), default='')
@@ -44,7 +45,7 @@ def extract_channel_info(polymer_json, tab):
     info['approx_subscriber_count'] = extract_approx_int(deep_get(response,
         'header', 'c4TabbedHeaderRenderer', 'subscriberCountText'))
 
-    # stuff from microformat (info given by youtube for every page on channel)
+    # stuff from microformat (info given by youtube for first page on channel)
     info['short_description'] = metadata.get('description')
     if info['short_description'] and len(info['short_description']) > 730:
         info['short_description'] = info['short_description'][0:730] + '...'
@@ -69,34 +70,99 @@ def extract_channel_info(polymer_json, tab):
     info['ctoken'] = None
 
     # empty channel
-    if 'contents' not in response and 'continuationContents' not in response:
-        return info
+    #if 'contents' not in response and 'continuationContents' not in response:
+    #    return info
 
-    if tab in ('videos', 'playlists', 'search'):
+    if tab in ('videos', 'shorts', 'streams', 'playlists', 'search'):
         items, ctoken = extract_items(response)
-        additional_info = {'author': info['channel_name'], 'author_url': info['channel_url']}
+        additional_info = {
+            'author': info['channel_name'],
+            'author_id': info['channel_id'],
+            'author_url': info['channel_url'],
+        }
         info['items'] = [extract_item_info(renderer, additional_info) for renderer in items]
         info['ctoken'] = ctoken
         if tab in ('search', 'playlists'):
             info['is_last_page'] = (ctoken is None)
     elif tab == 'about':
-        items, _ = extract_items(response, item_types={'channelAboutFullMetadataRenderer'})
-        if not items:
-            info['error'] = 'Could not find channelAboutFullMetadataRenderer'
-            return info
-        channel_metadata = items[0]['channelAboutFullMetadataRenderer']
+        # Latest type
+        items, _ = extract_items(response, item_types={'aboutChannelRenderer'})
+        if items:
+            a_metadata = deep_get(items, 0, 'aboutChannelRenderer',
+                'metadata', 'aboutChannelViewModel')
+            if not a_metadata:
+                info['error'] = 'Could not find aboutChannelViewModel'
+                return info
 
-        info['links'] = []
-        for link_json in channel_metadata.get('primaryLinks', ()):
-            url = remove_redirect(deep_get(link_json, 'navigationEndpoint', 'urlEndpoint', 'url'))
-            if not (url.startswith('http://') or url.startswith('https://')):
-                url = 'http://' + url
-            text = extract_str(link_json.get('title'))
-            info['links'].append( (text, url) )
+            info['links'] = []
+            for link_outer in a_metadata.get('links', ()):
+                link = link_outer.get('channelExternalLinkViewModel') or {}
+                link_content = extract_str(deep_get(link, 'link', 'content'))
+                for run in deep_get(link, 'link', 'commandRuns') or ():
+                    url = remove_redirect(deep_get(run, 'onTap',
+                        'innertubeCommand', 'urlEndpoint', 'url'))
+                    if url and not (url.startswith('http://')
+                            or url.startswith('https://')):
+                        url = 'https://' + url
+                    if link_content is None or (link_content in url):
+                        break
+                else: # didn't break
+                    url = link_content
+                    if url and not (url.startswith('http://')
+                            or url.startswith('https://')):
+                        url = 'https://' + url
+                text = extract_str(deep_get(link, 'title', 'content'))
+                info['links'].append( (text, url) )
 
-        info['date_joined'] = extract_date(channel_metadata.get('joinedDateText'))
-        info['view_count'] = extract_int(channel_metadata.get('viewCountText'))
-        info['description'] = extract_str(channel_metadata.get('description'), default='')
+            info['date_joined'] = extract_date(
+                a_metadata.get('joinedDateText')
+            )
+            info['view_count'] = extract_int(a_metadata.get('viewCountText'))
+            info['approx_view_count'] = extract_approx_int(
+                a_metadata.get('viewCountText')
+            )
+            info['description'] = extract_str(
+                a_metadata.get('description'), default=''
+            )
+            info['approx_video_count'] = extract_approx_int(
+                a_metadata.get('videoCountText')
+            )
+            info['approx_subscriber_count'] = extract_approx_int(
+                a_metadata.get('subscriberCountText')
+            )
+            info['country'] = extract_str(a_metadata.get('country'))
+            info['canonical_url'] = extract_str(
+                a_metadata.get('canonicalChannelUrl')
+            )
+
+        # Old type
+        else:
+            items, _ = extract_items(response,
+                item_types={'channelAboutFullMetadataRenderer'})
+            if not items:
+                info['error'] = 'Could not find aboutChannelRenderer or channelAboutFullMetadataRenderer'
+                return info
+            a_metadata = items[0]['channelAboutFullMetadataRenderer']
+
+            info['links'] = []
+            for link_json in a_metadata.get('primaryLinks', ()):
+                url = remove_redirect(deep_get(link_json, 'navigationEndpoint',
+                    'urlEndpoint', 'url'))
+                if url and not (url.startswith('http://')
+                                or url.startswith('https://')):
+                    url = 'https://' + url
+                text = extract_str(link_json.get('title'))
+                info['links'].append( (text, url) )
+
+            info['date_joined'] = extract_date(a_metadata.get('joinedDateText'))
+            info['view_count'] = extract_int(a_metadata.get('viewCountText'))
+            info['description'] = extract_str(a_metadata.get(
+                'description'), default='')
+
+            info['approx_video_count'] = None
+            info['approx_subscriber_count'] = None
+            info['country'] = None
+            info['canonical_url'] = None
     else:
         raise NotImplementedError('Unknown or unsupported channel tab: ' + tab)
 
@@ -163,7 +229,7 @@ def extract_playlist_metadata(polymer_json):
     if metadata['first_video_id'] is None:
         metadata['thumbnail'] = None
     else:
-        metadata['thumbnail'] = 'https://i.ytimg.com/vi/' + metadata['first_video_id'] + '/mqdefault.jpg'
+        metadata['thumbnail'] = f"https://i.ytimg.com/vi/{metadata['first_video_id']}/hqdefault.jpg"
 
     metadata['video_count'] = extract_int(header.get('numVideosText'))
     metadata['description'] = extract_str(header.get('descriptionText'), default='')
@@ -186,6 +252,19 @@ def extract_playlist_metadata(polymer_json):
         elif 'updated' in text:
             metadata['time_published'] = extract_date(text)
 
+    microformat = deep_get(response, 'microformat', 'microformatDataRenderer',
+                           default={})
+    conservative_update(
+        metadata, 'title', extract_str(microformat.get('title'))
+    )
+    conservative_update(
+        metadata, 'description', extract_str(microformat.get('description'))
+    )
+    conservative_update(
+        metadata, 'thumbnail', deep_get(microformat, 'thumbnail',
+                                        'thumbnails', -1, 'url')
+    )
+
     return metadata
 
 def extract_playlist_info(polymer_json):
@@ -193,13 +272,11 @@ def extract_playlist_info(polymer_json):
     if err:
         return {'error': err}
     info = {'error': None}
-    first_page = 'continuationContents' not in response
     video_list, _ = extract_items(response)
 
     info['items'] = [extract_item_info(renderer) for renderer in video_list]
 
-    if first_page:
-        info['metadata'] = extract_playlist_metadata(polymer_json)
+    info['metadata'] = extract_playlist_metadata(polymer_json)
 
     return info
 

youtube/yt_data_extract/watch_extraction.py

@@ -133,32 +133,59 @@ def _extract_from_video_information_renderer(renderer_content):
     return info
 
 def _extract_likes_dislikes(renderer_content):
-    info = {
-        'like_count': None,
-        'dislike_count': None,
-    }
-    for button in renderer_content.get('buttons', ()):
-        button_renderer = button.get('slimMetadataToggleButtonRenderer', {})
-
+    def extract_button_count(toggle_button_renderer):
         # all the digits can be found in the accessibility data
-        count = extract_int(deep_get(
-                    button_renderer,
-                    'button', 'toggleButtonRenderer', 'defaultText',
-                    'accessibility', 'accessibilityData', 'label'))
+        count = extract_int(multi_deep_get(
+            toggle_button_renderer,
+            ['defaultText', 'accessibility', 'accessibilityData', 'label'],
+            ['accessibility', 'label'],
+            ['accessibilityData', 'accessibilityData', 'label'],
+            ['accessibilityText'],
+        ))
 
         # this count doesn't have all the digits, it's like 53K for instance
-        dumb_count = extract_int(extract_str(deep_get(
-            button_renderer, 'button', 'toggleButtonRenderer', 'defaultText')))
+        dumb_count = extract_int(extract_str(multi_get(
+            toggle_button_renderer, ['defaultText', 'title'])))
 
         # The accessibility text will be "No likes" or "No dislikes" or
         # something like that, but dumb count will be 0
         if dumb_count == 0:
             count = 0
+        return count
 
-        if 'isLike' in button_renderer:
-            info['like_count'] = count
-        elif 'isDislike' in button_renderer:
-            info['dislike_count'] = count
+    info = {
+        'like_count': None,
+        'dislike_count': None,
+    }
+    for button in renderer_content.get('buttons', ()):
+        if 'slimMetadataToggleButtonRenderer' in button:
+            button_renderer = button['slimMetadataToggleButtonRenderer']
+            count = extract_button_count(deep_get(button_renderer,
+                                                  'button',
+                                                  'toggleButtonRenderer'))
+            if 'isLike' in button_renderer:
+                info['like_count'] = count
+            elif 'isDislike' in button_renderer:
+                info['dislike_count'] = count
+        elif 'slimMetadataButtonRenderer' in button:
+            button_renderer = button['slimMetadataButtonRenderer']
+            liberal_update(info, 'like_count', extract_button_count(
+                multi_deep_get(button_renderer,
+                    ['button', 'segmentedLikeDislikeButtonRenderer',
+                     'likeButton', 'toggleButtonRenderer'],
+                    ['button', 'segmentedLikeDislikeButtonViewModel',
+                     'likeButtonViewModel', 'likeButtonViewModel',
+                     'toggleButtonViewModel', 'toggleButtonViewModel',
+                     'defaultButtonViewModel', 'buttonViewModel']
+                )
+            ))
+            '''liberal_update(info, 'dislike_count', extract_button_count(
+                deep_get(
+                    button_renderer, 'button',
+                    'segmentedLikeDislikeButtonRenderer',
+                    'dislikeButton', 'toggleButtonRenderer'
+                )
+            ))'''
     return info
 
 def _extract_from_owner_renderer(renderer_content):
@@ -212,6 +239,36 @@ def _extract_metadata_row_info(renderer_content):
 
     return info
 
+def _extract_from_music_renderer(renderer_content):
+    # latest format for the music list
+    info = {
+        'music_list': [],
+    }
+
+    for carousel in renderer_content.get('carouselLockups', []):
+        song = {}
+        carousel = carousel.get('carouselLockupRenderer', {})
+        video_renderer = carousel.get('videoLockup', {})
+        video_renderer_info = extract_item_info(video_renderer)
+        video_id = video_renderer_info.get('id')
+        song['url'] = concat_or_none('https://www.youtube.com/watch?v=',
+                                     video_id)
+        song['title'] = video_renderer_info.get('title')
+        for row in carousel.get('infoRows', []):
+            row = row.get('infoRowRenderer', {})
+            title = extract_str(row.get('title'))
+            data = extract_str(row.get('defaultMetadata'))
+            if title == 'SONG':
+                song['title'] = data
+            elif title == 'ARTIST':
+                song['artist'] = data
+            elif title == 'ALBUM':
+                song['album'] = data
+            elif title == 'WRITERS':
+                song['writers'] = data
+        info['music_list'].append(song)
+    return info
+
 def _extract_from_video_metadata(renderer_content):
     info = _extract_from_video_information_renderer(renderer_content)
     liberal_dict_update(info, _extract_likes_dislikes(renderer_content))
@@ -235,6 +292,7 @@ visible_extraction_dispatch = {
     'slimVideoActionBarRenderer': _extract_likes_dislikes,
     'slimOwnerRenderer': _extract_from_owner_renderer,
     'videoDescriptionHeaderRenderer': _extract_from_video_header_renderer,
+    'videoDescriptionMusicSectionRenderer': _extract_from_music_renderer,
     'expandableVideoDescriptionRenderer': _extract_from_description_renderer,
     'metadataRowContainerRenderer': _extract_metadata_row_info,
     # OR just this one, which contains SOME of the above inside it
@@ -307,17 +365,18 @@ def _extract_watch_info_mobile(top_level):
         # https://www.androidpolice.com/2019/10/31/google-youtube-app-comment-section-below-videos/
         # https://www.youtube.com/watch?v=bR5Q-wD-6qo
         if header_type == 'commentsEntryPointHeaderRenderer':
-            comment_count_text = extract_str(comment_info.get('headerText'))
+            comment_count_text = extract_str(multi_get(
+                comment_info, 'commentCount', 'headerText'))
         else:
             comment_count_text = extract_str(deep_get(comment_info,
                 'header', 'commentSectionHeaderRenderer', 'countText'))
         if comment_count_text == 'Comments':    # just this with no number, means 0 comments
-            info['comment_count'] = 0
+            info['comment_count'] = '0'
         else:
-            info['comment_count'] = extract_int(comment_count_text)
+            info['comment_count'] = extract_approx_int(comment_count_text)
         info['comments_disabled'] = False
     else:   # no comment section present means comments are disabled
-        info['comment_count'] = 0
+        info['comment_count'] = '0'
         info['comments_disabled'] = True
 
     # check for limited state
@@ -369,26 +428,28 @@ def _extract_watch_info_desktop(top_level):
     return info
 
 def update_format_with_codec_info(fmt, codec):
-    if (codec.startswith('av')
-            or codec in ('vp9', 'vp8', 'vp8.0', 'h263', 'h264', 'mp4v')):
+    if any(codec.startswith(c) for c in ('av', 'vp', 'h263', 'h264', 'mp4v')):
         if codec == 'vp8.0':
             codec = 'vp8'
         conservative_update(fmt, 'vcodec', codec)
     elif (codec.startswith('mp4a')
-            or codec in ('opus', 'mp3', 'aac', 'dtse', 'ec-3', 'vorbis')):
+            or codec in ('opus', 'mp3', 'aac', 'dtse', 'ec-3', 'vorbis',
+                        'ac-3')):
         conservative_update(fmt, 'acodec', codec)
     else:
         print('Warning: unrecognized codec: ' + codec)
 
 fmt_type_re = re.compile(
-    r'(text|audio|video)/([\w0-9]+); codecs="([\w0-9\.]+(?:, [\w0-9\.]+)*)"')
+    r'(text|audio|video)/([\w0-9]+); codecs="([^"]+)"')
 def update_format_with_type_info(fmt, yt_fmt):
     # 'type' for invidious api format
     mime_type = multi_get(yt_fmt, 'mimeType', 'type')
     if mime_type is None:
         return
     match = re.fullmatch(fmt_type_re, mime_type)
-
+    if match is None:
+        print('Warning: Could not read mimetype', mime_type)
+        return
     type, fmt['ext'], codecs = match.groups()
     codecs = codecs.split(', ')
     for codec in codecs:
@@ -411,17 +472,32 @@ def _extract_formats(info, player_response):
     for yt_fmt in yt_formats:
         itag = yt_fmt.get('itag')
 
+        # Translated audio track
+        # Example: https://www.youtube.com/watch?v=gF9kkB0UWYQ
+        # Only get the original language for now so a foreign
+        # translation will not be picked just because it comes first
+        if deep_get(yt_fmt, 'audioTrack', 'audioIsDefault') is False:
+            continue
+
         fmt = {}
         fmt['itag'] = itag
         fmt['ext'] = None
         fmt['audio_bitrate'] = None
+        fmt['bitrate'] = yt_fmt.get('bitrate')
         fmt['acodec'] = None
         fmt['vcodec'] = None
         fmt['width'] = yt_fmt.get('width')
         fmt['height'] = yt_fmt.get('height')
-        fmt['file_size'] = yt_fmt.get('contentLength')
-        fmt['audio_sample_rate'] = yt_fmt.get('audioSampleRate')
+        fmt['file_size'] = extract_int(yt_fmt.get('contentLength'))
+        fmt['audio_sample_rate'] = extract_int(yt_fmt.get('audioSampleRate'))
+        fmt['duration_ms'] = yt_fmt.get('approxDurationMs')
         fmt['fps'] = yt_fmt.get('fps')
+        fmt['init_range'] = yt_fmt.get('initRange')
+        fmt['index_range'] = yt_fmt.get('indexRange')
+        for key in ('init_range', 'index_range'):
+            if fmt[key]:
+                fmt[key]['start'] = int(fmt[key]['start'])
+                fmt[key]['end'] = int(fmt[key]['end'])
         update_format_with_type_info(fmt, yt_fmt)
         cipher = dict(urllib.parse.parse_qsl(multi_get(yt_fmt,
             'cipher', 'signatureCipher', default='')))
@@ -437,6 +513,14 @@ def _extract_formats(info, player_response):
         for key, value in hardcoded_itag_info.items():
             conservative_update(fmt, key, value) # prefer info from YouTube
         fmt['quality'] = hardcoded_itag_info.get('height')
+        conservative_update(
+            fmt, 'quality',
+            extract_int(yt_fmt.get('quality'), whole_word=False)
+        )
+        conservative_update(
+            fmt, 'quality',
+            extract_int(yt_fmt.get('qualityLabel'), whole_word=False)
+        )
 
         info['formats'].append(fmt)
 
@@ -459,7 +543,7 @@ def extract_hls_formats(hls_manifest):
             if lines[i].startswith('#EXT-X-STREAM-INF'):
                 fmt = {'acodec': None, 'vcodec': None, 'height': None,
                     'width': None, 'fps': None, 'audio_bitrate': None,
-                    'itag': None, 'file_size': None,
+                    'itag': None, 'file_size': None, 'duration_ms': None,
                     'audio_sample_rate': None, 'url': None}
                 properties = lines[i].split(':')[1]
                 properties += ',' # make regex work for last key-value pair
@@ -546,6 +630,25 @@ def extract_watch_info(polymer_json):
     info['translation_languages'] = []
     captions_info = player_response.get('captions', {})
     info['_captions_base_url'] = normalize_url(deep_get(captions_info, 'playerCaptionsRenderer', 'baseUrl'))
+    # Sometimes the above playerCaptionsRender is randomly missing
+    # Extract base_url from one of the captions by removing lang specifiers
+    if not info['_captions_base_url']:
+        base_url = normalize_url(deep_get(
+            captions_info,
+            'playerCaptionsTracklistRenderer',
+            'captionTracks',
+            0,
+            'baseUrl'
+        ))
+        if base_url:
+            url_parts = urllib.parse.urlparse(base_url)
+            qs = urllib.parse.parse_qs(url_parts.query)
+            for key in ('tlang', 'lang', 'name', 'kind', 'fmt'):
+                if key in qs:
+                    del qs[key]
+            base_url = urllib.parse.urlunparse(url_parts._replace(
+                query=urllib.parse.urlencode(qs, doseq=True)))
+            info['_captions_base_url'] = base_url
     for caption_track in deep_get(captions_info, 'playerCaptionsTracklistRenderer', 'captionTracks', default=()):
         lang_code = caption_track.get('languageCode')
         if not lang_code:
@@ -635,6 +738,8 @@ def extract_watch_info(polymer_json):
 
     # other stuff
     info['author_url'] = 'https://www.youtube.com/channel/' + info['author_id'] if info['author_id'] else None
+    info['storyboard_spec_url'] = deep_get(player_response, 'storyboards', 'playerStoryboardSpecRenderer', 'spec')
+
     return info
 
 single_char_codes = {
@@ -714,10 +819,15 @@ def extract_watch_info_from_html(watch_html):
     return extract_watch_info(fake_polymer_json)
 
 
+def captions_available(info):
+    return bool(info['_captions_base_url'])
+
 
 def get_caption_url(info, language, format, automatic=False, translation_language=None):
     '''Gets the url for captions with the given language and format. If automatic is True, get the automatic captions for that language. If translation_language is given, translate the captions from `language` to `translation_language`. If automatic is true and translation_language is given, the automatic captions will be translated.'''
     url = info['_captions_base_url']
+    if not url:
+        return None
     url += '&lang=' + language
     url += '&fmt=' + format
     if automatic:
@@ -729,7 +839,7 @@ def get_caption_url(info, language, format, automatic=False, translation_languag
         url += '&tlang=' + translation_language
     return url
 
-def update_with_age_restricted_info(info, player_response):
+def update_with_new_urls(info, player_response):
     '''Inserts urls from player_response json'''
     ERROR_PREFIX = 'Error getting missing player or bypassing age-restriction: '