Commit Graph

95 Commits

Author SHA1 Message Date
Jesus E
0f4bf45cde Fix minor formatting issues 2023-06-17 16:14:59 -04:00
Jesus E
d7f934b7b2 Merge short and video parsing even further
Use multi_get and multi_deep_get for tag differences
Replace the duration check with conservative_update
2023-06-17 16:14:02 -04:00
Jesus E
a4299dc917 Merge short and video parsing 2023-06-17 16:10:59 -04:00
Jesus E
e6fd9b40f4 Fix parsing shorts
Add check for extracting duration for shorts
Make short duration extraction stricter
Fix handling shorts with no views
2023-06-17 16:08:52 -04:00
Jesus E
f322035d4a Add functional but preliminary channel tab support
Add channel tabs to the channel template and script
Update continuation token to request different tabs

Add support for 'reelItemRenderer' format required to extract shorts
2023-06-17 16:05:40 -04:00
Jesus E
74907a8183 Music list extraction: read from SONG field
This one is used when there is no corresponding YouTube video
for the track
2023-05-28 21:45:20 -04:00
Jesus E
aa57ace742 Fix music list extraction
Closes #160
2023-05-28 21:42:13 -04:00
Jesus E
e54596f3e9 Partially fix age restricted videos
Does not work for videos that require decryption because
decryption is not working (giving 403) for some reason.

Related invidious issue for decryption not working:
https://github.com/iv-org/invidious/issues/3245

Partial fix for #146
2023-05-28 21:30:51 -04:00
Jesus E
68752000f0 Update channel to new ctoken format
Huge thanks to @michaelweiser

Different sortings still don't work for videos and playlists
2023-05-28 21:04:36 -04:00
Jesus E
7b60751e99 Fix failure to detect vp9.2 and mp4v.20.3 codecs 2023-05-28 20:47:47 -04:00
Jesus E
9890617098 Fix fmt extraction mime_type regex failure as well as exceptions 2023-05-28 20:44:30 -04:00
Jesus E
0f78f07875 Remove leftover print statement 2023-05-28 20:40:25 -04:00
Jesus E
08545a29df Fix likes count 2023-05-28 20:39:11 -04:00
Jesús
1fbc0cdd46 Fix preview_thumbnails
use 'deep_get' for storyboard
2022-05-30 22:45:08 +08:00
James Taylor
79fd2966cd Extract captions base_url using different method when missing
The base url will be randomly missing.

Take one of the listed captions urls which already
has the &lang and automatic specifiers. Then remove these
specifiers.

Signed-off-by: Jesús <heckyel@hyperbola.info>
2022-03-30 00:41:30 +08:00
James Taylor
dcd4b0f0ae Fix exception when _captions_base_url is not present
Signed-off-by: Jesús <heckyel@hyperbola.info>
2022-03-30 00:37:43 +08:00
zrose584
a5ef801c07 handle missing storyboard
Signed-off-by: Jesús <heckyel@hyperbola.info>
2022-01-17 09:01:09 -05:00
zrose584
63c92e0c4e add preview thumbnails
Signed-off-by: Jesús <heckyel@hyperbola.info>
2022-01-09 16:39:50 -05:00
Jesús
a1d3cc5045 update formats 2021-12-27 13:05:54 -05:00
Jesús
92067638b1 Disable dislikes
Ref: https://blog.youtube/news-and-events/update-to-youtube/
2021-12-26 13:29:55 -05:00
Jesús
f3469b1ff4 Revert "Usage hqdefault thumbnail in related videos"
This reverts commit a0c3ca0159.
2021-09-14 16:35:04 -05:00
Jesús
a0c3ca0159 Usage hqdefault thumbnail in related videos 2021-09-14 15:58:13 -05:00
James Taylor
9c7e93ecf8 Redo av codec settings & selections to accomodate webm
Allows for ranked preferences for h264, av1, and vp9 codecs in
settings, along with equal preferences which are tiebroken using
smaller file size.

For each quality, gives av-merge a list of video sources
and audio sources sorted based on preference & file size. It
will pick the first one that the browser supports.

Closes #84

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-09-06 16:18:11 -05:00
James Taylor
2ab5b96178 Fix videos added to playlist from channel missing author_id
Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-08-31 18:29:37 -05:00
James Taylor
7c79f530a5 Support more audio and video qualities
Adds support for AV1-encoded videos, which includes any videos
above 1080p. These weren't getting included because they did
not have a quality entry in the format table at the top of
watch_extraction.py. So get the quality from the quality
labels of the format if it's not there.

Because YouTube often includes BOTH AV1 and H.264 (AVC) for each
quality, after these are included, there will be way too many
quality options and the code needs to choose which one to use.
The choice is somewhat hard: AV1 is encoded in fewer bytes than
H.264 and is patent-free, however, it has less hardware support,
so might be more difficult to play. For instance, on my system,
AV1 does not work on 1080p, but H.264 does. Adds a setting about
which to prefer, set to H.264 as the default.

Also adds support for the lower quality mp4 audio quality, which
now gets used at 144p to save network bandwidth. For similar
reasons, this was not getting included because it did not
have an audio_bitrate entry in the table. Prefer bitrate
instead for the quality.

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-08-31 16:40:19 -05:00
James Taylor
c9a75042d2 Add support for more qualities, merging video+audio using MSE
Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-08-29 18:48:56 -05:00
Jesús
e4af99fd17 Revert "Add support for more qualities, merging video+audio using MSE"
This reverts commit d56df02e7b.
2021-08-29 18:48:01 -05:00
James Taylor
d56df02e7b Add support for more qualities, merging video+audio using MSE
Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-08-29 18:44:26 -05:00
James Taylor
e76257f8b7 Fix comment reply url extraction due to youtube changes
Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-08-23 18:42:16 -05:00
James Taylor
4e556efa3d Fix comments extraction due to new response continuation key name
Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-08-23 18:40:52 -05:00
James Taylor
40fcee52c0 Fix description extraction in search results
Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-08-09 12:29:01 -05:00
James Taylor
2039972ab3 Fix (dis)like, music list extraction due to YouTube changes (again)
YouTube reverted the changes they made that prompted f9f5d5ba.

In case they change their minds again, this adds support for both
formats.

The liberal_update and conservative_update functions needed to be
modified to handle the cases of empty lists, so that
a successfully extracted 'music_list': [{'Author':...},...] will
not be overwritten by 'music_list': [] in the calls to
liberal_dict_update.

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-08-09 12:13:52 -05:00
James Taylor
3dee7ea0d1 Switch to new comments api now that old one is being disabled
watch_comment api periodically gives the error "Top level
comments mweb servlet is turned down."

The continuation items for the new api are in a different
arrangement in the json, so changes were necessary to the
extract_items function.

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-08-09 12:10:42 -05:00
James Taylor
f27105fa7f New age restriction bypass method since get_video_info was disabled
From
https://github.com/yt-dlp/yt-dlp/issues/574#issuecomment-887171136

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-07-28 23:48:54 -05:00
James Taylor
54b39f1303 Fix missing likes, dislikes, & music list due to Youtube changes
Also moves some microformat extraction from
_extract_watch_info_mobile to extract_watch_info where it belongs.
_extract_watch_info_mobile is really only for stuff visible on the
page, and thus specialized for either mobile or desktop.

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-07-28 23:47:41 -05:00
Jesús
7fd2c3474f Capitalize name app 2021-06-10 16:41:45 -05:00
James Taylor
a9edc54aac Use extract_approx_int for comment likes
Full digits no longer available

Closes #64

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-06-10 12:47:09 -05:00
James Taylor
e44647f106 Fix comment like extraction due to Youtube changes
Variable name changed from likeCount to voteCount

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-05-17 22:08:09 -05:00
James Taylor
f0cd170767 Fix videos added to playlist from channel page not having author
Information from additional_info was being overrided with None.

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-05-17 22:02:03 -05:00
James Taylor
0ee448c186 Channel about: Add http:// to links without it
So that the link is not interpretted as a relative link

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-05-06 15:21:36 -05:00
James Taylor
e549b5f67c Channel: Allow going to next pages of playlists page
Uses previous and next buttons. Now can view more than just
first page of playlists page

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-03-15 22:22:15 -05:00
James Taylor
2df4238924 Use new channel api endpoint now that browse_ajax is disabled
Fixes channel pages > 1

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-03-03 10:40:02 -05:00
James Taylor
00ef1c8627 Fix comment replies
Comment reply protobuf now requires the channel id of the uploader
of the video. Otherwise the endpoint returns 500.

Instead of making the protobuf ourselves and passing this data
around through query parameters, just use the ctoken provided to us
but modify the max_replies field from 10 to 250.

Fixes #53

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-02-26 11:39:23 -05:00
James Taylor
31fe1dac55 Fix signature decryption due to new base.js minifier rules
YouTube now includes e.g. {"fe": ...} instead of just {fe: ...}
in the javascript object entries in the object holding the
operation definitions.

Fixes #2

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-02-23 17:17:06 -05:00
James Taylor
1cc0ffcb20 yt_data_ext: support richGrid&richItem sometimes used on search
Some searches have these renderers instead of the usual ones

Signed-off-by: Jesús <heckyel@hyperbola.info>
2021-02-13 17:29:05 -05:00
James Taylor
6b6a6653a0 Fix youtube mixes
They cannot be viewed on their own, so change url in items to
go to the video+playlist instead

Signed-off-by: Jesús <heckyel@hyperbola.info>
2020-12-18 23:39:25 -05:00
James Taylor
43679efff5 channel: replace page #s w/ next page button using provided ctoken
Since yt doesn't accept page #'s when sorting by oldest

Signed-off-by: Jesús <heckyel@hyperbola.info>
2020-12-18 23:25:45 -05:00
James Taylor
5edcaa4f9d Improve ytInitialPlayerResponse extraction
Makes it work if there are additional javascripts statements
after the playerResponse variable

Signed-off-by: Jesús <heckyel@hyperbola.info>
2020-12-17 11:00:04 -05:00
James Taylor
9d0be82e74 Always extract from html watch page to get base.js url
Youtube removed the url from the pbj responses. They are now
only in the html page. Replaces previous fix for the missing
base.js issue.
2020-12-12 23:11:54 -08:00
James Taylor
6443cedf62 Retrieve base.js url from html watch page when it's missing
Fixes failure mode 3 in #22
2020-12-09 17:08:12 -08:00