YouTube is rolling out or A/B testing a new date format returned inside player
responses, which are precise to the second instead of the day.
This commit makes the StreamExtractor tests use these more precise dates.
This commit fixes the testRelatedItems test method by:
- accepting consent in the test class, in order to extract mixes in
recommendations;
- removing assertion of a music mix inside the recommendations, as YouTube
doesn't seem to return such mixes anymore, at least for the video used in the
test class.
Replace the video used in this test class with another one publicly available
and update the corresponding expected test values.
The test class's mocks will be updated in a different commit.
- Change CarouselHeader test channel to Sports system one, as the Coachella one
doesn't return this channel header anymore;
- Fix InteractiveTabbedHeader test by checking whether the test's channel
description is not empty instead of containing some words, as it is changing
frequently.
Video's title and tags have been changed by its uploader, so they have to be
updated.
Also make some package-private constants private, as they are not used outside
of the class, and remove unneeded test overrides.
These crisis resources are preventing search results to be returned. See
https://support.google.com/youtube/answer/10726080?hl=en for more info on them.
This commit changes search parameters to include the property allowing to show
search results.
YouTube returns sometimes videos inside channel search results. As we only want
results corresponding to the type we requested, this commits makes
YoutubeSearchExtractor ignoring non-requested search results we get, using the
extractor LinkHandler's first content filter value.
Also remove an unneeded exception throwing declaration in
YoutubeSearchExtractor.
This query parameter for which its value is set to false was not added to two
requests made in test classes of YoutubeMixPlaylistExtractorTest.
Also remove an unneeded ParsingException exception throwing declaration in a
test method.
This should make returned dates consistent between timezones and countries on
which the extractor is ran.
It was previously only set on YouTube Music search continuations.
For every InnerTube request:
- Always add a `request` object with the following properties:
- "internalExperimentFlags" set to an empty array;
- "useSsl" set to "true";
- "lockedSafetyMode" set to "false".
- Use proper TODO comment to provide a way to enable restricted mode on every
request and add it on requests on which it wasn't present.
For YouTube Music:
- Remove alt query parameter, as it is not used anymore by the website;
- Add prettyPrint query parameter with false value on YouTube Music search
continuations.
Default image qualities were removed in image URLs with the jpg extension,
causing the addition of the image suffix to full non-JPG images URLs and so to
invalid image URLs.
Only the image quality name with its leading "-" character and the "."
character after the name is now removed and replaced by a string format
replaced itself with the image quality name for each quality.
As the image suffixes do not contain the image extension, the name of image
qualities lists has been adapted with these changes and some related comments
have been also improved.
Some services may provide different image formats using the same suffix,
without we know what format the service provide. Enforcing an image extension
could so lead to provide invalid image URLs, like for SoundCloud PNG images
currently.
With this documentation change, it is now clear that users of this class decide
of whether they want to include image extensions in the suffix. The previous
behavior described in the Javadoc was not enforced.
The signature timestamp is used as a number by HTML5 clients, so it should be
used in the same way by the extractor too instead of being a string.
As the timestamp doesn't seem to exceed 5 digits, an integer is used to store
its value.
This commit is introducing breaking changes.
For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.
The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.
YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.
If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.
All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.
Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
The goal of this class is to decouple the extraction of signature timestamp and
signature deobfuscation function from YoutubeStreamExtractor.
The extraction of the signature deobfuscation function has been also adapted to
support the latest YouTube player versions.
This new class, YoutubeSignatureUtils, doens't store anything temporary such as
a copy of the player code, which has to be passed where required. It is not
public, as it will be used by a JavaScript player manager class in the future,
in order to handle in a better way fetching, caching and resetting cache of the
player code.
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.
BandcampChannelExtractorTest.testLength has been removed as the test is always
true.
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains f4.bcbits.com/img and ends with .jpg or .png.
To do so, a new non-instantiable final class has been added: BandcampTestUtils.
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains the string yt.
The JavaDoc of the class has been also updated to reflect the changes made in
it (it is now more general).
Two new methods have been added in ExtractorAsserts to check if a collection is
empty:
- assertNotEmpty(String, Collection<?>), checking:
- the non nullity of the collection;
- its non emptiness (if that's not case, an exception will be thrown using
the provided message).
- assertNotEmpty(Collection<?>), calling assertNotEmpty(String, Collection<?>)
with null as the value of the string argument.
A new one has been added to this assertion class to check the contrary:
assertEmpty(Collection<?>), checking emptiness of the collection only if it is
not null.
Three new methods have been added in ExtractorAsserts as utility test methods
for image collections:
- assertContainsImageUrlInImageCollection(String, Collection<Image>), checking
that:
- the provided URL and image collection are not null;
- the image collection contains at least one image which has the provided
string value as its URL (which is a string) property.
- assertContainsOnlyEquivalentImages(Collection<Image>, Collection<Image>),
checking that:
- both collections are not null;
- they have the same size;
- each image of the first collection has its equivalent in the second one.
This means that the properties of an image in the first collection must be
equal in an image of the second one.
- assertNotOnlyContainsEquivalentImages(Collection<Image>, Collection<Image>),
checking that:
- both collections are not null;
- one of the following conditions is met:
- they have different sizes;
- an image of the first collection has not its equivalent in the second one.
This means that the properties of an image in the first collection must
be not equal in an image of the second one.
These methods will be used by services extractors tests (and default ones) to
test image collections.
This new method, defaultTestImageList(List<Image), will check that the image
list is not null.
For each image, it will test that its URL is secure and its height and width
are more than or equal to their relevant unknown constants in the Image class
(HEIGHT_UNKNOWN and WIDTH_UNKNOWN).
These three new methods, added in MediaCCCParsingHelper,
getImageListFromImageUrl(String), getThumbnailsFromStreamItem(JsonObject) and
getThumbnailsFromLiveStreamItem(JsonObject) (the last two are based on a common
method, getThumbnailsFromObject(JsonObject, String, String)), return an empty
list if the case no image URL could be extracted.
Images returned have their height and width unknown and a resolution level
depending on the image key of the JSON API response.
Bandcamp images work with image IDs, which provide different resolutions.
Images on Bandcamp are not always squares, and some IDs respect aspect ratios
where some others not.
The extractor will only use the ones which preserve aspect ratio and will not
provide original images, for performance and size purposes.
Because of this aspect ratio preservation constraint, only one dimension will
be known at a time.
The image IDs with their respective dimension used are:
- 10: 1200w;
- 101: 90h;
- 170: 422h;
- 171: 646h;
- 20: 1024w;
- 200: 420h;
- 201: 280h;
- 202: 140h;
- 204: 360h;
- 205: 240h;
- 206: 180h;
- 207: 120h;
- 43: 100h;
- 44: 200h.
(Where w represents the width of the image and h the height of the image)
Note that these dimensions are theoretical because if the image size is less
than the dimensions of the image ID, it will be not upscaled but kept to its
original size.
All these resolutions are stored in a private static list of ThumbnailSuffixes
in BandcampExtractorHelper, in which the methods to get mutliple images have
been added:
- getImagesFromImageUrl(String): public method to get images from an image URL;
- getImagesFromImageId(long, boolean): public method to get images from an
image ID;
- getImagesFromImageBaseUrl(String): private utility method to get images from
the static list of ThumbnailSuffixes from a given image base URL, containing
the path to the image, a "a" letter if it comes from an album, its ID and an
underscore.
Some existing methods have been also edited:
- the documentation of getImageUrl(long, boolean) has been changed to reflect
the Bandcamp images findings;
- getThumbnailUrlFromSearchResult has been renamed to
getImagesFromSearchResult, and a documentation has been added to this method.
The method replaceHttpWithHttps of the Utils class has been also used in
BandcampExtractorHelper instead of doing manually what the method does.
The default avatar picture was used when no profile picture was found, but it
was removed and split in multiple images.
Thumbnails' size is not known, as this data is not provided by the API.
This method, getThumbnailsFromPlaylistOrVideoItem, has been added in
PeertubeParsingHelper and returns the two image variants for playlists and
videos.
Four new static methods have been added in PeertubeParsingHelper to do so:
- two public methods to get the corresponding image type:
getAvatarsFromOwnerAccountOrVideoChannelObject(String, JsonObject) and
getBannersFromAccountOrVideoChannelObject(String, JsonObject);
- two private methods as helper methods: getImagesFromAvatarsOrBanners(String,
JsonObject, String, String) and getImagesFromAvatarOrBannerArray(String,
JsonArray).
These new public and static methods, added in SoundcloudParsingHelper,
getAllImagesFromArtworkOrAvatarUrl(String) and
getAllImagesFromVisualUrl(String) (which call a common private method,
getAllImagesFromImageUrlReturned(String, List<ImageSuffix>, List<Image>)),
return an unmodifiable list of JPEG images containing almost every image
resolution provided by SoundCloud except the original size and the tiny
resolution (for artworks and avatars, as the image size is 20x20 for artworks
and 18x18 for avatars, so very close to or equal to the t20x20 resolution):
- for artworks and avatars:
- mini: 16x16;
- t20x20: 20x20;
- small: 32x32;
- badge: 47x47;
- t50x50: 50x50;
- t60x60: 60x60;
- t67x67: 67x67;
- large: 100x100;
- t120x120: 120x120;
- t200x200: 200x200;
- t240x240: 240x240;
- t250x250: 250x250;
- t300x300: 300x300;
- t500x500: 500x500.
- for visuals/user banners:
- t1240x260: 1240x260;
- t2480x520: 2480x520.
Duplicated code in two methods of SoundcloudParsingHelper
(getUsersFromApi(ChannelInfoItemsCollector, String) and
getStreamsFromApi(StreamInfoItemsCollector, String, boolean)) has been merged
into one common private method, getNextPageUrlFromResponseObject(JsonObject).
Splitting YoutubeMusicSearchExtractor's InfoItemExtractors into separate
classes (YoutubeMusicSongOrVideoInfoItemExtractor,
YoutubeMusicAlbumOrPlaylistInfoItemExtractor and
YoutubeMusicArtistInfoItemExtractor) allows to simplify
YoutubeMusicSearchExtractor,improves reading and applying changes to InfoItems
(no more losing at least quarter of a line due to indentations).
These InfoItems, in which the image changes have been applied, don't extend the
YouTube ones anymore, as most methods were overridden and the few ones that are
not don't apply in YouTube Music items responses, so it was useless to extend
them.
The code of YoutubeMusicSearchExtractor have been also improved a bit.
Unmodifiable lists of Images are returned, parsed from a given YouTube
"thumbnails" JSON array.
These methods will be used in all YouTube extractors and InfoItems, as the
structures between content types (videos, channels, playlists, ...) are common.
The goal of this utility class is to simply store suffixes which need to be
appended to image URLs, in order to get images at the suffix resolution.
This class contains four properties: the suffix (as a string), the height,
the width (as integers) and the estimated resolution level of the image
corresponding to the one represented by the suffix.
Objects of this serializable class contains four properties: a URL (as a
string), a width, a height (represented as integers) and an estimated
resolution level, which can be constructed from a given height.
Possible resolution levels are:
- UNKNOWN: for unknown heights or heights <= 0;
- LOW: for heights > 0 & < 175;
- MEDIUM: for heights >= 175 & < 720;
- HIGH: for heights >= 720.
Getters of these properties are available and the constructor needs these four
properties.
The addition of this support required to turn the isCarouselHeader boolean into
an enum containing all supported channel headers named HeaderType.
Also assert that the page has been fetched where needed to avoid
NullPointerExceptions when the channel page has been not fetched and remove the
getChannelHeaderJson method in YoutubeChannelExtractor, method for which its
code has been moved to its sole usage after the new headers support changes.
The non-null assertion was made on the exception message instead of the string
to check, causing a NullPointerException if the string to check was null.
Note that this introduces a "Raw use of parameterized class 'InfoItemsPage'" warning, but it can be ignored since the type missing would be <InfoItem>, and StreamInfoItem extends InfoItem
Only the first content filter of the ListLinkHandler instances provided is
used when collecting all channel tabs of the ListLinkHandler list, as channel
tabs implementations only use one content filter per ListLinkHandler instance.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
These changes should help to detect tests as tests, when running a subset of
tests or all tests.
They should be also implemented in these interfaces' implementations (new and
existing ones).
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Support of tags and videos, shorts, live, playlists and channels tabs has been
added for non-age restricted channels.
Age-restricted channels are now also supported and always returned the videos,
shorts and live tabs, accessible using system playlists. These tabs are the
only ones which can be accessed using YouTube's desktop website without being
logged-in.
The videos channel tab parameter has been updated to the one used by the
desktop website and when a channel extraction is fetched, this tab is returned
in the list of tabs as a cached one in the corresponding link handler.
Visitor data support per request has been added, as a valid visitor data is
required to fetch continuations with contents on the shorts tab. It is only
used in this case to enhance privacy.
A dedicated shorts UI elements (reelItemRenderers) extractor has been added,
YoutubeReelInfoItemExtractor. These elements do not provide the exact view
count, any uploader info (name, URL, avatar, verified status) and the upload
date.
All service's LinkHandlers are now using the singleton pattern and some code
has been also improved on the files changed.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
This is required to parse duration of YouTube's reelItemRenderers, returned
only inside accessibility data.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Support of tracks, playlists and albums has been added for users.
Also add the declaration of the UnsupportedOperationException exception to the
service's LinkHandlers.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
Support of tracks and albums has been added for artists.
Also use the singleton pattern and add the declaration of the
UnsupportedOperationException exception to the service's LinkHandlers and
improved some code in the files changed.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
Support of channels and videos has been added for accounts and support of
videos and playlists has been added for video channels.
The following changes have been also done:
- collectStreamsFrom method in PeertubeParsingHelper has been renamed to
collectItemsFrom;
- PeertubeChannelInfoItemExtractor.getStreamCount method has been fixed due to
ChannelExtractor's new inheritance;
- the declaration of the UnsupportedOperationException exception thrown has
been added to the service's LinkHandlers;
- a channel tab LinkHandlerFactory has been added,
PeertubeChannelTabLinkHandlerFactory;
- all service's LinkHandlers are now using properly the singleton pattern.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
This new ListLinkHandler, ReadyChannelTabListLinkHandler, should help saving
clients data, energy and time by helping to reduce duplicate requests.
Co-authored-by: Stypox <stypox@pm.me>
This change advertise to clients that channel tabs' link handler factories can
return an UnsupportedOperationException when a tab provided to them is
unsupported.
This commit introduces the following breaking changes:
- Three new classes have been added:
- ChannelTabExtractor, class extending ListExtractor<InfoItem>, which
extracts InfoItems from a channel tab;
- ChannelTabInfo extending ListInfo<InfoItem>, which extracts InfoItems from
a ChannelTabExtractor and returns them as a ChannelTabInfo;
- ChannelTabs, an immutable class containing all supported channel tabs.
- StreamingService implementations must implement new methods returning a
channel tab LinkHandlerFactory (getChannelTabsLHFactory) and a
ChannelTabExtractor (getChannelTabExtractor);
- ChannelExtractor inherits Extractor instead of ListExtractor<StreamInfoItem>
and ChannelInfo inherits Info instead of ListInfo<StreamInfoItem>;
- ChannelExtractor and ChannelInfo have now getters and/or setters of tabs.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
- Enhance documentation;
- Fix the regular expression fallback on HTML embed watch page;
- Use HTML scripts tag search first instead of the regular expression approach,
now used as a last resort;
- Compile regular expressions only once, in order to improve the performance of
subsequent extraction calls when clearing the cache;
- Provide original exceptions when fetching or parsing pages on which the base
JavaScript's player could be found failed, allowing clients to detect network
errors when they are the cause of the failures for instance;
- Remove delegate method which was not taking a video ID and hardcoding one, as
we can provide the video ID in all cases or do not provide a video ID at worse;
- Rename and make extraction methods package-private, as they are not intended
to be used publicly.
These breaking internal changes have been applied where needed, in
YoutubeJavaScriptExtractorTest and YoutubeStreamExtractor (in which an unneeded
initStsFromPlayerJsIfNeeded call have been removed).
- Fix testCheckAudioStreams test of
YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test class, by updating
the excepted audio track name test to use the updated English audio track name
(audio track type info has been added on the video tested);
- Fix YoutubeStreamExtractorDefaultTest.PublicBroadcasterTest test class by
using a different video from a French and German public broadcast channel, as
the channel Dinge Erklärt – Kurzgesagt is not affiliated with a public
broadcast channel anymore;
- Fix YoutubeStreamExtractorLivestreamTest test class, by updating the excepted
name of the livestream to the current one.
These parameters are the only ones currently known to bypass 403 HTTP issues
related to failure of passing Android client integrity checks, as the ones of
stories (and the base of the shorts ones) do not work anymore, which may be
related to end of this format on the service.
- Remove useless concatenation on the downloader path;
- Remove unneeded public test modifier;
- Update license header;
- Specify the service class tested instead of the generic class.
- Switch to the new domain used by YouTube for search suggestions,
suggestqueries-clients6.youtube.com, and add the xhr query parameter with the
t value, to allow getting responses without requiring trim;
- Use the Java 8 Stream API to collect search suggestions and improve invalid
response detection by checking whether the content type of the response
returned is JSON;
- Move the licence header at the top of the file.
When a description is missing, no description should be returned, even the ones
indicating there is no description. This behavior is represented by a null
return instead.
Also update PeertubeAccountExtractorTest to reflect these changes.
The tested comment has been removed, so it couldn't be found in the comments
list.
This comment has been replaced by a new one from the current comments of the
video.
Also, in the parent class PeertubeCommentsExtractorTest, final has been used as
much as possible and for-each loops of lists have been replaced by their
forEach method or the Stream API, in order to simplify code.
As the "No views" string is returned in the case there is no view on a video, a
number cannot be parsed in this case, so -1 was returned.
This string is now detected in all methods to get the view count of a stream.
Getting audio tracks locales by parsing their ID or their label, should not be
done by clients, but by the extractor.
This commit adds the ability to store the Locale of an AudioStream, which is
used to compare similar AudioStreams (in the equalStats method).
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.
The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).
As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.
URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.
As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
As like count is now returned by the extractor, we need to assert a positive
minimum like count, which is close to the actual value, in order to avoid test
failures due to lower like counts than the ones excepted.
- Remove unused imports;
- Replace wildcard imports by single class imports;
- Suppress "HTTP links are not secured" warnings from IDEA IDEs;
- Replace removed video jZViOEv90dI by an existing video, 9Dpqou5cI08 (the
corresponding test has been of course renamed).
- Move license header at the top;
- Use an unmodifiable set for the subpaths instead of a modifiable list;
- Add missing Nonnull and Nullable annotations;
- Improve exception messages.
The yt:channelId element doesn't provide the channel ID anymore and is empty,
like the id element, so we need now to extract it from the channel URL provided
in two elements: author -> uri and feed -> link.
Also avoid a NullPointerException in getUrl and getName methods.
- Return duration of video premieres;
- Add another non-localized method to determine whether a stream is a running
livestream;
- Return view count and upload date of videos in playlists;
- Store isPremiere result;
- Remove shorts workaround code, as it was only useful on channels and shorts
have been moved into a separated channel tab;
- Improve some other code.
This header was not sent partially before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.
Some code in the modified classes has been improved and / or deduplicated, and usages of the UTF_8 constant of the Utils class has been replaced by StandardCharsets.UTF_8 where possible.
Note that this header has been not added in except in YoutubeDashManifestCreatorsUtils, as an empty body is sent in the POST requests made by this class.
This header was not sent before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.
Also make use of StandardCharsets.UTF_8 when getting bytes of bodies instead of the platform default's charset, to make sure to prevent some encoding issues on some JVMs.
The video "Makani’s first commercial-scale energy kite" (video ID:
An8vtD1FDqs), which has this behavior, is used for the new test,
NoVisualMetadataVideoTest, added in YoutubeStreamExtractorDefaultTest.
Tests of elements who throw an exception in this case (subscriber count, like
count, uploader avatar URL) test if the ParsingException exception is thrown by
YoutubeStreamExtractor.
More non-channel paths have been also added to the excluded custom name paths,
documentation and exception messages have been improved and fixed in some
places, and the licence header of YoutubeChannelLinkHandlerFactory has been
moved to its beginning and updated.
Also update the mock of the next InnerTube endpoint response of the
YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing test class with an
attributedDescription instead of a regular description
The video count is now returned for this playlist, so it isn't unknown.
The testStreamCount method of this test class asserts now that the stream count
is greater than 40.
This new layout doesn't provide author thumbnails and is completely different
for metadata, so the code to get them has been refactored.
The code of learning playlists video count check has been also removed, as it
seems to be not relevant anymore (the video count seems to be returned for
these playlists with both layouts).
Finally, unneeded overrides of subchannel methods, which don't apply to the
YouTube service, have been removed.
Using the player parameters used to get stories seems to fix the issue, which
affects currently only certain countries such as UK.
This is a workaround and should be fixed in a better way (by changing the
InnerTube additional client used for videos or finding what is now required in
Android player requests).
Use the same possible characters for variables everywhere, in order to avoid
potential future throttling parameter decryption function name parsing issues
related to the usage of other letter(s) than b.
- Quote the function name, as it may contain special regex symbols, such as
dollar;
- Support multiple lines;
- Use what looks like the end of the function for the end of the regex (this
part is inspired from yt-dlp throttling parameter decryption regex);
- Move the throttling function body regex into a private and static constant.
This new structure allow us to filter easily Trending shorts and Recently
trending sections.
On the previous one, this Recently trending section is now filtered, by
checking whether sections have a title, which isn't the case for normal trends
contrary to the other ones.
This makes that the extractor returns now only the real 50 "Now" YouTube
trends.
Elements inside arrays are now extracted dynamically instead of only the ones
of the first index, using Java 8's Stream API.
The getInitialPage() method of YoutubeTrendingExtractor can now throw a
ParsingException if no selected tab (corresponding to the one of the trends
type extracted) has been found.
Finally, the licence header has been moved to the top of the file and updated.
In the new data model currently A/B tested or deployed, the like count button
data is the same, but its path has been changed.
The extraction of the like count has been also improved, by using the multiple
accessibility data available instead of the only one which was used before.
This accessibility data has been also deprioritized, because it is language
dependent when there is no view.
Like count is always known, even for age-restricted videos, so returning -1 if
the like count extraction fails on this type of videos is not needed and hides
an extraction error: that's the reason why this return has been removed and the
exception will always be thrown, even if a video is age-restricted.
YoutubeThrottlingDecrypter is now a non-instantiable final class and non-static
attributes have been removed.
The static attributes of this class have been renamed, in order to respect the
naming specification used.
AudricV