2022-03-06 19:10:11 +00:00
|
|
|
/*
|
|
|
|
* Created by Christian Schabesberger on 06.08.15.
|
|
|
|
*
|
2023-09-22 22:10:15 +00:00
|
|
|
* Copyright (C) 2019 Christian Schabesberger <chris.schabesberger@mailbox.org>
|
2022-03-06 19:10:11 +00:00
|
|
|
* YoutubeStreamExtractor.java is part of NewPipe Extractor.
|
|
|
|
*
|
|
|
|
* NewPipe Extractor is free software: you can redistribute it and/or modify
|
|
|
|
* it under the terms of the GNU General Public License as published by
|
|
|
|
* the Free Software Foundation, either version 3 of the License, or
|
|
|
|
* (at your option) any later version.
|
|
|
|
*
|
|
|
|
* NewPipe Extractor is distributed in the hope that it will be useful,
|
|
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
* GNU General Public License for more details.
|
|
|
|
*
|
|
|
|
* You should have received a copy of the GNU General Public License
|
|
|
|
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
|
|
|
|
*/
|
|
|
|
|
2018-05-08 19:19:03 +00:00
|
|
|
package org.schabi.newpipe.extractor.services.youtube.extractors;
|
2017-03-01 17:47:52 +00:00
|
|
|
|
2022-05-07 17:32:12 +00:00
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.ItagItem.APPROX_DURATION_MS_UNKNOWN;
|
2022-08-15 03:49:40 +00:00
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.ItagItem.CONTENT_LENGTH_UNKNOWN;
|
2024-03-31 20:54:47 +00:00
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeDescriptionHelper.attributedDescriptionToHtml;
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.CONTENT_CHECK_OK;
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.CPN;
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.RACY_CHECK_OK;
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.VIDEO_ID;
|
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.createDesktopPlayerBody;
|
2022-03-18 14:09:06 +00:00
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.fixThumbnailUrl;
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.generateContentPlaybackNonce;
|
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.generateTParameter;
|
2022-07-22 20:31:43 +00:00
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getImagesFromThumbnailsArray;
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getJsonAndroidPostResponse;
|
2022-03-26 19:02:35 +00:00
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getJsonIosPostResponse;
|
2022-03-18 14:09:06 +00:00
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getJsonPostResponse;
|
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getTextFromObject;
|
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.prepareAndroidMobileJsonBuilder;
|
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.prepareDesktopJsonBuilder;
|
2022-03-26 19:02:35 +00:00
|
|
|
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.prepareIosMobileJsonBuilder;
|
2022-03-18 14:09:06 +00:00
|
|
|
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
|
|
|
|
|
2017-11-22 17:39:38 +00:00
|
|
|
import com.grack.nanojson.JsonArray;
|
2017-08-16 02:40:03 +00:00
|
|
|
import com.grack.nanojson.JsonObject;
|
2021-05-23 15:55:19 +00:00
|
|
|
import com.grack.nanojson.JsonWriter;
|
2021-04-15 16:58:59 +00:00
|
|
|
|
2022-07-22 20:31:43 +00:00
|
|
|
import org.schabi.newpipe.extractor.Image;
|
2019-04-28 20:03:16 +00:00
|
|
|
import org.schabi.newpipe.extractor.MediaFormat;
|
2020-12-20 18:54:12 +00:00
|
|
|
import org.schabi.newpipe.extractor.MetaInfo;
|
2022-02-02 19:23:11 +00:00
|
|
|
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
|
2019-04-28 20:03:16 +00:00
|
|
|
import org.schabi.newpipe.extractor.StreamingService;
|
|
|
|
import org.schabi.newpipe.extractor.downloader.Downloader;
|
2021-04-15 16:58:59 +00:00
|
|
|
import org.schabi.newpipe.extractor.exceptions.AgeRestrictedContentException;
|
|
|
|
import org.schabi.newpipe.extractor.exceptions.ContentNotAvailableException;
|
|
|
|
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
|
|
|
|
import org.schabi.newpipe.extractor.exceptions.GeographicRestrictionException;
|
|
|
|
import org.schabi.newpipe.extractor.exceptions.PaidContentException;
|
|
|
|
import org.schabi.newpipe.extractor.exceptions.ParsingException;
|
|
|
|
import org.schabi.newpipe.extractor.exceptions.PrivateContentException;
|
|
|
|
import org.schabi.newpipe.extractor.exceptions.YoutubeMusicPremiumContentException;
|
2018-07-13 16:02:40 +00:00
|
|
|
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
|
2021-05-23 15:55:19 +00:00
|
|
|
import org.schabi.newpipe.extractor.localization.ContentCountry;
|
2019-11-03 18:45:25 +00:00
|
|
|
import org.schabi.newpipe.extractor.localization.DateWrapper;
|
2020-02-25 20:19:53 +00:00
|
|
|
import org.schabi.newpipe.extractor.localization.Localization;
|
2019-12-16 07:35:43 +00:00
|
|
|
import org.schabi.newpipe.extractor.localization.TimeAgoParser;
|
2020-02-25 20:19:53 +00:00
|
|
|
import org.schabi.newpipe.extractor.localization.TimeAgoPatternsManager;
|
2018-05-08 19:19:03 +00:00
|
|
|
import org.schabi.newpipe.extractor.services.youtube.ItagItem;
|
[YouTube] Refactor JavaScript player management API
This commit is introducing breaking changes.
For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.
The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.
YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.
If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.
All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.
Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-11 21:16:11 +00:00
|
|
|
import org.schabi.newpipe.extractor.services.youtube.YoutubeJavaScriptPlayerManager;
|
2023-12-07 19:57:44 +00:00
|
|
|
import org.schabi.newpipe.extractor.services.youtube.YoutubeMetaInfoHelper;
|
2020-04-10 08:51:05 +00:00
|
|
|
import org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper;
|
2020-05-30 08:25:43 +00:00
|
|
|
import org.schabi.newpipe.extractor.services.youtube.linkHandler.YoutubeChannelLinkHandlerFactory;
|
2022-03-18 14:09:06 +00:00
|
|
|
import org.schabi.newpipe.extractor.stream.AudioStream;
|
2022-03-06 19:10:11 +00:00
|
|
|
import org.schabi.newpipe.extractor.stream.DeliveryMethod;
|
2022-03-18 14:09:06 +00:00
|
|
|
import org.schabi.newpipe.extractor.stream.Description;
|
|
|
|
import org.schabi.newpipe.extractor.stream.Frameset;
|
|
|
|
import org.schabi.newpipe.extractor.stream.Stream;
|
|
|
|
import org.schabi.newpipe.extractor.stream.StreamExtractor;
|
|
|
|
import org.schabi.newpipe.extractor.stream.StreamSegment;
|
|
|
|
import org.schabi.newpipe.extractor.stream.StreamType;
|
|
|
|
import org.schabi.newpipe.extractor.stream.SubtitlesStream;
|
|
|
|
import org.schabi.newpipe.extractor.stream.VideoStream;
|
2020-02-09 10:59:23 +00:00
|
|
|
import org.schabi.newpipe.extractor.utils.JsonUtils;
|
2022-12-16 16:38:37 +00:00
|
|
|
import org.schabi.newpipe.extractor.utils.LocaleCompat;
|
2022-04-16 17:00:41 +00:00
|
|
|
import org.schabi.newpipe.extractor.utils.Pair;
|
2017-06-29 18:12:55 +00:00
|
|
|
import org.schabi.newpipe.extractor.utils.Parser;
|
2017-07-11 03:08:03 +00:00
|
|
|
import org.schabi.newpipe.extractor.utils.Utils;
|
2017-03-01 17:47:52 +00:00
|
|
|
|
|
|
|
import java.io.IOException;
|
2022-05-01 14:38:05 +00:00
|
|
|
import java.nio.charset.StandardCharsets;
|
2020-10-18 03:48:14 +00:00
|
|
|
import java.time.LocalDate;
|
|
|
|
import java.time.OffsetDateTime;
|
|
|
|
import java.time.format.DateTimeFormatter;
|
2022-03-18 14:09:06 +00:00
|
|
|
import java.util.ArrayList;
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
import java.util.Arrays;
|
2022-03-18 14:09:06 +00:00
|
|
|
import java.util.Collections;
|
|
|
|
import java.util.List;
|
|
|
|
import java.util.Locale;
|
|
|
|
import java.util.Map;
|
|
|
|
import java.util.Objects;
|
2022-05-01 14:38:05 +00:00
|
|
|
import java.util.stream.Collectors;
|
|
|
|
|
2022-03-18 14:09:06 +00:00
|
|
|
import javax.annotation.Nonnull;
|
|
|
|
import javax.annotation.Nullable;
|
2020-02-27 16:39:23 +00:00
|
|
|
|
2017-03-01 17:47:52 +00:00
|
|
|
public class YoutubeStreamExtractor extends StreamExtractor {
|
2022-01-15 16:25:00 +00:00
|
|
|
private static boolean isAndroidClientFetchForced = false;
|
|
|
|
private static boolean isIosClientFetchForced = false;
|
|
|
|
|
2020-10-29 17:44:05 +00:00
|
|
|
private JsonObject playerResponse;
|
2021-05-23 15:55:19 +00:00
|
|
|
private JsonObject nextResponse;
|
2021-05-29 12:43:26 +00:00
|
|
|
|
|
|
|
@Nullable
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
private JsonObject html5StreamingData;
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
@Nullable
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
private JsonObject androidStreamingData;
|
2022-01-15 16:25:00 +00:00
|
|
|
@Nullable
|
|
|
|
private JsonObject iosStreamingData;
|
|
|
|
|
2020-02-28 16:03:21 +00:00
|
|
|
private JsonObject videoPrimaryInfoRenderer;
|
|
|
|
private JsonObject videoSecondaryInfoRenderer;
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
private JsonObject playerMicroFormatRenderer;
|
2020-10-29 17:44:05 +00:00
|
|
|
private int ageLimit = -1;
|
2022-01-15 16:25:00 +00:00
|
|
|
private StreamType streamType;
|
2017-08-10 17:50:59 +00:00
|
|
|
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
// We need to store the contentPlaybackNonces because we need to append them to videoplayback
|
|
|
|
// URLs (with the cpn parameter).
|
|
|
|
// Also because a nonce should be unique, it should be different between clients used, so
|
|
|
|
// three different strings are used.
|
|
|
|
private String html5Cpn;
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
private String androidCpn;
|
2022-01-15 16:25:00 +00:00
|
|
|
private String iosCpn;
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
|
2021-02-24 16:06:38 +00:00
|
|
|
public YoutubeStreamExtractor(final StreamingService service, final LinkHandler linkHandler) {
|
2019-04-28 20:03:16 +00:00
|
|
|
super(service, linkHandler);
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|
|
|
|
|
2017-07-11 03:08:03 +00:00
|
|
|
/*//////////////////////////////////////////////////////////////////////////
|
|
|
|
// Impl
|
|
|
|
//////////////////////////////////////////////////////////////////////////*/
|
2017-03-01 17:47:52 +00:00
|
|
|
|
2017-11-25 00:10:04 +00:00
|
|
|
@Nonnull
|
2017-03-01 17:47:52 +00:00
|
|
|
@Override
|
2017-08-11 01:23:09 +00:00
|
|
|
public String getName() throws ParsingException {
|
2017-11-30 09:49:27 +00:00
|
|
|
assertPageFetched();
|
2022-06-02 08:57:52 +00:00
|
|
|
String title;
|
2020-07-18 07:50:22 +00:00
|
|
|
|
2022-06-02 08:57:52 +00:00
|
|
|
// Try to get the video's original title, which is untranslated
|
|
|
|
title = playerResponse.getObject("videoDetails").getString("title");
|
2020-02-29 16:18:50 +00:00
|
|
|
|
2020-05-11 09:40:24 +00:00
|
|
|
if (isNullOrEmpty(title)) {
|
[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.
The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).
As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.
URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.
As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
2023-02-20 12:21:55 +00:00
|
|
|
title = getTextFromObject(getVideoPrimaryInfoRenderer().getObject("title"));
|
2020-02-29 16:18:50 +00:00
|
|
|
|
2022-03-18 14:09:06 +00:00
|
|
|
if (isNullOrEmpty(title)) {
|
|
|
|
throw new ParsingException("Could not get name");
|
|
|
|
}
|
2017-11-25 00:10:04 +00:00
|
|
|
}
|
2020-02-29 16:18:50 +00:00
|
|
|
|
|
|
|
return title;
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|
|
|
|
|
2020-11-04 13:50:35 +00:00
|
|
|
@Nullable
|
2017-03-01 17:47:52 +00:00
|
|
|
@Override
|
2019-04-28 20:03:16 +00:00
|
|
|
public String getTextualUploadDate() throws ParsingException {
|
2022-08-15 03:49:40 +00:00
|
|
|
if (!playerMicroFormatRenderer.getString("uploadDate", "").isEmpty()) {
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
return playerMicroFormatRenderer.getString("uploadDate");
|
2022-08-15 03:49:40 +00:00
|
|
|
} else if (!playerMicroFormatRenderer.getString("publishDate", "").isEmpty()) {
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
return playerMicroFormatRenderer.getString("publishDate");
|
2022-05-01 14:38:05 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
final JsonObject liveDetails = playerMicroFormatRenderer.getObject(
|
|
|
|
"liveBroadcastDetails");
|
2022-08-15 03:49:40 +00:00
|
|
|
if (!liveDetails.getString("endTimestamp", "").isEmpty()) {
|
2022-05-01 14:38:05 +00:00
|
|
|
// an ended live stream
|
|
|
|
return liveDetails.getString("endTimestamp");
|
2022-08-15 03:49:40 +00:00
|
|
|
} else if (!liveDetails.getString("startTimestamp", "").isEmpty()) {
|
2022-05-01 14:38:05 +00:00
|
|
|
// a running live stream
|
|
|
|
return liveDetails.getString("startTimestamp");
|
|
|
|
} else if (getStreamType() == StreamType.LIVE_STREAM) {
|
|
|
|
// this should never be reached, but a live stream without upload date is valid
|
|
|
|
return null;
|
2020-04-16 14:08:14 +00:00
|
|
|
}
|
2020-02-25 08:50:22 +00:00
|
|
|
|
2022-11-04 17:35:53 +00:00
|
|
|
final String videoPrimaryInfoRendererDateText =
|
|
|
|
getTextFromObject(getVideoPrimaryInfoRenderer().getObject("dateText"));
|
|
|
|
|
|
|
|
if (videoPrimaryInfoRendererDateText != null) {
|
|
|
|
if (videoPrimaryInfoRendererDateText.startsWith("Premiered")) {
|
|
|
|
final String time = videoPrimaryInfoRendererDateText.substring(13);
|
|
|
|
|
|
|
|
try { // Premiered 20 hours ago
|
|
|
|
final TimeAgoParser timeAgoParser = TimeAgoPatternsManager.getTimeAgoParserFor(
|
2024-03-29 12:41:23 +00:00
|
|
|
new Localization("en"));
|
2022-11-04 17:35:53 +00:00
|
|
|
final OffsetDateTime parsedTime = timeAgoParser.parse(time).offsetDateTime();
|
|
|
|
return DateTimeFormatter.ISO_LOCAL_DATE.format(parsedTime);
|
|
|
|
} catch (final Exception ignored) {
|
|
|
|
}
|
2020-04-16 14:08:14 +00:00
|
|
|
|
2022-11-04 17:35:53 +00:00
|
|
|
try { // Premiered Feb 21, 2020
|
|
|
|
final LocalDate localDate = LocalDate.parse(time,
|
|
|
|
DateTimeFormatter.ofPattern("MMM dd, yyyy", Locale.ENGLISH));
|
|
|
|
return DateTimeFormatter.ISO_LOCAL_DATE.format(localDate);
|
|
|
|
} catch (final Exception ignored) {
|
|
|
|
}
|
|
|
|
|
|
|
|
try { // Premiered on 21 Feb 2020
|
|
|
|
final LocalDate localDate = LocalDate.parse(time,
|
|
|
|
DateTimeFormatter.ofPattern("dd MMM yyyy", Locale.ENGLISH));
|
|
|
|
return DateTimeFormatter.ISO_LOCAL_DATE.format(localDate);
|
|
|
|
} catch (final Exception ignored) {
|
|
|
|
}
|
2020-10-26 15:32:39 +00:00
|
|
|
}
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
|
2022-11-04 17:35:53 +00:00
|
|
|
try {
|
|
|
|
// TODO: this parses English formatted dates only, we need a better approach to
|
|
|
|
// parse the textual date
|
|
|
|
final LocalDate localDate = LocalDate.parse(videoPrimaryInfoRendererDateText,
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
DateTimeFormatter.ofPattern("dd MMM yyyy", Locale.ENGLISH));
|
|
|
|
return DateTimeFormatter.ISO_LOCAL_DATE.format(localDate);
|
2022-11-04 17:35:53 +00:00
|
|
|
} catch (final Exception e) {
|
|
|
|
throw new ParsingException("Could not get upload date", e);
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
}
|
2020-04-16 14:08:14 +00:00
|
|
|
}
|
2020-02-25 20:19:53 +00:00
|
|
|
|
2022-11-04 17:35:53 +00:00
|
|
|
throw new ParsingException("Could not get upload date");
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|
|
|
|
|
2019-04-28 20:03:16 +00:00
|
|
|
@Override
|
2019-11-03 18:45:25 +00:00
|
|
|
public DateWrapper getUploadDate() throws ParsingException {
|
2019-04-28 20:03:16 +00:00
|
|
|
final String textualUploadDate = getTextualUploadDate();
|
|
|
|
|
2020-05-11 09:40:24 +00:00
|
|
|
if (isNullOrEmpty(textualUploadDate)) {
|
2019-04-28 20:03:16 +00:00
|
|
|
return null;
|
|
|
|
}
|
|
|
|
|
2019-12-16 07:35:43 +00:00
|
|
|
return new DateWrapper(YoutubeParsingHelper.parseDateFrom(textualUploadDate), true);
|
2019-04-28 20:03:16 +00:00
|
|
|
}
|
|
|
|
|
2017-11-25 00:10:04 +00:00
|
|
|
@Nonnull
|
2017-03-01 17:47:52 +00:00
|
|
|
@Override
|
2022-07-22 20:31:43 +00:00
|
|
|
public List<Image> getThumbnails() throws ParsingException {
|
2017-11-30 09:49:27 +00:00
|
|
|
assertPageFetched();
|
2017-03-01 17:47:52 +00:00
|
|
|
try {
|
2022-07-22 20:31:43 +00:00
|
|
|
return getImagesFromThumbnailsArray(playerResponse.getObject("videoDetails")
|
2022-05-01 14:38:05 +00:00
|
|
|
.getObject("thumbnail")
|
2022-07-22 20:31:43 +00:00
|
|
|
.getArray("thumbnails"));
|
2021-02-24 16:06:38 +00:00
|
|
|
} catch (final Exception e) {
|
2022-07-22 20:31:43 +00:00
|
|
|
throw new ParsingException("Could not get thumbnails");
|
2017-08-11 01:23:09 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-11-25 00:10:04 +00:00
|
|
|
@Nonnull
|
2017-08-11 01:23:09 +00:00
|
|
|
@Override
|
2021-02-12 21:22:11 +00:00
|
|
|
public Description getDescription() throws ParsingException {
|
2017-11-30 09:49:27 +00:00
|
|
|
assertPageFetched();
|
2021-05-29 12:43:26 +00:00
|
|
|
// Description with more info on links
|
[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.
The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).
As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.
URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.
As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
2023-02-20 12:21:55 +00:00
|
|
|
final String videoSecondaryInfoRendererDescription = getTextFromObject(
|
|
|
|
getVideoSecondaryInfoRenderer().getObject("description"),
|
|
|
|
true);
|
|
|
|
if (!isNullOrEmpty(videoSecondaryInfoRendererDescription)) {
|
|
|
|
return new Description(videoSecondaryInfoRendererDescription, Description.HTML);
|
|
|
|
}
|
2022-09-25 20:06:37 +00:00
|
|
|
|
2024-03-31 20:54:47 +00:00
|
|
|
final String attributedDescription = attributedDescriptionToHtml(
|
[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.
The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).
As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.
URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.
As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
2023-02-20 12:21:55 +00:00
|
|
|
getVideoSecondaryInfoRenderer().getObject("attributedDescription"));
|
|
|
|
if (!isNullOrEmpty(attributedDescription)) {
|
|
|
|
return new Description(attributedDescription, Description.HTML);
|
2020-07-26 12:15:13 +00:00
|
|
|
}
|
2020-02-25 17:27:39 +00:00
|
|
|
|
2021-04-15 16:58:59 +00:00
|
|
|
String description = playerResponse.getObject("videoDetails")
|
|
|
|
.getString("shortDescription");
|
2021-02-12 21:22:11 +00:00
|
|
|
if (description == null) {
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
final JsonObject descriptionObject = playerMicroFormatRenderer.getObject("description");
|
2021-02-12 21:22:11 +00:00
|
|
|
description = getTextFromObject(descriptionObject);
|
|
|
|
}
|
|
|
|
|
2021-05-29 12:43:26 +00:00
|
|
|
// Raw non-html description
|
2021-02-12 21:22:11 +00:00
|
|
|
return new Description(description, Description.PLAIN_TEXT);
|
2017-08-11 01:23:09 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
@Override
|
2020-10-29 17:44:05 +00:00
|
|
|
public int getAgeLimit() throws ParsingException {
|
2022-05-01 14:38:05 +00:00
|
|
|
if (ageLimit != -1) {
|
|
|
|
return ageLimit;
|
2020-10-29 17:44:05 +00:00
|
|
|
}
|
2022-05-01 14:38:05 +00:00
|
|
|
|
|
|
|
final boolean ageRestricted = getVideoSecondaryInfoRenderer()
|
|
|
|
.getObject("metadataRowContainer")
|
|
|
|
.getObject("metadataRowContainerRenderer")
|
|
|
|
.getArray("rows")
|
|
|
|
.stream()
|
|
|
|
// Only JsonObjects allowed
|
|
|
|
.filter(JsonObject.class::isInstance)
|
|
|
|
.map(JsonObject.class::cast)
|
|
|
|
.flatMap(metadataRow -> metadataRow
|
|
|
|
.getObject("metadataRowRenderer")
|
|
|
|
.getArray("contents")
|
|
|
|
.stream()
|
|
|
|
// Only JsonObjects allowed
|
|
|
|
.filter(JsonObject.class::isInstance)
|
|
|
|
.map(JsonObject.class::cast))
|
|
|
|
.flatMap(content -> content
|
|
|
|
.getArray("runs")
|
|
|
|
.stream()
|
|
|
|
// Only JsonObjects allowed
|
|
|
|
.filter(JsonObject.class::isInstance)
|
|
|
|
.map(JsonObject.class::cast))
|
2022-08-15 03:49:40 +00:00
|
|
|
.map(run -> run.getString("text", ""))
|
2022-05-01 14:38:05 +00:00
|
|
|
.anyMatch(rowText -> rowText.contains("Age-restricted"));
|
|
|
|
|
|
|
|
ageLimit = ageRestricted ? 18 : NO_AGE_LIMIT;
|
2020-02-28 16:14:26 +00:00
|
|
|
return ageLimit;
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
@Override
|
2017-08-10 17:50:59 +00:00
|
|
|
public long getLength() throws ParsingException {
|
2017-11-30 09:49:27 +00:00
|
|
|
assertPageFetched();
|
2019-07-30 18:53:23 +00:00
|
|
|
|
2017-08-10 17:50:59 +00:00
|
|
|
try {
|
2021-05-29 12:43:26 +00:00
|
|
|
final String duration = playerResponse
|
2019-07-30 18:53:23 +00:00
|
|
|
.getObject("videoDetails")
|
|
|
|
.getString("lengthSeconds");
|
|
|
|
return Long.parseLong(duration);
|
2021-02-24 16:06:38 +00:00
|
|
|
} catch (final Exception e) {
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
return getDurationFromFirstAdaptiveFormat(Arrays.asList(
|
|
|
|
html5StreamingData, androidStreamingData, iosStreamingData));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
private int getDurationFromFirstAdaptiveFormat(@Nonnull final List<JsonObject> streamingDatas)
|
|
|
|
throws ParsingException {
|
|
|
|
for (final JsonObject streamingData : streamingDatas) {
|
|
|
|
final JsonArray adaptiveFormats = streamingData.getArray(ADAPTIVE_FORMATS);
|
|
|
|
if (adaptiveFormats.isEmpty()) {
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
final String durationMs = adaptiveFormats.getObject(0)
|
|
|
|
.getString("approxDurationMs");
|
|
|
|
try {
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
return Math.round(Long.parseLong(durationMs) / 1000f);
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
} catch (final NumberFormatException ignored) {
|
2020-01-06 19:45:57 +00:00
|
|
|
}
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
|
|
|
|
throw new ParsingException("Could not get duration");
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|
|
|
|
|
2017-08-11 01:23:09 +00:00
|
|
|
/**
|
|
|
|
* Attempts to parse (and return) the offset to start playing the video from.
|
|
|
|
*
|
|
|
|
* @return the offset (in seconds), or 0 if no timestamp is found.
|
|
|
|
*/
|
|
|
|
@Override
|
|
|
|
public long getTimeStamp() throws ParsingException {
|
2020-04-11 15:18:17 +00:00
|
|
|
final long timestamp =
|
2022-01-12 09:45:06 +00:00
|
|
|
getTimestampSeconds("((#|&|\\?)t=\\d*h?\\d*m?\\d+s?)");
|
2020-04-09 12:45:33 +00:00
|
|
|
|
|
|
|
if (timestamp == -2) {
|
2021-05-29 12:43:26 +00:00
|
|
|
// Regex for timestamp was not found
|
2020-04-09 12:45:33 +00:00
|
|
|
return 0;
|
|
|
|
}
|
2022-05-01 14:38:05 +00:00
|
|
|
return timestamp;
|
2017-08-11 01:23:09 +00:00
|
|
|
}
|
|
|
|
|
2017-03-01 17:47:52 +00:00
|
|
|
@Override
|
|
|
|
public long getViewCount() throws ParsingException {
|
[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.
The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).
As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.
URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.
As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
2023-02-20 12:21:55 +00:00
|
|
|
String views = getTextFromObject(getVideoPrimaryInfoRenderer().getObject("viewCount")
|
|
|
|
.getObject("videoViewCountRenderer").getObject("viewCount"));
|
2021-06-02 16:46:35 +00:00
|
|
|
|
|
|
|
if (isNullOrEmpty(views)) {
|
|
|
|
views = playerResponse.getObject("videoDetails").getString("viewCount");
|
|
|
|
|
2022-03-18 14:09:06 +00:00
|
|
|
if (isNullOrEmpty(views)) {
|
|
|
|
throw new ParsingException("Could not get view count");
|
|
|
|
}
|
2021-06-02 16:46:35 +00:00
|
|
|
}
|
|
|
|
|
2022-03-18 14:09:06 +00:00
|
|
|
if (views.toLowerCase().contains("no views")) {
|
|
|
|
return 0;
|
|
|
|
}
|
2020-03-17 14:00:07 +00:00
|
|
|
|
2020-02-29 16:18:50 +00:00
|
|
|
return Long.parseLong(Utils.removeNonDigitCharacters(views));
|
2020-02-24 15:04:01 +00:00
|
|
|
}
|
|
|
|
|
2017-03-01 17:47:52 +00:00
|
|
|
@Override
|
2017-08-11 01:23:09 +00:00
|
|
|
public long getLikeCount() throws ParsingException {
|
2017-11-30 09:49:27 +00:00
|
|
|
assertPageFetched();
|
2022-09-10 14:38:25 +00:00
|
|
|
|
|
|
|
// If ratings are not allowed, there is no like count available
|
|
|
|
if (!playerResponse.getObject("videoDetails").getBoolean("allowRatings")) {
|
2023-10-28 16:26:52 +00:00
|
|
|
return -1L;
|
2022-09-10 14:38:25 +00:00
|
|
|
}
|
|
|
|
|
2023-10-28 16:26:52 +00:00
|
|
|
final JsonArray topLevelButtons = getVideoPrimaryInfoRenderer()
|
|
|
|
.getObject("videoActions")
|
|
|
|
.getObject("menuRenderer")
|
|
|
|
.getArray("topLevelButtons");
|
2022-09-10 14:38:25 +00:00
|
|
|
|
2017-03-01 17:47:52 +00:00
|
|
|
try {
|
2023-10-28 16:26:52 +00:00
|
|
|
return parseLikeCountFromLikeButtonViewModel(topLevelButtons);
|
|
|
|
} catch (final ParsingException ignored) {
|
|
|
|
// A segmentedLikeDislikeButtonRenderer could be returned instead of a
|
|
|
|
// segmentedLikeDislikeButtonViewModel, so ignore extraction errors relative to
|
|
|
|
// segmentedLikeDislikeButtonViewModel object
|
|
|
|
}
|
|
|
|
|
|
|
|
try {
|
|
|
|
return parseLikeCountFromLikeButtonRenderer(topLevelButtons);
|
|
|
|
} catch (final ParsingException e) {
|
|
|
|
throw new ParsingException("Could not get like count", e);
|
|
|
|
}
|
|
|
|
}
|
2022-09-10 14:38:25 +00:00
|
|
|
|
2023-10-28 16:26:52 +00:00
|
|
|
private static long parseLikeCountFromLikeButtonRenderer(
|
|
|
|
@Nonnull final JsonArray topLevelButtons) throws ParsingException {
|
|
|
|
String likesString = null;
|
|
|
|
final JsonObject likeToggleButtonRenderer = topLevelButtons.stream()
|
|
|
|
.filter(JsonObject.class::isInstance)
|
|
|
|
.map(JsonObject.class::cast)
|
|
|
|
.map(button -> button.getObject("segmentedLikeDislikeButtonRenderer")
|
|
|
|
.getObject("likeButton")
|
|
|
|
.getObject("toggleButtonRenderer"))
|
|
|
|
.filter(toggleButtonRenderer -> !isNullOrEmpty(toggleButtonRenderer))
|
|
|
|
.findFirst()
|
|
|
|
.orElse(null);
|
|
|
|
|
|
|
|
if (likeToggleButtonRenderer != null) {
|
2022-09-10 14:38:25 +00:00
|
|
|
// Use one of the accessibility strings available (this one has the same path as the
|
|
|
|
// one used for comments' like count extraction)
|
|
|
|
likesString = likeToggleButtonRenderer.getObject("accessibilityData")
|
2021-11-30 18:52:51 +00:00
|
|
|
.getObject("accessibilityData")
|
|
|
|
.getString("label");
|
2021-11-19 20:36:03 +00:00
|
|
|
|
2022-09-10 14:38:25 +00:00
|
|
|
// Use the other accessibility string available which contains the exact like count
|
2021-11-19 20:36:03 +00:00
|
|
|
if (likesString == null) {
|
2022-09-10 14:38:25 +00:00
|
|
|
likesString = likeToggleButtonRenderer.getObject("accessibility")
|
|
|
|
.getString("label");
|
2017-08-11 01:23:09 +00:00
|
|
|
}
|
2021-11-19 20:36:03 +00:00
|
|
|
|
2022-09-10 14:38:25 +00:00
|
|
|
// Last method: use the defaultText's accessibility data, which contains the exact like
|
|
|
|
// count too, except when it is equal to 0, where a localized string is returned instead
|
|
|
|
if (likesString == null) {
|
|
|
|
likesString = likeToggleButtonRenderer.getObject("defaultText")
|
|
|
|
.getObject("accessibility")
|
|
|
|
.getObject("accessibilityData")
|
|
|
|
.getString("label");
|
|
|
|
}
|
|
|
|
|
|
|
|
// This check only works with English localizations!
|
2023-10-28 16:26:52 +00:00
|
|
|
if (likesString != null && likesString.toLowerCase().contains("no likes")) {
|
2021-12-27 15:50:08 +00:00
|
|
|
return 0;
|
|
|
|
}
|
2023-10-28 16:26:52 +00:00
|
|
|
}
|
2021-12-27 15:50:08 +00:00
|
|
|
|
2023-10-28 16:26:52 +00:00
|
|
|
// If ratings are allowed and the likes string is null, it means that we couldn't extract
|
|
|
|
// the full like count from accessibility data
|
|
|
|
if (likesString == null) {
|
|
|
|
throw new ParsingException("Could not get like count from accessibility data");
|
|
|
|
}
|
|
|
|
|
|
|
|
try {
|
|
|
|
return Long.parseLong(Utils.removeNonDigitCharacters(likesString));
|
|
|
|
} catch (final NumberFormatException e) {
|
|
|
|
throw new ParsingException("Could not parse \"" + likesString + "\" as a long", e);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
private static long parseLikeCountFromLikeButtonViewModel(
|
|
|
|
@Nonnull final JsonArray topLevelButtons) throws ParsingException {
|
|
|
|
// Try first with the current video actions buttons data structure
|
|
|
|
final JsonObject likeToggleButtonViewModel = topLevelButtons.stream()
|
|
|
|
.filter(JsonObject.class::isInstance)
|
|
|
|
.map(JsonObject.class::cast)
|
|
|
|
.map(button -> button.getObject("segmentedLikeDislikeButtonViewModel")
|
|
|
|
.getObject("likeButtonViewModel")
|
|
|
|
.getObject("likeButtonViewModel")
|
|
|
|
.getObject("toggleButtonViewModel")
|
|
|
|
.getObject("toggleButtonViewModel")
|
|
|
|
.getObject("defaultButtonViewModel")
|
|
|
|
.getObject("buttonViewModel"))
|
|
|
|
.filter(buttonViewModel -> !isNullOrEmpty(buttonViewModel))
|
|
|
|
.findFirst()
|
|
|
|
.orElse(null);
|
|
|
|
|
|
|
|
if (likeToggleButtonViewModel == null) {
|
|
|
|
throw new ParsingException("Could not find buttonViewModel object");
|
|
|
|
}
|
|
|
|
|
|
|
|
final String accessibilityText = likeToggleButtonViewModel.getString("accessibilityText");
|
|
|
|
if (accessibilityText == null) {
|
|
|
|
throw new ParsingException("Could not find buttonViewModel's accessibilityText string");
|
|
|
|
}
|
|
|
|
|
|
|
|
// The like count is always returned as a number in this element, even for videos with no
|
|
|
|
// likes
|
|
|
|
try {
|
|
|
|
return Long.parseLong(Utils.removeNonDigitCharacters(accessibilityText));
|
|
|
|
} catch (final NumberFormatException e) {
|
|
|
|
throw new ParsingException(
|
|
|
|
"Could not parse \"" + accessibilityText + "\" as a long", e);
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-11-25 00:10:04 +00:00
|
|
|
@Nonnull
|
2017-08-11 01:23:09 +00:00
|
|
|
@Override
|
|
|
|
public String getUploaderUrl() throws ParsingException {
|
2017-11-30 09:49:27 +00:00
|
|
|
assertPageFetched();
|
2020-04-16 14:08:14 +00:00
|
|
|
|
2021-05-29 12:43:26 +00:00
|
|
|
// Don't use the id in the videoSecondaryRenderer object to get real id of the uploader
|
|
|
|
// The difference between the real id of the channel and the displayed id is especially
|
|
|
|
// visible for music channels and autogenerated channels.
|
2021-02-24 16:06:38 +00:00
|
|
|
final String uploaderId = playerResponse.getObject("videoDetails").getString("channelId");
|
2020-07-18 07:50:22 +00:00
|
|
|
if (!isNullOrEmpty(uploaderId)) {
|
2020-04-16 14:08:14 +00:00
|
|
|
return YoutubeChannelLinkHandlerFactory.getInstance().getUrl("channel/" + uploaderId);
|
2020-07-18 07:50:22 +00:00
|
|
|
}
|
2020-04-16 14:08:14 +00:00
|
|
|
|
2020-02-25 09:05:53 +00:00
|
|
|
throw new ParsingException("Could not get uploader url");
|
2017-11-25 00:10:04 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
@Nonnull
|
2017-08-11 01:23:09 +00:00
|
|
|
@Override
|
|
|
|
public String getUploaderName() throws ParsingException {
|
2017-11-30 09:49:27 +00:00
|
|
|
assertPageFetched();
|
2020-07-18 07:50:22 +00:00
|
|
|
|
2021-04-15 16:58:59 +00:00
|
|
|
// Don't use the name in the videoSecondaryRenderer object to get real name of the uploader
|
|
|
|
// The difference between the real name of the channel and the displayed name is especially
|
|
|
|
// visible for music channels and autogenerated channels.
|
|
|
|
final String uploaderName = playerResponse.getObject("videoDetails").getString("author");
|
2022-03-18 14:09:06 +00:00
|
|
|
if (isNullOrEmpty(uploaderName)) {
|
|
|
|
throw new ParsingException("Could not get uploader name");
|
|
|
|
}
|
2020-02-29 16:18:50 +00:00
|
|
|
|
|
|
|
return uploaderName;
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|
|
|
|
|
2021-01-22 00:44:58 +00:00
|
|
|
@Override
|
|
|
|
public boolean isUploaderVerified() throws ParsingException {
|
2022-05-01 14:38:05 +00:00
|
|
|
return YoutubeParsingHelper.isVerified(
|
|
|
|
getVideoSecondaryInfoRenderer()
|
|
|
|
.getObject("owner")
|
|
|
|
.getObject("videoOwnerRenderer")
|
|
|
|
.getArray("badges"));
|
2021-01-22 00:44:58 +00:00
|
|
|
}
|
|
|
|
|
2017-11-25 00:10:04 +00:00
|
|
|
@Nonnull
|
2017-03-01 17:47:52 +00:00
|
|
|
@Override
|
2022-07-22 20:31:43 +00:00
|
|
|
public List<Image> getUploaderAvatars() throws ParsingException {
|
2017-11-30 09:49:27 +00:00
|
|
|
assertPageFetched();
|
2020-07-18 07:50:22 +00:00
|
|
|
|
2022-07-22 20:31:43 +00:00
|
|
|
final List<Image> imageList = getImagesFromThumbnailsArray(
|
|
|
|
getVideoSecondaryInfoRenderer().getObject("owner")
|
|
|
|
.getObject("videoOwnerRenderer")
|
|
|
|
.getObject("thumbnail")
|
|
|
|
.getArray("thumbnails"));
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
|
2022-07-22 20:31:43 +00:00
|
|
|
if (imageList.isEmpty() && ageLimit == NO_AGE_LIMIT) {
|
|
|
|
throw new ParsingException("Could not get uploader avatars");
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|
2020-07-18 07:50:22 +00:00
|
|
|
|
2022-07-22 20:31:43 +00:00
|
|
|
return imageList;
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|
|
|
|
|
2022-02-11 04:08:22 +00:00
|
|
|
@Override
|
|
|
|
public long getUploaderSubscriberCount() throws ParsingException {
|
2022-03-18 14:09:06 +00:00
|
|
|
final JsonObject videoOwnerRenderer = JsonUtils.getObject(videoSecondaryInfoRenderer,
|
|
|
|
"owner.videoOwnerRenderer");
|
2022-02-26 17:12:51 +00:00
|
|
|
if (!videoOwnerRenderer.has("subscriberCountText")) {
|
2022-02-12 18:00:54 +00:00
|
|
|
return UNKNOWN_SUBSCRIBER_COUNT;
|
2022-02-11 04:08:22 +00:00
|
|
|
}
|
2022-02-26 17:12:51 +00:00
|
|
|
try {
|
2022-03-18 14:09:06 +00:00
|
|
|
return Utils.mixedNumberWordToLong(getTextFromObject(videoOwnerRenderer
|
|
|
|
.getObject("subscriberCountText")));
|
2022-02-26 17:12:51 +00:00
|
|
|
} catch (final NumberFormatException e) {
|
|
|
|
throw new ParsingException("Could not get uploader subscriber count", e);
|
|
|
|
}
|
2022-02-11 04:08:22 +00:00
|
|
|
}
|
|
|
|
|
2018-02-25 21:03:32 +00:00
|
|
|
@Nonnull
|
2017-03-01 17:47:52 +00:00
|
|
|
@Override
|
|
|
|
public String getDashMpdUrl() throws ParsingException {
|
2017-11-30 09:49:27 +00:00
|
|
|
assertPageFetched();
|
2021-06-24 16:39:16 +00:00
|
|
|
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
// There is no DASH manifest available in the iOS clients and the DASH manifest of the
|
|
|
|
// Android client doesn't contain all available streams (mainly the WEBM ones)
|
2022-05-01 14:38:05 +00:00
|
|
|
return getManifestUrl(
|
|
|
|
"dash",
|
|
|
|
Arrays.asList(html5StreamingData, androidStreamingData));
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|
|
|
|
|
2018-02-25 21:03:32 +00:00
|
|
|
@Nonnull
|
|
|
|
@Override
|
|
|
|
public String getHlsUrl() throws ParsingException {
|
|
|
|
assertPageFetched();
|
2019-01-19 12:50:02 +00:00
|
|
|
|
2022-01-15 16:25:00 +00:00
|
|
|
// Return HLS manifest of the iOS client first because on livestreams, the HLS manifest
|
|
|
|
// returned has separated audio and video streams
|
|
|
|
// Also, on videos, non-iOS clients don't have an HLS manifest URL in their player response
|
2022-05-01 14:38:05 +00:00
|
|
|
return getManifestUrl(
|
|
|
|
"hls",
|
|
|
|
Arrays.asList(iosStreamingData, html5StreamingData, androidStreamingData));
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
@Nonnull
|
|
|
|
private static String getManifestUrl(@Nonnull final String manifestType,
|
|
|
|
@Nonnull final List<JsonObject> streamingDataObjects) {
|
|
|
|
final String manifestKey = manifestType + "ManifestUrl";
|
|
|
|
|
2022-05-01 14:38:05 +00:00
|
|
|
return streamingDataObjects.stream()
|
|
|
|
.filter(Objects::nonNull)
|
|
|
|
.map(streamingDataObject -> streamingDataObject.getString(manifestKey))
|
|
|
|
.filter(Objects::nonNull)
|
|
|
|
.findFirst()
|
2022-08-15 03:49:40 +00:00
|
|
|
.orElse("");
|
2018-02-25 21:03:32 +00:00
|
|
|
}
|
|
|
|
|
2017-03-01 17:47:52 +00:00
|
|
|
@Override
|
2022-05-01 14:38:05 +00:00
|
|
|
public List<AudioStream> getAudioStreams() throws ExtractionException {
|
2017-11-30 09:49:27 +00:00
|
|
|
assertPageFetched();
|
2022-03-06 19:10:11 +00:00
|
|
|
return getItags(ADAPTIVE_FORMATS, ItagItem.ItagType.AUDIO,
|
|
|
|
getAudioStreamBuilderHelper(), "audio");
|
2022-05-01 14:38:05 +00:00
|
|
|
}
|
2017-03-01 17:47:52 +00:00
|
|
|
|
2022-05-01 14:38:05 +00:00
|
|
|
@Override
|
|
|
|
public List<VideoStream> getVideoStreams() throws ExtractionException {
|
|
|
|
assertPageFetched();
|
2022-03-06 19:10:11 +00:00
|
|
|
return getItags(FORMATS, ItagItem.ItagType.VIDEO,
|
|
|
|
getVideoStreamBuilderHelper(false), "video");
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
@Override
|
2018-08-21 15:23:56 +00:00
|
|
|
public List<VideoStream> getVideoOnlyStreams() throws ExtractionException {
|
2017-11-30 09:49:27 +00:00
|
|
|
assertPageFetched();
|
2022-03-06 19:10:11 +00:00
|
|
|
return getItags(ADAPTIVE_FORMATS, ItagItem.ItagType.VIDEO_ONLY,
|
|
|
|
getVideoStreamBuilderHelper(true), "video-only");
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|
|
|
|
|
2022-02-01 18:52:41 +00:00
|
|
|
/**
|
[YouTube] Refactor JavaScript player management API
This commit is introducing breaking changes.
For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.
The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.
YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.
If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.
All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.
Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-11 21:16:11 +00:00
|
|
|
* Try to deobfuscate a streaming URL and fall back to the given URL, because decryption may
|
|
|
|
* fail if YouTube changes break something.
|
2022-08-12 14:17:13 +00:00
|
|
|
*
|
|
|
|
* <p>
|
2022-02-01 18:52:41 +00:00
|
|
|
* This way a breaking change from YouTube does not result in a broken extractor.
|
2022-08-12 14:17:13 +00:00
|
|
|
* </p>
|
|
|
|
*
|
[YouTube] Refactor JavaScript player management API
This commit is introducing breaking changes.
For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.
The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.
YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.
If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.
All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.
Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-11 21:16:11 +00:00
|
|
|
* @param streamingUrl the streaming URL to which deobfuscating its throttling parameter if
|
|
|
|
* there is one
|
2022-08-12 14:17:13 +00:00
|
|
|
* @param videoId the video ID to use when extracting JavaScript player code, if needed
|
2022-02-01 18:52:41 +00:00
|
|
|
*/
|
[YouTube] Refactor JavaScript player management API
This commit is introducing breaking changes.
For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.
The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.
YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.
If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.
All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.
Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-11 21:16:11 +00:00
|
|
|
private String tryDeobfuscateThrottlingParameterOfUrl(@Nonnull final String streamingUrl,
|
|
|
|
@Nonnull final String videoId) {
|
2022-02-01 18:52:41 +00:00
|
|
|
try {
|
[YouTube] Refactor JavaScript player management API
This commit is introducing breaking changes.
For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.
The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.
YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.
If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.
All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.
Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-11 21:16:11 +00:00
|
|
|
return YoutubeJavaScriptPlayerManager.getUrlWithThrottlingParameterDeobfuscated(
|
|
|
|
videoId, streamingUrl);
|
2022-03-15 16:10:05 +00:00
|
|
|
} catch (final ParsingException e) {
|
2022-08-12 14:17:13 +00:00
|
|
|
return streamingUrl;
|
2022-02-01 18:52:41 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-11-22 17:39:38 +00:00
|
|
|
@Override
|
2018-02-01 21:27:14 +00:00
|
|
|
@Nonnull
|
2020-10-29 17:44:05 +00:00
|
|
|
public List<SubtitlesStream> getSubtitlesDefault() throws ParsingException {
|
2018-09-24 19:04:22 +00:00
|
|
|
return getSubtitles(MediaFormat.TTML);
|
2017-11-23 15:33:03 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
@Override
|
2018-02-01 21:27:14 +00:00
|
|
|
@Nonnull
|
2020-10-29 17:44:05 +00:00
|
|
|
public List<SubtitlesStream> getSubtitles(final MediaFormat format) throws ParsingException {
|
2017-11-30 09:49:27 +00:00
|
|
|
assertPageFetched();
|
2020-10-29 17:44:05 +00:00
|
|
|
|
2022-03-06 19:10:11 +00:00
|
|
|
// We cannot store the subtitles list because the media format may change
|
|
|
|
final List<SubtitlesStream> subtitlesToReturn = new ArrayList<>();
|
2020-10-29 17:44:05 +00:00
|
|
|
final JsonObject renderer = playerResponse.getObject("captions")
|
|
|
|
.getObject("playerCaptionsTracklistRenderer");
|
|
|
|
final JsonArray captionsArray = renderer.getArray("captionTracks");
|
|
|
|
// TODO: use this to apply auto translation to different language from a source language
|
|
|
|
// final JsonArray autoCaptionsArray = renderer.getArray("translationLanguages");
|
|
|
|
|
|
|
|
for (int i = 0; i < captionsArray.size(); i++) {
|
|
|
|
final String languageCode = captionsArray.getObject(i).getString("languageCode");
|
|
|
|
final String baseUrl = captionsArray.getObject(i).getString("baseUrl");
|
|
|
|
final String vssId = captionsArray.getObject(i).getString("vssId");
|
|
|
|
|
|
|
|
if (languageCode != null && baseUrl != null && vssId != null) {
|
|
|
|
final boolean isAutoGenerated = vssId.startsWith("a.");
|
|
|
|
final String cleanUrl = baseUrl
|
2022-03-06 19:10:11 +00:00
|
|
|
// Remove preexisting format if exists
|
|
|
|
.replaceAll("&fmt=[^&]*", "")
|
|
|
|
// Remove translation language
|
|
|
|
.replaceAll("&tlang=[^&]*", "");
|
|
|
|
|
|
|
|
subtitlesToReturn.add(new SubtitlesStream.Builder()
|
|
|
|
.setContent(cleanUrl + "&fmt=" + format.getSuffix(), true)
|
|
|
|
.setMediaFormat(format)
|
|
|
|
.setLanguageCode(languageCode)
|
|
|
|
.setAutoGenerated(isAutoGenerated)
|
|
|
|
.build());
|
2020-10-29 17:44:05 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2022-03-06 19:10:11 +00:00
|
|
|
return subtitlesToReturn;
|
2017-11-22 17:39:38 +00:00
|
|
|
}
|
|
|
|
|
2017-03-01 17:47:52 +00:00
|
|
|
@Override
|
2020-11-04 13:50:35 +00:00
|
|
|
public StreamType getStreamType() {
|
2018-02-25 22:31:42 +00:00
|
|
|
assertPageFetched();
|
2021-06-24 16:39:16 +00:00
|
|
|
|
2022-01-15 16:25:00 +00:00
|
|
|
return streamType;
|
|
|
|
}
|
|
|
|
|
|
|
|
private void setStreamType() {
|
2022-03-15 10:19:13 +00:00
|
|
|
if (playerResponse.getObject("playabilityStatus").has("liveStreamability")) {
|
2022-01-15 16:25:00 +00:00
|
|
|
streamType = StreamType.LIVE_STREAM;
|
2022-03-15 10:19:13 +00:00
|
|
|
} else if (playerResponse.getObject("videoDetails").getBoolean("isPostLiveDvr", false)) {
|
|
|
|
streamType = StreamType.POST_LIVE_STREAM;
|
2022-01-15 16:25:00 +00:00
|
|
|
} else {
|
|
|
|
streamType = StreamType.VIDEO_STREAM;
|
2021-06-24 16:39:16 +00:00
|
|
|
}
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|
|
|
|
|
2020-10-24 19:17:58 +00:00
|
|
|
@Nullable
|
2017-03-01 17:47:52 +00:00
|
|
|
@Override
|
2022-02-02 19:23:11 +00:00
|
|
|
public MultiInfoItemsCollector getRelatedItems() throws ExtractionException {
|
2017-11-30 09:49:27 +00:00
|
|
|
assertPageFetched();
|
2020-02-28 15:40:50 +00:00
|
|
|
|
2020-10-24 19:17:58 +00:00
|
|
|
if (getAgeLimit() != NO_AGE_LIMIT) {
|
|
|
|
return null;
|
|
|
|
}
|
2020-02-28 15:40:50 +00:00
|
|
|
|
2017-03-01 17:47:52 +00:00
|
|
|
try {
|
2022-02-02 19:23:11 +00:00
|
|
|
final MultiInfoItemsCollector collector = new MultiInfoItemsCollector(getServiceId());
|
2020-05-30 08:25:43 +00:00
|
|
|
|
2022-05-01 14:38:05 +00:00
|
|
|
final JsonArray results = nextResponse
|
|
|
|
.getObject("contents")
|
|
|
|
.getObject("twoColumnWatchNextResults")
|
|
|
|
.getObject("secondaryResults")
|
|
|
|
.getObject("secondaryResults")
|
|
|
|
.getArray("results");
|
2020-02-18 16:04:22 +00:00
|
|
|
|
2019-04-28 20:03:16 +00:00
|
|
|
final TimeAgoParser timeAgoParser = getTimeAgoParser();
|
2022-05-01 14:38:05 +00:00
|
|
|
results.stream()
|
|
|
|
.filter(JsonObject.class::isInstance)
|
|
|
|
.map(JsonObject.class::cast)
|
|
|
|
.map(result -> {
|
|
|
|
if (result.has("compactVideoRenderer")) {
|
|
|
|
return new YoutubeStreamInfoItemExtractor(
|
|
|
|
result.getObject("compactVideoRenderer"), timeAgoParser);
|
|
|
|
} else if (result.has("compactRadioRenderer")) {
|
|
|
|
return new YoutubeMixOrPlaylistInfoItemExtractor(
|
|
|
|
result.getObject("compactRadioRenderer"));
|
|
|
|
} else if (result.has("compactPlaylistRenderer")) {
|
|
|
|
return new YoutubeMixOrPlaylistInfoItemExtractor(
|
|
|
|
result.getObject("compactPlaylistRenderer"));
|
|
|
|
}
|
|
|
|
return null;
|
|
|
|
})
|
|
|
|
.filter(Objects::nonNull)
|
|
|
|
.forEach(collector::commit);
|
2019-04-28 20:03:16 +00:00
|
|
|
|
2017-03-01 17:47:52 +00:00
|
|
|
return collector;
|
2021-02-24 16:06:38 +00:00
|
|
|
} catch (final Exception e) {
|
2017-03-01 17:47:52 +00:00
|
|
|
throw new ParsingException("Could not get related videos", e);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-06-29 18:12:55 +00:00
|
|
|
/**
|
2017-07-11 03:08:03 +00:00
|
|
|
* {@inheritDoc}
|
2017-06-29 18:12:55 +00:00
|
|
|
*/
|
2017-07-11 03:08:03 +00:00
|
|
|
@Override
|
|
|
|
public String getErrorMessage() {
|
2020-02-29 21:57:25 +00:00
|
|
|
try {
|
2021-04-15 16:58:59 +00:00
|
|
|
return getTextFromObject(playerResponse.getObject("playabilityStatus")
|
|
|
|
.getObject("errorScreen").getObject("playerErrorMessageRenderer")
|
|
|
|
.getObject("reason"));
|
[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.
The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).
As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.
URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.
As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
2023-02-20 12:21:55 +00:00
|
|
|
} catch (final NullPointerException e) {
|
2021-05-29 12:43:26 +00:00
|
|
|
return null; // No error message
|
2020-02-29 21:57:25 +00:00
|
|
|
}
|
2017-07-11 03:08:03 +00:00
|
|
|
}
|
2017-03-01 17:47:52 +00:00
|
|
|
|
2017-07-11 03:08:03 +00:00
|
|
|
/*//////////////////////////////////////////////////////////////////////////
|
2017-08-10 17:50:59 +00:00
|
|
|
// Fetch page
|
2017-07-11 03:08:03 +00:00
|
|
|
//////////////////////////////////////////////////////////////////////////*/
|
2017-03-01 17:47:52 +00:00
|
|
|
|
2019-09-11 18:04:28 +00:00
|
|
|
private static final String FORMATS = "formats";
|
|
|
|
private static final String ADAPTIVE_FORMATS = "adaptiveFormats";
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
private static final String STREAMING_DATA = "streamingData";
|
|
|
|
private static final String PLAYER = "player";
|
|
|
|
private static final String NEXT = "next";
|
2022-03-06 19:10:11 +00:00
|
|
|
private static final String SIGNATURE_CIPHER = "signatureCipher";
|
|
|
|
private static final String CIPHER = "cipher";
|
2017-07-11 03:08:03 +00:00
|
|
|
|
2017-08-06 20:20:15 +00:00
|
|
|
@Override
|
2020-10-29 17:44:05 +00:00
|
|
|
public void onFetchPage(@Nonnull final Downloader downloader)
|
|
|
|
throws IOException, ExtractionException {
|
2021-05-29 12:43:26 +00:00
|
|
|
final String videoId = getId();
|
2023-07-22 18:08:22 +00:00
|
|
|
|
2021-05-23 15:55:19 +00:00
|
|
|
final Localization localization = getExtractorLocalization();
|
|
|
|
final ContentCountry contentCountry = getExtractorContentCountry();
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
html5Cpn = generateContentPlaybackNonce();
|
2020-02-28 14:17:47 +00:00
|
|
|
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
playerResponse = getJsonPostResponse(PLAYER,
|
[YouTube] Refactor JavaScript player management API
This commit is introducing breaking changes.
For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.
The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.
YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.
If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.
All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.
Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-11 21:16:11 +00:00
|
|
|
createDesktopPlayerBody(
|
|
|
|
localization,
|
|
|
|
contentCountry,
|
|
|
|
videoId,
|
|
|
|
YoutubeJavaScriptPlayerManager.getSignatureTimestamp(videoId),
|
|
|
|
false,
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
html5Cpn),
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
localization);
|
2021-05-29 12:43:26 +00:00
|
|
|
|
|
|
|
// Save the playerResponse from the player endpoint of the desktop internal API because
|
|
|
|
// there can be restrictions on the embedded player.
|
|
|
|
// E.g. if a video is age-restricted, the embedded player's playabilityStatus says that
|
|
|
|
// the video cannot be played outside of YouTube, but does not show the original message.
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
final JsonObject youtubePlayerResponse = playerResponse;
|
2021-02-24 11:06:19 +00:00
|
|
|
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
if (playerResponse == null) {
|
2021-02-24 13:13:12 +00:00
|
|
|
throw new ExtractionException("Could not get playerResponse");
|
|
|
|
}
|
|
|
|
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
final JsonObject playabilityStatus = playerResponse.getObject("playabilityStatus");
|
2021-05-29 12:43:26 +00:00
|
|
|
|
2022-08-15 03:49:40 +00:00
|
|
|
final boolean isAgeRestricted = playabilityStatus.getString("reason", "")
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
.contains("age");
|
2021-05-29 12:43:26 +00:00
|
|
|
|
2022-01-15 16:25:00 +00:00
|
|
|
setStreamType();
|
|
|
|
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
if (!playerResponse.has(STREAMING_DATA)) {
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
try {
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
fetchTvHtml5EmbedJsonPlayer(contentCountry, localization, videoId);
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
} catch (final Exception ignored) {
|
|
|
|
}
|
|
|
|
}
|
2021-05-29 12:43:26 +00:00
|
|
|
|
2022-03-06 19:10:11 +00:00
|
|
|
// Refresh the stream type because the stream type may be not properly known for
|
|
|
|
// age-restricted videos
|
|
|
|
setStreamType();
|
|
|
|
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
if (html5StreamingData == null && playerResponse.has(STREAMING_DATA)) {
|
|
|
|
html5StreamingData = playerResponse.getObject(STREAMING_DATA);
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
}
|
|
|
|
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
if (html5StreamingData == null) {
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
checkPlayabilityStatus(youtubePlayerResponse, playabilityStatus);
|
|
|
|
}
|
2021-07-05 17:21:54 +00:00
|
|
|
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
// The microformat JSON object of the content is not returned on the client we use to
|
|
|
|
// try to get streams of unavailable contents but is still returned on the WEB client,
|
|
|
|
// so we need to store it instead of getting it directly from the playerResponse
|
|
|
|
playerMicroFormatRenderer = youtubePlayerResponse.getObject("microformat")
|
|
|
|
.getObject("playerMicroformatRenderer");
|
|
|
|
|
2023-06-18 19:54:52 +00:00
|
|
|
if (isPlayerResponseNotValid(playerResponse, videoId)) {
|
|
|
|
throw new ExtractionException("Initial player response is not valid");
|
|
|
|
}
|
|
|
|
|
2022-05-01 14:38:05 +00:00
|
|
|
final byte[] body = JsonWriter.string(
|
|
|
|
prepareDesktopJsonBuilder(localization, contentCountry)
|
|
|
|
.value(VIDEO_ID, videoId)
|
|
|
|
.value(CONTENT_CHECK_OK, true)
|
|
|
|
.value(RACY_CHECK_OK, true)
|
|
|
|
.done())
|
|
|
|
.getBytes(StandardCharsets.UTF_8);
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
nextResponse = getJsonPostResponse(NEXT, body, localization);
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
|
2022-06-20 20:04:09 +00:00
|
|
|
// streamType can only have LIVE_STREAM, POST_LIVE_STREAM and VIDEO_STREAM values (see
|
|
|
|
// setStreamType()), so this block will be run only for POST_LIVE_STREAM and VIDEO_STREAM
|
|
|
|
// values if fetching of the ANDROID client is not forced
|
|
|
|
if ((!isAgeRestricted && streamType != StreamType.LIVE_STREAM)
|
2022-01-15 16:25:00 +00:00
|
|
|
|| isAndroidClientFetchForced) {
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
try {
|
|
|
|
fetchAndroidMobileJsonPlayer(contentCountry, localization, videoId);
|
|
|
|
} catch (final Exception ignored) {
|
2022-06-20 20:04:09 +00:00
|
|
|
// Ignore exceptions related to ANDROID client fetch or parsing, as it is not
|
|
|
|
// compulsory to play contents
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
}
|
|
|
|
}
|
2022-01-15 16:25:00 +00:00
|
|
|
|
2022-03-06 19:10:11 +00:00
|
|
|
if ((!isAgeRestricted && streamType == StreamType.LIVE_STREAM)
|
2022-01-15 16:25:00 +00:00
|
|
|
|| isIosClientFetchForced) {
|
|
|
|
try {
|
|
|
|
fetchIosMobileJsonPlayer(contentCountry, localization, videoId);
|
|
|
|
} catch (final Exception ignored) {
|
2022-06-20 20:04:09 +00:00
|
|
|
// Ignore exceptions related to IOS client fetch or parsing, as it is not
|
|
|
|
// compulsory to play contents
|
2022-01-15 16:25:00 +00:00
|
|
|
}
|
|
|
|
}
|
2021-05-29 12:43:26 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
private void checkPlayabilityStatus(final JsonObject youtubePlayerResponse,
|
2022-03-18 14:09:06 +00:00
|
|
|
@Nonnull final JsonObject playabilityStatus)
|
2021-06-06 16:32:07 +00:00
|
|
|
throws ParsingException {
|
2021-02-24 11:06:19 +00:00
|
|
|
String status = playabilityStatus.getString("status");
|
2022-05-01 14:38:05 +00:00
|
|
|
if (status == null || status.equalsIgnoreCase("ok")) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2021-02-14 14:22:45 +00:00
|
|
|
// If status exist, and is not "OK", throw the specific exception based on error message
|
|
|
|
// or a ContentNotAvailableException with the reason text if it's an unknown reason.
|
2022-05-01 14:38:05 +00:00
|
|
|
final JsonObject newPlayabilityStatus =
|
|
|
|
youtubePlayerResponse.getObject("playabilityStatus");
|
|
|
|
status = newPlayabilityStatus.getString("status");
|
|
|
|
final String reason = newPlayabilityStatus.getString("reason");
|
|
|
|
|
|
|
|
if (status.equalsIgnoreCase("login_required")) {
|
|
|
|
if (reason == null) {
|
|
|
|
final String message = newPlayabilityStatus.getArray("messages").getString(0);
|
|
|
|
if (message != null && message.contains("private")) {
|
|
|
|
throw new PrivateContentException("This video is private.");
|
2021-02-14 14:22:45 +00:00
|
|
|
}
|
2022-05-01 14:38:05 +00:00
|
|
|
} else if (reason.contains("age")) {
|
|
|
|
// No streams can be fetched, therefore throw an AgeRestrictedContentException
|
|
|
|
// explicitly.
|
|
|
|
throw new AgeRestrictedContentException(
|
|
|
|
"This age-restricted video cannot be watched.");
|
2021-02-14 14:22:45 +00:00
|
|
|
}
|
2022-05-01 14:38:05 +00:00
|
|
|
}
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
|
2022-11-23 07:33:06 +00:00
|
|
|
if ((status.equalsIgnoreCase("unplayable") || status.equalsIgnoreCase("error"))
|
|
|
|
&& reason != null) {
|
2022-05-01 14:38:05 +00:00
|
|
|
if (reason.contains("Music Premium")) {
|
|
|
|
throw new YoutubeMusicPremiumContentException();
|
|
|
|
}
|
|
|
|
if (reason.contains("payment")) {
|
|
|
|
throw new PaidContentException("This video is a paid video");
|
|
|
|
}
|
|
|
|
if (reason.contains("members-only")) {
|
|
|
|
throw new PaidContentException("This video is only available"
|
|
|
|
+ " for members of the channel of this video");
|
|
|
|
}
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
|
2022-05-01 14:38:05 +00:00
|
|
|
if (reason.contains("unavailable")) {
|
|
|
|
final String detailedErrorMessage = getTextFromObject(newPlayabilityStatus
|
|
|
|
.getObject("errorScreen")
|
|
|
|
.getObject("playerErrorMessageRenderer")
|
|
|
|
.getObject("subreason"));
|
|
|
|
if (detailedErrorMessage != null && detailedErrorMessage.contains("country")) {
|
|
|
|
throw new GeographicRestrictionException(
|
|
|
|
"This video is not available in client's country.");
|
2022-11-26 21:09:08 +00:00
|
|
|
} else if (detailedErrorMessage != null) {
|
|
|
|
throw new ContentNotAvailableException(detailedErrorMessage);
|
2022-11-23 16:03:22 +00:00
|
|
|
} else {
|
|
|
|
throw new ContentNotAvailableException(reason);
|
2021-02-14 14:22:45 +00:00
|
|
|
}
|
|
|
|
}
|
2020-03-01 01:00:33 +00:00
|
|
|
}
|
2022-05-01 14:38:05 +00:00
|
|
|
|
|
|
|
throw new ContentNotAvailableException("Got error: \"" + reason + "\"");
|
2021-05-29 12:43:26 +00:00
|
|
|
}
|
|
|
|
|
2021-06-02 19:44:51 +00:00
|
|
|
/**
|
2022-01-15 16:25:00 +00:00
|
|
|
* Fetch the Android Mobile API and assign the streaming data to the androidStreamingData JSON
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
* object.
|
2021-06-02 19:44:51 +00:00
|
|
|
*/
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
private void fetchAndroidMobileJsonPlayer(@Nonnull final ContentCountry contentCountry,
|
|
|
|
@Nonnull final Localization localization,
|
|
|
|
@Nonnull final String videoId)
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
throws IOException, ExtractionException {
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
androidCpn = generateContentPlaybackNonce();
|
2022-05-01 14:38:05 +00:00
|
|
|
final byte[] mobileBody = JsonWriter.string(
|
|
|
|
prepareAndroidMobileJsonBuilder(localization, contentCountry)
|
|
|
|
.value(VIDEO_ID, videoId)
|
|
|
|
.value(CPN, androidCpn)
|
|
|
|
.value(CONTENT_CHECK_OK, true)
|
|
|
|
.value(RACY_CHECK_OK, true)
|
2023-07-22 18:22:16 +00:00
|
|
|
// Workaround getting streaming URLs which return 403 HTTP response code by
|
|
|
|
// using some parameters for Android client requests
|
2024-04-04 21:56:09 +00:00
|
|
|
.value("params", "CgIIAQ%3D%3D")
|
2022-05-01 14:38:05 +00:00
|
|
|
.done())
|
|
|
|
.getBytes(StandardCharsets.UTF_8);
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
final JsonObject androidPlayerResponse = getJsonAndroidPostResponse(PLAYER,
|
2022-01-15 16:25:00 +00:00
|
|
|
mobileBody, localization, "&t=" + generateTParameter()
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
+ "&id=" + videoId);
|
|
|
|
|
2022-08-08 20:40:49 +00:00
|
|
|
if (isPlayerResponseNotValid(androidPlayerResponse, videoId)) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
final JsonObject streamingData = androidPlayerResponse.getObject(STREAMING_DATA);
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
if (!isNullOrEmpty(streamingData)) {
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
androidStreamingData = streamingData;
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
if (html5StreamingData == null) {
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
playerResponse = androidPlayerResponse;
|
2021-05-29 12:43:26 +00:00
|
|
|
}
|
|
|
|
}
|
2017-07-11 03:08:03 +00:00
|
|
|
}
|
|
|
|
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
/**
|
2022-01-15 16:25:00 +00:00
|
|
|
* Fetch the iOS Mobile API and assign the streaming data to the iosStreamingData JSON
|
|
|
|
* object.
|
|
|
|
*/
|
|
|
|
private void fetchIosMobileJsonPlayer(@Nonnull final ContentCountry contentCountry,
|
|
|
|
@Nonnull final Localization localization,
|
|
|
|
@Nonnull final String videoId)
|
|
|
|
throws IOException, ExtractionException {
|
|
|
|
iosCpn = generateContentPlaybackNonce();
|
2022-05-01 14:38:05 +00:00
|
|
|
final byte[] mobileBody = JsonWriter.string(
|
|
|
|
prepareIosMobileJsonBuilder(localization, contentCountry)
|
2022-01-15 16:25:00 +00:00
|
|
|
.value(VIDEO_ID, videoId)
|
|
|
|
.value(CPN, iosCpn)
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
.value(CONTENT_CHECK_OK, true)
|
|
|
|
.value(RACY_CHECK_OK, true)
|
2022-01-15 16:25:00 +00:00
|
|
|
.done())
|
2022-05-01 14:38:05 +00:00
|
|
|
.getBytes(StandardCharsets.UTF_8);
|
2022-01-15 16:25:00 +00:00
|
|
|
|
|
|
|
final JsonObject iosPlayerResponse = getJsonIosPostResponse(PLAYER,
|
|
|
|
mobileBody, localization, "&t=" + generateTParameter()
|
|
|
|
+ "&id=" + videoId);
|
|
|
|
|
2022-08-08 20:40:49 +00:00
|
|
|
if (isPlayerResponseNotValid(iosPlayerResponse, videoId)) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2022-01-15 16:25:00 +00:00
|
|
|
final JsonObject streamingData = iosPlayerResponse.getObject(STREAMING_DATA);
|
|
|
|
if (!isNullOrEmpty(streamingData)) {
|
|
|
|
iosStreamingData = streamingData;
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
if (html5StreamingData == null) {
|
2022-01-15 16:25:00 +00:00
|
|
|
playerResponse = iosPlayerResponse;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
* Download the {@code TVHTML5_SIMPLY_EMBEDDED_PLAYER} JSON player as an embed client to bypass
|
|
|
|
* some age-restrictions and assign the streaming data to the {@code html5StreamingData} JSON
|
|
|
|
* object.
|
2022-01-15 16:25:00 +00:00
|
|
|
*
|
|
|
|
* @param contentCountry the content country to use
|
|
|
|
* @param localization the localization to use
|
|
|
|
* @param videoId the video id
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
*/
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
private void fetchTvHtml5EmbedJsonPlayer(@Nonnull final ContentCountry contentCountry,
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
@Nonnull final Localization localization,
|
|
|
|
@Nonnull final String videoId)
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
throws IOException, ExtractionException {
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
// Because a cpn is unique to each request, we need to generate it again
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
html5Cpn = generateContentPlaybackNonce();
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
final JsonObject tvHtml5EmbedPlayerResponse = getJsonPostResponse(PLAYER,
|
[YouTube] Refactor JavaScript player management API
This commit is introducing breaking changes.
For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.
The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.
YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.
If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.
All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.
Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-11 21:16:11 +00:00
|
|
|
createDesktopPlayerBody(localization,
|
|
|
|
contentCountry,
|
|
|
|
videoId,
|
|
|
|
YoutubeJavaScriptPlayerManager.getSignatureTimestamp(videoId),
|
|
|
|
true,
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
html5Cpn), localization);
|
2023-06-18 19:54:52 +00:00
|
|
|
|
|
|
|
if (isPlayerResponseNotValid(tvHtml5EmbedPlayerResponse, videoId)) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
final JsonObject streamingData = tvHtml5EmbedPlayerResponse.getObject(
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
STREAMING_DATA);
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
if (!isNullOrEmpty(streamingData)) {
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
playerResponse = tvHtml5EmbedPlayerResponse;
|
|
|
|
html5StreamingData = streamingData;
|
2022-01-15 16:25:00 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2022-08-08 20:40:49 +00:00
|
|
|
/**
|
2023-06-18 19:54:52 +00:00
|
|
|
* Checks whether a player response is invalid.
|
2022-08-08 20:40:49 +00:00
|
|
|
*
|
|
|
|
* <p>
|
|
|
|
* If YouTube detect that requests come from a third party client, they may replace the real
|
|
|
|
* player response by another one of a video saying that this content is not available on this
|
2023-06-18 19:54:52 +00:00
|
|
|
* app and to watch it on the latest version of YouTube. This behavior has been observed on the
|
|
|
|
* {@code ANDROID} client, see
|
|
|
|
* <a href="https://github.com/TeamNewPipe/NewPipe/issues/8713">
|
|
|
|
* https://github.com/TeamNewPipe/NewPipe/issues/8713</a>.
|
2022-08-08 20:40:49 +00:00
|
|
|
* </p>
|
|
|
|
*
|
|
|
|
* <p>
|
2023-06-18 19:54:52 +00:00
|
|
|
* YouTube may also sometimes for currently unknown reasons rate-limit an IP, and replace the
|
|
|
|
* real one by a player response with a video that says that the requested video is
|
|
|
|
* unavailable. This behaviour has been observed in Piped on the InnerTube clients used by the
|
|
|
|
* extractor ({@code ANDROID} and {@code WEB} clients) which should apply for all clients, see
|
|
|
|
* <a href="https://github.com/TeamPiped/Piped/issues/2487">
|
|
|
|
* https://github.com/TeamPiped/Piped/issues/2487</a>.
|
2022-08-08 20:40:49 +00:00
|
|
|
* </p>
|
|
|
|
*
|
|
|
|
* <p>
|
2023-06-18 19:54:52 +00:00
|
|
|
* We can detect this by checking whether the video ID of the player response returned is the
|
|
|
|
* same as the one requested by the extractor.
|
2022-08-08 20:40:49 +00:00
|
|
|
* </p>
|
|
|
|
*
|
2023-06-18 19:54:52 +00:00
|
|
|
* @param playerResponse a player response from any client
|
|
|
|
* @param videoId the video ID of the content requested
|
2022-08-08 20:40:49 +00:00
|
|
|
* @return whether the video ID of the player response is not equal to the one requested
|
|
|
|
*/
|
|
|
|
private static boolean isPlayerResponseNotValid(
|
2023-06-18 19:54:52 +00:00
|
|
|
@Nonnull final JsonObject playerResponse,
|
2022-08-08 20:40:49 +00:00
|
|
|
@Nonnull final String videoId) {
|
2023-06-18 19:54:52 +00:00
|
|
|
return !videoId.equals(playerResponse.getObject("videoDetails")
|
|
|
|
.getString("videoId"));
|
2022-08-08 20:40:49 +00:00
|
|
|
}
|
|
|
|
|
2017-08-10 17:50:59 +00:00
|
|
|
/*//////////////////////////////////////////////////////////////////////////
|
|
|
|
// Utils
|
|
|
|
//////////////////////////////////////////////////////////////////////////*/
|
|
|
|
|
2022-11-04 17:35:53 +00:00
|
|
|
@Nonnull
|
|
|
|
private JsonObject getVideoPrimaryInfoRenderer() {
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
if (videoPrimaryInfoRenderer != null) {
|
|
|
|
return videoPrimaryInfoRenderer;
|
2022-03-18 14:09:06 +00:00
|
|
|
}
|
2020-02-28 16:03:21 +00:00
|
|
|
|
2022-11-04 17:35:53 +00:00
|
|
|
videoPrimaryInfoRenderer = getVideoInfoRenderer("videoPrimaryInfoRenderer");
|
|
|
|
return videoPrimaryInfoRenderer;
|
2020-02-25 08:50:22 +00:00
|
|
|
}
|
|
|
|
|
2022-05-01 14:38:05 +00:00
|
|
|
@Nonnull
|
2022-11-04 17:35:53 +00:00
|
|
|
private JsonObject getVideoSecondaryInfoRenderer() {
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
if (videoSecondaryInfoRenderer != null) {
|
|
|
|
return videoSecondaryInfoRenderer;
|
2022-03-18 14:09:06 +00:00
|
|
|
}
|
2020-02-28 16:03:21 +00:00
|
|
|
|
2022-11-04 17:35:53 +00:00
|
|
|
videoSecondaryInfoRenderer = getVideoInfoRenderer("videoSecondaryInfoRenderer");
|
|
|
|
return videoSecondaryInfoRenderer;
|
|
|
|
}
|
|
|
|
|
|
|
|
@Nonnull
|
|
|
|
private JsonObject getVideoInfoRenderer(@Nonnull final String videoRendererName) {
|
|
|
|
return nextResponse.getObject("contents")
|
2022-05-01 14:38:05 +00:00
|
|
|
.getObject("twoColumnWatchNextResults")
|
|
|
|
.getObject("results")
|
|
|
|
.getObject("results")
|
|
|
|
.getArray("contents")
|
|
|
|
.stream()
|
|
|
|
.filter(JsonObject.class::isInstance)
|
|
|
|
.map(JsonObject.class::cast)
|
2022-11-04 17:35:53 +00:00
|
|
|
.filter(content -> content.has(videoRendererName))
|
|
|
|
.map(content -> content.getObject(videoRendererName))
|
2022-05-01 14:38:05 +00:00
|
|
|
.findFirst()
|
2022-11-04 17:35:53 +00:00
|
|
|
.orElse(new JsonObject());
|
2020-02-25 09:05:53 +00:00
|
|
|
}
|
|
|
|
|
2021-06-06 16:32:07 +00:00
|
|
|
@Nonnull
|
2022-03-06 19:10:11 +00:00
|
|
|
private <T extends Stream> List<T> getItags(
|
|
|
|
final String streamingDataKey,
|
|
|
|
final ItagItem.ItagType itagTypeWanted,
|
2022-05-18 10:39:41 +00:00
|
|
|
final java.util.function.Function<ItagInfo, T> streamBuilderHelper,
|
2022-03-06 19:10:11 +00:00
|
|
|
final String streamTypeExceptionMessage) throws ParsingException {
|
|
|
|
try {
|
2022-05-27 22:26:53 +00:00
|
|
|
final String videoId = getId();
|
2022-03-06 19:10:11 +00:00
|
|
|
final List<T> streamList = new ArrayList<>();
|
2022-05-27 22:26:53 +00:00
|
|
|
|
|
|
|
java.util.stream.Stream.of(
|
|
|
|
// Use the androidStreamingData object first because there is no n param and no
|
|
|
|
// signatureCiphers in streaming URLs of the Android client
|
|
|
|
new Pair<>(androidStreamingData, androidCpn),
|
|
|
|
new Pair<>(html5StreamingData, html5Cpn),
|
|
|
|
// Use the iosStreamingData object in the last position because most of the
|
|
|
|
// available streams can be extracted with the Android and web clients and also
|
|
|
|
// because the iOS client is only enabled by default on livestreams
|
|
|
|
new Pair<>(iosStreamingData, iosCpn)
|
|
|
|
)
|
|
|
|
.flatMap(pair -> getStreamsFromStreamingDataKey(videoId, pair.getFirst(),
|
|
|
|
streamingDataKey, itagTypeWanted, pair.getSecond()))
|
|
|
|
.map(streamBuilderHelper)
|
|
|
|
.forEachOrdered(stream -> {
|
|
|
|
if (!Stream.containSimilarStream(stream, streamList)) {
|
|
|
|
streamList.add(stream);
|
|
|
|
}
|
|
|
|
});
|
2022-03-06 19:10:11 +00:00
|
|
|
|
|
|
|
return streamList;
|
|
|
|
} catch (final Exception e) {
|
|
|
|
throw new ParsingException(
|
|
|
|
"Could not get " + streamTypeExceptionMessage + " streams", e);
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
}
|
2022-03-06 19:10:11 +00:00
|
|
|
}
|
|
|
|
|
2022-03-15 10:19:13 +00:00
|
|
|
/**
|
2022-05-18 10:39:41 +00:00
|
|
|
* Get the stream builder helper which will be used to build {@link AudioStream}s in
|
|
|
|
* {@link #getItags(String, ItagItem.ItagType, java.util.function.Function, String)}
|
2022-03-15 10:19:13 +00:00
|
|
|
*
|
|
|
|
* <p>
|
|
|
|
* The {@code StreamBuilderHelper} will set the following attributes in the
|
|
|
|
* {@link AudioStream}s built:
|
|
|
|
* <ul>
|
|
|
|
* <li>the {@link ItagItem}'s id of the stream as its id;</li>
|
|
|
|
* <li>{@link ItagInfo#getContent()} and {@link ItagInfo#getIsUrl()} as its content and
|
|
|
|
* and as the value of {@code isUrl};</li>
|
|
|
|
* <li>the media format returned by the {@link ItagItem} as its media format;</li>
|
|
|
|
* <li>its average bitrate with the value returned by {@link
|
|
|
|
* ItagItem#getAverageBitrate()};</li>
|
|
|
|
* <li>the {@link ItagItem};</li>
|
|
|
|
* <li>the {@link DeliveryMethod#DASH DASH delivery method}, for OTF streams, live streams
|
|
|
|
* and ended streams.</li>
|
|
|
|
* </ul>
|
|
|
|
* </p>
|
|
|
|
*
|
|
|
|
* <p>
|
|
|
|
* Note that the {@link ItagItem} comes from an {@link ItagInfo} instance.
|
|
|
|
* </p>
|
|
|
|
*
|
2022-05-18 10:39:41 +00:00
|
|
|
* @return a stream builder helper to build {@link AudioStream}s
|
2022-03-15 10:19:13 +00:00
|
|
|
*/
|
2022-03-06 19:10:11 +00:00
|
|
|
@Nonnull
|
2022-05-18 10:39:41 +00:00
|
|
|
private java.util.function.Function<ItagInfo, AudioStream> getAudioStreamBuilderHelper() {
|
|
|
|
return (itagInfo) -> {
|
|
|
|
final ItagItem itagItem = itagInfo.getItagItem();
|
|
|
|
final AudioStream.Builder builder = new AudioStream.Builder()
|
|
|
|
.setId(String.valueOf(itagItem.id))
|
|
|
|
.setContent(itagInfo.getContent(), itagInfo.getIsUrl())
|
|
|
|
.setMediaFormat(itagItem.getMediaFormat())
|
|
|
|
.setAverageBitrate(itagItem.getAverageBitrate())
|
2022-11-13 21:35:26 +00:00
|
|
|
.setAudioTrackId(itagItem.getAudioTrackId())
|
|
|
|
.setAudioTrackName(itagItem.getAudioTrackName())
|
2022-12-16 16:38:37 +00:00
|
|
|
.setAudioLocale(itagItem.getAudioLocale())
|
2023-03-27 22:02:20 +00:00
|
|
|
.setAudioTrackType(itagItem.getAudioTrackType())
|
2022-05-18 10:39:41 +00:00
|
|
|
.setItagItem(itagItem);
|
|
|
|
|
|
|
|
if (streamType == StreamType.LIVE_STREAM
|
|
|
|
|| streamType == StreamType.POST_LIVE_STREAM
|
|
|
|
|| !itagInfo.getIsUrl()) {
|
|
|
|
// For YouTube videos on OTF streams and for all streams of post-live streams
|
|
|
|
// and live streams, only the DASH delivery method can be used.
|
|
|
|
builder.setDeliveryMethod(DeliveryMethod.DASH);
|
2022-03-06 19:10:11 +00:00
|
|
|
}
|
2022-05-18 10:39:41 +00:00
|
|
|
|
|
|
|
return builder.build();
|
2022-03-06 19:10:11 +00:00
|
|
|
};
|
|
|
|
}
|
|
|
|
|
2022-03-15 10:19:13 +00:00
|
|
|
/**
|
2022-05-18 10:39:41 +00:00
|
|
|
* Get the stream builder helper which will be used to build {@link VideoStream}s in
|
|
|
|
* {@link #getItags(String, ItagItem.ItagType, java.util.function.Function, String)}
|
2022-03-15 10:19:13 +00:00
|
|
|
*
|
|
|
|
* <p>
|
|
|
|
* The {@code StreamBuilderHelper} will set the following attributes in the
|
|
|
|
* {@link VideoStream}s built:
|
|
|
|
* <ul>
|
|
|
|
* <li>the {@link ItagItem}'s id of the stream as its id;</li>
|
|
|
|
* <li>{@link ItagInfo#getContent()} and {@link ItagInfo#getIsUrl()} as its content and
|
|
|
|
* and as the value of {@code isUrl};</li>
|
|
|
|
* <li>the media format returned by the {@link ItagItem} as its media format;</li>
|
|
|
|
* <li>whether it is video-only with the {@code areStreamsVideoOnly} parameter</li>
|
|
|
|
* <li>the {@link ItagItem};</li>
|
|
|
|
* <li>the resolution, by trying to use, in this order:
|
|
|
|
* <ol>
|
|
|
|
* <li>the height returned by the {@link ItagItem} + {@code p} + the frame rate if
|
|
|
|
* it is more than 30;</li>
|
|
|
|
* <li>the default resolution string from the {@link ItagItem};</li>
|
2022-08-15 03:49:40 +00:00
|
|
|
* <li>an empty string.</li>
|
2022-03-15 10:19:13 +00:00
|
|
|
* </ol>
|
|
|
|
* </li>
|
|
|
|
* <li>the {@link DeliveryMethod#DASH DASH delivery method}, for OTF streams, live streams
|
|
|
|
* and ended streams.</li>
|
|
|
|
* </ul>
|
|
|
|
*
|
|
|
|
* <p>
|
|
|
|
* Note that the {@link ItagItem} comes from an {@link ItagInfo} instance.
|
|
|
|
* </p>
|
|
|
|
*
|
2022-05-18 10:39:41 +00:00
|
|
|
* @param areStreamsVideoOnly whether the stream builder helper will set the video
|
2022-03-15 10:19:13 +00:00
|
|
|
* streams as video-only streams
|
2022-05-18 10:39:41 +00:00
|
|
|
* @return a stream builder helper to build {@link VideoStream}s
|
2022-03-15 10:19:13 +00:00
|
|
|
*/
|
2022-03-06 19:10:11 +00:00
|
|
|
@Nonnull
|
2022-05-18 10:39:41 +00:00
|
|
|
private java.util.function.Function<ItagInfo, VideoStream> getVideoStreamBuilderHelper(
|
2022-03-06 19:10:11 +00:00
|
|
|
final boolean areStreamsVideoOnly) {
|
2022-05-18 10:39:41 +00:00
|
|
|
return (itagInfo) -> {
|
|
|
|
final ItagItem itagItem = itagInfo.getItagItem();
|
|
|
|
final VideoStream.Builder builder = new VideoStream.Builder()
|
|
|
|
.setId(String.valueOf(itagItem.id))
|
|
|
|
.setContent(itagInfo.getContent(), itagInfo.getIsUrl())
|
|
|
|
.setMediaFormat(itagItem.getMediaFormat())
|
|
|
|
.setIsVideoOnly(areStreamsVideoOnly)
|
|
|
|
.setItagItem(itagItem);
|
|
|
|
|
|
|
|
final String resolutionString = itagItem.getResolutionString();
|
|
|
|
builder.setResolution(resolutionString != null ? resolutionString
|
2022-08-15 03:49:40 +00:00
|
|
|
: "");
|
2022-05-18 10:39:41 +00:00
|
|
|
|
|
|
|
if (streamType != StreamType.VIDEO_STREAM || !itagInfo.getIsUrl()) {
|
|
|
|
// For YouTube videos on OTF streams and for all streams of post-live streams
|
|
|
|
// and live streams, only the DASH delivery method can be used.
|
|
|
|
builder.setDeliveryMethod(DeliveryMethod.DASH);
|
2022-03-06 19:10:11 +00:00
|
|
|
}
|
2022-05-18 10:39:41 +00:00
|
|
|
|
|
|
|
return builder.build();
|
2022-03-06 19:10:11 +00:00
|
|
|
};
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
@Nonnull
|
2022-05-27 22:26:53 +00:00
|
|
|
private java.util.stream.Stream<ItagInfo> getStreamsFromStreamingDataKey(
|
|
|
|
final String videoId,
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
final JsonObject streamingData,
|
2022-03-06 19:10:11 +00:00
|
|
|
final String streamingDataKey,
|
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2021-12-22 16:55:41 +00:00
|
|
|
@Nonnull final ItagItem.ItagType itagTypeWanted,
|
2022-05-27 22:26:53 +00:00
|
|
|
@Nonnull final String contentPlaybackNonce) {
|
2022-05-01 14:38:05 +00:00
|
|
|
if (streamingData == null || !streamingData.has(streamingDataKey)) {
|
2022-05-27 22:26:53 +00:00
|
|
|
return java.util.stream.Stream.empty();
|
2022-05-01 14:38:05 +00:00
|
|
|
}
|
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-07-28 21:55:09 +00:00
|
|
|
|
2022-05-27 22:26:53 +00:00
|
|
|
return streamingData.getArray(streamingDataKey).stream()
|
|
|
|
.filter(JsonObject.class::isInstance)
|
|
|
|
.map(JsonObject.class::cast)
|
|
|
|
.map(formatData -> {
|
|
|
|
try {
|
|
|
|
final ItagItem itagItem = ItagItem.getItag(formatData.getInt("itag"));
|
|
|
|
if (itagItem.itagType == itagTypeWanted) {
|
|
|
|
return buildAndAddItagInfoToList(videoId, formatData, itagItem,
|
|
|
|
itagItem.itagType, contentPlaybackNonce);
|
|
|
|
}
|
|
|
|
} catch (final IOException | ExtractionException ignored) {
|
|
|
|
// if the itag is not supported and getItag fails, we end up here
|
|
|
|
}
|
|
|
|
return null;
|
|
|
|
})
|
|
|
|
.filter(Objects::nonNull);
|
2022-03-15 10:19:13 +00:00
|
|
|
}
|
2022-03-06 19:10:11 +00:00
|
|
|
|
2022-05-27 22:26:53 +00:00
|
|
|
private ItagInfo buildAndAddItagInfoToList(
|
2022-03-15 10:19:13 +00:00
|
|
|
@Nonnull final String videoId,
|
|
|
|
@Nonnull final JsonObject formatData,
|
|
|
|
@Nonnull final ItagItem itagItem,
|
|
|
|
@Nonnull final ItagItem.ItagType itagType,
|
|
|
|
@Nonnull final String contentPlaybackNonce) throws IOException, ExtractionException {
|
|
|
|
String streamUrl;
|
|
|
|
if (formatData.has("url")) {
|
|
|
|
streamUrl = formatData.getString("url");
|
|
|
|
} else {
|
|
|
|
// This url has an obfuscated signature
|
|
|
|
final String cipherString = formatData.has(CIPHER)
|
|
|
|
? formatData.getString(CIPHER)
|
|
|
|
: formatData.getString(SIGNATURE_CIPHER);
|
|
|
|
final Map<String, String> cipher = Parser.compatParseMap(
|
|
|
|
cipherString);
|
|
|
|
streamUrl = cipher.get("url") + "&" + cipher.get("sp") + "="
|
[YouTube] Refactor JavaScript player management API
This commit is introducing breaking changes.
For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.
The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.
YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.
If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.
All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.
Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-11 21:16:11 +00:00
|
|
|
+ YoutubeJavaScriptPlayerManager.deobfuscateSignature(videoId, cipher.get("s"));
|
2022-03-15 10:19:13 +00:00
|
|
|
}
|
2022-05-01 14:38:05 +00:00
|
|
|
|
2022-03-15 10:19:13 +00:00
|
|
|
// Add the content playback nonce to the stream URL
|
|
|
|
streamUrl += "&" + CPN + "=" + contentPlaybackNonce;
|
2022-03-06 19:10:11 +00:00
|
|
|
|
2022-04-11 17:35:57 +00:00
|
|
|
// Decrypt the n parameter if it is present
|
[YouTube] Refactor JavaScript player management API
This commit is introducing breaking changes.
For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.
The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.
YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.
If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.
All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.
Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-11 21:16:11 +00:00
|
|
|
streamUrl = tryDeobfuscateThrottlingParameterOfUrl(streamUrl, videoId);
|
2022-03-06 19:10:11 +00:00
|
|
|
|
2022-03-15 10:19:13 +00:00
|
|
|
final JsonObject initRange = formatData.getObject("initRange");
|
|
|
|
final JsonObject indexRange = formatData.getObject("indexRange");
|
2022-08-15 03:49:40 +00:00
|
|
|
final String mimeType = formatData.getString("mimeType", "");
|
2022-03-15 10:19:13 +00:00
|
|
|
final String codec = mimeType.contains("codecs")
|
2022-08-15 03:49:40 +00:00
|
|
|
? mimeType.split("\"")[1] : "";
|
2022-03-15 10:19:13 +00:00
|
|
|
|
|
|
|
itagItem.setBitrate(formatData.getInt("bitrate"));
|
|
|
|
itagItem.setWidth(formatData.getInt("width"));
|
|
|
|
itagItem.setHeight(formatData.getInt("height"));
|
|
|
|
itagItem.setInitStart(Integer.parseInt(initRange.getString("start", "-1")));
|
|
|
|
itagItem.setInitEnd(Integer.parseInt(initRange.getString("end", "-1")));
|
|
|
|
itagItem.setIndexStart(Integer.parseInt(indexRange.getString("start", "-1")));
|
|
|
|
itagItem.setIndexEnd(Integer.parseInt(indexRange.getString("end", "-1")));
|
|
|
|
itagItem.setQuality(formatData.getString("quality"));
|
|
|
|
itagItem.setCodec(codec);
|
|
|
|
|
|
|
|
if (streamType == StreamType.LIVE_STREAM || streamType == StreamType.POST_LIVE_STREAM) {
|
|
|
|
itagItem.setTargetDurationSec(formatData.getInt("targetDurationSec"));
|
2022-06-16 09:04:24 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (itagType == ItagItem.ItagType.VIDEO || itagType == ItagItem.ItagType.VIDEO_ONLY) {
|
2022-03-15 10:19:13 +00:00
|
|
|
itagItem.setFps(formatData.getInt("fps"));
|
2022-05-27 22:26:53 +00:00
|
|
|
} else if (itagType == ItagItem.ItagType.AUDIO) {
|
2022-03-15 10:19:13 +00:00
|
|
|
// YouTube return the audio sample rate as a string
|
|
|
|
itagItem.setSampleRate(Integer.parseInt(formatData.getString("audioSampleRate")));
|
2022-06-16 09:04:24 +00:00
|
|
|
itagItem.setAudioChannels(formatData.getInt("audioChannels",
|
|
|
|
// Most audio streams have two audio channels, so use this value if the real
|
|
|
|
// count cannot be extracted
|
|
|
|
// Doing this prevents an exception when generating the
|
|
|
|
// AudioChannelConfiguration element of DASH manifests of audio streams in
|
|
|
|
// YoutubeDashManifestCreatorUtils
|
|
|
|
2));
|
2022-12-16 16:38:37 +00:00
|
|
|
|
|
|
|
final String audioTrackId = formatData.getObject("audioTrack")
|
|
|
|
.getString("id");
|
|
|
|
if (!isNullOrEmpty(audioTrackId)) {
|
|
|
|
itagItem.setAudioTrackId(audioTrackId);
|
|
|
|
final int audioTrackIdLastLocaleCharacter = audioTrackId.indexOf(".");
|
|
|
|
if (audioTrackIdLastLocaleCharacter != -1) {
|
|
|
|
// Audio tracks IDs are in the form LANGUAGE_CODE.TRACK_NUMBER
|
2024-03-29 12:41:23 +00:00
|
|
|
LocaleCompat.forLanguageTag(
|
|
|
|
audioTrackId.substring(0, audioTrackIdLastLocaleCharacter)
|
|
|
|
).ifPresent(itagItem::setAudioLocale);
|
2022-12-16 16:38:37 +00:00
|
|
|
}
|
2023-03-27 22:02:20 +00:00
|
|
|
itagItem.setAudioTrackType(YoutubeParsingHelper.extractAudioTrackType(streamUrl));
|
2022-12-16 16:38:37 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
itagItem.setAudioTrackName(formatData.getObject("audioTrack")
|
|
|
|
.getString("displayName"));
|
2022-03-15 10:19:13 +00:00
|
|
|
}
|
|
|
|
|
2022-05-07 17:32:12 +00:00
|
|
|
// YouTube return the content length and the approximate duration as strings
|
2022-03-15 10:19:13 +00:00
|
|
|
itagItem.setContentLength(Long.parseLong(formatData.getString("contentLength",
|
|
|
|
String.valueOf(CONTENT_LENGTH_UNKNOWN))));
|
2022-05-07 17:32:12 +00:00
|
|
|
itagItem.setApproxDurationMs(Long.parseLong(formatData.getString("approxDurationMs",
|
|
|
|
String.valueOf(APPROX_DURATION_MS_UNKNOWN))));
|
2017-08-10 17:50:59 +00:00
|
|
|
|
2022-03-15 10:19:13 +00:00
|
|
|
final ItagInfo itagInfo = new ItagInfo(streamUrl, itagItem);
|
|
|
|
|
|
|
|
if (streamType == StreamType.VIDEO_STREAM) {
|
2022-08-15 03:49:40 +00:00
|
|
|
itagInfo.setIsUrl(!formatData.getString("type", "")
|
2022-03-15 10:19:13 +00:00
|
|
|
.equalsIgnoreCase("FORMAT_STREAM_TYPE_OTF"));
|
|
|
|
} else {
|
|
|
|
// We are currently not able to generate DASH manifests for running
|
|
|
|
// livestreams, so because of the requirements of StreamInfo
|
|
|
|
// objects, return these streams as DASH URL streams (even if they
|
|
|
|
// are not playable).
|
|
|
|
// Ended livestreams are returned as non URL streams
|
|
|
|
itagInfo.setIsUrl(streamType != StreamType.POST_LIVE_STREAM);
|
|
|
|
}
|
|
|
|
|
2022-05-27 22:26:53 +00:00
|
|
|
return itagInfo;
|
2022-03-15 10:19:13 +00:00
|
|
|
}
|
2022-05-01 14:38:05 +00:00
|
|
|
|
2020-02-08 22:58:46 +00:00
|
|
|
@Nonnull
|
|
|
|
@Override
|
|
|
|
public List<Frameset> getFrames() throws ExtractionException {
|
|
|
|
try {
|
2020-04-09 13:52:42 +00:00
|
|
|
final JsonObject storyboards = playerResponse.getObject("storyboards");
|
2022-05-01 14:38:05 +00:00
|
|
|
final JsonObject storyboardsRenderer = storyboards.getObject(
|
|
|
|
storyboards.has("playerLiveStoryboardSpecRenderer")
|
|
|
|
? "playerLiveStoryboardSpecRenderer"
|
|
|
|
: "playerStoryboardSpecRenderer"
|
|
|
|
);
|
2020-04-09 13:52:42 +00:00
|
|
|
|
2021-06-08 18:30:13 +00:00
|
|
|
if (storyboardsRenderer == null) {
|
2021-06-15 19:58:00 +00:00
|
|
|
return Collections.emptyList();
|
2021-06-08 18:30:13 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
final String storyboardsRendererSpec = storyboardsRenderer.getString("spec");
|
|
|
|
if (storyboardsRendererSpec == null) {
|
2021-06-15 19:58:00 +00:00
|
|
|
return Collections.emptyList();
|
2021-06-08 18:30:13 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
final String[] spec = storyboardsRendererSpec.split("\\|");
|
2020-02-08 22:58:46 +00:00
|
|
|
final String url = spec[0];
|
2022-05-01 14:38:05 +00:00
|
|
|
final List<Frameset> result = new ArrayList<>(spec.length - 1);
|
2020-04-09 13:52:42 +00:00
|
|
|
|
2020-02-08 22:58:46 +00:00
|
|
|
for (int i = 1; i < spec.length; ++i) {
|
|
|
|
final String[] parts = spec[i].split("#");
|
2021-01-14 19:01:06 +00:00
|
|
|
if (parts.length != 8 || Integer.parseInt(parts[5]) == 0) {
|
2020-02-08 22:58:46 +00:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
final int totalCount = Integer.parseInt(parts[2]);
|
|
|
|
final int framesPerPageX = Integer.parseInt(parts[3]);
|
|
|
|
final int framesPerPageY = Integer.parseInt(parts[4]);
|
2021-04-15 16:58:59 +00:00
|
|
|
final String baseUrl = url.replace("$L", String.valueOf(i - 1))
|
|
|
|
.replace("$N", parts[6]) + "&sigh=" + parts[7];
|
2020-02-08 22:58:46 +00:00
|
|
|
final List<String> urls;
|
|
|
|
if (baseUrl.contains("$M")) {
|
2021-04-15 16:58:59 +00:00
|
|
|
final int totalPages = (int) Math.ceil(totalCount / (double)
|
|
|
|
(framesPerPageX * framesPerPageY));
|
2020-02-08 22:58:46 +00:00
|
|
|
urls = new ArrayList<>(totalPages);
|
|
|
|
for (int j = 0; j < totalPages; j++) {
|
|
|
|
urls.add(baseUrl.replace("$M", String.valueOf(j)));
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
urls = Collections.singletonList(baseUrl);
|
|
|
|
}
|
|
|
|
result.add(new Frameset(
|
|
|
|
urls,
|
2022-05-01 14:38:05 +00:00
|
|
|
/*frameWidth=*/Integer.parseInt(parts[0]),
|
|
|
|
/*frameHeight=*/Integer.parseInt(parts[1]),
|
2020-02-08 22:58:46 +00:00
|
|
|
totalCount,
|
2022-05-01 14:38:05 +00:00
|
|
|
/*durationPerFrame=*/Integer.parseInt(parts[5]),
|
2020-02-08 22:58:46 +00:00
|
|
|
framesPerPageX,
|
|
|
|
framesPerPageY
|
|
|
|
));
|
|
|
|
}
|
|
|
|
return result;
|
2021-02-24 16:06:38 +00:00
|
|
|
} catch (final Exception e) {
|
2021-05-29 12:43:26 +00:00
|
|
|
throw new ExtractionException("Could not get frames", e);
|
2020-02-08 22:58:46 +00:00
|
|
|
}
|
|
|
|
}
|
added metadata, fix descriptions, fix thumbnail, update tests
thumbnail: quality before: https://peertube.cpy.re/static/thumbnails/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
quality after: https://peertube.cpy.re/static/previews/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
description: we were getting about the first 260 characters, we now get full description (with fallback to first 260 chars if the get request for full description fails)
test: updated tests to match description, also changed some test: it was assertEquals(extracted, expected), but the proper way to do it is assertEquals(expected, extracted)
metadata: got host, privacy (public, private, unlisted), licence, language, tags
2020-01-19 11:45:52 +00:00
|
|
|
|
2020-02-25 08:07:22 +00:00
|
|
|
@Nonnull
|
added metadata, fix descriptions, fix thumbnail, update tests
thumbnail: quality before: https://peertube.cpy.re/static/thumbnails/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
quality after: https://peertube.cpy.re/static/previews/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
description: we were getting about the first 260 characters, we now get full description (with fallback to first 260 chars if the get request for full description fails)
test: updated tests to match description, also changed some test: it was assertEquals(extracted, expected), but the proper way to do it is assertEquals(expected, extracted)
metadata: got host, privacy (public, private, unlisted), licence, language, tags
2020-01-19 11:45:52 +00:00
|
|
|
@Override
|
2020-02-09 10:59:23 +00:00
|
|
|
public Privacy getPrivacy() {
|
2022-05-01 14:38:05 +00:00
|
|
|
return playerMicroFormatRenderer.getBoolean("isUnlisted")
|
|
|
|
? Privacy.UNLISTED
|
|
|
|
: Privacy.PUBLIC;
|
added metadata, fix descriptions, fix thumbnail, update tests
thumbnail: quality before: https://peertube.cpy.re/static/thumbnails/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
quality after: https://peertube.cpy.re/static/previews/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
description: we were getting about the first 260 characters, we now get full description (with fallback to first 260 chars if the get request for full description fails)
test: updated tests to match description, also changed some test: it was assertEquals(extracted, expected), but the proper way to do it is assertEquals(expected, extracted)
metadata: got host, privacy (public, private, unlisted), licence, language, tags
2020-01-19 11:45:52 +00:00
|
|
|
}
|
|
|
|
|
2020-02-25 08:07:22 +00:00
|
|
|
@Nonnull
|
added metadata, fix descriptions, fix thumbnail, update tests
thumbnail: quality before: https://peertube.cpy.re/static/thumbnails/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
quality after: https://peertube.cpy.re/static/previews/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
description: we were getting about the first 260 characters, we now get full description (with fallback to first 260 chars if the get request for full description fails)
test: updated tests to match description, also changed some test: it was assertEquals(extracted, expected), but the proper way to do it is assertEquals(expected, extracted)
metadata: got host, privacy (public, private, unlisted), licence, language, tags
2020-01-19 11:45:52 +00:00
|
|
|
@Override
|
2020-02-25 08:07:22 +00:00
|
|
|
public String getCategory() {
|
2022-08-15 03:49:40 +00:00
|
|
|
return playerMicroFormatRenderer.getString("category", "");
|
added metadata, fix descriptions, fix thumbnail, update tests
thumbnail: quality before: https://peertube.cpy.re/static/thumbnails/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
quality after: https://peertube.cpy.re/static/previews/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
description: we were getting about the first 260 characters, we now get full description (with fallback to first 260 chars if the get request for full description fails)
test: updated tests to match description, also changed some test: it was assertEquals(extracted, expected), but the proper way to do it is assertEquals(expected, extracted)
metadata: got host, privacy (public, private, unlisted), licence, language, tags
2020-01-19 11:45:52 +00:00
|
|
|
}
|
|
|
|
|
2020-02-25 08:07:22 +00:00
|
|
|
@Nonnull
|
added metadata, fix descriptions, fix thumbnail, update tests
thumbnail: quality before: https://peertube.cpy.re/static/thumbnails/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
quality after: https://peertube.cpy.re/static/previews/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
description: we were getting about the first 260 characters, we now get full description (with fallback to first 260 chars if the get request for full description fails)
test: updated tests to match description, also changed some test: it was assertEquals(extracted, expected), but the proper way to do it is assertEquals(expected, extracted)
metadata: got host, privacy (public, private, unlisted), licence, language, tags
2020-01-19 11:45:52 +00:00
|
|
|
@Override
|
2020-02-09 10:59:23 +00:00
|
|
|
public String getLicence() throws ParsingException {
|
|
|
|
final JsonObject metadataRowRenderer = getVideoSecondaryInfoRenderer()
|
2022-05-01 14:38:05 +00:00
|
|
|
.getObject("metadataRowContainer")
|
|
|
|
.getObject("metadataRowContainerRenderer")
|
2021-04-15 16:58:59 +00:00
|
|
|
.getArray("rows")
|
2022-05-01 14:38:05 +00:00
|
|
|
.getObject(0)
|
|
|
|
.getObject("metadataRowRenderer");
|
2020-02-09 10:59:23 +00:00
|
|
|
|
|
|
|
final JsonArray contents = metadataRowRenderer.getArray("contents");
|
|
|
|
final String license = getTextFromObject(contents.getObject(0));
|
2022-05-01 14:38:05 +00:00
|
|
|
return license != null
|
|
|
|
&& "Licence".equals(getTextFromObject(metadataRowRenderer.getObject("title")))
|
|
|
|
? license
|
|
|
|
: "YouTube licence";
|
added metadata, fix descriptions, fix thumbnail, update tests
thumbnail: quality before: https://peertube.cpy.re/static/thumbnails/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
quality after: https://peertube.cpy.re/static/previews/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
description: we were getting about the first 260 characters, we now get full description (with fallback to first 260 chars if the get request for full description fails)
test: updated tests to match description, also changed some test: it was assertEquals(extracted, expected), but the proper way to do it is assertEquals(expected, extracted)
metadata: got host, privacy (public, private, unlisted), licence, language, tags
2020-01-19 11:45:52 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
@Override
|
2020-02-25 08:07:22 +00:00
|
|
|
public Locale getLanguageInfo() {
|
2020-01-25 12:16:42 +00:00
|
|
|
return null;
|
added metadata, fix descriptions, fix thumbnail, update tests
thumbnail: quality before: https://peertube.cpy.re/static/thumbnails/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
quality after: https://peertube.cpy.re/static/previews/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
description: we were getting about the first 260 characters, we now get full description (with fallback to first 260 chars if the get request for full description fails)
test: updated tests to match description, also changed some test: it was assertEquals(extracted, expected), but the proper way to do it is assertEquals(expected, extracted)
metadata: got host, privacy (public, private, unlisted), licence, language, tags
2020-01-19 11:45:52 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
@Nonnull
|
|
|
|
@Override
|
2020-02-25 08:07:22 +00:00
|
|
|
public List<String> getTags() {
|
2021-04-15 16:58:59 +00:00
|
|
|
return JsonUtils.getStringListFromJsonArray(playerResponse.getObject("videoDetails")
|
|
|
|
.getArray("keywords"));
|
2020-01-23 13:19:22 +00:00
|
|
|
}
|
2020-12-12 09:24:29 +00:00
|
|
|
|
|
|
|
@Nonnull
|
|
|
|
@Override
|
|
|
|
public List<StreamSegment> getStreamSegments() throws ParsingException {
|
2022-05-01 14:38:05 +00:00
|
|
|
|
|
|
|
if (!nextResponse.has("engagementPanels")) {
|
|
|
|
return Collections.emptyList();
|
|
|
|
}
|
|
|
|
|
|
|
|
final JsonArray segmentsArray = nextResponse.getArray("engagementPanels")
|
|
|
|
.stream()
|
|
|
|
// Check if object is a JsonObject
|
|
|
|
.filter(JsonObject.class::isInstance)
|
|
|
|
.map(JsonObject.class::cast)
|
|
|
|
// Check if the panel is the correct one
|
|
|
|
.filter(panel -> "engagement-panel-macro-markers-description-chapters".equals(
|
|
|
|
panel
|
|
|
|
.getObject("engagementPanelSectionListRenderer")
|
|
|
|
.getString("panelIdentifier")))
|
|
|
|
// Extract the data
|
|
|
|
.map(panel -> panel
|
2021-04-15 16:58:59 +00:00
|
|
|
.getObject("engagementPanelSectionListRenderer")
|
2022-05-01 14:38:05 +00:00
|
|
|
.getObject("content")
|
|
|
|
.getObject("macroMarkersListRenderer")
|
|
|
|
.getArray("contents"))
|
|
|
|
.findFirst()
|
|
|
|
.orElse(null);
|
|
|
|
|
|
|
|
// If no data was found exit
|
|
|
|
if (segmentsArray == null) {
|
|
|
|
return Collections.emptyList();
|
|
|
|
}
|
|
|
|
|
|
|
|
final long duration = getLength();
|
|
|
|
final List<StreamSegment> segments = new ArrayList<>();
|
|
|
|
for (final JsonObject segmentJson : segmentsArray.stream()
|
|
|
|
.filter(JsonObject.class::isInstance)
|
|
|
|
.map(JsonObject.class::cast)
|
|
|
|
.map(object -> object.getObject("macroMarkersListItemRenderer"))
|
|
|
|
.collect(Collectors.toList())
|
|
|
|
) {
|
|
|
|
final int startTimeSeconds = segmentJson.getObject("onTap")
|
|
|
|
.getObject("watchEndpoint").getInt("startTimeSeconds", -1);
|
|
|
|
|
|
|
|
if (startTimeSeconds == -1) {
|
|
|
|
throw new ParsingException("Could not get stream segment start time.");
|
|
|
|
}
|
|
|
|
if (startTimeSeconds > duration) {
|
|
|
|
break;
|
2020-12-12 09:24:29 +00:00
|
|
|
}
|
|
|
|
|
2022-05-01 14:38:05 +00:00
|
|
|
final String title = getTextFromObject(segmentJson.getObject("title"));
|
|
|
|
if (isNullOrEmpty(title)) {
|
|
|
|
throw new ParsingException("Could not get stream segment title.");
|
|
|
|
}
|
2020-12-12 09:24:29 +00:00
|
|
|
|
2022-05-01 14:38:05 +00:00
|
|
|
final StreamSegment segment = new StreamSegment(title, startTimeSeconds);
|
|
|
|
segment.setUrl(getUrl() + "?t=" + startTimeSeconds);
|
|
|
|
if (segmentJson.has("thumbnail")) {
|
|
|
|
final JsonArray previewsArray = segmentJson
|
|
|
|
.getObject("thumbnail")
|
|
|
|
.getArray("thumbnails");
|
|
|
|
if (!previewsArray.isEmpty()) {
|
|
|
|
// Assume that the thumbnail with the highest resolution is at the last position
|
|
|
|
final String url = previewsArray
|
|
|
|
.getObject(previewsArray.size() - 1)
|
|
|
|
.getString("url");
|
|
|
|
segment.setPreviewUrl(fixThumbnailUrl(url));
|
2020-12-12 09:24:29 +00:00
|
|
|
}
|
|
|
|
}
|
2022-05-01 14:38:05 +00:00
|
|
|
segments.add(segment);
|
2020-12-12 09:24:29 +00:00
|
|
|
}
|
|
|
|
return segments;
|
|
|
|
}
|
2020-12-20 18:54:12 +00:00
|
|
|
|
|
|
|
@Nonnull
|
|
|
|
@Override
|
|
|
|
public List<MetaInfo> getMetaInfo() throws ParsingException {
|
2023-12-07 19:57:44 +00:00
|
|
|
return YoutubeMetaInfoHelper.getMetaInfo(nextResponse
|
2022-05-01 14:38:05 +00:00
|
|
|
.getObject("contents")
|
|
|
|
.getObject("twoColumnWatchNextResults")
|
|
|
|
.getObject("results")
|
|
|
|
.getObject("results")
|
|
|
|
.getArray("contents"));
|
2020-12-20 18:54:12 +00:00
|
|
|
}
|
2021-06-06 16:32:07 +00:00
|
|
|
|
2022-01-15 16:25:00 +00:00
|
|
|
/**
|
|
|
|
* Enable or disable the fetch of the Android client for all stream types.
|
|
|
|
*
|
|
|
|
* <p>
|
|
|
|
* By default, the fetch of the Android client will be made only on videos, in order to reduce
|
|
|
|
* data usage, because available streams of the Android client will be almost equal to the ones
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
* available on the {@code WEB} client: you can get exclusively a 48kbps audio stream and a
|
|
|
|
* 3GPP very low stream (which is, most of times, a 144p8 stream).
|
2022-01-15 16:25:00 +00:00
|
|
|
* </p>
|
|
|
|
*
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
* @param forceFetchAndroidClientValue whether to always fetch the Android client and not only
|
|
|
|
* for videos
|
2022-01-15 16:25:00 +00:00
|
|
|
*/
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
public static void forceFetchAndroidClient(final boolean forceFetchAndroidClientValue) {
|
|
|
|
isAndroidClientFetchForced = forceFetchAndroidClientValue;
|
2022-01-15 16:25:00 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* Enable or disable the fetch of the iOS client for all stream types.
|
|
|
|
*
|
|
|
|
* <p>
|
|
|
|
* By default, the fetch of the iOS client will be made only on livestreams, in order to get an
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
* HLS manifest with separated audio and video which has also an higher replay time (up to one
|
|
|
|
* hour, depending of the content instead of 30 seconds with non-iOS clients).
|
2022-01-15 16:25:00 +00:00
|
|
|
* </p>
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
*
|
2022-01-15 16:25:00 +00:00
|
|
|
* <p>
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
* Enabling this option will allow you to get an HLS manifest also for regular videos, which
|
|
|
|
* contains resolutions up to 1080p60.
|
2022-01-15 16:25:00 +00:00
|
|
|
* </p>
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
*
|
|
|
|
* @param forceFetchIosClientValue whether to always fetch the iOS client and not only for
|
|
|
|
* livestreams
|
2022-01-15 16:25:00 +00:00
|
|
|
*/
|
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 17:06:36 +00:00
|
|
|
public static void forceFetchIosClient(final boolean forceFetchIosClientValue) {
|
|
|
|
isIosClientFetchForced = forceFetchIosClientValue;
|
2022-01-15 16:25:00 +00:00
|
|
|
}
|
2017-03-01 17:47:52 +00:00
|
|
|
}
|