a18e98adbb
Remove copyright from original file
2025-07-13 20:59:20 +02:00
b07b3dae7c
Basic implementation of ``YoutubeStreamInfoItemLockupExtractor``
2025-07-13 20:59:20 +02:00
ed37a429d1
Use my nanojson version which has some shortcut methods
2025-07-13 20:59:20 +02:00
71b335128e
Merge pull request #1308 from watermelon42/3783_Import_Soundcloud_likes
...
[SoundCloud] Add support for likes channel tab
2025-07-13 03:16:34 -07:00
c0ff21af93
Merge pull request #1330 from litetex/speed-up-n-parameter-extraction
...
[YouTube] Speed up n parameter presence check
2025-07-12 05:10:49 -07:00
249f8f83e7
Updated mocks
2025-07-11 20:19:12 +02:00
0dfaeb20fa
Added doc for related code
2025-07-11 20:12:50 +02:00
1915dc6904
[YouTube] Allow both types of string quotes
2025-07-11 08:27:44 +02:00
a847fdb330
[YouTube] Allow lowercase global array names
2025-07-11 08:27:44 +02:00
d914e86a82
[YouTube] extract global variable for signature deobfuscation
...
Extracts the global variable used in newer signature deobfuscation functions,
such as the one found in the `59b252b9` player.
2025-07-11 08:27:44 +02:00
235dd7cc55
[YouTube] add new signature regex
...
Adds a new regex to find the signature deobfuscation function, which
works for the new `20830619` player.
Closes: https://github.com/TeamNewPipe/NewPipeExtractor/issues/1287
Ref: https://github.com/TeamNewPipe/NewPipeExtractor/issues/1287#issuecomment-2764573093
2025-07-11 08:27:44 +02:00
1b8864afb8
Merge pull request #1302 from FineFindus/fix/throttling-function-regex
...
[YouTube] add new deobfuscation function name regex
2025-07-10 23:48:08 +02:00
12f7615e4a
Updated mocks for new deobfuscation function name regex
2025-07-10 23:46:28 +02:00
5184231479
Do a quick check if the n parameter is even present before exec regex
2025-07-10 23:37:54 +02:00
26bcfcdeaf
[YouTube] Detect members first and members only shorts
...
This fixes view count extraction error for such content, as the view
count is replaced by a text describing the availability of the short.
Also made YoutubeShortsLockupInfoItemExtractor package private, as it
doesn't need to be public.
2025-07-09 18:05:06 +02:00
5e1a1d51b6
Handle liked playlists using the `SoundcloudPlaylistInfoItemExtractor`
2025-07-08 18:44:51 +03:00
22f7cd4a6f
Merge pull request #1309 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.21.1
...
Bump org.jsoup:jsoup from 1.19.1 to 1.21.1
2025-07-08 05:08:08 -07:00
f06b01f036
Bump org.jsoup:jsoup from 1.19.1 to 1.21.1
...
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup ) from 1.19.1 to 1.21.1.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.19.1...jsoup-1.21.1 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-version: 1.21.1
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-07-07 21:00:47 +00:00
8f28ab3fea
Bump com.google.code.gson:gson from 2.12.1 to 2.13.1
...
Bumps [com.google.code.gson:gson](https://github.com/google/gson ) from 2.12.1 to 2.13.1.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/main/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.12.1...gson-parent-2.13.1 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-version: 2.13.1
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-07-07 21:00:04 +00:00
83b12ba93a
Merge pull request #1317 from AudricV/soundcloud-drm-free-aac-160k
...
[Soundcloud] Add support for AAC 160k unprotected stream and update Creative Commons stream test
2025-07-07 13:35:22 -07:00
9e0fdf969e
Merge pull request #1318 from AudricV/yt_fix-subs-count-channels-with-pronouns
...
[YouTube] Fix parsing subscribers count of channels with pronouns
2025-07-07 12:32:43 -07:00
c4dcef4d51
Merge pull request #1319 from AudricV/peertube_support-video-only-streams-properly
...
[PeerTube] Detect video-only streams
2025-07-07 12:26:27 -07:00
3042e40059
[PeerTube] Detect video-only streams
2025-07-07 21:20:49 +02:00
8bab96afd6
[YouTube] Add a test of a channel with pronouns
...
This is to test subscribers count parsing in a page header when a
channel has pronouns.
2025-07-07 19:02:10 +02:00
53757ae97d
[YouTube] Support pronouns in subs count parsing of page channel headers
2025-07-07 18:56:01 +02:00
7dc7b06bab
[Soundcloud] Update Creative Commons stream test
...
The track which was used has been deleted, so replace it with another
track from the account.
The AAC 160k unprotected stream isn't present on this track, only recent
ones seem to have it.
2025-07-07 17:15:35 +02:00
9d88d84bff
[Soundcloud] Add support for AAC 160k unprotected stream
...
This stream isn't returned in DRM-protected tracks, of course.
2025-07-07 17:14:24 +02:00
cbbc6d2504
Added the @Nullable annotaions in ListExtractor.java
2025-07-06 18:06:22 +05:30
44882b393c
Added support for importing Soundcloud likes.
2025-06-18 16:44:58 +03:00
2dbf404507
Fix javadoc typo
2025-06-06 00:44:55 +02:00
b97c6555c0
[YouTube] add new deobfuscation function name regex
...
Adds a new regex to find the name of the deobfuscation function for throttling,
which works for the `59b252b9` player.
Closes: https://github.com/TeamNewPipe/NewPipeExtractor/issues/1289
2025-05-26 22:33:36 +02:00
fec89a2cf1
[YouTube] Fix extraction of continuations in initial playlist responses
2025-05-24 11:05:24 +02:00
f35874e76f
[YouTube] Fix crash on SABR-only player responses, do not use WEB client
2025-05-06 20:02:36 +02:00
4f94950feb
Bump org.jsoup:jsoup from 1.18.3 to 1.19.1
...
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup ) from 1.18.3 to 1.19.1.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.18.3...jsoup-1.19.1 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-03-16 20:36:54 +00:00
de4e27d055
Fix tests execution
...
See https://github.com/gradle/gradle/issues/32534#issuecomment-2678767645
2025-03-15 17:05:56 +01:00
d718f50e34
Fix compilation + Cleanup
2025-03-05 17:29:33 +01:00
4f16112d6d
Use Files methods in tests.
2025-03-05 17:23:45 +01:00
6bf0110e38
Merge pull request #1256 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.8.0
...
Bump org.mozilla:rhino from 1.7.15 to 1.8.0
2025-03-05 17:05:18 +01:00
da1e0e72b0
Consolidate Rhino Version
2025-03-05 16:20:50 +01:00
0c3ac0a308
Rework TokenStream
2025-03-05 16:07:05 +01:00
77ee25e3b6
Fix typo
2025-03-05 15:55:46 +01:00
907fc2ac52
Fix compile problems and optimize TokenStream
2025-03-05 15:45:08 +01:00
bcf24def8b
Cleanup test
2025-03-05 15:44:57 +01:00
470a719861
Fix compile problems in JavaScript class
2025-03-05 15:44:46 +01:00
b376539062
Also use rhino engine
2025-03-05 15:44:33 +01:00
2d3342bf56
Improve tests
2025-03-05 15:44:25 +01:00
7b5d41a741
Bump org.mozilla:rhino from 1.7.15 to 1.8.0
...
Bumps [org.mozilla:rhino](https://github.com/mozilla/rhino ) from 1.7.15 to 1.8.0.
- [Release notes](https://github.com/mozilla/rhino/releases )
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/mozilla/rhino/commits )
---
updated-dependencies:
- dependency-name: org.mozilla:rhino
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-02-17 21:11:54 +00:00
c4056f5d5c
Fix tests
...
Upstream changes are described here: https://github.com/jhy/jsoup/issues/1278
They are effectively irrelevant as YT removes the ``"``/attributes anyway.
2025-02-17 22:09:44 +01:00
7e21002838
Bump org.jsoup:jsoup from 1.17.2 to 1.18.3
...
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup ) from 1.17.2 to 1.18.3.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.17.2...jsoup-1.18.3 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-02-17 21:06:55 +00:00
b1ecb6882f
Merge pull request #1277 from litetex/fix-tests
...
Fix tests and cleanup other stuff
2025-02-15 15:23:27 +01:00
e4081243e7
Improve documentation
...
> > I think you can use YoutubeParsingHelper.getTextFromObject()
> That was used there before however I did not choose to use it again.
>
> ``runs`` returns an array with 3 items containing:
> 0: ``<artistname>``
> 1: ``•``
> 2: ``<subscriberCount>``
>
> There is 2 problems with that:
> 1. It's inefficient for extracting the subscription count
> 2. If for some reason the artist now has a name like "2k" the extraction malfunctions
See also: https://github.com/TeamNewPipe/NewPipeExtractor/pull/1277#discussion_r1955148742
2025-02-15 15:02:30 +01:00
677c048f1b
Only include ``jsr305`` and not complete spotbugs
...
Only ``javax.annotation.Nonull`` or ``javax.annotation.Nullable`` are currently in used which means that spotbugs can and should be removed.
See https://github.com/TeamNewPipe/NewPipeExtractor/issues/1278 for details
2025-02-15 14:23:33 +01:00
3ad9f17c53
Revert potentially breaking change
...
I expected this class to be located in ``youtube`` as it mostly contains YT specific playlist types, but this assumption was wrong.
See also: d9caa90d23 (r1955161108)
2025-02-13 21:24:05 +01:00
d9caa90d23
Fix okhttp deprecation
2025-02-12 22:29:06 +01:00
4c3918ac4c
Removed unused code
2025-02-12 22:09:03 +01:00
2f2e2df5aa
Add recorded mock data
2025-02-12 22:01:17 +01:00
daac243fc4
Use camelCase
2025-02-12 22:00:54 +01:00
69d5bf2a89
YouTube 100% Mock coverage
2025-02-12 21:56:02 +01:00
3b34b82e0f
Downloader: Don't force IOException
...
Use ``UncheckedIOException`` instead
This no longer forces one to always write ``throws IOException`` in tests
2025-02-12 21:46:45 +01:00
177262485a
Make all YT tests in MOCK mode use mock data
2025-02-11 22:19:53 +01:00
cf5df3fcff
Improve rate limiting
...
* Bandcamp also has rate-limiting (but a lot more req are allowed than with YT)
* Shorten default warm up time
* Fix typo
2025-02-11 21:50:38 +01:00
287123b0fd
Add rate limiter with default cold factor
2025-02-11 21:49:48 +01:00
4d7df1486c
Fix YT artist getSubscriberCount extraction
2025-02-11 21:35:19 +01:00
25e4a6f3cf
YT: Suggestion test is working again
2025-02-11 21:24:39 +01:00
ee6bc846ce
Re-Enable test as it's no longer broken
2025-02-11 21:17:46 +01:00
fa99003077
Fix duplicated ;
2025-02-11 21:16:49 +01:00
1966ad1e52
Use new Bandcamp autocomplete api
2025-02-11 21:16:30 +01:00
59a2f1aa18
No random errors
2025-02-11 21:10:31 +01:00
9f94a2912d
Remove recording retry/throttle as this is now handled in a more general way
2025-02-11 20:47:14 +01:00
460e0f77e6
Update test dependencies
2025-02-11 20:46:45 +01:00
351174a85f
Enforce modern TLS1.2+
...
This fixes the following problem:
Bandcamp/their Fastly CDN fails with 403 responses when running tests
* on Windows
* with Java 17+
* and OkHttp's APPROVED_CIPHER_SUITES (default) are used
2025-02-11 20:43:55 +01:00
ad5ca4eced
Backport/Sync code from NewPipe implementation
2025-02-09 22:06:41 +01:00
c892cfaafa
Fix PeerTube tests
2025-02-09 21:40:05 +01:00
e1b94c4148
Move into dedicated package
2025-02-09 20:57:21 +01:00
f1b90985da
Remove outdated comment
2025-02-09 20:45:17 +01:00
1a9df85213
MetaInfo Search is was removed or there is none active anymore
...
https://github.com/TeamNewPipe/NewPipeExtractor/issues/1274
2025-02-09 20:38:38 +01:00
61ef1699f8
Fix code style
2025-02-09 20:37:59 +01:00
d746d1b288
Disable irrelevant test
2025-02-09 20:37:50 +01:00
271a4656da
Add missing override annotations
2025-02-09 20:22:59 +01:00
51586c0957
Disable YT Shorts UI
...
https://github.com/TeamNewPipe/NewPipeExtractor/issues/1273
2025-02-09 20:21:52 +01:00
451bb1315c
YouTube channel mixes no longer exist
...
It looks like YT removed this around autumn 2024
Other mentions: https://www.reddit.com/r/youtube/comments/1dhuvyo/channel_mixes_disappeared/
2025-02-09 20:15:08 +01:00
ee7d8b042f
Channel has a new name
2025-02-09 20:08:49 +01:00
ddc7f3e33a
Fix unexpected error due to malformed url
2025-02-09 20:08:40 +01:00
6e82a80b26
Disabled age restricted test as it's currently not working
2025-02-09 20:08:29 +01:00
f0ed76c8c6
Implement rate-limiting
2025-02-09 19:57:04 +01:00
96911ae2a4
[YouTube] Fix usage of WEB client headers for all HTML5 URLs in DASH creators
...
Also use TVHTML5 user agent for requests from this client in these
DASH manifests creators.
2025-02-05 09:56:53 +01:00
c48d449853
[YouTube] Add ability to get TVHTML5 user agent used
...
Also make YoutubeParsingHelper.getOriginReferrerHeaders public, in
order to be used by other extractor classes and improve the name of a
parameter of YoutubeParsingHelper.getVisitorDataFromInnertube.
2025-02-05 09:56:52 +01:00
0952431121
[YouTube] Move InnertubeClientRequestInfo creations in class' methods
...
This commits provides methods to get InnertubeClientRequestInfo
instances, which can be used by extractor clients to get visitor data
to pass to PoTokenProvider implementations using YoutubeParsingHelper.
Ability to create custom instances has been removed, but returned
objects can be modified. This is what YoutubeStreamHelper now uses to
set the visitorData property.
2025-02-05 09:56:52 +01:00
4644e1744b
[YouTube] Add signatureTimestamp argument to TVHTML5 client requests
...
This argument, which has been forgot, is required to get valid
streaming URLs with this client.
2025-02-05 09:56:52 +01:00
61f67854ed
[YouTube] Get visitorData for player requests if not provided and do some fixes
...
visitorData are get using InnertubeClientRequestInfo and
YoutubeParsingHelper.prepareJsonBuilder, which is replacing the
corresponding method in YoutubeStreamHelper, removed in this commit.
Also fix some bugs like JsonBuilder usages in some places and remove
HLS manifest filtering for the iOS client, as we still use for now
the full player response, in the case having streams requiring poTokens
after some time as a last resort is useful.
2025-02-05 09:56:52 +01:00
d08331dbcf
[YouTube] Add ability to use the guide endpoint to get a visitorData
...
Some clients like TVHTML5 are not allowed on the visitor_id endpoint
(with this client, a 400 HTTP response is returned with a precondition
check failed error).
Also disable pretty printing for these requests, like we do for others.
2025-02-05 09:56:52 +01:00
862a607fc6
[YouTube] Add IOS and ANDROID client IDs in ClientsConstants
2025-02-05 09:56:52 +01:00
6533a333fa
[YouTube] Add ability to get a visitorData from InnerTube
2025-02-05 09:56:52 +01:00
9e45c8001a
[YouTube] Do not send a visitorData for every InnerTube request
...
As YouTube is disabling ability to use a random visitor ID in a
visitorData on player requests and BotGuard challenges, it
shouldn't matter if we use a random one or not for other
request types.
2025-02-05 09:56:52 +01:00
94541d2d94
[YouTube] Add utility data class to store client and device info
...
This new class, InnertubeClientRequestInfo, composed of two mutable
subclasses, ClientInfo and DeviceInfo, allows to store client and
device info in a better way, without requiring to pass more than 10
method parameters like in YoutubeStreamHelper currently.
Mutability has been added in order to allow changing some fields
easily, especially visitorData.
2025-02-05 09:56:52 +01:00
9333d7fcdc
[YouTube] Update DASH manifest creation clients' handling
...
- Use POST requests with the same body as official clients do;
- Update methods checking the client streaming URLs come from:
- Replace TVHTML5_SIMPLY_EMBEDDED_PLAYER by TVHTML5;
- Add WEB_EMBEDDED_PLAYER.
2025-02-05 09:56:52 +01:00
38e2b67cb7
[YouTube] Remove unused methods and constants in YoutubeParsingHelper
2025-02-05 09:56:52 +01:00
3878696b2c
[YouTube] Add support for poTokens, refactor player clients' fetching
...
Also improve detection of age-restricted statuses in playability error
messages returned by the service and provide version 7 of DASH
manifests.
2025-02-05 09:56:52 +01:00
9d2b840cfb
[YouTube] Add utility class to handle player requests fetching
...
This internal class, YoutubeStreamHelper, has to be public in order to
be used by subpackages of the service's root one.
It supports poTokens, HTML5 signatureTimestamp property, embed context
and multiple InnerTube clients. It is meant to replace the
corresponding methods in YoutubeParsingHelper, in order to reduce the
class' size, code duplicates and improve its readability.
It adds a new way to get age-restricted videos' streams, the only ones
which are playable in YouTube embeds, which is not very common.
2025-02-05 09:56:52 +01:00
1df0267440
[YouTube] Make client and origin HTTP headers methods package-private
...
This allows their internal usage in an upcoming new class to be used on
other InnerTube clients than the WEB one.
2025-02-05 09:56:52 +01:00
litetex