ytdl_hook fails due to large amounts of JSON data

[Asday]

Important Information

Provide following Information:

• mpv version

mpv 0.35.1-dirty Copyright © 2000-2023 mpv/MPlayer/mplayer2 projects
 built on Wed Mar 22 08:19:26 2023
FFmpeg library versions:
   libavutil       58.2.100
   libavcodec      60.3.100
   libavformat     60.3.100
   libswscale      7.1.100
   libavfilter     9.3.100
   libswresample   4.10.100
FFmpeg version: n6.0

• Linux Distribution and Version

Arch. uname -a:

Linux archfx 6.2.10-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 07 Apr 2023 02:10:43 +0000 x86_64 GNU/Linux

• Source of the mpv binary

Official pacman repositories. pacman -Qi mpv:

Name            : mpv
Version         : 1:0.35.1-4
Description     : a free, open source, and cross-platform media player
Architecture    : x86_64
URL             : https://mpv.io/
Licenses        : GPL3
Groups          : None
Provides        : libmpv.so=2-64
Depends On      : alsa-lib  libasound.so=2-64  desktop-file-utils  ffmpeg  libavcodec.so=60-64  libavdevice.so=60-64
                  libavfilter.so=9-64  libavformat.so=60-64  libavutil.so=58-64  libswresample.so=4-64  libswscale.so=7-64  glibc
                  hicolor-icon-theme  jack  libjack.so=0-64  lcms2  liblcms2.so=2-64  libarchive  libarchive.so=13-64  libass
                  libass.so=9-64  libbluray  libbluray.so=2-64  libcdio  libcdio-paranoia  libdrm  libdvdnav  libdvdread  libegl  libgl
                  libglvnd  libjpeg  libjpeg.so=8-64  libplacebo  libplacebo.so=229-64  libpulse  libpulse.so=0-64  libva
                  libva.so=2-64  libva-drm.so=2-64  libva-wayland.so=2-64  libva-x11.so=2-64  libvdpau  libx11  libxext  libxinerama
                  libxkbcommon  libxkbcommon.so=0-64  libxpresent  libxrandr  libxss  libxv  luajit  mesa  mujs  libpipewire
                  rubberband  librubberband.so=2-64  shaderc  libshaderc_shared.so=1-64  uchardet  vulkan-icd-loader  wayland  zlib
Optional Deps   : yt-dlp: for video-sharing websites playback [installed]
                  youtube-dl: for video-sharing websites playback
Required By     : None
Optional For    : anki-bin
Conflicts With  : None
Replaces        : None
Installed Size  : 6.09 MiB
Packager        : Christian Hesse <eworm@archlinux.org>
Build Date      : Wed 22 Mar 2023 20:19:26 GMT
Install Date    : Fri 24 Mar 2023 20:03:58 GMT
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

• If known which version of mpv introduced the problem

Unknown - I was watching similar videos yesterday, and with no system changes, this video I found today doesn't work.

• Window Manager and version

i3wm. i3 -v:

i3 version 4.22 (2023-01-02) © 2009 Michael Stapelberg and contributors

• GPU driver and version

amdgpu. Snipped lspci -v:

09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev e7) (prog-if 00 [VGA controller])
	Subsystem: Sapphire Technology Limited Nitro+ Radeon RX 570/580/590
	Flags: bus master, fast devsel, latency 0, IRQ 69, NUMA node 0
	Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Memory at d0000000 (64-bit, prefetchable) [size=2M]
	I/O ports at b000 [size=256]
	Memory at fe400000 (32-bit, non-prefetchable) [size=256K]
	Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu

• Possible screenshot or video of visual glitches

Not relevant.

Reproduction steps

Issue persists with --no-config.

• install mpv and yt-dlp;
• ensure yt-dlp 'https://www.youtube.com/watch?v=7Ip-zi-C66s' works and cancel it once you see it starting to download;
• run mpv 'https://www.youtube.com/watch?v=7Ip-zi-C66s'

Expected behavior

• mpv draws a window and plays a video

Actual behavior

• ytdl_hook fails with an error

[ytdl_hook]  
[ytdl_hook] youtube-dl failed: failed to parse JSON data: error 
Failed to recognize file format.

Exiting... (Errors when loading file)

Log file

output.txt (large)

Sample files

Not needed.

---

I notice that the amount of JSON ytdl_hook spits to the console is immense, and ends halfway through a string, breaking the JSON format. I also notice that the log file is suspiciously close to 64MiB in size. I am of course but a stupid idiot, but I wonder if there's something in ytdl_hook that's truncating something to that size, and this video for some reason overruns that limit.

Please note that mpv 'https://www.youtube.com/watch?v=aKNQPhVucVY' works fine so it's not all youtube videos, and mpv 'https://www.youtube.com/watch?v=k-9keDEstMg' works fine too, so it's not even live streams or recent live streams or stream archives that are the issue.

[Asday]

Here's from where the error output appears to come, with result.stdout coming from here, which runs mp.command_native(), which I think then goes into C and I get lost and my head spins.

[christoph-heinrich]

The json we get for that video is huge! mpv by default limits the captured output to 64MiB.
The json from yt-dlp --no-warnings --flat-playlist --sub-format ass/srt/best -J 'https://www.youtube.com/watch?v=7Ip-zi-C66s' is 134.8MiB in size (655956 lines when formatted with jq .).
Due to the truncation it's not a valid json anymore and the parsing fails. I've tried raising the limit to 1GiB with capture_size = 1073741824 here but then I get the following error.

[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Error when loading first fragment of playlist
[lavf] avformat_open_input() failed
[lavf] Leaking 1 nested connections (FFmpeg bug).
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Cannot get fragment, missing template URL
[ffmpeg/demuxer] dash: Error when loading first fragment of playlist
[lavf] avformat_open_input() failed
[lavf] Leaking 1 nested connections (FFmpeg bug).
EDL: Could not open source file 'https://manifest.googlevideo.com/api/manifest/dash/expire/1681179828/ei/VHA0ZJyYLJLfgAf6mZaoAQ/...<trucated very long url>'.
No video or audio streams selected.

Afaict the json is so big because it's a very long stream, which results in 16381 fragments for each video format, and each of those has a pretty long url (that is almost identical to all the others).

[sfan5]

When a live stream ends, YouTube puts the transcoding of a proper recording into a queue somewhere.
In the meantime you can still watch the recording but it is merely composed of the same fragments already produced during live streaming. At 1 second per fragment and 4 hours 33 minutes this makes about 16381 fragments, all of which yt-dlp dumps into the json output including their terribly long URL.
Then there are multiple formats and other stuff so you end up with 900000+ lines and 200+ MB of JSON (when reformatted).

If you have ffmpeg compiled with libxml2 – which is needed for DASH support – you can plug the manifest URL into mpv and watch the recording anyway. (this works for me, I checked)

mpv will also do this automatically if the ytdl_hook-use_manifests option is set.
With DASH support mpv can figure out the fragments on its own and doesn't need all that junk yt-dlp outputs. Unfortunately there are some other reasons why it's disabled by default and not everyone has the support.

A possible solution I see here is an option for yt-dlp to just skip enumerating all the fragments in the JSON output when it would blow everything out of proportion. mpv could then see that e.g. fragments is null and always fall back to the DASH manifest URL.

[Asday]

Afaict the json is so big because it's a very long stream

It might be the settings Shibuya HAL was using to stream perhaps, when I try to watch the video in browser I hit constant buffering, and I've watched longer live stream archives fine in MPV before (e.g.: https://www.youtube.com/watch?v=80Qcs_2zVhA, works fine as I type).

mpv will also do this automatically if the ytdl_hook-use_manifests option is set.

(default: no). It's disabled ("no") by default for performance reasons.

Is this something I - a fool - should be changing?

---

E: I've reread the following

In the meantime you can still watch the recording but it is merely composed of the same fragments already produced during live streaming.

Should I understand that to mean if I try the same repro steps tomorrow or the day after or what have you, the transcoding will have finished and a more sensible JSON output will be produced, fixing the issue? I can try to reproduce tomorrow if that would be helpful.

[sfan5]

Should I understand that to mean if I try the same repro steps tomorrow or the day after or what have you, the transcoding will have finished and a more sensible JSON output will be produced, fixing the issue?

Yes, that is likely.

[Asday]

Tested the repro steps again today, and it now works, presumably for the exact reasons mentioned by sfan5.