Handle bluray folders

[breezytm]

Hello Clinton,

Is there a way for nzbToMedia to transcode the bluray to mkv?

[clinton-hall]

I thought it did...

is this just the folder structure, or an iso?

[clinton-hall]

I have added the basics for this... but don't have anything to test it on here at the moment.
If you want to test, switch to bluray-1 branch and test it out.

[breezytm]

I will go head and switch my branch to nightly and test the changes you made. It is interesting that I had one iso that worked fine and two others failed. perhaps it could be folder structure difference. Please let me know if I should send you the folder structure of one of the failed one.

[clinton-hall]

Yes, any detaiks as to folder structure would help.

Please use bluray-1 branch, not nightly...

[breezytm]

I have not been able to switch to the latest branch but here's a log of one of the issues. It's not always the same.
https://ufile.io/kq4ez

:/opt/nzbget/downloads/completed/movie/A.Star.is.Born.2018.1080p.CEE.Blu-ray.AVC.TrueHD.Atmos.7.1-wezjhOurBits-Rakuvfinhel$ ls
A.Star.is.Born.2018.1080p.CEE.Blu-ray.AVC.TrueHD.Atmos.7.1-wezjhOurBits-Rakuvfinhel.iso
:/opt/nzbget/downloads/completed/movie/A.Star.is.Born.2018.1080p.CEE.Blu-ray.AVC.TrueHD.Atmos.7.1-wezjhOurBits-Rakuvfinhel$

The file is still an iso. It wasn't transcoded into an MKV

[clinton-hall]

Yes, the issue here is that there was a bug if you have "Concat" enabled and no .vob files are found.

This is fixed in bluray-1 branch (along with checking for bluray structure in addaition to DVD .vob structures.

You will need to switch to bluray-1 branch to test this. I can guarantee that nightly branch will not currently trasncode any bluray images.

[breezytm]

I am about to test this right now. I'll report back with my findings.

[breezytm]

What am I doing wrong?

admin@CYBERDYNE:/opt/nzbget/scripts/nzbToMedia$ sudo git checkout bluray-1
Branch bluray-1 set up to track remote branch bluray-1 from origin.
admin@CYBERDYNE:/opt/nzbget/scripts/nzbToMedia$ sudo git pull
remote: Enumerating objects: 8, done.
remote: Counting objects: 100% (8/8), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 8 (delta 2), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (8/8), done.
From https://github.com/clinton-hall/nzbToMedia
82d1ba8..0a3b2cd bluray-1 -> origin/bluray-1
cb3f61f..825b48a nightly -> origin/nightly
Updating 82d1ba8..0a3b2cd
Fast-forward
core/transcoder.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

https://ufile.io/5xze2

[breezytm]

Also can you merge the HVEC change to this branch as well. I could test them both in this use case.

[clinton-hall]

ok, I have merged h265 changes into bluray-1
You have correctly switched branch
probably worth doing another sudo git pull and then all should be ready to test

[breezytm]

What I was saying is I switched branch but it still failed. log file is attached at the bottom of my previous comment.

[clinton-hall]

ok, I see the error now. At least it isn't crashing, and now just says it cent find matching files to process.

What I need to see is just what the file structure is in that .iso....

I'll see tonight if I can download the same iso and check this to find out what it isn't matching....
are you able to email the .nzb file to [email protected]?

[breezytm]

Sure. Emailing right now

[clinton-hall]

thanks.... in about 7 hours I should be able to start testing!

Hopefully I'll get some time over the weekend to debug the bluray parsing and get this working for you.

[clinton-hall]

Does that image actually work to be played by bluray players (e.g. in kodi)?

When I use 7zip to view the contents, there is a com directory, a META-INF, and a kudu directory along with a whole stack of *.class files

There is not a single .mts file, there is no BDMV or SOURCE directory... so as it stands nzbToMedia can't do anything with this.

All searching I have done indicates that Bluray folder structure should have a BDMV and a CERTIFICATE directory at the root level.

https://en.wikipedia.org/wiki/Blu-ray#Directory_and_file_structure

So there could be a couple of things going on here:

• 1. This .iso is not a proper bluray image.
• 2. The bluray .iso is somehow encrypted and needs to be decrypted before the actual folder structure can be properly parsed.
• 3. 7zip can't read bluray .iso files correctly.

[clinton-hall]

ok... this is a limitation with 7zip.... (And I updated to v16.02)
When I mount the .iso I can view the BDMV and STREAM directories fine.

So I had another discussion somewhere in here recently about doing away with 7zip and using a python library that supports iso parsing and extraction... looks like I have some work ahead of me.

[clinton-hall]

Ok... I just tried using https://github.com/barneygale/isoparser

isoparser/source.py", line 93, in unpack_volume_descriptor
    raise SourceError("Wrong volume descriptor identifier")
isoparser.source.SourceError: Wrong volume descriptor identifier

I am wondering if we need to fall-back to mounting the iso and then parsing the contents.

[breezytm]

ok... this is a limitation with 7zip.... (And I updated to v16.02)
When I mount the .iso I can view the BDMV and STREAM directories fine.

So I had another discussion somewhere in here recently about doing away with 7zip and using a python library that supports iso parsing and extraction... looks like I have some work ahead of me.

That was me :)
#1302

[clinton-hall]

LOL... well, aren't you giving me a headache :P ?
Thanks for linking that issue back in. I knew I had looked at these...

So I can now confirm that isoparser does NOT support bluray iso files.
I have not had a chance to test https://github.com/clalancette/pycdlib yet

[breezytm]

LOL this is me helping you helping me helping everyone if that makes sense.

[breezytm]

In all fairness I don't believe that particular iso is a valid one. MakeMKV can read it though.

[clinton-hall]

What I will like do is keep the 7zip parsing (if it works, it is efficient) but then if that doesn't work, try a mount and parse (as a fall-back).

The only other things I'll need to check is if I need to try and make this smarter and actually read the various play-list files.... current methodology is to transcode all .mts files into a single video file. but that may very well see all additional materials embedded before.after the main feature movie...

[clinton-hall]

Ok. I have added the fall-back to mounting .iso files to parse (only for linux at this time).
This is very theoretical, as I am not able to specifically test this functionality (working remotely).

I would be fascinated to see the output of logging from the script against this same download (after updating to latest bluray-1 branch).

If it transcodes, the big question will be what does the video file look like? i.e. is the movie continuous with extra features added at the end, or is it all jumbled and out of order?

If it does not transcode, what does the logging show us to reveal what I have missed?

[clinton-hall]

as of this moment, using bluray-1 I am transcoding that very .iso :)
So that part is working.... once it has transcoded I'll then need to verify what the movie itself looks like

[clinton-hall]

wow... this has been transcoding for over 11 hours! The output video is only up to 3 GB!

So it appears to be working... but testing on a NAS is slow going.
I still need to verify just how the transcoded video looks and if it it continuous.

[breezytm]

Something is not right. MakeMKV does all of that in under 30 minutes for me but then again I am using a computer with decent specs. Let me use my server to test. I should post process much quicker. I am also wondering if it is transcoding the video as well since it is HVEC. I know you were supposed to merge the changes. Maybe it isn't working correctly. That could explain the longer transcoding time.

[breezytm]

Before I even redownload the file, can you please confirm that when I select MKV as the outputDefaultk, it would copy the video instead of transcoding it as requested in this thread #1592

[breezytm]

And yes I did do a pull and I did select mkv-bluray

remote: Enumerating objects: 26, done.
remote: Counting objects: 100% (26/26), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 26 (delta 11), reused 14 (delta 8), pack-reused 0
Unpacking objects: 100% (26/26), done.
From https://github.com/clinton-hall/nzbToMedia
066e93e..13523ea bluray-1 -> origin/bluray-1
bbcef52..0ba4b9d nightly -> origin/nightly
Updating 066e93e..13523ea
Fast-forward
autoProcessMedia.cfg.spec | 2 +-
core/init.py | 10 +++++++++-
core/transcoder.py | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------
nzbToCouchPotato.py | 2 +-
nzbToLidarr.py | 2 +-
nzbToMedia.py | 2 +-
nzbToNzbDrone.py | 2 +-
nzbToRadarr.py | 2 +-
nzbToSickBeard.py | 2 +-
9 files changed, 84 insertions(+), 27 deletions(-)

[clinton-hall]

ok... I am guessing this didn't mount the .iso

Can you see any files in /opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp?

what happens if you run the following command?

 mount -o loop /opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits.iso /opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp

[breezytm]

that folder temp seems to be read only

WARNING: device write-protected, mounted read-only.

[breezytm]

:sudo mount -o loop /opt/nzbget/downloads/completed/movie/Val erian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/Valerian.and .the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits.iso /opt/nzbget/downl oads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-w ezjhOurBits/temp [sudo] password for system-admin: mount: /opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray .HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp: WARNING: device write-protected, mounted read-only. :/opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp$

[clinton-hall]

so this mounts ok when you use sudo, but not otherwise?

[breezytm]

See below

`:/opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.US A.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp$ mount -o loop /opt/nzbget/downloads/completed/movie/Valerian .and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/Valerian.and.the. City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits.iso /opt/nzbget/downloads/ completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhO urBits/temp

mount: only root can use "--options" option
`

[clinton-hall]

ok... I see 2 ways of working through this:
-1. create a python mount using ctypes.... this may or may not work without using root
e.g.

import ctypes
import ctypes.util
import os

libc = ctypes.CDLL(ctypes.util.find_library('c'), use_errno=True)
libc.mount.argtypes = (ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p, ctypes.c_ulong, ctypes.c_char_p)

def mount(source, target, fs, options=''):
  ret = libc.mount(source, target, fs, 0, options)
  if ret < 0:
    errno = ctypes.get_errno()
    raise OSError(errno, "Error mounting {} ({}) on {} with options '{}': {}".
     format(source, fs, target, options, os.strerror(errno)))

mount('/dev/sdb1', '/mnt', 'ext4', 'rw')

-2. create an option for users to enter root password into config, and call sudo

[clinton-hall]

if you update to latest then edit autoProcessMedia.cfg

[General]
    sudo_password = 

enter the sudo password and then test to see if this works to mount and transcode.

[breezytm]

Before I do that, would you encrypt the password on the text .cfg file?

[breezytm]

Also how come it is working for you but not working for me? Do you nzbget or whatever tool you use running as root? I have nzbget running as nzbget user

[clinton-hall]

fair call.. I currently don't have any encryption in the .cfg file... I'll need to do some research as to how to handle that... personally, my biggest concern is around giving the script sudo access (as opposed to the sudo password being stored within the .cfg). encrypting this will prevent anyone with access to the .cfg learning the password, and also protect people when they post their .cfg here asking for help.

For me this works becasue my NAS does not accept su or sudo. NZBGet runs as admin user, but the way the busybox commands work mount and umount work when called by admin.

[labrys]

@clinton-hall From a security perspective I see this is as a very bad idea. The permissions of the user to run that command should be an environment configuration. For example you can use a combination of sudo and visudo to give passwordless access to the user for discretionary access to execute the command while not allowing other root commands. This should not be done by the application, there are too many ways it could be a security concern.

Edit: Just saw that this was related to mounting/unmounting - there's a good reason that mounting/unmounting requires root as it can be used for privilege escalation. I don't see an effective way of automating this without opening serious security concerns.

[clinton-hall]

@labrys
agreed... the 7zip methodology worked well, but the issue is 7zip does not support bluray .iso.
Likewise isoparser fails.

There has to be someway of using python to browse and extract from bluray .iso files, but this is way above my capabilities...

mount works fine for me on my NAS but obviously not on any system requiring sudo.

do you know if the ctypes library (snippet of code above that came from a sourceforge page) would allow mounting and parsing without requiring root privileges? or is that essentially going to emulate the mount command and have the same limitations?

[clinton-hall]

FWIW, I note a request on 7zip page to support BD iso. I have provided info there to see if this can be added.

[labrys]

I'm fairly certain that mounting with ctypes requires root.

[labrys]

Have you tried using rar? I'm fairly sure it can extract bluray isos.

[clinton-hall]

testing unrar at my end does not work. I have seen some indication online that winRAR does work, but unrar on linux does not.

I have provided more info to 7zip dev to see if 7zip can support UDF 2.50 and 2.60 iso files. If so, then I am confident the changes I have made will work for bluray images.

To Sum up:
-1. Bluray file structure is supported. - All video files are transoced in order (main movie followed by extra features)
-2. 7zip is called first for .iso files
-3. If 7zip failed to match DVD or Bluray file structure the script attempts to mount and then parse the folder structure.

So if mount works (some busybox systems or any system where the downloader runs as root) this will work.
Otherwise if/when 7zip is updated, this will work.

I'll remove the changes I have made for the sudo password...

[breezytm]

Sounds good. Is there a ticket on their side you can add to this thread for us to track the changes on there end.

[labrys]

@clinton-hall can you detect that the reason for failure is a bluray iso? If so then possibly raise warning informing user to manually extract?

[clinton-hall]

Ticket at 7zip:
https://sourceforge.net/p/sevenzip/discussion/45797/thread/21f213e2/

I can't currently detect bluray iso as the reason for failure. In fact the issue is that 7zip actually does produce a file list, but it isn't correct (as per thread linked above Igor has indicated this is actually parsing an internal zip file) and therefore doesn't look like the expected DVD / BluRay structure.

[clinton-hall]

@breezytm for what it is worth, I transcoded that same movie to 1080p (so it should now be playable). Remembering that this is for the full 3+ hours of bonus features etc as well... the transcoding on my NAS took about 48 hours!

[breezytm]

Hey I was out for a little while dealing with my mothers passing. Glad to see you open a case to get this included.

[clinton-hall]

@breezytm
Sorry to hear that. I hope you and your family are doing well.
As soon as I can get either 7zip that handles these, or another iso-parsing tool, I will update this further.