Testing Notes 2

[andrewfer000]

This is a new issue to start fresh from Testing Notes.

[markjfine]

Starting this new issue with:
Find out why/how Discogs is 403-ing search requests via Cloudflare. It's not being done by IP or User-Agent.
Also, find out why logos aren't showing right after being downloaded when not cached in stationLogos.json - noticed this after Discogs put the kabosh on automated cover art.

[markjfine]

While investigating the logo issue, I found a station that used 'jpeg' instead of 'jpg' for album art. The regex was only looking for jpg, png, and txt. So I knocked that one out straight away. Still investigating the other part tho.

[markjfine]

I've determined that the issue of why the logos don't show when downloaded at the start has to do with the XHDR going to 0 with no logo nor a cover downloaded yet. I'll add logic that automatically applies and displays the logo if it's downloaded, but there's no image showing at all, regardless of the XHDR setting. Could be tricky given the current way the images display.

[markjfine]

Fixed.

[andrewfer000]

Awesome! I'll test it out tomorrow.

[markjfine]

Trying to work TODO 3.a and a precursor to 4.a but reception is not cooperating. Dropping a lot of LOT and XHDR messages. This is like trying to work a radio in a SCIF. If I can't resolve my reception issues I may put everything on hold until I'm back in Virginia and I have my good antennas. This is just too painful.

[markjfine]

After a thoroughly frustrating morning, I think I got this logo thing sorted. Also managed to fix an annoying bug where the covers would show prematurely.

[andrewfer000]

I still need to do some testing, but I think there is a problem with album art that makes it take awhile for them to pop up or not pop up at all when the next song starts. I need to do more testing. Will leave an update tomorrow.

[markjfine]

If you do, you might want to un-comment some of the print statements I've added that show when covers and logos are received, when the LOT messages occur, as well as when the XHDR changes. This way you can see the events as they happen in a terminal window in real time instead of having to review a log file, well after the fact.

So you know what you'd be looking at:

1. The LOT messages are when files are received, including the travel/nav text and image/overlay files.
2. The XHDR messages show what's supposed to be displaying (cover or logo), and the LOT number of the file (logos are always a LOT number of -1 in XHDRs, regardless of the logo's actual lot number at the beginning of the file name).

It's possible that I may have missed a condition where the cover may be delayed, but have seen where covers are received before the XHDR changes while a different cover is showing, while the XHDR says a logo should be displaying, and well after the XHDR has changed to the correct LOT number for the cover when no cover existed.

In the first two cases, it immediately changes to the cover when the XHDR changes using the correct LOT number. However, if the XHDR changes to the correct LOT number before the cover has actually downloaded, it won't show until the station sends another cover (LOT message).

The delay you're seeing may be because:

1. The station hadn't sent the cover or failed to switch the XHDR away from the logo (or previous cover) in time - have seen those instances happen a lot.
2. nrsc5 had a problem with the cover file and threw it out - I assume this is a possibility. The app also has some error checking based on file size against what the LOT message says it should be.
3. nrsc5 didn't echo the appropriate LOT or XHDR status message, either due to a signal dropout or the station failed to send it. When that happens the app just thinks it never happened.

I have seen events 1 & 3 occur yesterday many times, which was frustrating the hell out of me - I assume 2 is additionally possible due to poor reception here. That's what prompted me to start adding the LOT number checks - to prevent showing new covers early, or old covers after the XHDR switched.

Previously, all of the gui apps just detected the XHDR switch to a cover and displayed whatever the last received cover was - not really the way it's supposed to work, and why I want to split cover and logo handling completely. It's just too messy.

This will also allow me to cache covers received for other streams, like I'm doing for logos now, as well as reuse covers in case the LOT numbers change. So if you happen to change streams it will show the appropriate cover and not the logo, if that's what's playing and the cover is available. Opens up a whole lot of other possibilities for reorganizing the cover art by station and the actual filename. Will also allow me to index the LOT number metadata for reference so the aas directory doesn't constantly fill up with duplicate images that have to be pruned.

[andrewfer000]

Okay so I figured out what was wrong and it's the stupidest thing ever. So, when I pull down a new version of nrsc5-dui from the repo, I rename the old version do nrsc5-dui-oldX and I extract the old one to the new location. Now due to some strange file system stuff, I need to restart my terminal for it to notice changed files (or else it references the old ones). Once I noticed that the problem was caused by that I restarted the terminal and the application and it seems like everything is working now.

This is a great fix! It's too bad the album art downloads from Discogs does not work anymore.

[markjfine]

How are you pulling them down?
In theory, you should just need to go to the root of where nrsc5-dui.py is then run a git pull of the main branch to update it. The versioning is kept intact, so you can always back the code out to a point if there's a problem, just like any other cvs.

I think the command is something like:
'git pull origin main'

[andrewfer000]

I download the zip file and extract it to my desktop after renaming the old version. Then I copy the new folder to my desktop. After that I copy my AAS, Configs, and Maps over to the new version and that's it.

I like to keep the old versions in case something goes wrong.

[markjfine]

That's a lot of work.
I run incremental backups every hour so if something goes wrong with the cfg, aas, or other data I can easily recover.

[markjfine]

btw: cheers for the reddit post. I don't do reddit, personally, but It also got picked up by rtl-sdr.com. Seems like it's getting a good share of viz.

[markjfine]

Note: Discogs uses Cloudflare for their content delivery. Apparently they have this new super duper bot buster. I have a feeling automated cover downloads is going to be a nice idea that can't be fully implemented because of things like super bot busters. Again, from a site security aspect and knowing what nasty things are out there, I can't blame them for doing it. I'd do it in a heartbeat.

Worthwhile to note: I think what it's doing may be similar to how WordFence determines it's a bot or not. If it just scrapes the html file (what this is doing), it's a bot. If it pulls down the associated css, js, and images, it's a browser. They don't seem to give a monkey's about the user-agent (and I block an f-ton of those). I'm just wondering if it's worth it to basically build a browser that caches all that with no display just to get past the bot buster. Would mean creating a tmp directory to save all that nonsense to, then deleting the piece parts when I was done with it. Seems like a wasteful endeavor for very little payoff.

[andrewfer000]

That's true. It's possible and it's a nice feature but it's not super necessary since the broadcaster usually supplies the album art anyway. That feature can be put on the back burner for now and maybe focusing on the smaller Quality of Life stuff should come first. I'm not sure if you noticed, but this program has been getting more attention recently. That's because I shared it on Reddit. I'm hoping this gets us some help in testing and implementing new features as time goes on.

[markjfine]

Looking at MusicBrainz as a possible replacement for Discogs. They also have an API that generates json, so I don't have to go through some ugly kabuki dance parsing web pages.
...and there's a Python package already written for it.

[markjfine]

Tempted to try and implement this now, but I know it'll be painful because of reception issues.

[andrewfer000]

I have good reception so I can do testing. MusicBrainz is a good service since it's like a "Wikipedia for music" and I'm assuming the information is free (as in Wikipedia free) and with a real API things would be more stable and controllable.

I feel like after the update you just did for the station art/album art makes the program feel so much more complete and better.

Also, I tested the program in the car today and let's just say there may be some problems but it might just had to do with the bad signals, my crap R820D based rtl-sdr, and the fact the program was trying to reach Discogs without an internet connection which may have lead to the program freezing I was experiencing.

[markjfine]

Think I'm going to disable all of the Discog stuff it until I get it re-wired for MB. Would rather not get a complaint that it doesn't work.

[markjfine]

Went ahead and added the MusicBrainz stuff as much as possible.

Had trouble with the part that actually downloads the image and saves it so used the MB URL in the old save routine.

Once I get that last part fixed, both urllib3 and OpenSSL will no longer be needed.

However, will still have to pip3 install musicbrainzngs for the new stuff to work.

[andrewfer000]

I like the new album art service so much better. You never cease to amaze me with your skill. Thank you for doing so much work for this project and giving it new life and features. I am super grateful for it.

Also, If you don't mind could you modify the console output to send a warning if the SDR gets disconnected. It would be a huge help when I'm testing different devices and cables.

[markjfine]

Cheers. With me it's really more determination and luck than skill.

The disconnect should turn the pilot red when nrsc5 generates a lost device message. I've noticed that sometimes these radios get locked up for no reason. Unfortunately there's no real good way to detect those.

[andrewfer000]

Hey so after doing a lot of testing today I learned a few things,

1. My SDR or Laptop's USB port is complete shit.
2. The album art download system needs a little tweaking. It cannot find some songs that I know should be in the DB since they are older and even if album art is not available it does not always display the one downloaded from the station.
3. Since Musicbrainz is a database and has a real API for python is there a way to gather the information about the album and genre for a song when the station does not provide it just from the song title and artist?

[andrewfer000]

Hey so I think I found a pattern, the downloaded album art only seems to work when the song was released as a single. I believe the best way to fix this is to get the album name from the DB for the song and use that version of the art. I hope that's possible but I'm not sure.

Edit: Yikes, Looking through the Musicbrainz database is difficult. Is there just a way to search a song + artist and return the first album the song showed up on? Like dang this is complicated!

Also I have seemed fo find an issue on the radio station's side. For example, they put the artist as "Blink 182" when it is "Blink-182" in the database (and that's how the band name is spelled). I think things like that are also causing issues with the Album art down-loader.

[markjfine]

The search parameters need to be tweaked. I just noticed a few things in the search I'm doing, which is different than the way the web site does it. Am currently looking at that while I rewicker the download part, which I'll post in a few. Right now it's focusing on singles, as you surmised. I thought I was looking at all releases that contained the title, which apparently is not the case.

Yes, I can pull the other info easily. It's returned within the json that I get.

[andrewfer000]

That's awesome! I hope everything works in your favor, Now I understand how great yet much of a pain these systems and databases can be

Tbh I find it kind of funny how this program written in python and uses an rtl-sdr works better than most $30,000+ cars with HD Radios. That just puts a smile on my face xD.

[andrewfer000]

so I tested your latest code and its still mostly the same. I know that this feature is going to take a lot of time and testing to prefect.

That being said, I think it would be smart to try and get the album name first (if not provided by the station) and use that as part of the search if possible. Obviously the program does not have eyes of it's own so it's not going to be prefect but something is better than nothing at this point in the feature's development.

[markjfine]

I know, I just uploaded a better query. That one just removed the urllib3 nonsense.
Getting the album name is trickier than you think... found a better way.

[markjfine]

So, next thing I'm going to do is clean this puppy up. It's now got more commented code than operational code.

[markjfine]

"It is what it is". I added the Strict option, set to On as default, provided the DL option is set. So, we'll see. Can never have too many levers to pull.

Am going to start posting my latest, since it looks like the cached album titles work... Just remember that it won't cache if you have existing album art, so you may want to move them off to the side if you want to keep them.

[markjfine]

done.

[andrewfer000]

Is there a way to detect if an album art sets or not? I wonder if there is a way to implement a system where it starts with the most strict queries and it that fails (on the occasions that it does) it goes to a less strict query (I found the best thing to do is take away the type=Album requirement) but I only want that to happen if the first query does not set an album art.

[markjfine]

If nothing is downloaded, imgSaved is False. You could take the whole thing from after
try: imgSaved=False
to just before
# If no match use the station logo

Move that into a def block with the appropriate arguments and returning things like imgSaved, the ['Album'] and ['Genre'] and just call that twice: once with Strict set, then if it returns False, run it again with Strict set to False.

I would think you'd want to keep the type=Album requirement, but setting Strict=False makes the priority on title or artist rather than type anyway. At least that's what I've found.

[markjfine]

If you want to emulate that, run it with the Strict engaged. Wait until something comes up blank. Click Stop, change the Strict setting, then hit Play. That will trigger the search again when the track info updates.

[markjfine]

Added a new TODO 4.f for a more manual approach to this.
Would prefer a drag/drop thing for the image URL, but that might take some doing.

[andrewfer000]

After this update, things seem to be working better. I'll leave it alone for now but if I notice problems I'll start messing around with it again.

[andrewfer000]

If you want to emulate that, run it with the Strict engaged. Wait until something comes up blank. Click Stop, change the Strict setting, then hit Play. That will trigger the search again when the track info updates.

I tried this and it works for more songs than you may think. I feel like the method of going from the strict to non-strict quarries should be a part of the code since it is a good catch. If you don't want to implement it I'll try and do it but it wont be pretty and you'll probably have to fix it up but hey. I'll try. The main issue with the current implantation is that when strict completely is disabled, it breaks more than fix for most songs.

My end goal with this is to have every song have an album art from the database if possible even if its just from an iTunes special or compilation yet ensure that the song gets the best and most accurate artwork available.

[markjfine]

Let me see what I can do

[markjfine]

Changed routine to run with Strict off if it first runs with it on and doesn't find anything.
I've left the print statements uncommented so you can see what's going on underneath the hood.

Edit: Also noted that it was adding the Album data, but not the Genre (even if it came up blank), so I added that just in case.

[andrewfer000]

I made some modifications to the code to add more filters, I am testing now and if all goes well I'll share it here.
Also, I am having some issues with station art,

1. If station art is downloaded from the internet, it will still default to the one the station set
2. MusicBrainz searches occur every time the Title and Artist switches back to the station info, The best way to fix this is to set up downloaded logos like artwork, as in using the Artist_-_Title format for the logo file.

[markjfine]

1. Yeah, this was really meant for stations that don't have logos. As you've seen, trying to roll your own logo could have implications.

2. Actually, that could cause a problem because you won't be able to differentiate between logos and cover art, and I'm in the process of trying to separate the two.

Another issue is that not every station provides the same thing. Some run ads and the title/artist changes the text to match the ad.

Probably a better way to keep it from doing a search is to use the XHDR flag, and only do searches when lastXHDR is set for a cover, not a logo.

Having to take care of a few non-coding things atm, so may not get to adding this until a bit later.

[andrewfer000]

Sounds good take as long as you need. I never ran into a station that changes text for ads but I wouldn't be surprised if some stations did that. I am just trying to find solutions based on what I know so you probably know a better solution.

As for my enhanced filter, it works well and is able to catch pretty much everything but I'm still testing it and like I said, I never used git before so I'll just post my code here when it's ready.

[markjfine]

Of course this XHDR thing only works if the station cooperates and sends the correct messages.
If they change the text but XHDR doesn't change to "0", they're basically saying that they want the logo to show, not a cover.

[andrewfer000]

Starting at around line 434 and the "If not, get it from MusicBrainz" comment. You'll understand. If you have any issues I'll share my whole nrsc5-dui.py file. Overall the filters work, I've gotten most things with the 1st filter but I got some others with filters 2,5, 6, and 7 as well.

 # if not, get it from MusicBrainz
        else:
            try:
                imgSaved = False
                i = 1

                while (not imgSaved):
                    print()
                    print("searching for {} - {}".format(searchArtist,newTitle))
  
                    try:
                      if (i==1 and (not imgSaved)): 
                        setStrict = True
                        result = musicbrainzngs.search_recordings(strict=setStrict, artist=searchArtist, recording=newTitle, type='Album', status='Official')
                        print("Running through filter 1")
                      if (i==2 and (not imgSaved)):
                        setStrict = False
                        result = musicbrainzngs.search_recordings(strict=setStrict, artist=searchArtist, recording=newTitle, type='Album', status='Official')
                        print("Running through filter 2")
                      if (i==3 and (not imgSaved)):
                        setStrict = True
                        result = musicbrainzngs.search_recordings(strict=setStrict, artist=searchArtist, recording=newTitle, status='Official')
                        print("Running through filter 3")
                      if (i==4 and (not imgSaved)):
                        setStrict = False
                        result = musicbrainzngs.search_recordings(strict=setStrict, artist=searchArtist, recording=newTitle, status='Official')
                        print("Running through filter 4")
                      if (i==5 and (not imgSaved)):
                        setStrict = True
                        result = musicbrainzngs.search_recordings(strict=setStrict, artist=searchArtist, recording=newTitle, type='Album')
                        print("Running through filter 5")
                      if (i==6 and (not imgSaved)):
                        setStrict = False
                        result = musicbrainzngs.search_recordings(strict=setStrict, artist=searchArtist, recording=newTitle, type='Album')
                        print("Running through filter 6")
                      if (i==7 and (not imgSaved)):
                        setStrict = True
                        result = musicbrainzngs.search_recordings(strict=setStrict, artist=searchArtist, recording=newTitle)
                        print("Running through filter 7")
                      if (i==8 and (not imgSaved)):
                        setStrict = False
                        result = musicbrainzngs.search_recordings(strict=setStrict, artist=searchArtist, recording=newTitle)
                        print("Running through filter 8")
                        
                    except:
                        print("MusicBrainz recording search error")
                        #pass
    
                    if (result is not None) and ('recording-list' in result) and (len(result['recording-list']) != 0):
                        print("got recording search result with {} recordings".format(len(result['recording-list'])))
    
                        # loop through the list until you get a match
                        for (idx, release) in enumerate(result['recording-list']):
                            #print(release)
                            resultID = self.check_value('id',release,"")
                            resultScore = self.check_value('ext:score',release,"0")
                            resultArtist = self.check_value('artist-credit-phrase',release,"")
                            resultTitle = self.check_value('title',release,"")
                            resultGenre = self.check_value('name',self.check_value('tag-list',release,""),"")
                            scoreMatch = (int(resultScore) > 90)
                            artistMatch = (newArtist.lower() in resultArtist.lower())
                            titleMatch = (newTitle.lower() in resultTitle.lower())
                            recordingMatch = (artistMatch and titleMatch and scoreMatch)
    
                            # don't bother dealing with releases if artist, title and score don't match
                            resultStatus = ""
                            resultType = ""
                            resultAlbum = ""
                            resultArtist2 = ""
                            releaseMatch = False
                            imageMatch = False
                            if recordingMatch and ('release-list' in release):
                                print("got release list with {} releases".format(len(release['release-list'])))
                                for (idx2, release2) in enumerate(release['release-list']):
                                    #print(release2)
                                    imageMatch = False
                                    resultID = self.check_value('id',release2,"")
                                    resultStatus = self.check_value('status',release2,"Official")
                                    resultType = self.check_value('type',self.check_value('release-group',release2,""),"")
                                    resultAlbum = self.check_value('title',release2,"")
                                    resultArtist2 = self.check_value('artist-credit-phrase',release2,"")
                                    typeMatch = (resultType in ['Single','Album','EP'])
                                    statusMatch = (resultStatus == 'Official')
                                    albumMatch = (not self.check_terms(resultAlbum, albumExclude))
                                    #artistMatch2 = (resultArtist2 != "") and (not ('Various' in resultArtist2))
                                    artistMatch2 = (not ('Various' in resultArtist2))
                                    releaseMatch = (artistMatch2 and albumMatch and typeMatch and statusMatch)
                                    print("#{} {}: Track: {} - {}, {}: {} - {}, {} {}% {}".format(idx, resultStatus, resultArtist, resultTitle, resultType, resultArtist2, resultAlbum, resultID, resultScore, resultGenre))                    
                                    # don't bother checking for covers unless album, type, and status match
                                    if releaseMatch:
                                        imageMatch = self.check_musicbrainz_cover(resultID)
                                    if (releaseMatch and imageMatch and ((idx2+1) < len(release['release-list']))):
                                        break
    
                            if (recordingMatch and releaseMatch and imageMatch):
     
                                # got a full match, now get the cover art
                                print("Found {}: Track: {} - {}, {}: {} - {}, {} {}% {}".format(resultStatus, resultArtist, resultTitle, resultType, resultArtist2, resultAlbum, resultID, resultScore, resultGenre))
                                if self.save_musicbrainz_cover(resultID,saveStr):
                                    self.coverImage = saveStr
                                    imgSaved = True
                                    if (self.streamInfo['Album'] == ""):
                                        self.streamInfo['Album']=resultAlbum
                                    if (self.streamInfo['Genre'] == ""):
                                        self.streamInfo['Genre']=resultGenre
                                    self.coverMetas[baseStr] = [self.streamInfo["Title"],self.streamInfo["Artist"],self.streamInfo["Album"],self.streamInfo["Genre"]]
                                print("recording search succeeded")
                                  
                            if (imgSaved) and ((idx+1) < len(result['recording-list'])) or (not scoreMatch):
                               break

                    i = i + 1
                    # if we got an image or all the filters were run through, there is no reason to keep looking.
                    if (imgSaved or i > 9):
                        break

[markjfine]

Don't mean to impede your initiative (which is commendable), but I'm concerned that if you have a number of people running this concurrently with that many potential queries, aside from a performance hit for people with slow, high-lag internet connections, the volume of hits could either get the app severely throttled or banned quicker than Discogs. They were pretty adamant on the site about that part.

So I have to ask if you've profiled this as far as a worst case turnaround time. I noticed quite a bit of delay just using two queries but it's not really noticeable when the print statements are commented out. I'm curious what 8 does before I put it in. Also, how many hits are we potentially pinging MB with for one cover?

[andrewfer000]

8 does the same thing is 7 but the strict is turned off. The number of hits is what I am worried about too. Especially since it happens every time a song ends and it switches back to the station's info. When it comes to songs however, there usually only needs to be the one query unless it's not in the first one. As long as MB has it and it is correct in the DB (and the radio station spelled everything right) then usually it won't have to go beyond filter 4 or filter 6.

Also it takes less than 10 seconds to go through all of these.

[andrewfer000]

I was reading their limit rules. I am kinda worried about the user-agent limits, but the IP limits will never be reached with a program like this since it's not like we are running a cloud-based service with many users. Each user has their own copy of nrsc5-dui running on their system on their own IP address. I mean we don't want to or mean to abuse their service or anything.

[markjfine]

10 seconds is kinda long, but we'll see.

The order they're run is really the priority. So according to your results, what I might do is swap the order of the 'Album'-only ones with the 'Official'-only ones so it has the potential to exit quicker. I may also combine them down to 4 lines and just flip-flop setStrict back and forth, much the way the current one uses one query line - just in case you wonder why it might look different.

[andrewfer000]

Alright give it a try and see what happens.

[andrewfer000]

I know this might sound kinda wrong, but is there a way to have a random user-Agent string generated on startup so everyone has their own limits? Maybe make it something like "nrsc5dui radomstring" where the randomstring is generated and saved on first run?

[markjfine]

I see malicious bots that use valid UA strings and fake googlebots all the time. Would really rather not game the system that way. I block those suckers straight away. lol

[andrewfer000]

But hey those are malicious bots, this program is good. But tbh I don't even think with a 100 users this program will break the limits just as long everything is optimized and we try to minimize pointless hits.

[markjfine]

a wise man once told me to dress for the job you want. if we look like a bot, we're a bot.
anyway... Arsenal look like crap. I may start on this just for the distraction.

[andrewfer000]

Also, don't forget the limits are based on averages and it's per second and this program makes 1 to 8 quick requests on average once every 120-180 seconds (if it's new data only) there really isn't too much to worry about. My main concern is that if there is nothing available it will keep trying which is only a bad thing when the song info changes back to the station info.

[andrewfer000]

Also this Testing Notes is becoming a bit big, I think it might be time to close this one and open another one. Maybe we should open an issue just for MB Stuff.

[markjfine]

XHDR is solving the logo issue. Only problem is that if the station doesn't remember to switch it back to 0, it will stay on the logo even though the text changes. May be a minor consolation.