Phillyman's Work In Progress...

[Phillyman]

On the inventory side, I expect to get my asset tags by this weekend. I had the guy make a change to the asset tags so I could cover both magazines and guides.

By appending a 3 character code to the front of the barcode I can differentiate between magazines and strategy guides. I chose a 6 digit code so that I never have to repeat a number. At the moment I have ordered 3000 magazine barcodes and 1000 strategy guide barcodes. Still in the process of writing a small graphical interface to give me a quick and easy way to fill out a form and hit submit as I go magazine by magazine. I haven't decided if I want to have one giant .CSV file with everything inside, or have it write out individual CSV files for each publication.

 

EDIT:

Not my worst script but it works

 

Which outputs a CSV file....

[Phillyman]

So before I start my great scan quest, I wanted to make sure that the Magazine Database and Download Manager were in sync with each other. Everything in the Download Manager was run thru a MD5 checksum generator and those values were attached to each release. Now that I have MD5 values for all 1344 releases, I am looking for a matching MD5 in the Magazine Database. I have run a query to extract any item in our Magazine Database without an accompanying MD5 checksum. The result was 303 entries in the MagDB that did not have an MD5 checksum.

So now I have that data extracted to Google Sheets and am going line by line to figure out why some of these are not populated. You can follow my progress here

https://docs.google.com/spreadsheets/d/1HLfXlkP0LWh73TUxtL99ROtDBcnmltNFPLM8TWz0N5I/edit?usp=sharing

Once I have this spreadsheet filled out, I should be able to dump the missing MD5 values into the MagDB. Then I should be able to run an extraction from the Download Manager and Magazine Database and run a compare query to ensure that the MD5s that exist in our Download Manager, also exist over in the Magazine DB. There are a few big reasons why items need to match between these two systems.....one of them will be a huge change for us

 

 

 

[Phillyman]

Oh boy, what fun

So after dropping in more MD5 values to the Magazine Database, I extracted the names and MD5 values from the Download Manager and Magazine Database. Performed a bit of Excel kung-fu on the values and removed anything that had a match. What I am left with is this.....

https://docs.google.com/spreadsheets/d/1jzE110PgDPpj5o0oope1Rud-8_MKh3mPesoasqj4sDc/edit?usp=sharing

The above document shows 334 unique MD5 values. Meaning....

1) A MD5 value exists in the Download Manager but not the Database. This can happen if someone forgets to copy that value over

2) A MD5 value exists in the Database but not the Download Manager. Probably more rare, but probably means a release never got reuploaded from FileFactory back to Retromags

3) A Member added MD5 values in the Database section, for items at another preservation site. These I just need to blow away. The MD5 value is just for our files.

I am going to go thru the above document and look for matches, the DLM value should take precedence over the MagDB value, as I know them to be correct.

[kitsunebi]

Just looked at your list.  Of the things I'm responsible for:

PlayStation Magazine Issue 106 September 3, 1999 - this is showing up as only in the DLM because it IS only in the DLM, as this title has no database category.  I scanned the only issue I owned, but have no info on the hundreds of other issues (seriously, YOU try googling "Playstation magazine" and see how many hits you get for this super-generically-titled Japanese mag), so I didn't even try to create a database for it.

Tech Gian Issue 014 (December 1997) - No idea what happened here.  The MD5 on the DLM matches both the file I have on my computer, as well as the file I just downloaded from RM to test.  I have no idea where the different MD5 listed in the DB came from, but I went ahead and corrected it.

[Phillyman]

14 hours ago, kitsunebi77 said:

Just looked at your list.  Of the things I'm responsible for:

PlayStation Magazine Issue 106 September 3, 1999 - this is showing up as only in the DLM because it IS only in the DLM, as this title has no database category.  I scanned the only issue I owned, but have no info on the hundreds of other issues (seriously, YOU try googling "Playstation magazine" and see how many hits you get for this super-generically-titled Japanese mag), so I didn't even try to create a database for it.

Tech Gian Issue 014 (December 1997) - No idea what happened here.  The MD5 on the DLM matches both the file I have on my computer, as well as the file I just downloaded from RM to test.  I have no idea where the different MD5 listed in the DB came from, but I went ahead and corrected it.

No worries

All this cleaning up and organizing is leading up to a huge change for us. I would rather be scanning items, but it would just further compound the disorganization at this point. I get closer to my end goal each day

[Phillyman]

Now everything is basically matched up, and I found two duplicate files in our DLM. They are different files, but the same issue.

 

[Melki]

On 15/8/2017 at 9:58 AM, kitsunebi77 said:

Now that's more like it.  You should check here though to make sure you don't waste time duplicating things Kiwi's been doing recently.  http://kiwis.world/index.php/component/jdownloads/category/108-computer-games?Itemid=-1

saw this issue in kiwis site. this one is about to be scanned. hope it isnt a problem.

 

8 hours ago, Phillyman said:

No worries

All this cleaning up and organizing is leading up to a huge change for us. I would rather be scanning items, but it would just further compound the disorganization at this point. I get closer to my end goal each day

i'm guessing this would be fixed.(attached file) 

 

and now that im writing about pizza and microwave oven it reminds me... of my onimusha issue @E-Day xD

Edited August 31, 2017 by Melki

[Phillyman]

I have two pretty big tasks in front of me at the moment which are somewhat related. The first is the naming convention issue and the second is writing those names into all the records. So in order to keep moving forward I am going to handle it this way.....

Today I am going to extract the record title, month and year from all the records in our Magazine Database. I will then recombine them....

Nintendo Power Issue 1, July/August, 1988

will become....

Nintendo Power Issue 1 (July-August 1988)

then those 18,000 or so newly created names will be pushed back into the Magazine Database as the Retromags Naming Field. After that is done I can run the renaming tool I am developing against the local copy of the Download Manager that I have on my hard drive. The collection tool will rename all my files and sort them into appropriate folders....

Retromags > Magazines > Nintendo Power > Nintendo Power Issue 1 (July-August 1988).cbr

Once my local copy of the Download Manager has been renamed and organized, I will be able to browse thru it and look for anything that does not look right. I will also do a directory dump and post the results in this thread so you guys can assist me in finding anything that is wrong.

It is at THIS POINT where we can then fine tune the naming scheme for files that exist on Retromags. This would be where we say

Sega Visions Issue 1 (June-July 1990).cbr 

should be actually called....

Sega Visions Volume 1, Number 1.cbr OR Sega Visions v1n1.cbr OR Sega Visions Vol1-Num1.cbr

It will be much easier to deal with oddballs and renames within the 1300 releases, rather than within the entire 18,000 record database. Once we decide how a file is named, we can then change out the remaining issues to follow suit.

 

[kitsunebi]

2 hours ago, Phillyman said:

Sega Visions Issue 1 (June-July 1990).cbr 

should be actually called....

Sega Visions Volume 1, Number 1.cbr OR Sega Visions v1n1.cbr OR Sega Visions Vol1-Num1.cbr

It will be much easier to deal with oddballs and renames within the 1300 releases, rather than within the entire 18,000 record database. Once we decide how a file is named, we can then change out the remaining issues to follow suit.

 

In situations like this, supposing that the date can be confirmed as reasonably accurate, I see no reason not to include it as well.  So while I agree that it should be identified by volume/number, since that is what is printed on the mag, adding the date as well allows people to get a general idea of what to expect from the contents.

So, Sega Visions Volume 1 , Number 1 (June-July 1990).cbr OR Sega Visions v1n1 (June-July 1990).cbr OR Sega Visions Vol1-Num1 (June-July 1990).cbr

[Melki]

14 hours ago, kitsunebi77 said:

In situations like this, supposing that the date can be confirmed as reasonably accurate, I see no reason not to include it as well.  So while I agree that it should be identified by volume/number, since that is what is printed on the mag, adding the date as well allows people to get a general idea of what to expect from the contents.

So, Sega Visions Volume 1 , Number 1 (June-July 1990).cbr OR Sega Visions v1n1 (June-July 1990).cbr OR Sega Visions Vol1-Num1 (June-July 1990).cbr

Sega Visions Vol1-Num1 (June-July 1990).cbr

not too large and easy to understand

[KiwiArcader]

Personally, I'd run with the naming convention on my site for this title

Sega Visions (1990-06) Vol.01-01.pdf for the file itself and likely simply Vol.01-01 on the product page. If the product changed mid-cycle from one naming convention to another I'd use 1990-06 release date as the title to ensure issues display correctly in Joomla.

 

[kitsunebi]

Keeping the database organized is important, but so far as the filenames go, to be honest, I don't care if we call it:

"Yo - this be mothaf%^#in' Sega Visions  ☯ jUnE 1990 ☯ VOL1 № 1 ☠ ☢ ☣ ☢ ☠ SONIC RULLEZZZZ.cbr"

I don't know about you guys, but I ALWAYS rename my files when I download them.  Retromags does things one way, Kiwi does things another, hundreds of other sites offering downloads do their own thing as well.  Since I don't download everything from a single website, the only way to keep things organized on my computer is to name everything using my own system.  So no matter what is decided here, I'm almost definitely going to continue to rename stuff my way when I download it.

All that's important to me is that the identifying information is present (title, some form of vol/num, date if available.)  Including the issue number is fine, too, if anyone really wants it there, but since it isn't printed anywhere on the mag, it should never be included in lieu of vol/no.

[E-Day]

I rename everything I download as well, because I prefer certain format.

[Phillyman]

14 hours ago, kitsunebi77 said:

Keeping the database organized is important, but so far as the filenames go, to be honest, I don't care if we call it:

"Yo - this be mothaf%^#in' Sega Visions  ☯ jUnE 1990 ☯ VOL1 № 1 ☠ ☢ ☣ ☢ ☠ SONIC RULLEZZZZ.cbr"

 

I get that some people may not care, but I am someone that loves renaming tools. MusicBrainz can rename MP3 files and fix the metadata, FileBot can rename movies and TV shows so that they pull the correct artwork for Kodi, Emby and Plex. GoodTools can rename video game roms so that they can pull the correct artwork for Hyperspin, Emustation, etc. So I am personally taking up the renaming of magazines

 

[Phillyman]

Still doing cleanup around the site at the moment. One thing I will say is this gives me a clearer picture of how things can be a bit more organized. Attached is a list of every file in our download manager and how it would be named once downloaded.

Download Manager List

I am running a renaming script against my local copy of the Download Manager. Right now I need to tidy up the Tips & Tricks / GamePro sections so that files are named correctly.

 

[kitsunebi]

So many underscores!  I feel like I'm looking at the contents of my hard drive from 20 years ago.

Don't worry - I'm not suggesting we change it, though.  I rarely download anything from here (and our new releases are pretty infrequent), so erasing a few underscores now and then is no big deal.  But good lord, what a nightmare I'd have on my hands if any of the several hundred new comic scans/digital .cbr files that are released each week were titled like that, since I download all of 'em (luckily none of them are).

[Phillyman]

4 minutes ago, kitsunebi77 said:

So many underscores!  I feel like I'm looking at the contents of my hard drive from 20 years ago.

Don't worry - I'm not suggesting we change it, though.  I rarely download anything from here (and our new releases are pretty infrequent), so erasing a few underscores now and then is no big deal.  But good lord, what a nightmare I'd have on my hands if any of the several hundred new comic scans/digital .cbr files that are released each week were titled like that, since I download all of 'em (luckily none of them are).

Its already on my todo list though!!!!

So one of the reasons for me putting the MD5 values in the Download Manager AND the Magazine Database, is so that I can pull the "recommended" file name across back into the Download Manager. In a few days all those underscore file names will be purged from the Download Manager, everything will follow the same format. I will also pull the Database links from the Magazine Database back into the Download Manager, so that users can move back to a database record from within the download pages.

All will be revealed soon.......this all leads up to a much bigger "surprise"

 

[E-Day]

2 hours ago, kitsunebi77 said:

So many underscores!  I feel like I'm looking at the contents of my hard drive from 20 years ago.

Don't worry - I'm not suggesting we change it, though.  I rarely download anything from here (and our new releases are pretty infrequent), so erasing a few underscores now and then is no big deal.  But good lord, what a nightmare I'd have on my hands if any of the several hundred new comic scans/digital .cbr files that are released each week were titled like that, since I download all of 'em (luckily none of them are).

 That's what programs like Bulk Rename Utility are for

[Phillyman]

8 minutes ago, E-Day said:

 That's what programs like Bulk Rename Utility are for

Didn't I banish you from Retromags last night? Get outta here! Scram.....Vamoose!

Also AntRenamer FTW!

[E-Day]

44 minutes ago, Phillyman said:

Didn't I banish you from Retromags last night? Get outta here! Scram.....Vamoose!

Also AntRenamer FTW!

 I'm on a train to Montreal right now. I don't leave for Italy until tonight. Then i will banish myself!

[Phillyman]

35 minutes ago, E-Day said:

 I'm on a train to Montreal right now. I don't leave for Italy until tonight. Then i will banish myself!

[Phillyman]

Today I believe I have finished up the corrections for Tips and Tricks and GamePro in our MagDB. I am getting better at quickly changing data fields in mass. The old Tips and Tricks records have been moved to a hidden GraveYard section just in case we have to grab something that was missed.

With those two sections corrected, my magazine collection tool should be able to rename all my files correctly. When I get home in a few hours I should be able to run it aganist all 1300+ magazines that I pulled from our Download Manager. If everything looks good after the rename, I will then script those filenames to drop into the Download Manager and everything will be uniform in that section as well

 

[Phillyman]

 

The Retromags renaming tool grabbed about 98% of the files and renamed them appropriately The only items it didn't find was the Guides and that is because there are so few at this moment that I have not set up the MD5's in that section. It also pointed out many flaws in the UGP section.....lots of duplicate files that need to be nixed!

https://docs.google.com/document/d/1YnX7ebJ5t-Pk85vfmtKCT0muw7KizRXC9MKnsvfhskY/edit?usp=sharing

Ignore the sorting as its because I used the command prompt to output the directory contents....in Windows the files sort correctly.

Anyhow just a bit more cleanup

For those who are bored, here is the output from the Retromags Collection Tool. Ignore the timestamp's as that is not working properly yet.

https://docs.google.com/document/d/1KbjlW2qeN1ejPWK2PMwuEXEksxQVA8gqU3L7zzLPrCU/edit?usp=sharing

 

 

[Phillyman]

Today we dedupe the Download Manager!

First up.....these two entries are identical. Deleting the newer of the two.

 

 

[Phillyman]

Next up for deduplication is this issue of Ultra Game Players which is listed as Issue 80 and 99. Going to delete the 80 version as it is incorrect. The only difference between these two files is they use different Retromags Thank You pages at the end.