retroprime

Thanks @E-DayI have not reached out to him yet, more than happy to chat. I still stand for the first message of making the data public, I just need to get user login under control, public means registered user, as I found out that even with no users, I'm still getting hit pretty bad by bots hitting the endpoints. I have also thought of maybe one day make it work with this site, integrate the backend of iremdb into retromags, not sure if this would be possible through this platform but it is a thought nevertheless.

https://iremdb.com I finally managed to get something out ... This has become a much harder effort that I could have ever imagined. Having to both index magazines (currently at 17k pages) and build that horrible website, has become a very hard job, Hope it somehow people enjoy it. I got the basics out, list platforms, publications, magazine, search feature, and a basic viewer. Some things you can find there: All pages that contains Super Mario Bros 3 : https://iremdb.com/platforms/nintendo-entertainment-system-nes/super-mario-bros-3 All famous Sega ads "Genesis does what nintendon't": https://iremdb.com/search?mode=ads&tags=123 All John Madden Football ads: https://iremdb.com/search?mode=content&games=38633&contentTypes=Ads All content published by "Andromeda" editor from Gamepro: https://iremdb.com/search?mode=editors&editors=43 The entire "Legend of Zelda" comic strips from nintendo power published over 12 issues https://iremdb.com/search?mode=content&tags=228&contentTypes=Comic I need now to integrate a user management into the site, so people can start contributing if they wish. Some progress I made in automating the whole thing: I had managed to do some neat tricks such as automatic ads detection, given that 50% of those magazines are ads, I used a perceptual hash algorithm to finger print each ad page, and for each new issue, it simply looks in the database and usually get 80-90% of all ads indexed (it does miss about 10%, and there's always some new ads on each issue). I have also played with using custom object detection, and it works fine, for instance I automated all detection of gamepro reviews by looking for those silly faces, problem is that data labeling to train a model is as expensive in time as actually, you know labeling the page I had set a goal to release this to public as soon as I reached March 1993 (exact 30 years ago) in magazines indexed. I'll keep pushing new issues, and adding more features, need to get user integration or this is just a waste of my time. Once again, if you want to participate ping me again, I had to shut the other site down. I may also have to one day keep the viewer behind a login screen, the site has never been shown to anyone else but somehow hundreds of bots from some countries are just crawling it nonstop, if I want to keep CDN costs at bay, I may need to just let the thumb images to be public, other pages, and future features such as like, collection, custom searches, to be only for registered users. Constructive (does not need to be good, just constructive) Feedback is welcome

Thanks again @E-Dayrest assured that every single page shown, will have a source link to this site, even to the specific issue if you desire. I'm planning to let users either search (main feature) or say browse a magazine, but with the capability to filter out pages since they are tagged (so one could read an issue and filter out all ads, mail, and just see content) For that use case I intend to make the retromags image (that is present in several magazines) to stick as an "ad" for a few seconds before the magazine loads, it is my way to give credit where it is due. If you recall, I'm still working on the magazine viewer, the internet archive one turned out to be a lot of trouble to work with. But as soon as I get something off ground you can test on the demo site.

Ok I think I may have had enough. Just to start I couldn't care less what you think of my motives. If you continue with accusations and just being plain nasty, you don't have (nor I would like you to) be part of this. "my not just add tags to the images already in retromags gallery" -> That is EXACTLY what I am doing, maybe you can't understand but in order to correlate the data I need the images in first place. Every single image has an unique name that can be traced back to the original image, the database refer to it, it is a no brainer get the dump of the data and use it. But I don't have access to the retromags db, and I believe the app I wrote was made with the clear intention of making indexing quite fast with all the shortcuts in place, something that would take way more time if done via this website which seems to be a community based platform, not built for this kind of work. I said this before, all those images are available via retrocdn or internet archive. If someone feels so strong about their scan, they can reach out to me. But that is a very thin line to walk, because the same can be said about the publisher. That is what you really don't get it do you? You also complain about other people's work, and I'm taking advantage of that. Do you ignore the time it takes to index one magazine? Or to build the site? Are you really that selfish? I'm done talking to you, there's always one person that can't simply accept that others want to improve upon work, they are sour, and I believe that you have said yourself your true motives "my app will get less crowd attention" Don't be like this, build your app to improve something not start trashing others.

Thanks @E-DayI also believe that, I honestly think this is more a way to let people find information than anything else. I don't believe it will drive the fury of attorneys out there. And even if it does, there's a case for both preservation and information retrieval here. Anyways, I keep just doing what I have been doing for the past 60 days, every morning before work, I try to index 1 magazine, and that has been the goal so far. I literally doubled the number of pages since my initial post. And whenever I have some free time I work on what could be a site to host it. The demo site is still a few versions behind, I have not automated any CI/CD pipeline to publish changes, will eventually get there, but so far just keep indexing those pages

The images are being hosted on a CDN backed by S3 which only really cares about bandwidth. Traditional hosts services will care about that but not any cloud provider out there. There's a 1TB allowance per each 250GB of data stored, which means about 1M pages could be downloaded per month. If I ever get to have that problem then I think it would be a nice problem to have

I'm not sure if I follow your comments @tcaud, right at the start I posted a link with the whole data being available to anyone to download. Just like I have downloaded the TGDB database they offer as the way to link the games during the search to each page. Just to start, I have zero expectations of making any money, fortune I think you may be overselling the number of people who actually cares about retro magazines. As per IP, well I know Nintendo is a big A*hole when it comes to IP, but the way I see, although this is not different than the Internet Archive that parsers each PDF page extract OCR information and allow search (I've done that too), it is just a way to tag inventory and allow more meaningful searches such as "All games by publisher X" or "All reviews by a given editor" those things are not currently possible. Initially I downloaded the magazines from other sites such as Internet Archive or retrocdn.net but they were pdf which required an extra step on breaking into individual jpegs. But the point being that it seems that unless otherwise required by the original publishers this almost became public domain. As per costs, again I think you are really overestimating the costs of this kind of effort. I don't believe this to be ever more popular than this website, I reached out actually to partner and make something that can co-exist not compete. But sure, there are costs, which I have not figured out how to pay, but I would just put some ads on the site. The total cost should never go above 60 bucks a month, and with enough page views, ads could offload some of that cost. I'm comfortable in taking some of the toll, not here to make a profit (unlike your thinking). Right now the total cost for it is 15 USD. It has a 250gb CDN storage (using 140gb), a shared database as a service that costs $10 month, and should probably be ok for a few million requests a month. The app itself runs as a serverless app, which should cost about $5 a month for 10 million requests. Hope this shows you and others what are the true intentions here

Hi folks, made a ton of progress and finally reached 10k indexed pages Site is up and running here: https://demo.openscans.org -> Please ping me for a username That is a demo site, data is erased frequently, but I thought it would be better if folks can get a grip of the app before starting modifying the contents. As promised, the data is always exported (every Friday for now) to the CDN link shared before. I'm looking forward to discuss what options we may have to publish that data, it would require a new site, not the one I built for this. And here it is a long help (sorry about that) on how to use the site. I'm sorry to upload a PDF, but I wrote a markdown file and did not plan properly how to present it. README (1).pdf

Oops sorry, somehow old content was loaded on editor. Wanted to give some updates, I made a lot of progress, and I think I'm very close to opening the site. All major problems have been addressed and I have the workflow ready. Automated the export of data (just not on cron yet, but the code exports it daily to https://openscans.nyc3.cdn.digitaloceanspaces.com/db/openscans.zip I just dumped new data there (old link was broken sorry) I'm now working on having proper DNS and the demo site (I setup a backup cluster for the data to open a demo) and finally get some basic help page with the many keyboard shortcuts I used to make tagging of the magazines as quick as humane possible In between coding I have also indexed May/June 1991 Confident I'll have a website to share with interested folks by early Jan. Happy Holidays

Hi folks, really glad you enjoyed, gotta start working on this to be ready for prime time So here's a link for the current database export (this uses a mongo db cloud atlas, so everything is a huge json) https://openscans.nyc3.cdn.digitaloceanspaces.com/db/GamaMagazineExport.zip Here's what I need to finish before getting this opened up for collaboration Automate the db export data Fix some login issues Provide minimum error handling for users (right now if something goes wrong I know where to look in the logs, not friendly) Get a basic magazine viewer in place (goal is to use the Internet Archive amazing book reader https://openlibrary.org/dev/docs/bookreader, but that will take time to integrate, need a step stone solution in between) Craft the approval workflow (once a user completes a review, an admin needs to approve) Get the demo site up and running (I want a demo site for people to try and play, data will be purged daily and restored to latest backup from the main site) I believe most of those if not all should be completed in the next 2-3 weeks, I will then keep you posted and start handling some users for the demo site, we can then start working on the main site to curate the data. Like I said before, what we do with this data is really up for grabs, not sure how much work would needed to integrate here, but once we have the data, everything else becomes easy, we just need to present it on a relevant manner. Looking forward for more collaboration

Hi folks, really glad you enjoyed, gotta start working on this to be ready for prime time So here's a link for the current database export (this uses a mongo db cloud atlas, so everything is a huge json) https://openscans.nyc3.cdn.digitaloceanspaces.com/db/GamaMagazineExport.zip Here's what I need to finish before getting this opened up for collaboration Automate the db export data Fix some login issues Provide minimum error handling for users (right now if something goes wrong I know where to look in the logs, not friendly) Get a basic magazine viewer in place (goal is to use the Internet Archive amazing book reader https://openlibrary.org/dev/docs/bookreader, but that will take time to integrate, need a step stone solution in between) Craft the approval workflow (once a user completes a review, an admin needs to approve) Get the demo site up and running (I want a demo site for people to try and play, data will be purged daily and restored to latest backup from the main site) I believe most of those if not all should be completed in the next 2-3 weeks, I will then keep you posted and start handling some users for the demo site, we can then start working on the main site to curate the data. Like I said before, what we do with this data is really up for grabs, not sure how much work would needed to integrate here, but once we have the data, everything else becomes easy, we just need to present it on a relevant manner. Looking forward for more collaboration

Hi folks, first thank you so much for this amazing trip to memory lane. I always wanted to check some of my childhood game magazines. So wanted to share a weekend/nighttime project I started. I wanted to catalog each retro magazine out there. I saw an entry here on the database, but just for fun I put up a magazine manager that I made my best to make it so quick to tag/index pages (uses a lot of keyboard shortcuts for instance). Over the last 50 days or so while building it I manage to index three main magazines (Nintendo Power, EGM and GamePro) up to April 1991 (so far) This yields about 6452 some fun stats about it: 2323 pages were ads 2366 general content 1698 game reviews 65 covers So tagging is quite quick, but I won't be able to do all by myself. so I was hoping others would like to help. Please take 3 min to watch a sample video of the app I made hoping it gather some attention: So, what would I like to do with that data? Give it back of course, should be free, open for anyone like TGDB for example (which I used to create the game finder database to find and match the games on the magazines) The app is not really available yet, I'm still setting up permissions, there are still a few annoying bugs that needs fixing (some minor UI glitches - I am NOT a UI developer by any chance, so apologies on the raw look) But with Holidays, I hope to have something ready by end of January, where folks can log in, and start contributing, and also a pipeline to dump the data daily into a CDN for people to download. I thought at first to create a front end site, but I may not have time for that, and it would make more sense to live here. So, if you like what you saw, please, send me a DM, I'd love to get more people involved in curating this data and making it publicly available, in any case, I'll just keep tagging them, and making daily dumps until I'm done one day (may take a few years I'm churning something like 800 pages a week, there are 110k pages on all those 3 publications) Happy Holidays and hope someone joins this