Image sub-folder Set code source

by **Rooger** » 06 Jan 2023, 23:26

I'm working on a small project to download Forge images using Scryfall's API. The goal is to also implement a standalone auditing tool similar to what is found ingame but be able to run it against Scryfall's info and grab only the missing sets/cards or do other kinds of data manipulation like deck building, etc. The tech stack is mainly just Blazor and C# as that's what I'm trying to learn with this.

I've run into a couple edge cases while debugging and I'm hoping that someone more knowledgeable about Forge's source code would help me out.

The local data sources i'm parsing are the /res/editions .txt files, I suppose that's where Forge reads the information for the available sets and respective cards. This gives me a nice overview of the entire Forge library.

After extracting the Code, Code2, Alias and ScryfallCode from the Metadata section, I compare those with all the Scryfall sets returned from the /sets endpoint, and also with all the sub-folders that exist in the /pics/cards folder.

In the end, I want to get the match between: Edition code - Scryfall Set - Card Pics Sub-Folder.
With that information I can easily count the number of image files inside the folder (if it exists), count the amount of cards in the [cards] section of the edition.txt file and make a get request to scryfall for the missing data, if any.

The problem I've run into, for now, is the mismatch of some folder names and the respective edition code, particularly on older sets. For instance, Unlimited Edition:

Code: Select all: [metadata] Code=2ED Code2=U ScryfallCode=2ED

This is it's relevant metadata information. Judging from this I would guess that the proper images folder name would be "2ed" but it's in fact "u". Renaming it to "2ed" will make the card's image appear missing even though that's the code used in the card information; check the Deck Editor for instance.

How do I make sure I'm targeting the proper folder name when downloading image files and distributing them to the game's path?
I did some testing and found that if there is a Code2 field in the Metadata, that should be the folder name (and only that), is that correct for all cases?
Shouldn't Code be the default value for the images subfolder name and all the other available codes possible fallbacks (including ScryfallCode)? This would mean that even if there is a mismatch between Scryfalls code and Forge edition Code we could use any of the paths

This however leads to the next problem I encountered but only spotted it one time for an exception thrown in my app. It's a code overlap between two distinct sets: Masters Edition and Mythic Edition - Ravnica Allegiance

Code: Select all: [metadata] Code=MED <= Name=Masters Edition ScryfallCode=ME1 [metadata] Code=MPS_RNA Name=Mythic Edition - Ravnica Allegiance ScryfallCode=med <=

There is an ambiguous reference between the ScryfallCode and Forge Code. Without proper matching, this could cause problems with an automated download process where the cards for the Mythic Edition would end up in the Masters Edition folder.

And just for fun I'll share some data about the amount of sections and metadata fields present in all of the edition files, I used this information to help me define C# models for the editions. (there are 2 personal custom editions I used for testing that shouldn't count but they are simple and only contain a cards section apart from the default metadata).

Code: Select all: Section - Featured in n Editions [metadata] - 549 [cards] - 548 [tokens] - 242 [promo] - 21 [showcase] - 20 [borderless] - 19 [buy a box] - 18 [extended art] - 17 [rebalanced] - 13 [precon product] - 11 [other] - 6 [Lands] - 5 [lands] - 4 [Foils] - 3 [dungeons] - 2 [jumpstart] - 2 [Basic Lands] - 2 [box topper] - 2 [alternate art] - 2 [retrocards] - 1 [retrolands] - 1 [Aether Revolt Inventions] - 1 [Legendaries] - 1 [Non-Legendary Uncommons] - 1 [Non-Legendary RareMythics] - 1 [oldframes] - 1 [spire] - 1 [foils] - 1 [mid] - 1 [vow] - 1 [midDFC] - 1 [vowDFC] - 1 [Kaladesh Inventions] - 1 [Snow Lands] - 1 [UrzaLands] - 1 [special slot] - 1 [alternate frame] - 1 [bundle] - 1 [NewToModern] - 1 [White CommonUncommon] - 1 [Blue CommonUncommon] - 1 [Black CommonUncommon] - 1 [Red CommonUncommon] - 1 [Green CommonUncommon] - 1 [Multi CommonUncommon] - 1 [Artifact Land CommonUncommon] - 1 [Pre M15] - 1 [Post M15 RareMythic] - 1 [etched] - 1 [lesson] - 1 [Commons] - 1 [Uncommons] - 1 [RareMythics] - 1 [ModalDoubleFaceCards] - 1

Code: Select all: Metadata Field - Featured in n Editions Code - 549 Date - 549 Name - 549 Type - 549 ScryfallCode - 543 Code2 - 168 Booster - 150 BoosterCovers - 148 FatPack - 76 FatPackExtraSlots - 67 ChaosDraftThemes - 61 Foil - 51 FoilAlwaysInCommonSlot - 31 Alias - 28 BoosterBox - 25 Border - 23 Prerelease - 17 CardLang - 16 FoilChanceInBooster - 9 AdditionalSetUnlockedInQuest - 8 AdditionalSheetForFoils - 7 BoosterMustContain - 4 DoublePick - 3 TreatAsSmallSet - 3 MciCode - 2 ChanceReplaceCommonWith - 2 DraftBooster - 2 #BoosterBox - 2 BoosterReplaceSlotFromPrintSheet - 2 SheetReplaceCardFromSheet - 1 SheetReplaceCardFromSheet2 - 1 CollectorBooster - 1

by **Rooger** » 07 Jan 2023, 03:08

Followup question regarding edition data. I'm currently working on the line parsing logic for the cards but I'm still 314 cards short for complete parity with the game, there is no point in developing other features without achieving that.

In game stats show that my current Forge version (a few days old by now) has a total of 61711 cards in the Deck Editor screen, after parsing all the editions text files I have a total of 61397 scraped objects (including Vanguard avatars has they fit the overall pattern but should be ignored for this count since they aren't included in the regular deck editor).

I think I've accounted for every possible scenario when it comes to line patterns but it looks like I'm missing something.

The most relevant and common pattern is:
CN R N @A
CN - card number, R - rarity (single character), N - card name, A - artist(s)

some omit the artist name:
CN R N

Logging the extra cases that didn't fall into said patterns I also noticed that some cards don't have the CN, the pattern is just:
R N
These are cards like Archenemy Schemes and for now I just ignore them but there are probably some cards that shouldn't be.

As I previously shared, there are some sections that aren't either [metadata], [cards] or [tokens], some of these lines have the following pattern
N|C
N - card name, C - set code
I'm not sure about all the cases but I think they are used to indicate foils, or link to other sets like Aether Revolt -> Aether Revolt Inventions and the other masterpiece series cards, either way, I don't think those lines should be counted as cards and more like pointers or extra data for things like Quest, boosters, etc. I exclude every line that contains the character "|" and is part of a sections other than "[cards]" (I also don't think there is a single line that contains that character and is in a [cards] section either)

Lastly, is there a way, in game, to see how many cards each individual set has? That would greatly help pinpoint the missing data and perhaps reveal some cards that I scraped but shouldn't (like the avatars). I could filter for each set in the Deck Editor and get the totals reading but that would be too time consuming and error prone.

If it helps here's the card parsing method i'm working on:

Code: Select all: private List<char> _cardRarities = new List<char>() { 'L', 'C', 'U', 'R', 'M', 'S' }; private ForgeCardModel ParseCardLine(string line, string section) { if (section != "[cards]") { if (line.Contains('|')) return null; } string number, rarity, name, artist; number = rarity = name = artist = string.Empty; // Artist names are prefixed by " @" and all the card lines that include an artist follow the same default pattern: // CN R N @A // CN - card_number, R - rarity (single character), N - card name, A - artist(s) // This shouldn't throw any out of range exceptions. if (line.Contains(" @")) { var groups = line.Split(" @"); artist = groups.Length > 1 ? groups[1] : null; groups = groups[0].Split(' '); number = groups[0]; rarity = groups[1]; name = string.Join(' ', groups.ToList().GetRange(2, groups.Length - 2)); } else { var groups = line.Split(' '); if (groups.Length > 1) { // Ignore cards that don't have a CN but start with R // Other sections data is not relevant either // This will exclude every other case, so there is no need to predict a diferent combination for the 3 variables if (_cardRarities.Contains(groups[0][0]) || section != "[cards]") return null; number = groups[0]; rarity = groups[1]; name = string.Join(' ', groups.ToList().GetRange(2, groups.Length - 2)); } // This excludes all lines with a single word // None of them had relevant card data and all were on a non-[cards] section else { return null; } } ForgeCardModel card = new ForgeCardModel() { Number = number, Rarity = rarity, Name = name, Artist = artist }; return card; }

EDIT: Turns out I was doing the counting improperly on a method influencing edition output and skipping some due to a couple conditions I was playing around with. However the amount still isn't correct, overshooting now by 1280 (now excluding Vanguard cards). I think that the easiest way to debug this would be by set Types, since there's only a dozen or so, I filtered by them in game and noted the card totals. Here is my comparison:

[Forge]

Boxed_Set: 875
Collector_Edition: 2395
Commander: 6718
Core: 6703
Draft: 6744
Duel_Deck: 1926
Expansion: 23306
Funny: 358
Multiplayer: 1004
Online: 4532
Promo: 1603
Reprint: 4686
Starter: 861
TOTAL CARDS: 61711

[App]

Boxed_Set: 875
Collector_Edition: 2396 | +1
Commander: 6719 | +1
Core: 6703
Draft: 6810 | +66
Duel_Deck: 1926
Expansion: 23318 | +12
Funny: 1314 | +956
Multiplayer: 1236 | +232
Online: 4534 | +2
Promo: 1611 | +8
Reprint: 4688 | +2
Starter: 861
TOTAL CARDS: 62991 | +1280

Looks like most of the extra cards are from Funny and Multiplayer type sets, I'm inclined to say that those are the unimplemented cards but I still need to check them to be sure about the amount. It actually makes sense to include all the cards in a set on the edition data, even if there is no corresponding script in the cardsfolder.zip. Either way it feels like I'm getting closer and Scryfall downloading should be fine now, the unused images might be useful in the future anyway.

by **Rooger** » 07 Jan 2023, 19:36

Small update. Please, excuse the persistence, I'm just logging my findings as I go but also trying to keep posts to a reasonable size.

I used the cardsfolder.zip files in the parsing and see if any file includes the corresponding implemented card "Name:" (split and double faced cards are only evaluated to the first Name: that appears in the card script, the Alternative version is ignored but it should give results either way).

It's now much closer to the in game stats but still not right for some types.

Forge

Boxed_Set: 875
Collector_Edition: 2395
Commander: 6718
Core: 6703
Draft: 6744
Duel_Deck: 1926
Expansion: 23306
Funny: 358
Multiplayer: 1004
Online: 4532
Promo: 1603
Reprint: 4686
Starter: 861

App + CardsFolder

Boxed_Set: 875
Collector_Edition: 2395
Commander: 6718
Core: 6703
Draft: 6768 | +24
Duel_Deck: 1926
Expansion: 23295 | -11
Funny: 361 | +3
Multiplayer: 1236 | +232
Online: 4531 | -1
Promo: 1603
Reprint: 4687 | +1
Starter: 861

This just probably means that there is something wrong with my comparison method but what I found interesting was the negative numbers for the Expansion and Online types, it means that the game is finding more implemented cards than I could with the script running through the card scripts. If someone is curious I logged out all the unimplemented cards I could find, just keep in mind that is not 100% accurate.

I think I'll keep developing Scryfall integration now, the stats aren't what I most wanted but it should be fine downloading a few extra images. I'm still unsure about the proper set code for the folder though but it looks like if Code2 is present I should use that, otherwise use Code. Cheers!