Quest Scraper

[Anderas]

I am working on a refinment of my quest scraper again. I guess If i would invest all the time in playing heroquest instead of investing it in fruitless projects, I would have finished HQ now three times over. But ok.

The Neural Net keeps mixing up Alchemist table with the Frozen Horror Throne Room, and the Bookcase with the Cupboard. The latter I do understand, in the one corner they really look alike. But the former? *sigh*

A more conventional image recognition based on analytic metrics (like a bar code dark-light pattern) would not make either of these mistakes and you could "train" it faster. This type of system has been used successfully in automotive to recognize speed limits from signs, and even distinguish locality in Europe by the font used for the numbers. Is there a thread for your scraper? What is your objective?

Suggestion: If you have found a bounding box for the whole map then size of the sub-element should be a constraint in your classification process. It should not be possible to confuse a room tile with a furniture item

The final logic will certainly be able to separate them out.

Currently I have:
A thing that cuts a png map from heroscribe down to single-square images (slightly bigger so that the doors are completely visible in every square)

And a logic that throws a (very small) Neural Net on these snippets to identify what's visible in there. Here is a constraint: if I throw in entire rooms or maps, I need a quite big Neural Net and I don't have the capacity at home to train it. So I use many small snippets for which I can easily train a small net in an hour, but then it can identify the rooms only square by square.

The bookcase-cupboard error was an error in my training set by the way. I corrected the assignment, now it works.

The throne room has one corner with some circles which look like those bottles on the alchemists table.

The target is to have a thing which takes a quest image and produces a heroscribe file from it. It doesn't have to be perfect, just time saving.

It is not time saving if I have to check everything. So I guess it has to be quite perfect.

Right now it is capturing around 100 symbols coming from the Frozen Horror, the Base Game and the Ogre Hordes. This works. Very good. 0.5% errors.

Now I need to add a hough transformation. With it I can identify the outlines of a randomly sized Quest image. Then I can scale it to fit with the rest, and here we go.

I don't have a solution for the rooms yet.

Do you have some google-able names for the methods you mentioned?

[whitebeard]

The final logic will certainly be able to separate them out.

Currently I have:
A thing that cuts a png map from heroscribe down to single-square images (slightly bigger so that the doors are completely visible in every square)

And a logic that throws a (very small) Neural Net on these snippets to identify what's visible in there. Here is a constraint: if I throw in entire rooms or maps, I need a quite big Neural Net and I don't have the capacity at home to train it. So I use many small snippets for which I can easily train a small net in an hour, but then it can identify the rooms only square by square.

The bookcase-cupboard error was an error in my training set by the way. I corrected the assignment, now it works.

The throne room has one corner with some circles which look like those bottles on the alchemists table.

The target is to have a thing which takes a quest image and produces a heroscribe file from it. It doesn't have to be perfect, just time saving.

It is not time saving if I have to check everything. So I guess it has to be quite perfect.

Right now it is capturing around 100 symbols coming from the Frozen Horror, the Base Game and the Ogre Hordes. This works. Very good. 0.5% errors.

Now I need to add a hough transformation. With it I can identify the outlines of a randomly sized Quest image. Then I can scale it to fit with the rest, and here we go.

I don't have a solution for the rooms yet.

Do you have some google-able names for the methods you mentioned?

I understand the need to keep your algorithm small and working on the single squares is smart.

Unfortunately, I can't point you to an image processing resource because I don't actually do the vision processing. But the methods I'm talking about are similar enough to traditional optimization. It is my understanding from what I've seen in practice at my last employer (who sold this stuff) is that one classical method is to normalize the bounding box of your element to say 16x16 pixels then scan across lines recording the black white pattern (you could also do lines through the center at fixed angular intervals which may be more robust?). Each icon (or one square unit there of) is then coded as an array of these vectors. And you can optimize a set weights (purely Bayesian approach) to realize a best fit classifier on a real set of data (mostly clean but with some dirty outliers). Effectively you are finding eigenvectors in the matrix of black and white which robustly correspond to the independent features of each icon. Then all you have to do is project any new unknown icon's array into each vector and the one with the biggest magnitude is your answer (a matrix multiply). And the second largest magnitude tells you how certain you are of the classification. In your case you would need to do this 4 times (0,90,180,270 degrees). This type of projection is FAST and can be implemented on cheap embedded hardware.

Hope that helps.

[Anderas]

Doing some feature engineering and using a kind of decision tree would be a serious alternative, yes.
In fact, I was using K-Means clustering in the beginning to do the first labelling of my dataset. K-Means is the easiest low-tech method there is in the world, and it already worked 95% correct.
My problem is, I want 99.95

In theory, everyone tells you the single really great advantage of NN is that you don't need self-engineered data features.
The reality is less rosy of course, so adding a couple of eigenvectors could be great indeed.

Taking the eigenvectors would be easy (there is numpy.linalg.eig), taking min and max color and a couple of others, I bet it would work

The medium sized snippets look like this one:


It should give two data points: 1. Spear Trap, 2. Door to the north.
So there is not one maximum value, but several results per square.
1 Main Symbol, in theory up to 4 doors but in reality up to two, finally 1 Room background which is sometimes barely visible behind all the others.

Currently I am trying to get all of them through a single pass of the NN, giving me all of the above at once.
It greatly helps that the snippet is a little bit bigger than a square, for furniture or other multi-square-things and of course the doors.

Currently the net has 1.4 million weights, which should be enough to find all the content of HQ!
An alternative would be to have something lots simpler, times 6, one for each symbol category.

If you know how to use pickle-files or are generally interested in my code, you can get the entire dataset here.