Noise in image to text data-set.

[Akhil Gupta]

@prodicus Dataset which contain text from images is not cleaned (i.e containing lots of special character and numbers) and not normalized as well.
for e.g. sambhar vada , vada sambar, vada with sambhar all are same but they are different in the dataset.

[Tasdik Rahman]

@codeorbit That would be a problem.

But it would be a pain to manually clean the dataset with the cases like sambhar vada. Skim through the dataset once, there are some properly OCR'd menus in there.

Does nltk (or any other module) have something which normalizes the data the way we want?

[Akhil Gupta]

@prodicus There is no such module for that in nltk. Everyone makes their own custom algorithm for normalization based on some pattern in the dataset.

[Tasdik Rahman]

@codeorbit So would your models not work with this data?

[Akhil Gupta]

Not only mine, any model will not work with noisy data.

[Tasdik Rahman]

What is the plan now then? OCR wont give be giving us the data in the form you want. Burrp hosts text menus on their website. So we could probably use them

[Akhil Gupta]

@prodicus Cool .. we can go for it 👍