I don't believe the rates that you see "concerning" vs "not-concerning" in the app match the population rates. That is, a random "mole-like spot or thingy" on a random person will have have a much lower base rate of being cancerous than the app would suggest.
Of course, this is necessary to make the learning efficient. But unless you pair it with base rate education it will create a bias for over-concern.
Yes you are right - the representation is biased due to the image dataset that I have used.
I don't think it would be useful to match the population distribution since the fraction of skin cancers would be tiny (less than 1:1000 of the images) so users would not learn what a skin cancer looks like, however in the next version I will make it closer to 50:50 and highlight the difference from the population distribution.
Yes. As I said matching the population base rate wouldn't be practical, so you'd need to educate on that separately from the identification learning.
Let's say I achieve a 95% on the app though. Most people would have a massively over-inflated sense of their correctness in the wild. If the actual fraction is only 1/1000 and I see a friend with a lesion I identify as concerning, then my actual success rate would be:
1*0.95 / (0.05*999 + 1*0.95)
So ~1.8%, not 95%. Few people understand Bayesian updating.
One concern:
I don't believe the rates that you see "concerning" vs "not-concerning" in the app match the population rates. That is, a random "mole-like spot or thingy" on a random person will have have a much lower base rate of being cancerous than the app would suggest.
Of course, this is necessary to make the learning efficient. But unless you pair it with base rate education it will create a bias for over-concern.