Puzzle rating

@achja, yeah, I don't like being so familiar with ... that awkward feeling ... of failing a 1600- puzzle after the glory of solving a 2000+ puzzle easily. *chuckles*

I'm a slow thinker so it usually takes me more than a couple of minutes to solve most things. I even get tricked by 1300 rated puzzles from time to time whenever I forget that I shouldn't "take anything for granted". Material puzzles are my bane. (Oh, and unlike game players, we don't get to know the puzzle's rating until after we fail/solve it so we can't form a prior expectation to the difficulty.)

In games, I even forget that I'm playing something with no increments and get lost in the enjoyment of simply playing (whether I'm winning or losing) ... until I carelessly time out and the dream ends. ^,^'

I saw a puzzle rated around 2600 before and I wish I could remember which number it was because the mate was really extraordinary. It looked much simpler that it actually was because of the massive amount of pins but it led to a pretty clever mate.

http://en.lichess.org/training/6456 - rated 2712 right now
http://en.lichess.org/training/38164 - rated 2536 right now
http://en.lichess.org/training/9892 - 2673

#12
You can go through your own puzzle statistics, and possible find it there.
#13 Nice one. Thanks for sharing !

I got the moves to the first idea in 6456, but got completely lost looking for the continuation. I think most people made the same mistake I did in 38164 since the idea there isn't too hard to recognize. Would be kind of interesting if puzzles recorded the moves people made and displayed the most common error paths.

@EugeneJudo, something like "Congratulations! You are the 10,000th person to make that mistake." >,<

Good idea. The puzzles themselves were borne from heaps of accumulated analyses after all. (Too bad, it probably isn't getting recorded so we likely see that coming into fruition.)

Well, I see I haven't made myself clear enough and that lead to misunderstandings and deviation of the point, so here I go again.
1)There's ELO inflation/deflation of the puzzles, due to the presence of multiaccounters (I'm one of them) who are already aware of the answer in several cases.
2)The computer always shows the very same puzzles (if there are thousands of them how comes I always solve the same ones), I assume because they are the most voted (of course nobody can vote for puzzles he never gets to see).
3) We need the computer to display the puzzles in random order, and a new way (probably, a group of human judges) to catalog them by difficulty, as the current system is far from accurate (it relies simply on an ELO that is fictional).

To point 2 in #17 there are a few simply explanations.

Firstly if you've already played for example 100 puzzles, in a system with 10,000 puzzles, and then you play 100 more on a new account there is around a 50% chance that you will encounter one of the same puzzles you played before (this is called the birthday paradox).

The pool of puzzles is even more narrow because you would be playing at the same rating, so there is a high chance that you will come across the same puzzles.

The moral really is just do tactics on only one account.

@AlcianGreen, in this era, there is currently no foolproof way of restricting one account per person so developers (and statisticians I suppose) have to design (and research) with that in mind.

The puzzle ratings are a (novel) indirect application of Glicko-2. Instead of player vs player, we have player vs puzzle. Ideally, it should have been puzzle vs puzzle if such a scheme can be implemented. Thus, the ratings will tend to be inaccurate even without the multi-accounts factor.

I agree that I would like to have the option to turn off popularity-influenced randomness. However, at some point in the (hopefully still very) far future, it won't matter as there are only 60,120 puzzles (and I'm only going to be served a fraction of that due to the ratings like EugeneJudo mentioned), I'll eventually run out of unique puzzles to solve (even if all of them somehow get used). I wonder how lichess has been set to handle that event.

One interesting idea to fix all the puzzle issues:

1. re-generate all the puzzles with the new engine upgrade and a bit more tolerance for equally winning but subpar moves not failing them outright (i.e. fail on a missed mate, but not on a longer mate...only true inaccuracies or mistakes failing the puzzles outright)

2. set the AI levels against the puzzles first before they are made available to the user base to get initial puzzle ratings.

3. have a team of masters (LM, NM, IM, GM) evaluate puzzles before they go live. Basically require 3 up votes from the masters team before it gets added to the user pool.

4. have a small team of users with a mix of ratings go at the puzzles before they go live as well. After AI, some masters and some beginners challenge the puzzles, (i.e. once the rating becomes stable) lock the puzzle's rating and release the puzzle to the general user base.

This would be a much longer process than was used to generate the puzzles initially, and it would mean only maybe 1000 puzzles added each week rather than 10,000 or so all at once, but I think it would make for MUCH more confidence in the quality of the puzzles here and significantly reduce the amount of posts made about "bad puzzles."