Errare humanum est? A Pilot Study to Evaluate the Human-Likeness of a AI Othello Playing Agent

Olivaw is an AI Othello playing agent which autonomously learns how to improve its gameplay by playing against itself. Some topnotch players (including former World Champions) reported that they had the impression that Olivaw’s gameplay was human-like.
To better investigate the processes related to these impressions,
we conducted a pilot study using the Othello Game Evaluation
App, a computer application we developed to evaluate pre-recorded Othello games in a controlled setting while assuring an adequate user experience. An exploratory analysis of the results shows that the participants mostly evaluated Olivaw as a human. When asked for a motivation for their choice, some of them reported that they evaluate poor game moves (and, consequently, losing the game) as an indication of the human-likeness of the player.