1 week Ago By Maya Posch
At first glance, trying to play chess against a large language model (LLM) seems like a daft idea, as its weighted nodes have, at most, been trained on some chess-adjacent texts. It has no concept of board state, stratagems, or even whatever a ‘rook’ or ‘knight’ piece is. This daftness is indeed demonstrated by [Dynomight] ( ), where the chess AI is pitted against a range of LLMs, from a small Llama model to GPT-3.
5. Although the the outcomes (see featured image) are largely as you’d expect, there is one surprise: the model, which seems quite capable of giving Stockfish a run for its money, albeit on Stockfish’s lower settings. Each model was given the same query, telling it to be a chess grandmaster, to use standard notation, and to choose its next move.
The stark difference between the instruct model and the others calls investigation. OpenAI describes the instruct model as an ‘InstructGPT 3.5 class model’, which on OpenAI’s site and an that describes how InstructGPT is effectively the standard GPT LLM model heavily fine-tuned using human feedback.
Ultimately, it seems that instruct models do better with instruction-based queries because they have been programmed that way using extensive tuning. A discusses the Turbo vs Instruct version of GPT 3.5.
That thread also uses chess as a comparison point. Meanwhile, , per OpenAI, using Reinforcement Learning from Human Feedback (RLHF), with presumably ChatGPT users now mostly providing said feedback. OpenAI notes repeatedly that InstructGPT nor ChatGPT provide correct responses all the time.
However, within the limited problem space of chess, it would seem that it’s good enough not to bore a dedicated chess AI into digital oblivion. If you want a digital chess partner, try your . Chess software doesn’t have to be as as an AI model.
.
