PongML started as my first attempt to get my feet wet with machine learning. I wanted something small enough to understand, where I could actually see
PongML started as my first attempt to get my feet wet with machine learning. I wanted something small enough to understand, where I could actually see what the computer was learning. Pong seemed reasonable. Two paddles, a ball, and very little ambiguity about whether you’d missed it. If I couldn’t teach a computer to play that, I figured I should find out before giving it anything with pedestrians.
What interested me was how much of the behavior I could leave for it to discover. Writing a rule that tells a paddle to follow a ball is straightforward enough. Giving a model information about the game, a way to move, and feedback about the results leaves considerably more room for interpretation. Apparently “please become good at Pong” requires some elaboration.
I started with a model trained against an easy programmed opponent, then copied it into three separate learners: James, Annika, and Brad. They began with the same ability. What changes is the experience each one gets.
James learns by playing people. His training uses his own decisions and their outcomes during those matches. I wanted to see how he’d adapt to human opponents, who might have a strategy, might be experimenting, or might have just discovered that the mouse also moves the paddle.
Annika gets a more structured education. Her training works through Easy, Medium, and Hard programmed opponents, requiring five consecutive qualification wins before advancing. After those stages comes self-play against versions of herself. I’m curious about what that produces: whether repeatedly facing a familiar opponent develops useful skills or simply makes her very well prepared to deal with Annika.
Brad studies human play. He learns from the moves people make across the different game modes, with more weight given to actions associated with successful returns and points. James gets feedback on his own attempts; Brad gets examples from ours. That gives me a way to explore learning through consequences alongside learning from demonstrations, while quietly hoping the demonstrations are worth learning from.
One of the more interesting problems has been deciding what to reward. Scoring a point is an obvious success, but a beginner can miss quite a few balls before receiving much encouragement. A small reward for returning one provides earlier feedback. Give that reward too much importance, though, and keeping a rally going could become more attractive than winning it. The computer has no particular obligation to infer what I meant from what I actually asked it to do.
There have also been mistakes I wouldn’t have thought to design. The bots picked up a shortcut where the other paddle’s position influenced their movement enough to interfere with following the ball. Moving the human paddle toward an edge could make them miss a reachable return. I had provided information that seemed useful, and they had found a remarkably unhelpful use for it.
That’s why I wanted a game this simple. I can hold the situation steady, change one thing, and investigate why the paddle suddenly feels strongly about being somewhere else. Saved versions of Annika also let you play against different points in her training. More practice doesn’t automatically mean better play, and I want the project to show that.
You can try it at dustinsnoap.com/apps/pongml. There’s ordinary computer play and local or online multiplayer too, if you’d prefer an opponent whose mistakes aren’t part of my education.
I went into this hoping to teach three bots a simple game. So far, they’ve been quite effective at showing me what I still need to learn.
Dustin