TehnoBlog

Article Last Updated: August 16, 2015

Komodo 9 vs Stockfish 6 Chess Engine Tournament (64-bit single-core CPU benchmark)

Two months ago we published an article about two top chess engines tournament in the world in 32-bit single-core single-CPU arena on Windows XP setup. Today, we present our latest test results under Windows 7 Ultimate 64-bit architecture / single-CPU dual-core environment: komodo-9-64bit.exe and stockfish-6-64.exe chess engines, respectively.

Classic Chess Board

FIXED TIME RESULTS (3 seconds/move)
Download results table as image file (PNG format)

* SB = Sonneborn Berger score

Legend:
0 … loss (0 points)
= … draw (0,5 points)
1 … win (1 point)

Total Games Played: 50 (50 games between each engine / total of 50 games per engine)
Maximum Theoretical Score: 50 points (50 wins)
Level: 3 seconds per move (fixed time)
Books: None
Ponder: OFF
Hash Table Size: 512 MB (per engine)
Komodo 9 Table Memory Size: 64 MB (factory default)
Hardware: Pentium(R) Dual-Core CPU E5200 @ 2.50GHz with 2.0 GB Memory
Operating System: Microsoft Windows 7 Ultimate 64 bit

COMMENTARY

Not much surprise here. Stockfish 6 continued its domination and again triumphed over Komodo 6. We were expecting this results, after our 32-bit tournament test last time, however, it is important to note that this victory was not so dominant, as it was under the older architecture and operating system.

Here is another tournament representation with table and chart analysis data. It will give you a little more insight into how the tournament has progressed over time (click on image thumbnail to enlarge):

TehnoBlog.org – Tournament 3 – StockFish 6 vs Komodo 9 x64 3s Fixed Time – Table & Chart Data

Above table shows score progress per each engine in 5-games steps. Think about it as a collection of 5-games mini-tournaments. Right-Top chart displays total points accumulated over each 5 games. Right-Bottom chart displays relative score gains over each successive 5 games.

TOURNAMENT TIME RESULTS (40 moves/2 minutes)
Download results table as image file (PNG format)

* SB = Sonneborn Berger score

Legend:
0 … loss (0 points)
= … draw (0,5 points)
1 … win (1 point)

Total Games Played: 60 (60 games between each engine / total of 60 games per engine)
Maximum Theoretical Score: 60 points (60 wins)
Level: 40/2 [40 moves in 2 minutes, loop]
Books: None
Ponder: OFF
Hash Table Size: 512 MB (per engine)
Komodo 9 Table Memory Size: 64 MB (factory default)
Hardware: Pentium(R) Dual-Core CPU E5200 @ 2.50GHz with 2.0 GB Memory
Operating System: Microsoft Windows 7 Ultimate 64 bit

COMMENTARY

W-O-W! Komodo 9 was constantly leading throughout entire tournament, with one anomaly point – at game #50 Stockfish has managed to score even against it, however, we decided to prolong the tournament for 10 more games and break the even. It was, after all, not fair to Komodo, because this engine was constantly leading through entire event. Additionally, we have repeated another 30 games, as a separate tournament competition, and Komodo again won with several games lead, so this results are very accurate.

It is worth noting that both Komodo 9 and Sotckfish 6 (64 bit editions) are on a very high level of chess, and only marginal differences exists between the two. However, Komodo 9 64-bit edition is obviously better optimized for variable clock speed, and manages, under the given circumstances, to score over the competition. Kudos!

Here is another tournament representation with table and chart analysis data. It will give you a little more insight into how the tournament has progressed over time (click on image thumbnail to enlarge):

TehnoBlog.org – Tournament 3 – StockFish 6 vs Komodo 9 x64 40-2 Variable Time – Table & Chart Data

Above table shows score progress per each engine in 5-games steps. Think about it as a collection of 5-games mini-tournaments. Right-Top chart displays total points accumulated over each 5 games. Right-Bottom chart displays relative score gains over each successive 5 games.

CONCLUSION

Well, we have mixed results this time. Stockfish 6 is obviously better at short-interval fixed time matches against Komodo 9, and the situation is completely reversed for variable-clock tournament style time allotments.

There is also a fact that Komodo developers are obviously completely focused on 64-bit architecture, and that they don’t care much about the older 32-bit one, where Stockfish still shines.

Stockfish is still our favorite engine, despite the fact it lost by slim margin in the last round — simply because it is open-source and free. And because it outperformed Komodo 6 in more than one occasion. However, for all those who are seeking top-notch engine with variable-clock superiority, Komodo is the obvious choice here.

Again, as a warranty, keep in mind that this tournament represents results on a specific test-system, comprised of hardware, operating system and different software revisions. Results may be completely different on other systems/computers, most importantly depending on architecture, numbers of CPUs and cores, as we have shown so far.

Fixed Time vs Variable Time

For chess engine tournaments, variable time seems to be a recommended method. We could not find a scientific explanation for this statement, except it is simply stated “as a fact”. We assume that it has to be something with engine efficiency.

If we take a comparison between chess engines and video compression algorithms, constant bit-rate will always lose against variable bit-rate in overall perceived video quality, which is, unlike constant setting, modulated for harder/more demanding frames, and allocated less for easy/static parts. Same could be applied to chess engine’s thinking time.

However, sometimes chess engine can make a wrong decision (“blunder” move), and that usually determines the winner later as the game/tournament progress.

DOWNLOAD

To stand to our claims and increase the credibility of the article, we provide a PGN files of the entire tournament for download (files can be opened with Notepad or similar text editor, or imported in chess clients): StockFish 6 vs Komodo 9 Chess Engine Tournament x64 (click to download zip archive).

Notice: we are deeply sorry for the inconvenience, PGN file with all played games between engines is available only for 40/2 tournament. Due to our technical error during setup, we are not able to provide PGN file for a 3 seconds / Fixed Time tournament.