Terra Help and Information

bullet1 Help!

bullet2 Opening books

bullet3 What is an Opening book

An opening book is a database that helps the program to play in the opening phase. Most chess programs has some kind of opening book with essentially the same main functions but with different implementations. When it's the programs turn to move it will first look in the opening database in order to find a move. If there is no move the ordinary search takes over. When we talk about learning functions it is almost always Book learning  that is meant.

bullet3 Terra's Opening books

Terra is using it's own format for opening books. The book contains both moves and statistics connected to each move. This statistics is used to make choices between the moves .

There are two types of move statistics:

Fixed values
When the opening book is created , it is by loading games in PGN-format. Together with each move the sum of all the results for that move is stored. These figures are not changed unless we merge in more games to the book .
Dynamic values
When a game is over Terra will go through all the selected book moves in the game and update the result for each of the positions. This statistics is based only on Terra's own results and is changing all the time.
We can also have User set values  (i.e. by the bookeval command or move nags like e4! h6?). When we have User set values, Terra's own results are not stored and will not have any effect on the Dynamic values.


It is possible to have more than one opening book, but Terra can only use one at a time. It is possible to merge two books together  into one.


The PostBook

This is a special book that Terra builds automatically. None of the functions applied to the ordinary book can be used here. Everything is automatic with the post book.


When the book line ends Terra has to start ordinary searching. The first about 10 moves after being out of book is Terra keeping track of. Depending of the evaluations and the game score, Terra is saving statistics together with these moves. In a way Terra is expanding it's book with this method. The PostBook format is not the same as the book format and the two are kept apart. The PostBook has the default name postbook.dat and is an ASCII-file in the Book directory. The name can be changed in Terra.ini or with the command Postfile filename. When upgrading Terra or after changing opening books, it is probably best to just delete postbook.dat. A new postbook will be build up by Terra.

If you get troubles with the postbook, just delete it.

  • How to create an opening book
    In order to create a new opening book you have to collect some good games and save them in a PGN-file. If you want a free product I certainly recommend Scid. From Scid you can export the PGN-files that are free from errors and that Terra can read.

    1. Find and collect good games
    2. Load them into Scid or some other database
    3. Produce (in Scid Export) a PGN-file and place it in the Book directory.
    4. Start Terra and run the command createbook xxxx yyyy.pgn s t  where xxxx is the new book file name, yyyy.pgn is the PGN-file, s is number of half moves that will at most be included in the book and t is the minimum number of games. The new book will be put in the Book directory.
    5. If you want to hand tune some opening lines (and I think you always should), i.e. set the opening moves e4 and d4 as always good whatever results Terra get, proceed as described in the next section . There is an example of such a handcoding in the file ownscore.pgn.
    6. In order to use the new opening book, set Bookfile=xxxx  in the file Terra.ini or run the command Bookfile xxxx  within Terra.


    When loading the PGN-file Terra uses the memory available for the hash tables. In order to get the book creation process as fast as possible, set the hash table size as large as possible.


    The Post Book

    The Post book is completely handled by Terra itself. The only thing you can do is to delete it. It is possible to edit this ASCII-file but then you have to have detailed knowledge about the format.


     
  • How to add more PGN files into an existing book
    If you want to expand the current book with more games it's possible. Be sure to use unique games and not the old ones from the existing book once again. It will make the statistics about moves in the book to be repeated and doubled. (If this isn't what you want...)

    Start Terra and run the command createbook xxxx yyyy.pgn s t  where xxxx is the  book file, yyyy.pgn is the PGN-file, s is number of half moves in the PGN file that at most will be used and t is the minimum number of games that the move has to be in (otherwise it will not be included in the book)


     
  • How to merge two books
    In order to merge two books into one use the command mergebooks xxxx yyyy z  where xxxx and yyyy are two book files placed in the book directory and z is the minimum number ov games(lines) that a move has to be in to be inincluded. The resulting merge will overwrite the first book xxxx.


     
  • How to make changes to the opening book
    There are two types of changes possible:

    Changing the Dynamic statistics
    It is possible to change this statistics manually, by commands or by moves with nags. The purpose is to control the opening lines by hand. The command to use is Bookeval . In fact it is possible to collect all the commands in a command file  and run the whole batch.
    Add more games
    It is possible to expand an existing book by adding another PGN-file .


    Changing the Dynamic statistics with the command BookEval

    It is possible to manually control what moves in the Book Terra will select. The command is Bookeval.  First make the moves including the move you want to set a value for. Now set your own value with Bookeval xxxx where xxxx is either a value between 0-100, clear or one of !,?,!!, !? or ?!.

    The translation is:

    cmd
    Value
    Meaning
    ?
    0
    never play this move.
    !?
    15
    very seldom play this move.
    !  
    75
    very often play this move.
    !!
    100
    always play this move. If there are more than one such move a random selection will take place.
    clear
    -
    Clear all User set values or Terra statistics for this move.

    By giving numbers instead of signs you can fine tune it a bit.


    In the position preceding the actual move (command undo) you can look at all moves and values with commands Bstat or Bmoves .


    Changing the Dynamic statistics with moves and nags

    It is possible to set your own values for a certain move by keeping a PGN-file with lines and moves that you want to control.

    The same table as above is valid, except from the clear command and the fact that you can give numbers in the BookEval command.

    Such a PGN-file could look like:

    [Event: "Line 1"]
    1. e4!! c6!? 2.d4!!
    [Event: "Line 2"]
    1. d4!! Nf6!
    2. c4! d6!
    [Event: "Line 3"]
    1. d4 d5!
    2. c4 a6?

    You will need at least one tag (in this case the Event tag) in order to separate the lines from each other. You can use whatever tag you want.i.e. [Result 1-0] or [Date: "2001-01-01"] etc.

    This file says that:

    • 1.e4 and 1.d4 are set to "always play" (they will be selected randomly and no other moves will be considered)
    • After 1.d4 we will consider both Nf6 and d5 as good moves to play often. In this case 75% each. See in " How Terra selects book moves " in order to find out how this is computed.
    • After 1. e4 we will play c6 very seldom (15% of the times we reach this position).
    • After 1.d4 d5 2.c4 we will never play a6. This move is placed in the Avoid list and will not be selected by PostBook or by Terra's search routine.
    • etc

    Note that 1.d4 had no nags in "Line 3" and the last nag was !! which will be used.  If we have conflicting nags i.e. one line with 1.f4! and later on 1.f4? the last found nag in the file will be used.




     
  • How to use different opening books
    It is possible to have many book files and switch between them.

    Just run the command BookFile file  or set BookFile=file  in terra.ini , where file is the Book file to use. In Both ways it will change to the new book file if there is such a file in the Book directory.


     
  • How Terra selects a move from books
    The information in the books together with the book and learning settings  controls which moves are selected.


    Let's see how Terra will select a move from either the book or the PostBook file.

    1. First of all Terra is looking in the Book. Do we have any book moves in the current position? If no book move is selected then Terra is looking in the PostBook file. If there are one or more moves in the PostBook file the best one (from previous results) is selected. Within the book we first chose from what group to select the move. Either the set values or the game statistics. Both the general game statistics (Fixed values) and Terra game statistics (Dynamic values) are grouped together. One move might have Terra statistics and another not. The first move will use the Terra (Dynamic) statistics and the second move will use the general (Fixed) statistics ( (if book and learning settings are met ).
    2. The values of RandomType, BookLimit, MinBookGames, MinTerraGames and TerraLim  are overall limiting what information to use and how.
    3. All book moves has at least the Fixed values and some have Dynamic values  Dynamic values are either "User set values" or Terra Game statistics. If there are any moves with Dynamic values that are set to the value 100, that means always play!  The move is immediately picked, regardless of RandomType. If there are more than one such move one of them is selected by random.

     
    • Selecting from "User set values"
      When a move has a " User set value " there will be no Terra learning for that move. The "User set value" will always be unchanged by Terra and used as described below.

      Terra will handle the group of moves with "User set values" apart from the other moves. In a certain position Terra will first decide from which group to select a move. From the "User set values" group or the other group (if there are moves from both groups of course). In order to select a group we try to evaluate the total weight of all the user set values. Each set value is between 0-100 and represents a weight telling us how often the move is supposed to be selected. v1, v2,....,vn are all these values form move# 1 - n. P = 1 - (100-v1)/100 * (100-v2)/100 *...* (100-vn)/100 gives the probability P that the "User set values" group will be selected this time. If Random(0,100) < (P*100) we will select the "User set values" group otherwise we will select the other one.


      The Roulette Wheel method

      If the "User set values" group is selected Terra will select the move from this group with the use of the Roulette Wheel method:

      • Take the sum U of all the values within the group. Set R = Random(0,U).
      • Then go through each move in random order and add the value v for each move. The sum S = S+v  for each move.
      • When that sums up to a value S >= R, we will select the move that made S to exceed R.


      Special case

      If we have at least one move with a set value = 100, then Terra will select that move without any selection method at all. In fact the method to select between "User set values" group and the other moves would anyway always select "User set values", if we have one or more moves with the value 100.

      If there are more than one such move, Terra will randomly select one of these.


       
    • Selecting from statistics
      The "Fixed values and Terra statistics group" (The opposite to "User set values") are treated as one group together. One move might have Win% from Terra and another move might have frequency or win% from General values. All these are compared in the same selection process.

      In order to select a move from game statistics, we have to take into consideration the settings of   RandomType, BookLimit, TerraLim, MinBookGames and MinTerraGames.   The BookLimit, TerraLim, MinBookGames and MinTerraGames tell us what moves to just sort out. If RandomType=1 (Terra's Win% + Fixed values frequency), the frequency has to be translated to something comparable with Win%. We do like this:

      Each frequency value is translated by 100*f/FSum where f is the frequency and FSum is the sum of all frequencies used.

      If RandomType is one of the "select best" choices, we just selects the best value from all moves meeting the settings mentioned above.

      If RandomType is one of the random choices we are using the Roulette Wheel method.

      See "Selecting from User set values"


       
    • Selecting from Postbook
      If nothing is selected from the book, the PostBook will be searched for a move.

      Within PostBook Terra saves information about moves outside the book. Depending on the saved information Terra will:

      • Completely ignore it
      • Place moves into the avoid list
      • Select the move with the best history

      No randomized selection process will take place with PostBook moves.


       
    • Example
      Example: White plays 1.e4 and now it's Terra's time with the following book moves


       
      Fixed values
      Terra values or Set values
      		  
      Move
      # of games
      Win%
      # of games
      Win%
      Remark
      e5
      5783
      43
      5
      60
      		  
      c5
      5230
      48
      127
      75
      Set to often play (=!)
      e6
      1329
      36
      1
      0
      		  
      c6
      748
      26
      		      
      Nf6
      276
      27
      127
      0
      Set to never play (=?)
      d5
      43
      29
      127
      15
      Set to play seldom (=?!)
      d6
      21
      34
      		      
      Nc6
      6
      48
      127
      0
      Set to never play (=?)
      b6
      5
      27
      		      
      g6
      2
      0
      		      
      a6
      2
      100
      		      

      Here we have a lot of moves. Some with set values (yellow rows) , some with Terra statistics and some with only the Fixed values. Let's follow the procedure and pick one move. Assume that RandomType=1 (Terra stat + frequency) and BookLimit=33, MinTerraGames=3, TerraLimit=40 and MinBookGames=4

      First: Do we have one or more moves with Set value=100 (always select)?  No. (Otherwise that move would be selected directly)

      Secondly: Compute the probability of using the group of moves with "Set values". Here c5 and d5. (Nf6 and Nc6 are set to never be played) P=75*(100-15)/100 = 63,75. Let's look at two different outcomes of rolling the dice.

      1st case) Random(0,100) returns 60, meaning that the "Set values" group is selected.

      Now we have c5 and d5 with the values 75 and 15 respectively. Using the Roulette Wheel method: The Sum is 85. Random(0,85) gives S=45. Start by taking the first move by random, say d5. Gives 15. Next move c5 gives total 85 and S is exceeded. The selected move is c5.

      2nd case) Random(0,100) returns 70, meaning that the other group of move is selected.

      So we have to select a move from the moves e5, e6, c6, d6, b6, g6 and a6.

      RandomType=1 tells us to use Terra values if there are any (enough due to MinTerraGames). What do we have now?

      Move
      What value to use
      Why
      e5
      60
      5 Terra games > MinTerraGames=3. Use the Terra Win% which exceeds the TerraLimit value
      e6
      1329
      Only 1 Terra game says select from fixed values. RandomType=1 says use Frequency
      c6
      -
      not selected du to BookLimit=33.
      d6
      21
      use Frequency due to RandomType=1
      b6
      -
      not selected du to BookLimit=33.
      g6
      -
      not selected due to MinBookGames=4
      a6
      -
      not selected due to MinBookGames=4

      Shaded means not selected for some reason.

      All the frequency values will be translated: 1329 /(FSum) * 100 = 43 and the others 10,  0.2 (1 will be used) when FSum=13445 giving us this new table:

      Move
      What value to use
      e5
      60
      e6
      63
      d6
      1

      Now select a move with the Roulette Wheel method ....

      S = 60+63+1 = 124. S=Random(0,124) Let's say 113.

      Randomly pick move after move until 113 is reached or exceeded.

      e6: 63     63

      d6:   1  64

      e5:    60 124   <==  Exceeded!

      N.B. If the value for e5 had not exceeded the value of TerraLimit the move would be placed into the Avoid List and not selected at all.


       

bullet3 How to set the book options

Books and learning are so integrated with each other so you'll better look into how to set the learning options .

bullet3 Learning and books

Book learning is the most used learning scheme for chess programs. Terra has both book learning and post book (after book) learning. The learning function, stores game results and/or the search score together with each move. This is a quite effective way to avoid players trying to win over and over again with the same opening variation. It is more fun to play a program that avoids bad experiences!



» See also: Learning

Please give your comments in WinBoard Forum or email me. This document was updated 2004-08-02.