Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
| SHA1 Hash: | 2077c9d152c6f593397b314b90945dc1557c55d6 |
|---|---|
| Date: | 2012-11-15 14:19:56 |
| User: | dan |
| Comment: | Add words to lsmusr.wiki. |
Tags And Properties
- branch=trunk inherited from [84d5dea8fd]
- sym-trunk inherited from [84d5dea8fd]
Changes
Changes to www/lsmusr.wiki
855 <p>This option can only be set if the connection does not currently have 855 <p>This option can only be set if the connection does not currently have 856 an open write transaction. 856 an open write transaction. 857 857 858 <p>The default value is 1 (true). 858 <p>The default value is 1 (true). 859 859 860 <dt> <a href=lsmapi.wiki#LSM_CONFIG_AUTOFLUSH>LSM_CONFIG_AUTOFLUSH</a> 860 <dt> <a href=lsmapi.wiki#LSM_CONFIG_AUTOFLUSH>LSM_CONFIG_AUTOFLUSH</a> 861 <dd> <p style=margin-top:0> 861 <dd> <p style=margin-top:0> > 862 When a client writes to an LSM database, changes are buffered in memory > 863 before being written out to the database file in a batch. This option > 864 is set to the size of the buffer in bytes. The default value is 1048576. > 865 Increasing this value may improve overall write throughput. Decreasing > 866 it reduces memory usage. > 867 862 868 863 <dt> <a href=lsmapi.wiki#LSM_CONFIG_AUTOCHECKPOINT>LSM_CONFIG_AUTOCHECKPOINT</ 869 <dt> <a href=lsmapi.wiki#LSM_CONFIG_AUTOCHECKPOINT>LSM_CONFIG_AUTOCHECKPOINT</ 864 <dd> <p style=margin-top:0> 870 <dd> <p style=margin-top:0> > 871 This option determines how often the database is checkpointed (synced to > 872 disk). A checkpoint is performed after each N bytes (approximately) are > 873 written to the database file, where N is the value of this option. The > 874 default value is 2097152 (2MB). 865 875 > 876 <p> > 877 Increasing this value (say to 4MB or even 8MB) may improve overall write > 878 throughput. However, it is important not to checkpoint too infrequently, > 879 as: > 880 <ul> > 881 <li> <p> > 882 Space in the log file may only be reused following a checkpoint. > 883 Once a checkpoint has been completed, all application data written > 884 before the checkpoint is safely stored in the database file. This > 885 means the contents of the log file are no longer required, and so > 886 the next database writer may start writing a new log into the > 887 start of the existing file (overwriting the old data). > 888 As well as consuming disk space, large log files are undesirable > 889 as often the entire log file must be read during database recovery > 890 following an application or system failure. > 891 > 892 <li> <p> > 893 Similarly, space in the database file freed by the continuous > 894 incremental reorganization of the database file that LSM performs > 895 cannot be reused until after a checkpoint has been performed. > 896 So increasing the value of this parameter causes the space used > 897 by the pre-reorganization versions of data structures within the > 898 file to be recycled more slowly. Increasing the overall size of > 899 database files under some circumstances. > 900 </ul> 866 </dl> 901 </dl> > 902 > 903 <p>The roles of the LSM_CONFIG_AUTOFLUSH and LSM_CONFIG_AUTOCHECKPOINT options > 904 are discussed in more detail in the following section. 867 905 868 <h2 id=using_worker_threads_or_processes>6.2. Using Worker Threads or Processes 906 <h2 id=using_worker_threads_or_processes>6.2. Using Worker Threads or Processes 869 907 870 <p><i>Todo: Fix the following </p> | 908 <p>Usually, the database file is updated on disk from within calls to > 909 write functions - lsm_insert(), lsm_delete(), lsm_delete_range() and > 910 lsm_commit(). This means that occasionally, a call to one of these four > 911 functions may take signicantly longer than usual as it pauses to write or > 912 sync the database file. <a href=#automatic_work_and_checkpoint_scheduling>See > 913 below</a> for details. 871 914 872 <p>The section above describes the three stages of transfering data written < 873 to the database from the application to persistent storage. A "writer" < 874 client writes the data into the in-memory tree and log file. Later on a < 875 "worker" client flushes the data from the in-memory tree to a new segment < 876 in the the database file. Additionally, a worker client must periodically < 877 merge existing database segments together to prevent them from growing too | 915 <p>The alternative to updating the database file from within calls to write 878 numerous. | 916 functions is to use one or more background threads or processes to perform the > 917 actual work of writing to the database file. > 918 > 919 <ul> > 920 <li> Have all client connections set the LSM_CONFIG_AUTOWORK option to 0. > 921 This stops them from writing to the database file. > 922 > 923 <li> Arrange for a background thread or process to connect to the database > 924 and call the <a href=lsmapi.wiki#lsm_work>lsm_work()</a> function > 925 periodically. > 926 below</a>). > 927 </ul> > 928 > 929 <p>Further explanation of, and example code for, the above is > 930 <a href=#explicit_work_and_checkpoint_scheduling>available below</a>. > 931 > 932 <p>The following sub-sections provide a high-level description of the > 933 LSM database architecture and descriptions of how various parameters > 934 affect the way clients write to the database file. While this may seem > 935 in some ways an inconvenient amount of detail, it may also be helpful > 936 when diagnosing performance problems or analyzing benchmark results. 879 937 880 <h3 id=architectural_overview>6.2.1. Architectural Overview </h3> 938 <h3 id=architectural_overview>6.2.1. Architectural Overview </h3> 881 939 882 <p> The LSM library implements two separate data structures that are used 940 <p> The LSM library implements two separate data structures that are used 883 together to store user data. When the database is queried, the library 941 together to store user data. When the database is queried, the library 884 actually runs parallel queries on both of these data stores and merges the 942 actually runs parallel queries on both of these data stores and merges the 885 results together to return to the user. The data structures are: 943 results together to return to the user. The data structures are: ................................................................................................................................................................................ 1121 is to explicitly schedule them. Possibly in a background thread or dedicated 1179 is to explicitly schedule them. Possibly in a background thread or dedicated 1122 application process. In order to disable automatic work, a client must set 1180 application process. In order to disable automatic work, a client must set 1123 the LSM_CONFIG_AUTOWORK parameter to zero. This parameter is a property of 1181 the LSM_CONFIG_AUTOWORK parameter to zero. This parameter is a property of 1124 a database connection, not of a database itself, so it must be cleared 1182 a database connection, not of a database itself, so it must be cleared 1125 separately by all processes that may write to the database. Otherwise, they 1183 separately by all processes that may write to the database. Otherwise, they 1126 may attempt automatic database work or checkpoints. 1184 may attempt automatic database work or checkpoints. 1127 1185 > 1186 <verbatim> > 1187 /* Disable auto-work on connection db */ > 1188 int iVal = 0; > 1189 lsm_config(db, LSM_CONFIG_AUTOWORK, &iVal); > 1190 </verbatim> > 1191 1128 <p>The lsm_work() function is used to explicitly perform work on the database: 1192 <p>The lsm_work() function is used to explicitly perform work on the database: 1129 1193 1130 <verbatim> 1194 <verbatim> 1131 int lsm_work(lsm_db *db, int nMerge, int nByte, int *pnWrite); 1195 int lsm_work(lsm_db *db, int nMerge, int nByte, int *pnWrite); 1132 </verbatim> 1196 </verbatim> 1133 1197 1134 <p>Parameter nByte is passed a limit on the number of bytes of data that 1198 <p>Parameter nByte is passed a limit on the number of bytes of data that ................................................................................................................................................................................ 1216 checkpoint (implying that the snapshot has not changed and there is no need 1280 checkpoint (implying that the snapshot has not changed and there is no need 1217 to write it into the database file), lsm_checkpoint() sets *pnCkpt to zero 1281 to write it into the database file), lsm_checkpoint() sets *pnCkpt to zero 1218 and returns immediately. Otherwise, it checkpoints the database and sets 1282 and returns immediately. Otherwise, it checkpoints the database and sets 1219 *pnCkpt to the number of bytes written to the database file since the 1283 *pnCkpt to the number of bytes written to the database file since the 1220 previous checkpoint. 1284 previous checkpoint. 1221 1285 1222 <p>The number of bytes written to the database since the most recent checkpoint 1286 <p>The number of bytes written to the database since the most recent checkpoint > 1287 can also be using the <a href=lsmapi.wiki#lsm_info>lsm_info()</a> API 1223 can also be using the lsm_info() API function. As follows: | 1288 function. As follows: 1224 1289 1225 <verbatim> 1290 <verbatim> 1226 int nCkpt; 1291 int nCkpt; 1227 rc = lsm_info(db, LSM_INFO_CHECKPOINT_SIZE, &nCkpt); 1292 rc = lsm_info(db, LSM_INFO_CHECKPOINT_SIZE, &nCkpt); 1228 </verbatim> 1293 </verbatim> 1229 1294 1230 <verbatim> 1295 <verbatim> 1231 int nOld, nLive; 1296 int nOld, nLive; 1232 rc = lsm_info(db, LSM_INFO_TREE_SIZE, &nOld, &nLive); 1297 rc = lsm_info(db, LSM_INFO_TREE_SIZE, &nOld, &nLive); 1233 </verbatim> 1298 </verbatim> 1234 1299 1235 <verbatim> < 1236 int lsm_flush(lsm_db *db); < 1237 </verbatim> < 1238 < 1239 <h3 id=compulsary_work_and_checkpoint_scheduling>6.2.4. Compulsary Work and Chec 1300 <h3 id=compulsary_work_and_checkpoint_scheduling>6.2.4. Compulsary Work and Chec 1240 1301 1241 <p>Apart from the scenarios described above, there are two there are two 1302 <p>Apart from the scenarios described above, there are two there are two 1242 scenarios where database work or checkpointing may be performed automatically, 1303 scenarios where database work or checkpointing may be performed automatically, 1243 regardless of the value of the LSM_CONFIG_AUTOWORK parameter. 1304 regardless of the value of the LSM_CONFIG_AUTOWORK parameter. 1244 1305 1245 <ul> 1306 <ul>