Reduce the Size of the Historic Granule Index

Each ClearSCADA server maintains an index of the historic data granules in that server (a granule is a file that holds the historic data for one item for a period of one week). The size of the index, and the amount of memory it requires, is dependent on:

• The number of items with historic data
• The amount of time for which the historic data is kept online.

To configure the Historic Granule Index feature:

1. Access the ClearSCADA Server Configuration Tool, (see Accessing the ClearSCADA Server Configuration Tool).
2. Expand the Historic Configuration branch.
3. Select the Historic Data entry.
4. Use the Index After check box to enable or disable the long term storage index feature. (Select to enable the feature; clear to disable the feature).

   

5. If you enable the long term storage index, use the Index after weeks field to define how long the historic data granules will be in the primary index. Granules that are older than the number of weeks you specify will be in the secondary index. The period is measured from the start of the week in which the server was last started. Unlike the Keep Online for, Archive After and Future Data time periods, the primary index time is not a sliding window (it does not move forward week-by-week as time goes by).

   The higher the number/duration, the more memory will be required for the index.

   When deciding how many weeks’ worth of historic data to store in the primary index, you should consider the following factors:

   • For how long is the historic data being kept online? This is defined by the Keep online for setting (see Specify the Duration to Retain Historic Data Online)
   • How much memory is available in the server?
   • How far back in time do you need to create new historic records, delete existing records or optimize granules?
   • How far back in time do you need to be able to transfer historic data to the server’s partners (if any).
   • If archiving is also enabled, then the period must be equal to or greater than the archive period (defined by the Archive After setting (see Define the Archive Time for Historic Data).

   Using this information you can determine the best compromise between the memory required by the index and the restrictions imposed by using the long term storage index. For more information on the amount of memory required, see Calculate the Amount of Memory Required by the Historic Granule Index.

   By using ClearSCADA's's long term storage index feature, you can reduce the amount of memory required by the index. However, using the long term storage index feature imposes certain restrictions.

   The long term storage index works by dividing the online period into two sections, called the primary index and the secondary index. The primary index contains the most recent historic data (for example, the last 12 months' data) and the secondary index contains the older historic data (for example, data for the 24 years previous to the 12 months). The granules in the secondary index take up much less memory, and so the index is smaller. For example, a 64-bit system with 100,000 historic aggregates kept online for 25 years with 1 year in the primary index and 24 years in the secondary index will require up to 1.4GB of memory. If the long term storage index is not used, the same system would require 11.7GB (so the long term storage index offers an 88% saving).

   The following table shows the memory required for each granule in the historic index:

   	64-bit Server	32-bit Server
   Primary Index	96 bytes per granule	68 bytes per granule
   Secondary Index	8 bytes per granule	8 bytes per granule

   When the long term storage index is enabled, granules that fall within the secondary index have certain restrictions:

   • The granules are read-only.
     • New records cannot be added to the granules. The CHistoryBase:LoadDataValuesEx() method cannot be used for adding records to the granules.
     • Existing records cannot be deleted from the granules.
     • Optimization cannot be applied to the granules (see Historic Optimization).
   • The granules are not transferred to any partner servers. However, a permanent standby server is able to receive updates to the granules in its secondary index from its partners. This allows the long term storage index to be used on a permanent standby server when it is not used on its partners. This is true even if there is an overlap between the secondary period on the permanent standby server and the online period on its partners.

     Example:

     If a granule in the Main server's Primary index is altered, the data is transferred to the Permanent Standby servers. The transfer occurs even if the granule falls in the time period covered by Permanent Standby servers' indexes.

     

   When the long term storage index is disabled, all granules are stored in the primary index – the secondary index is not used.

   When a granule in the secondary index is queried, it is promoted into the primary index (and therefore will require more memory). It is demoted back into the secondary index when it is purged from the loaded granule cache. (For more information on the loaded granule cache, (see Define the Historic Data Cache Size).

6. Right-click on the server icon and select the Apply Changes option to update the server’s configuration. The changes will come into effect the next time the server is restarted.

If you are using both archiving and the long term storage index, the granules in the secondary index can be archived. When an archive volume is mounted, all of the granules contained on the archive volume are stored in the primary index, regardless of their age.