Metatrader 5 Libraries | TickCompressor - with compression of 1 tick to 2-3 bytes on average

Compression of tick data for storage in a compact form up to 3.5 times more compact than .tcs MQ files. And for fast work with them, as it takes less time to read 3 bytes than 60 bytes of MqlTick structure.

The file size for 2023 with Ask, Bid, time with additional ZIP compression of data blocks can be seen on the screenshot:

File size in .tcs format for 2023:

3.56 times compression.

To store ticks, the differences of Ask and Bid prices from the previous price are used. Often (up to 50...70% of all ticks) it does not exceed (-8...7) points, and it can be recorded with 4 bits. Ask and Bid are combined into 1 byte.
Plus 1 byte to store time difference from 0 to 255 milliseconds (in the code up to 229, values above 229 are used to encode ticks that are beyond -8...7 points).

If prices or times differ by larger values, they are packed into a larger number of bytes.

For additional compression you can apply ZIP archiving. The size of the data is reduced by up to 2 times.

Alternatively, compression to 3 bytes can be done, with Ask and Bid from -129 to 128 compressed to 8 bits or 1 byte each. Plus 1 byte for time - totalling 3 bytes for most ticks.
Sometimes(https://www.mql5.com/ru/forum/499639/page6#comment_58544810), if there are more ticks compressed to 2 bytes than to 4 bytes, it is more efficient to compress to 3 bytes. You have to look at the instrument statistics.
You can switch the maximum compression to 3 bytes with the command:

#define  compressTo3Bytes // compress ticks to 3 bytes instead of 2.

Tick elements for storage in compressed form

3 variants of tick elements for storage are programmed:

1. Ask, Bid, time_msc
2. Ask, Bid, time_msc, volume_real
3. All elements Ask, Bid, Last, time_msc, volume_real, flags (int volume is calculated from volume_real).

They can also be additionally compressed in ZIP. There will be 6 variants in total

method=1;//1...6 BidAsk_=1, BidAskVolume_=2, All_=3, BidAsk_Zipped=4, BidAskVolume_Zipped=5, All_Zipped=6

Before starting the compression, you need to pass the variant of ticks storage and some standard parameters used for calculations and normalisation of prices to the class.

    TickCompressor Compressor2;
    double VolumeStep_=SymbolInfoDouble(Symbol(), SYMBOL_VOLUME_STEP);
    Compressor2.Start(method,_Point,VolumeStep_,_Digits);

If the Expert Advisor uses flags, they can be restored from price changes by the command

#define  RestoreFlags // restore tick flags from change ask, bid, volume - will add 7% to tick generation time 931 instead of 869ms

The Expert Advisor for compression test is attached, it will give statistics on speed and compression rate. You can see an example of compression and decompression of ticks in it.

An example of the Expert Advisor for trading can be viewed here https://www.mql5.com/en/code/65821.

Statistics for 2 and 3 bytes compression:

Compression to 2 bytes:

Compression to 3 bytes

Ticks: 47707712 Compressed size: 135718404 Compressed 2862666420 bytes into 135718404 bytes ==> 4.74% Compress performance: 764 MB/s Compress performance: 13.4 Ticks (millions)/sec. Compress performance criterion: 281.7 Decompress performance: 3550 MB/s Decompress performance: 62.0 Ticks (millions)/sec. Decompress performance criterion: 1308.8 Statistics from expert https://www. mql5.com/en/code/65821 for BTCUSDT -------------------- Statistics: -------------------- 2 bytes: 70.1%, 50705359 ticks 4 bytes: 17.1%, 12350966 ticks 5 bytes: 12.7%, 9185484 ticks 6 bytes: 0.0%, 15274 ticks 11 bytes: 0.1%, 46214 ticks 12 bytes: 0.0%, 1 ticks 24 bytes: 0.0%, 1 ticks Total: 72303299 ticks, 197342036 bytes. Average: 2.729 bytes per tick UnZipped size: 197342036. Zipped size: 108302550. ZIP compression: 54.9% Average: 1.498 bytes per tick For EURUSD -------------------- Statistics: -------------------- 2 bytes: 66.2%, 29694779 ticks 4 bytes: 2.3%, 1022937 ticks 5 bytes: 31.5%, 14106637 ticks 6 bytes: 0.0%, 25 ticks 7 bytes: 0.0%, 8 ticks 11 bytes: 0.0%, 800 ticks 12 bytes: 0.0%, 3 ticks 13 bytes: 0.0%, 4 ticks 24 bytes: 0.0%, 1 ticks Total: 44825194 ticks, 134023609 bytes. Average: 2.99 bytes per tick UnZipped size: 134023609. Zipped size: 95495454. ZIP compression: 71.3 % Average: 2.13 bytes per tick

Ticks: 47707712 Compressed size: 169378137 Compressed 2862462720 bytes into 169378137 bytes ==> 5.92% Compress performance: 623 MB/s Compress performance: 10.9 Ticks (millions)/sec. Compress performance criterion: 183.9 Decompress performance: 3225 MB/s Decompress performance: 56.4 Ticks (millions)/sec. Decompress performance criterion: 952.6 Correct = true Statistics from Expert https://www. mql5.com/en/code/65821 for BTCUSDT -------------------- Statistics: -------------------- 3 bytes: 86.6%, 62644158 ticks 4 bytes: 0.6%, 412167 ticks 5 bytes: 12.7%, 9185484 ticks 6 bytes: 0.0%, 15274 ticks 11 bytes: 0.1%, 46214 ticks 12 bytes: 0.0%, 1 ticks 24 bytes: 0.0%, 1 ticks Total: 72303299 ticks, 236108596 bytes. Average: 3.266 bytes per tick UnZipped size: 236108596. Zipped size: 105802525. ZIP compression: 44.8% Average: 1.463 bytes per tick For EURUSD 3 bytes: 66.5%, 29801633 ticks 4 bytes: 2.0%, 916083 ticks 5 bytes: 31.5%, 14106637 ticks 6 bytes: 0.0%, 25 ticks 7 bytes: 0.0%, 8 ticks 11 bytes: 0.0%, 800 ticks 12 bytes: 0.0%, 3 ticks 13 bytes: 0.0%, 4 ticks 24 bytes: 0.0%, 1 ticks Total: 44825194 ticks, 163611534 bytes. Average: 3.65 bytes per tick UnZipped size: 163611534. Zipped size: 96541155. ZIP compression: 59.0% Average: 2.154 bytes per tick

Code examples

Tick compression

Block by block:

    int ZIPpos=0;//compressed byte counter
    if(Amount>ticks_per_block){// > 1 block - gluing blocks from tmp to Ticks2
       for(int start=0; start<Amount; start+=ticks_per_block){
          Compressor2.Compress(Ticks, tmp, start, (Amount > start + ticks_per_block ? ticks_per_block : Amount - start));
          ZIPpos+=ArrayCopy(Ticks2,tmp,ZIPpos); //copy to the end of Ticks2
       }
    }else{//1 block - unpacking directly into Ticks2
       Compressor2.Compress(Ticks, Ticks2, 0, Amount);
    } 

If you set the number of ticks in 1 block to be greater than the total number of ticks in the array, it will be compressed into 1 block.

If you always need compression into 1 block, you can use

Compressor2.Compress(Ticks,Ticks2); 

But the speed of decompression of such a large or very large block may be 2 times slower. Also there will be a large memory consumption for a large block.

Unpacking ticks

When unpacking it is desirable to know the number of packed ticks. The receiver array must have this size.

ArrayResize(Ticks3,Amount);

The size can be saved in a file, for example. And then use it when unpacking.

If the size is unknown, you can change the size inside the loop by the number of ticks in the block

       //slow down 
ArrayResize(Ticks3,total_ticks+ticks_per_block,10000000); //resize a large array - works slower than overwriting a small block

This code gets ticks block by block. If there is only 1 big block, it counts it correctly too. Ticks are not collected into a large array, but can be processed immediately by your Strategy(Ticks3[j]) strategy;

    while (ZIPpos<ArraySize(Ticks2)){
       nextSize=Compressor3.ArrToInt(Ticks2,ZIPpos);//size the next block, increase ZIPpos by 4

       uint s = ArrayCopy(tmp,Ticks2,0,ZIPpos,nextSize); // copy new block to tmp with size nextSize
       //slower by a factor of 3 ArrayResize(Ticks3,total_ticks+ticks_per_block,10000000); //resize a large array - works slower than overwriting a small block
       //total_ticks=Compressor3.DeCompress(tmp,Ticks3,nextSize,total_ticks);//unpack the block and add it to Ticks3.
       
       total_ticks+=Compressor3.DeCompress(tmp,Ticks3,nextSize,0); //unpack the block and overwrite it in Ticks3
       ZIPpos+=nextSize;
       for (int j = 0; j < ticks; j++){ Strategy(Ticks3[j]);}//strategy
    };

Collects ticks from all blocks into one big array:

    while (ZIPpos<ArraySize(Ticks2)){
       nextSize=Compressor3.ArrToInt(Ticks2,ZIPpos);//size the next block, increase ZIPpos by 4

       uint s = ArrayCopy(tmp,Ticks2,0,ZIPpos,nextSize); // copy new block to tmp with size nextSize
       //slower ArrayResize(Ticks3,total_ticks+ticks_per_block,10000000); //resize a large array - works slower than overwriting a small block
       total_ticks=Compressor3.DeCompress(tmp,Ticks3,nextSize,total_ticks);//unpack the block and add it to Ticks3
       
       //total_ticks+=Compressor3.DeCompress(tmp,Ticks3,nextSize,0); //unpack the block and overwrite it in Ticks3
       ZIPpos+=nextSize;
       //for (int j = 0; j < ticks; j++){ Strategy(Ticks3[j]);}//strategy
    };

Or a single line. Only 1 block should be recorded. If more - use the 2 code variants above.

total_ticks=Compressor3.DeCompress(Ticks2,Ticks3);

66201

Best MetaTrader Indicators + Profitable Expert Advisors