This expert advisor code will scan the user's broker's market watch and extract symbols for which it will download all available ticks , or ticks up to a date.
It may help you to download all symbol history for your backtest , or create a custom chart from those ticks.
The terminals will cache the ticks on the data folder so make sure you have adequate hard drive space.
To facilitate download of the symbols we need a download manager first.
Structure CDownloadManager contains all the info we will need to retain.
struct CDownloadManager { bool m_started,m_finished; string m_symbols[],m_current; int m_index;
- The state of the download (started/finished)
- the list of the symbols to scan
- the current symbol
- and the index of the symbol being scanned
We will also need to read and write to the hard drive and since we are working with symbols we create 2 quick functions to write and read strings from binary files.
The save string to file function :
void writeStringToFile(int f,string thestring) { //save symbol string char sysave[]; int charstotal=StringToCharArray(thestring,sysave,0,StringLen(thestring),CP_ACP); FileWriteInteger(f,charstotal,INT_VALUE); for(int i=0;i<charstotal;i++) { FileWriteInteger(f,sysave[i],CHAR_VALUE); } }
It receives :
- file handle f , of a file opened for write and binary flags FILE_WRITE|FILE_BIN
- the string to write to the file
It writes an integer length of how many characters are in the string and then stores each character in the string.
The load string from file function:
string readStringFromFile(int f) { string result=""; //load symbol string char syload[]; int charstotal=(int)FileReadInteger(f,INT_VALUE); if(charstotal>0) { ArrayResize(syload,charstotal,0); for(int i=0;i<charstotal;i++) { syload[i]=(char)FileReadInteger(f,CHAR_VALUE); } result=CharArrayToString(syload,0,charstotal,CP_ACP); } return(result); }
It receives:
- file handle f of a file opened for reading as binary , flags FILE_READ|FILE_BIN
It reads an integer length of how many characters to expect at that point in the file.It proceeds to read each character into a char array and then create a string from that char array which is returned as the result of the load as a string.
Back to the CDownloadManager structure . We need a way to initialize the manager and fill it up from the market watch:
//+------------------------------------------------------------------+ //| grab symbols from the market watch | //+------------------------------------------------------------------+ void grab_symbols() { //! only from the mw ! int s=SymbolsTotal(true); ArrayResize(m_symbols,s,0); for(int i=0;i<ArraySize(m_symbols);i++) { m_symbols[i]=SymbolName(i,true); } }
Pretty straightforward :
- ask for how many symbols are in the market watch (active)
- resize our m_symbols array to receive them
- loop into the total symbols and request the name of the symbol
We are also responsible for managing the download of the symbol data so we'll need a function that is essentially the manager:
//+------------------------------------------------------------------+ //| Manage the download of symbols process | //+------------------------------------------------------------------+ void manage(string folder,string filename) { //essentially this starts or navigates to the next symbol //if set if(ArraySize(m_symbols)>0) { //if not started if(!m_started) { m_started=true; //go to first symbol m_current=m_symbols[0]; m_index=1; save(folder,filename); if(_Symbol!=m_current) { ChartSetSymbolPeriod(ChartID(),m_current,_Period); } else { ENUM_TIMEFRAMES new_period=PERIOD_M1; for(int p=0;p<ArraySize(TFS);p++) { if(_Period!=TFS[p]) { new_period=TFS[p]; break; } } ChartSetSymbolPeriod(ChartID(),m_current,new_period); } return; } //if started else { m_index++; if(m_index<=ArraySize(m_symbols)) { m_current=m_symbols[m_index-1]; save(folder,filename); if(_Symbol!=m_current) { ChartSetSymbolPeriod(ChartID(),m_current,_Period); } return; } else { m_finished=true; FileDelete(folder+"\\"+filename); Print("Finished"); ExpertRemove(); return; } } } else { Print("Please grab symbols first"); } //if set ends here }
How the system works :
- The chart opens , we need 1 chart , and a timer is set.
- That timer executes ,we cancel the timer
- We check if this is a new download or a continuing download
- If its a new download we set it up by grabbing all the symbols
- If its a continuing download we download data for the current symbol
This is the part of the code conducting the download on timer :
//+------------------------------------------------------------------+ //|Timer | //+------------------------------------------------------------------+ void OnTimer() { //--- if synchronized if(SymbolIsSynchronized(_Symbol)&&TerminalInfoInteger(TERMINAL_CONNECTED)==1) { EventKillTimer(); //--- load the system here if(MANAGER.load(MANAGER_FOLDER,MANAGER_STATUS_FILE)) { //--- system loaded so we are scanning a symbol here Comment("System loaded and we are processing "+MANAGER.m_current); //--- tick load //--- find the oldest tick available in the broker first int attempts=0; int ping=-1; datetime cursor=flatten(TimeTradeServer()); long cursorMSC=((long)cursor)*1000; long jump=2592000000;//60*60*24*30*1000; MqlTick receiver[]; long oldest=LONG_MAX; Comment("PleaseWait"); while(attempts<5) { ping=CopyTicks(_Symbol,receiver,COPY_TICKS_ALL,cursorMSC,1); if(ping==1) { if(receiver[0].time_msc==oldest) { attempts++; } else { attempts=0; } if(receiver[0].time_msc<oldest) { oldest=receiver[0].time_msc; } cursorMSC-=jump; if(limitDate&&receiver[0].time<=oldestLimit) { break; } } else { attempts++; } Sleep(44); Comment("Oldest Tick : "+TimeToString((datetime)(oldest/1000),TIME_DATE|TIME_MINUTES|TIME_SECONDS)+"\nCursor("+TimeToString((datetime)(cursorMSC/1000),TIME_DATE|TIME_MINUTES|TIME_SECONDS)+")\nAttempts("+IntegerToString(attempts)+")\nPlease wait for response..."); } //--- at this point we have the oldest tick //--- start requesting ticks from the oldest to the newest if(oldest!=LONG_MAX) { ArrayFree(receiver); datetime newest_tick=0; //--- receive the time of the last tick for this symbol stored in symbol_time datetime most_recent_candle=(datetime)SymbolInfoInteger(_Symbol,SYMBOL_TIME); while(newest_tick<most_recent_candle) { //--- request a new batch starting from the oldest time with the ticks limit specified int pulled=CopyTicks(_Symbol,receiver,COPY_TICKS_ALL,oldest,tick_packets); if(pulled>0) { //--- if we pull a new batch update our downloaded times newest_tick=receiver[pulled-1].time; oldest=receiver[pulled-1].time_msc; ArrayFree(receiver); } //--- timeout server requests , alter it if you want Sleep(44); Comment("Pulled up to "+TimeToString(newest_tick,TIME_DATE|TIME_MINUTES|TIME_SECONDS)+" so far"); } } else { Alert("Please close the terminal \n head over to the ticks folder \n and delete the empty folders"); ExpertRemove(); } //--- update the manager and move on MANAGER.manage(MANAGER_FOLDER,MANAGER_STATUS_FILE); } else { //--- grab the market watch symbols to start download Comment("Grabbing MW and starting"); MANAGER.grab_symbols(); MANAGER.manage(MANAGER_FOLDER,MANAGER_STATUS_FILE); } } }
