Yahoo! Data Source
5.0 FD_AddDataSource Parameters Details
1.0 Introduction
Yahoo! offers free historical data through internet. This data can be integrated into your unified database using two Yahoo! data sources explain in this document.
A small demo application is included (see c/src/demo/fd_yahoo). The makefile is in c/src/make. This tool allows to fetch historical data from the command line. Feel free to enhance or re-use that code in your own application.
You can add new symbols to your unified database in two ways:
(1) FD_YAHOO_WEB provides automatically an index using the FIDAL category/symbol guidelines.
(2) FD_YAHOO_ONE_SYMBOL adds Yahoo! symbols one at the time with a user defined category/symbol.
The two data sources type can co-exist. The recommendation is to start with FD_YAHOO_WEB and add the symbols that you find missing with one or many FD_YAHOO_ONE_SYMBOL sources. The advantage of FD_YAHOO_WEB is to have the index all build for you, while FD_YAHOO_ONE_SYMBOL is passing the responsibility to the FIDAL user. If you do not care about having an index, then using only FD_YAHOO_ONE_SYMBOL might be sufficient.
Terms Of Service
Data retrieved from Yahoo! must respect their terms of service (see http://www.yahoo.com for the legal details). In short, the data cannot be re-distributed. The FIDAL package does not contain any market data from Yahoo!, so you are solely responsible of the data you retrieve with your application.
Please visit http://www.yahoo.com and consider their multiple good services.
2.0 The Yahoo! Index
2.1 FD_YAHOO_WEB
In FIDAL, a data source must provide an index. Because Yahoo! does not provide such index directly, one must be build. With FD_YAHOO_WEB the index is automatically accessed and copied from fidalsoft.org (see "Advanced Feature" if you want to re-build that index yourself).
With this data source, the symbols DO NOT follow the Yahoo! convention, it is instead map to follow the Category Guideline and the Symbol Guideline. By following this guideline, you have a better chance to be able to switch to other data source without changing your list of stocks.
Example: Nortel on the Toronto Stock Exchange on Yahoo! is "NT.TO". When accessing from the FD_YAHOO_WEB data source, it is instead in the category "CA.TSE.STOCK" with the symbol "NT".
Because the number of symbols provided by Yahoo! can get very large, the list is decompose into multiple index. Someone wishing to access all available symbols will need to add these smaller index individually with FD_AddDataSource.
| List Of Index (param.location) | |
| us | United States of America |
| ca | Canada |
| uk | United Kingdom |
| de | Germany |
| es | Spain |
| fr | France |
| it | Italy |
| no | Norvegia |
| se | Sweden |
| dk | Danemark |
Here is an example building the index with both the American and Canadian stocks:
FD_AddDataSourceParam param;
memset( ¶m, 0, sizeof( FD_AddDataSourceParam ) );
param.id = FD_YAHOO_WEB;
param.location = "us";
FD_AddDataSource( unifiedDatabase, ¶m );
param.location = "ca";
FD_AddDataSource( unifiedDatabase, ¶m );
2.2 FD_YAHOO_ONE_SYMBOL
With this data source, you can map individual Yahoo! name to a category/symbol of your choice. You provide the Yahoo! name with the info field. The category and symbol can then be optionally specified. If the category field is NULL, the default category name is used ("ZZ.OTHER.OTHER"). If the symbol field is NULL, the info field is going to be used as the symbol.
Example(1): To add "2812.TW" using the default category/symbol and retrieve it later with FD_HistoryAlloc.
/* Add the symbol to the unified database */
FD_AddDataSourceParam param;
memset( ¶m, 0, sizeof( FD_AddDataSourceParam ) );
param.id = FD_YAHOO_ONE_SYMBOL;
param.info = "2812.TW";
FD_AddDataSource( unifiedDatabase, ¶m );
/* Retreive the data. */
FD_HistoryAllocParam paramHistory;
FD_History *history;
memset( ¶mHistory, 0, sizeof( FD_HistoryAllocParam ) );
paramHistory.symbol = "2812.TW";
FD_HistoryAlloc( unifiedDatabase, ¶mHistory, &history );
Example(2): To map "2812.TW", which is a Taiwanese stock, to the category "MyTaiwanStocks" and the symbol "Taichung".
/* Add the symbol to the unified database */
FD_AddDataSourceParam param;
memset( ¶m, 0, sizeof( FD_AddDataSourceParam ) );
param.id = FD_YAHOO_ONE_SYMBOL;
param.info = "2812.TW";
param.category = "MyTaiwanStocks";
param.symbol = "Taichung";
FD_AddDataSource( unifiedDatabase, ¶m );
/* Retreive the data. */
FD_HistoryAllocParam paramHistory;
FD_History *history;
memset( ¶mHistory, 0, sizeof( FD_HistoryAllocParam ) );
paramHistory.category = "MyTaiwanStocks";
paramHistory.symbol = "Taichung";
FD_HistoryAlloc( unifiedDatabase, ¶mHistory, &history );
3.0 Historical Data
Once the data source are added, the historical data can be retrieved like usual with FD_HistoryAlloc. Each call will cause a remote online access, there is no local caching of historical data.
The data can be provided in the following timeframe: FD_DAILY, FD_WEEKLY, FD_MONTHLY, FD_QUARTERLY and FD_YEARLY.
4.0 Advanced Feature
4.1 Index Local Caching
(This section applies only to FD_YAHOO_WEB)
Each time a data source is added, the index is fetched from fidalsoft.org. Although this takes only a few seconds, it may add up quickly if your application creates and destroys often its unified database. It is also noticeable if your application is stopped/restarted often (like a command line utility). To avoid to re-download constantly the index, this one can be stored on the local drive specified with FD_Initialize. The Yahoo! data source ALWAYS check first on that local drive (when available). The index files have the format "y_xx.dat" where 'xx' is the 2 letters country abbreviation.
If you want to force FIDAL to use a "fresh" data source from fidalsoft.org, simply erase all the y_xx.dat files on your local drive.
If the y_xx.dat file in your local drive is older than 4 days since it was originally build, FIDAL will always fetch a new fresh y_xx.dat file from fidalsoft.org and overwrite your existing one in the local drive.
4.2 Building Your Own Index
(This section applies only to FD_YAHOO_WEB)
Advanced user may want to build their own index because:
- they want fresh, very recently generated index.
- they do not want to have a dependency on fidalsoft.org
- they develop their own index for additional country.
It is easy to build your own index as fidalsoft.org do periodically. You just need to use the "gen_rdata" tool. Once all the .dat files are generated, just put these on your local drive cache or within the same directory of your application (see previous section). These .dat files will be used until they expire (~4 days later).
Take note that building your own index takes a LOT of time and bandwidth, in fact I simply do not encourage to do that if you do not have a broadband connection. Do "gen_rdata ?" to get the command line options. The easiest is to simply type "gen_rdata -u" to create the index for all the countries at once.
5.0 FD_AddDataSource Parameters Details
Here are a quick overview of how each FD_AddDataSource parameters are used for the Yahoo! data sources:
5.1 FD_YAHOO_WEB
'param.id'
Must be FD_YAHOO_WEB'param.location'
The country of the symbols added.
'param.info'
Unused. Must be NULL.
'param.category', 'param.country', 'param.exchange', 'param.type'
Unused. Must be NULL.
'param.symbol'
Unused. Must be NULL.
'param.username', 'param.password'
Unused. Must be NULL.'param.period'
Unused. Must be NULL.'param.flags'
Unused. Must be NULL.
5.2 FD_YAHOO_ONE_SYMBOL
'param.id'
Must be FD_YAHOO_ONE_SYMBOL'param.location'
Unused. Must be NULL.
'param.info'
Specify here the Yahoo! name. Example "2812.TW", "IBM", "^DJI", "NT.TO" etc...
'param.category'
Specify the category in the unified database. When NULL, the symbol will be put in the default category "ZZ.OTHER.OTHER".
'param.country', 'param.exchange', 'param.type'
Unused. Must be NULL.
'param.symbol'
Specify the symbol in the unified database. When NULL, the param.info field will be used as the symbol.
'param.username', 'param.password'
Unused. Must be NULL.'param.period'
Unused. Must be NULL.'param.flags'
Unused. Must be NULL.
