This documentation is divided into three parts: I Description of the dataset sp500.dat II Instructions for DOS users III Some uses of the dataset David Cummins cummins@stat.ncsu.edu I *********** THE DATASET *********** ** NAME ** sp500.dat.Z -- Daily S&P 500 Index, Daily NYSE Volume, 1928-1987 ** SIZE ** 16127 observations, 8 variables ** DESCRIPTION ** The dataset contains daily data on the trading volume and price changes on the New York Stock Exchange for the years 1928-1987. Unfortunately, the volume and price are for different things: volume is for NYSE composite and the price is the S&P 500 index. The first five lines of the file are shown below: S&P NYSE DATE DAY HOL WKND GAP INDEX %CHANGE VOLUME 01/04/28 3 0 0 1 17.72 -0.22523 3228 01/05/28 4 0 0 1 17.55 -0.95937 3385 01/06/28 5 0 0 1 17.66 0.62678 2932 The first two lines are header lines, so that a program reading this file needs to skip over those lines. VARIABLE DESCRIPTION: DAY = Day of the week (1=Monday, 2=Tuesday, 3=Wednesday, 4=Thursday, 5=Friday, 6=Saturday) HOL = Holiday dummy (1=preceded by a holiday, 0=otherwise) WKND = Weekend dummy (1=preceded by a weekend, 0=otherwise) GAP = No. of days since preceding trading day NYSE VOLUME = the New York Stock Exchange Composite Volume, in thousands. This data was originally in a quite different format. For instance the S&P index was not given, but rather the log differences, specifically, 100*( log X(t) - log X(t-1) ) . Also the volume was actually the log of the volume. The transormations were inverted, and the 1/4/1928 value was fed to the algorithm, which iterated until it reached 12/31/1987. Confidence in the data was increased when the value the algorithm ended up with for 12/31/1987 was 247.08, which is exactly what the S&P was on that date. Also the data were graphed and compared to a published graph of the S&P 500, and qualitatively the match was perfect. However, there can be no assurance that the data is exactly correct for every day in between 1/4/28 and 12/31/87. Also keep in mind that before 1957 the daily S&P 500 did not exist. Thus, from 1928 thru 1956 the data is from the S&P 90 (industrials and transpor- tation), adjusted to make a smooth and meaninful transition with the S&P 500. I also have the S&P 500 index from 4/2/90 to the present, but not from 12/31/87 so there is a very frustrating gap from jan 1988 thru march 1990. If anyone has this data, please send it to me and then I will tack on what I have and we will have the gap filled in nicely. The other frustration is of course that the volume is not the S&P 500 volume. II *********** FOR DOS USERS *********** I am including a file called DECOMP.EXE. This is the dos equivalent to the unix uncompress command. Also, the file COMPRESS.EXE is equivalent to the unix compress command. So you can download compressed files to a dos machine, and uncompress them there. Remember if you want to download the sp500.dat file to a dos machine, you need to use the unix2dos utility so that it will have the proper carriage control at the end of each line. uncompress it, convert it to dos format, then compress that file, then download it (use binary setting) to your dos machine, then decompress it using the dos decomp utility. SCATTER.EXE I am also including a program to make scatter plots from an input data file. Since the s&p file has over 16,000 records, it may not be possible to import it into your spreadsheet and graph it from there. This program can handle any size file, because it doesn't store its arrays in memory. The program itself teaches you how to use it, but for the sp500.dat file, if you wanted to graph the S&P index over time, you would notice that it is the sixth column in the file, and also that you don't want to graph the first two lines since they are header info, and you would enter the following: scatter sp500.dat 6 6 startrec=3 endrec=99999 at the dos prompt, in the directory where the sp500.dat file resides. The first 6 is the x axis, and the second 6 is the y axis. If you make the x and y the same, the program knows to map the y column against time. If you wanted to graph the index against volume, you could enter scatter sp500.dat 8 6, which would graph volume on the x axis and the index on the y axis. (not a pretty graph!) III *********** USES OF THE DATA *********** The day of the week fields can be used to experiment with anomoly hypotheses, such as the "Monday effect". To illustrate, below are some output from a program I wrote to summarize the S&P 500 with respect to these indicator variables: STANDARD STATISTICS OF THE VARIABLES Variable N Mean Std Dev Minimum Maximum ----------------------------------------------------------------------- GAP 16127 1.3587152 0.7455616 1.0000000 12.0000000 INDEX 16127 60.4954995 58.2018503 4.4000000 336.7700000 %CHANGE 16127 0.0230446 1.1580637 -20.4669000 16.6095900 VOL 16127 17130.27 36227.80 83.0000000 608100.00 ----------------------------------------------------------------------- Looking at the %CHANGE we can see that there was a one day drop of -20.467% This, as you could guess, was the great correction of October 1987, called Black Monday. That record is given below, with the day before and the day after: 10/16/87 5 0 0 1 282.70 -5.15969 338500 10/19/87 1 0 1 3 224.84 -20.4669 604300 <-- Black Monday 10/20/87 2 0 0 1 236.83 5.33268 608100 Below are some summary statistics designed to examine the effects of GAP, which is how long since the last trading day, DAY(in particular 1=Monday), and WKEND, which basically collapses the data into two halves: the first part contains regular business days and the second part contains Mondays and Tuesdays when Monday was not a trading day. The questions to consider are, does the stock market go down on Mondays? Does the market go down after a weekend? Does the market go down or up after a long gap in trading (caused by holidays, bad weather, etc.). Here are the summary statistics (no conjectures are offered): STANDARD STATISTICS OF THE %CHANGE, BY GAP ----------------------------------------------------------------------- | | %CHANGE | %CHANGE | %CHANGE | %CHANGE | | |------------+------------+------------+------------| | | N | MIN | MAX | MEAN | |-----------------+------------+------------+------------+------------| |GAP | | | | | |-----------------+------------+------------+------------+------------| |1 | 12686.00| -10.16| 12.53| 0.07| |-----------------+------------+------------+------------+------------| |2 | 1339.00| -12.34| 8.37| -0.15| ----------------------------------------------------------------------- |3 | 1873.00| -20.47| 9.63| -0.18| |-----------------+------------+------------+------------+------------| |4 | 223.00| -6.73| 3.98| -0.24| ----------------------------------------------------------------------- |5 | 5.00| 0.04| 1.16| 0.51| |-----------------+------------+------------+------------+------------| |12 | 1.00| 16.61| 16.61| 16.61| ----------------------------------------------------------------------- -------------------------------- | | %CHANGE | | |------------| | | STD | |-----------------+------------| |GAP | | |-----------------| | |1 | 1.10| |-----------------+------------| |2 | 1.55| -------------------------------- |3 | 1.11| |-----------------+------------| |4 | 1.17| -------------------------------- |5 | 0.53| |-----------------+------------| |12 | .| -------------------------------- Since I am a statistician I feel compelled to point out that these summary statistics only give us a cursory view of the data. We may get certain hunches from them which are not proven statistically. For instance if you look at the MEAN column, it looks like we get an increasing "Weekend Worry" effect as you go from 1 to 2, 3, and 4, but then for the larger gaps, it's as if people have had to wait too long and are anxious to do some trading now. This is mere conjecture which needs to be set to a statistical test. An interesting observation: apparently there was one trading day that had 12 days of no trading before it. Naturally since N=1 there cannot be a standard deviation calculated, thus the "." where a number should be. That record is given below, along with the trading days before and after: 03/02/33 4 0 0 1 5.65 -2.07972 1004 03/03/33 5 0 0 1 5.84 3.36283 1414 03/15/33 3 1 1 12 6.81 16.60959 3066 <--large gap: bad weather? 03/16/33 4 0 0 1 6.93 1.76211 3301 STANDARD STATISTICS OF THE %CHANGE, BY DAY ----------------------------------------------------------------------- | | %CHANGE | %CHANGE | %CHANGE | %CHANGE | | |------------+------------+------------+------------| | | N | MIN | MAX | MEAN | |-----------------+------------+------------+------------+------------| |DAY | | | | | |-----------------+------------+------------+------------+------------| |1 | 2932.00| -20.47| 8.14| -0.19| |-----------------+------------+------------+------------+------------| |2 | 3036.00| -10.16| 12.36| 0.03| ----------------------------------------------------------------------- |3 | 3056.00| -9.92| 16.61| 0.10| |-----------------+------------+------------+------------+------------| |4 | 3015.00| -8.88| 9.52| 0.05| ----------------------------------------------------------------------- |5 | 3004.00| -8.70| 8.29| 0.07| |-----------------+------------+------------+------------+------------| |6 | 1084.00| -5.25| 8.37| 0.15| ----------------------------------------------------------------------- -------------------------------- | | %CHANGE | | |------------| | | STD | |-----------------+------------| |DAY | | |-----------------+------------| |1 | 1.26| |-----------------+------------| |2 | 1.13| -------------------------------- |3 | 1.23| |-----------------+------------| |4 | 1.08| -------------------------------- |5 | 1.10| |-----------------+------------| |6 | 1.06| -------------------------------- Day=6: way back there, they did some trading on Saturdays. STANDARD STATISTICS OF THE %CHANGE, BY WKEND ----------------------------------------------------------------------- | | %CHANGE | %CHANGE | %CHANGE | %CHANGE | | |------------+------------+------------+------------| | | N | MIN | MAX | MEAN | |-----------------+------------+------------+------------+------------| |WKEND | | | | | |-----------------+------------+------------+------------+------------| |0 | 12998.00| -10.16| 12.53| 0.07| |-----------------+------------+------------+------------+------------| |1 | 3129.00| -20.47| 16.61| -0.18| ----------------------------------------------------------------------- -------------------------------- | | %CHANGE | | |------------| | | STD | |-----------------+------------| |WKEND | | |-----------------+------------| |0 | 1.11| |-----------------+------------| |1 | 1.32| -------------------------------- If we left out Black Monday, considering it a fluke and wanting to observe the market's NORMAL behavior, we get the following results. Actually, to truly remove the effects of that day, we would have to reconstruct the S&P index as if that day never happened, which I have not done here. Also, if that day never happened, it is certain that the daily %changes thereafter would have been different, due to a completely different state of market psychology. Thus, it is really not possible to adjust for Black Monday as a data outlier. However, here are the results, which seem to indicate that the Monday Effect (and other effects) still exists: ******* Leaving 10/19/87 out ********** STANDARD STATISTICS OF THE %CHANGE Variable N Mean Std Dev Minimum Maximum ----------------------------------------------------------------------- GAP 16126 1.3586134 0.7454727 1.0000000 12.0000000 ignore->INDEX 16126 60.4853082 58.1892633 4.4000000 336.7700000 %CHANGE 16126 0.0243152 1.1468028 -12.3362000 16.6095900 VOL 16126 17093.86 35932.61 83.0000000 608100.00 ----------------------------------------------------------------------- STANDARD STATISTICS OF THE %CHANGE, BY GAP ------------------------------------------------------------------------- | | %CHANGE | %CHANGE | %CHANGE | %CHANGE | | |------------+------------+------------+------------| | | N | MIN | MAX | MEAN | |-------------------+------------+------------+------------+------------| |GAP | | | | | |-------------------| | | | | |1 | 12686.00| -10.16| 12.53| 0.07| |-------------------+------------+------------+------------+------------| |2 | 1339.00| -12.34| 8.37| -0.15| |-------------------+------------+------------+------------+------------| |3 | 1872.00| -8.28| 9.63| -0.17| ------------------------------------------------------------------------- |4 | 223.00| -6.73| 3.98| -0.24| |-------------------+------------+------------+------------+------------| |5 | 5.00| 0.04| 1.16| 0.51| ------------------------------------------------------------------------- |12 | 1.00| 16.61| 16.61| 16.61| ------------------------------------------------------------------------- ---------------------------------- | | %CHANGE | | |------------| | | STD | |-------------------+------------| |GAP | | |-------------------| | |1 | 1.10| |-------------------+------------| |2 | 1.55| |-------------------+------------| |3 | 1.01| ---------------------------------- |4 | 1.17| |-------------------+------------| |5 | 0.53| ---------------------------------- |12 | .| ---------------------------------- STANDARD STATISTICS OF THE %CHANGE, BY DAY ------------------------------------------------------------------------- | | %CHANGE | %CHANGE | %CHANGE | %CHANGE | | |------------+------------+------------+------------| | | N | MIN | MAX | MEAN | |-------------------+------------+------------+------------+------------| |DAY | | | | | |-------------------| | | | | |1 | 2931.00| -12.34| 8.14| -0.18| |-------------------+------------+------------+------------+------------| |2 | 3036.00| -10.16| 12.36| 0.03| |-------------------+------------+------------+------------+------------| |3 | 3056.00| -9.92| 16.61| 0.10| ------------------------------------------------------------------------- |4 | 3015.00| -8.88| 9.52| 0.05| |-------------------+------------+------------+------------+------------| |5 | 3004.00| -8.70| 8.29| 0.07| ------------------------------------------------------------------------- |6 | 1084.00| -5.25| 8.37| 0.15| ------------------------------------------------------------------------- ---------------------------------- | | %CHANGE | | |------------| | | STD | |-------------------+------------| |DAY | | |-------------------| | |1 | 1.20| |-------------------+------------| |2 | 1.13| |-------------------+------------| |3 | 1.23| ---------------------------------- |4 | 1.08| |-------------------+------------| |5 | 1.10| ---------------------------------- |6 | 1.06| ---------------------------------- STANDARD STATISTICS OF THE %CHANGE, BY WKEND ------------------------------------------------------------------------- | | %CHANGE | %CHANGE | %CHANGE | %CHANGE | | |------------+------------+------------+------------| | | N | MIN | MAX | MEAN | |-------------------+------------+------------+------------+------------| |WKEND | | | | | |-------------------| | | | | |0 | 12998.00| -10.16| 12.53| 0.07| |-------------------+------------+------------+------------+------------| |1 | 3128.00| -12.34| 16.61| -0.18| ------------------------------------------------------------------------- ---------------------------------- | | %CHANGE | | |------------| | | STD | |-------------------+------------| |WKEND | | |-------------------| | |0 | 1.11| |-------------------+------------| |1 | 1.26| ----------------------------------